summaryrefslogtreecommitdiff
path: root/miniany/README.html
blob: ca5cdf7bbb1bbbe5edee8efbe9559d1c2e13aa74 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
<h1>CC - a self-hosting, bootstrappable, minimal C compiler</h1>
<h2>Introduction</h2>
<p>On the never-ending quest of a minimal system I found Swieros and C4 (the C compiler in 4 functions). Inspired and intrigued I started to implement my own.</p>
<p>For abaos (a small operating system of mine, also in C) I cloned the minimal C library, so we can build a freestanding version of C4.</p>
<p>C4 serves as a test whether my own CC is minimal enough and doesn't use silly functions. Additionally C4 as well as CC are compiled both in a (on Linux) hosted version and a freestanding version. We use a series of compilers like gcc, clang, tcc and pcc to make sure that we are not using more silly C constructs.</p>
<p>In order to be able to port easily we make almost no use of system calls, the ones we need are:</p>
<ul>
<li><i>brk</i>: for malloc/free, change the start address of the heap segment of the process, if the OS only assigns a single static space, then brk results in a NOP.</li>
<li><i>exit</i>: terminate the process, return does not always work in all combinations (for instance with pcc on Linux). Can be a NOP, we don't require any trickery as <i>atexit </i>and we don't use buffering anywhere (for instance flushing stdout on exit).</li>
<li><i>read/write</i>: read from stdin linearly, write to stdout linearly, this is essentially a model using an input and an output tape. Those two functions must really exist. This basically eliminates the need for a file system which we might not have during early bootstrapping.</li>
</ul>
<p>Similarly we simplify the C language to not use certain features which can cause trouble when bootstrapping:</p>
<ul>
<li><i>variable </i><i>arguments</i>: though simple in principle (just some pointers into the stack if you use a stack for function parameters), it is not typesafe. And the only example in practice it's really heavily used for is in printf-like functions.</li>
<li><i>preprocessor</i>: it needs a filesystem, we take this outside of the compiler by feeding it an (eventually) concatenated list of *.c files. Note: in the hosted environment we (and glibc) can use as many preprocessor features as they want, they just don't have to get visible in our code.</li>
<li><i>two </i><i>types</i>: int and char, so we can interpret memory as words or as bytes.</li>
</ul>
<h2>Local version of C4</h2>
<p>The local version of C4 has the following adaoptions and extensions:</p>
<ul>
<li>switch statement from the <i>switch-and-struct</i>s branch, adapted c4 itself to use switch statements instead of if's (as in the <i>switch-and-structs </i>branch)</li>
<li>struct support from <i>switch-and-structs</i></li>
<li>constants like <i>EOF</i>, <i>EXIT_SUCCESS</i>, <i>NULL</i></li>
<li>standard C block comments along to c++ end of line ones</li>
<li>negative enum initializers</li>
<li>do/while loops</li>
<li>more C functions like <i>isspace</i>, <i>getc</i>, <i>strcmp</i></li>
<li>some simplified functions for printing like <i>putstring</i>, <i>putint</i>, <i>putnl </i>replacing printf-like functions</li>
<li>BSD-style string functions like <i>strlcpy</i>, <i>strlcat</i></li>
<li>strict C89 conformance, mainly use standard comment blocks, also removed some warnings</li>
<li>some casts around malloc and memset to fit to non-void freestanding-libc</li>
<li>converted <i>print</i>f to <i>putstring/putint/putnl </i>and some helper functions for error reporting like error()</li>
<li>removed all memory leaks</li>
<li>de-POSIX-ified, no <i>open/read/close</i>, use <i>getchar </i>from stdin only (don't assume the existence of a file system), this also means we had to create sort of an old style tape-file with FS markers to separate the files piped to c4.</li>
</ul>
<p>The reason for all those adaptions is to minimize the dependency on the host system and to be able to use <i>libc-freestanding.c</i>.</p>
<p>Note: Only too late I discovered that there was a C5 version of the same compiler, which would maybe have served better as a basis.</p>
<h2>Examples</h2>
<h3>Running on the host system using the hosts C compiler</h3>
<p>Compiled in either hosted (host libc) or freestanding (our own libc, currently IA-32 Linux kernel only syscalls):</p>
<pre><code>./build.sh cc hostcc hosted d
./build.sh cc hostcc freestanding d
./cc &lt; test1.c &gt; test1.asm
</code></pre>
<p>Create a plain binary from the assembly code:</p>
<pre><code>fasm test1.asm test1.bin
</code></pre>
<p>Disassemble it to verify it's correctness:</p>
<pre><code>ndisasm -b32 -o1000000h -a test1.bin
</code></pre>
<p>You can choose <i>gcc</i>, <i>clang</i>, <i>tcc </i>or <i>pcc </i>as host compiler (<i>hostcc</i>).</p>
<h3>Running on the host in the C4 interpreter</h3>
<p>Running in C4 interpreter, again, the C4 program can be compiled in hosted or freestanding mode:</p>
<pre><code>./build.sh c4 hostcc hosted d
./build.sh c4 hostcc freestanding d
</code></pre>
<p>Here again you can choose the host compiler for compiling C4.</p>
<p>Then we have to create the standard input for C4 using:</p>
<pre><code>echo -n -e &quot;\034&quot; &gt; EOF
cat cc.c EOF hello.c | ./c4
cat c4.c EOF cc.c EOF hello.c | ./c4
cat c4.c4 EOF c4.c EOF cc.c EOF hello.c | ./c4
</code></pre>
<p>EOF contains the traditional FS (file separator) character in the ASCII character set. Every time c4 is invoked it reads exacly one input file up to the first FS character (or stops at the end of stdin).</p>
<p>We can also use <i>-s</i>, or <i>-d </i>on every level as follows:</p>
<pre><code>cat cc.c EOF hello.c | ./c4 -d
</code></pre>
<h2>Features and Requirements</h2>
<p>We have to careful what to put in a bootstrapping compiler, there is a tradeoff between</p>
<ul>
<li>costs to implement it in the language: code complexity and size must be kept at a minimun</li>
<li>required features from the C runtime: for instance support of the whole Unicde characters</li>
<li>required features from the operating system: for instance the requirement for a POSIX layer</li>
<li>ugliness of the resulting code: it doesn't make sense to omit features which are somewhat hard to implement but can render readability of the code much better: best examples are the <i>switch/case </i>vs. <i>if/else </i>and the funny array indexing vs. structures in C4.</li>
</ul>
<p>So we collect some ideas here about features we add or do not add and why. We also collect here what their implications are, when we are implementing them.</p>
<p>We also have to be careful what C4 can do for us and either add it there (but only if small enough) in order no to loose this test case.</p>
<h3>Preprocessor for modularisation</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li><i>#include </i><i>&lt;filename.h&gt;</i>: requires a filename and thus an operating system with a filesystem early on, needs directory functions and open/close/read/write on files</li>
<li>C4 ignores all prepocessor commands and treats them as comments till the end of line</li>
</ul>
<p>Alternative:</p>
<ul>
<li>have a <i>cat </i><i>*.c </i>concatenating all the source code</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>we could have sort of a special tape filesystem with a directory at the beginning and offset to allow the preprocessor to find the files for the early destination OS</li>
<li>there is no reason not to use the preprocessor on the host</li>
</ul>
<h3>Preprocessor for conditional compilation</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>especially simple #ifdefs would be simple to do, conditional compilation in the same file is questionable, there is nothing wrong with including files like <i>linux_386.c</i>, <i>linux_x86_64.c</i></li>
<li>they are not part of C, they are an addon (though they are nowadays sometimes treated as if they were part of C)</li>
<li>C4 ignores all prepocessor commands and treats them as comments till the end of line</li>
</ul>
<p>Alternative:</p>
<ul>
<li>Use special files and concatenate them at the right place, e. g. <i>_start-stub.c </i>for tcc</li>
</ul>
<h3>Preprocessor for constant declarations</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>enum constants serve the same purpose, we prefer those</li>
<li>even C89 can work with enum constants</li>
</ul>
<p>Caveats:</p>
<ul>
<li>C4 has only positive enums, we need negative ones for <i>EOF</i></li>
<li>enum constants are signed integer, so we have to be careful when using them for chars</li>
<li>we don't use enum for enumeration types</li>
</ul>
<h3>Variable Initializers</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>initializers of global and locals, not that important as we use C89 anyway, forcing us to separate declaration and usage of variables per scope</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>for example <i>symname[t]: </i>printing the symbol and not the number, requires static initializers for array of char* to make the code look nice. Of course we can also just initialize it in a piece of code.</li>
</ul>
<h3>Inline Assembly</h3>
<p>Implementation status: yes</p>
<p>Reasoning:</p>
<ul>
<li>somehow we have to communicate to the operating system, it's either inline assembly or external linking to assembly code</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>C4 has no inline assembly, we must add it there too</li>
</ul>
<p>Alternative:</p>
<ul>
<li>we could implement a linker and object format early on, we could use an external assembler as for instance <i>asm-i386 </i>(which understands a subset of <i>fasm</i>)</li>
<li>we could implement an early dump format for symbols (function signatures mainly) and use them during linking</li>
</ul>
<p>Some general notes:</p>
<p>GNU inline asm statement has become the de-facto standard (which is too complicated IMHO): I would require sort of a <i>.byte </i><i>0xX</i>X instruction only, for readablility maybe simple fasm-like syntax. We must be careful that our invention of an inline assembler can be mapped somehow to the GNU inline asm version, so that we can use that one on the host with gcc/clang/tcc/pcc..</p>
<p>c.c in swieros (the c4 successor) has <i>asm(NOP)</i>, this is something we could implement easily. u.h contains an enum with opcodes (most likely doable or an easy architecture like the one in swieros, I doubt this works for Intel opcodes, but we should check if it works for our simplified Intel opcode subset).</p>
<p>There should though be only one single point of information for opcodes per architecture, so asm gets sort of an inline string generator for the assembly output. Or we share a common C-file with enums for the opcodes and cat it to both the assembler and the compiler during the build (should not result in increaed code size, as those are enums).</p>
<p>The asm(x) or asm(x,y) constructs can be mapped on the host compilers to asm __volatile__ .byte ugliness. In cc and c4 we can take the swieros approach. This should give us nice lowlevel inline assembly in a really simplified way (basically embedding bytes).</p>
<p>Not having inline assembly means you need compilation units written and linked to the program in assembly, which - well - adds a linker and calling conventions, which might be too early in bootstrapping.</p>
<h3>Object formats and linkers</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>requires file systems and linker formats like ELF</li>
</ul>
<p>Alternative:</p>
<ul>
<li>make compilation units from a bunch of source code, this results in bigger binaries, as we cannot share too much. This might be acceptable for early bootstrapping. Later on it would be nice to add a linker as an optional addon (which can be used outside of bootstrapping).</li>
</ul>
<h3>Forward declarations of function prototypes</h3>
<p>Implementation status: yes (<b>TODO</b>)</p>
<p>Reasoning:</p>
<ul>
<li>Recursive descent parsing requires forward function declarations.</li>
</ul>
<p>Caveats:</p>
<ul>
<li>Forward function declarations are not that easy to implement, because you have to generate a placeholder for the call address before you get the whole definition of the forwarded function (especially its entry address). Or we have to create sort of a temporary jump into a jump table (sort of a GOT) which we patch when we know the address of the implementation of the function. Either way we loose the 1-pass output of the generated code. Having a global table at one place scales easier, as we don't have to keep the whole generated code around just for patching (remember, we have tapes and memory, no seek of files).</li>
<li>Must be added to c4 too, might not be that easy (<b>TODO</b>)</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>We could write a non-recursive parser using tables</li>
</ul>
<h3>Functions with variable arguments</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>not type-safe</li>
<li>only used for (s)printfs string manipulation and output functions</li>
<li>C4 doesn't implement variable arguments for defining functions, so we cannot bootstrap a freestanding version</li>
</ul>
<p>Requirements</p>
<ul>
<li>on the hosted Linux envorinment we need syscalls to <i>syscal</i>, <i>int </i><i>0x80</i>, etc.</li>
<li>we need inline assembly to create to syscalls</li>
</ul>
<p>Alternative:</p>
<ul>
<li>create simple functions like <i>putchar</i>, <i>putint</i>, <i>putstring </i>(one per basic type) and some helpers like <i>putnl</i></li>
<li>use <i>stlcat </i>and <i>stlcpy </i>should be enought to compose label names</li>
<li>instead of a generic <i>syscall(..) </i>we can easily work around this by having <i>syscall1, </i><i>syscall2, </i><i>syscall3, </i><i>... </i>with a fixed number of parameters</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>we deviate from the C standard, printf just belongs to C</li>
<li>printf is actually not hard to implement in a type-unsafe-way</li>
<li>syscalls are variable arguments</li>
</ul>
<h3>FILE* and stderr</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>requires FILE * structures, requires various write channels from the operating system</li>
<li>we can write error messages into the output stream as comment <i>; </i><i>ERROR </i><i>in </i><i>line </i><i>32, </i><i>pos </i><i>2: </i><i>generic </i><i>error </i>(<b>TODO</b>)</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>unbuffered FILE * is simple to implement</li>
<li>almost all operating systems have the notion of stdout and stderr, we can set them to the same channel if an OS doesn't have them.</li>
</ul>
<h3>Typedefs</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>typedefs are syntactic sugar for <i>typedef </i><i>struct </i><i>T </i>as T, not strictly necessary and they don't make our code look too ugly.</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>portability without preprocessor could make use of <i>typedef </i><i>ulong64 </i><i>unsigned </i><i>long </i><i>int </i>and similar constructs</li>
</ul>
<h3>For-loops</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>unless we start optimizing (SIMD) there is no real benefit, for a generic 'for', a strict for i=0 to N, i++ is easier to optimize, when you have a grammatical construct to help recognizing it.</li>
<li>the C for loop has a funny ending semicolon issue and you can add arbitraty code blocks into the three parts of the for, this is way too generic for a minimalistic language</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>for loops are not hard to implement</li>
</ul>
<h3>Passing arguments to main</h3>
<p>Implementation status: yes</p>
<p>Reasoning:</p>
<ul>
<li>this is just passing an <i>in</i>t (<i>argc</i>) and a <i>char** </i>(argv) from _start to main. This is also a convention shared with the operating system when it calls our process. Without that we would have a hard time to parametrize the calls to our process both on the host and in the target OS.</li>
</ul>
<p>Counter argument:</p>
<ul>
<li>Do really everything over the input stream, but this would feel a little bit too mainframe-ish.</li>
</ul>
<h3>bool</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>C89 has no boolean</li>
<li>useful, but not strigtly necessary, we can live with <i>int </i>holding a boolean value</li>
</ul>
<h3>Union</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>in the compiler there is little benefit of compressing parts of e. g. ASTs into unions</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>for bootstrapping the operating system we might need unions (as well as packing and alignment)</li>
</ul>
<h3>Dangling else</h3>
<p>Implementation status: no</p>
<p>Reasoning:</p>
<ul>
<li>Don't allow non-blocked if/else, just avoid then dangling else problem, it's uninteresting and in C not as bad as in PASCAL-like languages, as the block markers are just one character big.</li>
</ul>
<h3>Register Allocation</h3>
<p>Implementation status: yes</p>
<p>Reasoning:</p>
<ul>
<li>Compared to a stack machine even the simplest register allocation algorithm produces much better code</li>
</ul>
<h3>Abstract Syntax Trees</h3>
<p>Implementation status: yes</p>
<p>Reasoning:</p>
<ul>
<li>Delaying code generation is essential when doing an assignment (rvalue must be evaluated before lvalue), in const folding (do no generate code as you don't know the expression is a constant but instead just compute the current value of the constant expression).</li>
<li>Separate semantic operation <i>array </i><i>index </i><i>evaluation </i>from <i>definition </i><i>of </i><i>the </i><i>size </i><i>of </i><i>an </i><i>array </i>for instance with the '[' character (we do not want to react on the scanner symbol directly).</li>
<li>When evaluating a boolean expression we don't know yet it's context (can be in a <i>if/while </i>condition, in which case we would geneate a conditional jump or whether in an assignment where we would store the value to the boolean variable).</li>
</ul>
<p>Caveats:</p>
<ul>
<li>Try to keep the scope of an AST as small as possible and as big as necessary (the output of the parser should not be the complete source code). Minimal is an expression and some context, maybe maximal is the scope of a function.</li>
</ul>
<p>Counter arguments:</p>
<ul>
<li>Readable AST code needs some trees and pointers to structs, but we want those anyway for better readable code (C4 for instance is not that readable in it's original array indexing version).</li>
</ul>
<h3>Builtin functions</h3>
<p>Implementation status: yes</p>
<p>Reasoning:</p>
<ul>
<li>Adding <i>putint/getchar </i>style of functions as elements of the language is tempting, as it allows early debugging and testing (as in PASCAL). The fundemental conflict here is betweet Bootstrapping is better with stdout and stdin in the language (no function calls, no linker, etc. needed). But later on we want those functions be part of a language library and not of the langyage itself.</li>
</ul>
<p>Caveats:</p>
<ul>
<li>Avoid code duplication (inline assembly in the compiler for the keyword implementation and with intline assembly in the language library). (<b>TODO</b>)</li>
</ul>
<h2>References</h2>
<p>Compiler construction in general:</p>
<ul>
<li><i>&quot;Compiler </i><i>Construction&quot;</i>&quot;, Niklaus Wirth</li>
<li><a href="https://github.com/DoctorWkt/acwj">https://github.com/DoctorWkt/acwj</a>: a nice series on building a C compiler, step by step with lots of good explanations</li>
<li><a href="https://github.com/lotabout/write-a-C-interpreter/blob/master/tutorial/en/">https://github.com/lotabout/write-a-C-interpreter/blob/master/tutorial/en/</a>, tutorial based on C4 how to build a C interpreter, explains nicely details in C4.</li>
</ul>
<p>Some special compiler building topics:</p>
<ul>
<li><a href="https://www.engr.mun.ca/%7Etheo/Misc/exp">https://www.engr.mun.ca/~theo/Misc/exp</a>_parsing.htm#climbing, <a href="https://en.wikipedia.org/wiki/Operator-precedence">https://en.wikipedia.org/wiki/Operator-precedence</a>_parser#Precedence_climbing_method</li>
<li><a href="https://en.wikipedia.org/wiki/Strahler">https://en.wikipedia.org/wiki/Strahler</a>_number: justification for register numbers for register alloation (<b>TODO: </b><b>clarify</b>)</li>
</ul>
<p>C4:</p>
<ul>
<li><a href="https://github.com/rswier/c4.git">https://github.com/rswier/c4.git</a>, <i>C4 </i><i>- </i><i>C </i><i>in </i><i>four </i><i>functions</i>, Robert Swierczek, minimalistic C compiler running on an emulator on the IR, inspiration for this project</li>
<li><a href="https://github.com/rswier/c4/blob/switch-and-structs/c4.c">https://github.com/rswier/c4/blob/switch-and-structs/c4.c</a>, c4 adaptions to provide switch and structs</li>
<li><a href="https://github.com/EarlGray/c4">https://github.com/EarlGray/c4</a>: a X86 JIT version of c4</li>
<li><a href="https://github.com/jserv/amacc">https://github.com/jserv/amacc</a>: based on C4, JIT or native code, for ARM, quite well documented, also very nice list of compiler resources on Github page</li>
</ul>
<p>Other minimal compilers and systems:</p>
<ul>
<li><a href="http://selfie.cs.uni-salzburg.at/">http://selfie.cs.uni-salzburg.at/</a>: C* self-hosting C compiler (also emulator, hypervisor) for RISCV, inspiration for what makes up a minimal C language</li>
<li><a href="http://www.iro.umontreal.ca/%7Efelipe/IFT2030-Automne2002/Complements/tinyc.c">http://www.iro.umontreal.ca/~felipe/IFT2030-Automne2002/Complements/tinyc.c</a>, Marc Feeley, really easy and much more readable, meant as educational compiler</li>
<li><a href="https://github.com/rswier/swieros.git">https://github.com/rswier/swieros.git</a>: c.c in swieros, Robert Swierczek</li>
</ul>
<p>Assembly:</p>
<ul>
<li><a href="https://github.com/felipensp/assembly/blob/master/x86/itoa.s">https://github.com/felipensp/assembly/blob/master/x86/itoa.s</a>, for putint (early debugging keyword)</li>
<li><a href="https://baptiste-wicht.com/posts/2011/11/print-strings-integers-intel-assembly.htm">https://baptiste-wicht.com/posts/2011/11/print-strings-integers-intel-assembly.htm</a> (earldy debugging keyword)</li>
</ul>
<p>Documentation:</p>
<ul>
<li><a href="http://cowlark.com/wordgrinder/index.html">http://cowlark.com/wordgrinder/index.html</a>: the fabulous editor which just does what it should do</li>
<li><a href="https://github.com/mity/md4c">https://github.com/mity/md4c</a>: markdown to HTML in C</li>
</ul>