path: root/miniany/cc.wg

WordGrinder dumpfile v3: this is a text file; diff me!
.addons.autosave.enabled: false
.addons.autosave.pattern: "%F.autosave.%T.wg"
.addons.autosave.period: 10
.addons.htmlexport.bold_off: "</b>"
.addons.htmlexport.bold_on: "<b>"
.addons.htmlexport.italic_off: "</i>"
.addons.htmlexport.italic_on: "<i>"
.addons.htmlexport.underline_off: "</u>"
.addons.htmlexport.underline_on: "<u>"
.addons.pagecount.enabled: false
.addons.pagecount.wordsperpage: 250
.addons.scrapbook.document: "Scrapbook"
.addons.scrapbook.pattern: "Item from '%N' at %T:"
.addons.scrapbook.timestamp: true
.addons.smartquotes.doublequotes: false
.addons.smartquotes.leftdouble: "“"
.addons.smartquotes.leftsingle: "‘"
.addons.smartquotes.notinraw: true
.addons.smartquotes.rightdouble: "”"
.addons.smartquotes.rightsingle: "’"
.addons.smartquotes.singlequotes: false
.addons.spellchecker.enabled: false
.addons.spellchecker.usesystemdictionary: true
.addons.spellchecker.useuserdictionary: true
.clipboard.co: 1
.clipboard.cp: 1
.clipboard.cw: 1
.clipboard.margin: 0
.clipboard.viewmode: 1
.clipboard.wordcount: 6
.documents.1.co: 7
.documents.1.cp: 31
.documents.1.cw: 25
.documents.1.margin: 0
.documents.1.name: "main"
.documents.1.sticky_selection: false
.documents.1.viewmode: 1
.documents.1.wordcount: 2863
.fileformat: 8
.findtext: "Note"
.menu.accelerators.^@: "ZM"
.menu.accelerators.^B: "SB"
.menu.accelerators.BACKSPACE: "ZDPC"
.menu.accelerators.^C: "EC"
.menu.accelerators.DELETE: "ZDNC"
.menu.accelerators.^DOWN: "ZNP"
.menu.accelerators.DOWN: "ZD"
.menu.accelerators.^E: "ZDW"
.menu.accelerators.EC: "^C"
.menu.accelerators.ECadd: "^M"
.menu.accelerators.ECfind: "^L"
.menu.accelerators.EF: "^F"
.menu.accelerators.EG: "^G"
.menu.accelerators.EN: "^K"
.menu.accelerators.END: "ZE"
.menu.accelerators.EP: "^V"
.menu.accelerators.ER: "^R"
.menu.accelerators.Eredo: "^Y"
.menu.accelerators.ET: "^X"
.menu.accelerators.Eundo: "^Z"
.menu.accelerators.^F: "EF"
.menu.accelerators.FQ: "^Q"
.menu.accelerators.FS: "^S"
.menu.accelerators.^G: "EG"
.menu.accelerators.HOME: "ZH"
.menu.accelerators.^I: "SI"
.menu.accelerators.^K: "EN"
.menu.accelerators.^L: "ECfind"
.menu.accelerators.^LEFT: "ZWL"
.menu.accelerators.LEFT: "ZL"
.menu.accelerators.^M: "ECadd"
.menu.accelerators.^O: "SO"
.menu.accelerators.^P: "SP"
.menu.accelerators.^PGDN: "ZED"
.menu.accelerators.PGDN: "ZPGDN"
.menu.accelerators.^PGUP: "ZBD"
.menu.accelerators.PGUP: "ZPGUP"
.menu.accelerators.^Q: "FQ"
.menu.accelerators.^R: "ER"
.menu.accelerators.^RIGHT: "ZWR"
.menu.accelerators.RIGHT: "ZR"
.menu.accelerators.^S: "FS"
.menu.accelerators.SB: "^B"
.menu.accelerators.S^DOWN: "ZSNP"
.menu.accelerators.SDOWN: "ZSD"
.menu.accelerators.SEND: "ZSE"
.menu.accelerators.SHOME: "ZSH"
.menu.accelerators.SI: "^I"
.menu.accelerators.S^LEFT: "ZSWL"
.menu.accelerators.SLEFT: "ZSL"
.menu.accelerators.SO: "^O"
.menu.accelerators.SP: "^P"
.menu.accelerators.S^PGDN: "ZSED"
.menu.accelerators.SPGDN: "ZSPGDN"
.menu.accelerators.S^PGUP: "ZSBD"
.menu.accelerators.SPGUP: "ZSPGUP"
.menu.accelerators.S^RIGHT: "ZSWR"
.menu.accelerators.SRIGHT: "ZSR"
.menu.accelerators.SU: "^U"
.menu.accelerators.S^UP: "ZSPP"
.menu.accelerators.SUP: "ZSU"
.menu.accelerators.^U: "SU"
.menu.accelerators.^UP: "ZPP"
.menu.accelerators.UP: "ZU"
.menu.accelerators.^V: "EP"
.menu.accelerators.^W: "ZSW"
.menu.accelerators.^X: "ET"
.menu.accelerators.^Y: "Eredo"
.menu.accelerators.^Z: "Eundo"
.menu.accelerators.ZBD: "^PGUP"
.menu.accelerators.ZD: "DOWN"
.menu.accelerators.ZDNC: "DELETE"
.menu.accelerators.ZDPC: "BACKSPACE"
.menu.accelerators.ZDW: "^E"
.menu.accelerators.ZE: "END"
.menu.accelerators.ZED: "^PGDN"
.menu.accelerators.ZH: "HOME"
.menu.accelerators.ZL: "LEFT"
.menu.accelerators.ZM: "^@"
.menu.accelerators.ZNP: "^DOWN"
.menu.accelerators.ZPGDN: "PGDN"
.menu.accelerators.ZPGUP: "PGUP"
.menu.accelerators.ZPP: "^UP"
.menu.accelerators.ZR: "RIGHT"
.menu.accelerators.ZSBD: "S^PGUP"
.menu.accelerators.ZSD: "SDOWN"
.menu.accelerators.ZSE: "SEND"
.menu.accelerators.ZSED: "S^PGDN"
.menu.accelerators.ZSH: "SHOME"
.menu.accelerators.ZSL: "SLEFT"
.menu.accelerators.ZSNP: "S^DOWN"
.menu.accelerators.ZSPGDN: "SPGDN"
.menu.accelerators.ZSPGUP: "SPGUP"
.menu.accelerators.ZSPP: "S^UP"
.menu.accelerators.ZSR: "SRIGHT"
.menu.accelerators.ZSU: "SUP"
.menu.accelerators.ZSW: "^W"
.menu.accelerators.ZSWL: "S^LEFT"
.menu.accelerators.ZSWR: "S^RIGHT"
.menu.accelerators.ZU: "UP"
.menu.accelerators.ZWL: "^LEFT"
.menu.accelerators.ZWR: "^RIGHT"
.name: "/media/sd/abaumann/projects/compilertests/miniany/cc.wg"
.replacetext: ""
.statusbar: true
.current: 1
#clipboard
P TODO: add rest of REQUIREMENTS here!
.
#1
H1 CC - a self-hosting, bootstrappable, minimal C compiler
H2 Introduction
P On the never-ending quest of a minimal system I found Swieros and C4 (the C compiler in 4 functions). Inspired and intrigued I started to implement my own.
P For abaos (a small operating system of mine, also in C) I cloned the minimal C library, so we can build a freestanding version of C4.
P C4 serves as a test whether my own CC is minimal enough and doesn't use silly functions. Additionally C4 as well as CC are compiled both in a (on Linux) hosted version and a freestanding version. We use a series of compilers like gcc, clang, tcc and pcc to make sure that we are not using more silly C constructs.
P In order to be able to port easily we make almost no use of system calls, the ones we need are:
LB brk: for malloc/free, change the start address of the heap segment of the process, if the OS only assigns a single static space, then brk results in a NOP.
LB exit: terminate the process, return does not always work in all combinations (for instance with pcc on Linux). Can be a NOP, we don't require any trickery as atexit and we don't use buffering anywhere (for instance flushing stdout on exit).
LB read/write: read from stdin linearly, write to stdout linearly, this is essentially a model using an input and an output tape. Those two functions must really exist. This basically eliminates the need for a file system which we might not have during early bootstrapping.
P Similarly we simplify the C language to not use certain features which can cause trouble when bootstrapping:
LB variable arguments: though simple in principle (just some pointers into the stack if you use a stack for function parameters), it is not typesafe. And the only example in practice it's really heavily used for is in printf-like functions.
LB preprocessor: it needs a filesystem, we take this outside of the compiler by feeding it an (eventually) concatenated list of *.c files. Note: in the hosted environment we (and glibc) can use as many preprocessor features as they want, they just don't have to get visible in our code.
LB two types: int and char, so we can interpret memory as words or as bytes.
H2 Local version of C4
P The local version of C4 has the following adaoptions and extensions:
LB switch statement from the switch-and-structs branch, adapted c4 itself to use switch statements instead of if's (as in the switch-and-structs branch)
LB struct support from switch-and-structs
LB constants like EOF, EXIT_SUCCESS, NULL
LB standard C block comments along to c++ end of line ones
LB negative enum initializers
LB do/while loops
LB more C functions like isspace, getc, strcmp
LB some simplified functions for printing like putstring, putint, putnl replacing printf-like functions
LB BSD-style string functions like strlcpy, strlcat
LB strict C89 conformance, mainly use standard comment blocks, also removed some warnings
LB some casts around malloc and memset to fit to non-void freestanding-libc
LB converted printf to putstring/putint/putnl and some helper functions for error reporting like error()
LB removed all memory leaks
LB de-POSIX-ified, no open/read/close, use getchar from stdin only (don't assume the existence of a file system), this also means we had to create sort of an old style tape-file with FS markers to separate the files piped to c4.
P The reason for all those adaptions is to minimize the dependency on the host system and to be able to use libc-freestanding.c.
P Note: Only too late I discovered that there was a C5 version of the same compiler, which would maybe have served better as a basis.
H2 Examples
H3 Running on the host system using the hosts C compiler
P Compiled in either hosted (host libc) or freestanding (our own libc, currently IA-32 Linux kernel only syscalls):
PRE ./build.sh cc hostcc hosted d
PRE ./build.sh cc hostcc freestanding d
PRE ./cc < test1.c > test1.asm
P Create a plain binary from the assembly code:
PRE fasm test1.asm test1.bin
P Disassemble it to verify it's correctness:
PRE ndisasm -b32 -o1000000h -a test1.bin
P You can choose gcc, clang, tcc or pcc as host compiler (hostcc).
H3 Running on the host in the C4 interpreter
P Running in C4 interpreter, again, the C4 program can be compiled in hosted or freestanding mode:
PRE ./build.sh c4 hostcc hosted d
PRE ./build.sh c4 hostcc freestanding d
P Here again you can choose the host compiler for compiling C4.
P Then we have to create the standard input for C4 using:
PRE echo -n -e "\034" > EOF
PRE cat cc.c EOF hello.c | ./c4
PRE cat c4.c EOF cc.c EOF hello.c | ./c4
PRE cat c4.c4 EOF c4.c EOF cc.c EOF hello.c | ./c4
P EOF contains the traditional FS (file separator) character in the ASCII character set. Every time c4 is invoked it reads exacly one input file up to the first FS character (or stops at the end of stdin).
P We can also use -s, or -d on every level as follows:
PRE cat cc.c EOF hello.c | ./c4 -d
H2 Features and Requirements
P We have to careful what to put in a bootstrapping compiler, there is a tradeoff between
LB costs to implement it in the language: code complexity and size must be kept at a minimun
LB required features from the C runtime: for instance support of the whole Unicde characters
LB required features from the operating system: for instance the requirement for a POSIX layer
LB ugliness of the resulting code: it doesn't make sense to omit features which are somewhat hard to implement but can render readability of the code much better: best examples are the switch/case vs. if/else and the funny array indexing vs. structures in C4.
P So we collect some ideas here about features we add or do not add and why. We also collect here what their implications are, when we are implementing them.
P We also have to be careful what C4 can do for us and either add it there (but only if small enough) in order no to loose this test case.
H3 Preprocessor for modularisation
P Implementation status: no
P Reasoning:
LB #include <filename.h>: requires a filename and thus an operating system with a filesystem early on, needs directory functions and open/close/read/write on files
LB C4 ignores all prepocessor commands and treats them as comments till the end of line
P Alternative:
LB have a cat *.c concatenating all the source code
P Counter arguments:
LB we could have sort of a special tape filesystem with a directory at the beginning and offset to allow the preprocessor to find the files for the early destination OS
LB there is no reason not to use the preprocessor on the host
H3 Preprocessor for conditional compilation
P Implementation status: no
P Reasoning:
LB especially simple #ifdefs would be simple to do, conditional compilation in the same file is questionable, there is nothing wrong with including files like linux_386.c, linux_x86_64.c
LB they are not part of C, they are an addon (though they are nowadays sometimes treated as if they were part of C)
LB C4 ignores all prepocessor commands and treats them as comments till the end of line
P Alternative:
LB Use special files and concatenate them at the right place, e. g. _start-stub.c for tcc
H3 Preprocessor for constant declarations
P Implementation status: no
P Reasoning:
LB enum constants serve the same purpose, we prefer those
LB even C89 can work with enum constants
P Caveats:
LB C4 has only positive enums, we need negative ones for EOF
LB enum constants are signed integer, so we have to be careful when using them for chars
LB we don't use enum for enumeration types
H3 Variable Initializers
P Implementation status: no
P Reasoning:
P - initializers of global and locals, not that important as we use C89 anyway, forcing us to separate declaration and usage of variables per scope
P Counter arguments:
P - for example symname[t]: printing the symbol and not the number, requires static initializers for array of char* to make the code look nice. Of course we can also just initialize it in a piece of code.
H3 Inline Assembly
P Implementation status: yes
P Reasoning:
LB somehow we have to communicate to the operating system, it's either inline assembly or external linking to assembly code
P Counter arguments:
LB C4 has no inline assembly, we must add it there too
P Alternative:
LB we could implement a linker and object format early on, we could use an external assembler as for instance asm-i386 (which understands a subset of fasm)
LB we could implement an early dump format for symbols (function signatures mainly) and use them during linking
P Some general notes:
P GNU inline asm statement has become the de-facto standard (which is too complicated IMHO): I would require sort of a .byte 0xXX instruction only, for readablility maybe simple fasm-like syntax. We must be careful that our invention of an inline assembler can be mapped somehow to the GNU inline asm version, so that we can use that one on the host with gcc/clang/tcc/pcc..
P c.c in swieros (the c4 successor) has asm(NOP), this is something we could implement easily. u.h contains an enum with opcodes (most likely doable or an easy architecture like the one in swieros, I doubt this works for Intel opcodes, but we should check if it works for our simplified Intel opcode subset).
P There should though be only one single point of information for opcodes per architecture, so asm gets sort of an inline string generator for the assembly output. Or we share a common C-file with enums for the opcodes and cat it to both the assembler and the compiler during the build (should not result in increaed code size, as those are enums).
P The asm(x) or asm(x,y) constructs can be mapped on the host compilers to asm __volatile__ .byte ugliness. In cc and c4 we can take the swieros approach. This should give us nice lowlevel inline assembly in a really simplified way (basically embedding bytes).
P Not having inline assembly means you need compilation units written and linked to the program in assembly, which - well - adds a linker and calling conventions, which might be too early in bootstrapping.
H3 Object formats and linkers
P Implementation status: no
P Reasoning:
LB requires file systems and linker formats like ELF
P Alternative:
LB make compilation units from a bunch of source code, this results in bigger binaries, as we cannot share too much. This might be acceptable for early bootstrapping. Later on it would be nice to add a linker as an optional addon (which can be used outside of bootstrapping).
H3 Forward declarations of function prototypes
P Implementation status: yes (TODO)
P Reasoning:
LB Recursive descent parsing requires forward function declarations.
P Caveats:
LB Forward function declarations are not that easy to implement, because you have to generate a placeholder for the call address before you get the whole definition of the forwarded function (especially its entry address). Or we have to create sort of a temporary jump into a jump table (sort of a GOT) which we patch when we know the address of the implementation of the function. Either way we loose the 1-pass output of the generated code. Having a global table at one place scales easier, as we don't have to keep the whole generated code around just for patching (remember, we have tapes and memory, no seek of files).
LB Must be added to c4 too, might not be that easy (TODO)
P Counter arguments:
LB We could write a non-recursive parser using tables
H3 Functions with variable arguments
P Implementation status: no
P Reasoning:
LB not type-safe
LB only used for (s)printfs string manipulation and output functions
LB C4 doesn't implement variable arguments for defining functions, so we cannot bootstrap a freestanding version
P Requirements
LB on the hosted Linux envorinment we need syscalls to syscal, int 0x80, etc.
LB we need inline assembly to create to syscalls
P Alternative:
LB create simple functions like putchar, putint, putstring (one per basic type) and some helpers like putnl
LB use stlcat and stlcpy should be enought to compose label names
LB instead of a generic syscall(..) we can easily work around this by having syscall1, syscall2, syscall3, ... with a fixed number of parameters
P Counter arguments:
LB we deviate from the C standard, printf just belongs to C
LB printf is actually not hard to implement in a type-unsafe-way
LB syscalls are variable arguments
H3 FILE* and stderr
P Implementation status: no
P Reasoning:
LB requires FILE * structures, requires various write channels from the operating system
LB we can write error messages into the output stream as comment ; ERROR in line 32, pos 2: generic error (TODO)
P Counter arguments:
LB unbuffered FILE * is simple to implement
LB almost all operating systems have the notion of stdout and stderr, we can set them to the same channel if an OS doesn't have them.
H3 Typedefs
P Implementation status: no
P Reasoning:
LB typedefs are syntactic sugar for typedef struct T as T, not strictly necessary and they don't make our code look too ugly.
P Counter arguments:
LB portability without preprocessor could make use of typedef ulong64 unsigned long int and similar constructs
H3 For-loops
P Implementation status: no
P Reasoning:
LB unless we start optimizing (SIMD) there is no real benefit, for a generic 'for', a strict for i=0 to N, i++ is easier to optimize, when you have a grammatical construct to help recognizing it.
LB the C for loop has a funny ending semicolon issue and you can add arbitraty code blocks into the three parts of the for, this is way too generic for a minimalistic language
P Counter arguments:
LB for loops are not hard to implement
H3 Passing arguments to main
P Implementation status: yes
P Reasoning:
LB this is just passing an int (argc) and a char** (argv) from _start to main. This is also a convention shared with the operating system when it calls our process. Without that we would have a hard time to parametrize the calls to our process both on the host and in the target OS.
P Counter argument:
LB Do really everything over the input stream, but this would feel a little bit too mainframe-ish.
H3 bool
P Implementation status: no
P Reasoning:
LB C89 has no boolean
LB useful, but not strigtly necessary, we can live with int holding a boolean value
H3 Union
P Implementation status: no
P Reasoning:
LB in the compiler there is little benefit of compressing parts of e. g. ASTs into unions
P Counter arguments:
LB for bootstrapping the operating system we might need unions (as well as packing and alignment)
H3 Dangling else
P Implementation status: no
P Reasoning:
LB Don't allow non-blocked if/else, just avoid then dangling else problem, it's uninteresting and in C not as bad as in PASCAL-like languages, as the block markers are just one character big.
H3 Register Allocation
P Implementation status: yes
P Reasoning:
LB Compared to a stack machine even the simplest register allocation algorithm produces much better code
H3 Abstract Syntax Trees
P Implementation status: yes
P Reasoning:
LB Delaying code generation is essential when doing an assignment (rvalue must be evaluated before lvalue), in const folding (do no generate code as you don't know the expression is a constant but instead just compute the current value of the constant expression).
LB Separate semantic operation array index evaluation from definition of the size of an array for instance with the '[' character (we do not want to react on the scanner symbol directly).
LB When evaluating a boolean expression we don't know yet it's context (can be in a if/while condition, in which case we would geneate a conditional jump or whether in an assignment where we would store the value to the boolean variable).
P Caveats:
LB Try to keep the scope of an AST as small as possible and as big as necessary (the output of the parser should not be the complete source code). Minimal is an expression and some context, maybe maximal is the scope of a function.
P Counter arguments:
LB Readable AST code needs some trees and pointers to structs, but we want those anyway for better readable code (C4 for instance is not that readable in it's original array indexing version).
H3 Builtin functions
P Implementation status: yes
P Reasoning:
LB Adding putint/getchar style of functions as elements of the language is tempting, as it allows early debugging and testing (as in PASCAL). The fundemental conflict here is betweet Bootstrapping is better with stdout and stdin in the language (no function calls, no linker, etc. needed). But later on we want those functions be part of a language library and not of the langyage itself.
P Caveats:
LB Avoid code duplication (inline assembly in the compiler for the keyword implementation and with intline assembly in the language library). (TODO)
H2 References
P Compiler construction in general:
LB "Compiler Construction"", Niklaus Wirth
LB https://github.com/DoctorWkt/acwj: a nice series on building a C compiler, step by step with lots of good explanations
LB https://github.com/lotabout/write-a-C-interpreter/blob/master/tutorial/en/, tutorial based on C4 how to build a C interpreter, explains nicely details in C4.
P Some special compiler building topics:
LB https://www.engr.mun.ca/~theo/Misc/exp_parsing.htm#climbing, https://en.wikipedia.org/wiki/Operator-precedence_parser#Precedence_climbing_method
LB https://en.wikipedia.org/wiki/Strahler_number: justification for register numbers for register alloation (TODO: clarify)
P C4:
LB https://github.com/rswier/c4.git, C4 - C in four functions, Robert Swierczek, minimalistic C compiler running on an emulator on the IR, inspiration for this project
LB https://github.com/rswier/c4/blob/switch-and-structs/c4.c, c4 adaptions to provide switch and structs
LB https://github.com/EarlGray/c4: a X86 JIT version of c4
LB https://github.com/jserv/amacc: based on C4, JIT or native code, for ARM, quite well documented, also very nice list of compiler resources on Github page
P Other minimal compilers and systems:
LB http://selfie.cs.uni-salzburg.at/: C* self-hosting C compiler (also emulator, hypervisor) for RISCV, inspiration for what makes up a minimal C language
LB http://www.iro.umontreal.ca/~felipe/IFT2030-Automne2002/Complements/tinyc.c, Marc Feeley, really easy and much more readable, meant as educational compiler
LB https://github.com/rswier/swieros.git: c.c in swieros, Robert Swierczek
P Assembly:
LB https://github.com/felipensp/assembly/blob/master/x86/itoa.s, for putint (early debugging keyword)
LB https://baptiste-wicht.com/posts/2011/11/print-strings-integers-intel-assembly.htm (earldy debugging keyword)
P Documentation:
LB http://cowlark.com/wordgrinder/index.html: the fabulous editor which just does what it should do
LB https://github.com/mity/md4c: markdown to HTML in C
.