Age | Commit message (Collapse) | Author | |
---|---|---|---|
2015-03-05 | more lua fixes | Andreas Baumann | |
2014-10-03 | fixed setting data from source in fetcher modules | Andreas Baumann | |
2014-09-28 | first Lua config of crawler | Andreas Baumann | |
2014-09-28 | some lua work | Andreas Baumann | |
2014-09-28 | some testing and stabilizing | Andreas Baumann | |
2014-07-23 | added parsing of Sitemap in robots.txt | Andreas Baumann | |
2012-09-10 | fixed logger to liblogger renaming on Linux | Andreas Baumann | |
2012-09-10 | libutil move and liblogger rename on Windows | Andreas Baumann | |
2012-09-07 | added a libutil for porting stuff and helpers | Andreas Baumann | |
2012-09-07 | fixed some linker errors on FreeBSD (should also affect Linux, but there ↵ | Andreas Baumann | |
everything worked?) | |||
2012-09-06 | crawler fixed on Linux | Andreas Baumann | |
2012-09-06 | more splitting into libcrawl, crawl binary | Andreas Baumann | |
moved more public header to 'include' changed approach for dynamic linking on Windows | |||
2012-09-05 | split away util, logger, and module | Andreas Baumann | |
made a liblogger adapted all tests | |||
2012-09-05 | added Windows WinDbg log sink | Andreas Baumann | |
some renames and cleanups | |||
2012-09-05 | added syslog logger sink | Andreas Baumann | |
2012-09-04 | pimplified logger, hides list of sinks (internal implementation) | Andreas Baumann | |
made logger test link statically again (for valgrind and gdb) | |||
2012-09-02 | prefer dynamic to static linking (linking a non-PIC .a library | Andreas Baumann | |
into a module is illegal, but on Linux!) | |||
2012-08-22 | completly rewrote the logger as singleton | Andreas Baumann | |
2012-08-21 | Merge branch 'master' of ssh://andreasbaumann.dyndns.org:2222/crawler | Andreas Baumann | |
2012-08-21 | - | Andreas Baumann | |
2012-08-21 | - | Andreas Baumann | |
2012-08-19 | - | Andreas Baumann | |
2012-08-17 | added a common base class for spooling rewind input stream, adapted | Andreas Baumann | |
libfetch rewind input stream to use that one | |||
2012-08-15 | renamed | Andreas Baumann | |
2012-08-10 | first porting attempts to Windows: | Andreas Baumann | |
nmake support from Wolframe module loader adapted tests for typeinfo and template trickery | |||
2012-08-08 | added a file rewind input stream | Andreas Baumann | |
started to add MIME type detection and a module based on libmagic (not finished yet) | |||
2012-08-08 | modularized all other modules | Andreas Baumann | |
2012-08-08 | chain filter and modules with one ctor param work now | Andreas Baumann | |
2012-08-07 | started modularization of URL filters | Andreas Baumann | |
better registry function for loading the module (base class as signature) started to support variable arguments for registry create/constructor (work in progress) playing with some Alexandrescu idions :-) | |||
2012-08-07 | more reduction of module code and fixed dependency problem when building | Andreas Baumann | |
2012-08-07 | cleaned up url normalizer tests and made them use module loader | Andreas Baumann | |
2012-08-06 | first steps to make URL loader loadable | Andreas Baumann | |
2012-08-04 | rearanged google test1 and added a GoogleUrlNormalizer | Andreas Baumann | |
2012-07-29 | temporarily removed domain, domain filter is a host filter now | Andreas Baumann | |
2012-07-28 | started to add URL normalizers and testing environment for URLs | Andreas Baumann | |
2012-07-19 | some interface fixes | Andreas Baumann | |
2012-07-18 | added URLSeen component | Andreas Baumann | |
2012-07-15 | started to add URL filters | Andreas Baumann | |
2012-07-14 | first working crawler | Andreas Baumann | |
2012-07-14 | added streamhtmlparser | Andreas Baumann | |
2012-07-13 | added a test for a libfetch_streambuf | Andreas Baumann | |
2012-07-12 | added basic structure | Andreas Baumann | |
2012-07-12 | a restart | Andreas Baumann | |
2009-12-29 | more reading and abstract interfacing | Andreas Baumann | |
2009-12-25 | some starting here | Andreas Baumann | |