Age | Commit message (Collapse) | Author | |
---|---|---|---|
2014-10-16 | simple url normalizer in Lua on Windows | Andreas Baumann | |
2014-10-16 | .. | Andreas Baumann | |
2014-10-16 | testing with two urlnormalizer modules (google and simple) in crawl.conf | Andreas Baumann | |
2014-10-16 | changed some throw new to throw (caused stopping of crawler with an unknown ↵ | Andreas Baumann | |
exception) | |||
2014-10-14 | fixed url normalizer tests on Windows (by linking them statically). | Andreas Baumann | |
There is some improvement to make the WITH_MODULELOADER a make property and not hard-coded in the makefiles! | |||
2014-10-14 | adapted google url normlazier Lua binding for Windows | Andreas Baumann | |
adapting cralwer to link work with Lua on windows (work in progress) | |||
2014-10-09 | first running lua code with URL normalization, cleanup needed.. | Andreas Baumann | |
2014-10-09 | building curl fetcher only if WITH_CURL=1 is set | Andreas Baumann | |
2014-10-09 | adaptiosn to use local tolua | Andreas Baumann | |
2014-10-09 | first trials with a Google normalizer called from Lua, std::string is the ↵ | Andreas Baumann | |
problem currently and the missing wrapper for the URL class also added a local 'tolua', we will have to hack it | |||
2014-10-08 | compilation fix lua (how did it ever work!) | Andreas Baumann | |
2014-10-08 | some fixes on Windows | Andreas Baumann | |
2014-10-04 | adapted all interface in modules, but SEGV happens now in crawler, | Andreas Baumann | |
module tests pass though | |||
2014-10-03 | some curl fixes | Andreas Baumann | |
2014-10-03 | fixed setting data from source in fetcher modules | Andreas Baumann | |
2014-10-03 | .. | Andreas Baumann | |
2014-10-03 | added an experimental curl fetcher | Andreas Baumann | |
2014-07-25 | we also have to normilze with query | Andreas Baumann | |
2014-07-24 | .. | Andreas Baumann | |
2014-07-24 | reading complete sitemap indexes and sitemaps | Andreas Baumann | |
2014-07-24 | .. | Andreas Baumann | |
2014-07-24 | sitemap processing (work in progress) | Andreas Baumann | |
2014-07-23 | added parsing of Sitemap in robots.txt | Andreas Baumann | |
2014-04-30 | started a robots.txt parser | Andreas Baumann | |
2014-04-25 | fixed MIME detection using libmagic | Andreas Baumann | |
2012-09-10 | libutil move and liblogger rename on Windows | Andreas Baumann | |
2012-09-07 | fixed some linker errors on FreeBSD (should also affect Linux, but there ↵ | Andreas Baumann | |
everything worked?) | |||
2012-09-07 | fixed all tests on Windows, still issues with static libcrawl | Andreas Baumann | |
2012-09-06 | crawler fixed on Linux | Andreas Baumann | |
2012-09-06 | works on Windows, after all the moved | Andreas Baumann | |
2012-09-06 | more splitting into libcrawl, crawl binary | Andreas Baumann | |
moved more public header to 'include' changed approach for dynamic linking on Windows | |||
2012-09-06 | first properly working logger on Windows (singleton dll issue) | Andreas Baumann | |
2012-09-05 | started to solve the logger dll problem on Windows | Andreas Baumann | |
2012-09-05 | split away util, logger, and module | Andreas Baumann | |
made a liblogger adapted all tests | |||
2012-09-02 | prefer dynamic to static linking (linking a non-PIC .a library | Andreas Baumann | |
into a module is illegal, but on Linux!) | |||
2012-08-18 | some code cleanup in fetchers | Andreas Baumann | |
2012-08-17 | adapted winhttp fetcher to new spooling, but crashes | Andreas Baumann | |
2012-08-17 | added a common base class for spooling rewind input stream, adapted | Andreas Baumann | |
libfetch rewind input stream to use that one | |||
2012-08-17 | some project renames | Andreas Baumann | |
2012-08-15 | renamed | Andreas Baumann | |
2012-08-13 | solved static linking problem of module classes on Windows | Andreas Baumann | |
2012-08-13 | winhttp work on windows | Andreas Baumann | |
2012-08-12 | - | Andreas Baumann | |
2012-08-12 | implemented the winhttp fetcher, not working yet | Andreas Baumann | |
2012-08-12 | added a fetcher module test | Andreas Baumann | |
2012-08-12 | improved error handling in module loader | Andreas Baumann | |
crawlingwolf.exe starts on Windows, fetcher still missing | |||
2012-08-12 | better naming of modules | Andreas Baumann | |
2012-08-12 | we should use WinINet, not WinHttp, spendid! | Andreas Baumann | |
2012-08-12 | streamhtmlparser works on Windows | Andreas Baumann | |
2012-08-11 | google url normalization works on Windows, test1 must be improved: | Andreas Baumann | |
there are linking problems (/DSHARED in *.lib normalization libraries produce clashing registry structures) |