Mercator, the "Altavista" robot

http://mercator.comm.nsdlib.org/

authors working for Microsoft now :-)

Some Java roboter frameworks:

heritrix
crawler4j

mainly dead or unusable:

jspider
websphinx

A C++ web robot

http://code.google.com/p/whalebot/

Javascript support

phantomjs http://code.google.com/p/phantomjs/
https://github.com/mikeal/spider
https://github.com/joshfire/node-crawler

Php

http://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website/

Streams

http://www.mr-edd.co.uk/blog/beginners_guide_streambuf

Lua embedding

http://www.ibm.com/developerworks/linux/library/l-embed-lua/

Loadable modules in C++

http://www.isotton.com/devel/docs/C++-dlopen-mini-HOWTO/C++-dlopen-mini-HOWTO.html