blob: e69ed4bd7221f6a7096a385bf4e0017f04f30133 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
Mercator, the "Altavista" robot
http://mercator.comm.nsdlib.org/
authors working for Microsoft now :-)
heritrix
crawler4j
mainly dead or unusable:
jspider
websphinx
Javascript support
phantomjs http://code.google.com/p/phantomjs/
https://github.com/mikeal/spider
https://github.com/joshfire/node-crawler
Php
http://www.makeuseof.com/tag/build-basic-web-crawler-pull-information-website/
Streams
http://www.mr-edd.co.uk/blog/beginners_guide_streambuf
|