1 files changed, 8 insertions, 0 deletions
diff --git a/docs/LINKS b/docs/LINKS
index 7600d3a..4a24075 100755
--- a/docs/LINKS
+++ b/docs/LINKS
@@ -89,3 +89,11 @@ Singleton design:
 
 Linking with gcc and visibility:
 - http://gcc.gnu.org/wiki/Visibility
+
+Robots.txt:
+- http://www.nextthing.org/archives/2007/03/12/robotstxt-adventure
+- https://github.com/seomoz/reppy: in Python, but as source of inspiration quite nice
+
+Service for Crawling:
+- http://www.michaelnielsen.org/ddi/how-to-crawl-a-quarter-billion-webpages-in-40-hours/
+- http://commoncrawl.org/