From 12c50867c04b2c2a11f5026466bbea02d5406b70 Mon Sep 17 00:00:00 2001
From: Andreas Baumann <abaumann@yahoo.com>
Date: Wed, 30 Apr 2014 16:46:00 +0200
Subject: started a robots.txt parser

---
 TODOS | 4 ++++
 1 file changed, 4 insertions(+)

(limited to 'TODOS')

diff --git a/TODOS b/TODOS
index 949dc8a..6d28b36 100755
--- a/TODOS
+++ b/TODOS
@@ -12,3 +12,7 @@
   - content based type detection on Windows
     - port of libmagic?
     - something from Microsoft (around the index service)? 
+- robots.txt
+  - handle Sitemap
+- Parse URLs from sitemaps
+
-- 
cgit v1.2.3-54-g00ecf