The importance of having a robots.txt file

I didn’t really place much importance on having a robots.txt file hence some of my sites had one and some others didn’t.  Now the problem with not having one is that if Google cannot find your robots.txt file and doesn’t get a 404 page not found error, Google won’t spider (crawl/index) your site. 

This created the problem of my homepages not being indexed but the rest of the site was. I found this problem by logging into sitemaps and verifying my site.

Add comment June 3rd, 2007