The importance of having a robots.txt file

June 3rd, 2007

I didn’t really place much importance on having a robots.txt file hence some of my sites had one and some others didn’t.  Now the problem with not having one is that if Google cannot find your robots.txt file and doesn’t get a 404 page not found error, Google won’t spider (crawl/index) your site. 

 

This created the problem of my homepages not being indexed but the rest of the site was. I found this problem by logging into sitemaps and verifying my site.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.
  • Digg
  • del.icio.us
  • Netvouz
  • BlinkList
  • Reddit
  • Spurl
  • StumbleUpon
  • YahooMyWeb

Entry Filed under: Internet business

Leave a Comment

Required

Required, hidden

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed


Add to Technorati Favorites