Wordpress robots.txt file
March 8th, 2007
A robots.txt file is a file that is used to keep web pages and files from being indexed by search engines. I decided I needed one on my blog to stop duplicate the rss feeds being indexed.
To create your robots.txt file open up notepad and save the file as robots.txt.
Here is the code:
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
The first line “User-agent: *” means that all the search engine robots (from Google, Yahoo, MSN and so on). Search engine robots are what gather the data for search engines.
“Disallow: /wp-” means don’t index the pages in the root folder (also known as home directory) with the starting name wp-.
“Disallow: /feed/” don’t index the feed. This stops your blog getting penalized for duplicate content.
“Disallow: /trackback/” don’t index trackbacks
Entry Filed under: Internet business
1 Comment Add your own
1. David Bradley | May 1st, 2007 at 3:51 pm
There are reports elsewhere in the blogosphere that this tip boosts traffic pretty well. Embed the sitemap URL too and you should be quids in.
One caveat, your mileage may vary, and I read one blogger who reckoned he’d been sent to Google Hell after editing his robots.txt file. He could have made a mistake with it, of course.
Leave a Comment
Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
Trackback this post | Subscribe to the comments via RSS Feed