Wordpress robots.txt file

A robots.txt file is a file that is used to keep web pages and files from being indexed by search engines. I decided I needed one on my blog to stop duplicate the rss feeds being indexed.
To create your robots.txt file open up notepad and save the file as robots.txt.

Here is the code:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/

The first line “User-agent: *” means that all the search engine robots (from Google, Yahoo, MSN and so on). Search engine robots are what gather the data for search engines.

“Disallow: /wp-” means don’t index the pages in the root folder (also known as home directory) with the starting name wp-.

“Disallow: /feed/” don’t index the feed. This stops your blog getting penalized for duplicate content.

“Disallow: /trackback/” don’t index trackbacks

1 comment March 8th, 2007