WP SEO Tip: robot.txt

I was submitting site map links in Google Webmaster Central and came across a problem that brought to my attention what a poor job WP does with the robot.txt function.

*  If you have your blog set public and do not choose the option of adding your Google site map URL to the robot file, no robot file will be created by WordPress.

*  If you choose to add the URL to the site map then WP creates a "virtual" robot.txt file that looks like this:

User-agent: *
Disallow:

YourDomainURL/sitemap.xml.gz

What this does is allow all robots to crawl your site, the good and the bad, and lets them know where your zipped Google sitemap can be found.

* If you chose to make the site private while construction is going on, and you add a URL for the site map to the robot.txt file then WP creates a "virtual" robot.txt file that looks like this:

This, because of the "/" after disallow, will disallow all robots because it does not make a specification after the "/".  Therefore, you will not be indexed by Google or any other search engine.

So, for the standard WordPress install, that is it:

1- No robot.txt file
2- A Virtual Robot file allowing everyone to crawl
3- A Virtual Robot file shutting the doors and locking everyone out!

Now, the robot.txt file is an important SEO feature not to be overlooked.

It can block unwanted "Bad" robots from crawling your site looking for email addresses and information you would not want known.

It can allow the "Good" robots to have access to your site.

However, do you want the Good Guys to have access to everything?

On a standard WP blog allowing posters, you will have a persons post, which will also show up on a category page, on a tag page, on an author page, ... and Google sees all of this, and from what I have read, can cause some duplicate content issues.

There is also a lot of items Google or any of the other Good Guys have no need to see, such as login files or registration files, just as an example.

So a properly constructed robot.txt file can add to your goal of good SEO for you site and help it achieve better rankings.

There are free WP Plugins (just free from WP, this is not a sales pitch)

The one I use is PC Robots.txt Plugin

This plugin allows me to set up a virtual robot file that I can view an edit easily from the plugin settings page in my Admin dashboard. 

It comes with a default setup already typed into the robot.rxt file disallowing a large number of known to be bad Bots, specifically giving the Google Bots premission (confusing because it says disallowed but because there is no "/" after the disallow statement it is actually saying allowed) 

It also allows any Bot not included in the disallow:/ list

It will ask the good Bots not to index certain directories and files.

This is an important feature you should look into.

I am going to close.  I this info may be of help.  Below is what my current robot.txt file looks like for one of my sites which was created by the PC Robots.txt plugin and then edited by me to include some parameters that I found when researching articles on the best setups for robot.txt for WP Blogs.

Take Care:  (long list of bad boys, three direct invites to Google, then instructions on what to index at the bottom (very long, compare it to what WP gives as a default, LOL):