Optimizing your WordPress site for SEO is one of the most important things you can do. An optimized site is going to bring in much more traffic than a site that has not had any SEO work done on it. There are a number of different SEO techniques you can employ to bring in more traffic – adding keywords, using alt tags, and adding meta data are some of the most common ones. However, there are a few other techniques that are not as common. One of these is creating a robots.txt file for your WordPress site and optimizing it. If you do not have one of these files or are not sure if it is optimized, you have found the right WordPress tutorial to walk you through it.
What is a Robots.txt file?
This file is created by website owners (not just those who use WordPress) to talk directly to search engine websites. The full name for this is The Robots Exclusion Protocol. This file is checked before the search engine robot visits the website. It can be used to tell search engines to never visit a site, for example, although it is possible that this can be ignored. It is also possible to tell it to only skip over certain directories or to give robots full access to everything. This file is most often in the root folder of your website. You can use an FTP client to view and download it.
The format of your robots.txt file is very basic. It can be opened in Notepad or any other word processing program. It lists a User-Agent in the first line (often an asterisk to indicate all users) and then lists what directors are allowed or disallowed. Again, an asterisk can be used to allow or disallow everything.
Here is an example of a robots.txt file:
This instructs all search engine robots to exclude the temporary and junk directories. They will not index any pages in these two directories. They will, however, index everything in the wp-content/uploads directory (which is usually your images).
Some people do disallow access to their plugins directory because it is often the target for hackers looking for certain plugins that have security vulnerabilities built into them. If you use plugins that may be a security risk, consider excluding the directory. Many webmasters also exclude the readme.html file so that hackers cannot see what version of WordPress is being used and therefore try to exploit any security holes that version may have.
Optimizing the File
As you can see, this is not really going to help you very much with SEO. There are no keywords or anything else here. One way of using this file for SEO purposes is to follow Google’s guidelines. Google has stated that they frown upon webmasters using robots.txt to hide content. You should not use it to exclude things such as archives or category pages because Google may see this as trying to hide old or low-quality content.
One way of optimizing robots.txt is to add a sitemap to it. A plugin like Yaost’s WordPress SEO will add the entire sitemap to robots.txt. This will list every page in your website in the robots.txt file, which is useful because it makes certain the search engines know every page on your site. They will be instructed to index all of the content you have except what you have specifically disallowed.