Edit robots txt in Wordpress

How to Edit Easily a WordPress Robots.txt File

 


 

In general it’s easy to edit a robots.txt of a custom coded website in html or php. WordPress has dynamically generated robots.txt which can’t be edited in the usual way. You can’t file the robots.txt file when you check the root directory of your WordPress installation.

When constructing a website it is best to deny robots on your website. You should use the following code:

The above code doesn’t allow crawling bots to index your website.
Fortunately in WordPress you don’t have to edit your robots.txt because you can do this with an easy click in Settings -> Reading and going to Search Engine Visibility -> Discourage search engines from indexing this site.

 

 

It happens of course that we want to modify more complicated things like blocking specific URLs or a directory:

 

The plugin that should be used to edit WordPress robots.txt

Well, I’ve tried a WordPress plugin called WP Robots Txt and I like it very much. You can edit your dynamically generated WordPress robots.txt file by going to Settings -> Reading -> Search Engine Visibility and below the Discourage search engines from indexing this site checkbox you can see your editable Robots.txt Content.

You can witness some warning messages in your Google Webmasters account when you delete a post or you had set it to be expired after a time. You can easily cure it by adding the affected URL in the dynamic WordPress robots.txt file.

You can find yourself in a situation where you try a lot of stuff in your robots.txt file and at the end you don’t know what was originally included. For those who want to reset the modifications, just delete everything in the text area where the WordPress robots.txt content is and the values will be restored to default.

Just a remark: if you want to have your website secured during the construction period the best method is to block unauthorized access by using .htaccess .htpasswd blocking.

 


 

Posted by:

Balazs Hende
I am a blogger and the founder of this site. I have started sparesome.com on October 1, 2013. Have a pleasant stay here!

Related articles

Categories

Back to Top