Recently i have read on many blogs that robots.txt is much important for wordpress, as if search engines found duplicate contents on our blog they will penalize our blog and our blog prove us as a nightmar. There was a solution to use robots.txt to stop search engine finding duplicat contents and also include not to crawl some folders like "wp-admin", i didnt understand how to setup robots.txt for my blog as i am using %postname% as a permalink so plz help me making robots.txt, which commands to includes in it ?
Just open up a text editor and create a plain text file named robots.txt. Then upload it so it's in the root directory of your blog like this www.example.com/robots.txt. You will need to add what you want to disallow to the text file.
Thanks dear for reply but i didnt know what to put in robots.txt file for my wordpress blog. I am using %postname% OR %postname.html% permalink and i dont want that search engines get duplicate contents from my blog so plz tell me what should i have to put in robots.txt ?
I know how to put robots.txt in root of my blog site server but i didnt know what commands to put in it to stop search engines getting duplicate contents from my blog and also plz remember i am using %postname% permalink
I found a useful article here about duplicate content, the robots.txt file is only one part of the best way to do it correctly. http://blog.maddozza.com/wordpress-duplicate-content/16/ Late
you can see my robots.txt structure here in this thread. http://forums.digitalpoint.com/showpost.php?p=5399022&postcount=8 -cypher.
It stops search engines listing useless pages like /wp-login.php, it can also be used to stop duplicate content which affects your rank. Late
I use - User-agent: * Disallow: /wp- Allow: /wp-content/uploads/ Disallow: /trackback/ Disallow: /comments/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ Disallow: /2007/ Disallow: /2008/ Disallow: /2009/ Disallow: /cgi-bin/ Disallow: /category/*/*
there's an equally important one for vBulletin, that blocks all of the duplicate pages (print, archives, etc)
Just so you know, there is a plugin for the robots.txt file you can use in the WordPress codex. http://wordpress.org/extend/plugins/kb-robotstxt/ I also suggest you take a look at Category Base Killer as well.