hello friends all of you i have dynamic site and i am do not know how to create robots.txt so plz tell me these website according (webrecsol.com) how to create robots file tell me and anyone send me post create file my site according and what do know dynamic site use robots file yes or no plz tell me. Thanks in advance.
hi u can visit the website : www.google.com/webmasters/tools/ [ Click on Dashboard > Tools > Generate robots.txt file]
You can go at:- http://www.seochat.com/seo-tools/robots-generator/ and Generate robots.txt file I hope this is help full for you
Use of robot.txt gets a bit trickier when it comes to dynamic sites. Normally webmaster tend to rewrite their dynamic urls so that it appears like static and then follow that usual method of employing robot.txt. However with a fairly large site this could be a bit challenging. The other option you have is use wild card character while writing robot.txt instructions. You could visit this site which provides basic information about how to implement this at your end. Offcourse you could also Google it and you would find a lot of useful information on this subject. The only issue I see with using wild card characters is not all search engine spiders recognize them (MSN in particular) and would therefore be rendered useless in that case. But then not all search engines follow robot.txt either.
thanks for all of your good information but i am confused how to create file so i request you can make the file these website (webrecsol.com) all pages google bot allow and send post and my site inner pages dynamic.
A robots.txt file is nothing but a text file (notepad can be used). Copy the text below and edit it to what you need. The # is a "comment". And no blank lines (creates error) # Robots.txt file created by 10/03/2008 # For domain: http://www.catanich.com # # All robots will spider the domain User-agent: * Disallow: /_common/ # Don't index directory Disallow: /_private/ Disallow: /_ScriptLibrary/ Disallow: /resources/index.htm # Don't index this file Sitemap: http://www.catanich.com/site-map.xml Sitemap: http://www.catanich.com/site-map.xml.gz Also, there is no wildcard commands or symbols. The use of "Sitmap" is a way to tell the SEs where the xml Sitemap is.
Yeah, yeah, these funny robots.txt files ... You register a new domain - Then you place a provisional index.html on the server. Next step, installing a wordpress blog, taking care that all ping features are disabled and start to play with configurations and plug ins. Two weeks later do a search, just for fun - site:domain.com and what turns up, unbelievable but true - domain.com First thing I had to do was to check robots.txt - was, of cause there. Next step, uploading a sitemap and submit it to our beloved big G. What answer do you think I got? We can't grawl your site due to robots.txt exclusion! That's my last experience with robots.txt, just from the last two weeks.
you can try this way (yoursite.com/robot.txt) and see. it will mention all about robot.txt . You may also loggin to webmaster tool and see