Hi, im new to search engines. Im curious if robots.txt is needed if i have put <meta name="robots" content="index,follow"> ? What should my robots.txt looks like?
The three major search engines support XML sitemaps, and I use my robots.txt to point to it. Here's the info below... User-agent: * Sitemap: http://www.yourdomainnamehere.com/sitemap.xml Code (markup): If you nothing of XML sitemaps, go to the URL below and provide your website address. It will create a XML sitemap for you if you have under 500 pages. http://www.xml-sitemaps.com/ By the way, search engines will spider your entire site unless you have noindex and nofollow tags on the pages. A robots.txt file is really not needed, but having it point to a sitemap will help the search engines index your site faster - especially if its a large site.
You are right Snowbird this is the best way to set up the robots on every site. Thanks for sharing with us.
Even if I have nothing to enter in the robots.txt, that is to say, I want all the pages to be indexed, I always create a blank robots.txt. This is to reduce unwanted 404 errors (page not found), caused by bots trying to get this file.