What does this robots.txt file do? User-agent: * Disallow: I was just sitting in my folder, does it do anything at all? do I need it? If it had the slash it would stop all spiders, yes? Like this below. User-agent: * Disallow: / Peter
User-agent: googlebot Disallow:/images If we mention above code in the robots.txt file, then it will prevent google search engine to crawl images folder. Similarly we can prevent spiders of different engines to crawl unwanted folders or web pages. Thus with the help of robots.txt file we can increase the indexing of important pages of our website.
User-agent: the robot the following rule applies to Disallow: the URL you want to block User-agent: * Google uses several different bots (user-agents). The bot we use for our web search is Googlebot. Our other bots like Googlebot-Mobile and Googlebot-Image follow rules you set up for Googlebot, but you can set up specific rules for these specific bots as well Disallow: it will not Block any folder you have on the site
It will disallow Google from crawling any of the dynamically generated pages. It will allow the altavista scooter bot to access every page