If you don't want SEs to index a particular page of your site, then add the following code in your robots.txt file - User-agent: * Disallow: /folder/file.html (give the complete path for your file)
Robot.txt file has some simple directives which manage the bots.These are define as below. 1.User agent : The * is a wildcard which means all bots or googlebot for google. 2.Disallow : Define which folder or file will be executed.
[TABLE="width: 658"] [TR] [TD="width: 658"]http://www.robotstxt.org/robotstxt.html Refer this link for using robots.txt file. [/TD] [/TR] [/TABLE]
write the code in notepad and save the notepad in robot.txt User-agent: * Disallow: /search Disallow: /groups Disallow: /images Disallow: /admin If u don't want to crawl the folder and pages of websites then use robots.txt file
[TABLE="width: 100%"] [TR] [TD="width: 75%"]A "robots.txt" file is a text file placed on your server which contains a list of robots and "disallows" for those robots. Each disallow will prevent any address that starts with the disallowed string from being accessed.[/TD] [/TR] [/TABLE]
write the code in notepad and save notepad Name as robot.txt and put on root path (just like http://www.xyz.com/robot.txt) User-agent: * Disallow: /search Disallow: /groups Disallow: /images Disallow: /admin