Here's the situation, we have two related sites. One is the public portion where content is displayed and one is more like an intranet site for content to be generated. We want people to be able to find the public site (1mill+ page views a day, people can find it no problem). However, we don't want people stumbling across the intranet site. We have asked people not to link back to that site, but they do any way. We intentionally delisted our site from the Google index. For the life of me I cannot find any way to intentionally delist a site from the Bing/Yahoo results. Besides violating Bing rules and then pointing it out to them, anybody know of a way to do this? If I set up webmaster tools for Bing, can I do it from there?
Make a robot.txt file on your server and put something similar to this in it: User-agent: * Disallow: /search Disallow: /admin # End robots.txt file This will stop almost any search engine from listing the specific pages you don't want to list in it's index. As soon as Bing/Yahoo re-crawl your website they will pick up this file automatically and delete the pages/folders you specify in the robots.txt file that you don't want indexed.