Hi, I contacted Yahoo about my site not having as many pages in it's index to what it use to have and the following reply is what I had got. I can't make much sense of it apart from it mentioning about the robots.txt file when my text files are fine and then in the 2nd paragraph it mentions that other sites could have my link in their site and them pages could be indexed by Yahoo. It seems that they have not answered my question about why my site has nearly disappeared from their search results unless I have missed something. I now have a good feeling that I know what caused it and I am now currently working on it, but it has nothing to do with robots.txt as far as I know so why they mentioned that I don't know.
So what do you think the problem is? From my understanding from the above Yahoo are saying that your robots.txt file is telling the slurp not to index? But other sites/links maybe pointing to these pages so they will be indexed but NOT the content? Because you have inbound links to these pages you are 'possibly' excluding with the robots file Yahoo is still going to index them but the content will not be indexed? Strange
Ummm, very strange. I have my robots text file set up just like my other sites. I am getting crawled fine by Google. When I type in my domain it says that I have about 19 pages in the search engine, when there use to be well over 20,000. It thought it was due to haveing near duplicate content on 2 sites, which I am now changing and will still change even if it was not the cause of the problem.
Mmmm ... I had one of my main sites dropped about 4 months ago ... And i have put that down to having lots of pages (200,000+) so i think Yahoo triggers a spam issue if the site has more then xxxx pages? Shame really because most of the 200,000 pages were about individual products and the content was delivered using ajax so it never appeared in the html source so i guess yahoo say that most of my site was duplicates. What did i do? well nothing ... ignored it ... Yahoo delivered less then 1% of my traffic and while i dont like losing any traffic i was not going to change my site and risk losing anything in Google.
It seems that I am getting traffic from Yahoo still. URL: http://www.simplysearch4it.com. I may have been because of having duplicate content, but I have now cleared all that, and Yahoo would have mentioned it in the email I would have thought.
I have looked in my sent email folder and couldn't find the email. I can't find the email that I sent to them and can't remember the email address. Sorry. I am sure that you should be able to find it on their site or you could even try typing something like Yahoo contact into google.