Hopefully in the right category here. When search engine bots crawl sites, do they crawl other search engines? simple question?
In my opinion no. All the sites in search engines are stored at the backend which are only fetched with related keywords. I dont see search engines doing that on each other and also does not make logical sense.
Why does it not make sense? If you crawled google from various search terms or words, could you not find and cache a lot of sites?
They do not as there are no links to crawl (results are all dynamically generated). Also, if someone or a bot were able to crawl a search engine's database that would reveal a lot of proprietary information which would most likely not be desirable.
They do, but it's usually an accident and it rarely lasts. I have seen this happen. Once in a while you will find results of one engine in the SERPs of another as the result of someone linking directly to the results page. But I'm not talking about major engines getting indexed: it's more often a small search engine that gets indexed because Google, Yahoo, Live or whoever don't realise that it's not a directory.
No..it would not make sense. The search app would need to be built such that it generates all the possible keywords so as to fetch maximum sites. Then the search app would them have to be able to crawl all the possible pages which gets listed. Suppose if it manages to index a sizable no. of sites, from then on it would keep getting the same set of URLs and it would be a huge waste of resouce.
I think they do. Be it search engine or for that matter any other site, a web-site is a web-site, so the SE bots do crawl all the sites on the web.
I wouldn't have thought they would, as one of the other posters have said, it wouldn't really make much sense. But I am not 100%.
well i think they do depending on what it is but. Like they crawl search engines for recruitment and stuff. reason i know this is i have had this happen to me a few times on google ending up on a recruitment websites search results page
where's the point to crawl another search engine ? even if you do it you simply wasting resources ... the main point for any of search engines is to provide more accurate and better experience for the users ... not to giving same retard things over and over ...
I apologize for being a little forward here, but I actually know (not guessing or thinking) they do not, as opposed to all of those that speculate they do, they are just wrong, sorry.
I've seen otherwise, so I'm not speculating. I'm not talking about the search results of something like Yahoo! in Google though, it was some tiny engine I that I can't remember the name of. Or some scraper that had pages full of search results. They all try not to index search results, but they can't rule out accidents or cases of getting spammed.