Do search engines crawl each other?

Discussion in 'All Other Search Engines' started by GarthOddy, Jan 8, 2009.

  1. #1
    Hopefully in the right category here.

    When search engine bots crawl sites, do they crawl other search engines?

    simple question?
     
    GarthOddy, Jan 8, 2009 IP
  2. selbourne

    selbourne Peon

    Messages:
    465
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    0
    #2
    In my opinion no. All the sites in search engines are stored at the backend which are only fetched with related keywords. I dont see search engines doing that on each other and also does not make logical sense.
     
    selbourne, Jan 8, 2009 IP
  3. GarthOddy

    GarthOddy Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Why does it not make sense? If you crawled google from various search terms or words, could you not find and cache a lot of sites?
     
    GarthOddy, Jan 8, 2009 IP
  4. Badlands07

    Badlands07 Well-Known Member

    Messages:
    2,471
    Likes Received:
    100
    Best Answers:
    0
    Trophy Points:
    155
    #4
    They do not as there are no links to crawl (results are all dynamically generated). Also, if someone or a bot were able to crawl a search engine's database that would reveal a lot of proprietary information which would most likely not be desirable.
     
    Badlands07, Jan 8, 2009 IP
  5. Obelia

    Obelia Notable Member

    Messages:
    2,083
    Likes Received:
    171
    Best Answers:
    0
    Trophy Points:
    210
    #5
    They do, but it's usually an accident and it rarely lasts. I have seen this happen. Once in a while you will find results of one engine in the SERPs of another as the result of someone linking directly to the results page. But I'm not talking about major engines getting indexed: it's more often a small search engine that gets indexed because Google, Yahoo, Live or whoever don't realise that it's not a directory.
     
    Obelia, Jan 8, 2009 IP
  6. selbourne

    selbourne Peon

    Messages:
    465
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    0
    #6
    No..it would not make sense.
    The search app would need to be built such that it generates all the possible keywords so as to fetch maximum sites.
    Then the search app would them have to be able to crawl all the possible pages which gets listed.
    Suppose if it manages to index a sizable no. of sites, from then on it would keep getting the same set of URLs and it would be a huge waste of resouce.
     
    selbourne, Jan 9, 2009 IP
  7. videos

    videos Peon

    Messages:
    473
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #7
    yes they do
     
    videos, Jan 9, 2009 IP
  8. GarthOddy

    GarthOddy Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Some mix oppinions on this. Im still unclear
     
    GarthOddy, Jan 9, 2009 IP
  9. rohan_shenoy

    rohan_shenoy Active Member

    Messages:
    441
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    60
    #9
    They can crawl, but its difficult to guess how useful it would be.
     
    rohan_shenoy, Jan 9, 2009 IP
  10. BloggTime

    BloggTime Peon

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    I think they do. Be it search engine or for that matter any other site, a web-site is a web-site, so the SE bots do crawl all the sites on the web.
     
    BloggTime, Jan 9, 2009 IP
  11. BlueIce08

    BlueIce08 Member

    Messages:
    535
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    33
    #11
    I wouldn't have thought they would, as one of the other posters have said, it wouldn't really make much sense. But I am not 100%.
     
    BlueIce08, Jan 9, 2009 IP
  12. tentwenty

    tentwenty Well-Known Member

    Messages:
    1,030
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #12
    well i think they do depending on what it is but. Like they crawl search engines for recruitment and stuff. reason i know this is i have had this happen to me a few times on google ending up on a recruitment websites search results page
     
    tentwenty, Jan 9, 2009 IP
  13. trustnobodynever

    trustnobodynever Peon

    Messages:
    404
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #13
    where's the point to crawl another search engine ? even if you do it you simply wasting resources ... the main point for any of search engines is to provide more accurate and better experience for the users ... not to giving same retard things over and over :) ...
     
    trustnobodynever, Jan 9, 2009 IP
  14. Badlands07

    Badlands07 Well-Known Member

    Messages:
    2,471
    Likes Received:
    100
    Best Answers:
    0
    Trophy Points:
    155
    #14
    I apologize for being a little forward here, but I actually know (not guessing or thinking) they do not, as opposed to all of those that speculate they do, they are just wrong, sorry.
     
    Badlands07, Jan 9, 2009 IP
  15. Obelia

    Obelia Notable Member

    Messages:
    2,083
    Likes Received:
    171
    Best Answers:
    0
    Trophy Points:
    210
    #15
    I've seen otherwise, so I'm not speculating. I'm not talking about the search results of something like Yahoo! in Google though, it was some tiny engine I that I can't remember the name of. Or some scraper that had pages full of search results.

    They all try not to index search results, but they can't rule out accidents or cases of getting spammed.
     
    Obelia, Jan 10, 2009 IP