How has MSN and Yahoo got Google indexed?

Discussion in 'Websites' started by soapbath, Aug 29, 2005.

0
  1. #1
    Checking over my search engine spider logs I noticed Google has a robots.txt blocking all user agents. It also blocks there home page.. so how has Yahoo and MSN got google indexed if they abide by Googles Robots.txt?

     
    soapbath, Aug 29, 2005 IP
  2. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #2
    I don't see DisAllow:/ in that list...


    --
    Derek
     
    dkalweit, Aug 29, 2005 IP
  3. soapbath

    soapbath Peon

    Messages:
    909
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Oh yeah :p Im silly. But anyhow, why do they disallow to so much?
     
    soapbath, Aug 29, 2005 IP
  4. dkalweit

    dkalweit Well-Known Member

    Messages:
    520
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    150
    #4
    They don't want bots scraping their dynamic pages. It's against their TOS, so they put it in their robots.txt, too.


    --
    Derek
     
    dkalweit, Aug 29, 2005 IP