Google crawls CALENDER even if i say not to.

Discussion in 'Google' started by nishant.soni, Jan 4, 2010.

0
  1. #1
    I have got a forum running on mybb script. I have updated my robots.txt and put /calender.php to not to be crawled by google. But still i see its crawling it. Whats the reason?

    Thanks.
     
    nishant.soni, Jan 4, 2010 IP
  2. longcall911

    longcall911 Peon

    Messages:
    1,672
    Likes Received:
    87
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Robots.txt tells bot what they are allowed to *index*. It does not prevent them from crawling. Theoretically, bots should not bother to crawl, but they certainly can 'sniff' around.

    /*tom*/
     
    longcall911, Jan 4, 2010 IP
  3. Bohra

    Bohra Prominent Member

    Messages:
    12,573
    Likes Received:
    537
    Best Answers:
    0
    Trophy Points:
    310
    #3
    I Agree.. if u want bots to be prevented just block thier ips lol.. they mostly just see it but they wont index.. if they index update us
     
    Bohra, Jan 4, 2010 IP
  4. nishant.soni

    nishant.soni Active Member

    Messages:
    323
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    55
    #4
    got the point thanks.
     
    nishant.soni, Jan 4, 2010 IP