Are you blocking googlebot from crawling

Discussion in 'Google' started by mark1230, Feb 21, 2008.

  1. #1
    Well there was a post today on google blog for webmasters tating that some of the webmasters who try to stop spam bots are unknowlingly stopping google bot also to crawl their websites. Have a look at this post here
    googlewebmastercentral.blogspot.com/2008/02/7-must-read-webmaster-central-blog.html

    by Susan Moskwa
     
    mark1230, Feb 21, 2008 IP
  2. MarketExpert

    MarketExpert Well-Known Member

    Messages:
    1,513
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    150
    #2
    I never block them for just that reason.
     
    MarketExpert, Feb 21, 2008 IP
  3. w3bmaster

    w3bmaster Notable Member

    Messages:
    17,594
    Likes Received:
    416
    Best Answers:
    1
    Trophy Points:
    240
    #3
    Nope and if you bloc google bot you block money :)
     
    w3bmaster, Feb 21, 2008 IP
  4. marshalseo

    marshalseo Peon

    Messages:
    252
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Really this is not good technique to avoid spams. By doing this method we will stop our indexing progress. There is an IP Blocking system by which you can stop or avoid the spam IP address.
     
    marshalseo, Feb 21, 2008 IP
  5. lzy

    lzy Peon

    Messages:
    480
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thanks for sharing. Lots of things to look out for.
     
    lzy, Feb 21, 2008 IP
  6. mcaronan

    mcaronan Peon

    Messages:
    441
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Hehe... They didn't noticed that they are blocking google bot.. hehe
     
    mcaronan, Feb 21, 2008 IP
  7. JakeCohen

    JakeCohen Peon

    Messages:
    358
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I'm not, are YOU!
     
    JakeCohen, Feb 21, 2008 IP
  8. vvtv

    vvtv Well-Known Member

    Messages:
    818
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    110
    #8
    I last month unknowingly blocked google bot from accessing my site... I am using this code in my robots.txt file to prevent urls containing "%" from indexing...

    Disallow: /*%*
    Disallow: /*%

    Also I am having some urls containing "$" symbol. So I thought just put another 2 lines containing "$" for blocking urls containing "$" symbols. Within 3 days I noticed that google bot is blocked in webmaster tools... Then I removed those "$" symbols...
     
    vvtv, Feb 21, 2008 IP
  9. Vengence

    Vengence Peon

    Messages:
    100
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    no :):(:):(
     
    Vengence, Feb 21, 2008 IP
  10. enous

    enous Well-Known Member

    Messages:
    1,500
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    158
    #10
    I dont do that...
     
    enous, Feb 21, 2008 IP
  11. Abhik

    Abhik ..:: The ONE ::..

    Messages:
    11,337
    Likes Received:
    606
    Best Answers:
    0
    Trophy Points:
    410
    Digital Goods:
    2
    #11
    Well, I am using robots.txt file to allow a few spiders.. This is what all do.
     
    Abhik, Feb 21, 2008 IP
  12. redspace

    redspace Peon

    Messages:
    619
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #12
    let them block google. that way google bot could spider our site quicker.:)
     
    redspace, Feb 22, 2008 IP