Penalty for Large # of URLs restricted by robots.txt?

Discussion in 'Google' started by underdog, Jul 12, 2007.

  1. #1
    I just checked my Google Webmaster Central account and have over 800 URL restricted by robots.txt. I am currently being penalized by google do you think this could be the cause? Does google penalize a site that has a large number of URL restricted by robots.txt, my restricted pages are php redirects to affiliates.
     
    underdog, Jul 12, 2007 IP
  2. freeman

    freeman Active Member

    Messages:
    232
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    58
    #2
    I dont know, but I see one of my sites having ~10 ulrs restricted by robots.txt and its rankings gone to nowhere. I dont know whats the reason but It may be cause of that.
     
    freeman, Jul 12, 2007 IP
  3. ajsa52

    ajsa52 Well-Known Member

    Messages:
    3,426
    Likes Received:
    125
    Best Answers:
    0
    Trophy Points:
    160
    #3
    I don't think this could be the cause. But IMO is better this approach:
    - Put those URLs on a subdirectory (virtual) using Apache mod rewrite, and then
    - Restrict only the directory on your robots.txt file
     
    ajsa52, Jul 12, 2007 IP
  4. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #4
    i remember that i was having over 5k restricted by robots.txt nearly 2 weeks ago but i'm getting good traffic from google. it's not a punishment reason for google
     
    trichnosis, Jul 12, 2007 IP
  5. pixelpusher

    pixelpusher Peon

    Messages:
    77
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I don't think Google should care which URLs you don't want to be indexed...
     
    pixelpusher, Jul 12, 2007 IP
  6. SEO Madrid

    SEO Madrid Peon

    Messages:
    108
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #6
    It happened to me in the past, that I restricted a fairly large number of pages and didn´t get indexed a number of pages that were supposed to be spiderable. Then I used a tracking system (Tealeaf) that shows you graphically the path that the bot was using, and it seemed that the bot wasn´t dealing too good with the robots. It wasn´t a matter of being penalised but a matter of the spiderability of the site because of the robots.txt
     
    SEO Madrid, Jul 12, 2007 IP
  7. underdog

    underdog Peon

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Seems like Google may not be penalizing site for this. Do any of you know of a more elegant way to block Google's bot?
     
    underdog, Jul 12, 2007 IP
  8. totalmasala

    totalmasala Banned

    Messages:
    1,454
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #8
    i really don't know......penalty for large restricted URLS in robots.txt!!!!
     
    totalmasala, Jul 13, 2007 IP
  9. paladin2

    paladin2 Banned

    Messages:
    375
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I don't think but everything is possible...
     
    paladin2, Jul 13, 2007 IP