I just checked my Google Webmaster Central account and have over 800 URL restricted by robots.txt. I am currently being penalized by google do you think this could be the cause? Does google penalize a site that has a large number of URL restricted by robots.txt, my restricted pages are php redirects to affiliates.
I dont know, but I see one of my sites having ~10 ulrs restricted by robots.txt and its rankings gone to nowhere. I dont know whats the reason but It may be cause of that.
I don't think this could be the cause. But IMO is better this approach: - Put those URLs on a subdirectory (virtual) using Apache mod rewrite, and then - Restrict only the directory on your robots.txt file
i remember that i was having over 5k restricted by robots.txt nearly 2 weeks ago but i'm getting good traffic from google. it's not a punishment reason for google
It happened to me in the past, that I restricted a fairly large number of pages and didn´t get indexed a number of pages that were supposed to be spiderable. Then I used a tracking system (Tealeaf) that shows you graphically the path that the bot was using, and it seemed that the bot wasn´t dealing too good with the robots. It wasn´t a matter of being penalised but a matter of the spiderability of the site because of the robots.txt
Seems like Google may not be penalizing site for this. Do any of you know of a more elegant way to block Google's bot?