URLs restricted by robots.txt..... ?

jamie01

Mar 19th 2007, 4:05 am

HI

What it means in Google sitemap.... my website has 60 errors in this column..
Web crawl errors... like :
HTTP errors
0
--
Not found
0
--
URLs not followed
0
--
URLs restricted by robots.txt
60
Details »
URLs timed out
0
--
Unreachable URLs
0

Please explain me if it has any impact on SERP.......and which URL restricted by robots?

Thanks

Mwb2k

Mar 19th 2007, 4:10 am

Uhm, you can define the contents of robots.txt yourself, so how can we know? :)

abcde

Mar 19th 2007, 4:28 am

I guess its this site ?? --> ayhu.com

take a look at http://www.ayhu.com/robots.txt

User-agent: *
Disallow: admin/
Disallow: includes/
Disallow: templates/
Disallow: logout.php
Disallow: suggest-link.php
Disallow: suggest-category.php

jamie01

Mar 19th 2007, 4:30 am

Uhm, you can define the contents of robots.txt yourself, so how can we know? :)

So does google sitemap shows those URL which has been disallowed in robort.txt ? I am worried if this due to other Problems....pls let me know how to get ride from this ?

Thanks

jamie01

Mar 19th 2007, 4:32 am

I guess its this site ?? --> ayhu.com

No, sorry...not this one

abcde

Mar 19th 2007, 4:34 am

delete the robots.txt and allow to crawl the whole site. BUT ist not a problem or an error so theres nothing you can do.

daykevi

Mar 19th 2007, 4:35 am

HI

What it means in Google sitemap.... my website has 60 errors in this column..
Web crawl errors... like :
HTTP errors
0
--
Not found
0
--
URLs not followed
0
--
URLs restricted by robots.txt
60
Details »
URLs timed out
0
--
Unreachable URLs
0

Please explain me if it has any impact on SERP.......and which URL restricted by robots?

Thanks

No not at all, if you restricted some url using the robots.txt file it is not going to hurt u.

jamie01

Mar 19th 2007, 4:45 am

delete the robots.txt and allow to crawl the whole site. BUT ist not a problem or an error so theres nothing you can do.

HI abcde

Is it a hurt for me ? would it be better than now if the robot.txt deleted ?

Thanks

abcde

Mar 19th 2007, 4:49 am

HI abcde

Is it a hurt for me ? would it be better than now if the robot.txt deleted ?

Thanks

No, no need to be worried. Here some answers that may help you:

http://www.google.com/support/webmasters/bin/answer.py?answer=35235&hl=en

http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html

jamie01

Mar 19th 2007, 5:36 am

No, no need to be worried. Here some answers that may help you:

http://www.google.com/support/webmasters/bin/answer.py?answer=35235&hl=en

http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html

Thanks abcde