my site is showing robot.txt error

Discussion in 'robots.txt' started by dheeruyadav, Sep 23, 2008.

0
  1. #1
    what should i do my site is showing

    This crawl report is updated weekly, and we are unable to accommodate requests for more frequent crawling. Please also keep in mind that this report is not exhaustive.
    Blocked URL
    http:/ / www. a1-webmarks. com/ bm_edit. html? u= http%3A%2F%2Fthecricketingleague. blogspot. com%2F2008%2F09%2Ficl-kick-icc-right-in-their-ground. html&t= ICL+kick+ICC+right+in+their+ground

    Reason Blocked [?] Robots.txt File

    Last Crawl AttemptSep 18, 2008
    Please help me.
     
    dheeruyadav, Sep 23, 2008 IP
  2. Kuldeep1952

    Kuldeep1952 Active Member

    Messages:
    290
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #2
    Your robots.txt file has this line

    Disallow: /bm_edit

    delete this line
     
    Kuldeep1952, Oct 1, 2008 IP
  3. himtuna

    himtuna Member

    Messages:
    225
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    28
    #3
    its really painfull, even i get to many errors
    but if we are telling the crawler that some files/directories are not important ( in this case bm_edit.html?...)
    then why the hell is it producing errors
    Cant it read the first part: Disallow:
    and make sure you know this
    disallow: /abc everything starting with abc whether files or directories are not important
    disallow: /abc/ everythin in the directory abc is not important
    rest you can make out with php? and php
    ...
     
    himtuna, Oct 6, 2008 IP