Need help In my robots.txt

Discussion in 'robots.txt' started by IndianMonk, Apr 10, 2008.

0
  1. #1
    I found in google web master tool that 67 error occurred in my sites ..most of them not found.

    EXample:
    Other all url similar to them. Now I wanna block bots to crawl these urls. Pls help me how to do it. Here my robots.txt:


    Thanx in advance.
     
    IndianMonk, Apr 10, 2008 IP
  2. MeetHere

    MeetHere Prominent Member

    Messages:
    15,399
    Likes Received:
    994
    Best Answers:
    0
    Trophy Points:
    330
    #2
    Add Disallow: /Nature/ if there is no useful content in that.
     
    MeetHere, Apr 10, 2008 IP
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    User-agent: *
    Disallow: /function.*
    Disallow: /full.php?
    Disallow: /nature/

    Use this code...:)
     
    manish.chauhan, Apr 10, 2008 IP
  4. IndianMonk

    IndianMonk Notable Member

    Messages:
    2,666
    Likes Received:
    41
    Best Answers:
    0
    Trophy Points:
    225
    #4
    I have many sub category in Nature...So if I do it google will stop crawling all subcategory under nature.
    Is that right robots.txt:
     
    IndianMonk, Apr 11, 2008 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    In that case you need to exclude all the sub pages of nature folder separately...
     
    manish.chauhan, Apr 13, 2008 IP