I don't want crawl some links

Discussion in 'robots.txt' started by hi-nam, Jan 18, 2009.

  1. #1
    My site using mod_rewrite in all public URLs, but some URLs when browse in Admin side is not.

    Some URLs like this:

    domain.com/?id.....
    domain.com/index.php?id....
    domain.com/subpage/?id....

    More URLs like these, how to block all?

    Thanks!
     
    hi-nam, Jan 18, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    let me know full path of the URLs so that I can provide you with the exact details:)
     
    manish.chauhan, Jan 20, 2009 IP
  3. hi-nam

    hi-nam Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I have no exact URLs

    Because the site has to much URLs with:

    If domain.com/1/ then have domain.com/1/?id1

    Bot is the same content on the page.
     
    hi-nam, Jan 20, 2009 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    You can block it using:

    User-agent: *
    Disallow: /?id.....
    Disallow: /*?id.....
     
    manish.chauhan, Jan 20, 2009 IP
  5. hi-nam

    hi-nam Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #5
    and what happed with:

    domain.com/12/?id12
    domain.com/13/?id13
    domain.com/14/?id14
    domain.com/15/?id15
    ...................

    ?

    Thanks!
     
    hi-nam, Jan 20, 2009 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    Disallow:/*/?id*

    Try this
     
    manish.chauhan, Jan 20, 2009 IP
  7. hi-nam

    hi-nam Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    You can do with 10 thoudsans links like that :(
     
    hi-nam, Jan 20, 2009 IP
  8. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #8
    Yes..you can use this code for all the urls in the same fashion..:)
     
    manish.chauhan, Jan 20, 2009 IP
  9. hi-nam

    hi-nam Peon

    Messages:
    211
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I will try and let you know!

    Thank you!
     
    hi-nam, Jan 20, 2009 IP
  10. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #10
    I'll also wait for your response
     
    manish.chauhan, Jan 20, 2009 IP