Using robots.txt to disallow to spider dynamic URL

Discussion in 'robots.txt' started by MoneyTopList.com, Oct 21, 2009.

0
  1. #1
    If is use in my robots.txt this:

    User-agent: *
    Disallow: info.php

    ...will I disallow robots to spider also dynamic URLs like mydomain/info.php?id=something-dynamic?
     
    MoneyTopList.com, Oct 21, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    If you use this robots.txt code, it will block only one page that would be info.php. It won't block any other dynamic page.
     
    manish.chauhan, Oct 28, 2009 IP
  3. MoneyTopList.com

    MoneyTopList.com Active Member

    Messages:
    314
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    88
    #3
    OK, thank you for your reply. Is there any other solution how to block the dynamic URLs?
     
    MoneyTopList.com, Oct 28, 2009 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    Yes, you can simply block those urls like
    User-agent: *
    Disallow: /info.php
    Disallow: /info.php?id=*
     
    manish.chauhan, Oct 28, 2009 IP
    digitalpoint and MoneyTopList.com like this.
  5. yesrams

    yesrams Greenhorn

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #5
    Good answer
     
    yesrams, Oct 28, 2009 IP