How to block certain pages with url parameters from crawler's access using robots.txt

Discussion in 'robots.txt' started by mpchekuri, Feb 29, 2012.

  1. #1
    My web site is getting several links from other sites with some parameters with the reference to their sites. For example http:// mysite.com/?ref=www.agothu.com. These links are getting indexed in search engines and are causing unnecessary crawler access. How would I block certain urls with parameters from crawler's access using robots.txt.

    For blocking I used
    Disallow : /folder/

    But I don't know how to block urls with parameters. Can some one help?
     
    Last edited: Feb 29, 2012
    mpchekuri, Feb 29, 2012 IP
  2. arunalande

    arunalande Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I think use of robot.txt in a right way is beneficial for you. you may find your answer on this link. check it.


    http://eadvertisements.blogspot.in/2012/03/removeblock-unwanted-pages-using.html

    Thanks.
     
    arunalande, Mar 16, 2012 IP
  3. Kurl Walls

    Kurl Walls Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Simply add a nofollow tag in your Robot.txt file
     
    Kurl Walls, Mar 17, 2012 IP
  4. mpchekuri

    mpchekuri Member

    Messages:
    40
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    26
    #4
    Can you explain the syntax.
     
    mpchekuri, Mar 24, 2012 IP
  5. pindians007

    pindians007 Greenhorn

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #5
    just used disallow for the url which you want to block
     
    pindians007, Mar 26, 2012 IP