Need Help With Robots.txt File

Discussion in 'Search Engine Optimization' started by danielwood, Feb 5, 2013.

0
  1. #1
    I want to block certain type of pages that are generated dynamically through robots.txt.

    Format for such pages is:

    domain.com/page.php?variable1=value
    or
    domain.com/page.php?variable2=value
    or
    domain.com/page.php?variable3=value

    If i add:
    Disallow: /page.php?*

    would it work?Please suggest.
     
    danielwood, Feb 5, 2013 IP
  2. abyse

    abyse Notable Member

    Messages:
    346
    Likes Received:
    9
    Best Answers:
    1
    Trophy Points:
    245
    #2
    Disallow: /page.php?* will block all urls like:

    domain.com/page.php?variable1=value
    domain.com/page.php?other-words.php

    You can check and test your existing robots.txt via Google Webmaster Tools > Site Configuration > Crawler access.

    Also, make sure that your robots.txt file is valid: http://tool.motoricerca.info/robots-checker.phtml
     
    abyse, Feb 5, 2013 IP
  3. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #3
    Is this to avoid duplicate content (for example, ?variable1=value changes the order a product list, but still all the same products are on the page)? If so, you can use rel=canonical so that Google knows example.com/product-list.php and example.com/productlist.php?sort=asc are the same.

    More information.
     
    ryan_uk, Feb 5, 2013 IP