Using Robots.txt to focus search results?

Discussion in 'Search Engine Optimization' started by northcave, Oct 21, 2009.

  1. #1
    If I have a site with many products all very similar with a page for each product in a different size then would it be a good idea to only include the main set of products in the XML sitemap and also negate all the others in a robots.txt file?

    The though I suppose would be to focus the google bot on the main pages?

    Has anyone had an experience with this?
     
    northcave, Oct 21, 2009 IP
  2. webdev007

    webdev007 Active Member

    Messages:
    1,037
    Likes Received:
    13
    Best Answers:
    3
    Trophy Points:
    88
    #2
    Its good to restrict all other page that are almost similar in contents with the main page through robots.txt file and also not to include them in sitemap file. If they have different contents then you should let them indexed by SEs.
     
    webdev007, Oct 21, 2009 IP