I have a retail store, using Volusion, with over 100 products. I'm reading mixed things on what to "Disallow" in the Robots.txt file. One source claims the file should look like this: Sitemap:http://www.YOURSITE.com/google_sitemap.asp User-agent:* Disallow: /cgi-bin/ Disallow: /AccountSettings.asp Disallow: /Affiliate_info.asp Disallow: /Affiliate_signup.asp Disallow: /Affiliate_thankyou.asp Disallow: /catalog_subscribe.asp Disallow: /donate.asp Disallow: /EmailaFriend.asp Disallow: /Email_Me_When_Back_In_Stock.asp Disallow: /FileUpload/TextObject.aspx Disallow: /GiftOptions.asp Disallow: /help.asp Disallow: /Help_EmailBetterPrice.asp Disallow: /Help_FreeShipping.asp Disallow: /kb_results.asp Disallow: /login_sendpass.asp Disallow: /Login.asp Disallow: /mailinglist_subscribe.asp Disallow: /mailinglist_unsubscribe.asp Disallow: /myaccount.asp Disallow: /MyAccount.asp Disallow: /OrderFinished.asp Disallow: /one-page-checkout.asp Disallow: /orders.asp Disallow: /ProductDetails.asp Disallow: /PhotoDetails.asp Disallow: /PlaceOrder.asp Disallow: /Returns.asp Disallow: /Register.asp Disallow: /Receipt.asp Disallow: /SearchResults.asp Disallow: /ShoppingCart.asp Disallow: /shoppingcart.asp Disallow: /Terms.asp Disallow: /Terms_privacy.asp Disallow: /Ticket_List.asp Disallow: /Ticket_New.asp Disallow: /TrackPackage.asp Disallow: /WishList.asp I am confused. Any help or clarification would be very appreciated. I'm just worried about Google penalizing me for duplicate content.... as some people seem to believe happens if you allow all pages within your site to be crawled. Thanks! Moose77
I did not understand which part of it were incorrect, if you could give me a link, maybe I could help... 1 more thing, Google never penalize any duplicate content, stated on google blog that they will index any duplicate content, so every user that find that content will get the best result on their search.
I was just asking if it is good practice to Disallow everything in the Robots.txt file I listed above. Just trying to better understand the right way to use that file. Thank you for your response.