robots.txt

Merry Christmas To All,

I need my seo urls pages searched to achieve this should i allow bots into my Ultimate SEO urls folder??

Below is my robots.txt file please advise if it is correctly written

User-agent: *
Disallow:
Disallow: /admin/
Disallow: /includes/
Disallow: /cgi-bin/
Disallow: /account.php
Disallow: /account_edit.php
Disallow: /account_history.php
Disallow: /account_history_info.php
Disallow: /account_password.php
Disallow: /activations.php
Disallow: /add_checkout_success.php
Disallow: /address_book.php
Disallow: /address_book_process.php
Disallow: /advanced_search.php
Disallow: /checkout_confirmation.php
Disallow: /checkout_payment.php
Disallow: /checkout_payment_address.php
Disallow: /checkout_process.php
Disallow: /checkout_shipping.php
Disallow: /checkout_shipping_address.php
Disallow: /checkout_success.php
Disallow: /contact_bean.php
Disallow: /contact-us-pm-8.html
Disallow: /cookie_usage.php
Disallow: /.htaccess
Disallow: /__index.php
Disallow: /__product_info.php
Disallow: /__stylesheet.css
Disallow: /CHANGELOG
Disallow: /stylesheet2.css
Disallow: /stylesheet1.css
Disallow: /stylesheet.css
Disallow: /specials.php
Disallow: /shopping_cart2.php
Disallow: /shopping_cart1.php
Disallow: /shipping.php
Disallow: /rjrdisco_osc2.sql
Disallow: /review_notice.php
Disallow: /redirect.php
Disallow: /password_forgotten.php
Disallow: /m1_loader.php
Disallow: /m1_export_tracker.php
Disallow: /m1_export.php
Disallow: /logoff.php
Disallow: /login2.php
Disallow: /login1.php
Disallow: /info.php
Disallow: /info_shopping_cart.php
Disallow: /google_ipn.php
Disallow: /google_checkout_process.php
Disallow: /gc_return.php
Disallow: /featured_products.php
Disallow: /extra_info_pages.php
Disallow: /download.php
Disallow: /discount_preview.php
Disallow: /column_right.php
Disallow: /create_account.php
Disallow: /create_account_success.php
Disallow: /login.php
Disallow: /password_forgotten.php
Disallow: /popup_image.php
Disallow: /shopping_cart.php
Disallow: /product_reviews_write.php
Disallow: /source/
Disallow: /xs_autoinstaller/
Disallow: /phpforms/
Disallow: /pub/
Disallow: /templates/
Disallow: /Ultimate_Seo_Urls/
Disallow: /backup/
Disallow: /googlecheckout/
Disallow: /gclog/
Disallow: /ext/
Disallow: /download/
Disallow: /images/
Disallow: /buttons/
Disallow: /mail/

Thanks
Best Wishes For 2010
Dunn

 

Remove the first Disallow and add a "Sitemap: " at the bottom with your sitemap.xml url.

As for the ultimate seo urls, you can do something like this.

Disallow: /Ultimate_Seo_Urls/
Allow: /Ultimate_Seo_Urls/*.php

Or whatever extension your site uses, that will allow bots to crawl .php pages while blocking everything else in that folder.

 

User-agent: *
Disallow:

this is standard format for robots.txt

 

Hello

Just wondering if I need to add in the html on my web pages that I see in the source code of some websites related to robots.txt do follow or follow.

Is this important to do or will google find my pages without a robots.txt file?

Also is it O.K. to have 2 sitemaps? One is called sitemap.xml and the other one is sitemap.html I have a sitemap for one of my pages I built and I think the other is from the built in sitemap generator.

Thanks for your help.

Best Regards
Barb

 

Your webpages do not need extensions. Google will find your pages fine without a robots.txt unless your site is huge and has a lot of parameters like products.php?item_id=3251a&page=4&sess_id=345a334.

There's no point in having 2 identical sitemaps. If your website has separate entities (forum, blog and a main website), or is (again) huge - then multiple sitemaps make sense.

 

Try using google wbmasters tools to check your robots configuration.

 

still good for the web because you still can in the index either by robots

 

this is enough