www and non-www crawl by bots

Discussion in 'robots.txt' started by iMax, Aug 9, 2008.

  1. #1
    I want bots to index mydomain.com
    I dont want bots to index www.domain.com

    what can I do in robots.txt to have bots perform that???
     
    iMax, Aug 9, 2008 IP
  2. scribby

    scribby Active Member

    Messages:
    497
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    65
    #2
    Don't know if it can be done via robots.txt but I know you can do it using .htaccess

     
    scribby, Aug 9, 2008 IP
  3. iMax

    iMax Well-Known Member

    Messages:
    136
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #3
    iMax, Aug 9, 2008 IP
  4. scribby

    scribby Active Member

    Messages:
    497
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    65
    #4
    Try...

    
    DirectoryIndex index.php
    <FilesMatch .\.php>
    ForceType application/x-httpd-php
    </FilesMatch>
    Options -MultiViews -Indexes +FollowSymlinks
    RewriteEngine On
    RewriteBase /
    RewriteCond %{HTTP_HOST} !^www\.
    RewriteRule (.*) http://%{HTTP_HOST}/$1 [R=301,L] 
    RewriteRule ^adser|ref|product\.php - [L]
    RewriteCond $1 !(\.css)|(\.js)|(\.ico)|(\.swf)|(\.jpg)|(\.png)|(\.gif)|(^widgets.*)$ [NC]
    RewriteRule ^(.*)$ index.php [QSA,NC,L]
    
    Code (markup):
     
    scribby, Aug 9, 2008 IP
  5. iMax

    iMax Well-Known Member

    Messages:
    136
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #5
    problem solved
    it should look like this:
    
    DirectoryIndex index.php
    <FilesMatch .\.php>
    ForceType application/x-httpd-php
    </FilesMatch>
    Options -MultiViews -Indexes +FollowSymlinks
    RewriteEngine On
    RewriteBase /
    [COLOR="Red"]RewriteCond %{HTTP_HOST} ^www.example.com$ [NC]
    RewriteRule ^(.*)$ http://example.com/$1 [R=301,L][/COLOR]
    RewriteRule ^adser|ref|product\.php - [L]
    RewriteCond $1 !(\.css)|(\.js)|(\.ico)|(\.swf)|(\.jpg)|(\.png)|(\.gif)|(^widgets.*)$ [NC]
    RewriteRule ^(.*)$ index.php [QSA,NC,L]
    
    Code (markup):
    and it WORKS!

    thanks for helping me out - I would never guess!
     
    iMax, Aug 9, 2008 IP
  6. udayns

    udayns Peon

    Messages:
    237
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    you can set in google webmaster tools. Gollow the folling steps:

    1. logine google webmaster tools
    2. Go to tools
    3. that click on "Set preferred domain"

    Here you can set.
     
    udayns, Aug 11, 2008 IP