Want to block other domain from crawling

Discussion in 'robots.txt' started by tanmay.kamath1992, Jun 22, 2013.

0
  1. #1
    I registered and did the DNS setting
    my domain is .com and is set up on wordpress
    I have added my site and google spider bot has crawled my site
    But it is also crawling .wordpress.com
    Please tell me what to disallow in robots.txt .
     
    tanmay.kamath1992, Jun 22, 2013 IP
  2. iulian.pw

    iulian.pw Active Member

    Messages:
    71
    Likes Received:
    3
    Best Answers:
    2
    Trophy Points:
    65
    #2
    I think you can do this with .htaccess, redirect all traffic from .wordpress.com subdomain to your .com domain, and this way the spiders won't revisit your .wordpress.com subdomain again.
     
    iulian.pw, Jun 22, 2013 IP
  3. tanmay.kamath1992

    tanmay.kamath1992 Member

    Messages:
    203
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    35
    #3
    Can u please give me any tutorial link ?
     
    tanmay.kamath1992, Jun 22, 2013 IP
  4. topnichewebsites

    topnichewebsites Member

    Messages:
    35
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    36
    #4
    Take a look at this site:
    http://www.robotstxt.org/robotstxt.html
     
    topnichewebsites, Jun 28, 2013 IP
  5. Whitecap SEO

    Whitecap SEO Active Member

    Messages:
    83
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    80
    #5
    you can use robot meta tag, in that case to nofollow the other domain in your page
     
    Whitecap SEO, Jun 30, 2013 IP
  6. Stephen Moyers

    Stephen Moyers Member

    Messages:
    42
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #6

    Hey you will find plenty of 301 redirection examples on the web. Just modify the basic things and you will done with your redirection. But if you have some pages in that .wordpress.com then you need to be careful while doing the redirection.
     
    Stephen Moyers, Jul 4, 2013 IP