in a website robots.txt

Discussion in 'robots.txt' started by GENIUSTECH, Oct 26, 2012.

0
  1. #1
    in a website robots.txt allows all search engine, but disallow link which is at facebook link at site, this action what effects the website.
     
    GENIUSTECH, Oct 26, 2012 IP
  2. TheCreator

    TheCreator Banned

    Messages:
    372
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #2
    Means you disallow google to crawl link which is at Facebook link at site...
     
    TheCreator, Oct 26, 2012 IP
  3. selectaupairs

    selectaupairs Greenhorn

    Messages:
    59
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #3
    Your site allowed all search engine to crawl.

    But where facebook link find they do not crawl that page this means if search engine find any facebook link don't crawl that page.
     
    selectaupairs, Oct 30, 2012 IP
  4. dimitarang

    dimitarang Well-Known Member

    Messages:
    713
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    128
    Digital Goods:
    2
    #4
    just make it allowing all links that's the best option,don't include text in robots.txt,just leave the file empty
     
    dimitarang, Oct 30, 2012 IP