1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What is robot.txt file?

Discussion in 'robots.txt' started by James101, Apr 1, 2014.

0
  1. #1
    What is robot.txt file, how to remove spam links?
     
    Solved! View solution.
    James101, Apr 1, 2014 IP
  2. HassanKhalid

    HassanKhalid Active Member

    Messages:
    158
    Likes Received:
    6
    Best Answers:
    4
    Trophy Points:
    90
    #2
    (Robot.txt files) specify which pages the search engine spiders can or cannot crawl.
     
    HassanKhalid, Apr 1, 2014 IP
  3. Ryan Viola

    Ryan Viola Member

    Messages:
    108
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    43
    #3
    Sorry there is no file called robot.txt it is robots.txt

    The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to advising cooperating web crawlers and other web robots about accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
     
    Ryan Viola, Apr 1, 2014 IP
  4. Mikhail19

    Mikhail19 Member

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    41
    #4
    Please be aware that robots.txt can not protect your site against annoying spamming bots - they simply ignore robots.txt
    You can use Registration with Admin approval and / or good captcha in this not easy fight.
     
    Mikhail19, Apr 2, 2014 IP
  5. #5
    Robots.txt file is used to put on the link that you don't want the Google Spiders to crawl, it tells the Google Spiders not to crawl the specific page or link. I don't think that it can help you in removing the SPAM links ???? It's just there to direct the crawlers which page to crawl or not. For fighting with spam Google has provided you the Google Disavow tool.
     
    John Dave, Apr 2, 2014 IP