1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What is robot.txt flie?

Discussion in 'robots.txt' started by sofiaisabella01, Feb 19, 2012.

  1. #1
    What is robot.txt file and what is the advantages and disadvantages. how con apply in website.
     
    sofiaisabella01, Feb 19, 2012 IP
  2. Icecube_media

    Icecube_media Peon

    Messages:
    656
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    HI
    these are the file which used to tell the search engine what to crawl and what to not.
     
    Icecube_media, Feb 19, 2012 IP
  3. fazyforum

    fazyforum Peon

    Messages:
    21
    Likes Received:
    1
    Best Answers:
    2
    Trophy Points:
    0
    #3
    Robot.txt is a file which you can generate by your self and upload it to your root folder to tell the bots what should they crawl and what not? if you want to disallow something to crawl them simply put disallow and rite the url name or directory name to be restricted by the robots. Rest the good way to apply it go to your webmaster tools and apply there in robots.txt file if n0t using Google webmaster tools then create a new one :)
     
    fazyforum, Feb 22, 2012 IP
  4. JohnnyMazuma

    JohnnyMazuma Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    The robots.txt file contains instructions for search engine spiders. These instructions tell the search engines to ignore directories, files, and even directories/files containing specific character strings. Althoug most people don't get involved enough to warrant explaining specific character strings.Think of the robots.txt file as a container for a set of instructions based upon what you might normally add to an individual webpage using the robots meta string. For example, if you don't want the search engines to follow links or index a particular webpage, you would add: if you were using HTML. If you were using XHTML or HTML5, add the / prior to the >.The advantages to the robots.txt file are:* Reduced work because you're not adding the robots meta tag to each webpage* Ability to tell the search engines to stay out of particular directories* The search engines typically request the robots.txt file as they enter your website each day. When they enter multiple times per day, they normally only ask the first time - not every time.The robots.txt file allows you to provide specific instructions to each spider. For example, you may want the image spiders to enter a particular directory and avoid all others. You may want the blog spiders to enter the blog directory and no others. You may want the standard spider to stay out of those areas.How you use the robots.txt file helps search engines understand how you want them to index your website.I hope this helps.Johnny Mazuma
     
    JohnnyMazuma, Feb 23, 2012 IP
  5. adseo

    adseo Greenhorn

    Messages:
    56
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    16
    #5
    robots.txt is the file to give access to bot to crawl your site, also if you have copy content then you can save those pages through the robots.txt by disallow robots
     
    adseo, Feb 24, 2012 IP
  6. farout666

    farout666 Peon

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    thanks for the info everyone
     
    farout666, Feb 27, 2012 IP
  7. yester123

    yester123 Peon

    Messages:
    360
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #7
    this file should be there in your site because it shows search engines what content to be crawled and what not to be crawled
     
    yester123, Feb 27, 2012 IP
  8. farout666

    farout666 Peon

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    found it and repaired it....thanks again
     
    farout666, Feb 28, 2012 IP
  9. irfan.goodluck

    irfan.goodluck Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    thanks i understant about robots.txt
     
    irfan.goodluck, Feb 29, 2012 IP
  10. irfan.goodluck

    irfan.goodluck Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    thanks i understand about robots.txt
     
    irfan.goodluck, Feb 29, 2012 IP
  11. sisatel

    sisatel Active Member

    Messages:
    1,391
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    90
    #11
    sisatel, Feb 29, 2012 IP
  12. bdthanh

    bdthanh Peon

    Messages:
    28
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #12
    There are two important considerations when using /robots.txt:

    robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
    the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
     
    bdthanh, Mar 4, 2012 IP
  13. madison37

    madison37 Greenhorn

    Messages:
    98
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #13
    It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page , you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty.
     
    madison37, Mar 6, 2012 IP
  14. SPA assurance

    SPA assurance Peon

    Messages:
    93
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    robot.txt file is a text file. This is used for search engine crawling.. Mainly used to improve ur website's score while crawling.
     
    SPA assurance, Mar 8, 2012 IP
  15. perfectbazar

    perfectbazar Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Great information friend thanks for sharing with us.
     
    perfectbazar, Mar 16, 2012 IP
  16. p.caspian

    p.caspian Peon

    Messages:
    964
    Likes Received:
    6
    Best Answers:
    1
    Trophy Points:
    0
    #16
    Robot is a text file which tells robots (bots) what to crawl and what not to crawl.
     
    p.caspian, Mar 21, 2012 IP
  17. anna30

    anna30 Well-Known Member

    Messages:
    280
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #17
    This is simple text file, saved with the name robots.txt to give instruction to web crawler which pages they should visit and which pages not. There is no disadvantages if it is correctly implement. However, there is big loss if it is incorrectly implemented. Search Engine Crawler will never visit your site if you have incorrectly Disallowed for whole pages. see for more details: http://www.robotstxt.org/robotstxt.html
     
    anna30, Apr 3, 2012 IP
  18. perfectbazar

    perfectbazar Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Robots.txt is the file to give access to bot to crawl your site or not crawl.
     
    perfectbazar, Apr 6, 2012 IP
  19. Artuurs

    Artuurs Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #19
    - Not working!
     
    Artuurs, Apr 7, 2012 IP
  20. mbitsol

    mbitsol Guest

    Messages:
    101
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #20
    The location of robots.txt is very important. It must be in the main directory because otherwise search engines will not be able to find it.
     
    mbitsol, Apr 7, 2012 IP