1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt

Discussion in 'Search Engine Optimization' started by softonaseo, Jul 23, 2012.

  1. #1
    Do you know anything about robots.txt?
    SEMrush
     
    softonaseo, Jul 23, 2012 IP
    SEMrush
  2. seothevirtual

    seothevirtual Greenhorn

    Messages:
    56
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #2
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.For example, you may not want Google to crawl the images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth."Robots.txt" lets you tell Google just that.
     
    seothevirtual, Jul 23, 2012 IP
  3. Mike55

    Mike55 Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    This file does help to tell search engine spiders to allow/disallow page which you want to take action. If there are any contents on your website which would harmful for website ranking then you can disallow that page in robots file instead of to remove that page.
     
    Mike55, Jul 23, 2012 IP
  4. sisatel

    sisatel Active Member

    Messages:
    1,391
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    90
    #4
    sisatel, Aug 4, 2012 IP
  5. easypr

    easypr Member

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #5
    Robots.txt is just allow or disallow the search engine spider to index your website pages, folder, css, images etc by write a simple code in robots.txt. For example if you want to allow search engine your all website than code is-:

    For Allow-:
    ---------------
    User-agent: *
    Allow: /

    If you want to disallow search engine to index certain page then code is

    User-agent: *
    Disallow: /services
    Disallow: /services.php

    Or your want to disallow folder then code is -:

    User-agent: *
    Disallow: /search
    Disallow: /catalogues
     
    easypr, Aug 4, 2012 IP
  6. indinoida2012

    indinoida2012 Peon

    Messages:
    56
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    indinoida2012, Aug 4, 2012 IP
  7. vickygvt

    vickygvt Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    robots.txt is a text you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
     
    vickygvt, Aug 4, 2012 IP
  8. nptifitness

    nptifitness Peon

    Messages:
    239
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Robots.txt is used for tell to crawler to index the pages of your sites or keep away some pages from indexing the sites.

    for eg.


    User-agent: *
    Allow:/

    Or

    User-agent: *
    Disallow: /abc.html
     
    nptifitness, Aug 4, 2012 IP
  9. techstalks

    techstalks Peon

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    what does robots.txt do?
     
    techstalks, Aug 4, 2012 IP
  10. queuemobile

    queuemobile Greenhorn

    Messages:
    99
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    6
    #10
    Robots.txt file in root directory ,helping to securing purpose.you have saying to search engines ,follow this files and folders etc.
     
    queuemobile, Aug 4, 2012 IP
  11. sebastianpick

    sebastianpick Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    It's especially good, if you have a print preview for each of your sites. So to avoid dublicate contant you can determine with the robots.txt, that all the print previews shouldn't be crawled.
     
    sebastianpick, Aug 7, 2012 IP
  12. RickyChhajed

    RickyChhajed Peon

    Messages:
    73
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Robots.txt file is a set of instructions that tell search engine robots which pages of your site to be crawled and indexed. In most cases, your site is consist of many files or folders i.e. admin folders, cgi-bin, image folder, which are not relevant to the search engines. Robots.txt helps tell spiders what is useful and public for sharing in the search engine indexes and what is not. It should also be noted that not all search spiders will follow your instructions left in the robots.txt file. In addition, a poorly done robots.txt file can stop the search spiders from crawling and indexing your website properly.
     
    RickyChhajed, Aug 7, 2012 IP
  13. jyotsna01

    jyotsna01 Peon

    Messages:
    93
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Update the coding of robot.txt in your website root directory...
    It will let google not crawl the content on ur website which you dnt want it to do...
    update it if its very neccesary...
     
    jyotsna01, Aug 7, 2012 IP
  14. marketingzen1

    marketingzen1 Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Hello

    Through robot.txt file website owners gives instructions to web robots about that website. This file serves as a request that specified robots ignore specified files or directories when crawling a site.
     
    marketingzen1, Aug 21, 2012 IP