1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

What is robot.txt files.

Discussion in 'robots.txt' started by mithu.singh, May 22, 2010.

Thread Status:
Not open for further replies.
0
  1. #1
    I want to know more about robot.txt files............
    Plz tell me about robot.txt files.........
     
    mithu.singh, May 22, 2010 IP
  2. Amator

    Amator Well-Known Member

    Messages:
    1,424
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    165
    #2
    Amator, May 22, 2010 IP
  3. pandorasbox

    pandorasbox Active Member

    Messages:
    140
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #3
    It basically is a set of instructions for search engine bots, spiders
     
    pandorasbox, May 22, 2010 IP
  4. DIA_8497

    DIA_8497 Well-Known Member

    Messages:
    410
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    130
    #4
    It is used to call the search engines to crawl the website.
     
    DIA_8497, May 23, 2010 IP
  5. alfredwinston

    alfredwinston Peon

    Messages:
    220
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    robots.txt file is a special text file that is located in web servers root directory.It defines the search engine spiders what to follow and what to not.
     
    alfredwinston, May 24, 2010 IP
  6. hannahx

    hannahx Peon

    Messages:
    107
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    in robot.txt file you tell the crawler which pages to crawl or which not to crawl
     
    hannahx, May 24, 2010 IP
  7. saching1

    saching1 Active Member

    Messages:
    131
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #7
    robot.txt is a text file, in this it is defined that spider will follow this and won't follow that.
     
    saching1, May 24, 2010 IP
  8. goldenminds

    goldenminds Peon

    Messages:
    202
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    it's important value for search engines. it determines the crawling behavior and authority which directory or folder can search engine have authority or not. Webmaster can give authority or denied search engines to access specific folders or pages. it's very important to specific optimization over internet.
     
    goldenminds, May 24, 2010 IP
  9. mst124

    mst124 Peon

    Messages:
    187
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    robots.txt file for a website tells the crawler about which page is to be crawled and which not to be crawled
     
    mst124, May 25, 2010 IP
  10. ahmadjordan

    ahmadjordan Member

    Messages:
    88
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #10
    used to include some pages in serach and others not
     
    ahmadjordan, May 25, 2010 IP
  11. vishalgaikar

    vishalgaikar Active Member

    Messages:
    90
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    61
    #11
    Even u can check the example of my Robots.txt file
     
    vishalgaikar, May 25, 2010 IP
  12. Jeff Collision

    Jeff Collision Peon

    Messages:
    1,020
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Sometimes, a content from your website can be copied to any blog submission pages. you can able to know that by checking. So you can disallow the duplicate copy of your content using robots.txt Then, there is no need to visit your cached pages to be visited by search engine bots. You can also disallow those pages using robots.txt
     
    Jeff Collision, May 25, 2010 IP
  13. kukote

    kukote Peon

    Messages:
    125
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #13
    robots.txt file is a text file that search engine bots check if they are allowed to index the sites or not. Google, Yahoo, Bing complies with robots.txt file.
     
    kukote, May 25, 2010 IP
  14. goldenminds

    goldenminds Peon

    Messages:
    202
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    yes i'm fully agree with you.
     
    goldenminds, May 25, 2010 IP
Thread Status:
Not open for further replies.