Robots.txt

Discussion in 'Search Engine Optimization' started by seorupesh, Dec 23, 2010.

0
  1. #1
    Can anyone define Robots.txt. Thanks in advance.
     
    seorupesh, Dec 23, 2010 IP
  2. SarahPerez10

    SarahPerez10 Greenhorn

    Messages:
    39
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #2
    Robots.txt is a text file written in not pad to define the pages which are allowed or not to be cache by search engines.
     
    SarahPerez10, Dec 23, 2010 IP
  3. arunthe

    arunthe Peon

    Messages:
    174
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    yes i agree with SarahPerez10
    Robots.txt is a text file written in not pad to define the pages which are allowed or not to be cache by search engines.
     
    arunthe, Dec 23, 2010 IP
  4. tonyran

    tonyran Peon

    Messages:
    953
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #4
    it doesnt have to be written in notepad
     
    tonyran, Dec 23, 2010 IP
  5. AirForce1

    AirForce1 Peon

    Messages:
    1,325
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Hi,

    It is a text file resided at the root of sites used in guiding the behaviors of all bots in how to crawling your sites. This is a authority source for your purpose: robotstxt.org :)

    all the best,
     
    AirForce1, Dec 24, 2010 IP
  6. Baby Prakash

    Baby Prakash Peon

    Messages:
    232
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Robot.txt file help to index your pages easily.
     
    Baby Prakash, Dec 24, 2010 IP
  7. jitendraag

    jitendraag Notable Member

    Messages:
    3,982
    Likes Received:
    324
    Best Answers:
    1
    Trophy Points:
    270
    #7
    Correction, robots.txt doesn't control 'caching', it controls whether search engine bots (you can also choose specific bots) should crawl a part of your website.
     
    jitendraag, Dec 24, 2010 IP
  8. wakeupseo

    wakeupseo Member

    Messages:
    191
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    30
    #8
    Most site contain page that should not be indexed by search engine like contact.html, so we use the Robat.text file to tell the search engine spider to ignore it.
     
    wakeupseo, Dec 24, 2010 IP