Help required for ROBOTS.txt?

Discussion in 'robots.txt' started by nshadab, Jan 22, 2008.

  1. #1
    I am very new to this word...ROBOTS.txt

    How this file is helpful to our website as far as the search engines are concern?

    Is this helpfull to gain good ranking at search engine? If yes; What important changes we must make, so that we have good ranking?

    Any other suggestions; we must take care with regards to ROBORTS.txt.

    Thanks
     
    nshadab, Jan 22, 2008 IP
  2. Dr. XYZ

    Dr. XYZ Guest

    Messages:
    226
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Robots.txt has nothing to do with pagerank. This is a file that helps you to allow/avoid search engines crawl certain directories of files that you can specify.

    Learn more about robots.txt here:www.robotstxt.org


    ·​
     
    Dr. XYZ, Jan 22, 2008 IP
  3. nshadab

    nshadab Well-Known Member

    Messages:
    1,120
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    185
    #3
    Thanks for the update.
    What if; i dont want search engines crawl a particular directory.....
    and a condition i want search engines crawl a particular directory

    is there any particluar keywords to make the search engines crawls a particular directory?
     
    nshadab, Jan 22, 2008 IP
  4. Rivers

    Rivers Peon

    Messages:
    276
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #4
    bots by default are set to crawl everything in your website unless you specified them not to.

    to block a directory:

    User-agent: *
    Disallow: /directoryname/

    hope that helps :D
     
    Rivers, Jan 24, 2008 IP
  5. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #5
    catanich, Feb 2, 2008 IP