Help me to create my robots.txt

Discussion in 'robots.txt' started by Abhik, Nov 23, 2009.

0
  1. #1
    Hello..
    Please help me to create a robots.txt file for my vBulletin Forum

    I want all spiders to crawl my site, but want then NOT to crawl and index some of the folders.
    And, I'd also like to put a link to sitemap file in it.. How do I do it?

    Thanks
     
    Abhik, Nov 23, 2009 IP
  2. saimoux

    saimoux Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    <p><noindex>text</p></noindex>
     
    saimoux, Nov 23, 2009 IP
  3. saimoux

    saimoux Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Examples

    This example allows all robots to visit all files because the wildcard "*" specifies all robots:
    User-agent: *
    Disallow:

    This example keeps all robots out:
    User-agent: *
    Disallow: /
     
    saimoux, Nov 23, 2009 IP
  4. saimoux

    saimoux Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    The next is an example that tells all crawlers not to enter four directories of a website:
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /images/
    Disallow: /tmp/
    Disallow: /private/

    Example that tells a specific crawler not to enter one specific directory:
    User-agent: BadBot # replace the 'BadBot' with the actual user-agent of the bot
    Disallow: /private/
     
    saimoux, Nov 23, 2009 IP
  5. saimoux

    saimoux Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Example that tells all crawlers not to enter one specific file:
    User-agent: *
    Disallow: /directory/file.html

    Note that all other files in the specified directory will be processed.

    Example demonstrating how comments can be used:
    # Comments appear after the "#" symbol at the start of a line, or after a directive
    User-agent: * # match all bots
    Disallow: / # keep them out
     
    saimoux, Nov 23, 2009 IP
  6. Abhik

    Abhik ..:: The ONE ::..

    Messages:
    11,337
    Likes Received:
    606
    Best Answers:
    0
    Trophy Points:
    410
    Digital Goods:
    2
    #6
    Ah!! Thanks..
    I already managed to create one.
     
    Abhik, Nov 23, 2009 IP
  7. hostimizer

    hostimizer Well-Known Member

    Messages:
    548
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    150
    #7
    simply open google webmaster there have easy option to make robots.txt

    thanks
     
    hostimizer, Nov 25, 2009 IP
  8. varul

    varul Member

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #8
    Not every search engines and crawlers following this robot.txt exclusion protocol standard. Only the genuine search engines and spiders are following this protocol standard.
    Preventing the unwanted robots apart from your sites help you to save more bandwidths
    Configure your robots.txt in good manner. Google, Yahoo, and Bing always look for robot.txt file in every websites.
    There are some examples I given below

    1. Allows web robots to visit all files

    User-agent:*
    Disallow:
    2. Allows all web robots out

    User-agent:*
    Disallow: /
    3. By specifying the directory name between two slashes in Disallow, denies the web spiders to access to those specified directory.

    User-agent:*
    Disallow: /cgi-bin/
    Disallow: /images/
     
    varul, Nov 30, 2009 IP
  9. redesignunit

    redesignunit Peon

    Messages:
    467
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    redesignunit, Dec 4, 2009 IP