1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.
  2. Better Analytics for WordPress Get It Free

robots.txt

Discussion in 'robots.txt' started by schwab, Nov 12, 2007.

  1. #1
    Hi, im new to search engines. Im curious if robots.txt is needed if i have put <meta name="robots" content="index,follow"> ?
    What should my robots.txt looks like?
     
    schwab, Nov 12, 2007 IP
  2. ssandecki

    ssandecki Peon

    Messages:
    5,927
    Likes Received:
    269
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I always use robots.txt and not the meta tags.

    www.robotstxt.org
     
    ssandecki, Nov 12, 2007 IP
  3. snowbird

    snowbird Notable Member

    Messages:
    3,033
    Likes Received:
    394
    Best Answers:
    0
    Trophy Points:
    240
    #3
    The three major search engines support XML sitemaps, and I use my robots.txt to point to it. Here's the info below...

    User-agent: *
    Sitemap: http://www.yourdomainnamehere.com/sitemap.xml
    Code (markup):
    If you nothing of XML sitemaps, go to the URL below and provide your website address. It will create a XML sitemap for you if you have under 500 pages.

    http://www.xml-sitemaps.com/

    By the way, search engines will spider your entire site unless you have noindex and nofollow tags on the pages. A robots.txt file is really not needed, but having it point to a sitemap will help the search engines index your site faster - especially if its a large site.
     
    snowbird, Nov 12, 2007 IP
  4. mariuspompier

    mariuspompier Peon

    Messages:
    324
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You are right Snowbird this is the best way to set up the robots on every site.
    Thanks for sharing with us.
     
    mariuspompier, Nov 12, 2007 IP
  5. Kuldeep1952

    Kuldeep1952 Active Member

    Messages:
    290
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #5
    Even if I have nothing to enter in the robots.txt, that is to say, I want
    all the pages to be indexed, I always create a blank robots.txt. This
    is to reduce unwanted 404 errors (page not found), caused by bots
    trying to get this file.
     
    Kuldeep1952, Nov 13, 2007 IP
  6. schwab

    schwab Peon

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Ok, got it. Thanks for the replies =)
     
    schwab, Nov 13, 2007 IP