1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt

Discussion in 'robots.txt' started by schwab, Nov 12, 2007.

  1. #1
    Hi, im new to search engines. Im curious if robots.txt is needed if i have put <meta name="robots" content="index,follow"> ?
    What should my robots.txt looks like?
     
    schwab, Nov 12, 2007 IP
  2. astup1didiot

    astup1didiot Notable Member

    Messages:
    5,926
    Likes Received:
    270
    Best Answers:
    0
    Trophy Points:
    280
    #2
    I always use robots.txt and not the meta tags.

    www.robotstxt.org
     
    astup1didiot, Nov 12, 2007 IP
  3. snowbird

    snowbird Notable Member

    Messages:
    3,036
    Likes Received:
    395
    Best Answers:
    0
    Trophy Points:
    290
    #3
    The three major search engines support XML sitemaps, and I use my robots.txt to point to it. Here's the info below...

    User-agent: *
    Sitemap: http://www.yourdomainnamehere.com/sitemap.xml
    Code (markup):
    If you nothing of XML sitemaps, go to the URL below and provide your website address. It will create a XML sitemap for you if you have under 500 pages.

    http://www.xml-sitemaps.com/

    By the way, search engines will spider your entire site unless you have noindex and nofollow tags on the pages. A robots.txt file is really not needed, but having it point to a sitemap will help the search engines index your site faster - especially if its a large site.
     
    snowbird, Nov 12, 2007 IP
  4. mariuspompier

    mariuspompier Peon

    Messages:
    323
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You are right Snowbird this is the best way to set up the robots on every site.
    Thanks for sharing with us.
     
    mariuspompier, Nov 12, 2007 IP
  5. Kuldeep1952

    Kuldeep1952 Active Member

    Messages:
    290
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #5
    Even if I have nothing to enter in the robots.txt, that is to say, I want
    all the pages to be indexed, I always create a blank robots.txt. This
    is to reduce unwanted 404 errors (page not found), caused by bots
    trying to get this file.
     
    Kuldeep1952, Nov 13, 2007 IP
  6. schwab

    schwab Peon

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Ok, got it. Thanks for the replies =)
     
    schwab, Nov 13, 2007 IP