1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robot.txt help

Discussion in 'robots.txt' started by blankeyecue, Jul 27, 2010.

0
  1. #1
    blankeyecue, Jul 27, 2010 IP
  2. bigmax

    bigmax Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    A robots.txt doesn't have any keywords. It's just a file telling the search bots which directories to look at and which to ignore. It also indicates the location of your site's sitemap.

    An example robots.txt may contain

    User-agent: *
Allow: /
Sitemap: http://buycheap-oem-software.com/sitemap.xml
    Code (markup):
    and would be stored as "robots.txt" in the root directory of your domain (i.e. the top-most location).
     
    bigmax, Aug 1, 2010 IP
  3. lightup

    lightup Peon

    Messages:
    221
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Ive heard that some people think that adding a sitemap pointed such as this may cause the robots.txt file to become un validated?
    What is your opinion of this?

    Kind regards
     
    lightup, Sep 16, 2010 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    All crap..:) this is the ideal format one should use in his website's robots.txt. The reason sitemap should be put in the robots file is that since robots visit robots.txt file first when they arrive on a website, so they would also visit our sitemap.xml file as well and could crawl our website better.
     
    manish.chauhan, Sep 16, 2010 IP
  5. The_Hatta

    The_Hatta Greenhorn

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #5
    Adding a Site map to your Robots.txt is a normal way of adding a sitemap to your website, all major Search engines use it ( Bing ASK Google) and will not make your Robots.txt un-verified

    Heres an example of mine
    
User-agent: *
Allow: /
Sitemap: http://www.fta-n-more.net/sitemap.xml
    Code (markup):
    When I ran all the test on it , it says it's valid , and Google's webmaster tools Now caries my sitemap, so you can use the same formate...

    If you need to create a sitemap, theres tons of FREE websites to do it for you...
    Best of luck ;)
     
    The_Hatta, Sep 20, 2010 IP