1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How Robots.txt works

Discussion in 'robots.txt' started by Alex Decosta, Apr 14, 2012.

0
  1. #1
    Alex Decosta, Apr 14, 2012 IP
  2. Knocks

    Knocks Well-Known Member

    Messages:
    85
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    100
    #2
    You should really learn the basics of HTML before trying to write this. Also learn about how opening and closing html tags.
     
    Knocks, Apr 20, 2012 IP
  3. AlisaRozzy

    AlisaRozzy Active Member

    Messages:
    103
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #3
    Hello your robots.txt file looking sound good. But there is a issue you can add your XML site map which can be more beneficial for your website ranking.

     
    AlisaRozzy, May 19, 2012 IP
  4. ronnydavis

    ronnydavis Member

    Messages:
    38
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #4
    you need to allow each and every url in your website then you can easily get rank for your website.
     
    ronnydavis, Jul 5, 2012 IP
  5. ryan_uk

    ryan_uk Illustrious Member

    Messages:
    3,983
    Likes Received:
    1,022
    Best Answers:
    33
    Trophy Points:
    465
    #5
    There is no such parameter as "Allow:" for robots.txt. Remove that line. Really, the robots.txt is used for disallowing (or telling robots where the sitemap file is).

    Read up on it here: http://www.robotstxt.org/orig.html
     
    ryan_uk, Aug 5, 2012 IP
  6. mygold

    mygold Peon

    Messages:
    1,164
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #6
    As robots.txt is for disallowing links to be crawled so there is no room in robots.txt for the word "allow:"
     
    mygold, Aug 5, 2012 IP
  7. XpErT

    XpErT Member

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #7
    check this out here is a good document about robots.txt seomoz.org/beginners-guide-to-seo/search-engine-tools-and-services
     
    XpErT, Aug 5, 2012 IP