1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Layman's guide to robots.txt (and sitemap)

Discussion in 'robots.txt' started by ljk6802, Jan 24, 2011.

0
  1. #1
    I currently have the following robots.txt in the root level of my site:

    User-agent: *
    Allow: /

    Will this allow the indexing of all directories below the root? Is there anything else I should add?

    also (somewhat related).. how does the sitemap.xml impact the indexing?
     
    ljk6802, Jan 24, 2011 IP
  2. illianafrazer

    illianafrazer Well-Known Member

    Messages:
    393
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    138
    #2
    Yes it will allow indexing/ or in other words will permit the search ngines to visit your site. but it is left to the crawlers to decide if they should index an what they should index.

    Robots.txt is for the so called Good robots only :)
     
    illianafrazer, Feb 3, 2011 IP
  3. ljk6802

    ljk6802 Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ok. thanks
     
    ljk6802, Feb 3, 2011 IP
  4. srisen2

    srisen2 Peon

    Messages:
    359
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    srisen2, Feb 4, 2011 IP
  5. ljk6802

    ljk6802 Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    thanks!

    I've noticed that blogger.com has "deny all" in the automatically generated robots.txt. Any way around that?
     
    ljk6802, Feb 4, 2011 IP
  6. srisen2

    srisen2 Peon

    Messages:
    359
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    not if your sites on their servers youd have to take that up with them
     
    srisen2, Feb 7, 2011 IP
  7. jitendraag

    jitendraag Notable Member

    Messages:
    3,982
    Likes Received:
    324
    Best Answers:
    1
    Trophy Points:
    270
    #7
    Most people also like to specify location of their sitemap file in robots.txt

    By default an empty file or a missing robots.txt would also result into your site getting indexed.
     
    jitendraag, Feb 7, 2011 IP
  8. nasul

    nasul Peon

    Messages:
    102
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    It is good so. You should not modify it if you don not know the codes.
     
    nasul, Feb 7, 2011 IP
  9. ljk6802

    ljk6802 Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I would want google to pickup my blogger.com pages.. but the default is "DENY ALL" - is there a way to change that?
     
    ljk6802, Feb 7, 2011 IP