Robots.txt question !

Discussion in 'Search Engine Optimization' started by Martin Taylor, Jan 5, 2010.

0
  1. #1
    Can anybody explain me what does this robots.txt code means?

    User-Agent: Googlebot
    Disallow: /*/*/*
    Allow: /

    Thanks in advance !!:):)
     
    Martin Taylor, Jan 5, 2010 IP
  2. Hosting24

    Hosting24 Member

    Messages:
    431
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #2
    This code doesn't make much sense for me. I believe it blocks googlebot from crawling ALL directories under level 3 or deeper, and allows crawling all other directories. But I'm not 100% sure.
     
    Hosting24, Jan 6, 2010 IP
  3. Hosting24

    Hosting24 Member

    Messages:
    431
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #3
    ash05, all of us know what robots file is, but this person has asked a DIFFERENT thing.
     
    Hosting24, Jan 7, 2010 IP
  4. king_me

    king_me Peon

    Messages:
    67
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I am having same problem with blogger.I cant use them to index.
    Can anybody help me in how can I change the robots.txt file in order
    to allow my blogs on search engine?
     
    king_me, Jan 7, 2010 IP
  5. Hosting24

    Hosting24 Member

    Messages:
    431
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #5
    Do you have any robots file now?
     
    Hosting24, Jan 7, 2010 IP
  6. Martin Taylor

    Martin Taylor Active Member

    Messages:
    172
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #6
    I didn't get my answer yet! Please help me to know this, This code exists at my website and I m thinking to remove it but not sure....can I have the logical answer from any body.
    Thanks in advance !!
     
    Martin Taylor, Jan 7, 2010 IP
  7. Hosting24

    Hosting24 Member

    Messages:
    431
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #7
    If you want all pages to be indexed properly, then you should definitely remove it.
     
    Hosting24, Jan 7, 2010 IP
  8. Canonical

    Canonical Well-Known Member

    Messages:
    2,223
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    110
    #8
    If you want your entire site indexed then either
    1) remove the robots.txt file above from your site OR
    2) change it so that it contains ONLY the following so that all crawlers will crawl your entire site:

     
    Canonical, Jan 7, 2010 IP
  9. desperateboy

    desperateboy Peon

    Messages:
    610
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    without robot.txt file website can index by crawler. We can mainly use to restrict some files to index by spider.
     
    desperateboy, Jan 7, 2010 IP
  10. config_error

    config_error Well-Known Member

    Messages:
    1,719
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    130
    #10
    hello there,
    this is just a quick explanation of robot.txt, it is some kind of a file that will tell search engine bots what to crawl and what to index. also it is useful if you dont want Search Engine bot to list your downloads in the search engine results. well this is the part where you will use robots.txt anyway if you need more information about robot.txt you can find more here. http://search-engine-optimization-starter.blogspot.com/2009/12/robotstxt-what-is-robotstxt.html

    i hope this helps.
    regards;
     
    config_error, Jan 7, 2010 IP