Disallow: / ????

Discussion in 'robots.txt' started by Ridhdhi, Jul 3, 2008.

0
  1. #1
    Hi all,

    I have placed below code robot.txt of my site:

    User-agent: *
    Disallow: /

    Will this stop crawling of my home page supposed name as www.hfg.com/ ???

    Please give me a quick ans??

    I am in great need of this.
     
    Ridhdhi, Jul 3, 2008 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    yes..it'll block your whole website....For more information visit Robots.txt
     
    manish.chauhan, Jul 4, 2008 IP
  3. Ridhdhi

    Ridhdhi Banned

    Messages:
    184
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Ok then what should i write to just make my home page crawl and make other pages disallow?
     
    Ridhdhi, Jul 4, 2008 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    Use these instructions in your Robots.txt:
    User-agent: *
    Disallow: /
    Allow: /index.shtml

    This'll only index your Domain and restrict other internal pages and subfolders....:)
     
    manish.chauhan, Jul 4, 2008 IP
    Ridhdhi likes this.
  5. Ridhdhi

    Ridhdhi Banned

    Messages:
    184
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Hey Manish,
    Thanks for quick reply...reputation added ;)
    can u please tell me what does this shtml stand for???

     
    Ridhdhi, Jul 4, 2008 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    Your welcome....:)
    SHTML stands for Server-parsed HTML. It is a file extension for Web pages that contain server side includes (SSIs)...
    I hope you got your answer...:)
     
    manish.chauhan, Jul 4, 2008 IP
  7. Ridhdhi

    Ridhdhi Banned

    Messages:
    184
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Ridhdhi, Jul 4, 2008 IP
  8. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #8
    manish.chauhan, Jul 4, 2008 IP
  9. Ridhdhi

    Ridhdhi Banned

    Messages:
    184
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Hi Manish,

    That domain name was just for example.I am having an other domain.
    Sorry for miss conception :(


     
    Ridhdhi, Jul 4, 2008 IP
  10. seoorganizers

    seoorganizers Peon

    Messages:
    111
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    all your pages are stopped by you, bots cannot crawls youuuuuu site, remove it firstt
     
    seoorganizers, Aug 13, 2008 IP
  11. khepri

    khepri Peon

    Messages:
    84
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    What best robots.txt config for backlinks.com on wordpress weblog?
     
    khepri, Aug 13, 2008 IP