robots.txt file - causing problems??

Discussion in 'Search Engine Optimization' started by leroyboseevus, Mar 29, 2008.

  1. #1
    Hi guys

    I've been trying to get Google to update its listings for my site but it won't. Been trying for weeks and weeks!

    Here is the robots.txt file I am using:

    User-agent: *
    Disallow: /private.html

    Is this causing a problem? I am trying to block only private.html from being listing, not the whole site!

    It's really frustrating!!!
     
    leroyboseevus, Mar 29, 2008 IP
  2. cashmachine101

    cashmachine101 Peon

    Messages:
    637
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Create a sitemap and add the following line to your robots.txt file.

    Sitemap: http:// yourdomain dot com / sitemap.xml (without the spaces)

    Also, make sure in your index.php or index.html you have the following line.

    <meta name="robots" content="follow" />
     
    cashmachine101, Mar 29, 2008 IP
  3. delhidon

    delhidon Banned

    Messages:
    567
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #3
    You have added correct line. The suggestion you have received is ok. I can write you the name of a website where you can verify your website. It is http://www.robotstxt.org/
     
    delhidon, Mar 29, 2008 IP
  4. coolslko

    coolslko Active Member

    Messages:
    177
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    58
    #4
    Do not use space after user agent: and Disallow:

    use as below-

    User-agent:*
    Disallow:/private.html
     
    coolslko, Mar 29, 2008 IP
  5. everythingisplanned

    everythingisplanned Peon

    Messages:
    365
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #5
    everythingisplanned, Mar 29, 2008 IP