1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Yahoo ignoring robots.txt

Discussion in 'Yahoo' started by jeff123, Aug 16, 2005.

  1. #1
    I have the following in my robots.txt

    User-agent: *
    Crawl-delay: 3
    Disallow: /verifypic.php
    Disallow: verifypic.php

    Yet, when I do a site command on Yahoo, it lists thousands of urls (urls only - no cache or title) of this script. Could anyone explain what's going on? Maybe this is how they've increase their index size. By the way, this script is used to serve an image and is only used like
    <img widht="50" height="16" src="/verifypic.php?v=1124209720">
     
    jeff123, Aug 16, 2005 IP
  2. Willy

    Willy Peon

    Messages:
    281
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Have you had the robots.txt in place all the time, or did you just recently implement it? If the latter, perhaps it takes time for Y! to delete the disallowed URLs?
     
    Willy, Aug 16, 2005 IP
  3. jeff123

    jeff123 Well-Known Member

    Messages:
    251
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    108
    #3
    It's been in place before Yahoo indexed the first page - 8 or 9 months.
     
    jeff123, Aug 16, 2005 IP
  4. 802networks

    802networks Guest

    Messages:
    132
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I have this very same problem. I've also noticed it with Google too. From what i've read, is that the robots.txt only tells the search engines which pages not to crawl. It doesn't mean they won't index the page without a title or description. They also 'claim' that this has no effect on your rankings.
     
    802networks, Aug 18, 2005 IP
  5. programmer

    programmer Guest

    Messages:
    444
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    0
    #5
    that's right - Yahoo and google both ignore it as my experience says -
     
    programmer, Aug 18, 2005 IP