1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt query

Discussion in 'robots.txt' started by ian_ok, Nov 2, 2004.

0
  1. #1
    Hi all,

    Just a quick question here.

    If I was to link to the site below *** replaces the real website, would the bot not crawl the directory I would be listed in?

    Page I would be listed in....

    http://www.*****.com/directory/dir-26-0.html

    Robots file...
    Disallow: /directory/
    Disallow: /directory/links/

    Thanks

    Ian
     
    ian_ok, Nov 2, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Right, it would not be crawled.
     
    digitalpoint, Nov 2, 2004 IP
  3. ian_ok

    ian_ok Peon

    Messages:
    551
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Just thought you might like to know the url from where the link is:

    http://www.carslots.com

    Remember to always check the robots file...

    Thanks for confirming this Shawn..

    Not sure if you have a Black list Shawn?

    Ian
     
    ian_ok, Nov 2, 2004 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    Black list of what, sites that block spiders? I have other things to worry about (I don't do link exchanges so...) :)
     
    digitalpoint, Nov 2, 2004 IP