Robots.txt query

Discussion in 'robots.txt' started by ian_ok, Nov 2, 2004.

0
  1. #1
    Hi all,

    Just a quick question here.

    If I was to link to the site below *** replaces the real website, would the bot not crawl the directory I would be listed in?

    Page I would be listed in....

    http://www.*****.com/directory/dir-26-0.html

    Robots file...
    Disallow: /directory/
    Disallow: /directory/links/

    Thanks

    Ian
     
    ian_ok, Nov 2, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Right, it would not be crawled.
     
    digitalpoint, Nov 2, 2004 IP
  3. ian_ok

    ian_ok Peon

    Messages:
    551
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Just thought you might like to know the url from where the link is:

    http://www.carslots.com

    Remember to always check the robots file...

    Thanks for confirming this Shawn..

    Not sure if you have a Black list Shawn?

    Ian
     
    ian_ok, Nov 2, 2004 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    Black list of what, sites that block spiders? I have other things to worry about (I don't do link exchanges so...) :)
     
    digitalpoint, Nov 2, 2004 IP