Robots exclusion

Discussion in 'Search Engine Optimization' started by frio, Jul 26, 2007.

0
  1. #1
    How would I know if robots are excluding my folders for crawling?

    I've already created a robots.txt wherein I disallow the folders that I don't want to be crawled
     
    frio, Jul 26, 2007 IP
  2. SEMSpot

    SEMSpot Peon

    Messages:
    513
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Easiest way to tell is to do a search for www.domainname/folder and see if any results come up. However if you have it in your robots.txt file then chances are that part of your site is not indexed.
     
    SEMSpot, Jul 26, 2007 IP
  3. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    437
    Best Answers:
    0
    Trophy Points:
    0
    #3
    You'd have to check your server's logs to see if the spiders are indeed not crawling the pages.
     
    Dan Schulz, Jul 26, 2007 IP
  4. MasterG

    MasterG Peon

    Messages:
    252
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Yes you can use google to check your crawl. Easiest. Enjoy :)
     
    MasterG, Jul 26, 2007 IP
  5. frio

    frio Guest

    Messages:
    40
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    frio, Jul 26, 2007 IP
  6. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    437
    Best Answers:
    0
    Trophy Points:
    0
    #6
    What that means is that your links are pointing to www. instead of /

    What does your robots.txt file look like?
     
    Dan Schulz, Jul 26, 2007 IP