Problem with robots following links

Discussion in 'Search Engine Optimization' started by aeiouy, Sep 5, 2005.

  1. #1
    I am using a directory script and having some problems. I have identified an issue where robots do not follow the secondary "more info" links. I have come up with all kinds of crazy thoughts, but none of them have worked out.

    Been following this for about a month or so, and am using the script for several different sites, and in no case have I seen a spider hit any of these pages.

    The actual code from the script is:

    <b>{$links.url}&nbsp;</b>
    	   {if $_output.mod_rewrite}
    	    
    	       <a class = 'cat' href='{$_output.this_cat_url}listing{$links.ID}.html'>{$_skalinks_lang.link.more_info}</a>&nbsp;
    	       
    	       {else}
    	       
    	       <a class = 'cat' href='{$_output.this_cat_url}listing.php?link_id={$links.ID}'>{$_skalinks_lang.link.more_info}</a>&nbsp;
    		  
    		{/if}
    Code (markup):
    It is using the smarty template engine, and I believe this whole write is part of a bigger form. Which I think might be part of the problem but I am really blind guessing at this point.

    The output in the source shoulds good anchor text, so I don't know whyt he robots won't follow the second links. Basically the listings look like this:

    <a href="http://www .url . com">Anchor text</a>
    www .url .com [not anchored] - <a href="http://internal url .com">More Info</a>

    I have run it through a few search engine emulators and such and some of the more detailed ones will show the anchor text for the linkback to the site, then the rest doesn't show up at all.

    I am hoping I am just missing something real simple here, and someone can point me in the right direction. I have wasted way too much time at this point trying to straighten this out.

    I have contacted the script author several times on this particular issue, but have not really gotten a response, and he doesn't seem to really see the problem.

    Again, my current thought process is that it has something to do with the form post but I clearly don't have enough understanding to determine that for sure.

    Thanks in advance...
     
    aeiouy, Sep 5, 2005 IP
  2. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #2
    Can you provide the URL?
     
    minstrel, Sep 5, 2005 IP
  3. aeiouy

    aeiouy Peon

    Messages:
    2,876
    Likes Received:
    275
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Here is a link to one of the pages:

    http: //lineage 2.linkedlinks. com/general/

    The more info links are the ones not being followed. Normally I have a mod-rewrite mod turned on that puts them in html, but I recently turned it off.

    I have been doing some searching through other sites using the script and it seems like none of them have any of these pages listed, so it has to be something with the script, I just have no idea what the problem is.
     
    aeiouy, Sep 6, 2005 IP
  4. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #4
    <a class = 'cat' href='http://lineage2.linkedlinks.com/general/listing.php?link_id=5'>More info</a>&nbsp;
    Code (markup):
    Try changing to

    <span class="cat"><a href="http://lineage2.linkedlinks.com/general/listing.php?link_id=5">More info</a></span>&nbsp;
    Code (markup):
     
    minstrel, Sep 6, 2005 IP
  5. aeiouy

    aeiouy Peon

    Messages:
    2,876
    Likes Received:
    275
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thanks Minstrel... I will give make the change and let you know how it works out.
     
    aeiouy, Sep 6, 2005 IP
  6. TechEvangelist

    TechEvangelist Guest

    Messages:
    919
    Likes Received:
    140
    Best Answers:
    0
    Trophy Points:
    133
    #6
    Hi Minstrel

    Why would moving the class attribute to a span tag make a difference with spiderability? Have you seen this affect link spiderability or are you speculating?

    It's pretty common to include a class attribute in an anchor tag, but I have never seen it affect spiderability. The only difference with the way I use it is that the class attibute is normally placed after the src attribute, but that really should not make a difference.

    It does seen strange that the spiders are not following aeiouy's links.
     
    TechEvangelist, Sep 6, 2005 IP
  7. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #7
    Troubleshooting mode. I don't know that's the problem but I do know that the suggested change should work.

    As an aside, in the original with the class attribute, should those be single ' marks or double " marks? I don't use that format...
     
    minstrel, Sep 6, 2005 IP
  8. aeiouy

    aeiouy Peon

    Messages:
    2,876
    Likes Received:
    275
    Best Answers:
    0
    Trophy Points:
    0
    #8
    The quotation thing has me wondering now. In the source that link has everything with single quotes, but I look at other anchor text and it is all using double-quotes around the variables.

    I did see, though, that the category links on the main page use the same <a class="cat" href='> format as the more info pages, and the spiders seem to be getting through as well.. Anyways.. made the changes will wait and see what happens. :)
     
    aeiouy, Sep 6, 2005 IP
  9. noppid

    noppid gunnin' for the quota

    Messages:
    4,246
    Likes Received:
    232
    Best Answers:
    0
    Trophy Points:
    135
    #9
    The comments are not formatted properly thus the parser chokes.
    There is no character encoding meta tag
    There are unbalanced tags.
    There are no alt tags on some images.
    There are duplucate ID tags.
    The document declaration spans multiple lines.


    The parser is getting so many errors G probably just runs away. :/
     
    noppid, Sep 6, 2005 IP
  10. aeiouy

    aeiouy Peon

    Messages:
    2,876
    Likes Received:
    275
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Yeah I did run it through a validator early this morning and saw all that...

    Not sure how much of that stuff is impacting, if any... will take it one step at a time...but you do raise a valid point.

    I am still fairly new to all this so I just don't have a good grasp on the intracacies of what does and does not cause problems.. The only thing I know is spread across a bunch of sites with the same software, I have never seen a search engine go to one of these extended links.. So I will take the first changes.. and then go from there...

    Again thanks to the feedback from everybody.
     
    aeiouy, Sep 6, 2005 IP
  11. aeiouy

    aeiouy Peon

    Messages:
    2,876
    Likes Received:
    275
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Just to follow up, beyond those changes, I also made an html sitemap for my directories... I had been using the google sitemap previously, and it didn't seem to encourage google.

    Since then I have seen a few spiders get to my individual listings, so I am just going to wait and see at this point. I just saw yahoo find a few this morning and MSN found some the other day. Haven't seen google land on any yet, but I will give it some more time. seems like they are at least locatable now.

    Thanks for the help.
     
    aeiouy, Sep 11, 2005 IP