1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Slurp Sees my files - w/o links??

Discussion in 'Yahoo' started by shina21, May 3, 2004.

  1. #1
    Hello,

    I am wondering how it is possible that slurp could see my links even though I have used mod_rewrite to make them very nice urls.

    For example, I am using wordpress blog software. I have checked all the links on all of the pages, to make sure of course the mod_rewrite rules worked. Somehow Slurp (and google) have actually spotted a blog posting site.com/index.ph?p=9 when there is really no way they could have found them!

    Obviously there is a way, or they wouldn't have found him. But what really sucks, is for some reason the post is now indexed under that URL!! The whole reason I have used to mod_rewrite rules is so that they wouldn't be - and my post title URLs would be instead! Arggggh...

    And I have checked the logs - the referer is "-" ... which stinks. I am not sure how they are finding these.

    Google is also finding the posts like this, but with a blank "-" referer also. However there is no leak, I promise! The blog is Here.

    I am very frustrated because I have no links, and for some reason it defies logic. Also, for some reason my page "site.com/" displays adsense ads that are different (and a PSA) then "site.com/index.php" - which are properly targeted ads. I am uncertain how fast the mediapartner googlebot comes to update, but on my other site it seems to happen very often.

    This is driving me nuts, because I don't want the wrong URLs indexed. The index.php?p=9.php one is on Google now too. Whacky...

    Finally, my friend and I submitted a total of 3 sites to yahoo with the free submission at the same time the other day. Two of our sites our indexed and have been visited a lot by slurp. The third hasn't even been visited once. That is screwy too - is it just the URL or how is slurp determining whether or not to visit? Is the site possibly blacklisted? We have not submitted it before. However, I am wondering if the submission went thru. I don't want to submit again and get blacklisted, but it is the one we want indexed most! =)
    SEMrush
    Thanks in advance for any help!
     
    shina21, May 3, 2004 IP
    SEMrush
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,326
    Likes Received:
    2,614
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Well, it has to have a link somewhere. My guess is someone offsite if linking to the p=9 page.

    - Shawn
     
    digitalpoint, May 3, 2004 IP
  3. shina21

    shina21 Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks for the reply Shawn. Of course I have tried the link-path in google, where it is also indexed, and found nothing. I assume that is because anyone linking to it may not have a PR 4 or whatever. I thought I read that somewhere - something about a minimum pagerank for inbounds being recorded. Ah ... whacky. Also, I am not certain how anyone would be able to link to it if I used mod_rewrite to change the links ahead of time .... Also if someone were linking to it, wouldn't it show a referer in the logs? Instead it shows googlebot and "-" as the referer. Also, Googlebot is somehow seeing the other posts with no referer "?p=4" etc... I really have no clue but it doesnt add up. Perhaps I am simply a moron.

    Edit ** well i am still a moron, but i did find out that somehow the Geourl bot .. www.geourl.screwing with my brain ... is finding the posts as individuals. It is probably linking to me. So I will have to kill this bot, and make sure it knows whos the boss.

    Also, any ideas why #1 - Slurp chooses only 2 of 3 sites to visit immediately? Is it based on the URL or something (after submission on the yahoo add url) and #2 why in the world would adsense display targeted ads on www.site.com/index.php and different ads (a psa) on www.site.com/ (the more important one)?

    Thx so much!
     
    shina21, May 3, 2004 IP
  4. eCommando

    eCommando Well-Known Member

    Messages:
    115
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    103
    #4
    Link will show up as backlink on google if it's on the PR4+ page. Otherwise, it's there and counted, but won't show up on google backlink list.
     
    eCommando, May 14, 2004 IP
  5. dsr771

    dsr771 Peon

    Messages:
    65
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I have seen where google/msn have there own database of links the found in the past, and when I changed the page, they still looked at the old links. And since they still worked, I guess they will keep them.

    So, maybe google had previously seen that page before your mod-rewite.
     
    dsr771, May 16, 2004 IP
  6. nlopes

    nlopes Guest

    Messages:
    103
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    You can do a 301 redirect on those dynamic pages to the static ones.
     
    nlopes, May 17, 2004 IP
  7. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,326
    Likes Received:
    2,614
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #7
    Even if you 301 a webpage, Yahoo still keeps the old one. I've had a website 301ed for almost a year, and Yahoo shows the old, not the new in the results. Lots of little things like that really makes me realize how much better Google is from a technical standpoint. I for one took Google for granted until I started seeing some technical limitations of Yahoo's bot.

    - Shawn
     
    digitalpoint, May 17, 2004 IP