Hello, I am wondering how it is possible that slurp could see my links even though I have used mod_rewrite to make them very nice urls. For example, I am using wordpress blog software. I have checked all the links on all of the pages, to make sure of course the mod_rewrite rules worked. Somehow Slurp (and google) have actually spotted a blog posting site.com/index.ph?p=9 when there is really no way they could have found them! Obviously there is a way, or they wouldn't have found him. But what really sucks, is for some reason the post is now indexed under that URL!! The whole reason I have used to mod_rewrite rules is so that they wouldn't be - and my post title URLs would be instead! Arggggh... And I have checked the logs - the referer is "-" ... which stinks. I am not sure how they are finding these. Google is also finding the posts like this, but with a blank "-" referer also. However there is no leak, I promise! The blog is Here. I am very frustrated because I have no links, and for some reason it defies logic. Also, for some reason my page "site.com/" displays adsense ads that are different (and a PSA) then "site.com/index.php" - which are properly targeted ads. I am uncertain how fast the mediapartner googlebot comes to update, but on my other site it seems to happen very often. This is driving me nuts, because I don't want the wrong URLs indexed. The index.php?p=9.php one is on Google now too. Whacky... Finally, my friend and I submitted a total of 3 sites to yahoo with the free submission at the same time the other day. Two of our sites our indexed and have been visited a lot by slurp. The third hasn't even been visited once. That is screwy too - is it just the URL or how is slurp determining whether or not to visit? Is the site possibly blacklisted? We have not submitted it before. However, I am wondering if the submission went thru. I don't want to submit again and get blacklisted, but it is the one we want indexed most! =) Thanks in advance for any help!
Well, it has to have a link somewhere. My guess is someone offsite if linking to the p=9 page. - Shawn
Thanks for the reply Shawn. Of course I have tried the link-path in google, where it is also indexed, and found nothing. I assume that is because anyone linking to it may not have a PR 4 or whatever. I thought I read that somewhere - something about a minimum pagerank for inbounds being recorded. Ah ... whacky. Also, I am not certain how anyone would be able to link to it if I used mod_rewrite to change the links ahead of time .... Also if someone were linking to it, wouldn't it show a referer in the logs? Instead it shows googlebot and "-" as the referer. Also, Googlebot is somehow seeing the other posts with no referer "?p=4" etc... I really have no clue but it doesnt add up. Perhaps I am simply a moron. Edit ** well i am still a moron, but i did find out that somehow the Geourl bot .. www.geourl.screwing with my brain ... is finding the posts as individuals. It is probably linking to me. So I will have to kill this bot, and make sure it knows whos the boss. Also, any ideas why #1 - Slurp chooses only 2 of 3 sites to visit immediately? Is it based on the URL or something (after submission on the yahoo add url) and #2 why in the world would adsense display targeted ads on www.site.com/index.php and different ads (a psa) on www.site.com/ (the more important one)? Thx so much!
Link will show up as backlink on google if it's on the PR4+ page. Otherwise, it's there and counted, but won't show up on google backlink list.
I have seen where google/msn have there own database of links the found in the past, and when I changed the page, they still looked at the old links. And since they still worked, I guess they will keep them. So, maybe google had previously seen that page before your mod-rewite.
Even if you 301 a webpage, Yahoo still keeps the old one. I've had a website 301ed for almost a year, and Yahoo shows the old, not the new in the results. Lots of little things like that really makes me realize how much better Google is from a technical standpoint. I for one took Google for granted until I started seeing some technical limitations of Yahoo's bot. - Shawn