Referrer logs show odd referrer: somesite.com\robots.txt

[Tim_Myth]

This probably doesn't really go here, but I can;t think of a better place.

I'm getting a huge number of referrals from www.[somesite].com\robots.txt showing up in my server logs. For example:

blog.360.yahoo.com/robots.txt
blognichi.blogs.friendster.com/robots.txt
p.moreover.com/robots.txt
rotoauthority.blogs.com/robots.txt
z.about.com/robots.txt
www.vertical-imbalance.com/robots.txt (nofollow)
www.smarttravelnews.com/robots.txt (nofollow)


What the heck?

[Sem-Advance]

I clicked the links and they are just a redirect

I surfed to the site and did it without clicking the link

Placing the URL then the robots.txt taked you to Typepads robots.txt

http://6a.typepad.com/robots.txt

User-agent: *
Disallow: /t/comments
Disallow: /t/stats
Disallow: /t/app

# weird MSIE thing that keeps hammering
User-agent: Active Cache Request
Disallow: *
I dont know what the heck they have done bit somethings wrong

[Tim_Myth]

Thanks, I hadn't actually followed any of the lnks. Why would these be showing up as referrer URLs in my referrer logs?

[Sem-Advance]

Hard to say

Dont know your site ...cant see you hosting panel

could be someone is trying to hijack a website

Could be a blog post somewhere else on typepad and that is what any referal link from a typedpad hosted site looks like??

Problem with websites is their is no off the hat answer usually... without seeing more information most answers are best guess...

Certainly not something I would want to pin down if I am not very sure

Hope that help

[Tim_Myth]

Well, I suppose they could be hoping my logs are public, and thus they would be getting several free links to typad. My logs aren't public, so it looks like this is probably just a waste of someone's time. I guess I just missed the obvious by not checking the links.

[Sem-Advance]

No problems

Glad I could help even though I didn't

:D

[minstrel]

Well, I suppose they could be hoping my logs are public, and thus they would be getting several free links to typad. My logs aren't public, so it looks like this is probably just a waste of someone's time.
That's pretty common. I suspect it's done by scripts anyway so they're not even bothering to check whether the links will ever be seen by anyone but the webmaster.

[Tim_Myth]

Yeah, I have seen it several times, but I've never seen the referrer url be a robots.txt file. Usually I'd see a real URL like joespornpalace.com. They must be getting sneakier. Figuringing that someone would delete joespornpalace but leave a somesite.com/robots.txt link.