This probably doesn't really go here, but I can;t think of a better place. I'm getting a huge number of referrals from www.[somesite].com\robots.txt showing up in my server logs. For example: blog.360.yahoo.com/robots.txt blognichi.blogs.friendster.com/robots.txt p.moreover.com/robots.txt rotoauthority.blogs.com/robots.txt z.about.com/robots.txt www.vertical-imbalance.com/robots.txt www.smarttravelnews.com/robots.txt What the heck?
I clicked the links and they are just a redirect I surfed to the site and did it without clicking the link Placing the URL then the robots.txt taked you to Typepads robots.txt http://6a.typepad.com/robots.txt User-agent: * Disallow: /t/comments Disallow: /t/stats Disallow: /t/app # weird MSIE thing that keeps hammering User-agent: Active Cache Request Disallow: * I dont know what the heck they have done bit somethings wrong
Thanks, I hadn't actually followed any of the lnks. Why would these be showing up as referrer URLs in my referrer logs?
Hard to say Dont know your site ...cant see you hosting panel could be someone is trying to hijack a website Could be a blog post somewhere else on typepad and that is what any referal link from a typedpad hosted site looks like?? Problem with websites is their is no off the hat answer usually... without seeing more information most answers are best guess... Certainly not something I would want to pin down if I am not very sure Hope that help
Well, I suppose they could be hoping my logs are public, and thus they would be getting several free links to typad. My logs aren't public, so it looks like this is probably just a waste of someone's time. I guess I just missed the obvious by not checking the links.
That's pretty common. I suspect it's done by scripts anyway so they're not even bothering to check whether the links will ever be seen by anyone but the webmaster.
Yeah, I have seen it several times, but I've never seen the referrer url be a robots.txt file. Usually I'd see a real URL like joespornpalace.com. They must be getting sneakier. Figuringing that someone would delete joespornpalace but leave a somesite.com/robots.txt link.