Hello, I own blitzgamer.com and currently have been getting spam bots coming to my site through google (I think) and writing reviews on my site that are just plain spam about various things. What I don't understand is that you have to be a member to write a review, the reviews in the admin panel shows the review to not have a member associated with it. SO they are not a member.. I am pretty confused on the whole thing. If anyone can help much thanks and rep will be awarded! The site is being ran with a custom php script that the person I bought the site from created. Regards, Zack B BlitzGamer.com
I got this robots.txt from another site having the same problem, just drop it into your root directory. User-agent: BotRightHere User-agent: larbin User-agent: b2w/0.1 User-agent: Copernic User-agent: psbot User-agent: Python-urllib User-agent: NetMechanic User-agent: URL_Spider_Pro User-agent: CherryPicker User-agent: EmailCollector User-agent: EmailSiphon User-agent: WebBandit User-agent: EmailWolf User-agent: ExtractorPro User-agent: CopyRightCheck User-agent: Crescent User-agent: SiteSnagger User-agent: ProWebWalker User-agent: CheeseBot User-agent: LNSpiderguy User-agent: Alexibot User-agent: Teleport User-agent: TeleportPro User-agent: MIIxpc User-agent: Telesoft User-agent: Website Quester User-agent: WebZip User-agent: moget/2.1 User-agent: WebZip/4.0 User-agent: WebStripper User-agent: WebSauger User-agent: WebCopier User-agent: NetAnts User-agent: Mister PiX User-agent: WebAuto User-agent: TheNomad User-agent: WWW-Collector-E User-agent: RMA User-agent: libWeb/clsHTTP User-agent: asterias User-agent: httplib User-agent: turingos User-agent: spanner User-agent: InfoNaviRobot User-agent: Harvest/1.5 User-agent: Bullseye/1.0 User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 User-agent: CherryPickerSE/1.0 User-agent: CherryPickerElite/1.0 User-agent: WebBandit/3.50 User-agent: NICErsPRO User-agent: DittoSpyder User-agent: Foobot User-agent: SpankBot User-agent: BotALot User-agent: lwp-trivial/1.34 User-agent: lwp-trivial User-agent: BunnySlippers User-agent: URLy Warning User-agent: Wget/1.6 User-agent: Wget/1.5.3 User-agent: Wget User-agent: LinkWalker User-agent: cosmos User-agent: moget User-agent: hloader User-agent: humanlinks User-agent: LinkextractorPro User-agent: Offline Explorer User-agent: Mata Hari User-agent: LexiBot User-agent: Web Image Collector User-agent: The Intraformant User-agent: True_Robot/1.0 User-agent: True_Robot User-agent: BlowFish/1.0 User-agent: JennyBot User-agent: MIIxpc/4.2 User-agent: BuiltBotTough User-agent: ProPowerBot/2.14 User-agent: BackDoorBot/1.0 User-agent: toCrawl/UrlDispatcher User-agent: suzuran User-agent: TightTwatBot User-agent: VCI WebViewer VCI WebViewer Win32 User-agent: VCI User-agent: Szukacz/1.4 User-agent: Openfind data gatherer User-agent: Openfind User-agent: Xenu's Link Sleuth 1.1c User-agent: Xenu's User-agent: Zeus User-agent: RepoMonkey Bait & Tackle/v1.01 User-agent: RepoMonkey User-agent: Openbot User-agent: URL Control User-agent: Zeus Link Scout User-agent: Zeus 32297 Webster Pro V2.9 Win32 User-agent: Webster Pro User-agent: EroCrawler User-agent: LinkScan/8.1a Unix User-agent: Keyword Density/0.9 User-agent: Kenjin Spider User-agent: Iron33/1.0.2 User-agent: Bookmark search tool User-agent: GetRight/4.2 User-agent: FairAd Client User-agent: Gaisbot User-agent: Aqua_Products User-agent: Radiation Retriever 1.1 User-agent: Flaming AttackBot User-agent: Curl User-agent: Web Reaper User-agent: Firefox User-agent: Opera User-agent: Netscape User-agent: WebVulnCrawl User-agent: WebVulnScan Disallow: / Code (markup): The last two in the list are well know for crawling your site in the quest of vulnerabilities.
So let me see if I understand what you are suggesting. Basically that robots.txt will replace mine which allows all robots.. The list that you provied only allows those robots and no others? While my robots.txt file allows all bots which is leading to the spam.. Is there any disadvantage of limiting to just those bots? Like what if a new search engine comes out or a smaller search engine that isn't listed in the robots.txt file? Anyone else have any ideas opinions on the problem and the robots.txt file? Regards, Zack B
That list avoid the above robots crawl your site. If you want to allow other mix this with your current robot.txt
I don't know much about it.. but it seems adding captcha in the registration page and review pages, should help you. I was getting a lot of spam from the feedback page, i just added captcha and now it is good for me. Let's see if that works for you as well. For ASP: http://sourceforge.net/projects/asp-captcha For PHP: http://www.webhosting.net/forum/showthread.php?p=154 Have a great day!
If it's a custom made script then it has a hole somewhere. The bots are smart enough to search out and post.. I would suggest both adding a captcha and tightening your code. O
Check you forms...75% of all coders leave loose forms,not by fault but they just aren't aware of the security that is required I have bought so many scripts in the past, and the forms on the site were so wide that I could stick an elephant through it Have a PROFESSIONAL coder look at your forms, and tighten them all up I used to have this issue as well, and then I hired some really good people that new php real well and they locked my shit up tight
i had a simple solution to my forums i killed them First 1. Update your php version on your forums - Especially PHPBB - mine got Owned Several Times make sure they are up to date 2. administrator approval on all messages or blog entrys 3. Or just kill the forums from your website - takes the fun out thou....
Hey I have a question to that list you posted. Will that block all users having these browsers or do I missunderstand something?
robots.txt will not help against spammers at all because they ignore the directives. spambots must be fought using technology not polite suggestions.
lol, yes, the robots.txt def didn't stop the spamming.. does anyone know of a good php coder who has knowledge of putting in image verification for posting comments...? Thanks
im using the Textual Confirmation Mod and it has been able to block 95% of spam registrations in the site.
Is that only for phpbb forums? I am looking for just a form submit image verification system. Not for a forum.
You need to go back to the person who wrote it and make them fix it. This has nothing to do with spam bots. There is a simple problem here : Members should be the only ones posting reviews and yet, non-members can post reviews. That is a bad script. There will ALWAYS be spammers, but you don't always have to have a bad script. Blocking spam bots by useragent name means nothing at all. I could crawl your site now with a useragent name from google. You could tell by IP that I was not google, but it would go right by that ridiculously long robots.txt. robots.txt is NOT for security.. AT ALL.
A couple thoughts: 1) A robots.txt will NEVER block spambots because they never obey robots.txt. 2) Implement CAPTCHA on your forms and make sure people have to be logged in to post. Log their IP addresses, and if one of them spams, ban them. 3) When it is released, install Crawlwall. If you want to know more about banning bots, read these blogs: 1) http://willmacc.wordpress.com/ 2) http://spamhuntress.com/category/bots/ 3) http://incredibill.blogspot.com/