Hi All, For the last 6 months our site has been under severe brute force, syn flood attack. They keep bombarding a single URL of the server and it is xml file. They are not attacking any other URL. e.g. http://www.example.com/rss123/attackedfilename.xml We have removed the xml page from our site but still they keep on sending requests, this is for the last 6 months non stop. The IP has been changed just to see and they are sending several thousand requests per second. The requests come from different IPS and different ranges, so you can not even block the IP’s. They seem to be coming from a legitimate IP’s. Due to this I have had to pay for an extremely expensive server which holds 8 GB of RAM and quad core processor etc, however, even with this the server server still reaches critical level, just because these requests are eating up my resources. Our technical team has been working on all aspects of apache server security, external modules, firewall, hardware firewall from beginning but still we are not able to stop them. We have installed following modules. 1) mod_security 2) mod_evasive 3) Firewall We have worked with the hosting company and their technical team leader, he installed the best CISCO hardware firewall and tried to stop them, but in vain. We have checked our server to see if anything from our site is causing the request, no extra file uploaded on to the server. For example if some file has been upload or some text has been added to the file (checked if we’ve been hacked). Even though we checked for any hacks, I am still wondering if there is something we do not know about. Can a hack lead to huge amounts of traffic? We need some help to stop these attacks. We have searched a lot and have found that sites that get attacked like this have only one option is to shut down till it stops. I really hope that will not be the case for us. Please let us know if any one has any ideas to deal with this. We are willing to try any small suggestion which might help from coding to scripting to modules to firewall. So please provide suggestion/solutions and way to get us out of this. Also could it be our own part of php code which can do this? We are ready to check every php file to make sure it does not have any line of code which can be dangerous? Thank you for your help in advance! Help! Regards, Sam
Hi, I just had a long meeting with my technical team and following are the updates. 1) SYN cookies is already Enable but not making much difference. 2) I will provide the packet details soon for the request to give the extra idea for the attack. 3) Following are the details I have got from my linux admin. This will help you to trace the issue in better way. Problem: Apache SYN_RECV OS - RHEL5 kernels - 2.6.18-92.1.22.el5-x86_64 2.6.18-92.el5-x86_64 rpms:- kernel-devel-2.6.18-92.el5 kernel-headers-2.6.18-92.1.22.el5 kernel-devel-2.6.18-92.1.22.el5 kernel-2.6.18-92.1.22.el5 kernel-2.6.18-92.el5 OS Type: cat /etc/issue Red Hat Enterprise Linux Server release 5.2 (Tikanga) > cat /proc/version Linux version 2.6.18-92.1.22.el5 (mockbuild@hs20-bc2-5.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Fri Dec 5 09:28:22 EST 2008 We are providing 403 code for the URL request. Following is part of Access Log 94.70.118.139 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; FunWebProducts; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)" 89.216.230.148 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; sr; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10" 82.81.54.226 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; CT2077543_4.5.191.15)" 85.229.15.86 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 2.0.50727; CT2088752_4.5.188.7)" 92.237.189.17 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10" 84.106.127.218 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)" 87.93.30.98 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.188.7)" 93.86.61.247 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.0" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; CT2077543_4.5.188.7)" 91.152.228.27 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; Creative ZENcast v2.00.13; CT2088347_4.5.188.7)" 94.69.164.32 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322; .NET CLR 2.0.50727; CT2088700_4.5.191.15)" 82.201.180.177 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6; CT2077543_4.5.189.28)" 83.248.2.230 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SIMBAR={46BC3752-9118-483D-8E88-CD3E89FCD192}; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2088752_4.5.188.7)" 99.235.137.30 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; CT2077543_4.5.191.15)" 216.155.136.84 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506; .NET CLR 1.1.4322; CT2077543_4.5.188.7)" 217.123.166.205 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088433_4.5.188.7)" 86.96.227.88 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; CT2077543_4.5.188.7)" 203.115.189.77 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; CT2077543_1.5.48.2; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10" 203.82.79.102 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; CT2088347_4.5.188.7)" 88.195.52.126 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.5.30729; .NET CLR 3.0.30618; CT2088347_4.5.189.24)" 77.81.114.171 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; SIMBAR={B471FCBA-22ED-11DE-91A3-00196693641D}; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; CT2077543_4.5.191.15)" 92.84.250.65 - - [11/Jun/2009:04:46:32 -0500] "GET /rss/test.xml HTTP/1.1" 403 292 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; CT2077543_4.5.191.15)" netstat: tcp 0 0 domain.com:http 85-156-91-20.elisa-mo:55168 SYN_RECV tcp 0 0 domain.com:http 220.255.7.227:27183 SYN_RECV tcp 0 0 domain.com:http 5e03cbc4.bb.sky.com:51086 SYN_RECV tcp 0 0 domain.com:http 79.126.234.198:18139 SYN_RECV tcp 0 0 domain.com:http 78.148.175.148:11115 SYN_RECV tcp 0 0 domain.com:http 83-154-143-68.rev.lib:61479 SYN_RECV tcp 0 0 domain.com:http ABTS-North-Static-248:54775 SYN_RECV tcp 0 0 domain.com:http 90-230-131-95-no130.tb:1134 SYN_RECV tcp 0 0 domain.com:http static-host119-73-6-2:49538 SYN_RECV tcp 0 0 domain.com:http 222.127.130.238:gtp-control SYN_RECV tcp 0 0 domain.com:http acl1-1571bts.gw.smartbr:g5m SYN_RECV tcp 0 0 domain.com:http athedsl-282427.home.o:60002 SYN_RECV tcp 0 0 domain.com:http CPE-58-166-77-138.nsw:60067 SYN_RECV tcp 0 0 domain.com:http C-59-101-99-107.syd.c:51097 SYN_RECV tcp 0 0 domain.com:http ti0111a380-2667.bb.on:60993 SYN_RECV tcp 0 0 domain.com:http 92.81.2.242:60451 SYN_RECV tcp 0 0 domain.com:http 118.100.120.248server SYN_RECV tcp 0 0 domain.com:http triband-del-59.178.84:50140 SYN_RECV tcp 0 0 domain.com:http cpc4-leds5-0-0-cust82brpd SYN_RECV tcp 0 0 domain.com:http ALyon-153-1-8-78.w86-:59494 SYN_RECV tcp 0 0 domain.com:http 120.28.199.183:3comnetman SYN_RECV tcp 0 0 domain.com:http h248.4.16.98.dynamic.:60758 SYN_RECV tcp 0 0 domain.com:http 89.211.205.59:64217 SYN_RECV tcp 0 0 domain.com:http CPE-124-187-26-30.qld:ff-sm SYN_RECV tcp 0 0 domain.com:http frw.Gloworld.com:59104 SYN_RECV tcp 0 0 domain.com:http 220.255.7.182:winpoplanmess SYN_RECV tcp 0 0 domain.com:http srisaionline180.excell:1232 SYN_RECV tcp 0 0 domain.com:http CPE-60-230-16-150.vic:52611 SYN_RECV tcp 0 0 domain.com:http 203.82.91.102:41318 SYN_RECV tcp 0 0 domain.com:http 69.171.165.50:32454 SYN_RECV tcp 0 0 domain.com:http dsl-TN-static-195.:corbaloc SYN_RECV tcp 0 0 domain.com:http 210.186.66.179:49330 SYN_RECV tcp 0 0 domain.com:http ABTS-North-D:xinuexpansion3 SYN_RECV tcp 0 0 domain.com:http c122-106-133-46.livrp:49273 SYN_RECV tcp 0 0 domain.com:http 173.subnet125-1:nssalertmgr SYN_RECV tcp 0 0 domain.com:http 121.246.52.30.dynamic:63977 SYN_RECV tcp 0 0 domain.com:http mobile-3G-dyn-BC-179-1:4464 SYN_RECV tcp 0 0 domain.com:http crd48.neoplus.adsl.t:aminet SYN_RECV Following we have done till now is mentioned below for the configurations. ############### sysctl.conf ############## # Kernel sysctl configuration file for Red Hat Linux # # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and # sysctl.conf(5) for more details. # Controls IP packet forwarding net.ipv4.ip_forward = 0 # Controls source route verification net.ipv4.conf.default.rp_filter = 1 # Do not accept source routing net.ipv4.conf.default.accept_source_route = 0 # Controls the System Request debugging functionality of the kernel kernel.sysrq = 0 # Controls whether core dumps will append the PID to the core filename # Useful for debugging multi-threaded applications kernel.core_uses_pid = 1 # Controls the use of TCP syncookies net.ipv4.tcp_syncookies = 1 # Controls the maximum size of a message, in bytes kernel.msgmnb = 65536 # Controls the default maxmimum size of a mesage queue kernel.msgmax = 65536 # Controls the maximum shared segment size, in bytes kernel.shmmax = 68719476736 # Controls the maximum number of shared memory segments, in pages kernel.shmall = 4294967296 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_synack_retries = 2 # Enable IP spoofing protection, turn on Source Address Verification net.ipv4.conf.all.rp_filter = 1 # Enable TCP SYN Cookie Protection net.ipv4.tcp_syncookies = 1 # 65536 seems to be the max it will take net.ipv4.ip_conntrack_max = 1048576 net.ipv4.tcp_rmem = 4096 87380 8388608 net.ipv4.tcp_wmem = 4096 87380 8388608 net.core.rmem_max = 8388608 net.core.wmem_max = 8388608 net.core.netdev_max_backlog = 5000 net.ipv4.tcp_window_scaling = 1 ############# fwsnort, bfd burnintest chkrootkit ddos faf lsm nobody_check sim apf ############# modsecurity-apache LoadModule evasive20_module /usr/lib64/httpd/modules/mod_evasive20.so <IfModule mod_evasive20.c> DOSHashTableSize 3097 DOSPageCount 3 DOSSiteCount 50 DOSPageInterval 1 DOSSiteInterval 1 DOSBlockingPeriod 30 </IfModule> LoadModule security_module /usr/lib64/httpd/modules/mod_security.so <IfModule mod_evasive20.c> DOSHashTableSize 3097 DOSPageCount 3 DOSSiteCount 50 DOSPageInterval 1 DOSSiteInterval 1 DOSBlockingPeriod 30 </IfModule> #################### Again, Hope this helps you to see the issue in details. I have also put the latest configurations to keep site going on. If it may be where server has some maleware or spyware, please let me know the solution and we can scan the pc for that too. Besides these, please let me know for any suggestion you think will be helpful. Thanks again for your time and have a wonderful day. Sam
Hi, Thanks for the details. I will post the details there too. Yesterday, I sent few details from the server configurations and access log files. Please review them to see if anything I can check or update. Can it be some malware or virus hosted on the server? We have simple antivirus scanner but if needed we can try or buy the latest one. Please advice on this. Again, PHP code I have checked but if you know any piece of code which can be problematic, please let me know and we can check for that too. Thanks in advance and have a great day. Sam
This is easy to mitigate. Create a script to log every IP hitting the test.xml. Have that IP added to your firewall. Run this on a cron job every 2 minutes. Start gathering the IPs and see if you can't see a pattern like a certain ISP or country. You can't stop the attack..you can only nullify it's effects. I deal with DDOS attacks all the time and have mitigated as much as 50,000 pps. And you might be hacked. It's incredibly rare that for 6 months an attack like this would go on. Can you PM me the url of the actual site? I will see if I can find anything. I run hackforums.net so I can put the word out and see who knows what.
Hi, @RectangleMan, Thanks for your valuable suggestions. This solution is already on and we are blocking such IP from firewall as they are coming. Its really sad that we can't stop such attack which can stop any site. Yeah, attack is now running more then 6 months its really rare but that what happening to us. As you have handle biggest attack of Ddos, can you please pass me the steps you have taken and I can verify if we have taken all those steps already. Sam
If your site is up and running normally then you probably have done all you can on your end. You can consider creating a template email to send out to ISP's that run the IP addresses associated with the attack. It's bothersome but you may not have a choice. Do you know if there is a specific reason for the attack? Is this a competitor? Is your site international or basically English based? What you can do is use your firewall as an allow only instead of deny. Allow US, CA, UK, AU or whatever other countries you feel are important. This would eliminate a large portion of the attack probably.
Hi, Thanks. We will work on regarding sending details to ISP; however, how can they help us? Any idea? Also please pass on the details how to contact them and if they can help us stopping this attack, we will surly go for this option, no matter, how much complex it is. Also, regarding the firewall, Its a good idea but we are general public site and will love to be global so allowing only some of countries may effect our traffic and popularity. You have any idea for Firewall which drops all request at entry point for only single url? The best firewalls we have tried, all are IP based and they make rules around them only.. Sam
Most ISPs use abuse@ for contact. They can help by contacting the user informing them of the compromised system. Believe it or not some of them do care. The fact their network is being used to attack you can present a lawsuit where they are liable if they do nothing about it. You could potentially log all your contacts and start a suit if they don't respond favorably. I would carefully word the contact to be appealing and yet threatening. Make it known the importance of this and how your business is being adversely effected by their network and users. Demand results and a response.
Hi, Great. Thanks for the wonderful details. We will see in detail for the same. This is something new and very sensitive so we need to take steps carefully. Besides this, again, you have any idea for Firewall which will drop request coming for my site URL? If this is done, we are done with the attack and site will be restored again. Thanks again for all your time and details. Sam
Hi all, Scanned the server with rootkit antispyware, no infection found. Regarding the firewall, put on BFD firewall over APF, still requests are not getting down. Also IP table is getting full of new ips and it is keeping network and site slow. Please advice for next steps to improve the performace. Sam
mod_security is very good as an IDS, but like every application firewall it takes the request an applies hundreds of regex to it, before it can be rejected if recognized as an attack. This can take a lot of cpu power. It is my opinion that in your case you should do anything possible to save cpu power. Try disabling mod_security and see if more resourced become available. I don't know how you are stopping those request, but if mod_security does hundreds/thousands of checks on each one you are wasting resources. The flood request must not reach mod_security. Imho after 6 months you are in damage control mode. If you couldn't stop the flood till now, what makes you think anything is going to change. Contain the situation and save resources.