1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

ERR_CONNECTION_RESET for some users

Discussion in 'Apache' started by LuckyTurnips, Aug 2, 2019.

  1. #1
    I have a website on Centos 7.4 running Apache 2.4.6
    Recently I'm getting some users unable to view the site. On trying to load any page they instantly get an ERR_CONNECTION_RESET and the page does not load.

    I managed to recreate this using my 4G (mobile network) connection on my phone, when I use WIFI on my phone the site works fine.
    SEMrush
    Any suggestions for things I could check which could be causing this?

    I use mod_ssl with virtual hosts if that's relevant.

    I cannot find anything in any error logs relating to this error.
     
    LuckyTurnips, Aug 2, 2019 IP
    SEMrush
  2. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #2
    This is an indication that something is interrupted and the connection is reset while trying to load the page. The issue probably lies in your system blocking specific URL and may be a result of changes in the registry, TCP IP or other network configurations or settings. Being technically challenged as I am, I can identify the problem but will leave it to others in how to solve it. Good luck to you with this.
     
    Last edited: Aug 2, 2019
    Spoiltdiva, Aug 2, 2019 IP
  3. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #3
    Thanks for your reply. Network config isn't really my forte either!

    When using the mobile network, I cannot access any page on the site, all give the same error. So it's not a specific URL.
     
    LuckyTurnips, Aug 2, 2019 IP
  4. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #4
    What could be causing problems with your visitors could be as simple as using an antivirus that is non compatible, or pc optimization software or a third party firewall.
     
    Spoiltdiva, Aug 2, 2019 IP
  5. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #5
    I don't think it's an anti-virus on the users part or anything like that. I can recreate the problem on my mobile phone (using the same browser etc.) simply by whether I'm using 4G or WIFI.
     
    LuckyTurnips, Aug 2, 2019 IP
  6. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #6
    Have a second look at your fail2ban [apache] settings.
     
    Spoiltdiva, Aug 2, 2019 IP
  7. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #7
    I've ruled the fail2ban possibility out now after looking at which IPs were currently banned in the apache jail (none). I haven't made any changes to fail2ban in a long time. The issue in this thread started a few days ago.
     
    LuckyTurnips, Aug 2, 2019 IP
  8. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #8
    I believe now that this issue is due to some TCP problem.

    The connection is being reset, but I'm not sure how to debug why.

    Using netstat -s is a bit tricky being that I cannot isolate a failed connection.

    Any way to log more information when a TCP connection is reset?
     
    LuckyTurnips, Aug 3, 2019 IP
  9. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #9
    If you're getting a TCP RESET-1 in the log message it probably means that the server from the inside is sending a reset to the PIX which then instructs the ASA firewall to drop the connection. The ASA then drops the connection and logs RESET-1.
     
    Spoiltdiva, Aug 3, 2019 IP
    tsc likes this.
  10. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #10
    I'm not getting any log messages about anything. I'm trying to get some information about TCP resets. How do I log that info?
     
    LuckyTurnips, Aug 3, 2019 IP
  11. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #11
    As to your problems routers could be doing this, particularly NAT. Anyway try running a packet sniffer such as Wireshark.
     
    Last edited: Aug 3, 2019
    Spoiltdiva, Aug 3, 2019 IP
  12. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #12
    This means nothing to me but it's a tcpdump of a failed (reset) connection attempt:

    12:26:37.001292 IP [IP].threembb.co.uk.55071 > mydomain.com.https: Flags [S], seq 2751361480, win 65535, options [mss 1326,sackOK,TS val 138818001 ecr 0,nop,wscale 8], length 0
    12:26:37.001384 IP mydomain.com.https > [IP].threembb.co.uk.55071: Flags [S.], seq 2839564584, ack 2751361481, win 28960, options [mss 1460,sackOK,TS val 229549639 ecr 138818001,nop,wscale 7], length 0
    12:26:37.004298 IP [IP].threembb.co.uk.55072 > mydomain.com.https: Flags [S], seq 1324045316, win 65535, options [mss 1326,sackOK,TS val 138818003 ecr 0,nop,wscale 8], length 0
    12:26:37.004344 IP mydomain.com.https > [IP].threembb.co.uk.55072: Flags [S.], seq 4077890442, ack 1324045317, win 28960, options [mss 1460,sackOK,TS val 229549642 ecr 138818003,nop,wscale 7], length 0
    12:26:37.155320 IP [IP].threembb.co.uk.55072 > mydomain.com.https: Flags [.], ack 1, win 343, options [nop,nop,TS val 138818059 ecr 229549642], length 0
    12:26:37.155497 IP [IP].threembb.co.uk.55072 > mydomain.com.https: Flags [R.], seq 1, ack 1, win 32120, length 0
    12:26:37.162855 IP [IP].threembb.co.uk.55071 > mydomain.com.https: Flags [.], ack 1, win 343, options [nop,nop,TS val 138818059 ecr 229549639], length 0
    12:26:37.163614 IP [IP].threembb.co.uk.55071 > mydomain.com.https: Flags [R.], seq 1, ack 1, win 32120, length 0 
    Code (markup):
    note: I have replaced the connecting IP with [IP] and my domain name with mydomain.com
     
    LuckyTurnips, Aug 3, 2019 IP
  13. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #13
    ^^^I noticed all the Flags. Your flags tell me that the receiver has become confused and therefore wants to abort the connection. I believe there is some router along the way that is responsible.
     
    Spoiltdiva, Aug 3, 2019 IP
  14. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #14
    So I used tshark to analyse the connection on the same device (my mobile phone) using the mobile network (where I cannot access the site) and the WIFI (where I can access the site).

    The mobile network attempt only has TCP entries, the WIFI connection has many more entries which include stuff like:
    SSL 612 Client Hello
    TLSv1.2 207 Server Hello, Change Cipher Spec, Encrypted Handshake Message
    TLSv1.2 117 Change Cipher Spec, Hello Request, Hello Request
    TLSv1.2 1504 Application Data

    I added logging for the ssl_engine, ssl_request, ssl_access in apache but they don't come up with anything, so something must be failing before that?
     
    LuckyTurnips, Aug 3, 2019 IP
  15. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #15
    Have you made any recent changes in registry, tcpip or any other network settings?....(VPN)
     
    Spoiltdiva, Aug 3, 2019 IP
  16. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #16
    Nope, I did update the kernel, but I think that was a day or two after these problems started (from what the reporting user has said).
     
    LuckyTurnips, Aug 3, 2019 IP
  17. Spoiltdiva

    Spoiltdiva Prominent Member

    Messages:
    6,371
    Likes Received:
    1,834
    Best Answers:
    50
    Trophy Points:
    395
    #17
    I'm sure you've tried it all, but just in case you haven't.......Changes in files that have permissions can screw things up, check that. Also turn off any automatic detect settings. Reset your IP address. (why not? it's worth a try) Clean up all your unnecessary registry entries.
     
    Spoiltdiva, Aug 3, 2019 IP
  18. mmerlinn

    mmerlinn Notable Member

    Messages:
    2,242
    Likes Received:
    283
    Best Answers:
    6
    Trophy Points:
    260
    #18
    Do you know when the problem started? If so, roll back all of your changes to somewhere before that time. At that point everything should work as it has in the past. After you verify that your problem no longer exists, then make your changes again, one at a time until the problem resurfaces. At that point you will know exactly what is causing your issues.
     
    Last edited: Aug 3, 2019
    mmerlinn, Aug 3, 2019 IP
  19. LuckyTurnips

    LuckyTurnips Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #19
    Thank you all for your responses.

    So I've finally got to the bottom of this error, after a few days of scratching my head and spending a lot of time researching and fiddling with the server config.
    It turns out the website has been blocked by a couple of ISPs who have decided to now deem the website as containing adult content (it's doesn't and has never had this issue before in its 10 year history).

    Rather than providing any kind of useful redirect to let a user know why they can't access the site, they simply don't send a hello packet and up comes the ERR_CONNECTION_RESET.

    I must say, this is something that had crossed my mind early on in my investigations. However, rather unhelpfully, when logging into my mobile phone account it said the adult content filter was off. I discovered this wasn't in fact the case when I called them to double check!
     
    LuckyTurnips, Aug 4, 2019 IP