Apache CentOS Crash - Core Dump Analysis & Backtrace

Discussion in 'Apache' started by Silver89, Nov 30, 2013.

  1. #1
    I've been seeing lots of the following error in my error_log recently, I've enabled core dumps and have managed to analyse one using gdb:

    [Fri Nov 29 20:44:41.318847 2013] [core:notice] [pid 2307] AH00051: child pid 7504 exit signal Segmentation fault (11), possible coredump in /tmp/core-dump
    [Fri Nov 29 20:44:41.318937 2013] [core:notice] [pid 2307] AH00051: child pid 7623 exit signal Segmentation fault (11), possible coredump in /tmp/core-dump
    [Fri Nov 29 20:44:43.324859 2013] [core:notice] [pid 2307] AH00051: child pid 7724 exit signal Segmentation fault (11), possible coredump in /tmp/core-dump
    Code (markup):
    GDB Analysis of Core Dump File
    Core was generated by `/usr/local/apache/bin/httpd -k start -DSSL'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x00007fcc9b259798 in apr_uri_parse () from /usr/local/apache/lib/libaprutil-1.so.0
    Code (markup):
    Backtrace
    #0  0x00007fcc9b259798 in apr_uri_parse () from /usr/local/apache/lib/libaprutil-1.so.0
    #1  0x00000000004f1e09 in stapling_renew_response ()
    #2  0x00000000004f2f58 in stapling_cb ()
    #3  0x0000003f4022b431 in ssl_check_clienthello_tlsext () from /usr/lib64/libssl.so.10
    #4  0x0000003f4021b9e5 in ssl3_get_client_hello () from /usr/lib64/libssl.so.10
    #5  0x0000003f4021db6f in ssl3_accept () from /usr/lib64/libssl.so.10
    #6  0x0000003f402290d9 in ssl23_get_client_hello () from /usr/lib64/libssl.so.10
    #7  0x0000003f402298b3 in ssl23_accept () from /usr/lib64/libssl.so.10
    #8  0x00000000004dc63c in ssl_io_filter_handshake ()
    #9  0x00000000004dd641 in ssl_io_filter_input ()
    #10 0x000000000043e61c in ap_get_brigade ()
    #11 0x0000000000440d8e in ap_rgetline_core ()
    #12 0x000000000044161f in read_request_line ()
    #13 0x0000000000442e30 in ap_read_request ()
    #14 0x0000000000497045 in ap_process_http_sync_connection ()
    #15 0x00000000004970a3 in ap_process_http_connection ()
    #16 0x000000000046ba8f in ap_run_process_connection ()
    #17 0x000000000046bf66 in ap_process_connection ()
    #18 0x000000000051c378 in child_main ()
    #19 0x000000000051c58e in make_child ()
    #20 0x000000000051c943 in perform_idle_server_maintenance ()
    #21 0x000000000051d0b5 in prefork_run ()
    #22 0x000000000043bf35 in ap_run_mpm ()
    #23 0x000000000043423c in main ()
    Code (markup):
    It seems the function apr_uri_parse() causes the issues and is located in /usr/local/apache/lib/.

    I don't really know what to do next with regards to fixing this? The mentioned file is editable as it is `.so` and I cannot find any other occurrences of it with other people?

    My software is all up to date, including CentOS, Apache, php.

    A new apache build did not fix this issue either.
     
    Silver89, Nov 30, 2013 IP
  2. ABCname.com

    ABCname.com Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #2
    Check if apache using correct library, for example:
    root@node5 [~]# ldd /usr/local/apache/bin/httpd  |grep libaprutil
            libaprutil-1.so.0 => /usr/local/apache/lib/libaprutil-1.so.0 (0x00007f6578a06000)
    root@node5 [~]# ll /usr/local/apache/lib/libaprutil-1.so.0
    lrwxrwxrwx 1 root root 21 Nov 17 17:24 /usr/local/apache/lib/libaprutil-1.so.0 -> libaprutil-1.so.0.5.2*
    
    Code (markup):
     
    ABCname.com, Dec 7, 2013 IP
  3. BreezeHost

    BreezeHost Member

    Messages:
    139
    Likes Received:
    1
    Best Answers:
    1
    Trophy Points:
    28
    #3
    Enable coredump in httpd.conf. For that add the following directive to /usr/local/apache/conf/httpd.conf :

    CoreDumpDirectory /tmp/httpdump
     
    BreezeHost, Dec 9, 2013 IP