Hello Apache Kings and Queens, Im stuck in an issue where i freshly built apache just wont simply forward my requests to my 2 JBoss servers clustered servers. Im using Mod_JK 1.2.27 Apache 2.2.21 Linux RHEL 5.7 Boss 7 I keep getting 503 errors-Service Temporarily Unavailable on the browser Even though I can hit the servlets and JSPs on there just fine if I hit the JBoss server individually. I get these errors in the mod_jk.log file [Thu Apr 04 01:54:34 2013] [6068:3860842240] [debug] ajp_send_request::jk_ajp_common.c (1520): (worker1) request body to send 0 - request body to resend 0 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] jk_shutdown_socket::jk_connect.c (680): About to shutdown socket 14 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] jk_shutdown_socket::jk_connect.c (731): Shutdown socket 14 and read 0 lingering bytes [Thu Apr 04 01:55:34 2013] [6068:3860842240] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1104): (worker1) can't receive the response message from tomcat, tomcat (10.1.1.222:8180) has forced a connection close for socket 14 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [error] ajp_get_reply::jk_ajp_common.c (1920): (worker1) Tomcat is down or refused connection. No response has been sent to the client (yet) [Thu Apr 04 01:55:34 2013] [6068:3860842240] [info] ajp_service::jk_ajp_common.c (2407): (worker1) sending request to tomcat failed (recoverable), (attempt=2) [Thu Apr 04 01:55:34 2013] [6068:3860842240] [error] ajp_service::jk_ajp_common.c (2426): (worker1) connecting to tomcat failed. [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_reset_endpoint::jk_ajp_common.c (743): (worker1) resetting endpoint with sd = 4294967295 (socket shutdown) [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_done::jk_ajp_common.c (2850): recycling connection pool slot=0 for worker worker1 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1347): service failed, worker worker1 is in error state [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] service::jk_lb_worker.c (1358): recoverable error... will try to recover on other worker [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] get_most_suitable_worker::jk_lb_worker.c (969): found best worker worker2 (worker2) using method 'Request' [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] service::jk_lb_worker.c (1133): service worker=worker2 route=worker1 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_get_endpoint::jk_ajp_common.c (2922): acquired connection pool slot=0 after 0 retries [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_marshal_into_msgb::jk_ajp_common.c (605): ajp marshaling done [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_service::jk_ajp_common.c (2246): processing worker2 with 2 retries [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_send_request::jk_ajp_common.c (1461): (worker2) all endpoints are disconnected. [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] jk_open_socket::jk_connect.c (451): socket TCP_NODELAY set to On [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] jk_open_socket::jk_connect.c (575): trying to connect socket 14 to 10.1.1.111:8180 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] jk_open_socket::jk_connect.c (601): socket 14 connected to 10.1.1.111:8180 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connect_to_endpoint::jk_ajp_common.c (931): Connected socket 14 to (10.1.1.111:8180) [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): sending to ajp13 pos=4 len=509 max=8192 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): 0000 12 34 01 F9 02 02 00 08 48 54 54 50 2F 31 2E 31 - .4......HTTP/1.1 [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): 0010 00 00 16 2F 68 65 6C 6C 6F 77 6F 72 6C 64 2F 48 - .../helloworld/H [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): 0020 65 6C 6C 6F 57 6F 72 6C 64 00 00 0D 31 37 32 2E - elloWorld...173. [Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): 0030 33 31 2E 34 35 2E 32 34 39 00 FF FF 00 21 74 30 - 11.42.222....!Thu Apr 04 01:55:34 2013] [6068:3860842240] [debug] ajp_connection_tcp_send_message::jk_ajp_common.c (1033): 0040 31 64 72 77 31 34 30 30 2E 64 73 6D 2E 74 69 63 - server1 . . . . . [Thu Apr 04 01:56:34 2013] [6068:3860842240] [debug] ajp_send_request::jk_ajp_common.c (1520): (worker2) request body to send 0 - request body to resend 0 [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] jk_shutdown_socket::jk_connect.c (680): About to shutdown socket 14 [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] jk_shutdown_socket::jk_connect.c (731): Shutdown socket 14 and read 0 lingering bytes [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1104): (worker2) can't receive the response message from tomcat, tomcat (10.1.1.111:8180) has forced a connection close for socket 14 [Thu Apr 04 01:57:34 2013] [6068:3860842240] [error] ajp_get_reply::jk_ajp_common.c (1920): (worker2) Tomcat is down or refused connection. No response has been sent to the client (yet) [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] ajp_service::jk_ajp_common.c (2407): (worker2) sending request to tomcat failed (recoverable), (attempt=2) [Thu Apr 04 01:57:34 2013] [6068:3860842240] [error] ajp_service::jk_ajp_common.c (2426): (worker2) connecting to tomcat failed. [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] ajp_reset_endpoint::jk_ajp_common.c (743): (worker2) resetting endpoint with sd = 4294967295 (socket shutdown) [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] ajp_done::jk_ajp_common.c (2850): recycling connection pool slot=0 for worker worker2 [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1347): service failed, worker worker2 is in error state [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] service::jk_lb_worker.c (1358): recoverable error... will try to recover on other worker [Thu Apr 04 01:57:34 2013] [6068:3860842240] [debug] service::jk_lb_worker.c (1101): retry 1, sleeping for 100 ms before retrying [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] force_recovery::jk_lb_worker.c (577): worker worker1 is marked for forced recovery [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] force_recovery::jk_lb_worker.c (577): worker worker2 is marked for forced recovery [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1399): Forcing recovery once for 2 workers [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=0, retry=1) [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1416): All tomcat instances failed, no more workers left (attempt=1, retry=1) [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] service::jk_lb_worker.c (1427): All tomcat instances are busy or in error state [Thu Apr 04 01:57:34 2013] [6068:3860842240] [error] service::jk_lb_worker.c (1432): All tomcat instances failed, no more workers left [Thu Apr 04 01:57:34 2013] [6068:3860842240] [info] jk_handler::mod_jk.c (2476): Service error=0 for worker=loadbalancer [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1035): Attempting to map URI '/favicon.ico' from 4 maps [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/HelloWorld/*=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/HelloWorld=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/*=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] jk_translate::mod_jk.c (3361): no match for /favicon.ico found [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1035): Attempting to map URI '/favicon.ico' from 4 maps [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/HelloWorld/*=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/HelloWorld=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld/*=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] find_match::jk_uri_worker_map.c (849): Attempting to map context URI '/helloworld=loadbalancer' source 'JkMount' [Thu Apr 04 01:57:35 2013] [6070:3976877824] [debug] jk_map_to_storage::mod_jk.c (3428): no match for /favicon.ico found Here are my workers.properties # Define a worker named 'worker1' (more workers can be added as comma separated values) #worker.list=loadbalancer,status worker.list=loadbalancer # Node #1 properties worker.worker1.type=ajp13 worker.worker1.host=server1.com worker.worker1.port=8180 worker.worker1.lbfactor=1 # Node #2 properties(will be configured later when apache and node 1 start working) worker.worker2.type=ajp13 worker.worker2.host=server2.com worker.worker2.port=8180 worker.worker2.lbfactor=1 # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=worker1,worker2 # Status worker for managing load balancer #worker.status.type=status #worker.loadbalancer.sticky_session=true mod_jk.conf # Set path to workers.properties. JkWorkersFile conf/jk/workers.properties # Set path to jk shared memory. Generally, you'll want this to point to your local state or logs directory. JkShmFile /apps/logs/mod_jk.shm # Set path to jk logs. This path should point to the same logs directory as the HTTPD access_log. JkLogFile /apps/logs/mod_jk.log # Set the jk log level. Valid values for this setting are 'debug', 'error', or 'info'. JkLogLevel debug # Set timestamp log format. Use provided variables to customize. JkLogStampFormat "[%a %b %d %H:%M:%S %Y] " JkMount /helloworld/* loadbalancer JkMount /helloworld loadbalancer JkMount /helloworld/HelloWorld/* loadbalancer JkMount /helloworld/HelloWorld loadbalancer httpd.conf # # This is the main Apache HTTP server configuration file. It contains the # configuration directives that give the server its instructions. # See <URL:http://httpd.apache.org/docs/2.2> for detailed information. # In particular, see # <URL:http://httpd.apache.org/docs/2.2/mod/directives.html> # for a discussion of each configuration directive. # # Do NOT simply read the instructions in here without understanding # what they do. They're here only as hints or reminders. If you are unsure # consult the online docs. You have been warned. # # Configuration and logfile names: If the filenames you specify for many # of the server's control files begin with "/" (or "drive:/" for Win32), the # server will use that explicit path. If the filenames do *not* begin # with "/", the value of ServerRoot is prepended -- so "logs/foo_log" # with ServerRoot set to "/apps/apache" will be interpreted by the # server as "/apps/apache/logs/foo_log". # # ServerRoot: The top of the directory tree under which the server's # configuration, error, and log files are kept. # # Do not add a slash at the end of the directory path. If you point # ServerRoot at a non-local disk, be sure to point the LockFile directive # at a local disk. If you wish to share the same ServerRoot for multiple # httpd daemons, you will need to change at least LockFile and PidFile. # ServerRoot "/apps/apache" # # Listen: Allows you to bind Apache to specific IP addresses and/or # ports, instead of the default. See also the <VirtualHost> # directive. # # Change this to Listen on specific IP addresses as shown below to # prevent Apache from glomming onto all bound IP addresses. # Listen 8080 # # Dynamic Shared Object (DSO) Support # # To be able to use the functionality of a module which was built as a DSO you # have to place corresponding `LoadModule' lines at this location so the # directives contained in it are actually available _before_ they are used. # Statically compiled modules (those listed by `httpd -l') do not need # to be loaded here. # # Example: # LoadModule foo_module modules/mod_foo.so # LoadModule jk_module modules/mod_jk.so <IfModule !mpm_netware_module> <IfModule !mpm_winnt_module> # # If you wish httpd to run as a different user or group, you must run # httpd as root initially and it will switch. # # User/Group: The name (or #number) of the user/group to run httpd as. # It is usually good practice to create a dedicated user and group for # running httpd, as with most system services. # User coresvcd Group coregrpd </IfModule> </IfModule> # 'Main' server configuration # # The directives in this section set up the values used by the 'main' # server, which responds to any requests that aren't handled by a # <VirtualHost> definition. These values also provide defaults for # any <VirtualHost> containers you may define later in the file. # # All of these directives may appear inside <VirtualHost> containers, # in which case these default settings will be overridden for the # virtual host being defined. # # # ServerAdmin: Your address, where problems with the server should be # e-mailed. This address appears on some server-generated pages, such # as error documents. e.g. # ServerAdmin # # ServerName gives the name and port that the server uses to identify itself. # This can often be determined automatically, but we recommend you specify # it explicitly to prevent problems during startup. # # If your host doesn't have a registered DNS name, enter its IP address here. # #ServerName www.example.com:80 # # DocumentRoot: The directory out of which you will serve your # documents. By default, all requests are taken from this directory, but # symbolic links and aliases may be used to point to other locations. # DocumentRoot "/apps/apache/htdocs" # # Each directory to which Apache has access can be configured with respect # to which services and features are allowed and/or disabled in that # directory (and its subdirectories). # # First, we configure the "default" to be a very restrictive set of # features. # <Directory /> Options FollowSymLinks AllowOverride None Order deny,allow Deny from all </Directory> # # Note that from this point forward you must specifically allow # particular features to be enabled - so if something's not working as # you might expect, make sure that you have specifically enabled it # below. # # # This should be changed to whatever you set DocumentRoot to. # <Directory "/apps/apache/htdocs"> # # Possible values for the Options directive are "None", "All", # or any combination of: # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews # # Note that "MultiViews" must be named *explicitly* --- "Options All" # doesn't give it to you. # # The Options directive is both complicated and important. Please see # http://httpd.apache.org/docs/2.2/mod/core.html#options # for more information. # Options Indexes FollowSymLinks # # AllowOverride controls what directives may be placed in .htaccess files. # It can be "All", "None", or any combination of the keywords: # Options FileInfo AuthConfig Limit # AllowOverride None # # Controls who can get stuff from this server. # Order allow,deny Allow from all </Directory> # # DirectoryIndex: sets the file that Apache will serve if a directory # is requested. # <IfModule dir_module> DirectoryIndex index.html </IfModule> # # The following lines prevent .htaccess and .htpasswd files from being # viewed by Web clients. # <FilesMatch "^\.ht"> Order allow,deny Deny from all Satisfy All </FilesMatch> # # ErrorLog: The location of the error log file. # If you do not specify an ErrorLog directive within a <VirtualHost> # container, error messages relating to that virtual host will be # logged here. If you *do* define an error logfile for a <VirtualHost> # container, that host's errors will be logged there and not here. # ErrorLog "/apps/logs/error_log" # # LogLevel: Control the number of messages logged to the error_log. # Possible values include: debug, info, notice, warn, error, crit, # alert, emerg. # LogLevel debug <IfModule log_config_module> # # The following directives define some format nicknames for use with # a CustomLog directive (see below). # LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined LogFormat "%h %l %u %t \"%r\" %>s %b" common <IfModule logio_module> # You need to enable mod_logio.c to use %I and %O LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio </IfModule> # # The location and format of the access logfile (Common Logfile Format). # If you do not define any access logfiles within a <VirtualHost> # container, they will be logged here. Contrariwise, if you *do* # define per-<VirtualHost> access logfiles, transactions will be # logged therein and *not* in this file. # CustomLog "/apps/logs/access_log" common # # If you prefer a logfile with access, agent, and referer information # (Combined Logfile Format) you can use the following directive. # #CustomLog "logs/access_log" combined </IfModule> <IfModule alias_module> # # Redirect: Allows you to tell clients about documents that used to # exist in your server's namespace, but do not anymore. The client # will make a new request for the document at its new location. # Example: # Redirect permanent /foo http://www.example.com/bar # # Alias: Maps web paths into filesystem paths and is used to # access content that does not live under the DocumentRoot. # Example: # Alias /webpath /full/filesystem/path # # If you include a trailing / on /webpath then the server will # require it to be present in the URL. You will also likely # need to provide a <Directory> section to allow access to # the filesystem path. # # ScriptAlias: This controls which directories contain server scripts. # ScriptAliases are essentially the same as Aliases, except that # documents in the target directory are treated as applications and # run by the server when requested rather than as documents sent to the # client. The same rules about trailing "/" apply to ScriptAlias # directives as to Alias. # ScriptAlias /cgi-bin/ "/apps/apache/cgi-bin/" </IfModule> <IfModule cgid_module> # # ScriptSock: On threaded servers, designate the path to the UNIX # socket used to communicate with the CGI daemon of mod_cgid. # #Scriptsock logs/cgisock </IfModule> # # "/apps/apache/cgi-bin" should be changed to whatever your ScriptAliased # CGI directory exists, if you have that configured. # <Directory "/apps/apache/cgi-bin"> AllowOverride None Options None Order allow,deny Allow from all </Directory> # # DefaultType: the default MIME type the server will use for a document # if it cannot otherwise determine one, such as from filename extensions. # If your server contains mostly text or HTML documents, "text/plain" is # a good value. If most of your content is binary, such as applications # or images, you may want to use "application/octet-stream" instead to # keep browsers from trying to display binary files as though they are # text. # DefaultType text/plain <IfModule mime_module> # # TypesConfig points to the file containing the list of mappings from # filename extension to MIME-type. # TypesConfig conf/mime.types # # AddType allows you to add to or override the MIME configuration # file specified in TypesConfig for specific file types. # #AddType application/x-gzip .tgz # # AddEncoding allows you to have certain browsers uncompress # information on the fly. Note: Not all browsers support this. # #AddEncoding x-compress .Z #AddEncoding x-gzip .gz .tgz # # If the AddEncoding directives above are commented-out, then you # probably should define those extensions to indicate media types: # AddType application/x-compress .Z AddType application/x-gzip .gz .tgz # # AddHandler allows you to map certain file extensions to "handlers": # actions unrelated to filetype. These can be either built into the server # or added with the Action directive (see below) # # To use CGI scripts outside of ScriptAliased directories: # (You will also need to add "ExecCGI" to the "Options" directive.) # #AddHandler cgi-script .cgi # For type maps (negotiated resources): #AddHandler type-map var # # Filters allow you to process content before it is sent to the client. # # To parse .shtml files for server-side includes (SSI): # (You will also need to add "Includes" to the "Options" directive.) # #AddType text/html .shtml #AddOutputFilter INCLUDES .shtml </IfModule> # # The mod_mime_magic module allows the server to use various hints from the # contents of the file itself to determine its type. The MIMEMagicFile # directive tells the module where the hint definitions are located. # #MIMEMagicFile conf/magic # # Customizable error responses come in three flavors: # 1) plain text 2) local redirects 3) external redirects # # Some examples: #ErrorDocument 500 "The server made a boo boo." #ErrorDocument 404 /missing.html #ErrorDocument 404 "/cgi-bin/missing_handler.pl" #ErrorDocument 402 http://www.example.com/subscription_info.html # # # MaxRanges: Maximum number of Ranges in a request before # returning the entire resource, or 0 for unlimited # Default setting is to accept 200 Ranges #MaxRanges 0 # # EnableMMAP and EnableSendfile: On systems that support it, # memory-mapping or the sendfile syscall is used to deliver # files. This usually improves server performance, but must # be turned off when serving from networked-mounted # filesystems or if support for these functions is otherwise # broken on your system. # #EnableMMAP off #EnableSendfile off # Supplemental configuration # # The configuration files in the conf/extra/ directory can be # included to add extra features or to modify the default configuration of # the server, or you may simply copy their contents here and change as # necessary. # Server-pool management (MPM specific) #Include conf/extra/httpd-mpm.conf # Multi-language error messages #Include conf/extra/httpd-multilang-errordoc.conf # Fancy directory listings #Include conf/extra/httpd-autoindex.conf # Language settings #Include conf/extra/httpd-languages.conf # User home directories #Include conf/extra/httpd-userdir.conf # Real-time info on requests and configuration #Include conf/extra/httpd-info.conf # Virtual hosts #Include conf/extra/httpd-vhosts.conf # Local access to the Apache HTTP Server Manual #Include conf/extra/httpd-manual.conf # Distributed authoring and versioning (WebDAV) #Include conf/extra/httpd-dav.conf # Various default settings #Include conf/extra/httpd-default.conf # Secure (SSL/TLS) connections #Include conf/extra/httpd-ssl.conf # # Note: The following must must be present to support # starting without SSL on platforms with no /dev/random equivalent # but a statically compiled-in mod_ssl. # Include conf/jk/mod_jk.conf <IfModule ssl_module> SSLRandomSeed startup builtin SSLRandomSeed connect builtin </IfModule> Any help is very much appreciate. King Koopa