8 Replies Latest reply on Jun 8, 2012 10:21 AM by jfclere

    Apache Crash / 1.2.0 Final / jboss 7.1.0

    black_dev

      Ok, my story is:

       

      I have apache with mod_cluster 1.2.0 Final and two jboss AS 7.1.0 in cluster with sticky-session

       

      I started all servers.

       

      Apache have five proces thread ( 0.0 % cpu busy per proces)

      Jboss about 2-5% cpu busy.

       

      After some time ( one time more , one time less ) under production load apache spawn new thread but:

      - this tread not serve a content ( nothing in logs )

      - have 100% cpu busy

      - I think this proces was zombie/or not function.

       

      like This:

       

      apache   10912  0.0  0.1 186288  3524 ?    Sl   11:28   0:00 /usr/sbin/httpd
      apache   10917  0.0  0.1 186284  3556 ?    Sl   11:28   0:00 /usr/sbin/httpd
      apache   10921 99.7  0.1 186148  3196 ?    Rl   11:29  17:24 /usr/sbin/httpd
      apache   10953  0.0  0.1 251820  3564 ?    Sl   11:30   0:00 /usr/sbin/httpd
      apache   10981  0.0  0.1 186284  3520 ?    Sl   11:33   0:00 /usr/sbin/httpd
      apache   10995 98.9  0.1 186148  3196 ?    Rl   11:44   2:27 /usr/sbin/httpd

       

       

      ok I make some debug thinks:

       

      pstack <pid>

      Thread 2 (Thread 0x7f7b5a835700 (LWP 15404)):

      #0  0x00007f7b638928b3 in select () from /lib64/libc.so.6

      #1  0x00007f7b63d83155 in apr_sleep () from /usr/lib64/libapr-1.so.0

      #2  0x00007f7b5ad4e6fd in proxy_cluster_watchdog_func () from /etc/httpd/modules/mod_proxy_cluster.so

      #3  0x00007f7b63b4b7f1 in start_thread () from /lib64/libpthread.so.0

      #4  0x00007f7b63899ccd in clone () from /lib64/libc.so.6

      Thread 1 (Thread 0x7f7b652657e0 (LWP 15403)):

      #0  0x00007f7b5ad5024e in remove_session_route () from /etc/httpd/modules/mod_proxy_cluster.so

      #1  0x00007f7b5ad5086f in proxy_cluster_pre_request () from /etc/httpd/modules/mod_proxy_cluster.so

      #2  0x00007f7b5c1a20d6 in proxy_run_pre_request () from /etc/httpd/modules/mod_proxy.so

      #3  0x00007f7b5c1a9026 in ap_proxy_pre_request () from /etc/httpd/modules/mod_proxy.so

      #4  0x00007f7b5c1a62c2 in ?? () from /etc/httpd/modules/mod_proxy.so

      #5  0x00007f7b6529fb00 in ap_run_handler ()

      #6  0x00007f7b652a33be in ap_invoke_handler ()

      #7  0x00007f7b652aea30 in ap_process_request ()

      #8  0x00007f7b652ab8f8 in ?? ()

      #9  0x00007f7b652a7608 in ap_run_process_connection ()

      #10 0x00007f7b652b3807 in ?? ()

      #11 0x00007f7b652b3b1a in ?? ()

      #12 0x00007f7b652b479c in ap_mpm_run ()

      #13 0x00007f7b6528b900 in main ()

       

       

      gdb -q -p <pid>

       

      Attaching to process 14970

      Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.

      Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libm.so.6

      Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libpcre.so.0

      Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libselinux.so.1

      Reading symbols from /usr/lib64/libaprutil-1.so.0...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libaprutil-1.so.0

      Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libcrypt.so.1

      Reading symbols from /lib64/libexpat.so.1...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libexpat.so.1

      Reading symbols from /lib64/libdb-4.7.so...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libdb-4.7.so

      Reading symbols from /usr/lib64/libapr-1.so.0...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libapr-1.so.0

      Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.

      [Thread debugging using libthread_db enabled]

      [New Thread 0x7f7b5a835700 (LWP 14996)]

      Loaded symbols for /lib64/libpthread.so.0

      Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libc.so.6

      Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libdl.so.2

      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/ld-linux-x86-64.so.2

      Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libuuid.so.1

      Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libfreebl3.so

      Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libnss_files.so.2

      Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libldap-2.4.so.2

      Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/liblber-2.4.so.2

      Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libresolv.so.2

      Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libssl3.so

      Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libsmime3.so

      Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libnss3.so

      Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libnssutil3.so

      Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libplds4.so

      Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libplc4.so

      Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.

      Loaded symbols for /lib64/libnspr4.so

      Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.

      Loaded symbols for /usr/lib64/libsasl2.so.2

      Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.

      ...

      ...

      ...

      Loaded symbols for /etc/httpd/modules/mod_substitute.so

      Reading symbols from /etc/httpd/modules/mod_rewrite.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_rewrite.so

      Reading symbols from /etc/httpd/modules/mod_proxy.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy.so

      Reading symbols from /etc/httpd/modules/mod_proxy_ftp.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so

      Reading symbols from /etc/httpd/modules/mod_proxy_http.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy_http.so

      Reading symbols from /etc/httpd/modules/mod_proxy_ajp.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy_ajp.so

      Reading symbols from /etc/httpd/modules/mod_proxy_connect.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so

      Reading symbols from /etc/httpd/modules/mod_cache.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_cache.so

      Reading symbols from /etc/httpd/modules/mod_suexec.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_suexec.so

      Reading symbols from /etc/httpd/modules/mod_disk_cache.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_disk_cache.so

      Reading symbols from /etc/httpd/modules/mod_cgi.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_cgi.so

      Reading symbols from /etc/httpd/modules/mod_version.so...(no debugging symbols found)...done.

      Loaded symbols for /etc/httpd/modules/mod_version.so

      Reading symbols from /etc/httpd/modules/mod_slotmem.so...done.

      Loaded symbols for /etc/httpd/modules/mod_slotmem.so

      Reading symbols from /etc/httpd/modules/mod_manager.so...done.

      Loaded symbols for /etc/httpd/modules/mod_manager.so

      Reading symbols from /etc/httpd/modules/mod_proxy_cluster.so...done.

      Loaded symbols for /etc/httpd/modules/mod_proxy_cluster.so

      Reading symbols from /etc/httpd/modules/mod_advertise.so...done.

      Loaded symbols for /etc/httpd/modules/mod_advertise.so

      0x00007f7b5ad50250 in remove_session_route (r=0x7f7b67280cc8, name=0x7f7b67285f63 "jsessionid") at mod_proxy_cluster.c:2472

              in mod_proxy_cluster.c

       

       

       

       

       

      backtrace

       

       

      Reading symbols from /etc/httpd/modules/mod_advertise.so...done.

      Loaded symbols for /etc/httpd/modules/mod_advertise.so

      Core was generated by `/usr/sbin/httpd'.

      #0  0x00007f7b638928b3 in select () from /lib64/libc.so.6

      Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-15.el6.centos.1.x86_64

      (gdb) backtrace

      #0  0x00007f7b638928b3 in select () from /lib64/libc.so.6

      #1  0x00007f7b63d83155 in apr_sleep () from /usr/lib64/libapr-1.so.0

      #2  0x00007f7b5ad4e6fd in proxy_cluster_watchdog_func (thd=0x7f7b67271080, data=0x0) at mod_proxy_cluster.c:1783

      #3  0x00007f7b63b4b7f1 in start_thread () from /lib64/libpthread.so.0

      #4  0x00007f7b63899ccd in clone () from /lib64/libc.so.6

       

       

       

      my system is:

       

      uname -a

      Linux jb-web-04 2.6.32-220.7.1.el6.x86_64 #1 SMP Wed Mar 7 00:52:02 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux

       

      our apache + we add mod_cluster modul

       

      httpd -V

      Server version: Apache/2.2.15 (Unix)

      Server built:   Feb 13 2012 22:31:42

      Server's Module Magic Number: 20051115:24

      Server loaded:  APR 1.3.9, APR-Util 1.3.9

      Compiled using: APR 1.3.9, APR-Util 1.3.9

      Architecture:   64-bit

      Server MPM:     Prefork

        threaded:     no

          forked:     yes (variable process count)

       

       

       

      before crash - nothing bad in log files.

       

      pls help... meybe somethink tips how to debug... or... change in system.

        • 1. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
          jfclere

          MODCLUSTER-312 could you try with httpd-2.2.17?

          • 2. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
            black_dev

            Can I try with 2.2.22 ? It is ok ?

            because i have it from package,  2.2.17 I must compile to test it.

            • 3. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
              black_dev

              On apache 2.2.22 this same bad.

              • 4. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
                jfclere

                Ok that is a bug... How do you reach the problem (CPU load etc) so that I can try to reproduce it. Your dump don't help: proxy_cluster_watchdog_func() is called even if there is no request that is normal (there is one thread per process doing that every ~ 500 ms) and remove_session_route() is a normal request processing.

                • 5. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
                  black_dev

                  Ok... some more information:

                   

                  I have one box ( jb-web-04 ) with Apache 2.2.22 , and second box ( jb-web-05 ) with jboss 7.1.0

                   

                  On Apache I have 3 vhosts:

                  one for mod_cluster manager

                  one for frontend application ( forward to jboss )

                  one for backend application ( forward to jboss )

                   

                  vhost for cluster manager

                   

                  <pre>

                  LoadModule slotmem_module modules/mod_slotmem.so

                  LoadModule manager_module modules/mod_manager.so

                  LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

                  LoadModule advertise_module modules/mod_advertise.so

                   

                  <VirtualHost cluster:80>

                   

                          <Directory />

                                  Order deny,allow

                                  Allow from all

                          </Directory>

                   

                          KeepAliveTimeout 60

                          MaxKeepAliveRequests 0

                   

                      ServerAdvertise On

                      AdvertiseFrequency 30

                      AdvertiseGroup    y.z.x.106:23366

                      AdvertiseSecurityKey cluster-poller-v3

                      AdvertiseBindAddress cluster

                          EnableMCPMReceive

                     

                      LBstatusRecalTime 30

                   

                        <Location /mod_cluster-manager>

                             SetHandler mod_cluster-manager

                   

                             Order deny,allow

                             Deny from all

                             Allow from all

                         </Location>

                   

                      <Location /server-status>

                          SetHandler server-status

                          Order deny,allow

                          Deny from all

                          Allow from x.x.x.

                      </Location>

                     

                    ErrorLog logs/cluster-error.log

                    CustomLog logs/cluster-access.log combined

                    LogLevel info

                   

                    AllowDisplay on

                   

                  </VirtualHost>

                  </pre>

                   

                  Vhost for app:

                   

                  <VirtualHost g.g.g.253:80>

                   

                      ServerName app.local

                      ProxyPass / balancer://cluster-prod-01/ stickysession=JSESSIONID|jsessionid nofailover=On

                      ProxyPassReverse / balancer://cluster-prod-01/ stickysession=JSESSIONID|jsessionid

                      ProxyPreserveHost On

                   

                   

                   

                      KeepAliveTimeout 60

                      MaxKeepAliveRequests 0

                   

                    ErrorLog logs/app-error.log

                    LogFormat "PID:%P %h %{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" app

                   

                   

                    CustomLog logs/app-access.log app

                   

                    LogLevel info

                   

                    RewriteEngine On

                    #RewriteLogLevel 3

                    #RewriteLog /var/log/httpd/app-rewrite.log

                   

                    # Przekierowanie wybranych linkow na stara wersje gpwis

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio\/html\.run\?_Instance=cms_gpw\.x\.pl\&_PageID=35 [NC,OR]

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio\/html\.run\?_Instance=cms_gpw\.x\.pl\&_PageID=6\&_EspiID= [NC,OR]

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio\/html\.media\?_Instance=cms_gpw\.x\.pl\&_Option=Other\&_Connector=espi\&_tableName=espi_attachments [NC,OR]

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio\/html\.media\?_Instance=cms_gpw\.x\.pl\&_Option=Palio\&_ID= [NC,OR]

                    RewriteCond %{THE_REQUEST} ^GET\ \/PAPWebService\/WSTransmisje\.jws [NC,OR]

                    RewriteCond %{THE_REQUEST} ^GET\ \/WSRelacjeInwestorskie\/WSRelacjeInwestorskie\.jws [NC]

                    RewriteRule ^/(.*)$ http://a.b.c.d/$1 [L,P]

                   

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio\/html\.run\?_Instance=cms_gpw\.x\.pl\ .*$ [NC]

                    RewriteRule ^/.*$ /GPWIS2/ [L,R]

                   

                   

                    RewriteCond %{THE_REQUEST} ^GET\ \/palio [NC]

                    ## RewriteRule ^/.*$ http://www.pap.pl/euro2012? [L,R=301]

                    RewriteRule ^/.*$ /GPWIS2/images/logof.png? [L,R]

                   

                   

                    RewriteCond %{REQUEST_URI} !^/APP [NC]

                    RewriteRule ^/.*$ /APP/ [L,R]

                   

                   

                  <Location /APPCMS >

                    Order deny,allow

                    Deny from all

                  </Location>

                   

                  </VirtualHost>

                   

                   

                  and this is a jboss standalone-ha.xml

                   

                   

                   

                  <?xml version='1.0' encoding='UTF-8'?>

                   

                  <server xmlns="urn:jboss:domain:1.1">

                   

                      <extensions>

                          <extension module="org.jboss.as.clustering.infinispan"/>

                          <extension module="org.jboss.as.clustering.jgroups"/>

                          <extension module="org.jboss.as.configadmin"/>

                          <extension module="org.jboss.as.connector"/>

                          <extension module="org.jboss.as.deployment-scanner"/>

                          <extension module="org.jboss.as.ee"/>

                          <extension module="org.jboss.as.ejb3"/>

                          <extension module="org.jboss.as.jaxrs"/>

                          <extension module="org.jboss.as.jdr"/>

                          <extension module="org.jboss.as.jmx"/>

                          <extension module="org.jboss.as.jpa"/>

                          <extension module="org.jboss.as.logging"/>

                          <extension module="org.jboss.as.mail"/>

                          <extension module="org.jboss.as.modcluster"/>

                          <extension module="org.jboss.as.naming"/>

                          <extension module="org.jboss.as.osgi"/>

                          <extension module="org.jboss.as.pojo"/>

                          <extension module="org.jboss.as.remoting"/>

                          <extension module="org.jboss.as.sar"/>

                          <extension module="org.jboss.as.security"/>

                          <extension module="org.jboss.as.threads"/>

                          <extension module="org.jboss.as.transactions"/>

                          <extension module="org.jboss.as.web"/>

                          <extension module="org.jboss.as.webservices"/>

                          <extension module="org.jboss.as.weld"/>

                      </extensions>

                   

                      <management>

                          <security-realms>

                              <security-realm name="ManagementRealm">

                                  <authentication>

                                      <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>

                                  </authentication>

                              </security-realm>

                              <security-realm name="ApplicationRealm">

                                  <authentication>

                                      <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>

                                  </authentication>

                              </security-realm>

                          </security-realms>

                          <management-interfaces>

                              <native-interface security-realm="ManagementRealm">

                                  <socket-binding native="management-native"/>

                              </native-interface>

                              <http-interface security-realm="ManagementRealm">

                                  <socket-binding http="management-http"/>

                              </http-interface>

                          </management-interfaces>

                      </management>

                   

                      <profile>

                          <subsystem xmlns="urn:jboss:domain:logging:1.1">

                              <console-handler name="CONSOLE">

                                  <level name="INFO"/>

                                  <formatter>

                                      <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

                                  </formatter>

                              </console-handler>

                              <periodic-rotating-file-handler name="FILE">

                                  <formatter>

                                      <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>

                                  </formatter>

                                  </formatter>

                                  <file relative-to="jboss.server.log.dir" path="server.log"/>

                                  <suffix value=".yyyy-MM-dd"/>

                                  <append value="true"/>

                              </periodic-rotating-file-handler>

                              <logger category="com.arjuna">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="org.apache.tomcat.util.modeler">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="sun.rmi">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="jacorb">

                                  <level name="WARN"/>

                              </logger>

                              <logger category="jacorb.config">

                                  <level name="ERROR"/>

                              </logger>

                              <root-logger>

                                  <level name="INFO"/>

                                  <handlers>

                                      <handler name="CONSOLE"/>

                                      <handler name="FILE"/>

                                  </handlers>

                              </root-logger>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:configadmin:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:datasources:1.0">

                              <datasources>

                                

                                 [....]

                                

                              </datasources>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">

                              <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:ee:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:ejb3:1.2">

                              <session-bean>

                                  <stateless>

                                      <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                                  </stateless>

                                  <stateful default-access-timeout="5000" cache-ref="simple" clustered-cache-ref="clustered"/>

                                  <singleton default-access-timeout="5000"/>

                              </session-bean>

                              <mdb>

                                  <resource-adapter-ref resource-adapter-name="hornetq-ra"/>

                                  <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>

                              </mdb>

                              <pools>

                                  <bean-instance-pools>

                                      <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                                      <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                                  </bean-instance-pools>

                              </pools>

                              <caches>

                                  <cache name="simple" aliases="NoPassivationCache"/>

                                  <cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>

                                  <cache name="clustered" passivation-store-ref="infinispan" aliases="StatefulTreeCache"/>

                              </caches>

                              <passivation-stores>

                                  <file-passivation-store name="file"/>

                                  <cluster-passivation-store name="infinispan" cache-container="ejb"/>

                              </passivation-stores>

                              <async thread-pool-name="default"/>

                              <timer-service thread-pool-name="default">

                                  <data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>

                              </timer-service>

                              <remote connector-ref="remoting-connector" thread-pool-name="default"/>

                              <thread-pools>

                                  <thread-pool name="default">

                                      <max-threads count="10"/>

                                      <keepalive-time time="100" unit="milliseconds"/>

                                  </thread-pool>

                              </thread-pools>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:infinispan:1.1" default-cache-container="cluster">

                              <cache-container name="cluster" aliases="ha-partition" default-cache="default">

                                  <transport lock-timeout="60000"/>

                                  <replicated-cache name="default" mode="SYNC" batching="true">

                                      <locking isolation="REPEATABLE_READ"/>

                                  </replicated-cache>

                              </cache-container>

                              <cache-container name="web" aliases="standard-session-cache" default-cache="repl">

                                  <transport lock-timeout="60000"/>

                                  <replicated-cache name="repl" mode="SYNC" batching="true">

                                      <file-store/>

                                  </replicated-cache>

                                  <replicated-cache name="sso" mode="SYNC" batching="true"/>

                                  <distributed-cache name="dist" mode="SYNC" batching="true">

                                      <file-store/>

                                  </distributed-cache>

                              </cache-container>

                              <cache-container name="ejb" aliases="sfsb sfsb-cache" default-cache="repl">

                                  <transport lock-timeout="60000"/>

                                  <replicated-cache name="repl" mode="SYNC" batching="true">

                                      <eviction strategy="LRU"/>

                                      <file-store/>

                                  </replicated-cache>

                                  <replicated-cache name="remote-connector-client-mappings" mode="SYNC" batching="true"/>

                                  <distributed-cache name="dist" mode="SYNC" batching="true">

                                      <eviction strategy="LRU"/>

                                      <file-store/>

                                  </distributed-cache>

                              </cache-container>

                              <cache-container name="hibernate" default-cache="local-query">

                                  <transport lock-timeout="60000"/>

                                  <local-cache name="local-query">

                                      <transaction mode="NONE"/>

                                      <eviction strategy="LRU" max-entries="10000"/>

                                      <expiration max-idle="100000"/>

                                  </local-cache>

                                  <invalidation-cache name="entity" mode="SYNC">

                                      <transaction mode="NON_XA"/>

                                      <eviction strategy="LRU" max-entries="10000"/>

                                      <expiration max-idle="100000"/>

                                  </invalidation-cache>

                                  <replicated-cache name="timestamps" mode="SYNC">

                                      <transaction mode="NONE"/>

                                      <eviction strategy="NONE"/>

                                  </replicated-cache>

                              </cache-container>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:jca:1.1">

                              <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                              <bean-validation enabled="false"/>

                              <default-workmanager>

                                  <short-running-threads>

                                      <core-threads count="50"/>

                                      <queue-length count="50"/>

                                      <max-threads count="50"/>

                                      <keepalive-time time="10" unit="seconds"/>

                                  </short-running-threads>

                                  <long-running-threads>

                                      <core-threads count="50"/>

                                      <queue-length count="50"/>

                                      <max-threads count="50"/>

                                      <keepalive-time time="10" unit="seconds"/>

                                  </long-running-threads>

                              </default-workmanager>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">

                              <stack name="udp">

                                  <transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>

                                  <protocol type="PING"/>

                                  <protocol type="MERGE2"/>

                                  <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                                  <protocol type="FD"/>

                                  <protocol type="VERIFY_SUSPECT"/>

                                  <protocol type="BARRIER"/>

                                  <protocol type="pbcast.NAKACK"/>

                                  <protocol type="UNICAST2"/>

                                  <protocol type="pbcast.STABLE"/>

                                  <protocol type="pbcast.GMS"/>

                                  <protocol type="UFC"/>

                                  <protocol type="MFC"/>

                                  <protocol type="FRAG2"/>

                              </stack>

                              <stack name="tcp">

                                  <transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>

                                  <protocol type="MPING" socket-binding="jgroups-mping"/>

                                  <protocol type="MERGE2"/>

                                  <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                                  <protocol type="FD"/>

                                  <protocol type="VERIFY_SUSPECT"/>

                                  <protocol type="BARRIER"/>

                                  <protocol type="pbcast.NAKACK"/>

                                  <protocol type="UNICAST2"/>

                                  <protocol type="pbcast.STABLE"/>

                                  <protocol type="pbcast.GMS"/>

                                  <protocol type="UFC"/>

                                  <protocol type="MFC"/>

                                  <protocol type="FRAG2"/>

                              </stack>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jmx:1.1">

                              <show-model value="true"/>

                              <remoting-connector/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:jpa:1.0">

                              <jpa default-datasource=""/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:mail:1.0">

                              <mail-session jndi-name="java:jboss/mail/Default">

                                  <smtp-server outbound-socket-binding-ref="mail-smtp"/>

                              </mail-session>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                              <mod-cluster-config advertise-socket="modcluster" sticky-session="true" sticky-session-remove="true" balancer="cluster-prod-01">

                                  <dynamic-load-provider history="9" decay="2">

                                      <load-metric type="cpu" weight="2" capacity="1"/>

                                  </dynamic-load-provider>

                              </mod-cluster-config>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:naming:1.1"/>

                          <subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="lazy">

                              <properties>

                                  <property name="org.osgi.framework.startlevel.beginning">

                                      1

                                  </property>

                              </properties>

                              <capabilities>

                                  <capability name="javax.servlet.api:v25"/>

                                  <capability name="javax.transaction.api"/>

                                  <capability name="org.apache.felix.log" startlevel="1"/>

                                  <capability name="org.jboss.osgi.logging" startlevel="1"/>

                                  <capability name="org.apache.felix.configadmin" startlevel="1"/>

                                  <capability name="org.jboss.as.osgi.configadmin" startlevel="1"/>

                              </capabilities>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:remoting:1.1">

                              <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:resource-adapters:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:sar:1.0"/>

                          <subsystem xmlns="urn:jboss:domain:security:1.1">

                              <security-domains>

                                  <security-domain name="other" cache-type="default">

                                      <authentication>

                                          <login-module code="Remoting" flag="optional">

                                              <module-option name="password-stacking" value="useFirstPass"/>

                                          </login-module>

                                          <login-module code="RealmUsersRoles" flag="required">

                                              <module-option name="usersProperties" value="${jboss.server.config.dir}/application-users.properties"/>

                                              <module-option name="rolesProperties" value="${jboss.server.config.dir}/application-roles.properties"/>

                                              <module-option name="realm" value="ApplicationRealm"/>

                                              <module-option name="password-stacking" value="useFirstPass"/>

                                          </login-module>

                                      </authentication>

                                  </security-domain>

                                  <security-domain name="jboss-web-policy" cache-type="default">

                                      <authorization>

                                          <policy-module code="Delegating" flag="required"/>

                                      </authorization>

                                  </security-domain>

                                  <security-domain name="jboss-ejb-policy" cache-type="default">

                                      <authorization>

                                          <policy-module code="Delegating" flag="required"/>

                                      </authorization>

                                  </security-domain>

                              </security-domains>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:threads:1.1"/>

                          <subsystem xmlns="urn:jboss:domain:transactions:1.1">

                              <core-environment>

                                  <process-id>

                                      <uuid/>

                                  </process-id>

                              </core-environment>

                              <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>

                              <coordinator-environment default-timeout="300"/>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">

                              <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" enabled="true"/>

                              <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

                              <virtual-server name="default-host" enable-welcome-root="true">

                                  <alias name="localhost"/>

                                  <alias name="example.com"/>

                              </virtual-server>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:webservices:1.1">

                              <modify-wsdl-address>true</modify-wsdl-address>

                              <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>

                              <endpoint-config name="Standard-Endpoint-Config"/>

                              <endpoint-config name="Recording-Endpoint-Config">

                                  <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">

                                      <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>

                                  </pre-handler-chain>

                              </endpoint-config>

                          </subsystem>

                          <subsystem xmlns="urn:jboss:domain:weld:1.0"/>

                      </profile>

                   

                      <interfaces>

                          <interface name="management">

                              <inet-address value="${jboss.bind.address.management:web05node1}"/>

                          </interface>

                          <interface name="public">

                              <inet-address value="${jboss.bind.address:web05node1}"/>

                          </interface>

                      </interfaces>

                   

                      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

                          <socket-binding name="ajp" port="8009"/>

                          <socket-binding name="http" port="8080"/>

                          <socket-binding name="https" port="8443"/>

                          <socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.51.75" multicast-port="7500"/>

                          <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.51.4}" multicast-port="45700"/>

                          <socket-binding name="jgroups-tcp" port="7600"/>

                          <socket-binding name="jgroups-tcp-fd" port="57600"/>

                          <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.51.4}" multicast-port="45688"/>

                          <socket-binding name="jgroups-udp-fd" port="54200"/>

                          <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>

                          <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>

                          <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>

                          <socket-binding name="modcluster" port="0" multicast-address="224.0.51.106" multicast-port="23366"/>

                          <socket-binding name="osgi-http" interface="management" port="8090"/>

                          <socket-binding name="remoting" port="4447"/>

                          <socket-binding name="txn-recovery-environment" port="4712"/>

                          <socket-binding name="txn-status-manager" port="4713"/>

                          <outbound-socket-binding name="mail-smtp">

                              <remote-destination host="localhost" port="25"/>

                          </outbound-socket-binding>

                      </socket-binding-group>

                   

                      <deployments>

                           <deployment name="APPCMS.war" runtime-name="APPCMS.war">

                              <content sha1="06554e6391a5716305e2d79c57165fe638a4863e"/>

                          </deployment>

                          <deployment name="APP.war" runtime-name="APP.war">

                              <content sha1="df735cbf34b75fdf2c9f923f1bd51c7e45d8a8a6"/>

                          </deployment>

                      </deployments>

                  </server>

                   

                   

                  The problem is when i think apache spawn new worker ( apache is in default configuration in httpd).

                   

                  example:

                  apache have 5 proces...

                   

                  apache spawn next one but next one have 100%cpu busy.

                   

                  apache have 6 proces ( 5 ok, 1 bad )

                   

                  in log is: 5 work, 1 not work ( not giveing a request )

                  in log any information about 1bad ( nothing )

                   

                  I think not matter many request to apache or small request to apache...

                   

                  make the: httpd reload - resolve the problem with 100% busy cpu ( 1bad was removed from proces )

                   

                  I can make additional tests for you if you prefer.

                  • 6. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
                    jfclere

                    I am not able to reproduce it... Could you compile mod_cluster modules using your httpd and try again?

                    • 7. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
                      black_dev

                      ok, I can, but Could you give ma a step by step how to make this, because informations in sources zip file with mod cluster is to small .

                       

                      And I compile by this scenario, meybe you have more information about problem.

                       

                      My second questions is - Is my configuration is ok ? meybe i not configure some parameters .

                      • 8. Re: Apache Crash / 1.2.0 Final / jboss 7.1.0
                        jfclere