5 Replies Latest reply on Oct 19, 2010 10:19 AM by peterj

    JBoss 4.2 high thread consumption

    isagarran

      Hello,

      I'm rather new on JBoss and I saw momething that i identified as a strange behaviour for JBoss ... and perhaps a malfunction.

      JBoss is consuming when i follow the activeThreadCount an increasing number of threads hence i have no longer any activity in the JVM.

      For example, i have an activity in the morning then at lunch, no one is working, and the number of threads is not going down. When the activity restart, the number of threads is raising.

      I checked the JDBC connection pool and i have only the minimum pool settings that is connected to Oracle (15 connections . i checked them using lsof)

      I issued the "kill -3 " command and i had 1850 threads and 1600 threads were  RequestProcessor threads. I've no idea of what could be these kind of threads and why they are so numberous.

      Could you help me and tell me if it is a corrct behaviour and tell me also where i should dig to remove this problem is it is ?

      JBoss 4.2 - AIX - JDK 6 SR7 - Only EJB inside this instance, jdbc connection pool.

      regards

      isagarran

        • 1. Re: JBoss 4.2 high thread consumption
          wdfink

          Hi,

          the behaviour depends on many things, so please give more informations, e.g. attach Threaddump and configuration files.

          What exact JBoss version you use, community or EAP?

          What server configuration (all, defaul...) you use?

          What is your addon configuration?

           

          You can check a little by yourself, start with a new JBoss and add step by step your configuration and check what happen.

          • 2. Re: JBoss 4.2 high thread consumption
            peterj

            Also try taking a JVM thread dump, that should tell you what those threads are doing (OS-level thread status such as "RequestProcessor" are usually meaningless for Java apps.)

            • 3. Re: JBoss 4.2 high thread consumption
              isagarran

              Hello,

              Thanks for your quick replies. I did a mistake for my JBoss level. I post all i have :

              Jboss startup :

              JBoss Bootstrap Environment JBOSS_HOME: /opt/jboss JAVA: /opt/java/bin/java JAVA_OPTS: -Dprogram.name=run.sh -Xms512m -Xmx512m -Xcompressedrefs -Xoptionsfile=/app/jvm.conf CLASSPATH: /opt/jboss/bin/run.jar:/opt/java/lib/tools.jar *****
              01:00:58,734 INFO  [Server] Starting JBoss (MX MicroKernel)... 01:00:58,915 INFO  [Server] Release ID: JBoss [EAP] 4.3.0.GA_CP07 (build: SVNTag=JBPAPP_4_3_0_GA_CP07 date=200911251949) ….
              01:01:04,258 INFO  [ServerInfo] Java version: 1.6.0,IBM Corporation 01:01:04,258 INFO  [ServerInfo] Java VM: IBM J9 VM 2.4,IBM Corporation 01:01:04,258 INFO  [ServerInfo] OS-System: AIX 5.3,ppc64 …....
              01:02:01,147 [main] INFO  [AjpProtocol] Initializing Coyote AJP/1.3 on ajp-173.5.1.80-8009 …....
              01:02:01,215 [main] INFO  [StandardEngine] Starting Servlet Engine: JBossWeb/2.0.0.GA_CP

               

              then file /app/jvm.conf

               

              -Dcom.ibm.tools.attach.enable=no -Xlp4k -Xgcpolicy:subpool

               

              then jacorb.properties (where thread pool is defined)

              #########################
              #                       #
              #   POA Configuration   #
              #                       #
              #########################

               

              # displays a GUI monitoring tool for servers
              jacorb.poa.monitoring=off

               

              # thread pool configuration for request processing
              jacorb.poa.thread_pool_max=500
              jacorb.poa.thread_pool_min=200

               

              # if set, request processing threads in thePOA
              # will run at this priority. If not set or invalid,
              # MAX_PRIORITY will be used.
              #jacorb.poa.thread_priority=

               

              # Properties controlling the POA's request queue.  If queue_wait is off,
              # then if there are more than queue_max requests in the queue, the
              # client gets TRANSIENT exceptions for any requests.  If queue_wait is on,
              # then the call blocks at the server side until no more than queue_min
              # requests are left in the queue.  The new request is then delivered as usual.
              jacorb.poa.queue_wait=off
              jacorb.poa.queue_max=100
              jacorb.poa.queue_min=10

              Then oracle datasource

              <datasources>
                <local-tx-datasource>
                  <jndi-name>MyDS</jndi-name>
                  <use-java-context>false</use-java-context>
                  <connection-url>jdbc:oracle:thin:@myHost:1521:DBname</connection-url>
                  <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
                  <user-name>myName</user-name>
                  <password>mypassword</password>
                  <min-pool-size>15</min-pool-size>
                  <max-pool-size>1000</max-pool-size>
                  <new-connection-sql>ALTER SESSION SET CONSTRAINTS = DEFERRED</new-connection-sql>
                  <valid-connection-checker-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleValidConnectionChecker</valid-connection-checker-class-name>
                  <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
                  <prefill>true</prefill>
                  <metadata>
                      <type-mapping>Oracle9i</type-mapping>
                  </metadata>
                </local-tx-datasource>
              </datasources>

               

              Normally, i expect 200 (corba thread pool) + 15 (datasource) + Jboss thread pool = around 300 threads.

              I followed the activethreadcount and the JVM start at

              276 at 01H00 up to 7:30

              then 670 from 07:30 to 07:40

              then 1078 to 1500 from 07:40 to 08:00

              This number doesn't go down. In some JVM, it increases. I don't know why i get them as i don't have too much activity on it. I took a thread dump and i saw no activity with 1000 requestProcessor doing nothing. I can post the thread as an attached file but i don't know how.

              regards.

              • 4. Re: JBoss 4.2 high thread consumption
                isagarran

                Hello,

                I can post one part of the thread dump if it is helpful :

                For information, all my requestProcessor are in this state :

                3XMTHREADINFO      "RequestProcessor-1545" J9VMThread:0x0000000031359400, j9thread_t:0x0000000159CAD9A0, java/lang/Thread:0x0000000041DDF6B0, state:CW, prio=10
                3XMTHREADINFO1            (native thread ID:0x4EE0BF, native priority:0xA, native policy:UNKNOWN)
                3XMTHREADINFO3           Java callstack:
                4XESTACKTRACE                at java/lang/Object.wait(Native Method)
                4XESTACKTRACE                at java/lang/Object.wait(Object.java:167(Compiled Code))
                4XESTACKTRACE                at org/jacorb/poa/RequestProcessor.run(Bytecode PC:3(Compiled Code))
                3XMTHREADINFO3           Native callstack:
                4XENATIVESTACK               _event_wait+0x248 (0x0900000000191FD0 [libpthreads.a+0x12fd0])
                4XENATIVESTACK               _cond_wait_local+0x3a4 (0x090000000019DF00 [libpthreads.a+0x1ef00])
                4XENATIVESTACK               _cond_wait+0x50 (0x090000000019E4AC [libpthreads.a+0x1f4ac])
                4XENATIVESTACK               pthread_cond_wait+0x178 (0x090000000019EF3C [libpthreads.a+0x1ff3c])
                4XENATIVESTACK               (0x0900000001BC4B64 [libj9thr24.so+0x4b64])
                4XENATIVESTACK               (0x0900000001BC4928 [libj9thr24.so+0x4928])
                4XENATIVESTACK               (0x0900000001BC4994 [libj9thr24.so+0x4994])
                4XENATIVESTACK               (0x09000000024A74A4 [libjclscar_24.so+0x264a4])
                4XENATIVESTACK               (0x09000000024A80F8 [libjclscar_24.so+0x270f8])
                4XENATIVESTACK               (0x0900000001B43FEC [libj9vm24.so+0xcfec])
                4XENATIVESTACK               (0x0900000001BD3B74 [libj9prt24.so+0x3b74])
                4XENATIVESTACK               (0x0900000001B43F08 [libj9vm24.so+0xcf08])
                4XENATIVESTACK               (0x0900000001BC2658 [libj9thr24.so+0x2658])
                4XENATIVESTACK               _pthread_body+0xdc (0x09000000001824F8 [libpthreads.a+0x34f8])

                regards

                • 5. Re: JBoss 4.2 high thread consumption
                  peterj

                  My guess is that the JVM fired up this thread on its own accord in support of CORBA. Or it could be simthing releated to an app you have deployed. JBoss AS thread dumps on Windows and Linux don't show any threads with a similar stack trace.

                   

                  I don't know anything about the JVM on AIX - you might have to get someone who knows that JVM to take a look at this.