1 2 Previous Next 16 Replies Latest reply on Feb 22, 2010 4:50 AM by edlam_2000 Go to original post
      • 15. Re: AJP13 Request Hang Intermittently
        edlam_2000

        Thank you, besides the thread dump file, threads.html, attached in post#14, I have just taken another thread dump file, threads2010020501.html as attached from another machine experiencing the same kind of Recv-Q thread hung problem.

         

        The JMX http://localhost:8080/jboss?status=full is showing

             Max threads: 1000 Current thread count: 42 Current thread busy: 4
             Max processing time: 46824 ms Processing time: 11450.632 s Request count: 1499921 Error count: 2854 Bytes received: 0.00 MB Bytes sent: 2933.76 MB

        The netstat shows that most Recv-Q are hung, except 3 are still ok as below:-

        Proto Recv-Q Send-Q Local Address               Foreign Address             State

        tcp      179      0 127.0.0.1:8009              127.0.0.1:49943             ESTABLISHED
        tcp      516      0 127.0.0.1:8009              127.0.0.1:49686             ESTABLISHED
        tcp      500      0 127.0.0.1:8009              127.0.0.1:49678             ESTABLISHED
        tcp      179      0 127.0.0.1:8009              127.0.0.1:49932             ESTABLISHED
        tcp      500      0 127.0.0.1:8009              127.0.0.1:49410             ESTABLISHED
        tcp      297      0 127.0.0.1:8009              127.0.0.1:49921             ESTABLISHED
        tcp      223      0 127.0.0.1:8009              127.0.0.1:49920             ESTABLISHED
        tcp      300      0 127.0.0.1:8009              127.0.0.1:49214             ESTABLISHED
        tcp      499      0 127.0.0.1:8009              127.0.0.1:49213             ESTABLISHED
        tcp      223      0 127.0.0.1:8009              127.0.0.1:49978             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:49977             ESTABLISHED
        tcp      223      0 127.0.0.1:8009              127.0.0.1:49976             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:49975             ESTABLISHED
        tcp      179      0 127.0.0.1:8009              127.0.0.1:49974             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50254             ESTABLISHED
        tcp      179      0 127.0.0.1:8009              127.0.0.1:49996             ESTABLISHED
        tcp      179      0 127.0.0.1:8009              127.0.0.1:49484             ESTABLISHED
        tcp      374      0 127.0.0.1:8009              127.0.0.1:49482             ESTABLISHED
        tcp      179      0 127.0.0.1:8009              127.0.0.1:49481             ESTABLISHED
        tcp      368      0 127.0.0.1:8009              127.0.0.1:49223             ESTABLISHED
        tcp      298      0 127.0.0.1:8009              127.0.0.1:49477             ESTABLISHED
        tcp      366      0 127.0.0.1:8009              127.0.0.1:49513             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:49512             ESTABLISHED
        tcp      223      0 127.0.0.1:8009              127.0.0.1:50023             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50322             ESTABLISHED
        tcp      300      0 127.0.0.1:8009              127.0.0.1:50057             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50355             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:50098             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:50097             ESTABLISHED
        tcp      223      0 127.0.0.1:8009              127.0.0.1:50086             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50139             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50138             ESTABLISHED
        tcp      499      0 127.0.0.1:8009              127.0.0.1:49406             ESTABLISHED
        tcp        0      0 127.0.0.1:8009              127.0.0.1:50168             ESTABLISHED
        tcp      500      0 127.0.0.1:8009              127.0.0.1:49651             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:49650             ESTABLISHED
        tcp      219      0 127.0.0.1:8009              127.0.0.1:49649             ESTABLISHED

        • 16. Re: AJP13 Request Hang Intermittently
          edlam_2000

          I have greatly improved the situation by setting the connection_pool_timeout to the same number of seconds as the connectionTimeout of the AJP 8009 port connection in the server.xml, however, occasionally, there're still some data stuck in the Recv-Q for reason unknown.

           

          It appears that if these 2 values are different, then the JBOSS might have closed the connection without acknowledging the tomcat workers on the Apache side, and thus whent the tomcat worker sent a AJP13 request to the JBOSS end, the Recv-Q is stuck with data without it being serviced by the JBOSS thread.  This is just my logical guess to explain why the data got hung in the Recv-Q for a long time.

          1 2 Previous Next