12 Replies Latest reply on Dec 24, 2011 11:57 AM by andrea.ippo

    Problem for Timer of BisocketServerInvoker

      Timer of BisocketServerInvoker is a static object.
      When BisocketServerInvoker start, it will create a controlMonitorTimerTask and insert it to Timer.
      Then BisocketServerInvoker cleanup, it will call controlMonitorTimerTask's shutdown method which will set its status to CANCELLED.
      Thus, controlMonitorTimerTask will still has its reference in static Timer, even its status is CANCELLED after BisocketServerInvoker destory. So Timer's size will increase until jboss crash, right?

        • 1. Re: Problem for Timer of BisocketServerInvoker
          ron_sigal

          Well, I've heard of that problem, but I'm not sure it really exists.

          1. I've profiled the bisocket code without finding any leak.

          2. When java.util.TimerTask.cancel() is called it sets its "state" variable to "CANCELLED". Now, java.util.TimerThread.mainLoop() does this:

           task = queue.getMin();
           synchronized(task.lock) {
           if (task.state == TimerTask.CANCELLED) {
           queue.removeMin();
           continue; // No action required, poll queue again
           }
          


          so the first time the cancelled TimerTask gets run, it should be removed from the java.util.Timer's queue.

          3. BisocketSocketInvoker will recreate its Timer if the Timer has shut itself down, but the reference to the old Timer would be lost, so the old Timer should get garbage collected eventually.

          If I'm wrong about anything here, or if I'm missing something, please let me know.

          • 2. Re: Problem for Timer of BisocketServerInvoker

            I think maybe I got why cancelled controlMonitorTimerTask not be release at my side.
            First time, timer schedule a task,

            timer.schedule(controlMonitorTimerTask, pingFrequency, pingFrequency);
            

            pingFrequency's value is 214748364 that's configure in jboss\server\default\deploy\jboss-messaging.sar\remoting-bisocket-service.xml, then java.util.TimerThread.mainLoop() do
            synchronized(task.lock) {
             ....
             currentTime = System.currentTimeMillis();
             executionTime = task.nextExecutionTime;
             if (taskFired = (executionTime<=currentTime)) {
             .... }
             }
             }
             if (!taskFired) // Task hasn't yet fired; wait
             queue.wait(executionTime - currentTime);
            

            so queue will wait for 214748364ms , thus cancelled controlMonitorTimerTask will not be release until in that long time.
            Could you tell me what value I should set pingFrequency to, as after set it a small value such as 10000, I get a lot of exception while starting jboss

            • 3. Re: Problem for Timer of BisocketServerInvoker
              ron_sigal

              Ahaaa. JBossMessaging essentially disables the BisocketServerInvoker.ControlMonitorTimerTask by setting its frequency to a huge number. I had forgotten about that. So ControlMonitorTimerTask gets removed the next time it runs after getting cancelled, but it "never" runs so it "never" gets cancelled.

              I've created a JIRA issue to track this problem: JBREM-1005: "Prevent build up of cancelled TimerTasks in bisocket transport" (http://jira.jboss.com/jira/browse/JBREM-1005). When it's ready I'll attach a jboss-remoting.jar with the fix to the JIRA issue.

              Until then

              "qingsongjian" wrote:

              Could you tell me what value I should set pingFrequency to, as after set it a small value such as 10000, I get a lot of exception while starting jboss


              I would say set "pingFrequency" to the smallest number that doesn't lead to exceptions. Maybe 60000?

              • 4. Re: Problem for Timer of BisocketServerInvoker

                Hi,
                i have the same problem in Remoting 2.5.1.

                Server crashes after some times with out of memory or runs very slowly.

                memory dump have lot of objects:

                org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask @ 0xe12ef690


                bisocket pingTimerFrequency is 60000. But this seeting seems dont work.

                Attached jar in JBREM-1005 works perfectly. But i need the fix for 2.5.1, because http-Invoker dont work in attached jar.

                any ideas ?

                • 5. Re: Problem for Timer of BisocketServerInvoker
                  ron_sigal

                  Are you still having this problem?

                  Note:

                  1. The fix for JBREM-1005 is in Remoting 2.5.1.

                  2. The parameter you want is "pingFrequency", not "pingTimerFrequency".

                  • 6. Re: Problem for Timer of BisocketServerInvoker
                    j_smith

                    Hi, I have same problem with jboss-remoting 2.5.2. On backend side (jboss 5.1.0.GA) there is a JmsXA configured like here http://community.jboss.org/message/438007#438007 and client side there is spring configuration DefaultmessageListenerContainer with concurrentConsumers=1,  cacheLevelName property set to CACHE_CONSUMER and jboss-remoting 2.5.2. I have tried to set another value of pingFrequency but like Quanfu Li I get a lot exceptions.

                    I have made memory dump of my app, attaching screens. Any idea what goes wrong? Maybe some library on backend side should be replaced?

                     

                    2.png1.png

                    • 7. Problem for Timer of BisocketServerInvoker
                      ron_sigal

                      Hi John,

                       

                      Could you say more about your environment?  I guess you're saying that you have about 16000 instances of a TimerTask referencing BisocketServerInvokers.  What kind of TimerTasks are we talking about?  Note that ControlMonitorTimerTasks run on the client side.  On which side did you take the memory dump?

                       

                      Thanks,

                      Ron

                      • 8. Problem for Timer of BisocketServerInvoker
                        j_smith

                        Hi Ron,

                         

                        thaks for answer.

                         

                        My environment:

                        -backend: first machine with Jboss 5.1.0.GA

                        -client: sencond machine with Tomcat 6.0.29

                        Feel free to ask about any other details because I dont know what can be important for You.

                         

                        I dont understand Your question about type of TimerTasks, how can I check this? All information I have are on screens in my previous post.

                         

                        Memory dump was made on client side. I thought  that problem is in org.springframework.jms.listener.DefaultMessageListenerContainer but I have tried many configurations,and problem still occured. People from Spring Community didn't answer my question so I thought that maybe problem is on backend side..

                         

                        I can show You my remoting-bisocket-service.xml file or any other if it will help. I really have no idea whats wrong.

                         

                        Thanks,

                        John

                        • 9. Problem for Timer of BisocketServerInvoker
                          ron_sigal

                          Hi John,

                           

                          re: "I dont understand Your question about type of TimerTasks, how can I check this?"

                           

                          I wanted to verify that we're talking about ControlMonitorTimerTasks.  I use JProfiler to look at memory used by running code, which shows you all the accessible objects, their types, and their contents.  Not sure what you used to get your memory dump.

                           

                          re: "Could you say more about your environment?"

                           

                          Well, that was pretty vague, wasn't it?  I was wondering if your application would normally create around 16000 JMS connections.  If so, how many connections would you expect to exist at any given time?

                           

                          -Ron

                          • 10. Re: Problem for Timer of BisocketServerInvoker
                            j_smith

                            Hi,

                             

                            re: "I wanted to verify that we're talking about ControlMonitorTimerTasks"

                             

                            Yes.

                             

                            re: "Not sure what you used to get your memory dump."

                             

                            I use jmap, and Eclipse memory analyzer

                             

                            re: "I was wondering if your application would normally create around 16000 JMS connections.  If so, how many connections would you expect to exist at any given time?"

                             

                            Backend send a lot of objects (even 200.000) to jms queue few times per day, and client takes them from queue as fast as it is possible. Normally number of connections shouldn't be higher than number of concurrentConsumers set in spring configuration on client side. I think JMS connections are not closed after procesing jms message.

                             

                            John

                            • 11. Problem for Timer of BisocketServerInvoker
                              ron_sigal

                              Hi John,

                               

                              Have you made any progress?

                               

                              re: "Backend send a lot of objects (even 200.000) to jms queue few times per day, and client takes them from queue as fast as it is possible. Normally number of connections shouldn't be higher than number of concurrentConsumers set in spring configuration on client side."

                               

                              Ok, so there aren't many connections at one time.  Do you think 16000 would be created over a period of time?

                               

                              re: "I use jmap, and Eclipse memory analyzer"

                               

                              I don't I've ever used the Eclipse memory analyzer.  In JProfiler, you can trace each object back to a "root" object that prevents it from being garbage collected.  Can you do the same thing with the Eclipse memory analyzer?  If so, that might help to clarify the situation.

                               

                              -Ron

                              • 12. Re: Problem for Timer of BisocketServerInvoker
                                andrea.ippo

                                Hi John, I had the same problem (some thousand BisocketServerInvoker and a similar number of InvokerLocator) and I just found the solution for my case (tests still in progress).

                                 

                                My environment: Jboss 5.0.1 GA, JMS acting for P2P communication between a single class that writes to a custom queue and an MDB.

                                Well, I found out that the Java class writing to the queue was not releasing the queue resources properly. In fact, the close() methods of QueueSender, QueueSession and QueueConnection were never invoked!

                                So I compared the behavior of this class as it is with one that calls close() on these 3 objects (though maybe calling close() on the QueueConnection would suffice), and while for the first case MAT showed one BisocketServerInvoker per message I wrote to the queue (with a dummy app), in the second case there were far less BisocketServerInvokers, leading also to a smaller heap usage (Jvisualvm now shows a beatiful heap graph).

                                 

                                So have a look if it could be the case for you too and let me know