12 Replies Latest reply: Dec 24, 2011 11:57 AM by Andrea Ippolito RSS

Problem for Timer of BisocketServerInvoker

Quanfu Li Newbie

Timer of BisocketServerInvoker is a static object.
When BisocketServerInvoker start, it will create a controlMonitorTimerTask and insert it to Timer.
Then BisocketServerInvoker cleanup, it will call controlMonitorTimerTask's shutdown method which will set its status to CANCELLED.
Thus, controlMonitorTimerTask will still has its reference in static Timer, even its status is CANCELLED after BisocketServerInvoker destory. So Timer's size will increase until jboss crash, right?

  • 1. Re: Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Well, I've heard of that problem, but I'm not sure it really exists.

    1. I've profiled the bisocket code without finding any leak.

    2. When java.util.TimerTask.cancel() is called it sets its "state" variable to "CANCELLED". Now, java.util.TimerThread.mainLoop() does this:

     task = queue.getMin();
     synchronized(task.lock) {
     if (task.state == TimerTask.CANCELLED) {
     queue.removeMin();
     continue; // No action required, poll queue again
     }
    


    so the first time the cancelled TimerTask gets run, it should be removed from the java.util.Timer's queue.

    3. BisocketSocketInvoker will recreate its Timer if the Timer has shut itself down, but the reference to the old Timer would be lost, so the old Timer should get garbage collected eventually.

    If I'm wrong about anything here, or if I'm missing something, please let me know.

  • 2. Re: Problem for Timer of BisocketServerInvoker
    Quanfu Li Newbie

    I think maybe I got why cancelled controlMonitorTimerTask not be release at my side.
    First time, timer schedule a task,

    timer.schedule(controlMonitorTimerTask, pingFrequency, pingFrequency);
    

    pingFrequency's value is 214748364 that's configure in jboss\server\default\deploy\jboss-messaging.sar\remoting-bisocket-service.xml, then java.util.TimerThread.mainLoop() do
    synchronized(task.lock) {
     ....
     currentTime = System.currentTimeMillis();
     executionTime = task.nextExecutionTime;
     if (taskFired = (executionTime<=currentTime)) {
     .... }
     }
     }
     if (!taskFired) // Task hasn't yet fired; wait
     queue.wait(executionTime - currentTime);
    

    so queue will wait for 214748364ms , thus cancelled controlMonitorTimerTask will not be release until in that long time.
    Could you tell me what value I should set pingFrequency to, as after set it a small value such as 10000, I get a lot of exception while starting jboss

  • 3. Re: Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Ahaaa. JBossMessaging essentially disables the BisocketServerInvoker.ControlMonitorTimerTask by setting its frequency to a huge number. I had forgotten about that. So ControlMonitorTimerTask gets removed the next time it runs after getting cancelled, but it "never" runs so it "never" gets cancelled.

    I've created a JIRA issue to track this problem: JBREM-1005: "Prevent build up of cancelled TimerTasks in bisocket transport" (http://jira.jboss.com/jira/browse/JBREM-1005). When it's ready I'll attach a jboss-remoting.jar with the fix to the JIRA issue.

    Until then

    "qingsongjian" wrote:

    Could you tell me what value I should set pingFrequency to, as after set it a small value such as 10000, I get a lot of exception while starting jboss


    I would say set "pingFrequency" to the smallest number that doesn't lead to exceptions. Maybe 60000?

  • 4. Re: Problem for Timer of BisocketServerInvoker
    6767 6767 Newbie

    Hi,
    i have the same problem in Remoting 2.5.1.

    Server crashes after some times with out of memory or runs very slowly.

    memory dump have lot of objects:

    org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask @ 0xe12ef690


    bisocket pingTimerFrequency is 60000. But this seeting seems dont work.

    Attached jar in JBREM-1005 works perfectly. But i need the fix for 2.5.1, because http-Invoker dont work in attached jar.

    any ideas ?

  • 5. Re: Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Are you still having this problem?

    Note:

    1. The fix for JBREM-1005 is in Remoting 2.5.1.

    2. The parameter you want is "pingFrequency", not "pingTimerFrequency".

  • 6. Re: Problem for Timer of BisocketServerInvoker
    John Smith Newbie

    Hi, I have same problem with jboss-remoting 2.5.2. On backend side (jboss 5.1.0.GA) there is a JmsXA configured like here http://community.jboss.org/message/438007#438007 and client side there is spring configuration DefaultmessageListenerContainer with concurrentConsumers=1,  cacheLevelName property set to CACHE_CONSUMER and jboss-remoting 2.5.2. I have tried to set another value of pingFrequency but like Quanfu Li I get a lot exceptions.

    I have made memory dump of my app, attaching screens. Any idea what goes wrong? Maybe some library on backend side should be replaced?

     

    2.png1.png

  • 7. Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Hi John,

     

    Could you say more about your environment?  I guess you're saying that you have about 16000 instances of a TimerTask referencing BisocketServerInvokers.  What kind of TimerTasks are we talking about?  Note that ControlMonitorTimerTasks run on the client side.  On which side did you take the memory dump?

     

    Thanks,

    Ron

  • 8. Problem for Timer of BisocketServerInvoker
    John Smith Newbie

    Hi Ron,

     

    thaks for answer.

     

    My environment:

    -backend: first machine with Jboss 5.1.0.GA

    -client: sencond machine with Tomcat 6.0.29

    Feel free to ask about any other details because I dont know what can be important for You.

     

    I dont understand Your question about type of TimerTasks, how can I check this? All information I have are on screens in my previous post.

     

    Memory dump was made on client side. I thought  that problem is in org.springframework.jms.listener.DefaultMessageListenerContainer but I have tried many configurations,and problem still occured. People from Spring Community didn't answer my question so I thought that maybe problem is on backend side..

     

    I can show You my remoting-bisocket-service.xml file or any other if it will help. I really have no idea whats wrong.

     

    Thanks,

    John

  • 9. Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Hi John,

     

    re: "I dont understand Your question about type of TimerTasks, how can I check this?"

     

    I wanted to verify that we're talking about ControlMonitorTimerTasks.  I use JProfiler to look at memory used by running code, which shows you all the accessible objects, their types, and their contents.  Not sure what you used to get your memory dump.

     

    re: "Could you say more about your environment?"

     

    Well, that was pretty vague, wasn't it?  I was wondering if your application would normally create around 16000 JMS connections.  If so, how many connections would you expect to exist at any given time?

     

    -Ron

  • 10. Re: Problem for Timer of BisocketServerInvoker
    John Smith Newbie

    Hi,

     

    re: "I wanted to verify that we're talking about ControlMonitorTimerTasks"

     

    Yes.

     

    re: "Not sure what you used to get your memory dump."

     

    I use jmap, and Eclipse memory analyzer

     

    re: "I was wondering if your application would normally create around 16000 JMS connections.  If so, how many connections would you expect to exist at any given time?"

     

    Backend send a lot of objects (even 200.000) to jms queue few times per day, and client takes them from queue as fast as it is possible. Normally number of connections shouldn't be higher than number of concurrentConsumers set in spring configuration on client side. I think JMS connections are not closed after procesing jms message.

     

    John

  • 11. Problem for Timer of BisocketServerInvoker
    Ron Sigal Master

    Hi John,

     

    Have you made any progress?

     

    re: "Backend send a lot of objects (even 200.000) to jms queue few times per day, and client takes them from queue as fast as it is possible. Normally number of connections shouldn't be higher than number of concurrentConsumers set in spring configuration on client side."

     

    Ok, so there aren't many connections at one time.  Do you think 16000 would be created over a period of time?

     

    re: "I use jmap, and Eclipse memory analyzer"

     

    I don't I've ever used the Eclipse memory analyzer.  In JProfiler, you can trace each object back to a "root" object that prevents it from being garbage collected.  Can you do the same thing with the Eclipse memory analyzer?  If so, that might help to clarify the situation.

     

    -Ron

  • 12. Re: Problem for Timer of BisocketServerInvoker
    Andrea Ippolito Newbie

    Hi John, I had the same problem (some thousand BisocketServerInvoker and a similar number of InvokerLocator) and I just found the solution for my case (tests still in progress).

     

    My environment: Jboss 5.0.1 GA, JMS acting for P2P communication between a single class that writes to a custom queue and an MDB.

    Well, I found out that the Java class writing to the queue was not releasing the queue resources properly. In fact, the close() methods of QueueSender, QueueSession and QueueConnection were never invoked!

    So I compared the behavior of this class as it is with one that calls close() on these 3 objects (though maybe calling close() on the QueueConnection would suffice), and while for the first case MAT showed one BisocketServerInvoker per message I wrote to the queue (with a dummy app), in the second case there were far less BisocketServerInvokers, leading also to a smaller heap usage (Jvisualvm now shows a beatiful heap graph).

     

    So have a look if it could be the case for you too and let me know