Strange netty error when sending a lot of messages| JBoss.org Content Archive (Read Only)

15. Re: Strange netty error when sending a lot of messages

timfox Oct 11, 2009 11:19 AM (in response to rnicholson10)

Sorry, thought you were leaking connections via your MDB.

If this is happening when sending message, like we mentioned before, please replicate this in a simple test program and someone will take a look.

Regarding applying fixes to beta5 code - of course this is totally up to you. You can just find the svn revisions for any particular fix and apply it to what ever code you want.

Of course this is done totally at your own risk, we make no guarantees that this will or won't work

16. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 11, 2009 11:41 AM (in response to rnicholson10)

Understood.

I have someone working on a simple test program that can replicate the issue. Once it's ready I'll let you know.

I know how to checkout the Beta5 release, but I'm unsure how to figure out what changes were made to fix the MDB leak.

Currently I think that only one line was added:

line 999 added to src/main/org/hornetq/ra/HornetQRASessionFactoryImpl.java


info.setDefaults(((HornetQResourceAdapter)mcf.getResourceAdapter()).getProperties());

Sorry , I havn't used FishEye before...

17. Re: Strange netty error when sending a lot of messages

timfox Oct 11, 2009 12:44 PM (in response to rnicholson10)

Go to JIRA, find the task, click on the subversion commits tab. That will tell you the svn revision number.

Then

1) checkout beta 5:

svn co http://anonsvn.jboss.org.repos/messaging/tags/<name of tag>
svn merge -c <revision number> http://anon.svn.jboss.org/repos/messaging/trunk

Or something like that.

svn help will tell you more, or try googling it.

18. Re: Strange netty error when sending a lot of messages

thammoud Oct 11, 2009 12:48 PM (in response to rnicholson10)

I think that you are hitting a TCP issue that is being caused by the way you are creating a connection per message.

TCP TIME-WAIT Delay

When a Transmission Control Protocol (TCP) connection is closed, the socket pair associated with the connection is placed into a state known as TIME-WAIT, which prevents other connections from using that protocol, source Internet Protocol (IP) address, destination IP address, source port, and destination port for a period of time.

This functionality presents a resource-related denial of service opportunity. Because the ports affected are not immediately returned to the system's pool of available ports, network applications that perform many outbound connections in a short time can use up all available ports before the ports can be recycled. At this point, the application either pauses, waiting for ports to become available, or ends with an error.

The fact that you are having issues at around 65K (Port limits). Your code is creating a connection per message. You need to reuse the same connection.

19. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 11, 2009 1:05 PM (in response to rnicholson10)

First of all, thanks Tim for the svn merge howto!

@Thammoud, yes I agree. But I had started to reuse connections via a pool for sending. I am now using a pool in every place I send a message. We will run a test in the morning to see if this works. I will also try the patch for the MDB leak to see if that makes a difference (as all messages are received via an MDB, so after 65L messages...).

I'll report back tomorrow to see if either solution fixes the problem.

20. Re: Strange netty error when sending a lot of messages

timfox Oct 11, 2009 2:07 PM (in response to rnicholson10)

If you're creating a connection per message, that would indeed by a problem.

This is probably the most common messaging anti-pattern - it's a definite no-no. Take a look at the user manual chapter on performance tuning for more words on this.

21. Re: Strange netty error when sending a lot of messages

timfox Oct 11, 2009 2:10 PM (in response to rnicholson10)

Think of a JMS connection like a JDBC connection. You wouldn't create a connection just to run a single query then close it again, then create another one to run the next query. It's the same for JMS connections.

JMS connections are heavyweight resources that, as Thamoud said, are designed to be re-used.

22. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 11, 2009 2:25 PM (in response to rnicholson10)

@Timfox

Would the same apply to sessions and producers on a connection?

Should I pool these along with the connections? In my use case we don't have many different queues, therefore I could essentially reuse the sessions and their producers also. Or would it be best practice not to do this.

Hmm, can a single connection supports multiple sessions at one time, and from there multiple producers? If I take it this far I would only ever need one connection. I think I'm talking myself out of the whole last sentence!

What I'd really like is a weekend when I'm not working. I can vaguely remember what sleep and rest were like...

;)

23. Re: Strange netty error when sending a lot of messages

timfox Oct 12, 2009 6:07 AM (in response to rnicholson10)

"rnicholson10" wrote:
@Timfox

Would the same apply to sessions and producers on a connection?

Should I pool these along with the connections? In my use case we don't have many different queues, therefore I could essentially reuse the sessions and their producers also.

Yes you should re-use them. But bear in mind a session/producer is a single threaded object and should only be accessed by a single thread at any one time

Hmm, can a single connection supports multiple sessions at one time, and from there multiple producers?

Yes, but remember the single threaded limitation.

I'd suggest taking a look at the Sun JMS tutorial - it should help you with understanding the relation between connections, sessions, producers etc

http://java.sun.com/products/jms/tutorial/1_3_1-fcs/doc/copyright.html

24. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 12, 2009 9:41 AM (in response to rnicholson10)

After all this we now believe it's the paging file not being read!

The last message we receive on the other side of the bridge is 52114. We are waiting for message 52115.

If I open the next paging file (which is the the first one) that was never sent the first id in there is 52115. If I restart the appserver, everything is sent.

Our paging files are set to 100Mib memory, 10Mib Files, as per the documentation.

We never saw this before as we had never sent enough data to require paging to take effect.

I will test with a very small paging buffer to confirm the problem.

25. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 12, 2009 4:28 PM (in response to rnicholson10)

The memory for the queue is becoming full hence the paging file being created.

I think every message that is sent is being kept in memory. Is it possible that no acks are being received for these messages?

If we switch off paging this works but a lot of memory is still being used.

26. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 12, 2009 5:09 PM (in response to rnicholson10)

I just reproduced this on a smaller scale.

Setup a bridge from A to B.

In your address settings on A set max-size-bytes to 2 MiB and page-size-bytes to 1MiB.

All sessions are created as follows:

 session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

When sending message set the same Group ID and priority:

 message.setStringProperty(HornetQMessage.JMSXGROUPID, "SAME_GROUP_ALWAYS");

 producer.send(message, DeliveryMode.PERSISTENT, 9, 0);

And that's it. Now simply send enough messages to fill the memory buffer. Once a paging file has been created no more messages will be sent.

If you restart your server messages will be sent from the paging file until memory is once more full. Keeping restarting until all messages have been sent. All acknowledge modes seem to act the same, even PRE_ACKNOWLEDGE! Maybe it's not messages acks but that's where I'm leaning at the moment...

Regardless the memory allocated to the queue is never cleared unless the server is restarted.

27. Re: Strange netty error when sending a lot of messages

timfox Oct 12, 2009 7:48 PM (in response to rnicholson10)

Please create a JIRA with a test program attached and someone will investigate

28. Re: Strange netty error when sending a lot of messages

mreasy Oct 15, 2009 5:54 AM (in response to rnicholson10)

I'm going into the same direction as thammoud.
Even if you would create one connection per send, this should not result in a problem.
I recommend to check your Operating System's timeout for TCP-connections that are in state=TIME_WAIT. Don't know your system, here are some links:
Overview HP-UX, Win, Linux, Solaris http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.edge.doc/cp/admingd45.htm
AIX: http://www-01.ibm.com/support/docview.wss?uid=swg21142422
Windows has a much lower limit, see http://support.microsoft.com/kb/196271/en-us

29. Re: Strange netty error when sending a lot of messages

rnicholson10 Oct 15, 2009 6:12 AM (in response to rnicholson10)

The problem was never anything to do with TIME_WAIT. I am now using a connection pool (which cleared up all the time waits), and then I changed to the in-vm connector/acceptor.

The TIME_WAITs were simply masking the issue. I have someone working on an example program to demonstrate the issue we are having whereby the queue memory is never cleared when using a bridge to a remote machine.

Once it's ready I'll post a JIRA with the example so the guys can take a look at it.