1 2 3 4 Previous Next 51 Replies Latest reply on Nov 23, 2006 2:27 PM by ovidiu.feodorov Go to original post
      • 30. Re: Incorporating Remoting http transport into Messaging
        ron_sigal

        I've checked in some changes to the HTTP branch. Tomorrow I'll discuss them more extensively.

        • 31. Re: Incorporating Remoting http transport into Messaging
          ovidiu.feodorov

          Could you please give us an overview?

          • 32. Re: Incorporating Remoting http transport into Messaging
            ron_sigal

            I. The central change has been to push callbacks back down into the Remoting layer. In particular, where on HEAD ServerConsumerEndpoint.Deliverer gets the ServerInvokerCallbackHandler's Client and delivers messages to the client side using Client.invoke(), on the HTTP branch it calls ServerInvokerCallbackHandler.handleCallback(). As a result, Messaging can take advantage of Remoting's variety of callback options. In particular, the relevant choices are:

            (a) push callbacks: a Connector (complete with ServerSocket) is created on the client side, and calling ServerInvokerCallbackHandler.handleCallback() causes an invocation to be made from server side to client side.

            (b) poll callbacks: calling ServerInvokerCallbackHandler.handleCallback() causes the callback to be stored (possibly on disk) until the client side polls for pending callbacks. No ServerSocket is created on the client side.

            It is my understanding that option (a) is suitable for the socket transport, and option (b) is suitable for the HTTP transport. Switching between the two is done by the following code in JMSRemotingConnection.start():

             boolean doPushCallbacks = "socket".equals(serverLocator.getProtocol());
             if (doPushCallbacks)
             {
             if (log.isTraceEnabled()) log.trace("doing push callbacks");
             HashMap metadata = new HashMap();
             metadata.put(InvokerLocator.DATATYPE, "jms");
             metadata.put(InvokerLocator.SERIALIZATIONTYPE, "jms");
             client.addListener(callbackManager, metadata, null, true);
             }
             else
             {
             if (log.isTraceEnabled()) log.trace("simulating push callbacks");
             HashMap metadata = new HashMap();
             metadata.put(CallbackPoller.CALLBACK_POLL_PERIOD, "100");
             client.addListener(callbackManager, metadata);
             }
            


            The implementation of Client.addListener() does the rest.

            Changes related to callbacks may be found in the following classes:

            org.jboss.jms.client.remoting.CallbackManager
            org.jboss.jms.client.remoting.CallbackServerFactory (no longer necessary)
            org.jboss.jms.client.remoting.JMSRemotingConnection
            org.jboss.jms.server.container.InjectionAspect
            org.jboss.jms.server.endpoint.ServerConnecionEndpoint
            org.jboss.jms.server.endpoint.ServerConsumerEndpoint

            In order to replace invocations with callbacks, a callback acknowledgement facility was added to Remoting. When ServerConsumerEndpoint.Deliverer sends a callback, it also stores the callback in a pendingCallbacks HashSet, from which it is removed upon acknowledgement. ServerConsumerEndpoint.stop() waits for pendingCallbacks to go empty before returning. A callback is acknowledged in one of two ways:

            - In the case of push callbacks, ServerInvokerCallbackHandler.handleCallback() does the acknowledgement after the callback has been pushed to the client side. At this point, the callback has been processed by MessageCallbackHandler.

            - In the case of polled callbacks, the acknowledgement is initiated from the client side after the callback has been processed by MessageCallbackHandler.

            TODO:

            1. A callback ID should be stored in pendingCallbacks instead of the callback itself.
            2. Unlike Client.invoke(), ServerInvokerCallbackHandler.handleCallback() does not return a response, so ServerConsumerEndpoint.Deliverer gets no feedback from the client side about throttling message flow.


            II. The new Messaging specific serialization structure interacts with callbacks to generate some added complexity in JMSWireFormat. Because of the two forms of callbacks, there are two delivery modes for callbacks, and each has to be unwound to expose the ClientDelivery. For example, callback polling delivers an ArrayList of pending callbacks, and each ClientDelivery in the list has to be processed separately. There's also another related issue. In pull or poll mode, callbacks are stored on the server, which might involve writing them to secondary storage. Currently, the application's designated serialization type determines how the callbacks are transferred to and from disk, but MessagingObjectInputStream and MessagingObjectOutputStream are derived from the standard java classes, which can't handle Messaging non-Serializable messages. For now I've worked around the problem in my local version of Remoting, but the problem still needs to be resolved. Is there any reason the Messaging can't use JBoss Serialization?

            III. I changed org.jboss.test.messaging.tools.jmx.ServiceContainer and org.jboss.test.messaging.tools.jmx.ServiceContainerConfiguration so that they can pick up the remoting transport from a <remoting-transport> element in tests/etc/container.xml.

            IV. Right now the HTTP branch runs with my own private version of Remoting. I've checked in a copy of jboss-remoting.jar in the lib directory.

            V. It's 4am and right now it looks like everything's broken. Great. Well, it WAS working.

            • 33. Re: Incorporating Remoting http transport into Messaging
              ovidiu.feodorov

               

              Ron wrote:

              I. The central change has been to push callbacks back down into the Remoting layer. In particular, where on HEAD ServerConsumerEndpoint.Deliverer gets the ServerInvokerCallbackHandler's Client and delivers messages to the client side using Client.invoke(), on the HTTP branch it calls ServerInvokerCallbackHandler.handleCallback(). As a result, Messaging can take advantage of Remoting's variety of callback options.


              That's good. Using the callback abstraction correctly is the right way to go.

              Ron wrote:

              (a) push callbacks: a Connector (complete with ServerSocket) is created on the client side, and calling ServerInvokerCallbackHandler.handleCallback() causes an invocation to be made from server side to client side.


              Not an option. Or not anymore, at this stage. The socket transport will be rendered irrelevant soon, one way or another, so we don't want to waste any more time on that. See http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3979236 If it currently works with the socket transport, that's good, but we need to get the multiplex transport working ASAP.

              Ron wrote:

              (b) poll callbacks: calling ServerInvokerCallbackHandler.handleCallback() causes the callback to be stored (possibly on disk) until the client side polls for pending callbacks. No ServerSocket is created on the client side.


              Yes, but no disk. For non persistent messages we don't care anyway, and for persistent messages, they're already stored by the channel, so we're covered. Disk at that level can only slow down things. Obviously this can be configured. Right?


              Ron wrote:

              In order to replace invocations with callbacks, a callback acknowledgement facility was added to Remoting. When ServerConsumerEndpoint.Deliverer sends a callback, it also stores the callback in a pendingCallbacks HashSet, from which it is removed upon acknowledgement.


              According to Tim, we don't need that. Sending the message from server to client should be fully asynchronous, so we don't need acknowledgments. More on this here http://www.jboss.org/index.html?module=bb&op=viewtopic&t=92954. Care to comment, Tim?

              Ron wrote:

              II. The new Messaging specific serialization structure interacts with callbacks to generate some added complexity in JMSWireFormat. Because of the two forms of callbacks, there are two delivery modes for callbacks, and each has to be unwound to expose the ClientDelivery. For example, callback polling delivers an ArrayList of pending callbacks, and each ClientDelivery in the list has to be processed separately. There's also another related issue. In pull or poll mode, callbacks are stored on the server, which might involve writing them to secondary storage. Currently, the application's designated serialization type determines how the callbacks are transferred to and from disk, but MessagingObjectInputStream and MessagingObjectOutputStream are derived from the standard java classes, which can't handle Messaging non-Serializable messages. For now I've worked around the problem in my local version of Remoting, but the problem still needs to be resolved. Is there any reason the Messaging can't use JBoss Serialization?


              You lost me here. I'll look tomorrow at the code.

              Ron wrote:

              IV. Right now the HTTP branch runs with my own private version of Remoting. I've checked in a copy of jboss-remoting.jar in the lib directory.


              Good. As soon as we agree on changes, I would asume that Tom can release an Alpha (or Beta) so we can pick the jar from repository.


              Ron wrote:

              V. It's 4am and right now it looks like everything's broken. Great. Well, it WAS working.


              It will work again :)



              • 34. Re: Incorporating Remoting http transport into Messaging
                ovidiu.feodorov
                • 35. Re: Incorporating Remoting http transport into Messaging
                  ron_sigal

                   

                  "ovidiu wrote:

                  "Ron" wrote:


                  (a) push callbacks: a Connector (complete with ServerSocket) is created on the client side, and calling ServerInvokerCallbackHandler.handleCallback() causes an invocation to be made from server side to client side.


                  Not an option. Or not anymore, at this stage. The socket transport will be rendered irrelevant soon, one way or another, so we don't want to waste any more time on that. See http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3979236 If it currently works with the socket transport, that's good, but we need to get the multiplex transport working ASAP.



                  The polling style of callbacks will work just as well with the socket transport, which as I mention on http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3979219
                  is an alternative to the multiplex transport.

                  "ovidiu" wrote:

                  Yes, but no disk. For non persistent messages we don't care anyway, and for persistent messages, they're already stored by the channel, so we're covered. Disk at that level can only slow down things. Obviously this can be configured. Right?


                  Actually, Tom wrote a "callback store" which blocks instead of writing to disk when it's running out of heap. I think that was done at Tim's request. It can be specified with the Connector configuration

                  "ovidiu" wrote:


                  Ron wrote:

                  In order to replace invocations with callbacks, a callback acknowledgement facility was added to Remoting. When ServerConsumerEndpoint.Deliverer sends a callback, it also stores the callback in a pendingCallbacks HashSet, from which it is removed upon acknowledgement.


                  According to Tim, we don't need that. Sending the message from server to client should be fully asynchronous, so we don't need acknowledgments.


                  The first real problem I ran up against in Messaging was that sometimes ServerConsumerEndpoint.stop() would return before the client side had a chance to acknowledge all of its delivered messages. I put in acknowledgements to prevent that problem. Now, stop() waits until all outstanding deliveries have been handed over to the MessageCallbackHandler. This duplicates the earlier behavior when callbacks were handled explicitly by Messaging.

                  • 36. Re: Incorporating Remoting http transport into Messaging
                    ron_sigal

                    Here's an update on the current state of the http experimental branch.

                    With a couple of minor exceptions, the remote tests, in-vm tests, crash tests, and stress tests are passing for both socket and http transports. I haven't tried the smoke test yet.

                    1. Remote tests:
                    (a) socket: all pass
                    (b) http: all pass

                    2. In-vm tests:
                    (a) socket: 1 failure: a ClassCastException in org.jboss.test.messaging.core.plugin.postoffice.cluster.DefaultClusteredPostOfficeTest
                    (b) http: 1 failure: same

                    3. Crash tests:
                    (a) socket: all pass
                    (b) http: 1 failure: org.jboss.test.messaging.jms.crash.CallbackFailureTest.testCallbackFailure() fails. This test disables the mechanism which detects that a connection has failed and then tests that a failed message delivery (in ServerConsumerEndpoint.Deliverer) would lead to the same result, i.e., cleaning up the connection's resources. The problem is that when the http transport is being used, there is no client invocation on the server side - messages are pulled from the server to the client. I'm not sure what to do about this failure, but then, I don't really understand the test, since normally the connection listener would not be disabled.

                    4. Stress tests:
                    (a) socket: all pass
                    (b) http: all pass

                    There are some cases in which error messages appear in messaging-tests-remote.trace.log, even though the tests pass, but I have verified that they also occurred in the http experimental branch immediately after it was forked. Perhaps they have already been fixed on HEAD.

                    • 37. Re: Incorporating Remoting http transport into Messaging
                      clebert.suconic

                      Ron:

                      What's the relationship between multiple invokers (HTTP, Socket), ConnectionFactories and ServerPeer?

                      I'm working with HA support, and I need to know how we would have these relationships on the code.



                      Clebert Suconic

                      • 38. Re: Incorporating Remoting http transport into Messaging
                        timfox

                        The connection factory specifies which connector to use

                        http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/guide-1.0.1.GA/html/configuration.html#conf.connections

                        There can be many connectors active at any one time per server peer.

                        • 39. Re: Incorporating Remoting http transport into Messaging
                          clebert.suconic

                          I wanted to know if anything was changed regarding ConnectionFactories for the HTTP support.

                          • 40. Re: Incorporating Remoting http transport into Messaging
                            timfox

                            If you want to use the HTTP transport you just specify the connector in the cf. This should be no different than any other transport.

                            • 41. Re: Incorporating Remoting http transport into Messaging
                              ron_sigal

                              Hi Clebert,

                              "clebert" wrote:

                              What's the relationship between multiple invokers (HTTP, Socket), ConnectionFactories and ServerPeer?


                              I didn't make any changes to ConnectionFactories or ServerPeer. [Actually, I made a slight change to ConnectionFactoryJNDIMapper and then removed it.]

                              -Ron

                              • 42. Re: Incorporating Remoting http transport into Messaging
                                ovidiu.feodorov

                                Ron, are all your changes checked in on the branch?

                                • 43. Re: Incorporating Remoting http transport into Messaging
                                  ron_sigal

                                   

                                  "ovidiu.feodorov@jboss.com" wrote:
                                  Ron, are all your changes checked in on the branch?


                                  1. All of the current source code changes are checked in. I'm looking at a change to callback acknowledgements that would entail a slight change in Messaging, but I haven't done that yet.

                                  2. I'm not sure how to change the contents of the thirdparty directory.

                                  • 44. Re: Incorporating Remoting http transport into Messaging
                                    ovidiu.feodorov

                                    Ron,

                                    I've done a first pass over your changes. I have several questions/things to fix:

                                    1. Is jboss-remoting.jar in lib 2.2.0.Alpha3, or it contains private changes? If it contains private changes, I need the source code, or better still, check in your changes on the Remoting 2x branch and tag it somehow, so I can get the right snapshot.

                                    2. Looks like we need tomcat-http.jar, tomcat-util.jar, etc. This is fine, but Messaging should not guess what version is needed, the build process should pick this information automatically from Remoting's component-info.xml in the repository. Take a look at serialization's http://repository.jboss.com/jboss/serialization/1.0.3.GA/component-info.xml , see how Clebert declared a dependency of a specifc release of trove. Remoting needs to do the same thing with tomcat. Now I am using an arbitrary tomcat version declared in build-thirdparty.xml, and this is not correct.

                                    3. CallbackManager: why is the lookup value calculated based on consumerID AND serverID. Can a CallbackManager receive callbacks originating from different servers? Why is serverID needed there?