1 2 Previous Next 19 Replies Latest reply on Apr 17, 2012 2:03 AM by nishant_hadole

    Parallel invocations of JaxWS services and getPort

    adinn

      The JBossTS/XTS multi-threaded test is failing with the trunk XTS and trunk AS. I investigated the problem and found that it relates to the use of client proxies in parallel. I think there is a big issue here which needs clarifying and probably requires a change in the Service/proxy implementation. Here's the situation and the symptoms:

       

      The test runs within a web app in the same container as the XTS service. The test thread creates 10 child threads each of which executes a TX start and TX commit for a Web Services Atomic Transaction (WS-AT) then exits. The parent joins each child thread. The problem is that some of the commit messages never get delivered so the client thread waits forever fo ra committed response and, the test hangs.

       

      The start and commit operations are implemented using JaxWS to invoke services provided by the XTS implementation. I will not go into the details of the actual sevice configuration as they are not particularly important. The important detail is that the threads all employ a specific JaxWS service -- ATTermination -- to commit the TX, invoking a one way operation. The service responds by routing a reply back to the client via another JaxWS service -- ATTerminationInitiator -- again using a one way operation.

       

      My code employs one instance of the ATTermination Service class. Each thread obtains this Service instance and calls getPort() to obtain a client proxy. The thread configures a handler via the BindingProvider API. It also obtains the message properties from the proxy and installs addressing property data using my MAP abstraction API. It then casts the proxy to the service interface and invokes the remote operation. The MAP data includes a replyto endpoint for theATTerminationInitiator configured with a reference parameter which identifies the thread/client making the request. The ATTermination service retrieves this data from the incoming request and uses it to address and tag the one way message to the ATTerminationInitiator service. The latter can use the tag to dispatch the result to the relevant thread.

       

      The problem is that on some occasions the messages received at the ATTermination Service end have the wrong tag. I traced the calls and found examples where, say, two threads would supply tag "<blah blah>:2fa" and "<blah blah>:2fe" but the service would receive two incoming requests with the same tag <blah blah>:2fe". It appears that the proxy returned to each thread is either the same object or, at least, shares the same message context with the result that one thread updates the MAP data on the request message context while another thread is in the middle of invoking the remote operation.

       

      I checked the JaxWS spec and it does not clarify whether the port returned from the getPort call is thread safe or not. My assumption has been that each getPort call is supposed to return a new port object which can be configured and invoked by a client thread independent of any parallel configuration and invocation of a port returned by a different call to getPort . Whatever the status of the spec it does not look like this is happening in our native implementation.

       

      One other way this behaviour migth be specified would be to require that each client thread employ its own instance of the Service. Thsi would also require the implementation to ensure that proxies obtained from different services could be used in a thread safe manner. That seems a bit perverse to me since the threads are actually using the same Sevrice -- they merely want to employ independent channels for communicating with it -- the port seems to me to be the correct level at which to achieve this.

       

      Also, creating the Service instance requires checking the WSDL, initialising all the endpoint and operation info etc so this appears to be an expensive operation which I don't want to do for every JaxWS request. Yet the WS-AT protocol rtequires the use of 6 different services in a given TX but rarely involves more than one, or in some cases two, JaxWS invocation per service. I could maybe mitigate some of the creation costs using a cache to store Service instances per-thread but that would still multiply the Sevrice instances unnecessaril;y. It woudl also require use of a WeakHashMap to ensure that the cache was garbage collected. This has its own awful performance implications fo rgarbage collection so I don;t want to have to go down that route. If this is the expecte dmodel then it's a pretty unattractive proposition. Anyway, it seems to me to be inappropriate since I don't need lots of copies of the service I need lots of copies of a port (i.e. proxy) associated with the same service.

       

      This thread-safe/unsafe behaviour is a critical issue which ought to be clearly documented in our code at least (it ought to be in the spec in bold font but the spec is pretty half-assed about many things so no surpirse I can't find it). Note that I cannot resolve this problem simply by introducing synchronization around the invocation of the proxy method. If I have to resort to inserting my own synchronization then I would have to synchronize from the point at which I obtain the port, maintain a lock while configuring the bindings and addressing properties and retain it throughout the duration of the call. In the case of an RPC message, I would also need keep the proxy locked while I grabbed the reply message context and retrieved and processed any data attached to it. Bye bye throughput.

       

      So, any comments regarding what should be specified here and any ideas about what is actually implemented anbd whetehr it can be made thread-safe?

        • 1. Re: Parallel invocations of JaxWS services and getPort
          ropalka

          Hi Andrew,

           

             this is the problem of broken by design JAX-WS specification

           

          Explanation:

             One of the problems when reusing proxy concurently is JAX-WS specification allows you to set invocation related properties on the proxy using BindingProvider methods. Consider the following code:

           

          public void configureSession(EchoService serviceProxy, boolean keepSession)

          {

             BindingProvider bp = (BindingProvider)serviceProxy;

             bp.getRequestContext().put( BindingProvider.SESSION_MAINTAIN_PROPERTY, keepSession );

          }


          If you would call above method on one proxy instance in e.g. 2 threads concurrently and each thread will set different keepSession policy, how would you implement this properly for the following scenario?

           

          thread1.configureSession(proxy, true);

          thread2.configureSession(proxy, false);

          thread1.callService(proxy);

          thread2.callService(proxy);

          assertThereAreSessions(thread1);

          assertThereAreNoSession(thread2);

           

          Do you see what I mean?

           

          Suggestion:

           

          Create/use service proxy per thread, not reuse it for multiple threads

           

          Related issues:

           

          https://jira.jboss.org/jira/browse/JBWS-1519

          https://jira.jboss.org/jira/browse/JBWS-2192

          https://jira.jboss.org/jira/browse/JBWS-2681

          • 2. Re: Parallel invocations of JaxWS services and getPort
            ropalka
            Note: Using ThreadLocals would not help because different threads can just configure the service proxy and different threads can just invoke the proxy.
            • 3. Re: Parallel invocations of JaxWS services and getPort
              adinn
              richard.opalka@jboss.com wrote:

               

              Hi Andrew,

               

                 this is the problem of broken by design JAX-WS specification

               

              Explanation:

                 One of the problems when reusing proxy concurently is JAX-WS specification allows you to set invocation related properties on the proxy using BindingProvider methods. Consider the following code:

               

              public void configureSession(EchoService serviceProxy, boolean keepSession)

              {

                 BindingProvider bp = (BindingProvider)serviceProxy;

                 bp.getRequestContext().put( BindingProvider.SESSION_MAINTAIN_PROPERTY, keepSession );

              }


              If you would call above method on one proxy instance in e.g. 2 threads concurrently and each thread will set different keepSession policy, how would you implement this properly for the following scenario?

               

              thread1.configureSession(proxy, true);

              thread2.configureSession(proxy, false);

              thread1.callService(proxy);

              thread2.callService(proxy);

              assertThereAreSessions(thread1);

              assertThereAreNoSession(thread2);

               

              Do you see what I mean?


              Yes, I understand that is the problme. That was what I was trying tyo explain.

               

              richard.opalka@jboss.com wrote:

               

              Suggestion:

               

              Create/use service proxy per thread, not reuse it for multiple threads

               

               

              Hm, I obviously did nto make myself clear. I am creating a new proxy for each thread. The problem is that they still interfere with each other. My code looks like this

               

              Service myService = getMyService(name, wsdl)

              final MyPort port1 = myService.getPort(endpointRef, MyPort.class, new AddressingFeature(true));

              final MyPort port2 = myService.getPort(endpointRef, MyPort.class, new AddressingFeature(true));

              Thread thread1 = new Thread() {
                configurePort(port1);
                port1.myMethod()
              }
              Thread thread2 = new Thread() {
                configurePort(port2);
                port2.myMethod()
              }
              thread1.start();
              thread2.start();

               

              So, the problem appears to be happening even when I create a different port for each thread. Occasionally, the data associated with port 1 is overwritten and I end up with two calls on the server side both using the data from port 2. The data in question is set by a JaxWS handler on the calling side and read by a JaxWS handler on the server side. The client adds a header and the server removes it and uses it to set up the current transaction for the thread in a ThreadLocal.

               

              I traced the code using Byteman to bytecode-inject print statements into the client just before the call to the proxy method and into the server as it enters the service method. Wen I run with 10 threads in parallel I occasionally see distinct values on the client side getting replaced with one value on the server side. If I switch on any other trace or inject more printouts the error does not normmally show because the timing of the threads changes.

              • 4. Re: Parallel invocations of JaxWS services and getPort
                asoldano

                Please see my comment on https://jira.jboss.org/jira/browse/JBWS-2324 . Starting from JBWS Native 3.2.0, the Service should be thread safe. The proxy is still not thread safe and the jaxws spec does not requires it to be thread safe. This said:

                - threre's been some discussion about this with Darran in the past and we might even think about working on this to make the port/proxy thread safe under some given assumptions (but this work does not have high priority)

                - I'd need to re-read your post Andrew to see if this might fit your usecase (perhaps it does not provide a good solution in terms of perf there), but I know support is suggesting users to create pools of proxies to be accessed by consumers threads. That should work with the current implementation; if that's not the case, we have a bug which need to be raised and fixed.

                • 5. Re: Parallel invocations of JaxWS services and getPort
                  ropalka

                  Hi Andrew,

                   

                     I thought this was fixed with https://jira.jboss.org/jira/browse/JBWS-2681

                   

                  What version of JBossWS are you using? I guess AS trunk i.e. JBossWS 3.3.0.Beta3, right?

                   

                  If yes, could you provide the test case and you think JBWS-2681 should fix it, then reopen it, please.

                  Otherwise create new issue in JBossWS and provide test case there.

                   

                  Richard

                  • 6. Re: Parallel invocations of JaxWS services and getPort
                    asoldano
                    ok Andrew, that's a bug. Isn't it the same as JBWS-2192 ?
                    • 7. Re: Parallel invocations of JaxWS services and getPort
                      adinn

                      My test case is a little bit too complicated to just hand over. It is one of the WSTX tests which is implemented in the XTS WSTX test war which is deployed to the maven repo along with the XTS release. The broken test is one of about 30 tests implemented in the war. These  tests only run as a complete test suite using a web form to drive the run and report outcomes. Also, you need to deploy the XTS sar wth the test war as the broken test relies upon communicating with 4 of the XTS services. To make it more complicated the problem does not show up reliably on every run as it is dependent upon the timing of the service requests.

                       

                      I will see if I can create a simpler test which isolates the problem. I suspect it will be possible to improve repeatability by adding more threads to the mix.

                       

                      And yes I am running my tests on the latest AS trunk so the WS version is 3.3.0.Beta3

                       

                      I am not sure it is the same bug as JBWS-2681. The problem there appears to happen when ports are created. This problem happens when two independently created ports are used.

                       

                      It also seems to me to be different to JBWS-2192 which appears to relate to using a single port for a series of Async calls. However, I guess that depends upon what is going on under the covers. One interesting thing about this case is that the error manifests when I am invoking OneWay requests. Is there something similar about how these requests are excuted and the way Async requests get executed which might be causing te problem?

                       

                      By the way, I didn't mention this in the preceding notes but I occasionally saw another intermittent and unpredictable error on the server side. My server code employs an injected WebServiceContext resource. Occasionally, the service call falls over with a null pointer exception when attempting to dereference the injected context. I don't know if this relates to the same problem or is some other issue. Anyway, it also appears to be related to the parallelism involved in this test. Could the problem be that the same service endpoint implementation instance is being entered in parallel below multiple service requests? It is not clear to me what the JaxWS spec requires but I have assumed that each service invocation is executed using a newly created instance. Is this correct?

                       

                       

                       


                      • 8. Re: Parallel invocations of JaxWS services and getPort
                        ropalka

                        Q: Each service invocation is executed using a newly created instance. Is this correct?

                        A: No. Service is reused since JBossWS 3.1.1.GA, see: https://jira.jboss.org/jira/browse/JBWS-2486

                        • 9. Re: Parallel invocations of JaxWS services and getPort
                          ropalka
                          • 10. Re: Parallel invocations of JaxWS services and getPort
                            ropalka

                            Andrew Dinn wrote:

                             

                            I will see if I can create a simpler test which isolates the problem.

                             

                            Please, try it and provide the necessary info.

                             

                            Richard

                            • 11. Re: Parallel invocations of JaxWS services and getPort
                              adinn

                              richard.opalka@jboss.com wrote:

                               

                              Q: It is not clear to me what the JaxWS spec requires but I have assumed that each service invocation is executed using a newly created instance. Is this correct?

                              A: No. Service is reused.

                               

                              Hmm, okay, so that explains what is going on here. My service method is obtaining the message context from the WebServiceContext. If two messages come in, the first in thread A and the second in thread B then the WebServiceContext can get overwritten as follows

                               

                              Thread A                   Thread B
                              enter myMethod
                              inject
                              WebServiceContext
                                                         enter myMethod
                                                         inject WebServiceContext
                              read WebServiceContext
                                                       read WebServiceContext

                              uses data for 2nd call

                               

                              This seems to me to be a tad incoherent. Even if the service method makes no reference to the state of the service implementation instance the injected data is not able to be used safely without introducing some sort of synchronization.

                               

                              If injection happens inside the service method (e.g. using an AOP transform which does the injection inside a synchronization) then it could be made thread safe by making myMethod synchronized.

                              Thread A                   Thread B
                              enter myMethod
                              inject
                              WebServiceContext
                              read WebServiceContext
                              uses data fro 1st call
                              . . .
                              exits
                                                         enter myMethod
                                                         inject WebServiceContext
                                                         read WebServiceContext

                               

                              If the injection happens before calling myMethod (e.g. using an AOP  transform which does the injection then calls the synchronized method) then  the service implementation cannot do anything about this:

                               

                              Thread A                   Thread B
                              inject
                              WebServiceContext
                                                         inject WebServiceContext
                              enter myMethod
                              read WebServiceContext
                              uses data for 2nd call
                                                         enter myMethod
                                                       read WebServiceContext

                               

                              So, which way round does it work?

                              • 12. Re: Parallel invocations of JaxWS services and getPort
                                ropalka

                                Andrew Dinn wrote:

                                 

                                Q: So, which way round does it work?


                                I'm affraid this is another question for our JAX-WS EG

                                We implemented https://jira.jboss.org/jira/browse/JBWS-2486

                                because of performance and lifecycle issues.

                                There's @PostConstruct and @PreDestroy lifecycle requirement

                                in JAX-WS 2.2 spec that defines these lifecycle methods.

                                We had users who were complaining these methods were called

                                so many times, how many requests came in, see:

                                http://community.jboss.org/message/337861#3327860


                                However your usecase requires it to not be shared and be created instance per request.

                                This introduces real performance penalty that incorporates:

                                * bean instantiation (calling constructor) (per request)

                                * scanning the bean class for lifecycle annotations (@PostConstruct & @PreDestroy) (per request)

                                * calling the @PostConstruct before method invocation (per request)

                                * calling the @PreDestroy after method invocation (per request)


                                • 13. Re: Parallel invocations of JaxWS services and getPort
                                  adinn

                                  Ok, I'm beginning to get the message here: the EG obviously didn't do a very thorough job up to 2.1. What boggles me is that all these issues have been thought about when it comes to EJBs. Let's hope Alessio can resolve some of these problems.

                                   

                                  The web servcie context and its uderlying message  context is the specified way of passing state between handler and service bean so it is important to make this work efficiently. It is still possible to make accesses  thread safe. Make the injected context is a single static wrapper instance which contains a ThreadLocal. Aroudn each call the Web Service code can install and then remove thread-specific implementation instances of the current service context. The wrapper can indirect API calls to these per-thread instances to retrieve thread specific data.

                                   

                                  That avoids the need to synchronize the whole web method when all that is really required is thread-safe access to the underlying message context and, from there, data cached by the handler. This will significantly benefit throughput. This is extremely important because your decision to use only a single bean is a serious potential bottleneck. Even if the bean itself does not contain any shared state it cannot employ the message context without synchronizing all service methods and hence serialising all calls. Obviously the handler and thread could use their own thread local to communicate data in a thread-safe manner. However, rather than require each application to replicate a mechanism already defined in the spec it would be better to make the specified mechanism work safely.

                                  • 14. Re: Parallel invocations of JaxWS services and getPort
                                    ropalka

                                    Hi Andrew,

                                     

                                       you're right, I was thinking too much in our implementation details

                                    Cross reference issue: https://jira.jboss.org/jira/browse/JBWS-2934

                                     

                                    Rio

                                    1 2 Previous Next