9 Replies Latest reply on Nov 2, 2016 2:34 PM by spatwary04

    [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"

    sunil_dixit

      jboss-ejb-client (Version 1.0.18.Final and above) has one bug (garbage collection related) in "public void unregisterEJBReceiver(final EJBReceiver receiver)" API.

       

      This API uses "ExecutorService" to send close notification to registered listeners (if any) asynchronously.

      Now if your InitialContext is eligible for garbage collection then it JVM calls RemoteContext.finalize() method which internally result into calling "org.jboss.ejb.client.EJBReceiverContext.close()" which triggers

      "org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver()" .

       

      Before "org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver() " gets executed "org.jboss.ejb.client.EJBClientContext.ejbClientContextTasksExecutorService" state is changed from RUNNING to TERMINATED i.e. ( runState is being set to 3) which result into "java.util.concurrent.RejectedExecutionException".

       

       

      ThreadPoolExecutor.addIfUnderMaximumPoolSize() API will always return false as " if (poolSize < maximumPoolSize && runState == RUNNING)" prevents to create new thread.

       

       

      Finally caller APIs gets following exception

       

       

      java.util.concurrent.RejectedExecutionException

              at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)

              at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

              at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)

              at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:432)

              at org.jboss.ejb.client.EJBReceiverContext.close(EJBReceiverContext.java:59)

              at org.jboss.ejb.client.remoting.ChannelAssociation.notifyBrokenChannel(ChannelAssociation.java:404)

              at org.jboss.ejb.client.remoting.ChannelAssociation.access$100(ChannelAssociation.java:59)

              at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:118)

              at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:110)

              at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)

              at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:515)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

              at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:390)

              at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:231)

              at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAction(RemoteConnectionHandler.java:376)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

              at org.jboss.remoting3.ConnectionImpl.closeAction(ConnectionImpl.java:52)

              at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

              at org.jboss.naming.remote.client.HaRemoteNamingStore.closeAsync(HaRemoteNamingStore.java:385)

              at org.jboss.naming.remote.client.NamingStoreCache.release(NamingStoreCache.java:113)

              at org.jboss.naming.remote.client.NamingStoreCache$1.close(NamingStoreCache.java:98)

              at org.jboss.naming.remote.client.RemoteContext.finalize(RemoteContext.java:199)

              at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)

              at java.lang.ref.Finalizer.runFinalizer(Unknown Source)

              at java.lang.ref.Finalizer.access$100(Unknown Source)

              at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

       

       

       

       

                Below code in "org.jboss.ejb.client.EJBClientContext" is added into new versions and causing this problem

       

       

                      // we *don't* want to send these notification to listeners synchronously since the listeners can be any arbitrary

                      // application code and can potential block for a long time. So invoke the listeners asynchronously via our ExecutorService

                      final Collection<EJBClientContextListener> listeners = new ArrayList<EJBClientContextListener>(this.ejbClientContextListeners);

                      for (final EJBClientContextListener listener : listeners) {

                          this.ejbClientContextTasksExecutorService.submit(new Runnable() {

                              @Override

                              public void run() {

                                  try {

                                      listener.receiverUnRegistered(receiverContext);

                                  } catch (Throwable t) {

                                      // log and ignore

                                      logger.debug("Exception trying to invoke EJBClientContextListener " + listener + " for EJB client context " + EJBClientContext.this + " on un-registertation of EJBReceiver " + receiver, t);

                                  }

                              }

                          });

                      }

                  }

        • 1. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
          jmeyer_community

          Sunil,

           

          I believe I may be seeing this issue as well.  Do we know if a bug has been reported for it?  Also, do we know the impact of this "bug?"  Is it potentially harmful to the application when the error is seen?

           

          Thanks!

           

          -Jason-

          • 2. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
            sunil_dixit

            Hi Jason,

             

            I have posted the same thing on red hat support team but they are suggested different approach to prevent this exception. In this approach "Programmer has to make InitialContext  class level with static variable instead of method/block level". But I don't found this appropriate.

            So your code should look like

             

            Class EJBLookupUtil

            {

            private static Context ctx;

             

            public static Object doLookup(String jndiName)

            {

            if(null == ctx)

            {

            // steps to initialize InitailContext

            ctx = new InitialContex();

            }

            return ctx.lookup(jndiName);

            }

             

            }

             

            To fix the problem I have supplied new implementation of "java.util.concurrent.ThreadPoolExecutor" class where "defaultHandler" is changed from "AbortPolicy " to "DiscardPolicy".

            i.e. line

            private static final RejectedExecutionHandler defaultHandler =  new AbortPolicy();

            is changed with

            private static final RejectedExecutionHandler defaultHandler =  new DiscardPolicy();

             

            This new class implementation should be used by JVM and "-Xbootclasspath:/p" can be used to override JVM classes.

             

            I hope this helps.

             

            Thanks,

            Sunil Dixit

            • 3. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
              henrikdeluxe

              Hi,

               

              got the same thing here on my clients.

              It makes no difference if i call close() explicit on my InitialContext or if GarbageCollctor does - the exception is the same.

               

              An Workaround like override JVM-Classes wont be my first choice.

               

              I'm using jboss-ejb-client version 1.0.23.Final-redhat-1  (EAP 6.1.1)

              Does anyone know if there is an issue in jiira?

              • 4. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
                jaikiran

                Please go ahead and file a bug in EJBCLIENT JIRA project

                • 5. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
                  henrikdeluxe

                  Hi, i did not found the rigth Jira to file AS7 Bugs, i only found one for Wildfly

                   

                  I will contact my RedHat Support with that subject today - may they will provide a patch.

                  • 7. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
                    henrikdeluxe

                    Hi Wolf-Dieter,

                     

                    so I tell my anxious customers that thier production systems might not be at risk when I supress that exception in thier error logfile?

                    That isn't exactly my imagination of an professional customer support :-(

                     

                    Whats my benefit of paying for subscription, when getting the bugfix as recently as anybody else with the next lgpl release?

                    • 8. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
                      wdfink

                      The EJBCLIENT-98 is the community upstream issue for that problem. If you use a community version you might be ignore the ERROR, use a provided ejb-client library which contain the fix, or use github integrate the bugfix and rebuild the server by yourself.

                       

                      If you pay for a subscription this is a different story and you need to address that in the customer support portal, you might reference this issue and ask for a fix for your special version if needed. That is the benefit if you pay with a sbscription for professional support.

                      • 9. Re: [Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"
                        spatwary04

                        I am using wildfly 10 with  jboss-ejb-client-2.1.4. I am getting the same issue. This is happening  when i am creating the initialcontext on method level and closing it in finally block.

                         

                        2016-11-02 13:45:23,621 SEVERE [com.test.TestBean] (EJB default - 21) Error: Service Locator Access Exception: java.io.IOException: Channel Channel ID caa8971a (outbound) of Remoting connection 5c95caa5 to /hostip:port has been closed

                            at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:123) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

                            at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:115) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

                            at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)

                            at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:514)

                            at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:419)

                            at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:290)

                            at org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:274)

                            at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)

                            at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:154)

                            at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.disassociate(RemotingConnectionEJBReceiver.java:266) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

                            at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:446) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

                            at org.jboss.ejb.client.EJBClientContext.close(EJBClientContext.java:1295) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

                            at org.jboss.naming.remote.client.ejb.RemoteNamingStoreEJBClientHandler$RemoteNamingEJBClientContextCloseTask.close(RemoteNamingStoreEJBClientHandler.java:118) [jboss-remote-naming-2.0.4.Final.jar:2.0.4.Final]

                            at org.jboss.naming.remote.client.RemoteContext.finalize(RemoteContext.java:299) [jboss-remote-naming-2.0.4.Final.jar:2.0.4.Final]

                            at java.lang.System$2.invokeFinalize(System.java:1267) [rt.jar:1.8.0_25]

                            at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98) [rt.jar:1.8.0_25]

                            at java.lang.ref.Finalizer.access$100(Finalizer.java:34) [rt.jar:1.8.0_25]

                            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210) [rt.jar:1.8.0_25]