4 Replies Latest reply on Feb 29, 2012 4:53 PM by flavia.rainone

    Local Vars Info Missing after Retransformation

    flavia.rainone

      Hi!

      I have a rule that refers to the local var $connHandlerFuture on its body:

       

      RULE retrieve ConnectionHandlerFactory Future
      CLASS org.jboss.remoting3.EndpointImpl
      METHOD doConnect
      HELPER org.jboss.remoting3.test.racecondition.FutureHelper
      AT INVOKE org.jboss.remoting3.spi.ConnectionProvider.connect
      IF TRUE
      DO
          debug("retrieving ConnectionHandlerFactory"),
          trackFuture(org.jboss.remoting3.test.racecondition.FutureHelper.CONNECTION_HANDLER_FACTORY, $connHandlerFuture.getIoFuture())
      ENDRULE
      
      

       

      This rule is part of CancelOpeningChannelTestCase and, while the rule above works if I run the test isolated, the rule stops from working if I run all tests in a batch, with mvn test.

       

      After some investigation, I realized the failure is related with the previous test in the batch, CloseConnectingEndpointTestCase. Such test contains a rule that is applied to the same EndpointImpl.doConnect method:

       

      RULE doConnect calls resourceUntick
      CLASS org.jboss.remoting3.EndpointImpl
      METHOD doConnect
      AFTER INVOKE org.jboss.remoting3.EndpointImpl.resourceUntick
      IF TRUE
      DO
          debug("ResourceUntick invoked by doConnect... waking closeAction"),
          signalWake("resourceUntick for connection", true),
          debug("ResourceUntick waiting for closeAction to complete before proceeding"),
          # waitFor should expire if connect and closeAction are correctly synchronized, otherwise this test will hang
          waitFor("connections to Array", 100)
      ENDRULE
      
      
      

       

      Debugging showed me that, somehow, on the first attempt to transform the bytecodes of EndpointImpl, for CloseConnectingEndpointTestCase, all local variables are accounted for (org.jboss.byteman.agent.adapter.RuleMethodAdapter.visitLocalVariable is invoked during the EndpointImpl bytecode visit process). But, when transformation for the problematic CancelOpeningChannelTestCase begins, no local variable is visited, and that's why my rule is not being applied to EndpointImpl.doConnect.

       

      Could the retransformation process, performed to revert the transformation in the end of CloseConnectingEndpointTestCase execution, be erasing the local var info from the class? This is the impression I get while debugging, that the EndpointImpl.doConnect bytecodes are lacking the local var info when Byteman attempts to transform EndpointImpl for CancelOpeningChannelTestCase.

        • 1. Re: Local Vars Info Missing after Retransformation
          adinn

          Hi Flavia,

          Flavia Rainone wrote:

           

          Hi!

          I have a rule that refers to the local var $connHandlerFuture on its body:

           

          RULE retrieve ConnectionHandlerFactory Future
          CLASS org.jboss.remoting3.EndpointImpl
          METHOD doConnect
          HELPER org.jboss.remoting3.test.racecondition.FutureHelper
          AT INVOKE org.jboss.remoting3.spi.ConnectionProvider.connect
          IF TRUE
          DO
              debug("retrieving ConnectionHandlerFactory"),
              trackFuture(org.jboss.remoting3.test.racecondition.FutureHelper.CONNECTION_HANDLER_FACTORY, $connHandlerFuture.getIoFuture())
          ENDRULE
          
          

           

          This rule is part of CancelOpeningChannelTestCase and, while the rule above works if I run the test isolated, the rule stops from working if I run all tests in a batch, with mvn test.

           

          When you say run the test isolated what exactly do you mean? Are you using BMUnit to run it? Or are you using Byteman from the Java command line? I suspect that this relates to JIRA BYTEMAN-28 but I would like to know the details of the failure case to be sure I understand what is happening.

           

          Flavia Rainone wrote:

           

          After some investigation, I realized the failure is related with the previous test in the batch, CloseConnectingEndpointTestCase. Such test contains a rule that is applied to the same EndpointImpl.doConnect method:

           

          RULE doConnect calls resourceUntick
          CLASS org.jboss.remoting3.EndpointImpl
          METHOD doConnect
          AFTER INVOKE org.jboss.remoting3.EndpointImpl.resourceUntick
          IF TRUE
          DO
              debug("ResourceUntick invoked by doConnect... waking closeAction"),
              signalWake("resourceUntick for connection", true),
              debug("ResourceUntick waiting for closeAction to complete before proceeding"),
              # waitFor should expire if connect and closeAction are correctly synchronized, otherwise this test will hang
              waitFor("connections to Array", 100)
          ENDRULE
          
          
          
          
          

           

          Debugging showed me that, somehow, on the first attempt to transform the bytecodes of EndpointImpl, for CloseConnectingEndpointTestCase, all local variables are accounted for (org.jboss.byteman.agent.adapter.RuleMethodAdapter.visitLocalVariable is invoked during the EndpointImpl bytecode visit process). But, when transformation for the problematic CancelOpeningChannelTestCase begins, no local variable is visited, and that's why my rule is not being applied to EndpointImpl.doConnect.

           

          Could the retransformation process, performed to revert the transformation in the end of CloseConnectingEndpointTestCase execution, be erasing the local var info from the class? This is the impression I get while debugging, that the EndpointImpl.doConnect bytecodes are lacking the local var info when Byteman attempts to transform EndpointImpl for CancelOpeningChannelTestCase.

          This may indeed be a possibility. When I last looked at the JVM code (i.e.when BYTEMAN-28 was reported) I thought the situation was as follows.

           

          If the JVM loaded a class and a transformer modified it at load time then the JVM would retain the complete original bytecode. So, any subsequent attempt to retransform the class would start with the bytecode obtained from the class file and this would include the local variable information.

           

          If the bytecode was not transformed at load time then the JVM would drop it. So, any later attempt to retransform the class required the JVM to reconstitute the bytecode from the internal in-memory represenation of the class and method code, meaning the bytecode would not include local variable info.

           

          So,  my impression  was that if the bytecode was retained at load then it would be kept for every subsequent retransformation. However, from your evidence it sounds like it may throw away bytecode if some intermediate transform decided not to modify it. This may be a recent change or may be how it always was. In order to explain what I mean by that here is the scenario I am talking about in detail.

           

          BMUNit executes the first test

          The agent installs the rule for class EndpointImpl

          Since EndpointImpl has not yet been loaded it cannot be scheduled for retransformation

          The test code runs and causes EndpointImpl to be loaded

          The agent transformer is passed the original loaded byecode for EndpointImpl and modifies it

          EndpointImpl is transformed at load so the original bytecode is retained by the JVM

          The test completes

          The agent uninstalls the rule for class EndpointImpl

          The agent schedules EndpointImpl for retransformation

          The agent is passed the original loaded byecode for EndpointImpl and returns it unmodified

          A) ?????

           

          BMUNit executes the second test

          The agent installs the 2nd rule for class EndpointImpl

          Since EndpointImpl has already been loaded it is scheduled for retransformation

          B) The agent is passed ????? byecode for EndpointImpl and modifies it

          . . .

           

          So, how do we fill  in the blanks at A) and B)  What I assumed on first reading the code was

           

          A) The JVM retains the original bytecode

           

          B) The agent is passed the original byecode for EndpointImpl and modifies it

           

          However, what might be happening is

           

          A) The JVM drops the original loaded bytecode because it has not been transformed

           

          B) The agent is passed the reconstituted byecode for EndpointImpl and fails to modify it

           

          I suspect this is what is happening but I'll have to check the JVM code to be sure.

           

          Anyway, the upshot of this is that until BYTEMAN-28 is fixed you can only reliably use $localVar in your rules if you drive Byteman from the Java command line. Note that this is not the only way things can go wrong, merely a new wrinkle on the original problem. Imagine your first test did not use a rule attached to EndpointImpl but instead that it ran some code which caused the class to be loaded. Your second test would still fail because the class would have been loaded and the original bytecode dropped.

           

          I did implement a JVM patch for this a long time ago but for reasons far too complex to go into have not yet been able to get it into OpenJDK. I'm hoping I will be able to fix this in the next few months, assuming I don't get diverted from OpenJDK work yet again. It's a nice small patch which I can use as a way of learning how to jump all the hurdles whihc are in the way of pushing code upstream into OpenJDK.

          1 of 1 people found this helpful
          • 2. Re: Local Vars Info Missing after Retransformation
            flavia.rainone

            Andrew Dinn wrote:

             

            Hi Flavia,

             

            When you say run the test isolated what exactly do you mean? Are you using BMUnit to run it? Or are you using Byteman from the Java command line? I suspect that this relates to JIRA BYTEMAN-28 but I would like to know the details of the failure case to be sure I understand what is happening.

            I mean running the test using either Eclipse JUnit's plugin or using mvn test with the -Dtest option. The test uses  BMUnit plugin.

            If you need to take a look at the tests, or you if would like to see the failure in action, you can check this commit:

            https://github.com/fl4via/jboss-remoting/commit/7ea021db00874acd302eeb0ebbabea171e20b68d

             

            Running the tests with mvn clean install or mvn test will cause CancelOpeningConnectionTestCase to fail. If you run the failing test with

             

             mvn test -Dtest=org.jboss.remoting3.test.racecondition.CancelOpeningConnectionTestCase
            

             

            the test will pass or it will freeze sometimes (there is a deadlock I'm investigating right now).

             

            Andrew Dinn wrote:

             

            So,  my impression  was that if the bytecode was retained at load then it would be kept for every subsequent retransformation. However, from your evidence it sounds like it may throw away bytecode if some intermediate transform decided not to modify it. This may be a recent change or may be how it always was. In order to explain what I mean by that here is the scenario I am talking about in detail.

             

            I am not sure if there is an intermediate transformation deciding not to modify it. Based on my debugging observations, I think the affected method is transformed when the first test loads EndpointImpl, then is reverted to the original code after the test finishes, and is retransformed when the second test starts.

             

            For that reason, I think it could be related to BYTEMAN-28, but I am not sure of it. Anyway, I'll try to change the rule to avoid this bug for now.

            • 3. Re: Local Vars Info Missing after Retransformation
              adinn

              Flavia Rainone wrote:


              I am not sure if there is an intermediate transformation deciding not to modify it. Based on my debugging observations, I think the affected method is transformed when the first test loads EndpointImpl, then is reverted to the original code after the test finishes, and is retransformed when the second test starts.

               

              Given how you say the test is being run the frist rule for EndpointImpl will be unloaded after  the first test completes. The agent listener will schedule the class for a retransform and the agent transformer will be asked if it wants to transform the original bytecode for EndpointImpl and will say no because it has no rule for the class. So, this will definitely present the JVM with the option to drop the untransformed bytecode between the first and second test. Whether it does I don't yet know but dropping ti at this point sems to be the obvious explanation for the outcome you have seen. Anyway, if I can fix the JVM bug identfiied in BYTEMAN-28 this problem will definitely go away because the local var info will be included in the reconstituted bytecode. So that is the priority here.

               

              regards,

               

               

              Andrew Dinn

              • 4. Re: Local Vars Info Missing after Retransformation
                flavia.rainone

                Thanks for the explanation, Andrew, now I know what you mean!

                In that case, it certainly is related to BYTEMAN-28.