4 Replies Latest reply: Feb 29, 2012 4:53 PM by Flavia Rainone RSS

Local Vars Info Missing after Retransformation

Flavia Rainone Master

Hi!

I have a rule that refers to the local var $connHandlerFuture on its body:

 

RULE retrieve ConnectionHandlerFactory Future
CLASS org.jboss.remoting3.EndpointImpl
METHOD doConnect
HELPER org.jboss.remoting3.test.racecondition.FutureHelper
AT INVOKE org.jboss.remoting3.spi.ConnectionProvider.connect
IF TRUE
DO
    debug("retrieving ConnectionHandlerFactory"),
    trackFuture(org.jboss.remoting3.test.racecondition.FutureHelper.CONNECTION_HANDLER_FACTORY, $connHandlerFuture.getIoFuture())
ENDRULE

 

This rule is part of CancelOpeningChannelTestCase and, while the rule above works if I run the test isolated, the rule stops from working if I run all tests in a batch, with mvn test.

 

After some investigation, I realized the failure is related with the previous test in the batch, CloseConnectingEndpointTestCase. Such test contains a rule that is applied to the same EndpointImpl.doConnect method:

 

RULE doConnect calls resourceUntick
CLASS org.jboss.remoting3.EndpointImpl
METHOD doConnect
AFTER INVOKE org.jboss.remoting3.EndpointImpl.resourceUntick
IF TRUE
DO
    debug("ResourceUntick invoked by doConnect... waking closeAction"),
    signalWake("resourceUntick for connection", true),
    debug("ResourceUntick waiting for closeAction to complete before proceeding"),
    # waitFor should expire if connect and closeAction are correctly synchronized, otherwise this test will hang
    waitFor("connections to Array", 100)
ENDRULE


 

Debugging showed me that, somehow, on the first attempt to transform the bytecodes of EndpointImpl, for CloseConnectingEndpointTestCase, all local variables are accounted for (org.jboss.byteman.agent.adapter.RuleMethodAdapter.visitLocalVariable is invoked during the EndpointImpl bytecode visit process). But, when transformation for the problematic CancelOpeningChannelTestCase begins, no local variable is visited, and that's why my rule is not being applied to EndpointImpl.doConnect.

 

Could the retransformation process, performed to revert the transformation in the end of CloseConnectingEndpointTestCase execution, be erasing the local var info from the class? This is the impression I get while debugging, that the EndpointImpl.doConnect bytecodes are lacking the local var info when Byteman attempts to transform EndpointImpl for CancelOpeningChannelTestCase.

  • 1. Re: Local Vars Info Missing after Retransformation
    Andrew Dinn Master

    Hi Flavia,

    Flavia Rainone wrote:

     

    Hi!

    I have a rule that refers to the local var $connHandlerFuture on its body:

     

    RULE retrieve ConnectionHandlerFactory Future
    CLASS org.jboss.remoting3.EndpointImpl
    METHOD doConnect
    HELPER org.jboss.remoting3.test.racecondition.FutureHelper
    AT INVOKE org.jboss.remoting3.spi.ConnectionProvider.connect
    IF TRUE
    DO
        debug("retrieving ConnectionHandlerFactory"),
        trackFuture(org.jboss.remoting3.test.racecondition.FutureHelper.CONNECTION_HANDLER_FACTORY, $connHandlerFuture.getIoFuture())
    ENDRULE
    
    

     

    This rule is part of CancelOpeningChannelTestCase and, while the rule above works if I run the test isolated, the rule stops from working if I run all tests in a batch, with mvn test.

     

    When you say run the test isolated what exactly do you mean? Are you using BMUnit to run it? Or are you using Byteman from the Java command line? I suspect that this relates to JIRA BYTEMAN-28 but I would like to know the details of the failure case to be sure I understand what is happening.

     

    Flavia Rainone wrote:

     

    After some investigation, I realized the failure is related with the previous test in the batch, CloseConnectingEndpointTestCase. Such test contains a rule that is applied to the same EndpointImpl.doConnect method:

     

    RULE doConnect calls resourceUntick
    CLASS org.jboss.remoting3.EndpointImpl
    METHOD doConnect
    AFTER INVOKE org.jboss.remoting3.EndpointImpl.resourceUntick
    IF TRUE
    DO
        debug("ResourceUntick invoked by doConnect... waking closeAction"),
        signalWake("resourceUntick for connection", true),
        debug("ResourceUntick waiting for closeAction to complete before proceeding"),
        # waitFor should expire if connect and closeAction are correctly synchronized, otherwise this test will hang
        waitFor("connections to Array", 100)
    ENDRULE
    
    
    
    
    

     

    Debugging showed me that, somehow, on the first attempt to transform the bytecodes of EndpointImpl, for CloseConnectingEndpointTestCase, all local variables are accounted for (org.jboss.byteman.agent.adapter.RuleMethodAdapter.visitLocalVariable is invoked during the EndpointImpl bytecode visit process). But, when transformation for the problematic CancelOpeningChannelTestCase begins, no local variable is visited, and that's why my rule is not being applied to EndpointImpl.doConnect.

     

    Could the retransformation process, performed to revert the transformation in the end of CloseConnectingEndpointTestCase execution, be erasing the local var info from the class? This is the impression I get while debugging, that the EndpointImpl.doConnect bytecodes are lacking the local var info when Byteman attempts to transform EndpointImpl for CancelOpeningChannelTestCase.

    This may indeed be a possibility. When I last looked at the JVM code (i.e.when BYTEMAN-28 was reported) I thought the situation was as follows.

     

    If the JVM loaded a class and a transformer modified it at load time then the JVM would retain the complete original bytecode. So, any subsequent attempt to retransform the class would start with the bytecode obtained from the class file and this would include the local variable information.

     

    If the bytecode was not transformed at load time then the JVM would drop it. So, any later attempt to retransform the class required the JVM to reconstitute the bytecode from the internal in-memory represenation of the class and method code, meaning the bytecode would not include local variable info.

     

    So,  my impression  was that if the bytecode was retained at load then it would be kept for every subsequent retransformation. However, from your evidence it sounds like it may throw away bytecode if some intermediate transform decided not to modify it. This may be a recent change or may be how it always was. In order to explain what I mean by that here is the scenario I am talking about in detail.

     

    BMUNit executes the first test

    The agent installs the rule for class EndpointImpl

    Since EndpointImpl has not yet been loaded it cannot be scheduled for retransformation

    The test code runs and causes EndpointImpl to be loaded

    The agent transformer is passed the original loaded byecode for EndpointImpl and modifies it

    EndpointImpl is transformed at load so the original bytecode is retained by the JVM

    The test completes

    The agent uninstalls the rule for class EndpointImpl

    The agent schedules EndpointImpl for retransformation

    The agent is passed the original loaded byecode for EndpointImpl and returns it unmodified

    A) ?????

     

    BMUNit executes the second test

    The agent installs the 2nd rule for class EndpointImpl

    Since EndpointImpl has already been loaded it is scheduled for retransformation

    B) The agent is passed ????? byecode for EndpointImpl and modifies it

    . . .

     

    So, how do we fill  in the blanks at A) and B)  What I assumed on first reading the code was

     

    A) The JVM retains the original bytecode

     

    B) The agent is passed the original byecode for EndpointImpl and modifies it

     

    However, what might be happening is

     

    A) The JVM drops the original loaded bytecode because it has not been transformed

     

    B) The agent is passed the reconstituted byecode for EndpointImpl and fails to modify it

     

    I suspect this is what is happening but I'll have to check the JVM code to be sure.

     

    Anyway, the upshot of this is that until BYTEMAN-28 is fixed you can only reliably use $localVar in your rules if you drive Byteman from the Java command line. Note that this is not the only way things can go wrong, merely a new wrinkle on the original problem. Imagine your first test did not use a rule attached to EndpointImpl but instead that it ran some code which caused the class to be loaded. Your second test would still fail because the class would have been loaded and the original bytecode dropped.

     

    I did implement a JVM patch for this a long time ago but for reasons far too complex to go into have not yet been able to get it into OpenJDK. I'm hoping I will be able to fix this in the next few months, assuming I don't get diverted from OpenJDK work yet again. It's a nice small patch which I can use as a way of learning how to jump all the hurdles whihc are in the way of pushing code upstream into OpenJDK.

  • 2. Re: Local Vars Info Missing after Retransformation
    Flavia Rainone Master

    Andrew Dinn wrote:

     

    Hi Flavia,

     

    When you say run the test isolated what exactly do you mean? Are you using BMUnit to run it? Or are you using Byteman from the Java command line? I suspect that this relates to JIRA BYTEMAN-28 but I would like to know the details of the failure case to be sure I understand what is happening.

    I mean running the test using either Eclipse JUnit's plugin or using mvn test with the -Dtest option. The test uses  BMUnit plugin.

    If you need to take a look at the tests, or you if would like to see the failure in action, you can check this commit:

    https://github.com/fl4via/jboss-remoting/commit/7ea021db00874acd302eeb0ebbabea171e20b68d

     

    Running the tests with mvn clean install or mvn test will cause CancelOpeningConnectionTestCase to fail. If you run the failing test with

     

     mvn test -Dtest=org.jboss.remoting3.test.racecondition.CancelOpeningConnectionTestCase
    

     

    the test will pass or it will freeze sometimes (there is a deadlock I'm investigating right now).

     

    Andrew Dinn wrote:

     

    So,  my impression  was that if the bytecode was retained at load then it would be kept for every subsequent retransformation. However, from your evidence it sounds like it may throw away bytecode if some intermediate transform decided not to modify it. This may be a recent change or may be how it always was. In order to explain what I mean by that here is the scenario I am talking about in detail.

     

    I am not sure if there is an intermediate transformation deciding not to modify it. Based on my debugging observations, I think the affected method is transformed when the first test loads EndpointImpl, then is reverted to the original code after the test finishes, and is retransformed when the second test starts.

     

    For that reason, I think it could be related to BYTEMAN-28, but I am not sure of it. Anyway, I'll try to change the rule to avoid this bug for now.

  • 3. Re: Local Vars Info Missing after Retransformation
    Andrew Dinn Master

    Flavia Rainone wrote:


    I am not sure if there is an intermediate transformation deciding not to modify it. Based on my debugging observations, I think the affected method is transformed when the first test loads EndpointImpl, then is reverted to the original code after the test finishes, and is retransformed when the second test starts.

     

    Given how you say the test is being run the frist rule for EndpointImpl will be unloaded after  the first test completes. The agent listener will schedule the class for a retransform and the agent transformer will be asked if it wants to transform the original bytecode for EndpointImpl and will say no because it has no rule for the class. So, this will definitely present the JVM with the option to drop the untransformed bytecode between the first and second test. Whether it does I don't yet know but dropping ti at this point sems to be the obvious explanation for the outcome you have seen. Anyway, if I can fix the JVM bug identfiied in BYTEMAN-28 this problem will definitely go away because the local var info will be included in the reconstituted bytecode. So that is the priority here.

     

    regards,

     

     

    Andrew Dinn

  • 4. Re: Local Vars Info Missing after Retransformation
    Flavia Rainone Master

    Thanks for the explanation, Andrew, now I know what you mean!

    In that case, it certainly is related to BYTEMAN-28.