5 Replies Latest reply on Mar 21, 2012 6:11 AM by adinn

    Keeping byteman scripts in sync with code they target

    waterscolin

      Hi,

       

      First I want to say a big thank you for releasing byteman. I've managed to use it successfully in the last few days to "augment" production code with additional trace functionality with no impact on readability of functional code. Really cool stuff.

       

      One I've been thinking about was keeping the scripts I have written in-sync with the code that they are targetting. Ideally I was aiming to have a JUnit test which can take the scripts, take the code the scripts point at - then give me a red bar if the scripts didn't compile any more.

       

      I saw the TestScript class which was cool as it can check the script compilation. But the tools text output is tricky to doing Asserts with in JUnit. So I wanted to run an idea past the group for comment:

      Would it be possible to split TestScript into the command line utility and an underlying class with a public method which can return compilation results for rules in something like a (boolean ruleCompiledOk, String compileMsgForRule) form?

       

      This then allows you to easily write a JUnit test which can Assert ruleCompiledOk flags for each rule and so you can get your green bar if scripts are in sync with code / red bar if they are not.

       

      I managed to achieve the same effect by wrapping TestScript in another class which intercepts the console output and processes it into that form. But I guess that would be kind of fragile as the script compiler itself changes over time. I've created a small project (attached) to illustrate my current wrapping of TestScript. It is not the way to go - but it does illustrate the structured form of the compile results + their usage in JUnit.

       

      Happy to contribute if this idea makes sense.

       

      All the best and thanks again for building such a cool tool!

      Col.

        • 1. Re: Keeping byteman scripts in sync with code they target
          adinn

          Hi Colin,

          Colin Waters wrote:

           

          First I want to say a big thank you for releasing byteman. I've managed to use it successfully in the last few days to "augment" production code with additional trace functionality with no impact on readability of functional code. Really cool stuff.

           

          Thank you very much. Now please go say this to all your friends  . . .  :-)

          Colin Waters wrote:

           

          One I've been thinking about was keeping the scripts I have written in-sync with the code that they are targetting. Ideally I was aiming to have a JUnit test which can take the scripts, take the code the scripts point at - then give me a red bar if the scripts didn't compile any more.

           

          I saw the TestScript class which was cool as it can check the script compilation. But the tools text output is tricky to doing Asserts with in JUnit. So I wanted to run an idea past the group for comment:

          Would it be possible to split TestScript into the command line utility and an underlying class with a public method which can return compilation results for rules in something like a (boolean ruleCompiledOk, String compileMsgForRule) form?

           

          This then allows you to easily write a JUnit test which can Assert ruleCompiledOk flags for each rule and so you can get your green bar if scripts are in sync with code / red bar if they are not.

           

          This feature has actually been requested already and JIRA issue  Byteman-190 has been raised for it. Actually, what you outline is just the preliminary step for this JIRA, the second step being to implement maven/ant plugins which run before the test phase and use the programmatic API to validate rules.

           

          Colin Waters wrote:


          I managed to achieve the same effect by wrapping TestScript in another class which intercepts the console output and processes it into that form. But I guess that would be kind of fragile as the script compiler itself changes over time. I've created a small project (attached) to illustrate my current wrapping of TestScript. It is not the way to go - but it does illustrate the structured form of the compile results + their usage in JUnit.

           

          Happy to contribute if this idea makes sense.

           

          Thank you and well done for trying this approach and also for posting your code. However, you are right that this is probably not the best solution. If you want to work on the JIRA then I will be happy to help with the design, review any changes you make and, eventually, incorporate them into the project master repo. Contributors to the project are always welcome. I suspect that all that is needed is to extend TestScript with a suitable API which returns a set of results identifying either a successful, failed or in-doubt transformation outcome for each rule and then to modify the main routine to simply print the results returned from this API.

           

          I'll note that I can only install your patches in the Byteman master repo if you sign a JBoss contributor agreement (an individual agreement if you are doing this in your own time or a corporate one if you are doing it on behalf of your employer). Basically, this affirms that the code you have produced is your own work and is not encumbered by undeclared copyrights or by any  patents you own i.e. it ensures Red Hat (and, therefore, everyone) can continue to use the code without any legal hindrance. Note that it does not take away any of your rights to use the code. You don't need to sign this agreement in order to copy and modify Byteman but I cannot pull your contribution into the master repo without it.

          • 2. Re: Keeping byteman scripts in sync with code they target
            waterscolin

            Hi Andrew,

             

            I've been thinking about how I might report compilation status back from the TestScript tool for JIRA-190. I have some design thoughts that I wanted to run by you.

             

            1-Easiest way of getting structured compile information back from TestScript is changing the signature of testScript(...) to return a Collection of CompileResults. This would allow the script to still call in via main() without change, but allow a Maven plugin to call testScript(...) directly and get some results that it can do something with. Is there a policy on changing publically visible method signatures though? This approach would also require TestScript() constructor to be public, would that be an issue?

            2-For design of the objects that will hold compile results: At a basic level I want to capture the compile result for a single rule.

               This can be in the form:

               RuleCompileResult

                -String ruleName

               -CompileStatus (SUCCEED, WARNING, FAIL)

               -String compileMsg

            I then thought about how you would collate these at the level of a Script file. It would be quite natural to hold them in a Map<String, RuleCompileResult>, but rather than just offer this raw Collection it would be nicer to wrap that in a ScriptCompileResult. That would allow you wrap some methods around the collection so that it is just one call such as ScriptCompileResult.didCompileOk() to determine if your compile succeeded at the file level. You could also hold error, warning and success counts here. Other benefits of having a ScriptCompileResult class is that sometimes you will get a compile fail at the script rather than rule level (i.e. when you can't read in a file), so it gives you somewhere that you can set that flag and message.

             

            I have started writing some code around this approach to see how it fits, but thought it wise to get some feedback on design direction before spending too much time heading in that direction.

             

            All the best,

            Colin.

            • 3. Re: Keeping byteman scripts in sync with code they target
              adinn

              Hi Colin,

               

              Thanks very much for taking on this task. I'm happy to offer what advice I can.

               

              Addressing point 1:

               

              • Yes, a good starting point for the restructuring is testScript(). You need to convert it and, below it, checkRules to return a result set.
              • You could make the constructor public. However, you could instead provide a static method to creates the instance and ten call testScript() -- the static could take an extra boolean argument supplying the verbose flag passed to the constructor. Since the plugin will only call testScript() once the latter is just as good and will be simpler for the caller.

               

              Addressing point 2:

               

              There are a variety of errors which are not related to parse/typecheck problems e.g. a script file may not be found or may not be readable. So, not all results can be of type RuleCompileResult. I think there are two options here: either you provide an outcome class (or aset fo such classes which can incorporate a range of different details specific to the outcome in qestion; or you reduce the outcome to its essentials and use instances of that one class to generate maven output and an overall result.

               

              For example the first approach would include amongst the different possible outcomes both a FileError containing the name of the script file and the relevant FileException and a TypeCheckWarning containing the scriptfile, line number and name of the offending rule, the name of the trigger class and method into which the rule was injected plus a message detailing the type mismatch which raised the warning and the TypeWarningException thrown by the type checker. By contrast the second approach would simply employ a a notification type (INFO or [TYPE] ERROR, to continue with the examples above), a detail message string including all the other relevant info and, possibly, a releated Exception instance.

               

              Now, maven and other tools which might drive the checker will mostly want to count and display the warnings and errors without really caring about how they arise i.e. they would only really need approach 2. The only thing I can think of which might want to adopt approach 1 is an IDE plugin.

               

              So, the question is what are the essential details that the client requires in each approach? If you look at how TestScript currently notifies results you will see that it counts various types of errors and warning, it prints a message appropriate to the error/warning at hand and it also sometimes provides details of a thrown exception. So, I think the basic elements you need in your result are

               

                NotificationType: INFO, WARNING, ERROR

                SubType: NONE, PARSE, TYPE

                Message: "..."

                Throwable: possibly null

               

              The maven plugin can count the warnings and/or errors to decide whether or not to continue or fail the build. Obviously maven should stop when there are errors but, depending upon policy (i.e. the setting of a boolean attribute in the plugin), it may allow the build to continue when there are only warnings.

               

              The details of the rule, script file, line number and error would simply be folded into the message with this approach (as happens at present). So, the maven plugin would simply print to an output file the type of each outcome followed by its message text . Alternatively it could just simply print this as part of the build output.  It can also use these counts to print a summary at the end as main currently does. If a throwable is provided then maven could print the throwable with the associated message. So, theoutput woudl look like

               

                ERROR : Unable to open script file foo.btm

                TYPE WARNING : bar.btm line 7 : No matching trigger point for trigger class org.my.Bar method myMethod(String)

                org.jboss.byteman.rule.exception.TypeWarningException : no matching trigger point for trigger class org.my.Bar method myMethod(String)

               

              n.b. I just changed the trigger injecton code to fix BYTEMAN-199 and, as a result, I modified testScript slightly just to print the exception but not the exception backtrace

               

              Now, if you want to be able to feed say an IDE plugin then you would probably also need to include these fields in the result

               

                ScriptFile (possibly null)

                ScriptLine: only significant for SubType in { PARSE, TYPE }

                TriggerClass: only significant for SubType in { TYPE }

                TriggerMethod: only significant for SubType in { TYPE } (and still possibly null)

               

              In this case the message would have to omit these details and there would have to be some standard way of formatting the output to include them e.g. they could be printed in a standard format like:

               

                 <SubType> <Type> <ScriptFile> line <ScriptLine> : <message> for trigger class <TriggerClass> method <TriggerMethod>

               

              obviously, omitting elements which are not present in the result.

              • 4. Re: Keeping byteman scripts in sync with code they target
                waterscolin

                Hi Andrew,

                 

                Thanks for the advice.

                 

                So far I've gone with the second approach that you mention - the general container from which it is possible to generate counts.

                The work is still in progress for sure, but I thought I'd share the code as that makes thing more concrete and allows for good feedback from

                you and the community.

                 

                Attached is a git patch containing the code modifications that I have made so far. I'm pretty new to both git and the open source world, so I'm not totally sure what form code is normally passed about in.

                Hopefully this will work, but I can send a straight zip if it is better.

                 

                The patch contains the work-in-progress on JIRA-190. So far I have looked at the CompileResult container and the modification to TestScript, but I have not started thinking about the Maven plugin.

                Now what are the main things to signpost...

                • I have CompileResults classes at several levels.
                  • RuleCompileResult : Simple container and holds Rulename, CompileStatus, CompileSubStatus, CompileMsg and Throwable as you suggested.
                  • ScriptCompileResult : Holds a Map of Rulename -> <List of ScriptCompileResult>

                               The reason for the List of ScriptCompileResult was that I found some cases where you could get several errors against a single rule. (I think this was on InstallParams, but I need to remember)

                               This class also holds a flag for failure at the script level - such as the file not being found error you mentioned in your mai

                               Finally it also holds a CompileCounts object which tracks successful, error or warning compiles + the parse and type sub counts. I've tried to have it that the counts are incremented automatically when a                RuleCompileResult is added into the ScriptCompileResult container.    

                  • CompileResults : This is the top level container for holding all CompileResults from a TestScript run. It exists to hold a Map of filename -> ScriptCompileResult but also as a place to implement aggregation for counts and isCompiledOk() & getCompileStatus()
                • Also included is a modification of TestScript to actually use the above CompileResults objects.
                • There is a TestCompileCheck test that I've added along with some associated scripts.
                  • The initial idea was to try and trigger every different type of compile error, but that is the part I am still working on (and struggling with in some cases).
                  • As it's a coverage task, I've listed the locations of all the different cases I could find in the attached spreadsheet + the associated test and whether I've figured out how to write a script that produces that compile error yet!
                  • FYI: Current status for TestCompileCheck is 12 failures, 13 pass, with the failure coming from non-complete tests.

                 

                There are a number of things in the API that feel a bit odd at the moment

                • Having both getCompileStatus(), isCompiledOk(). Are both necessary?
                • Separate CompileStatus and CompileSubStatus parameters. Might feel nicer if combined into a single CompileStatus object
                • The check for existing RuleCompileResult that have to do before can confirm that status=SUCCESS feels a bit ugly.

                 

                Anyway, I'm open to feedback, so let me know your thoughts.

                 

                Colin.

                 

                PS: One misc question I have is is about running the existing unit tests. Is there a clever way to run all of them from Eclipse? (or do you just run them from Maven?)

                The reason I ask is that I could setup a number of Run Configurations to get individual tests to pass, or sub-batches of tests to pass but I hit issues on the misc package where one script would interfere with another. So I can run them all, but I am wondering if I've missed a trick.

                • 5. Re: Keeping byteman scripts in sync with code they target
                  adinn

                  Hi Colin,

                   

                  Thanks for the progress report. I'm afraid I am not going to be able to look at this until next week -- I am busy preparing for a planning meeting this Thursday/Friday. I'll try to look at the code on Monday.

                   

                  In answer to your initial question about git, yes a git patch works. However, what others have done in the past might be easier for both you and others in the long run.

                   

                  Previously contributors have cloned the master repo into their own github copy, made changes in a local copy of their clone and then made local commits visible by uploading (i.e. git pushing) them to their own github remote repo (for example, I have a clone of GIT master in GIT adinn). Working this way makes it easy for other developers (in particular, me) to pull your commits into their own local copy and test/reconcile them with any changes they may have been making in parallel -- or evenjust to run tests/debug and sanity check that everything is working as advertised. Bing able to share change sets like this is one of git's main strengths and using a public repo to make the change sets available makes it much easier to manage.

                   

                  n.b. you would not need to edit and then publish your changes to github in the master branch. You can always do your work in a different branch.

                   

                  regards,

                   

                   

                  Andrew Dinn