1 2 3 Previous Next 67 Replies Latest reply on Apr 4, 2006 7:26 AM by adrian.brock

    JBossRetro weaver issues

    starksm64

      I'm working with the JBossRetro prototype Adrian checked in:

      "adrian@jboss.org" wrote:

      I haven't done any extra work on it, except wrapping it in an ant task - I just cloned Bill's AopC task.

      It even uses itself, in that some of the code is Java5 but the generated jar contains code compatibile with Java1.4. :-)

      Basic TODOs left in the source code
      see also the JIRA task:
      http://jira.jboss.com/jira/browse/JBBUILD-188

      * Externalise the class mapping, e.g.
      java.lang.StringBuilder -> java.lang.StringBuffer or better for JBoss java.lang.StringBuilder -> org.jboss.util.JBossStringBuilder

      * Externalise the editor expressions:
      e.g. like the one that came up yesterday
      java.util.Collections.list(Enumeration) ->
      org.jboss.util.collection.CollectionsUtil.list(Enumeration)

      * Complete the LDC rewrite to cope with large constant pools i.e. LDC_W

      To get the project:
      cvs -d:ext:user@cvs.forge.jboss.com:/cvsroot/jboss co jbossretro


      The current Weaver.doWeave(ClassLoader, CompilerClassInfo, Map, ExprEditor[]) seems a broken because there is a usage of a javassist ClassFile representation and then an incomplete usage of the ExprEditors. The problem is that there is no bidirectional consistency between the CompilerClassInfo.ClassFile and CompilerClassInfo.CtClass as well as no writing of the ExprEditor instrument results. Even if there was a write of the ExprEditor instrument results, it seems that this would simply be overwriten by the class renaming and ConstantPool mods done on the ClassFile.

      I saw that the CtClass updates were not be written out when I tried to add a CtClass.instrument(CodeConverter) for the new autoboxing utility method calls(java.lang.Integer.valueOf(int) mapped to a jboss AutoboxCodeConverter.valueOf(int) method for example).

      It seems that we should only be dealing with the CtClass representation when performing modifications and then invoking CtClass.write(...) to commit the changes on return from doWeave. I'm putting together some tests to validate the basic functionality as currently there none and will be looking to make this change.


        • 1. Re: JBossRetro weaver issues
          starksm64

          I completed support for enums and LDC_W handling. The enum support requires jdk5 Class apis so the weaver is itself no longer usable under jdk1.4.x, and the weaving of the non-runtime support classes has been removed.

          Note that Thomas found yet another translator as noted on the JBBUILD-188 case, http://retrotranslator.sourceforge.net/, and this seems quite complete. I'm going to build out a comprehensive tests collection to validate it as well as a jboss retroweaver if we need to move forward with our own.

          • 2. Re: JBossRetro weaver issues
            starksm64

            The expanded unit tests run using Retrotranslator-0.9.5 are descent, with every failure matched by a warning during the translation. The failures are due to missing ported class features:

            retro:
             [retro] Transforming 27 file(s) from C:\cvs\JBossHead\jbossretro\output\test
            -classes to C:\cvs\JBossHead\jbossretro\output\test-classes14.
             [retro] Transformation of 27 file(s) completed successfully.
             [retro] Verifying 27 file(s) in C:\cvs\JBossHead\jbossretro\output\test-clas
            ses14.
             [retro] org.jboss.test.enums.Card:
             [retro] Cannot find java.util.EnumMap
             [retro] org.jboss.test.enums.SimpleEnumTypesTestCase:
             [retro] Cannot find method: static net.sf.retrotranslator.runtime.java.util.EnumSet_ range(net.sf.retrotranslator.runtime.java.lang.Enum_,net.sf.retrotrans
            lator.runtime.java.lang.Enum_) in net.sf.retrotranslator.runtime.java.util.EnumSet_
             [retro] org.jboss.test.util.AllConcurrentTestCase:
             [retro] Cannot find method: boolean add(edu.emory.mathcs.backport.java.util.concurrent.Delayed) in edu.emory.mathcs.backport.java.util.concurrent.DelayQueue
             [retro] Verification of 27 file(s) completed with 3 warning(s).
            
            tests:
             [junit] Running org.jboss.test.annotations.AnnotationsTestCase
             [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.485 sec
             [junit] Running org.jboss.test.enums.SimpleEnumTypesTestCase
             [junit] Tests run: 6, Failures: 0, Errors: 2, Time elapsed: 0.11 sec
             [junit] Test org.jboss.test.enums.SimpleEnumTypesTestCase FAILED
             [junit] Running org.jboss.test.generics.SimpleGenericTypesTestCase
             [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.047 sec
             [junit] Running org.jboss.test.other.AutoboxTestCase
             [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.062 sec
             [junit] Running org.jboss.test.util.AllConcurrentTestCase
             [junit] Tests run: 11, Failures: 0, Errors: 1, Time elapsed: 0.625 sec
             [junit] Test org.jboss.test.util.AllConcurrentTestCase FAILED
            
            BUILD SUCCESSFUL
            Total time: 11 seconds
            [starksm@banshee9100 jbossretro]$
            


            I have asked the webservices team to look into getting jdk14 translated binaries of the jbossws-1.0 codebase released to the repository for testing against the webservice tests for a real evaluation.

            The Retrotranslator is using the same org.objectweb.asm.* package as cglib for the bytecode manipulation.


            • 3. Re: JBossRetro weaver issues
              starksm64

              From, http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossBuild

              There is a retrotranslator task definition in the jboss build tools module that allows for the following prototype usage:

               <property name="jdk14.home" value="/usr/java/j2sdk1.4.2_09"/>
              ...
              
               <target name="compile"
               description="Compile all source files."
               depends="_default:compile-classes,
               _default:compile-etc"
               >
               <!-- Translate the jdk5 classes to jdk14 compatible classes -->
               <mkdir dir="${build.classes}-jdk14"/>
               <retrotranslator
               destdir="${build.classes}-jdk14"
               verify="true"
               failonwarning="true">
               <!-- Needs the 14 rt.jar -->
               <classpath location="${jdk14.home}/jre/lib/rt.jar"/>
               <classpath refid="javac.classpath"/>
               <classpath refid="retrotranslator.task.classpath"/>
               <src path="${build.classes}"/>
               </retrotranslator>
               </target>
              


              There is a retrotranslator binary component in the repository.jboss.com repository that contains the runtime jars. At a minimum the retrotranslator-runtime.jar is required at runtime. If the java.util.concurrent.* package is used, the backport-util-concurrent.jar is also required. Still working on the limitations and drawbacks of the translated code.


              • 4. Re: JBossRetro weaver issues

                I am going to giveup committing prototypes and just commit full tested code in future.
                Everytime I do it, somebody comes along and says we should use this
                practically unsupported or crappy project over here.

                If you really want to introduce more code that requires unsupportable thirdparty libraries
                with all the associated version problems
                including yet another dependency on a bytecode library go ahead.

                I'm sure you can Hibernate to agree on the versions :-)

                We already have code to parse annotations in class files in AOP based on javassist.
                Which is why I wanted to use javassist as the basis for the weaver/runtime library.
                Although, AOP has an outstanding task to just keep the relevant information rather
                than the whole javassist model of the class.

                Duplicating this work with some other library on every loaded class makes no sense to me. Especially when we can get direct support for javassist problems.

                • 5. Re: JBossRetro weaver issues

                   

                  "scott.stark@jboss.org" wrote:

                  I saw that the CtClass updates were not be written out when I tried to add a CtClass.instrument(CodeConverter) for the new autoboxing utility method calls(java.lang.Integer.valueOf(int) mapped to a jboss AutoboxCodeConverter.valueOf(int) method for example).


                  This ExprEditor stuff worked in the original prototype I did last year.
                  Maybe I broke it converting it to ant and introducing the CompilerClassInfo?
                  I obviously didn't test it, because I took out the all the crappy test expression editors
                  for the ant task.

                  • 6. Re: JBossRetro weaver issues
                    starksm64

                    This was just an expansion of the translation prototype to guage the scope of what has been done vs what needs to be done. There is an alternate discussion to moving hibernate to javassist which further pushes us in the direction of our own weaver. My interest was in getting to testing a real module such as webservices to understand whether retroweaving is really something worth pursuing.

                    There is no decision on retrotranslator vs our own at this point.

                    • 7. Re: JBossRetro weaver issues

                      So I've done some work on the prototype to prove that it is going to work.

                      I think this proves that everything we need to do is possible,
                      it just needs either:
                      * Somebody spending time completing it
                      * Complete it incrementally based on usage

                      The main thing that is missing is some kind of post processing to validate
                      the weaved classes aren't making references to classes/methods that we
                      don't have retro processing for.

                      The currently failing tests are:

                      * Annotations - I've implemetented Classes but not methods
                      * Enums - I've shelled out EnumMap/Set but not implemented them
                      * Concurrent - this test fails because backport-concurrent-util
                      actually implements

                      DelayQueue<Object>
                      when it should be
                      DelayQueue<E extends Delayed>
                      this can be trivially made to
                      work by either fixing or extending the class with
                      public void add(Delayed delayed)
                      {
                       add((Object) delayed);
                      }
                      


                      The other thing I need to investigate is whether the Enum modifier
                      is not being set because:
                      * JDK1.4 masks it
                      * javassist is not setting it correctly.
                      * We need to reset the modifier after the weaved class

                      • 8. Re: JBossRetro weaver issues

                        Also, I only added the main package of java.util.conncurrent to the list of weaved classes.

                        • 9. Re: JBossRetro weaver issues

                          What I'm really aiming for is to be able to say to developers that aren't maintaining
                          concurrent code branches in JBoss3/4 that is it is ok to use
                          Java5 features.

                          If we need to backport to JDK1.4 we will retroweave.

                          This requires qa infrastructure in place to validate that this is reliable.
                          i.e.
                          * The weaved implementations work reliably
                          * We catch things we are not weaving
                          * We can fix the weaving problems => our own implemetation
                          * We can extend the weaving infrastructure to add new features, e.g. AOP/MC
                          adding annotations dynamically => our own implementation

                          • 10. Re: JBossRetro weaver issues

                            A related issue is validating the testsuite for backport-concurrent-util:
                            http://dcl.mathcs.emory.edu/cgi-bin/viewcvs.cgi/software/util/backport-util-concurrent/test/
                            and probably cloning a JDK5 version that doesn't reference the backport classes directly.

                            • 11. Re: JBossRetro weaver issues
                              starksm64

                              Agreed. Let's see if QA can take ownership as part of the build codebase.

                              • 12. Re: JBossRetro weaver issues

                                javassist has a Dump program that is "a better javap".

                                This could be used as a basis for an ant task that checks
                                classes and methods signatures exist in the JDK1.4 runtime?

                                • 13. Re: JBossRetro weaver issues

                                  I've got all the tests working in the jbossretro project.

                                  So these can be added to cruisecontrol now.

                                  The next step is to try the retro/retrocheck tasks against JBossWS
                                  to see how much weaving we are missing.

                                  Also, the annotations stuff needs properly sorting out before this is used in
                                  production. I currently used a single pool to cache this information,
                                  when this should be a weak cache by classloader.
                                  This really needs to be redone as a javassit implemetation of this:
                                  http://docs.jboss.org/nightly/microkernel/docs/reference/en/html/reflection.html

                                  • 14. Re: JBossRetro weaver issues

                                     

                                    "adrian@jboss.org" wrote:

                                    The next step is to try the retro/retrocheck tasks against JBossWS
                                    to see how much weaving we are missing.


                                    Ryan, can you take this? I look at the TypeInfo abstraction as part of the work
                                    of the MC/AOP integration.


                                    1 2 3 Previous Next