10 Replies Latest reply: Aug 3, 2010 5:40 AM by Kabir Khan RSS

Benchmarking classloaders

Kabir Khan Master

For AS 6 we might want to look at setting up imports and exports for the classloaders, which might be faster than using the current ImportAll/Big-ball-of-mud setting. However, that sounds like a very big task, so I we should benchmark this in isolation to see if imports/exports will actually be faster. I have something simple working for ImportAll, but need to think a bit about how to do this with imports and exports.

  • 1. Re: Benchmarking classloaders
    David Bosschaert Expert

    Hi Kabir,

     

    I just implemented something like this for the OSGi performance benchmarks. Here's a quick description of the bundle dependency topology that it creates:

    5 (Versioned) Common Bundles
       - Exports org.jboss.osgi.test.common;version=x
    5 Numbered (but not versioned) Util Bundles
       - Imports org.jboss.osgi.test.common
       - Exports org.jboss.osgi.test.util[x];uses="org.jboss.osgi.test.common"
    5 Versioned Interfaces Bundles
       - Exports org.jboss.osgi.test.versioned;version=x
    5 Versioned Impl Bundles
       - Imports org.jboss.osgi.test.common;version=[x,x]
       - Imports org.jboss.osgi.test.versioned;version=[x,x]
       - Imports org.jboss.osgi.test.util[x]
       - Exports org.jboss.osgi.test.versioned.impl;version=x;uses=org.jboss.osgi.test.util[x]
    a large number of test bundles (number configurable)
       - Imports org.jboss.osgi.test.common;version=[x,x]
       - Imports org.jboss.osgi.test.versioned;version=[x,x]
       - Imports org.jboss.osgi.test.versioned.impl;version=[x,x]

    Where x is a number [1..5].

     

    Each test bundle loads a class of each of its 3 dependency packages in its activator. This also triggers an indirect load on the Util[x] class.

     

    It's available on master in jbosgi on github: BundleInstallAndStartBenchmark.java

     

    You might be able to reuse some of this...

     

    David

  • 2. Re: Benchmarking classloaders
    Kabir Khan Master
    I have got some basic benchmarks working. Instead of using AS classes, I am now generating my own using Chiba's ClassFileWriter to have better control over what exists where. The classes are only generated if the target/generated-classes and target/generated-jars directories don't exist already. You tell it how many jars to create, how many packages per jar and how many classes per package. If you change these numbers, be sure to delete the target/generated-* directories. So, if 2 is selected for each of these it then creates the following jars:
    interface0.jar:
    org.jboss.test.interface0.pkg0.Interface0.class
    org.jboss.test.interface0.pkg0.Interface1.class
    org.jboss.test.interface0.pkg1.Interface0.class
    org.jboss.test.interface0.pkg1.Interface1.class
     
     
    interface1.jar:
    org.jboss.test.interface1.pkg0.Interface0.class
    org.jboss.test.interface1.pkg0.Interface1.class
    org.jboss.test.interface1.pkg1.Interface0.class
    org.jboss.test.interface1.pkg1.Interface1.class
     
     
    abstract0.jar:
    org.jboss.test.abstract0.pkg0.Abstract0.class
    org.jboss.test.abstract0.pkg0.Abstract1.class
    org.jboss.test.abstract0.pkg1.Abstract0.class
    org.jboss.test.abstract0.pkg1.Abstract1.class
     
     
    abstract1.jar:
    org.jboss.test.abstract1.pkg0.Abstract0.class
    org.jboss.test.abstract1.pkg0.Abstract1.class
    org.jboss.test.abstract1.pkg1.Abstract0.class
    org.jboss.test.abstract1.pkg1.Abstract1.class
     
     
    impl0.jar:
    org.jboss.test.impl0.pkg0.Impl0.class
    org.jboss.test.impl0.pkg0.Impl1.class
    org.jboss.test.impl0.pkg1.Impl0.class
    org.jboss.test.impl0.pkg1.Impl1.class
     
     
    impl1.jar:
    org.jboss.test.impl1.pkg0.Impl0.class
    org.jboss.test.impl1.pkg0.Impl1.class
    org.jboss.test.impl1.pkg1.Impl0.class
    org.jboss.test.impl1.pkg1.Impl1.class
    
    An example of the expected inheritance hierarchies is:
    class org.jboss.test.impl1.pkg1.Impl1 extends org.jboss.test.abstract1.pkg1.Abstract1{}
    class org.jboss.test.abstract1.pkg1.Abstract1 implements org.jboss.test.interface1.pkg1.Interface1{}
    
    I then create a classloader for each jar, with different classloading rules per test and try to load the Impl classes from the implx.jar loaders, which triggers searches for the superclasses/interfaces in the other classloaders.
    Ales, can you take a look at how I am setting up the VFSClassLoaderFactories? I do this in the TestCase classes. The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.
    Running this a few times on a set of 50 jars with 10 packages per jar and 10 classes per package, I get the following results.:

     

    Import/Export packages:
    Deploying the VFSClassLoaderFactories and creating the loaders: 1862ms 1693ms 1847ms 1725ms 1706ms
    Loading the classes: 5008ms 4883ms 4744ms 4893ms 5012ms
     
     
    Export/ImportAll:
    Deploying the VFSClassLoaderFactories and creating the loaders: 1657ms 1627ms 1659ms 1720ms 1573ms
    Loading the classes: 4858ms 4693ms 4796ms 4714ms 4899ms
     
     
    Import/Export module:
    Deploying the VFSClassLoaderFactories and creating the loaders: 1785ms 1736ms 1766ms 1819ms 1901ms
    Loading the classes: 4992ms 5609ms 5510ms 4991ms 5626ms
    
    
  • 3. Re: Benchmarking classloaders
    Ales Justin Master
    Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

    Should be fine.

     

    So to get the tests straight.

    Why the three level hierarchy depth?

    Who creates requirements and who provides the capabilities?

    Why the re-export on module and package?

     

    My first feeling is that this could be made simpler -- more straight fwd; no grandparent, parent, etc.

     

    The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

    What happens if you don't do this?

  • 4. Re: Benchmarking classloaders
    David Bosschaert Expert

    FYI the OSGi benchmarks that I wrote work on the following three platforms:

      MC-based JBoss OSGi (-Dframework=jbossmc)

      MSC / jboss-modules based JBoss OSGi (-Dframework=jbmsc)

      Felix (-Dframework=felix)

     

    There are various populations available (and its trivial to add more). E.g. you can run the bundle tests with 750 bundles for all three framework with the following command lines:

    jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbossmc

    jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=jbmsc

    jbosgi/testsuite/performance> mvn install -Dtest=Bundle750TestCase -Dframework=felix

     

    A number of populations for both bundle tests and service tests are run mightly in this hudson job: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix

     

    For the latest results, see here: http://jbmuc.dyndns.org:8280/hudson/job/jbosgi-performance-matrix/ws/testsuite/performance/target/performance-report.html

  • 5. Re: Benchmarking classloaders
    Kabir Khan Master

    Ales Justin wrote:

     

    Ales, can you take a look at how I am setting up the VFSClassLoaderFactories?

    Should be fine.

     

    So to get the tests straight.

    Why the three level hierarchy depth?

    The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

    Ales Justin wrote:

    Who creates requirements and who provides the capabilities?

    Why the re-export on module and package?

    The requirements and capabilities are set up in the individial tests. I'm  not sure about the re-export, is it needed when Impl (loaderA) loads its superclass AbstractImpl (loaderB) which in turn brings in its interface (loaderC)?

    The thing I found strange was that for the Module test I had to specify the packages as a capability as well as the module, but I'm not that familiar with how this works so maybe that's how it should be.

    What happens if you don't do this?

    I get an exception, IIRC NoClassDefFoundError when trying to load the superclass which comes from a loader that only exports the module and not also the packages. But maybe there is another setting to get around this?

  • 6. Re: Benchmarking classloaders
    David Bosschaert Expert

    Kabir Khan wrote:

     

    Ales Justin wrote:

     

    Why the three level hierarchy depth?

    The three levels was inspired by David's stuff, as I understood it a load of one class triggers load of some others. It can certainly be made simpler though, so I can do a one-level set tomorrow.

    I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

     

    A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

  • 7. Re: Benchmarking classloaders
    Kabir Khan Master

    David Bosschaert wrote:

    I created the bundle bench mark test model to follow typical OSGi deployments. Multiple levels are very common in that case, for instance:

     

    A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

    I understand and agree, but I think for this Ales wants a bare-bones comparison between exact classloading and the full visibility model, in this case it will probably be clearer with just one level of classes. I'm working on this now, and should hopefully have something in time for today's meeting.

  • 8. Re: Benchmarking classloaders
    Ales Justin Master
    A customer bundle uses a product delivered as OSGi bundles. The product has an API and an Implementation. The product uses some services from the compendium which are implemented by a number of util bundles. Voila - that's the test case structure.

    This doesn't sound like it would need 3-level hierarchy, but rather just 3 properly wired bundles at the same level.

  • 9. Re: Benchmarking classloaders
    Kabir Khan Master

    The new simpler scenario has a 100 jars with 10 packages and 15 classes per package, one loader per jar.

     

    The tests are called SiblingVFSXXXBenchmarkTestCase and they run as follows:

     

    Use Loader1 to load all its own classes and those from Loader2

    Use Loader2 to load all its own classes and those from Loader3

    ...

     

    For the exact classloading setups I make Loader 1 import the module/packages exported by Loader2 etc.

     

    I run the tests individually using e.g.

    mvn install -Dtest=SiblingVFSImportPackageLoaderBenchmarkTestCase

     

    Running each test individually, I get these average results over 7 runs

     

     

     


    Deploy (ms)Load Classes (ms)
    Big ball of mud14394485
    Package13214880
    Module14614874

     

    The code lives in svn under https://svn.jboss.org/repos/jbossas/projects/cl-benchmark/trunk

     

    Each family of tests uses its own AbstractTestSetCreator implementation which generates jars under their own directory, e.g. ThreeDeepTestSetCreator and SiblingTestCreator. The tests currently

  • 10. Re: Benchmarking classloaders
    Kabir Khan Master

    The times with the latest jboss-cl trunk:

     

     


    Deploy (ms)First Load (ms)Load Cached
    Big ball of mud97766912
    Package109370860
    Module120571060

     

    • Deploy is the time taken to install the classloader
    • First Load is the time taken to load the class first time
    • Load Cached is the time taken to load the class once it is already loaded

    In all cases the class comes from another loader than the initiating loader