Hi,

I am in the process of running performance tests to evaluate different JCR implementations. To do this I have ported the jackrabbit-jcr-benchmark package from the Jackrabbit project so that I can build it against Jackrabbit 1.6 and 2.3 as well as Modeshape.

All the tests run on Modeshape with the Memory connector, but most of them hang when using Jpa (Hsqldb or Mysql). I'm not quite sure what is happening, but it looks like there is some kind of infinite loop during session.save(). Thes tests have been run against 2.6.0-FINAL, 2.7.0-FINAL and the current 2.8.0-SNAPSHOT (ModeShape-modeshape-4110d75).

I have attached full sources.

In the class below you'll see from the comment in runTest() how to make this test succeed - otherwise it hangs in afterSuite().

To run the tests, build the bits you want (i.e. jcr-perf-suite and jcr-perf-modeshape) and run com.mom.jcrperf.TestRunner. Example configurations are in jcr-perf-modeshape/config.

At the end of this post you will also find two sets of results: one for Jackrabbit 1.5.7 (a slightly wacky custom build) running against a MySql database in Amazon, the other Modeshape 2.7.0-FINAL with an in-memory connector. As you can see, the results are fairly comparable. Results using the Jpa connector (once the tests have been modified to allow them to run) are off by orders of magnitude. Is there something broken in my configuration, or is Modeshape really this much slower than Jackrabbit?

Thanks for any help,

Duncan

public class TransientSomeChildNodesTest extends PerformanceTest {


    private static final int CHILD_COUNT = 100;


    private static final int ITERATIONS = 100;


    private Session session;


    private Node node;


    public void beforeSuite() throws RepositoryException {
        session = getRepository().login(getCredentials());
        node = session.getRootNode().addNode("testnode", "nt:unstructured");
        for (int i = 0; i < CHILD_COUNT; i++) {
            node.addNode("node" + i, "nt:unstructured");
        }
    }


    public void beforeTest() throws RepositoryException {
    }


    public void runTest() throws Exception {
        for (int i = 0; i < ITERATIONS; i++) {
            node.addNode("onemore", "nt:unstructured").remove();
        }
        // Test will run to completion on Modeshape if the following line is uncommented 
        // (but then it's not a Transient test..)
        // session.save();
    }


    public void afterTest() throws RepositoryException {
    }


    public void afterSuite() throws RepositoryException {
              session.getRootNode().getNode("testnode").remove();
              // Test will hang here when running on Modeshape
              session.save();
              session.logout();
    }


}

Test environment
  JCR: Jackrabbit version 1.5.7 by Apache Software Foundation
    Persistence Manager Configuration
          schema:mysql
          driver:com.mysql.jdbc.Driver
          bundleCacheSize:2
          schemaObjectPrefix:version_
  JRE: 1.6.0_20 by Sun Microsystems Inc.
  JVM: OpenJDK 64-Bit Server VM version 19.0-b09 by Sun Microsystems Inc.
   OS: Linux version 2.6.35.14-97.44.amzn1.x86_64 for amd64

Test case                               Mean    Sdev     Min     Max   Count
----------------------------------------------------------------------------
1000 x login()                           100     149      39     656      27
1000 x login().logout()                  108     156      45     635      49
1 000 000 x refresh(false)               150      28     104     172      34
ConcurrentReadTest                       431     158     305     825      12
ConcurrentReadWriteTest                  667     435     316    1418       8
CreateManyChildNodesTest               14772       0   14772   14772       1
TransientManyChildNodesTest               27      20      18      79     189
UpdateManyChildNodesTest                 100      78      49     333      25
VersionedTreeTest                       6028       0    6028    6028       1
VersionedTreeTest                       7126       0    7126    7126       1
SimpleSearchTest                          12      14       8      70     418
1000 x read a 10kB file                 1733      53    1695    1794       3
100 x write a 10kB file                 1250     156    1046    1405       5
read a 100MB file                          1       2       0      62    4475
write a 100MB file                      3762      30    3740    3783       2


Modeshape/memory
Test case                               Mean    Sdev     Min     Max   Count
----------------------------------------------------------------------------
1000 x login()                            83      18      70     140      59
1000 x login().logout()                   77      12      70     129      65
1 000 000 x refresh(false)                87       1      86      90      58
ConcurrentReadTest                        97      33      70     227      52
ConcurrentReadWriteTest                  343     607     132    2519      15
CreateManyChildNodesTest               18224       0   18224   18224       1
TransientManyChildNodesTest               50      10      37      88     102
TransientSomeChildNodesTest               37       5      35      67     135
UpdateManyChildNodesTest                 415       8     404     423       4
VersionedTreeTest                       5190     717    4683    5697       2
VersionedBundleTest                    11734       0   11734   11734       1
SimpleSearchTest                         233       7     228     249      22
1000 x read a 10kB file                   14       2      13      31     345
100 x write a 10kB file                   37       1      35      46      90
read a 100MB file                         37       1      36      38     137
write a 100MB file                      1337     126    1240    1513       4