1 2 Previous Next 18 Replies Latest reply on Jul 29, 2015 2:32 PM by garage_logician

    Can we introduce a new managed connection pool implementation that reduces thread contention under load?

    johara

      We have been profiling a number of JEE benchmarks that we run on WF (IJ 1.1) and EAP (IJ 1.0), and in a couple of cases we see the most contended object within the JVM is a java.util.ArrayList object (cls).  High thread contention on any object causes us scaling issues as we increase throughput, more threads on the critical path are blocked waiting to synchronize on the intrinsic lock and this creates a performance bottleneck preventing us from scaling higher.

       

      The method calls where most contention occurs are;

      • org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.isSize(int)
      • org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(Subject, ConnectionRequestInfo)

       

      IsSize is called a lot within SemaphoreArrayListManagedConnectionPool.returnConnection(ConnectionListener, boolean, boolean), so we are seeing most contention when returning a connection back to the connection pool.

       

      We have been looking at a potential alternative implementation to SemaphoreArrayListManagedConnectionPool that does not require extensive intrinsic locking on key data structures within the connection pool to eliminate the thread contention we see withing the  SemaphoreArrayListManagedConnectionPool.  The alternative implementation could be selected via the -Dironjacamar.mcp system property, leaving SemaphoreArrayListManagedConnectionPool as the default implementation.

       

      The alternative implementation that we have been working on (NonBlockingManagedConnectionPool) can be found here;

       

      https://github.com/johnaoahra80/ironjacamar/commits/1.2_nonBlockingMcp

       

      We have been testing this NonBlockingManagedConnectionPool implementation with our benchmarks and can see that the thread contention has been eliminated. The implementation is passing all the junit tests within the testsuite (changes are required to the byteman tests from SemaphoreArrayListManagedConnectionPool to NonBlockingManagedConnectionPool in order for the byteman scripts to be injected into the correct class).

       

      Running the performance testsuite, it shows higher throughput mainly in circumstances where the connection pool is highly contended.

       

      NarayanaFilePerfTestCaseNarayanaMemPerfTestCaseNoopTSPerfTestCase
      DefaultNon Blocking% differenceDefaultNon Blocking% differenceDefaultNon Blocking% difference
      Time elapsed: 7.547 sec 4.94 sec 7.468 sec 4.893 sec 6.443 sec 3.536 sec
      PERF-DATA:LocalTransaction185998,413-2.2%8,5798,6320.6%19,59319,513-0.4%
      PERF-DATA:LocalTransaction106148169,31412.7%68,51956,991-16.8%119,156140,11317.6%
      PERF-DATA:LocalTransaction25127345129,8502.0%137,307138,2390.7%129,795196,72251.6%
      PERF-DATA:LocalTransaction50156398158,8781.6%166,465204,38622.8%201,999419,632107.7%
      PERF-DATA:LocalTransaction100178719347,79994.6%208,539323,92255.3%332,415501,22450.8%
      PERF-DATA:LocalTransaction150264262524,75098.6%272,095584,342114.8%289,2531,023,686253.9%
      PERF-DATA:LocalTransaction200267673568,580112.4%262,701562,689114.2%286,7301,064,672271.3%
      PERF-DATA:LocalTransaction250252210544,979116.1%258,285574,405122.4%264,239947,305258.5%
      PERF-DATA:LocalTransaction300243594642,158163.6%256,127635,422148.1%245,277919,591274.9%
      PERF-DATA:NoTransaction173447,3440.0%73017,3020.0%1822918,8553.4%
      PERF-DATA:NoTransaction107404673,724-0.4%75,89572,543-4.4%59,277138,621133.9%
      PERF-DATA:NoTransaction2583172194,889134.3%90,805164,35081.0%116,145279,093140.3%
      PERF-DATA:NoTransaction50140919174,07323.5%132,684220,18065.9%443,173747,56568.7%
      PERF-DATA:NoTransaction100164386127,074-22.7%157,417121,349-22.9%508,124765,73650.7%
      PERF-DATA:NoTransaction1503188231,251,051292.4%298,0131,280,842329.8%466,1421,302,376179.4%
      PERF-DATA:NoTransaction2003364791,127,861235.2%352,9691,104,635213.0%495,5381,063,839114.7%
      PERF-DATA:NoTransaction2503554031,410,671296.9%360,4491,359,957277.3%498,2981,237,114148.3%
      PERF-DATA:NoTransaction3003400001,310,081285.3%338,7201,302,579284.6%470,3991,140,757142.5%
      PERF-DATA:XATransaction13809435,099-7.9%37,79839,3334.1%175,625279,41259.1%
      PERF-DATA:XATransaction10232277216,117-7.0%235515259,33310.1%321395880,533174.0%
      PERF-DATA:XATransaction25270222647,514139.6%232,424609,369162.2%392,324947,380141.5%
      PERF-DATA:XATransaction50288038658,361128.6%279,379651,952133.4%346,978909,624162.2%
      PERF-DATA:XATransaction100273321605,051121.4%271,742609,769124.4%275,187856,115211.1%
      PERF-DATA:XATransaction150262343655,093149.7%274,100653,479138.4%299,671950,689217.2%
      PERF-DATA:XATransaction200248658649,045161.0%272,798624,137128.8%292,491983,790236.3%
      PERF-DATA:XATransaction250256476652,559154.4%263,626637,521141.8%264,8841,036,293291.2%
      PERF-DATA:XATransaction300239161626,282161.9%259,713601,973131.8%243,0821,028,967323.3%
      Testcase:testNotook2.491.827.7%2.5441.76930.5%1.8781.19936.2%
      Testcase:testLocaltook2.3921.4838.1%2.3341.4737.0%2.1231.01452.2%
      Testcase:testXAtook2.0971.08548.3%2.0271.10245.6%1.9580.83257.5%
        1 2 Previous Next