1 Reply Latest reply: Jun 7, 2011 2:35 PM by Tariq Lahyani RSS

OutOfMemoryError while loading pojo cache

Tariq Lahyani Newbie

Hello -

 

We have an application using JTreeCache and I am working on a project to replace that by pojo cache.  When I start the application in our Dev box (using WebLogic 10.3), it starts loading the cache first and then I start getting this error

 

INFO [2011-06-07 11:11:07,753] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - Loading IMMPoint c

ache data ...

INFO [2011-06-07 11:11:35,001] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 50000 IMM points l

oaded

INFO [2011-06-07 11:12:02,487] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 100000 IMM points

loaded

INFO [2011-06-07 11:12:29,354] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 150000 IMM points

loaded

ERROR [2011-06-07 11:13:05,218] [Timer-1,10.31.73.5:49384] org.jgroups.protocols.UDP - failed sending message to null (67 bytes)

java.lang.Exception: dest=/228.0.1.3:39877 (70 bytes)

        at org.jgroups.protocols.UDP._send(UDP.java:353)

        at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:294)

        at org.jgroups.protocols.TP.doSend(TP.java:1478)

        at org.jgroups.protocols.TP.send(TP.java:1468)

        at org.jgroups.protocols.TP.down(TP.java:1189)

        at org.jgroups.protocols.PING.sendMcastDiscoveryRequest(PING.java:275)

        at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:256)

        at org.jgroups.protocols.Discovery$PingSenderTask$1.run(Discovery.java:406)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.InterruptedIOException: operation interrupted

        at java.net.PlainDatagramSocketImpl.send(Native Method)

        at java.net.DatagramSocket.send(DatagramSocket.java:612)

        at org.jgroups.protocols.UDP._send(UDP.java:344)

        ... 16 more

INFO [2011-06-07 11:14:34,458] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.PojoCacheManager - 200000 IMM points

loaded

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor38]

[Unloading class sun.reflect.GeneratedConstructorAccessor43]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor51]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor54]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor35]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor18]

[Unloading class sun.reflect.GeneratedMethodAccessor25]

 

After a while, the server runs out of memory:

 

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor136]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor133]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor134]

[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor131]

Heap dump file created [4266774323 bytes in 310.488 secs]

ERROR [2011-06-07 11:42:58,939] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] remedy.dispatcher - Error initializing servlet.

java.lang.OutOfMemoryError: Java heap space

        at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)

        at java.lang.StringCoding.decode(StringCoding.java:173)

        at java.lang.String.<init>(String.java:444)

        at java.lang.String.<init>(String.java:516)

        at com.sybase.jdbc3.utils.PureConverter.toUnicode(Unknown Source)

        at com.sybase.jdbc3.tds.TdsDataObject.readString(Unknown Source)

        at com.sybase.jdbc3.tds.TdsJdbcInputStream.getString(Unknown Source)

        at com.sybase.jdbc3.jdbc.SybResultSet.getString(Unknown Source)

        at weblogic.jdbc.wrapper.ResultSet_com_sybase_jdbc3_jdbc_SybResultSet.getString(Unknown Source)

        at com.jpmc.remedy.data.ResultSetReader$StringColumnReader.readColumn(ResultSetReader.java:146)

        at com.jpmc.remedy.data.ResultSetReader.readObject(ResultSetReader.java:72)

        at com.jpmc.remedy.rearc.cache.pojocache.PojoCacheManager.loadIMMPointFromDB(PojoCacheManager.java:1806)

        at com.jpmc.remedy.rearc.cache.pojocache.PojoCacheManager.loadCacheData(PojoCacheManager.java:1103)

        at com.jpmc.remedy.comm.RemedyConfig.loadCacheManager(RemedyConfig.java:281)

 

Below is my configuration file:

 

<?xml version="1.0" encoding="UTF-8"?>

<server>

   <mbean code="org.jboss.cache.pojo.jmx.PojoCacheJmxWrapper"

          name="jboss.cache:service=PojoCache">

 

 

      <depends>jboss:service=Naming</depends>

      <depends>jboss:service=TransactionManager</depends>

 

      <attribute name="TransactionManagerLookupClass">org.jboss.cache.transaction.GenericTransactionManagerLookup

      </attribute>

 

      <attribute name="IsolationLevel">READ_COMMITTED</attribute>

 

      <attribute name="CacheMode">REPL_ASYNC</attribute>

 

      <attribute name="UseReplQueue">false</attribute>

 

      <attribute name="ReplQueueInterval">0</attribute>

 

      <attribute name="ReplQueueMaxElements">0</attribute>


      <attribute name="ClusterName">Remedy-DEV-REM-PojoCache-Cluster</attribute>

 

      <!--Uncomment next three statements to enable JGroups multiplexer.

This configuration is dependent on the JGroups multiplexer being

registered in an MBean server such as JBossAS.  -->

      <!--

      <depends>jgroups.mux:name=Multiplexer</depends>

      <attribute name="MultiplexerService">jgroups.mux:name=Multiplexer</attribute>

      <attribute name="MultiplexerStack">fc-fast-minimalthreads</attribute>

      -->

 

      <!-- JGroups protocol stack properties.

         ClusterConfig isn't used if the multiplexer is enabled and successfully initialized.

      -->

      <attribute name="ClusterConfig">

         <config>

            <UDP mcast_addr="337.0.5.3"

                 mcast_port="39877"

                 tos="8"

                 ucast_recv_buf_size="20000000"

                 ucast_send_buf_size="640000"

                 mcast_recv_buf_size="25000000"

                 mcast_send_buf_size="640000"

                 loopback="false"

                 discard_incompatible_packets="true"

                 max_bundle_size="64000"

                 max_bundle_timeout="30"

                 use_incoming_packet_handler="true"

                 ip_ttl="2"

                 enable_bundling="false"

                 enable_diagnostics="true"

                 use_concurrent_stack="true"

                 thread_naming_pattern="pl"

                 thread_pool.enabled="true"

                 thread_pool.min_threads="1"

                 thread_pool.max_threads="25"

                 thread_pool.keep_alive_time="30000"

                 thread_pool.queue_enabled="true"

                 thread_pool.queue_max_size="10"

                 thread_pool.rejection_policy="Run"

                 oob_thread_pool.enabled="true"

                 oob_thread_pool.min_threads="1"

                 oob_thread_pool.max_threads="4"

                 oob_thread_pool.keep_alive_time="10000"

                 oob_thread_pool.queue_enabled="true"

                 oob_thread_pool.queue_max_size="10"

                 oob_thread_pool.rejection_policy="Run"/>

 

            <PING timeout="2000" num_initial_members="3"/>

            <MERGE2 max_interval="30000" min_interval="10000"/>

            <FD_SOCK/>

            <FD timeout="10000" max_tries="5" shun="true"/>

            <VERIFY_SUSPECT timeout="1500"/>

            <pbcast.NAKACK max_xmit_size="60000"

                           use_mcast_xmit="false" gc_lag="0"

                           retransmit_timeout="300,600,1200,2400,4800"

                           discard_delivered_msgs="true"/>

            <UNICAST timeout="300,600,1200,2400,3600"/>

            <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                           max_bytes="400000"/>

            <pbcast.GMS print_local_addr="true" join_timeout="5000"

                        join_retry_timeout="2000" shun="false"

                        view_bundling="true" view_ack_collection_timeout="5000"/>

            <FRAG2 frag_size="60000"/>

            <pbcast.STREAMING_STATE_TRANSFER use_reading_thread="true"/>

            <!-- <pbcast.STATE_TRANSFER/> -->

            <pbcast.FLUSH timeout="0"/>

         </config>

      </attribute>

 

      <attribute name="FetchInMemoryState">true</attribute>

 

      <attribute name="StateRetrievalTimeout">30000</attribute>

 

      <attribute name="SyncReplTimeout">20000</attribute>

 

      <!-- Max number of milliseconds to wait for a lock acquisition -->

      <attribute name="LockAcquisitionTimeout">50000</attribute>

 

      <attribute name="UseRegionBasedMarshalling">true</attribute>

   </mbean>

 

   <!--  Uncomment to get a graphical view of the TreeCache MBean above -->

   <!--   <mbean code="org.jboss.cache.TreeCacheView" name="jboss.cache:service=TreeCacheView">-->

   <!--      <depends>jboss.cache:service=TreeCache</depends>-->

   <!--      <attribute name="CacheService">jboss.cache:service=TreeCache</attribute>-->

   <!--   </mbean>-->

 

</server>

 

 

Any idea what could be causing this? Any help would be greatly appreciated!

 

Thanks,

Tariq

  • 1. Re: OutOfMemoryError while loading pojo cache
    Tariq Lahyani Newbie

    Also, here is a sample method that loads one of the database tables:

     

        /**

         * Loads point data from the database.

         */

        protected void loadIMMPointFromDB() {

            logger.info("Loading IMMPoint cache data ...");

                  Connection conn = null;

                  PreparedStatement pstmt = null;

            ResultSet result = null;

            int count = 0;

            try {

                conn = getConnection();

                pstmt = DataAccessor.getPreparedStatement(conn, getIMMPointSql);

                // get all IMMPoints from table

                result = pstmt.executeQuery();

                ResultSetReader resultSetReader = ResultSetReader.getInstance(IMMPoint.class, result);

                // for each record load into cache

                while (result.next()) {

                          IMMPoint point = (IMMPoint)resultSetReader.readObject(result);

                          String id = point.getCurveId().trim();

                          String tenor = point.getTenor().trim();

                          pojoCache.attach(IMMPOINTS_FQN + IMMPOINTS_FQN_PREFIX + id + TENOR_FQN_PREFIX + tenor, point);

                          // output some debugging info

                          if (count > 0 && count % 50000 == 0) {

                            logger.info(count + " IMM points loaded");

                          }

                          count++;

                }

            }

            catch (SQLException e) {

                logger.fatal("SQL Error loading IMMPoint data - " + e.getMessage(), e);

            }

            catch (Exception e) {

                logger.fatal("Error loading IMMPoint data - " + e.getMessage(), e);

            }

            finally {

                DataAccessor.close(result, pstmt, conn);

            }

            logger.info(count + " IMMPoint entries loaded from database");

        }