4 Replies Latest reply on Jan 28, 2013 7:15 AM by rhusar

Cluster Environment Session Locks

mrspinto Jan 24, 2013 6:16 PM

Hi,

I need some help to resolve a problem with session locks in a cluster environment.

We have two Liferay 6.1 over JBoss 7.1.1 in standalone-ha configuration (build 7.1.1 Final) with infinispan "web" cache-container, invalidation-cache and jdbc store. The load balancer is configured in non sticky session mode.

Problem: when a node processes requests that manipulate session's data, in some cases the node locks the session and never unlock it, preventing other node from processing requests for that session. The affected node never regain the locked session and keep throwing the following exception for all subsequent requests and only recover a session when other node shutdown:

Note: we also tried replication-cache and distributed-cache and all locking modes but without success.

Exception:

16:29:52,278 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--127.0.0.1-8080-2) An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of ruxDFIayi1vrsCejzwnVQJl3

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:534) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:81) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_31]

Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //default-host//ruxDFIayi1vrsCejzwnVQJl3 from cluster

at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)

at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:560) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:532) [jboss-as-web-7.1.1.Final-custom.jar:7.1.1.Final]

... 11 more

Infinispan modules:

Infinispan Core 5.1.6 Final and Infinispan Cachestore JDBC 5.1.6.FINAL

The standalone-ha.xml "web" cache-container config:

<cache-container name="web" default-cache="database">

<invalidation-cache name="database" mode="SYNC" batching="true">

<mixed-keyed-jdbc-store datasource="java:jboss/datasources/LPORTAL_DS" shared="true" preload="true" passivation="false" purge="true">

<property name="databaseType">oracle</property>

<string-keyed-table prefix="stringbased">

<id-column name="id" type="VARCHAR2(200)"/>

<data-column name="datum" type="BLOB"/>

<timestamp-column name="version" type="NUMBER"/>

</string-keyed-table>

<binary-keyed-table prefix="binarybased">

<id-column name="id" type="VARCHAR2(200)"/>

<data-column name="datum" type="BLOB"/>

<timestamp-column name="version" type="NUMBER"/>

</binary-keyed-table>

</mixed-keyed-jdbc-store>

</invalidation-cache>

</cache-container>

Thanks,

Manuel

1. Re: Cluster Environment Session Locks

mrspinto Jan 24, 2013 6:32 PM (in response to mrspinto)

The correct log is:

17:39:00,174 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--172.16.250.105-8080-4) An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of Cvn-K+r-cBGesIBoDrakJhrO
        at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:528) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:496) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:81) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //default-host//Cvn-K+r-cBGesIBoDrakJhrO from cluster
        at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)
        at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)
        at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
        ... 12 more
Actions
2. Re: Cluster Environment Session Locks

rhusar Jan 25, 2013 9:32 AM (in response to mrspinto)

Is there any reason why not to turn on sticky sessions? It is very recommended to use sticky sessions.

Also, try with the latest version, like 7.1.3.Final.

Maybe AS7-4260 could help a bit too.
Actions
3. Re: Cluster Environment Session Locks

mrspinto Jan 25, 2013 11:42 AM (in response to rhusar)

Hi Radoslav,

Is there any reason why not to turn on sticky sessions? It is very recommended to use sticky sessions.

Customer requirement...

Also, try with the latest version, like 7.1.3.Final.

Where i can download this version?

Maybe AS7-4260 could help a bit too.

We have applied that fix but without success.

Thanks,
Manuel
Actions
4. Re: Cluster Environment Session Locks

rhusar Jan 28, 2013 7:15 AM (in response to mrspinto)

Where i can download this version?
You will have to build yourself. Checkout this branch from Git https://github.com/jbossas/jboss-as/tree/7.1.3.Final and then see building instructions.

Customer requirement...
I recommend talking to the customer and explaining that the overhead/performance cost of such decision is significant.
Actions

Go to original post