-
1. Re: TEIID Services Not available if JBoss AS Running for Couple of Hours.
rareddy Aug 26, 2015 9:39 AM (in response to akshy_harale)Akshay,
1) This means that server could not get the connection to the underlying source in a given amount time. Have you checked the source it self, is that is active? Typically idle connections will be flushed after defined time, then new connections are made when requests come in.
2) What operation are you doing when you saw this exception?
3) Not sure about this one.
4) This occurs when client makes a connection, but after that server may have been re-started but the previously made connection is still being used.
We have running servers for days, months together without any issues, looks like there may be something else that may be re-starting your server in between. Have you done any kind of automation with CLI scripts etc to manage the server perhaps? What I would suggest is to leave the server running in DEBUG mode and check the log when you see the symptoms as to what happened to the server prior to that.
Ramesh..
-
2. Re: TEIID Services Not available if JBoss AS Running for Couple of Hours.
guydavis Sep 2, 2015 4:50 PM (in response to rareddy)Hi Ramesh, I'm following up for Akshay. We profiled the problematic workflow in Yourkit and found high CPU usage in BufferManagerImpl.doEvictions(). In particular, when the system become unresponsive, this method was consuming 94% of CPU time in this JBoss instance. Please see below screenshot:
The VDB is a complicated multi-level one connecting to multiple MS SQL Server databases. Query and procedure results seem to require a substantial amount of buffer space. Is such CPU usage to be expected in this case? If this is not expected, what would you suggest we investigate next? What tuning could be performed to improve BufferManager behavior?
Thanks in advance,
Guy
-
yourkit_teiid.png 81.9 KB
-
-
3. Re: TEIID Services Not available if JBoss AS Running for Couple of Hours.
shawkins Sep 2, 2015 6:13 PM (in response to guydavis)More than likely you are seeing [TEIID-3106] BufferManager Cleaner consuming 90% of total CPU time - JBoss Issue Tracker which was addressed for 8.7.1 and 8.9+.
-
4. Re: TEIID Services Not available if JBoss AS Running for Couple of Hours.
akshy_harale Sep 16, 2015 8:59 AM (in response to shawkins)Hello Steven/Ramesh,
We applied the patch on 8.7 for the slow performance issue and we get it resolved. Now we are facing the invalid session issue as mentioned in my question above.
We have a multi-layered VDB architecture where we have Teiid datasource which is connecting to the underlying VDB which is connecting to other datasource like sqlserver.
When I do the connection I get the data but at same time in server.log file I get session invalid exception. I looked into the log file but unable to locate the origin of the session creation.
2015-09-16 06:35:11,255 WARN [org.teiid.SECURITY] (New I/O worker #7) TEIID40011 Processing exception 'TEIID40042 Invalid Session L1JLlDb3PZU9. Session may have already been terminated.' for session null. Exception type org.teiid.client.security.InvalidSessionException thrown from org.teiid.services.SessionServiceImpl.getSessionInfo(SessionServiceImpl.java:368).: org.teiid.client.security.InvalidSessionException: TEIID40042 Invalid Session L1JLlDb3PZU9. Session may have already been terminated.
-
5. Re: TEIID Services Not available if JBoss AS Running for Couple of Hours.
shawkins Sep 16, 2015 2:05 PM (in response to akshy_harale)> When I do the connection I get the data but at same time in server.log file I get session invalid exception. I looked into the log file but unable to locate the origin of the session creation.
Are you seeing unexpected client exceptions? In some/most circumstances that message is benign. There were some related cleanups and modifications to the logging of it in later releases, most recently with https://issues.jboss.org/browse/TEIID-3697