-
1. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
ataylor Mar 25, 2010 11:03 AM (in response to mzeijen)1 of 1 people found this helpfulcan't say i have ever seen this happen, is the HornetQ instance standalone or embedded in the AS?
-
2. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
mzeijen Mar 25, 2010 11:13 AM (in response to ataylor)The HornetQ is embedded in a JBoss AS 5.1 instance.
The problem occurres in the org.hornetq.core.client.impl.FailoverManagerImpl class.
The Deadlock occurres because of the failoverLock object
The HornetQ Thread is in the 'createSession' method in the synchronized block of the failoverLock. This thread is probably from the RA because when the RA is nog being used then the problem doesn't occur.
The JBoss Shutdown Thread is in the 'removeSession' method and tries to entry the synchronized block of the failoverLock.
Tomorrow I can provide Stacktraces of both threads if you want.
-
3. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
clebert.suconic Mar 25, 2010 3:42 PM (in response to mzeijen)yes... First thing would be the analyze the stack trace from a kill -3.
It would be great if you could provide one.
-
4. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
mzeijen Mar 26, 2010 3:53 AM (in response to clebert.suconic)I did a shutdown of the JBoss Server and extracted the stack traces from both threads:
Daemon Thread [JBoss Shutdown Hook] FailoverManagerImpl.removeSession(ClientSessionInternal) line: 495 ClientSessionImpl.doCleanup() line: 1614 ClientSessionImpl.close() line: 811 DelegatingSession.close() line: 144 HornetQMessageHandler.teardown() line: 195 HornetQActivation.teardown() line: 282 HornetQActivation.stop() line: 245 HornetQResourceAdapter.endpointDeactivation(MessageEndpointFactory, ActivationSpec) line: 155 RARDeployment.endpointDeactivation(MessageEndpointFactory, ActivationSpec) line: 346 RARDeployment.internalInvoke(String, Object[], String[]) line: 293 RARDeployment(ServiceDynamicMBeanSupport).invoke(String, Object[], String[]) line: 156 RawDynamicInvoker.invoke(String, Object[], String[]) line: 164 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 668 JBoss42ActivationBridge.deactivate() line: 208 JcaMessageAwareListener(BaseJcaInflow<T>).doStop() line: 242 JcaMessageAwareListener(AbstractManagedLifecycle).stop() line: 216 ManagedLifecycleController.stopInstances() line: 164 ManagedLifecycleController.stopAndDestroy(boolean) line: 255 ManagedLifecycleController.stop() line: 92 EsbDeployment.stop() line: 132 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 597 ReflectionUtils.invoke(Method, Object, Object[]) line: 59 ReflectMethodInfoImpl.invoke(Object, Object[]) line: 150 BasicMethodJoinPoint.dispatch() line: 66 KernelControllerContextAction$JoinpointDispatchWrapper.execute() line: 241 KernelControllerContextAction$JoinpointDispatchWrapper(ExecutionWrapper).execute(AccessControlContext) line: 47 KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContext, ExecutionWrapper) line: 109 KernelControllerContextAction.dispatchJoinPoint(KernelControllerContext, Joinpoint) line: 70 StartStopLifecycleAction(LifecycleAction).uninstallActionInternal(KernelControllerContext) line: 249 StartStopLifecycleAction(InstallsAwareAction).uninstallAction(KernelControllerContext) line: 157 StartStopLifecycleAction(InstallsAwareAction).uninstallAction(ControllerContext) line: 42 StartStopLifecycleAction(SimpleControllerContextAction<T>).simpleUninstallAction(T) line: 79 StartStopLifecycleAction(AccessControllerContextAction<S,T>).uninstall(ControllerContext) line: 131 KernelControllerContextActions(AbstractControllerContextActions).uninstall(ControllerContext, ControllerState, ControllerState) line: 58 AbstractKernelControllerContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354 AbstractKernelController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234 AbstractKernelController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827 AbstractKernelController(AbstractController).change(ControllerContext, ControllerState) line: 553 ServiceController.doChange(KernelController, ServiceControllerContext, ControllerState, String) line: 688 ServiceController.stop(ObjectName) line: 510 ServiceDeployer.stop(ObjectName) line: 170 ServiceDeployer.undeploy(DeploymentUnit, ServiceMetaData) line: 150 ServiceDeployer.undeploy(DeploymentUnit, Object) line: 46 ServiceDeployer(AbstractSimpleRealDeployer<T>).internalUndeploy(DeploymentUnit) line: 69 ServiceDeployer(AbstractRealDeployer).undeploy(DeploymentUnit) line: 112 DeployerWrapper.undeploy(DeploymentUnit) line: 196 DeployersImpl.doUndeploy(Deployer, DeploymentUnit) line: 1469 DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1376 DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1369 DeployersImpl.doUninstallParentLast(Deployer, DeploymentContext, boolean, boolean) line: 1356 DeployersImpl.uninstall(ControllerContext, ControllerState, ControllerState) line: 1331 DeploymentControllerContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354 AbstractKernelController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664 AbstractKernelController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275 AbstractKernelController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827 AbstractKernelController(AbstractController).change(ControllerContext, ControllerState) line: 553 DeployersImpl.process(List<DeploymentContext>, List<DeploymentContext>) line: 694 MainDeployerImpl.process() line: 679 MainDeployerAdapter.process() line: 117 ProfileDeployAction.uninstall(Profile) line: 95 ProfileDeployAction(AbstractProfileAction).uninstall(ProfileContext) line: 70 AbstractProfileService.uninstall(ControllerContext, ControllerState, ControllerState) line: 375 ProfileContext(AbstractControllerContext).uninstall(ControllerState, ControllerState) line: 354 ScopedProfileServiceController(AbstractController).uninstall(ControllerContext, ControllerState, ControllerState) line: 1664 ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1275 ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, ControllerState, boolean) line: 1179 ScopedProfileServiceController(AbstractController).uninstallContext(ControllerContext, boolean) line: 1234 ScopedProfileServiceController(AbstractController).change(ControllerContext, ControllerState, boolean) line: 827 ScopedProfileServiceController(AbstractController).change(ControllerContext, ControllerState) line: 553 AbstractProfileService.deactivateProfile(ProfileKey) line: 402 ProfileServiceBootstrap.deactivateProfiles(Collection<ProfileKey>) line: 388 ProfileServiceBootstrap.shutdown(Server) line: 345 ServerImpl(AbstractServerImpl).shutdownServer() line: 571 AbstractServerImpl$ShutdownHook.run() line: 909
Daemon Thread [Thread-0 (group:HornetQ-client-global-threads-32315681)] Thread.sleep(long) line: not available [native method] FailoverManagerImpl.getConnectionWithRetry(int) line: 821 FailoverManagerImpl.reconnectSessions(RemotingConnection, int) line: 758 FailoverManagerImpl.failoverOrReconnect(Object, HornetQException) line: 682 FailoverManagerImpl.handleConnectionFailure(Object, HornetQException) line: 548 FailoverManagerImpl.access$600(FailoverManagerImpl, Object, HornetQException) line: 69 FailoverManagerImpl$DelegatingFailureListener.connectionFailed(HornetQException) line: 1111 RemotingConnectionImpl.callFailureListeners(HornetQException) line: 445 RemotingConnectionImpl.fail(HornetQException) line: 250 FailoverManagerImpl$Channel0Handler$1.run() line: 1079 OrderedExecutorFactory$OrderedExecutor$1.run() line: 96 ThreadPoolExecutor$Worker.runTask(Runnable) line: 886 ThreadPoolExecutor$Worker.run() line: 908 Thread.run() line: 619
If you want I can create a Jira Issue for this problem.
-
5. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
ataylor Mar 26, 2010 4:30 AM (in response to mzeijen)Yes, if you could raise a jira and refernce this thread that would be cool.
-
6. Re: JBoss won't shutdown when HornetQ Server and HornetQ RA are used on that same server
mzeijen Mar 26, 2010 5:43 AM (in response to ataylor)I created the following Jira Issue: HORNETQ-339
I also created a workaround. You can find it in the attached zip file.
It is a small Service that you deploy on the JBoss AS service. It makes it possible to create dependencies on the deploy cycle of the HornetQ service. To make it work you need to create a dependency to this service. When JBoss Server is being shut down then the dependency on this service makes sure that your service is stopped first. This is only needed when you require a JMX dependency. If your application is build for the JBoss MC then you can create a direct depedency on the HornetQ service.
The JMX dependency that you need to make is to: com.smies.hornetq:service=HornetQMBeanHook
For example in the JBoss ESB deployment you need to add the following line to the deployment.xml: <depends>com.smies.hornetq:service=HornetQMBeanHook</depends>. However in case of JBoss ESB then it is best when you make sure that all your packages have the <depends>jboss.esb:deployment=jbossesb.esb</depends> dependency and the jbossesb.esb package has the dependency on the HornetQMBeanHook
This HornetQMBeanHook also does one extra thing namely it makes sure that the hornetq-ra.rar or the jms-ra.rar deployment is stopped before the HornetQServer is stopped.
-
hornetq-mbean-hook.zip 9.2 KB
-