1 2 Previous Next 15 Replies Latest reply on Apr 16, 2015 4:45 PM by gopinathang

    Unexpected shutdown of service

    patrickawilson

      Things move along normally, sometimes for an hour or more ...

      ^[[0m^[[0m11:50:05,624 INFO  [stdout] (http-/0.0.0.0:8088-1) Sent message to OpsCenter

      ^[[0m^[[0m11:55:03,924 INFO  [stdout] (http-/0.0.0.0:8088-1) Sent message to OpsCenter

      ^[[0m^[[0m12:00:07,591 INFO  [stdout] (http-/0.0.0.0:8088-1) Sent message to OpsCenter

      ^[[0m^[[0m12:05:03,937 INFO  [stdout] (http-/0.0.0.0:8088-1) Sent message to OpsCenter

      ^[[0m^[[0m12:10:09,138 INFO  [stdout] (http-/0.0.0.0:8088-1) Sent message to OpsCenter

       

      Then the service is shut down, which is not desirable.

      ^[[0m^[[0m12:11:18,333 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) JBAS010410: Unbound JCA ConnectionFactory [java:jboss/ConsertOpsCenterConnectionFactory]

      ^[[0m^[[0m12:11:18,327 INFO  [org.apache.catalina.core] (MSC service thread 1-1) JBWEB001079: Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started

      ^[[0m^[[0m12:11:18,340 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003075: Coyote HTTP/1.1 pausing on: http-/0.0.0.0:8088

      ^[[0m^[[0m12:11:18,346 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003077: Coyote HTTP/1.1 stopping on : http-/0.0.0.0:8088

      ^[[0m^[[0m12:11:18,386 INFO  [org.jboss.as.webservices] (MSC service thread 1-4) JBAS015540: Stopping service jboss.ws.endpoint."GenericGSCIMService.deployment".GenericGSCIMService

      ^[[0m^[[0m12:11:18,391 INFO  [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022051: Endpoint unregistered: jboss.ws:context=GenericGSCIMService,endpoint=GenericGSCIMService

      ^[[0m^[[0m12:11:18,395 INFO  [org.jboss.as.webservices] (MSC service thread 1-4) JBAS015540: Stopping service jboss.ws.endpoint."OpsCenterUndeliveredMessagesService.deployment".OpsCenterUndeliveredMessagesService

      ^[[0m^[[0m12:11:18,396 INFO  [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022051: Endpoint unregistered: jboss.ws:context=OpsCenterUndeliveredMessagesService,endpoint=OpsCenterUndeliveredMessagesService

      etc. etc. etc. as shutdown proceeds.

       

      It looks like the connection factory might have lost the connection, but why would that cause the entire service to collapse?  What I want is for the service to remain up and running and for Switchyard to keep trying.  I have code in place for failure handling, but that all gets shut down if the whole service is stopped.  Is there something that I can configure to prevent this?

       

      Thanks

        • 1. Re: Unexpected shutdown of service
          patrickawilson

          I wanted to bring this to the top again.  The service runs properly for several hours and then suddenly collapses.

           

          This is an absolutely critical issue.  This product has to be 24/7 reliable with a message throughput of thousands per minute.  At the moment I can't get it to stay up for more than a few hours with just a dozen messages per hour. 

           

          The root error:

          (MSC service thread 1-1) JBAS010410: Unbound JCA ConnectionFactory [java:jboss/ConsertOpsCenterConnectionFactory]

           

          Why would this service work for hours and then die?  It seems to me that if the connection factory is unbound then it should not work at all.

           

          Thanks again.

          • 2. Re: Unexpected shutdown of service
            kcbabo

            This was posted in the wrong forum.  Please see this note:

            Before You Post to SwitchYard Development

             

            I will move it to the user forum and answer there.

            • 3. Re: Unexpected shutdown of service
              kcbabo

              There is no code in SwitchYard which shuts down the server in the event of failure and I would be pretty surprised if any such code existed in EAP either.  The log output looks like a clean shutdown initiated through the management API or by terminating the Java process.  Another possibility is that you have code which invokes System.exit() either directly in your SY app or indirectly (e.g. through a JCA adapter deployed to the server, another application deployed, etc.).

               

              I have not heard any reports of the server randomly shutting down due to load or time up, so I would take an inventory of what's deployed on your instance and see if something might be introducing this behavior.

              • 4. Re: Unexpected shutdown of service
                patrickawilson

                Thanks for the response and apologies for posting to the wrong forum.  That was unintentional.

                 

                The JCA adapter is wmq.jmsra.rar (IBM MQ).


                Definitely no call to System.exit anywhere in our code base.  This software is supposed to run 24/7.

                The server is not being explicitly shut down via admin console.  It is running in a pretty isolated lab.

                There have been four occurrences in four attempted runs.  Always the same issue.  The longest continuous execution run was about 4 hours.  The shortest just over 1. 

                There are no error messages in in the server log.  Everything appears to be going normally and then the sequence in the OP, starting with the info log on the JCA shutdown.

                 

                I will continue to test on my end.  I have a run going now that has lasted 4.5 hours (new record).  If we continue to see the issue then the next step is to raise the log level to debug.  That might offer an explanation as to why the shutdown is happening.

                 

                I am aware that this might have nothing at all to do with Switchyard.  However, since Switchyard is the highest level component it gets the nod as to where to post.  Our system uses Switchyard over EAP with Camel, HornetQ and the IBM MQ adapter.  The amount of software is fairly trivial at this point as we are just getting off the ground.

                • 5. Re: Re: Unexpected shutdown of service
                  dward

                  Perhaps in your code, at some point during initialization, you could add something like:

                   

                  Runtime.getRuntime().addShutdownHook(new Thread(new Runnable(){

                    public void run() {

                        new Exception("**** Who called System.exit()? ****").printStackTrace();

                    }

                  }));

                   

                  I'm thinking that might give you a hint as to who called System.exit()?

                  • 6. Re: Unexpected shutdown of service
                    patrickawilson

                    Thanks for the tip.  I will give that a shot over the next couple of days.

                     

                    Latest is that over the weekend it failed after 8 hours (longest run to date).  I now have it running with lots of debug turned on.  Hopefully this does not cause it to work (random nature of the failure has the smell of a timing problem).  If the debug logs fail to isolate the issue I will insert that code segment to see what it uncovers.

                     

                    Just to confirm previous statements: nobody is explicitly shutting the JBoss server down and at no point does any of my software explicitly exit.  It is designed to be a continuous, 24/7 application.

                     

                    The code is relatively simple JMS.  JMS objects are contained in a POJO object that I wrote.  The POJO is instantiated within the bean.  Connection, session, and message producer objects are created once and then retained in the POJO which in turn is a private member of the bean instance.  An external data packet comes in through a web service and gets routed to the bean.   The bean then a transforms the data and uses the message producer to send to an external destination.  This can go on from 1-8 hours and then it fails per the details in the OP.

                     

                    In the long run this system has to handle up to 30000 messages a minute, which should not be an issue for a properly performing JMS system.  At the moment it receives only 1 message every five minutes, and it is still failing.

                    • 7. Re: Unexpected shutdown of service
                      kcbabo

                      I have never seen this in my environment, in our stress test environment, or through user reports.  We are happy to help issues that relate to SY, but as of now there are no indications whatsoever that this is related to something in the runtime.  Even in your original post there's no indication that there's an error.  This line:

                       

                      ^[[0m^[[0m12:11:18,333 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) JBAS010410: Unbound JCA ConnectionFactory [java:jboss/ConsertOpsCenterConnectionFactory]


                      Is not an error.  It's an indication that the connection factory is unbound from JNDI, which happens every time during a normal shutdown of the server. 

                      • 8. Re: Unexpected shutdown of service
                        dward

                        Also:

                        1. Did you customize the jboss startup script, where the java process might get terminated early?
                        2. Is there the chance that there is some other process running on the OS that is reaping processes (by sending it a kill signal) that it incorrectly thinks doesn't need to be running? This could also include something that kills entire user sessions, and possibly, the processes that were spawned from it.

                        Just some thoughts...

                        • 9. Re: Unexpected shutdown of service
                          patrickawilson

                          I understand and that is a big part of my problem - there is no error.  At least none indicated in the logs.

                           

                          Per a previous post, I completely agree that it might not be a Switchyard issue.  The error could be in any component: Switchyard, Camel, JBoss, or the WebSphere MQ Adapter.  Or it could be in my code (I have been doing this long enough to never discount the possibility that I may be doing something wrong ).  I am posting here because Switchyard is the highest level component in my stack.  I absolutely do not intend to impugn Switchyard or other component.  Just looking for diagnostic tips.

                           

                          Another interesting note: we are only receiving one message every five minutes, at regular intervals, which means that the code should really only be active at that time.  However, the shutdowns do not correspond with message processing points.  That indicates a cause other than the message handling chain since that code would be idle when the shutdown occurs. 

                           

                          I have edited standalone-full.xml to add configuration, but that is the only edited config file.  I have diffed that file to check for inadvertent changes and there are none.  There is no reaper process running on our labs.

                           

                          Running with debug enabled has been going for five hours now with no failure.  Let's see what kind of information that produces.  With a bit of luck it will help to isolate the issue.  If not we continue on.

                           

                          I do appreciate the responses. 

                          • 10. Re: Unexpected shutdown of service
                            patrickawilson

                            So I determined the cause of the issue ... me.  Embarrassing, but really the best possible outcome as my own mistakes are the easiest to fix.  I was running the standalone script to start the the server directly from the command line as a background process.  I thought running it as a background process would keep it active when the shell shut down.  I was wrong.  The standalone script remains active with the shell as its parent.  The server process has the standalone script as its parent.  When I terminate the shell a HUP is sent to the child processes - thus the normal shutdown. 

                             

                            Solution is to execute the standalone script from another startup script and then start the startup script as a background process.  The startup script actually terminates, which divorces the child processes from the shell.  Then the server survives the shell termination.

                             

                            Did I mention that I was better at software development than sys admin?

                             

                            Anyhow, thanks for the responses.  Hopefully this will help somebody else.

                            • 11. Re: Unexpected shutdown of service
                              dward

                              W00t! I feel pretty good about my guesses then. Glad you got things working.

                              • 12. Re: Unexpected shutdown of service
                                santosh2027

                                Patrick,

                                 

                                Can you please provide what we need to update in standalone script?

                                • 13. Re: Unexpected shutdown of service
                                  synclpz

                                  just start it with nohup :)

                                   

                                  nohup ./standalone.sh  >/dev/null &

                                  • 14. Re: Unexpected shutdown of service
                                    heenak

                                    Hi All,

                                     

                                    I am also facing same problem. My Jboss is getting closed without displaying any error. I am running my Jboss as a windows service.It gets shuts down randomly; 1st two days it shut down 1 time and yesterday it shut down 3 time. Cannot figure out where exactly the problem is? Is it due to overloading of transactions or some network related issues?

                                    1 2 Previous Next