1 Reply Latest reply on Jan 31, 2014 3:49 PM by tcunning

    JBoss dropping responses from web service

    jloyd

      We are having an interesting issue with our JBoss EAP 5.0 installation (5.0.0 GA).

       

      We have a web service used by our public web site to gather rating information from internal systems and return it for display on the web site. The service is set up as a simple ESB service that has the following in the jboss-esb.xml file:

       

      <?xml version="1.0"?>

      <jbossesb parameterReloadSecs="5"

          xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.2.0.xsd"

          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.2.0.xsd http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.2.0.xsd">

          <globals>

              <war-security domain="java:/jaas/JBossWS" method="BASIC" />

          </globals>

          <providers>

              <http-provider name="RatingService-Http-Provider">

                  <http-bus busid="RatingService_Bus_Id">

                      <protected-methods>

                          <method name="GET" />

                          <method name="POST" />

                      </protected-methods>

                      <allowed-roles>

                          <role name="friend" />

                      </allowed-roles>

                  </http-bus>

              </http-provider>

          </providers>

          <services>

              <service category="com.mem.integration.esb.ratingservice"

                  description="Access Rating Engine" invmScope="GLOBAL" name="RatingServicESB">

                  <listeners>

                      <http-gateway busidref="RatingService_Bus_Id" name="RatingService"

                          urlPattern="getRating/*" >

                          <property name="synchronousTimeout" value="1500000"/>

                      </http-gateway>

                  </listeners>

                  <actions mep="RequestResponse">

                      <action class="org.jboss.soa.esb.actions.SystemPrintln" name="Smooks-Before">

                          <property name="message" value="Smooks-Before--->" />

                      </action>

                      <action class="org.jboss.soa.esb.smooks.SmooksAction" name="RatingServiceESB">

                          <property name="smooksConfig"

                              value="/com/mem/integration/esb/ratingservice/transformation/Ratingservicesmooks-config.xml" />

                          <property name="resultType" value="JAVA" />

                      </action>

                      <action

                          class="com.mem.integration.esb.ratingservice.action.RatingServiceAction"

                          name="RatingServiceProcessor" />

                      <action class="org.jboss.soa.esb.actions.SystemPrintln" name="Smooks-11111111--">

                          <property name="message" value="After--1111111------->" />

                      </action>

                      <action class="org.jboss.soa.esb.smooks.SmooksAction" name="PCtoRatingServiceAction">

                          <property name="smooksConfig"

                              value="/com/mem/integration/esb/ratingservice/transformation/PolicyPeriod-config.xml" />

                          <property name="resultType" value="JAVA" />

                      </action>

                      <action class="org.jboss.soa.esb.actions.SystemPrintln" name="Smooks---">

                          <property name="message" value="After--------->" />

                      </action>

                      <action

                          class="com.mem.integration.esb.ratingservice.action.XmlTransAction"

                          name="RatingServiceProcessor1" />

                  </actions>

              </service>

          </services>

      </jbossesb>

       

      The issue we have is that with no changes to the server the service stops responding to clients. Logging shows me that the service is getting all the way through the com.mem.integration.esb.ratingservice.action.XmlTransAction (AbstractActionPipelineProcessor) class and that class reaches the point where it returns the Message object fro the process method. The calling app does not recive the message. Restarting JBoss corrects this issue.

       

      It seems to me that there is something going when my service hands the message back to JBoss to where JBoss decides not to return the message. The service this is on has several other similar services running and they do not have this issue.

       

      The funny thing is that we have not made any changes to this service recently. We added a new service to the mix and I think that it where the trouble lies, I just am not sure what to look for. The new service is pretty vanella, but is not a JBoss specifc web service. The web.xml for that service looks like:

       

      <?xml version="1.0" encoding="UTF-8"?>

      <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">

        <listener>

          <listener-class>

            com.sun.xml.ws.transport.http.servlet.WSServletContextListener

          </listener-class>

        </listener>

        <servlet>        

            <servlet-name>cognosSdkService</servlet-name>        

            <servlet-class>com.sun.xml.ws.transport.http.servlet.WSServlet</servlet-class>   

            <load-on-startup>1</load-on-startup>    

        </servlet>    

        <servlet-mapping>        

            <servlet-name>cognosSdkService</servlet-name>        

            <url-pattern>/cognosSdkService</url-pattern>    

        </servlet-mapping>    

        <session-config>        

            <session-timeout>30</session-timeout>    

        </session-config>

      </web-app>

       

      So far here are the lists of things we have tried to figure it out:

       

      1. Check for library conflicts between the new service and JBoss. No issues there. A new library was put in the server/default/lib directory. It was saaj-impl-1.3.2. I did not see a conflicting version on the server.

      2. Dump the memory from JBoss. Nothing of note in that. The server has 4 GB of memory and is only using 2 GB.

      3. Trace every thing from everywhere. Nothing to show. Basically, the following netowrk map is aht we see:

       

      Web -> firewall -> database (oracle, makes soap request to service) - > proxy server - > firewall - > JBoss rating service - > policy/builling system -> JBoss rating service -> ???????????

       

      Network traces show that the call is coming into JBoss through the proxy server but it is not returning to the proxy server.

       

      Unanswered questions:

      1. Why does the service work for a while (days) and then start having issues?

      2. Why does a restart of JBoss resolve the issues?

      3. What is causing the service to stop responding?

       

      Please help! Please ask any questions you need.

       

      Thanks

       

      John

        • 1. Re: JBoss dropping responses from web service
          tcunning

          John,

           

          What version of JBossESB are you using?    If you aren't on 4.12, the first thing I'm going to suggest is trying 4.12 + AS to see if you still see the issue there.

           

          Are you using ESB on top of EAP?   Or are you using SOA-P (SOA Platform)?     5.0.0 is a pretty old version of EAP too, I might try upgrading that as well.