Hello Rado,

actually I did update both parts (httpd and java). I have 2 httpds (both updated) and, underneath, 3 servers running several JBoss java instancies.

I'm checking now that everything is correctly upgraded but I'm pretty sure everything is correct.

Also, I have noticed that some threads down from this one, there is a mention to the "index out of range: -1".

http://community.jboss.org/thread/172549?tstart=0

Would this be the same problem?

Thank you.

Joao

Hello.

I checked all my configuration and all seems to be correctçy migrated.

Affter the expection that I posted there is another (different) error. For example:

2011-10-31 17:27:51,534 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (Incoming-8,192.168.150.42:55900) Error parsing response header for command CONFIG

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

    at java.lang.String.substring(String.java:1937)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequests(DefaultMCMPHandler.java:476)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:417)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:377)

    at org.jboss.modcluster.ha.ClusteredMCMPHandlerImpl.updateServersFromMasterNode(ClusteredMCMPHandlerImpl.java:146)

    at org.jboss.modcluster.ha.HAModClusterService$RpcHandler.getClusterCoordinatorState(HAModClusterService.java:907)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

    at java.lang.reflect.Method.invoke(Method.java:597)

    at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:331)

    at org.jboss.ha.framework.server.ClusterPartition$RpcHandler.handle(ClusterPartition.java:2314)

    at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)

    at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)

    at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)

    at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)

    at org.jgroups.JChannel.up(JChannel.java:1336)

    at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)

    at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:490)

    at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)

    at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)

    at org.jgroups.protocols.FC.up(FC.java:473)

    at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)

    at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)

    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)

    at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)

    at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)

    at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:709)

    at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)

    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)

    at org.jgroups.protocols.FD.up(FD.java:284)

    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:307)

    at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)

    at org.jgroups.protocols.Discovery.up(Discovery.java:264)

    at org.jgroups.protocols.PING.up(PING.java:273)

    at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)

    at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)

    at org.jgroups.protocols.TP.access$100(TP.java:50)

    at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1831)

    at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)

    at java.lang.Thread.run(Thread.java:619)

2011-10-31 17:27:51,534 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (Incoming-8,192.168.150.42:55900) Error [null: null: {4}] sending command CONFIG to proxy 192.168.150.8/192.168.150.8:6666, configuration will be reset

Error null null does not sound right. What should such a log message show instead of null null?

Thank you.

Joao

Hi, sorry for the delay on the reply but had to revert the whole system to 1.0.10GA.

My config under 1.1.3:

Frontend, 2 httpds, running on Windows 2008 Server R2

httpd config is:

# Using the Apache 2.2.21 original mod_proxy.so and not the one shipped with mod_cluster 1.1.3

LoadModule proxy_module modules/mod_proxy.so

# Modules for JBoss mod_cluster

LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

LoadModule slotmem_module modules/mod_slotmem.so

LoadModule manager_module modules/mod_manager.so

LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

LoadModule advertise_module modules/mod_advertise.so

# JBoss mod_manager (mod_cluster) configuration.

<IfModule manager_module>

    Maxnode 60

    Maxhost 40

    Maxcontext 200

</IfModule>

# Virtual host for JBoss mod_cluster (port 6666 will be used to receive information from JBoss servers)

Listen 192.168.150.8:6666

<VirtualHost 192.168.150.8:6666>

    KeepAliveTimeout 60

    MaxKeepAliveRequests 0

    ManagerBalancerName ApacheHttpdBalancer

    ServerAdvertise Off

    <Location />   

        Order deny,allow

        Deny from all

        Allow from 192.168.150.

    </Location>

</VirtualHost>

The applications servers are JBoss EAP 5.1 and Tomcat 6.0.32, all under Windows 2008 Server R2.

Each JBoss EAP has modcluster.sar deployed. The only change I did to the config was on the following two lines of "mod_cluster-jboss-beans.xml" file:

<property name="proxyList">192.168.150.8:6666,192.168.150.9:6666</property>

<property name="advertise">false</property>

Each Tomcat 6.0.32 has the following files added to the lib folder:

jboss-logging-jdk.jar

jboss-logging-spi.jar

mod_cluster.jar

and the follwing line added to "server.xml":

<Listener className="org.jboss.modcluster.catalina.ModClusterListener" advertise="false" proxyList="192.168.150.8:6666,192.168.150.9:6666" excludedContexts="ROOT,manager" />

Please notice that this a medium size system that accounts for (each httpd, as seen in mod_cluster manager):

24 nodes

24 Vhosts

58 Contexts

Not all contexts are unique. From the 58 contexts, 6 are repeated and belong to the same (heavy-loaded app) and 2 are also repeated (belong to another moderately loaded app). The rest are unique contexts.

As for the SO, the fact that I am running Windows 2008 does not seem to be a factor since other reports on the same (?) problem seem to indicate other SOs.

Please let me know if you require any further information.

Joao

As for your read of the code, I belive you are correct. On line 654 of DefaultMCMPHandler.java one can read the following comment:

// Parse the line, which is formed like HTTP/1.x YYY Message

and on line 666 the HTTP code is parsed to variable responseStatus:

String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

A well formed response should be something like

HTTP/1.1 200 OK

The java.lang.StringIndexOutOfBoundsException: String index out of range: -1 is thrown because line 666 is not finding the second space. This is what fails:

line.indexOf(' ', spaceIndex + 1)

I haven't debuged but I'll take a look at the CONFIG HTTP messages. I'm pretty sure that at this point in the code they are not formated as HTTP/1.x YYY Message.

I would say that it could be enough to find the first space and read the 3 chars that follow is the HTTP code is allways formated like XXX. Something like this:

String responseStatus = line.substring(spaceIndex + 1, spaceIndex + 4);

Joao

Hello Yves,

I may be wrong but I disagree that the cause for the exception are the "1f82" chars.

Please notice the following:

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

    at java.lang.String.substring(String.java:1937)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

Line 666 of the code:

String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

It seems that the code is trying to get an HTTP responseStatus. This is also why the coment a few lines up is:

// Parse the line, which is formed like HTTP/1.x YYY Message

The String index out of range: -1 happens over the parsing of variable line (variable type is String).

Looking at the code, line is the result of

line = sendRequest(proxy, proxyhead, body);

Also looking at sendRequest (look at lines 530 to 542), line is the result of

return proxy.getConnectionReader().readLine();

Here is the complete sendRequest method:

   private String sendRequest(Proxy proxy, String command, String body) throws IOException

   {

      Writer writer = proxy.getConnectionWriter();

      writer.append(command).append(NEW_LINE);

      writer.append("Content-Length: ").append(String.valueOf(body.length())).append(NEW_LINE);

      writer.append("User-Agent: ClusterListener/1.0").append(NEW_LINE);

      writer.append("Connection: Keep-Alive").append(NEW_LINE);

      writer.write(NEW_LINE);

      writer.write(body);

      writer.write(NEW_LINE);

      writer.flush();

      // Read the first response line and skip the rest of the HTTP header

      return proxy.getConnectionReader().readLine();

   }

As you can see, the comment above the return statement says: "// Read the first response line and skip the rest of the HTTP header"

I belive the first response line, from your debug is:

HTTP/1.1 200 OK

If what I wrote above is correct, your TCPdump would not cause this exception;

java.lang.StringIndexOutOfBoundsException: String index out of range: -1

    at java.lang.String.substring(String.java:1937)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:666)

In fact, based on your TCPdump, the result of

String responseStatus = line.substring(spaceIndex + 1, line.indexOf(' ', spaceIndex + 1));

would be

responseStatus = "200"

log.trace("Parsing responseStatus: "+line);

2011-09-20 09:10:45,134 TRACE [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Parsing responseStatus:1f53

Well, that's strange. What happened to the HTTP header????