For a while we had 2 instances of 4.0.5 running where instance A was calling EJB services of instance B. We upgraded instance A to 5.1 and at that point started getting the InvokationException with the serial number mismatch between classes in the 2 different versions of jbossas-client.jar.
After doing a little research I started bringing up instance A with the property -Djboss.remoting.pre_2_0_compatible=true and the error I started seeing was
java.lang.Exception: Can not make remoting client invocation due to
not being connected to server.
When I googled this I found a seemingly helpful thread speaking about
the new req in remoting 2.x that connect() be called before invoke()
and how this did not use to be required in remoting 1.x
here is the thread: http://community.jboss.org/message/368917#368917
but I actually can't figure out how we are doing this since
we aren't writing a client and invoking from the container.
also, fyi, as a test upgrading instance B to 5.1 as well... everything works fine.
Any help would be greatly appreciated.
What you say is very plausible. When you download an EJB, you're downloading a proxy with an invocation handler, so you're running an invocation handler targeted to Remoting 1.4.x and executing it in an environment where it's calling Remoting 2.4/2.5.
You are correct that the Remoting call in the EJB invocation handler isn't accessible to you.
When I do a Remoting release, I run a test suite that ensures that the new version is compatible with the old version, but the compatibility is at the Remoting level. Your issue occurs outside of Remoting.
Given that the missing Client.connect() call is in the EJB invocation handler, I would suggest that you pose the question in the EJB forum. I know that there are compatibility test suites somewhere, but I don't remember what they guarantee regarding AS 4 and AS 5. It might be something like, AS 4 can make invocations on AS 5 but not vice versa. By the way, EJB2 and EJB3 are separate subsystems, so be sure to pose the question on the appropriate forum.
One suggestion I can make would be to replace the jboss-remoting.jar that comes with AS 5.1 with the one in AS 4.0.5. I can't guarantee that you won't discover other problems. For example, JBossMessaging, EJB2, and EJB3 all use Remoting, and it's possible that JBossMessaging depends on Remoting 2.4/2.5. If you're not using JBossMessaging, you might be OK.
In the end, you might have to upgrade instance B, the server.
Sorry I couldn't give you a simpler answer.