5 Replies Latest reply on Oct 30, 2009 9:14 PM by kkoster

    lookupLocaly failing

    kkoster

      I am attempting to deploy an EAR that contains annotated EJB3 beans. I have already done this on several instances of JBoss, but this one is causing me problems. I am getting the following exception.

      javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeException: Exception while trying to locate proxy factory in JNDI, at key ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR]
      


      I have checked the JNDI tree in the web-console using JNDIView and can see the name that I am looking up (i.e. kdr/remote/UserContextManagement) registered with what appear to the the appropriate bindings.

      I turned up the logging to trace on org.jboss.ha.jndi and got the following output:
      2009-10-27 13:38:56,522 TRACE [org.jboss.ha.jndi.HANamingService] (JBoss System Threads(1)-4) Accepted bootstrap client: Socket[addr=/192.168.11.3,port=51953,localport=1100]
      2009-10-27 13:38:57,086 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(4)-192.168.11.3) lookup, name=kdr/remote/UserContextManagement
      2009-10-27 13:38:57,087 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally, name=kdr/remote/UserContextManagement
      2009-10-27 13:38:57,124 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(4)-192.168.11.3) lookup, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
      2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
      2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
      javax.naming.NameNotFoundException: ProxyFactory not bound
       at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
       at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
       at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
       at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
       at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
       at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.ha.framework.server.HARMIServerImpl.invoke(HARMIServerImpl.java:209)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
       at sun.rmi.transport.Transport$1.run(Transport.java:159)
       at java.security.AccessController.doPrivileged(Native Method)
       at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
       at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:619)
      2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) calling lookupLocally(ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR) on HAJNDI cluster
      2009-10-27 13:38:57,132 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) Returned results size: 0
      


      It looks like an internal caching issue of the proxy. This is going a bit deeper than I can spend time looking at right now. If anyone has any ideas what is going on here, I would be grateful for some suggestions.

      Karl

        • 1. Re: lookupLocaly failing
          kkoster

          Sorry, forgot to mention this is on JBoss 5.1.0.GA

          • 2. Re: lookupLocaly failing
            kkoster

            Just to add more detail. I turned up the logging on org.jnp.server to TRACE (it produces a lot of output) and found this in the log:

            2009-10-27 19:09:23,284 TRACE [org.jnp.server.NamingServer] (main) bind UserContextManagement=Reference Class Name: Proxy for: org.kdr.security.user.UserContextManagementRemote
            Type: ProxyFactoryKey
            Content: ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
            Type: EJB Container Name
            Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3
            Type: Proxy Factory is Local
            Content: false
            Type: Remoting Host URL
            Content: socket://GFL1:3873/?
            Type: Remote Business Interface
            Content: org.kdr.security.user.UserContextManagementRemote
            Type: Cluster Partition Name
            Content: KDR
            Type: Cluster Proxy Factory Load Balance Policy
            Content: org.jboss.ha.client.loadbalance.RoundRobin
            Type: Cluster Family Name
            Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3socketKDR
            Type: Cluster Target Remoting Host URL
            Content: socket://GFL1:3873/?
            , Proxy for: org.kdr.security.user.UserContextManagementRemote
            
            2009-10-27 19:09:23,285 TRACE [org.jnp.server.NamingServer] (main) org.jnp.server.NamingServer@66bc8644, setBinding: name=UserContextManagement, obj=Reference Class Name: Proxy for: org.kdr.security.user.UserContextManagementRemote
            Type: ProxyFactoryKey
            Content: ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
            Type: EJB Container Name
            Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3
            Type: Proxy Factory is Local
            Content: false
            Type: Remoting Host URL
            Content: socket://GFL1:3873/?
            Type: Remote Business Interface
            Content: org.kdr.security.user.UserContextManagementRemote
            Type: Cluster Partition Name
            Content: KDR
            Type: Cluster Proxy Factory Load Balance Policy
            Content: org.jboss.ha.client.loadbalance.RoundRobin
            Type: Cluster Family Name
            Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3socketKDR
            Type: Cluster Target Remoting Host URL
            Content: socket://GFL1:3873/?
            , className=Proxy for: org.kdr.security.user.UserContextManagementRemote
            


            This looks like the been and the relevant interfaces/classes/names are being parsed out correctly. I later see them in what looks like the existing bindings with the name server when new artifacts are being bound. Example:

            2009-10-27 19:09:23,469 TRACE [org.jnp.server.NamingServer] (main) org.jnp.server.NamingServer@7841514b, No binding for: BusinessCalendarManagementBean in context risk-server, bindings:
            EnvironmentObjectSourceManagementBean=org.jnp.server.NamingServer@422f4736
            ParameterConstructorManagementBean=org.jnp.server.NamingServer@5711297a
            EnvironmentObjectFactoryContextManagementBean=org.jnp.server.NamingServer@550c45fe
            InstrumentEnvironmentMatcherManagementBean=org.jnp.server.NamingServer@3fbd15da
            TradeManagementBean=org.jnp.server.NamingServer@69d7135a
            RoleContextManagementBean=org.jnp.server.NamingServer@ebab1b6
            PerturbationRuleManagementBean=org.jnp.server.NamingServer@1d89e611
            TranslationSourceManagementBean=org.jnp.server.NamingServer@553d70ce
            SpotCurveHistoryStateManagementBean=org.jnp.server.NamingServer@780bfb81
            TradeMatchStateManagementBean=org.jnp.server.NamingServer@27f7ee30
            TradeMatchContextManagementBean=org.jnp.server.NamingServer@565de293
            ObjectStateManagementBean=org.jnp.server.NamingServer@3fa472bf
            AggregatorNameManagementBean=org.jnp.server.NamingServer@3822ee3c
            TradeSettlementApplyRuleManagementBean=org.jnp.server.NamingServer@435f7db5
            EnvironmentObjectFactoryConfigurationManagementBean=org.jnp.server.NamingServer@3f834f84
            TradeReaderBuilderManagementBean=org.jnp.server.NamingServer@37f0d3ce
            ObjectTypeManagementBean=org.jnp.server.NamingServer@577c76cd
            DateRangeToDayRuleManagementBean=org.jnp.server.NamingServer@423425c1
            TradeLinkTypeManagementBean=org.jnp.server.NamingServer@221c02f5
            TradeMatchRuleManagementBean=org.jnp.server.NamingServer@163542ef
            ValuatorManagementBean=org.jnp.server.NamingServer@3e6df2b0
            TradeSettlementRuleManagementBean=org.jnp.server.NamingServer@1b9f1a8b
            UnitManagementBean=org.jnp.server.NamingServer@20e38957
            EnvironmentObjectFactoryManagementBean=org.jnp.server.NamingServer@4fc74c82
            ModelTypeManagementBean=org.jnp.server.NamingServer@7a70c042
            EnvironmentObjectTypeManagementBean=org.jnp.server.NamingServer@7d065976
            ValuationAggregatorSpecificationManagementBean=org.jnp.server.NamingServer@6b091424
            PerturbationContextManagementBean=org.jnp.server.NamingServer@5622ea37
            InstrumentTypeManagementBean=org.jnp.server.NamingServer@44b89dd0
            CashTypeManagementBean=org.jnp.server.NamingServer@6811c805
            TradeFeeRuleContextManagementBean=org.jnp.server.NamingServer@361227f
            RoleManagementBean=org.jnp.server.NamingServer@1884bb5
            ObjectTypeStateManagementBean=org.jnp.server.NamingServer@dd0ca4f
            TradeMatchManagementBean=org.jnp.server.NamingServer@3a135e2a
            CurveStrategyContextManagementBean=org.jnp.server.NamingServer@60648df2
            SynchWorkerManagementBean=org.jnp.server.NamingServer@61df79fe
            TradeLinkStateManagementBean=org.jnp.server.NamingServer@1523db0
            TradeFeeApplyRuleManagementBean=org.jnp.server.NamingServer@793044e
            TradeFeeRuleManagementBean=org.jnp.server.NamingServer@e5ac92f
            LocationManagementBean=org.jnp.server.NamingServer@104c2c06
            TradeFeeSuppressionRuleManagementBean=org.jnp.server.NamingServer@188d7d8b
            JMSEmitterBean=org.jnp.server.NamingServer@840509b
            ValuatorContextManagementBean=org.jnp.server.NamingServer@6a562541
            TradeFeeContextManagementBean=org.jnp.server.NamingServer@4dd2c7f5
            ValuationJobSpecificationManagementBean=org.jnp.server.NamingServer@3ae6f61c
            BucketerManagementBean=org.jnp.server.NamingServer@3ea6da02
            TradeSettlementRuleContextManagementBean=org.jnp.server.NamingServer@6ba964d4
            ValueTypeManagementBean=org.jnp.server.NamingServer@6dc4a8dd
            ScenarioManagementBean=org.jnp.server.NamingServer@19dad138
            UnitConvertManagementBean=org.jnp.server.NamingServer@2cafed2f
            PerturbationRuleContextManagementBean=org.jnp.server.NamingServer@1d836fe
            TradeSourceManagementBean=org.jnp.server.NamingServer@cf72664
            HolidayTableManagementBean=org.jnp.server.NamingServer@77df9558
            TranslationManagementBean=org.jnp.server.NamingServer@2a9d704c
            PerturbationApplyRuleManagementBean=org.jnp.server.NamingServer@2486e1cd
            InstrumentTemplateManagementBean=org.jnp.server.NamingServer@5c31d43
            AggregatorExtractorManagementBean=org.jnp.server.NamingServer@18979865
            DayCountManagementBean=org.jnp.server.NamingServer@44130dba
            CurveStrategyManagementBean=org.jnp.server.NamingServer@204d57a2
            TranslationTypeManagementBean=org.jnp.server.NamingServer@260f204b
            PortfolioManagementBean=org.jnp.server.NamingServer@7bc08345
            KeywordRendererManagementBean=org.jnp.server.NamingServer@6c9de34c
            TradeSettlementContextManagementBean=org.jnp.server.NamingServer@4807f3e2
            TimeContextManagementBean=org.jnp.server.NamingServer@54e5ebc5
            DateRangeToDateRangeRuleManagementBean=org.jnp.server.NamingServer@744f439d
            UserContextManagementBean=org.jnp.server.NamingServer@65821e23
            GroupContextManagementBean=org.jnp.server.NamingServer@78d96b4b
            ProductManagementBean=org.jnp.server.NamingServer@5b19393d
            ShareClassManagementBean=org.jnp.server.NamingServer@40b96b99
            


            But when the remote application makes a JNDI lookup the following details emerge:
            2009-10-27 19:11:50,589 TRACE [org.jboss.ha.jndi.HANamingService] (JBoss System Threads(1)-4) Accepted bootstrap client: Socket[addr=/192.168.11.3,port=57934,localport=1100]
            2009-10-27 19:11:51,619 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(3)-192.168.11.3) lookup, name=kdr/remote/UserContextManagement
            2009-10-27 19:11:51,620 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally, name=kdr/remote/UserContextManagement
            2009-10-27 19:11:51,786 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(3)-192.168.11.3) lookup, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
            2009-10-27 19:11:51,786 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
            2009-10-27 19:11:51,786 TRACE [org.jnp.server.NamingServer] (RMI TCP Connection(3)-192.168.11.3) org.jnp.server.NamingServer@7e628e42, No binding for: ProxyFactory in context , bindings:
            UserTransactionSessionFactory=org.jnp.interfaces.MarshalledValuePair@1951dea9
            risk-server=org.jnp.server.NamingServer@7841514b
            UUIDKeyGeneratorFactory=org.jnp.interfaces.MarshalledValuePair@36c2b1d3
            HAPartition=org.jnp.server.NamingServer@1206cda0
            SecureManagementView=org.jnp.server.NamingServer@2113cfd1
            SecureDeploymentManager=org.jnp.server.NamingServer@1624bdb0
            HiLoKeyGeneratorFactory=org.jnp.interfaces.MarshalledValuePair@435b92dd
            XAConnectionFactory=org.jnp.interfaces.MarshalledValuePair@72381307
            topic=org.jnp.server.NamingServer@32ae8db4
            kdr=org.jnp.server.NamingServer@4f840e24
            ClusteredConnectionFactory=org.jnp.interfaces.MarshalledValuePair@7cb9682a
            ProfileService=org.jnp.interfaces.MarshalledValuePair@10f1abd1
            HASessionState=org.jnp.server.NamingServer@1d7275cd
            SecureProfileService=org.jnp.server.NamingServer@4127e183
            queue=org.jnp.server.NamingServer@56d6d6c6
            ClusteredXAConnectionFactory=org.jnp.interfaces.MarshalledValuePair@7065dd5e
            UserTransaction=Reference Class Name: org.jboss.tm.usertx.client.ClientUserTransaction
            
            invokers=org.jnp.server.NamingServer@5e0fae9
            ConnectionFactory=org.jnp.interfaces.MarshalledValuePair@536e5b7b
            jmx=org.jnp.server.NamingServer@755392d5
            JAXR=org.jnp.interfaces.MarshalledValuePair@5edbde1
            persistence.unit:unitName=risk-server.ear=org.jnp.server.NamingServer@14dd3d
            TomcatAuthenticators=org.jnp.interfaces.MarshalledValuePair@40b29eda
            console=org.jnp.server.NamingServer@20550f84
            
            2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
            javax.naming.NameNotFoundException: ProxyFactory not bound
             at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
             at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
             at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
             at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
             at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
             at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at org.jboss.ha.framework.server.HARMIServerImpl.invoke(HARMIServerImpl.java:209)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:597)
             at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
             at sun.rmi.transport.Transport$1.run(Transport.java:159)
             at java.security.AccessController.doPrivileged(Native Method)
             at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
             at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
             at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
             at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
             at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
             at java.lang.Thread.run(Thread.java:619)
            2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) calling lookupLocally(ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR) on HAJNDI cluster
            2009-10-27 19:11:51,791 TRACE [org.jboss.ha.framework.interfaces.HAPartition.KDR] (RMI TCP Connection(3)-192.168.11.3) callMethodOnCluster(true), objName=HAJNDI, methodName=lookupLocally, members=[]
            2009-10-27 19:11:51,791 TRACE [org.jboss.ha.framework.server.ClusterPartition$RpcHandler] (RMI TCP Connection(3)-192.168.11.3) destination list of HAJNDI.lookupLocally() is empty: no need to send message
            2009-10-27 19:11:51,791 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) Returned results size: 0
            


            It looks like the only bindings are the root of the name tree. I have this application running without any problems on Windows but when I moved to Linux, this problem arose. The annotations on this bean are:

            @Stateless
            @Clustered(loadBalancePolicy = "org.jboss.ha.client.loadbalance.RandomRobin", partition = "KDR")
            @Local( { UserContextManagementLocal.class })
            @LocalBinding(jndiBinding = "kdr/local/UserContextManagement")
            @Remote( { UserContextManagementRemote.class })
            @RemoteBinding(jndiBinding = "kdr/remote/UserContextManagement")
            public class UserContextManagementBean implements UserContextManagementLocal,
             UserContextManagementRemote, Serializable {
            


            I am stumped as to what is going on.

            • 3. Re: lookupLocaly failing
              jaikiran

               

              at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
              at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)


              Are you running the "all" server configuration?

              javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeExcep
              tion: Exception while trying to locate proxy factory in JNDI, at key ProxyFactory/risk-server/UserCo
              ntextManagementBean/kdr/remote/UserContextManagement/KDR]


              Please post this entire exception stacktrace.

              • 4. Re: lookupLocaly failing
                kkoster

                I have copied all to a different name and am running that. I don't like to touch the baseline configurations so I have something to compare to if something goes wrong.

                • 5. Re: lookupLocaly failing
                  kkoster

                  The entire stack trace of the exception thrown is the next to last code posting and starts with

                  2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookup
                  Locally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManage
                  ment/KDR
                  javax.naming.NameNotFoundException: ProxyFactory not bound
                  


                  I have additional TRACE level logging coming from the server but is larger than 1MB by the the JNDI lookup fails. I can send it, but I believe it will be too large to post.