1 2 3 4 Previous Next 51 Replies Latest reply on Jul 18, 2013 12:23 PM by kasik

    Infinispan on EC2

    ravinderrana

      Hi,

       

      I am using infinispan for cache in clustering mode on amazon ec2. I've configured it to use s3_ping for discovery. Currently, i've only 2 nodes but discovery is not working. I can see that they are creating proper files in the given s3 bucket - folder name is the clustername given in config file and inside that folder there are 2 files and these files are constantly updated by the nodes as well (though the update is not always in sync, meaning, sometime i see that both updated the files at the same time and at times there's 10-15 seconds of difference between the updates).

       

      My config files are attached. Any help on this is appreciated.

       

      Thanks

        • 1. Infinispan on EC2
          genman

          If the store config seems to work BUT the JGroups config is working, it's likely a problem in your JGroups config.

           

          Most helpful for diagnosis: You should work on preparing useful, annotated log files showing the interactions between your host and Amazon S3. Probably I would turn on JGroups logging and tune as necessary.

           

          There may also be a way to override the discovery and just list the EC2 hosts you have allocated.

          • 2. Infinispan on EC2
            guytom19

            A note from our experience with the same thing-

             

            We launched with the S3_PING and it wasn't very stable, especially if you expect your service to be elastic e.g. be able to turn servers on and off separately.

             

            We lately switched to JDBC_PING which has been officially released in the latest jgroups version and it seems more stable (although not 100% either).

             

            Guy

            • 3. Infinispan on EC2
              sannegrinovero

              Hi Guy,

              I'm the one who contributed the JDBC_PING protocol, I'd love to hear what is wrong with it, we can improve if needed, or maybe some more logging could be helpful. I wouldn't mind knowing more about your problems with S3_PING either, do you have a clue about which issues you where having?

               

              Regards,

              Sanne

              • 4. Re: Infinispan on EC2
                guytom19

                Sanne,

                 

                I sent you a private message about it sometime ago.

                 

                Generally when we play enough with either solution, restarting servers etc. we get to situation that servers fail to start and get stuck trying to handshake non existing servers.

                 

                Right now we have scenario with JDBC_PING but not with infinispan (we tried with infinispan previously as well), attched is the log I have, we can recreate it and send more logs.

                 

                Guy

                 

                Message was edited by: Guy Tomer

                • 5. Infinispan on EC2
                  ravinderrana

                  Thanks a lot Elias and Guy, i dropped my plans to use S3_PING for now and got JDBC_PING working.

                  • 6. Re: Infinispan on EC2
                    sannegrinovero

                    Hi Guy,

                    sorry I didn't mean to ignore the private message, I never use them and it seems I had disabled private notifications to avoid spam.

                    I'd prefer to discuss stuff in public, if you have no specific requirements to do otherwise.

                     

                    Thank you for the report and logs, I'll inspect the cleanup routines. If you happen to see something suspicious in the code patch proposals are welcome, as always.

                     

                    I'd also like to know what's wrong with S3_PING, if you happen to have identified anything.

                     

                    Sanne

                    • 7. Re: Infinispan on EC2
                      guytom19

                      Hi Sanne,

                      One reproducible scenario we found is by causing the tomcat to hot deploy the application using jgroups. This seem to constantly bring us to this retry scenario.

                      With S3_PING we experienced similar problems, it seemed to happen more then with the JDBC_PING but I can't be certain.

                      Guy

                      • 8. Infinispan on EC2
                        belaban

                        Works fine for me (tried with JGroups 3.0.0.Alpha1, but 2.12.0 or 2.12.1 should be the same).

                         

                        I did make 2 changes though:

                        1. location="sandbox_clustering" to "sandboxclustering" though as underscores are not valid bucket names
                        2. I removed access_key and secret_access_key and made the sandboxclustering bucket PUBLIC
                        • 9. Infinispan on EC2
                          itayhindy

                          Hi Bela Ban,

                           

                          I have make some more tests with jgroups (2.12.0.Final) and JDBC_PING and here are my results:

                          1 - Starting servers A,B and C (A is the first one to start) all using JDBC_PING for discovery.

                          2 - There is no problem in adding more servers.

                          3 - There is no problem with restarts on servers B and C.

                          4 - If I try to restart server A (A is the first started server) it will fail to start with the next messages repeating all the time:

                          [2011-04-27 14:58:27,918] [main] WARN  org.jgroups.protocols.TCP  - hp-HP-25896: no physical address for cc16337b-1bdb-078e-9d37-1b1c11ba325b, dropping message.

                          5 - From that point (restart server A) all the other servers restarts will also fail with the above message.

                           

                          Can it be that because the first started server is the coordinator we can not restart it?

                           

                          Thanks,

                          Itay

                          • 10. Re: Infinispan on EC2
                            hussam.galal

                            I have been unable to run infinispan on EC2, tried different tricks yet the same problem. An exception is thrown when using S3_PING protocol. I have created an S3 bucket and I was able to access the bucket through bucketExplorer tool to make sure there are no issues accessing the bucket and the credentials are correct.

                             

                            The exception implies that for some reason JGroups by default is trying to use an HTTPs connection:

                             

                             

                            2011-05-09 10:08:30,012 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/cachelab-0.1.1-SNAPSHOT]] (main) StandardWrapper.Throwable

                            org.infinispan.CacheException: Unable to invoke method public abstract void org.infinispan.remoting.transport.Transport.start() on object

                                at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:174)

                                at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)

                                at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:687)

                                at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:589)

                                at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:137)

                                at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:137)

                                at org.infinispan.CacheDelegate.start(CacheDelegate.java:323)

                                at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

                                at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:439)

                                at myapp.cache.dataloader.SearchCache.init(SearchCache.java:48)

                                at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1048)

                                at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:950)

                                at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4122)

                                at org.apache.catalina.core.StandardContext.start(StandardContext.java:4417)

                                at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:321)

                                at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:145)

                                at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461)

                                at org.jboss.web.deployers.WebModule.startModule(WebModule.java:118)

                                at org.jboss.web.deployers.WebModule.start(WebModule.java:97)

                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                                at java.lang.reflect.Method.invoke(Method.java:597)

                                at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

                                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

                                at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

                                at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

                                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

                                at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)

                                at $Proxy38.start(Unknown Source)

                                at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)

                                at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)

                                at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

                                at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

                                at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                                at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                                at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)

                                at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                                at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                                at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

                                at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

                                at org.jboss.system.ServiceController.doChange(ServiceController.java:688)

                                at org.jboss.system.ServiceController.start(ServiceController.java:460)

                                at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)

                                at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)

                                at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)

                                at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

                                at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

                                at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

                                at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

                                at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

                                at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

                                at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

                                at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                                at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                                at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                                at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

                                at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

                                at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

                                at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

                                at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

                                at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

                                at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

                                at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

                                at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                                at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                                at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                                at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                                at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

                                at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

                                at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

                                at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

                                at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

                                at org.jboss.Main.boot(Main.java:221)

                                at org.jboss.Main$1.run(Main.java:556)

                                at java.lang.Thread.run(Thread.java:662)

                            Caused by: java.lang.reflect.InvocationTargetException

                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                                at java.lang.reflect.Method.invoke(Method.java:597)

                                at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

                                ... 80 more

                            Caused by: org.infinispan.CacheException: org.jgroups.ChannelException: unable to setup the protocol stack

                                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.buildChannel(JGroupsTransport.java:261)

                                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannel(JGroupsTransport.java:211)

                                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannelAndRPCDispatcher(JGroupsTransport.java:227)

                                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:154)

                                ... 85 more

                            Caused by: org.jgroups.ChannelException: unable to setup the protocol stack

                                at org.jgroups.JChannel.init(JChannel.java:1728)

                                at org.jgroups.JChannel.<init>(JChannel.java:249)

                                at org.jgroups.JChannel.<init>(JChannel.java:215)

                                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.buildChannel(JGroupsTransport.java:258)

                                ... 88 more

                            Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching myapp.cache.infinispan.lab.s3.amazonaws.com found.

                                at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)

                                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)

                                at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)

                                at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)

                                at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)

                                at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)

                                at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)

                                at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)

                                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)

                                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)

                                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)

                                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149)

                                at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)

                                at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)

                                at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)

                                at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)

                                at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)

                                at org.jgroups.protocols.S3_PING$AWSAuthConnection.checkBucketExists(S3_PING.java:407)

                                at org.jgroups.protocols.S3_PING.init(S3_PING.java:95)

                                at org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:892)

                                at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:538)

                                at org.jgroups.JChannel.init(JChannel.java:1725)

                                ... 91 more

                            Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching my.app.clustering.lab.s3.amazonaws.com found.

                                at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:193)

                                at sun.security.util.HostnameChecker.match(HostnameChecker.java:77)

                                at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:264)

                                at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:250)

                                at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1185)

                                ... 108 more

                            • 11. Infinispan on EC2
                              vblagojevic

                              Your S3_PING configuration is invalid. Go to https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key#access_credentials of your Amazon EC2 account and use values of Access Key ID and Secret Access Key in S3_PING attributes. Also location attribute has to match the name of your bucket.

                               

                              Have fun

                              • 12. Re: Infinispan on EC2
                                hussam.galal

                                Thank for your reply. But I still couldnt celebrate getting to work.

                                 

                                I was initially suspecting its a problem with the credentials, but as I mentioned earlier I tried the exact same credentials using bucketexplorer and I was able to connect. Kindly find my S3_PING configuration file attached I have slightly tweaked the credentials so they are unusable. Im wondering if anything looks fishy in my configuration.

                                 

                                Could it be that special characters in the configuration file like '+' and '/' are not handled thats why credentials are not being passed correctly ?

                                • 13. Infinispan on EC2
                                  vblagojevic

                                  Hi,

                                   

                                  Yes, include all the characters in secret key and access key. Also make sure that EC2 nodes have a firewall policy that opens tcp ports 7800-7850 and tcp port 9777. Make sure that bind_port in TCP jgroups config is specified to be 7800 and start_port attribute of FD_SOCK is 9777.

                                   

                                  Regards,

                                  Vladimir

                                  • 14. Re: Infinispan on EC2
                                    hussam.galal

                                    Thanks once again for your reply, this is frustrating as I still could not get inifinispan to run on ec2. Im running on Jboss-eap-.5.1 and by default is uses an older version of JGroups, therefore i have included jgroups-2.12.1.Final.jar in my ../WEB-INF/lib directory and to override the default class loading behaviour of Jboss I have added cache-config.xml. I have also tried it on Jboss 6 and I get the exact same error.

                                     

                                    I have the following TCP ports opened: 9777, 7500-7850 in my security group

                                     

                                    I dont think it's a port problem as Im not getting socket errors. I have attached my cache-config.xml and jboss-classloading.xml

                                    1 2 3 4 Previous Next