1 2 Previous Next 15 Replies Latest reply on Sep 5, 2013 8:37 AM by ctomc

    Domain deployment slow

    mackerman

      Hi, I have a scenario where we have several servers in Amazon, and a Domain manager locally.  When uploading an app to the Domain manager, that happens in a matter of seconds.  However, when I perform the operation to add the app to the Server Group, which uploads the app to the hosts in the cluster it is VERY slow, about 15 mins for a 40 meg file.  While I recognize that JBoss has no way to affect the underlying network, it is of note that other protocols can upload to those servers much faster.  For example Filezilla can upload the same file in about 1 min.  We have also noticed that secure protocols, such as SCP, are slower. 

       

      Does the deployment protocol utilize a security layer that could be disabled?  I have ensured that the management interfaces are not using a security realm, though I think that is only for authentication.  Does anyone have any other suggestions as to how to improve domain deployment speeds?  (note that placing the domain manager on a local network was an attempt to improve the situation, which it did, but only marginally).

       

      Unfortunately, for development, a 15 min turnaround to deploy the app is too much of a burden, so in the meantime we are going to revert to managing our own cluster with the servers running in standalone mode & using filezilla, or an alternative mechanism, to upload the app.

       

      thanks for any suggestions, Mitchell

        • 1. Re: Domain deployment slow
          brian.stansberry

          Do you know how long it takes if there's only a single host in the domain (besides the local master)?

           

          Do you know how long it takes to deploy to a standalone server using the CLI?

          • 2. Re: Domain deployment slow
            mackerman

            For one server (other than the master) it appears to take the same length of time as for 4 servers, i.e., about 15 mins

             

            Can't say that I have tried using the CLI, i'll give that a shot tomorrow.

             

            thanks, Mitchell

            • 3. Re: Domain deployment slow
              mackerman

              I also forgot to add that we are currently using JBoss 7.0.2.  Have there been any improvements to deployment speeds in 7.1.1?

              • 4. Re: Domain deployment slow
                emuckenhuber

                We did not look at deployment speed in particular, but there have been a couple of other related changes and improvements.

                 

                So it's hard to say if this is still an issue. While we start investing this a bit, do you have an easy way to test this scenario with 7.1.1?

                • 5. Re: Domain deployment slow
                  safetytrick

                  I'm having a similar problem with very slow upload speeds when using the domain web interface and also when using the jboss-cli. When using the web interface I'm running into this timeout issue: https://issues.jboss.org/browse/AS7-4234 when i scp the file to the same server the upload completes in a few seconds. I tried scping the file to the same server as the domain controller and using the cli command "deploy /home/somefile.ear" but even locally the deploy command never completes.

                   

                  The only error message I've seen comes from the web interface when the file upload times out.

                   

                  I have a second domain controller setup that isn't experiencing these issues so i'm not sure what's wrong?

                   

                  I'm using JBoss 7.1.1 on linux. Any pointers on how to figure this out?

                   

                  Michael

                  • 6. Re: Domain deployment slow
                    emuckenhuber

                    In general you cannot compare deploying a deployment using the mgmt API to scp or a simple copy. The deploy operation will wait until the deployment is completely deployed on all servers in the specified groups before returning. So it also depends on the size of the server group and the deployment time of the application.

                     

                    Can you check the logs and once you see a log message like:  JBAS015876: Starting deployment of "deployment.war" would mean that the content was copied successfully. Perhaps the issues are when trying to deploy the actual deployment?

                    • 7. Re: Domain deployment slow
                      safetytrick

                      This turned out to be a combination of issues for me. One the web upload was timing out after 15 seconds, Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet), and Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.

                       

                      I'm still looking for answers on the remote jboss-cli.sh deploy issue.

                       

                      Michael

                      • 8. Re: Domain deployment slow
                        safetytrick

                        One interesting bottleneck seems to be ProcessControllerConnectionService.THREAD_POOL_MAX_SIZE = 4; copying ears out to 6 hosts runs 4 at a time. Would it make sense to increase this constant?

                        • 9. Re: Domain deployment slow
                          emuckenhuber

                          One the web upload was timing out after 15 seconds

                          Yeah, that should be fixed already in upstream and available in a nightly snapshot build.

                          Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet)

                          When that happens can you take a thread-dump and also check the memory usage of the domain-controller?

                          Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.

                          Hmm, good point we should improve that.

                          • 10. Re: Domain deployment slow
                            mackerman

                            Sorry it's taken me so long to try this scenario with 7.1.1.  We just upgraded our servers to 7.1.1.Final, and I can unfortunately report that the upload speed is no better.  Our war file has shrunk a little, but to upload to the servers still takes in excess of 11 mins, compared to < 2 for filezilla.

                            • 11. Re: Domain deployment slow
                              emuckenhuber

                              Hi, interresting timing - since i've just had some time in the last couple of days to investigate this. It seems to be a problem in the configuration of our remoting protocol, using a too small chunk size. So i am currently working on a patch for AS to speed things up.

                              1 of 1 people found this helpful
                              • 12. Re: Domain deployment slow
                                mackerman

                                great!  thanks

                                • 13. Re: Domain deployment slow
                                  emuckenhuber

                                  FYI we also found an issue where the slave host-controller was synchronizing deployment content althought it was already available locally. This should be fixed upstream soon as well.

                                  • 14. Re: Domain deployment slow
                                    ymartin

                                    I got the same issue with 7.1.1 even using

                                     

                                    export JAVA_OPTS=-Xmx1024m

                                     

                                    to run jboss-cli.sh

                                     

                                    The host controller has logged

                                     

                                    JBAS014900: Content added at location /opt/jboss/domain/data/content/f5/88b80e83b11449a016e695e638d0dfdc6b0d90/content

                                     

                                    ...but everything remains stuck without any explanation.

                                     

                                    So I decided to delete everything and get back to a clean JBoss, run my deployment script and it ends then successfully. Probably some internal state was messed up.

                                    1 2 Previous Next