1 2 Previous Next 15 Replies Latest reply: Sep 5, 2013 8:37 AM by Tomaz Cerar RSS

Domain deployment slow

Mitchell Ackerman Newbie

Hi, I have a scenario where we have several servers in Amazon, and a Domain manager locally.  When uploading an app to the Domain manager, that happens in a matter of seconds.  However, when I perform the operation to add the app to the Server Group, which uploads the app to the hosts in the cluster it is VERY slow, about 15 mins for a 40 meg file.  While I recognize that JBoss has no way to affect the underlying network, it is of note that other protocols can upload to those servers much faster.  For example Filezilla can upload the same file in about 1 min.  We have also noticed that secure protocols, such as SCP, are slower. 

 

Does the deployment protocol utilize a security layer that could be disabled?  I have ensured that the management interfaces are not using a security realm, though I think that is only for authentication.  Does anyone have any other suggestions as to how to improve domain deployment speeds?  (note that placing the domain manager on a local network was an attempt to improve the situation, which it did, but only marginally).

 

Unfortunately, for development, a 15 min turnaround to deploy the app is too much of a burden, so in the meantime we are going to revert to managing our own cluster with the servers running in standalone mode & using filezilla, or an alternative mechanism, to upload the app.

 

thanks for any suggestions, Mitchell

  • 1. Re: Domain deployment slow
    Brian Stansberry Master

    Do you know how long it takes if there's only a single host in the domain (besides the local master)?

     

    Do you know how long it takes to deploy to a standalone server using the CLI?

  • 2. Re: Domain deployment slow
    Mitchell Ackerman Newbie

    For one server (other than the master) it appears to take the same length of time as for 4 servers, i.e., about 15 mins

     

    Can't say that I have tried using the CLI, i'll give that a shot tomorrow.

     

    thanks, Mitchell

  • 3. Re: Domain deployment slow
    Mitchell Ackerman Newbie

    I also forgot to add that we are currently using JBoss 7.0.2.  Have there been any improvements to deployment speeds in 7.1.1?

  • 4. Re: Domain deployment slow
    Emanuel Muckenhuber Master

    We did not look at deployment speed in particular, but there have been a couple of other related changes and improvements.

     

    So it's hard to say if this is still an issue. While we start investing this a bit, do you have an easy way to test this scenario with 7.1.1?

  • 5. Re: Domain deployment slow
    Michael Nielson Newbie

    I'm having a similar problem with very slow upload speeds when using the domain web interface and also when using the jboss-cli. When using the web interface I'm running into this timeout issue: https://issues.jboss.org/browse/AS7-4234 when i scp the file to the same server the upload completes in a few seconds. I tried scping the file to the same server as the domain controller and using the cli command "deploy /home/somefile.ear" but even locally the deploy command never completes.

     

    The only error message I've seen comes from the web interface when the file upload times out.

     

    I have a second domain controller setup that isn't experiencing these issues so i'm not sure what's wrong?

     

    I'm using JBoss 7.1.1 on linux. Any pointers on how to figure this out?

     

    Michael

  • 6. Re: Domain deployment slow
    Emanuel Muckenhuber Master

    In general you cannot compare deploying a deployment using the mgmt API to scp or a simple copy. The deploy operation will wait until the deployment is completely deployed on all servers in the specified groups before returning. So it also depends on the size of the server group and the deployment time of the application.

     

    Can you check the logs and once you see a log message like:  JBAS015876: Starting deployment of "deployment.war" would mean that the content was copied successfully. Perhaps the issues are when trying to deploy the actual deployment?

  • 7. Re: Domain deployment slow
    Michael Nielson Newbie

    This turned out to be a combination of issues for me. One the web upload was timing out after 15 seconds, Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet), and Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.

     

    I'm still looking for answers on the remote jboss-cli.sh deploy issue.

     

    Michael

  • 8. Re: Domain deployment slow
    Michael Nielson Newbie

    One interesting bottleneck seems to be ProcessControllerConnectionService.THREAD_POOL_MAX_SIZE = 4; copying ears out to 6 hosts runs 4 at a time. Would it make sense to increase this constant?

  • 9. Re: Domain deployment slow
    Emanuel Muckenhuber Master

    One the web upload was timing out after 15 seconds

    Yeah, that should be fixed already in upstream and available in a nightly snapshot build.

    Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet)

    When that happens can you take a thread-dump and also check the memory usage of the domain-controller?

    Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.

    Hmm, good point we should improve that.

  • 10. Re: Domain deployment slow
    Mitchell Ackerman Newbie

    Sorry it's taken me so long to try this scenario with 7.1.1.  We just upgraded our servers to 7.1.1.Final, and I can unfortunately report that the upload speed is no better.  Our war file has shrunk a little, but to upload to the servers still takes in excess of 11 mins, compared to < 2 for filezilla.

  • 11. Re: Domain deployment slow
    Emanuel Muckenhuber Master

    Hi, interresting timing - since i've just had some time in the last couple of days to investigate this. It seems to be a problem in the configuration of our remoting protocol, using a too small chunk size. So i am currently working on a patch for AS to speed things up.

  • 12. Re: Domain deployment slow
    Mitchell Ackerman Newbie

    great!  thanks

  • 13. Re: Domain deployment slow
    Emanuel Muckenhuber Master

    FYI we also found an issue where the slave host-controller was synchronizing deployment content althought it was already available locally. This should be fixed upstream soon as well.

  • 14. Re: Domain deployment slow
    Yves Martin Newbie

    I got the same issue with 7.1.1 even using

     

    export JAVA_OPTS=-Xmx1024m

     

    to run jboss-cli.sh

     

    The host controller has logged

     

    JBAS014900: Content added at location /opt/jboss/domain/data/content/f5/88b80e83b11449a016e695e638d0dfdc6b0d90/content

     

    ...but everything remains stuck without any explanation.

     

    So I decided to delete everything and get back to a clean JBoss, run my deployment script and it ends then successfully. Probably some internal state was messed up.

1 2 Previous Next