1 2 3 Previous Next 132 Replies Latest reply on Apr 18, 2011 3:13 PM by brian.stansberry

    Thoughts on filesystem action driven hot deployment

    brian.stansberry

      Some thoughts on hot deployment support for further discussion (summarizing some thoughts from David, Jason, John, Jesper, all mistakes are my own)

       

      First, terminology: what's being discussed here is hot-deploy in the traditional AS sense of detecting deployment archive changes and redeploying. It's not things like replacing classes in a running application. I assume though we want to support the exploded web archive resource (including jsp file) replacement we've traditionally supported (unless the JBW say otherwise).

       

      Hot deploy support will be limited to the standalone server use case. Hot deployment will not be supported via the Domain Controller; users should use the regular deployment API.

       

      For zipped archives:

       

      Scanner will detect an archive in the hot-deploy dir, e.g. hot-deploy/foo.war. Scanner will use the same deployment API that an external client would use to register the content with the server and deploy it. Once the content is registered the server will maintain it's own copy in an internal directory (e.g. server/data/deployments). Scanner will replace the hot-deploy/foo.war file with a simple marker file hot-deploy/foo.war.deployed.

       

      User can trigger a redeploy of the existing foo.war content by touching hot-deploy/foo.war.deployed.

       

      User can trigger an undeploy of foo.war by deleting hot-deploy/foo.war.deployed.

       

      User can trigger a replacement of foo.war (undeploy old, deploy new) by copying a new foo.war into hot-deploy/

       

      For exploded archives:

       

      User can copy an exploded archive into hot-deploy/ (e.g. hot-deploy/foo.war/) but this will not trigger a deployment.

       

      It will trigger an INFO level message in the log noting that the exploded archive has been detected.

       

      To trigger a deployment user must copy a marker file jboss.deploy to META-INF.

       

      So, user can copy a large deployment without worrying that it will be deployed when only partly copied.

       

      User can trigger a redeploy of the existing foo.war content by touching hot-deploy/foo.war/META-INF/jboss-deploy

       

      User can trigger an undeploy of foo.war by deleting hot-deploy/foo.war/META-INF/jboss-deploy

       

      Undeploy by deleting the entire hot-deploy/foo.war/ directory is not supported! The user should delete hot-deploy/foo.war/META-INF/jboss-deploy to trigger undeploy, after which they can delete the entire directory if they so choose. If they leave the directory, that will not result in risk of another deploy (e.g. after server restart), since the META-INF/jboss-deploy won't be there.

       

      Users can replace content in the exploded archive (e.g. hot-deploy/foo.war/index.html). We need to clarify exactly what the results of this will be; e.g. whether we support the current AS 6 behavior, something more, something less...

       

      If we support something like the AS 6 behavior where some content changes are reflected in the runtime without a redeploy, the Scanner will be responsible to informing the server of the new content. It will do this via a separate internal API, not the regular end-user deployment API. If we end up adding methods needed for updating content without redeploy to the regular end-user deployment API, or to some other end-user API, the scanner should use that API. But we will not add methods to an end-user API solely to support scanner operations.

       

      The current assumption is the scanner will tell the server to mount the exploded archive directly from the hot-deploy dir, rather than copying it. But that is an internal implementation detail subject to change.

        • 1. Re: Thoughts on hot deployment
          jason.greene

          Excellent summary!

           

          One small thing to nail down is what happens when a user undeploys a file using the console or the CLI that was deployed via hot-deploy. Under that scenario you could end up with a leftover foo.war.deployed.

          • 2. Re: Thoughts on hot deployment
            brian.stansberry

            Perhaps some listener API so service coordinating the scanner can get a notification of undeployment and delete the file?

            • 3. Re: Thoughts on hot deployment
              jason.greene

              Brian Stansberry wrote:

               

              Perhaps some listener API so service coordinating the scanner can get a notification of undeployment and delete the file?

               

              Yeah it could also poll as well on startup to "cleanup" any stale entries.

              • 4. Re: Thoughts on hot deployment
                jaikiran

                Nice!

                 

                Here's some questions I have:

                Brian Stansberry wrote:

                 

                For zipped archives:

                 

                Scanner will detect an archive in the hot-deploy dir, e.g. hot-deploy/foo.war. Scanner will use the same deployment API that an external client would use to register the content with the server and deploy it. Once the content is registered the server will maintain it's own copy in an internal directory (e.g. server/data/deployments). Scanner will replace the hot-deploy/foo.war file with a simple marker file hot-deploy/foo.war.deployed.

                Would it be a "rename" or would the foo.war.deployed be a completely different file? From what I understand, I believe  it would be a completely different file without any .war semantics, but just wanted to be sure. Furthermore, where would be original foo.war be moved to? Are users expected to know where it's moved to? If not, then they probably will have to maintain a copy of their original deployment somewhere (I know most of them already do that, but again I just wanted to be clear about the expectations).

                 

                 

                Brian Stansberry wrote:

                 

                User can trigger a replacement of foo.war (undeploy old, deploy new) by copying a new foo.war into hot-deploy/

                Later in your post, I see that special care has to taken to avoid spurious hot deploys (i.e. picking up a deployment even before it is completely copied to hot-deploy folder) for exploded deployments. How do we handle such cases for these zipped archives? i.e. What happens when a zipped foo.war is still being copied to hot-deploy and the scanner picks it up for deployment?

                 

                Brian Stansberry wrote:

                 

                For exploded archives:

                 

                User can trigger an undeploy of foo.war by deleting hot-deploy/foo.war/META-INF/jboss-deploy

                 

                Undeploy by deleting the entire hot-deploy/foo.war/ directory is not supported! The user should delete hot-deploy/foo.war/META-INF/jboss-deploy to trigger undeploy, after which they can delete the entire directory if they so choose.

                Any specific reason why deleting the hot-deploy/foo.war directory shouldn't undeploy the application? If foo.war is deleted it effectively means foo.war/META-INF/jboss.deploy too is deleted, so I'm not sure why we can't support it. I guess there's something I'm missing

                 

                 

                In addition to the above, I can think of a few more cases:

                 

                "Leaf" deployments:

                 

                What do we plan to do with "leaf" file hot-deployments? For example, consider a hot-deploy/mail-service.xml (I know for AS-7, *-service.xml may not be relevant, but let's consider it just as an example). How is hot deployment handled for such files? Or for that matter, how is deployment handled for such "leaf" files in the hot-deploy folder? Would it too create a mail-service.xml.deployed marker? Do we plan to (somehow) avoid spurious hot deploys for such files?

                 

                Nested deployments:

                 

                In case of nested deployments, I believe only the top level META-INF/jboss.deploy marker file plays a role. Any marker files in sub-deployments of the top-level deployment, will not have any effect on the hot-deploy semantics. Am I right?

                 

                 

                Finally, are the *.deployed and jboss.deploy marker file semantics only specific to a hot deployment enabled folder (like hot-deploy)? Considering a simple foo.war zipped archive in a normal deploy folder (without hot deployment), would it too be replaced with a marker foo.war.deployed file? And would jboss.deploy play a role in exploded deployment in a normal deploy folder?

                • 5. Re: Thoughts on hot deployment
                  jaikiran

                  jaikiran pai wrote:

                   


                  Brian Stansberry wrote:

                   

                  For zipped archives:

                   

                  Scanner will detect an archive in the hot-deploy dir, e.g. hot-deploy/foo.war. Scanner will use the same deployment API that an external client would use to register the content with the server and deploy it. Once the content is registered the server will maintain it's own copy in an internal directory (e.g. server/data/deployments). Scanner will replace the hot-deploy/foo.war file with a simple marker file hot-deploy/foo.war.deployed.

                  Would it be a "rename" or would the foo.war.deployed be a completely different file? From what I understand, I believe  it would be a completely different file without any .war semantics, but just wanted to be sure. Furthermore, where would be original foo.war be moved to? Are users expected to know where it's moved to? If not, then they probably will have to maintain a copy of their original deployment somewhere (I know most of them already do that, but again I just wanted to be clear about the expectations).

                   

                   

                  Just to be a little more clear about that question - I know that the server will internally maintain it's own copy in an internal directory (server/data/deployments). But the part which I wasn't sure was - would that internal copy in server/data/deployments be an extracted version (like we do in AS6 in server/tmp folder) or the original version? I assumed it would be an extracted and hence that question on locating the "original" deployment. (Note that in our current versions of AS even though we maintain an internal copy of the deployment in tmp folder, we don't change anything directly in the original copy, which means that the user still has access to the original unchanged deployment).

                  • 6. Re: Thoughts on hot deployment
                    dimitris

                    What is the rationale for the .deployed marker? It's really counter-intuitive. Just make a copy to avoid locking/classloading issues and leave the original in place.

                     

                    Also, for exploded deployments have a different META-INF/jboss-deploy marker to trigger the deployment? Counter-intuitive as well. The whole idea with the ./deploy dir (or whatever is called) is you look there and you know what gets deployed, you don't need to traverse directories to find out. Or unzip something and it gets deployed, no need for extra files.

                     

                    IMO the AS4 model of hot-deployment (no VFS) was the most practival and successful. People knew that to avoid the "early deployment" problem for large or nested deployments they just have to do an atomic filesystem mv. Through stuff in and it just gets deployed, no markers or any other type of magic. I don't see why we need to re-discover the wheel.

                    • 7. Re: Thoughts on hot deployment
                      aloubyansky

                      I am with Dimitris here.

                      What I see a deployment marker could do is be a visual indicator to the admin whether the given package is deployed (successfully or at all) or not while if you look just at the deploy dir in, e.g., AS5 you see only a list of deployments and you can't tell whether one of them failed w/o checking the logs or admin console.

                      And for exploaded deployments it solves the deploy-before-fully-copied issue.

                      But I would still prefer the traditional way if it were just for the advantages above.

                       

                      Plus, as Jaikiran says, if the deployments are really substituted, what if you are unsure and want to check the exact deployment descriptors and resources deployed? How would you do that?

                      • 8. Re: Thoughts on hot deployment
                        jason.greene

                        Ok so, I will first start off by saying that the file system is a broken API, for many reasons that I will get into, and that we basically have a bunch of trade-offs here. There is no perfect solution, but I think we should prefer one that is robust. Ultimately we really should push people to using the CLI because then we can get truly intuitive behavior.

                        Dimitris Andreadis wrote:

                         

                        What is the rationale for the .deployed marker? It's really counter-intuitive. Just make a copy to avoid locking/classloading issues and leave the original in place.

                         

                        The main reason was to support reusing information like indexes after a restart.

                         

                        Also, for exploded deployments have a different META-INF/jboss-deploy marker to trigger the deployment? Counter-intuitive as well. The whole idea with the ./deploy dir (or whatever is called) is you look there and you know what gets deployed, you don't need to traverse directories to find out. Or unzip something and it gets deployed, no need for extra files.

                         

                        IMO the AS4 model of hot-deployment (no VFS) was the most practival and successful. People knew that to avoid the "early deployment" problem for large or nested deployments they just have to do an atomic filesystem mv. Through stuff in and it just gets deployed, no markers or any other type of magic. I don't see why we need to re-discover the wheel.

                         

                        Pretty much exploded deployments have always been broken. The atomic move only solves half of the problem, the deploy part, and it requires that it be on the same partition of an OS that supports atomic moves. During undeploy though, removing the directory causes classloader operations (namely getResource) to fail. This lead to a hack on AS5 where it would detect if the deployment used seam and then copy the whole thing, and then try to monitor and copy changed files.

                         

                        The other issue is how you detect a redeploy operation. Historically this was touching the TLD, but since EE5 there is no gauranteed TLD (ejb deplyments could be pure annotations). Further modern frameworks use other descriptors besides the TLD which lead to AS5 detecting a change to ANY file and redeploying. This was a huge problem for our users, since a change doesn't necessarily mean you want a redeploy and we had to move back to the AS4 behavior which still doesnt fix the problem, since sometimes you have no way to cause a redeploy, and it also doesn't handle staged updates to files including the TLD. The marker though allows for someone to say I really actually want you to redeploy this.

                        • 9. Re: Thoughts on hot deployment
                          aloubyansky
                          Ultimately we really should push people to using the CLI because then we can get truly intuitive behavior.

                          Why not start with CLI then? And if needed add a deploy dir-based one later?

                          • 10. Re: Thoughts on hot deployment
                            jason.greene

                            jaikiran pai wrote:

                             

                            Nice!

                             

                            Here's some questions I have:

                            Brian Stansberry wrote:

                             

                            For zipped archives:

                             

                            Scanner will detect an archive in the hot-deploy dir, e.g. hot-deploy/foo.war. Scanner will use the same deployment API that an external client would use to register the content with the server and deploy it. Once the content is registered the server will maintain it's own copy in an internal directory (e.g. server/data/deployments). Scanner will replace the hot-deploy/foo.war file with a simple marker file hot-deploy/foo.war.deployed.

                            Would it be a "rename" or would the foo.war.deployed be a completely different file? From what I understand, I believe  it would be a completely different file without any .war semantics, but just wanted to be sure. Furthermore, where would be original foo.war be moved to? Are users expected to know where it's moved to? If not, then they probably will have to maintain a copy of their original deployment somewhere (I know most of them already do that, but again I just wanted to be clear about the expectations).

                             

                            Brian meant a different file, and they shouldn't necessarily be poking in the data area, which I admit is not intuitive. That's a very good point. It's also  inconsistent with the exploded dir, which preserves the original. We could alternatively leave the file there and just add the .deployed as an indicator that it was deployed. The only side effect is that it would require a copy vs a move, but IMO not a huge deal for zips.

                             

                            Brian Stansberry wrote:

                             

                            User can trigger a replacement of foo.war (undeploy old, deploy new) by copying a new foo.war into hot-deploy/

                            Later in your post, I see that special care has to taken to avoid spurious hot deploys (i.e. picking up a deployment even before it is completely copied to hot-deploy folder) for exploded deployments. How do we handle such cases for these zipped archives? i.e. What happens when a zipped foo.war is still being copied to hot-deploy and the scanner picks it up for deployment?

                            What I had in my head for that scenario, was that we would just detect an incomplete zip, and wait for the next few intervals before reporting the failure. It's pretty easy to do this because zip files store the directory at the end.

                             

                            Nested deployments:

                             

                            In case of nested deployments, I believe only the top level META-INF/jboss.deploy marker file plays a role. Any marker files in sub-deployments of the top-level deployment, will not have any effect on the hot-deploy semantics. Am I right?

                             

                             

                            Finally, are the *.deployed and jboss.deploy marker file semantics only specific to a hot deployment enabled folder (like hot-deploy)? Considering a simple foo.war zipped archive in a normal deploy folder (without hot deployment), would it too be replaced with a marker foo.war.deployed file? And would jboss.deploy play a role in exploded deployment in a normal deploy folder?

                             

                            Correct. We would basically treat the outer deployment as an atomic entity, and only look for the markers relevant to it.

                            • 11. Re: Thoughts on hot deployment
                              brian.stansberry

                              That's a fair question in terms of organization of work. I think hot deploy is on the agenda because:

                               

                              1) We have a mini-goal of recreating the functionality of the AS 6 'minimal' profile which includes hot deployment.

                               

                              2) When thinking about the deployment API and the implementation we need to think about hot deployment, to make sure we cover the requirements.

                               

                              #2 is IMHO much more important; mini-goals like #1 should shift to match whatever is most productive in getting to the overall M1 goals. But if once we feel comfortable we've adequately considered hot deployment, I have no problem focusing on CLI or programatic deployments.

                              • 12. Re: Thoughts on hot deployment
                                jason.greene

                                Alexey Loubyansky wrote:

                                 

                                Ultimately we really should push people to using the CLI because then we can get truly intuitive behavior.

                                Why not start with CLI then? And if needed add a deploy dir-based one later?

                                 

                                That has always been my preference. However, many think we should preserve it, and I am certainly sympathetic to it. I guess really what we need to figure out is can we find the magic balance of trade-offs that makes it intuitive enough, but still not cause application crashes.

                                 

                                Note that for reference Glassfish uses a similar marker file approach, and I haven't heard any major complaints from it. One thing we could also do is ship a HOWTO file in that directory that tells you the semantics.

                                • 13. Re: Thoughts on hot deployment
                                  aloubyansky

                                  Actually, be it CLI-based, prorgammatic or deploy dir - it's all hot deploy, right? It's just CLI is not the original JBoss deploy dir style hot deploy. Hot deploy means being able to deploy w/o server restart.

                                  • 14. Re: Thoughts on hot deployment
                                    aloubyansky

                                    My opinion is if CLI is what we will have to add sooner or later anyway and it might be a more intuitive way than the deploy dir then I would start with CLI.

                                    1 2 3 Previous Next