1 2 Previous Next 20 Replies Latest reply on Dec 3, 2010 9:50 AM by burmanm

    JCRMessageStoreImpl store only message body?

    maeste

      Hi folks,
      I'm taking a look to JCRMessageStoreImpl class, because I'm evalueting to write a JBossDNA's sequencers on message stored in JCR repository by esb.
      I noticed current implementation only store Message Body and properties in JSR-170 repository. Are there some reason for that?
      I would need all information available about message (Header, context, fault, attachment) to extract and correlate any information useful from a SOA governance point of view. For example EPRs in Header would be very useful, isn't it?

        • 1. Re: JCRMessageStoreImpl store only message body?
          burrsutter

          Are you looking to place all "in-flight" messages of the ESB into the DNA repo?

          And I agree, if you wish to store the messages then you would need to store the whole Message. This implementation originally came from another external contributor and I suspect their needs were only for the Body.


          • 2. Re: JCRMessageStoreImpl store only message body?
            kconner

            Yes, storing the complete message makes sense.

            Burr is correct, this code was contributed by an external contributor and would have been done to fit their requirements. We should revisit this code.

            • 3. Re: JCRMessageStoreImpl store only message body?
              maeste

               

              "burrsutter" wrote:
              Are you looking to place all "in-flight" messages of the ESB into the DNA repo?

              Good question...we are discussing about. It would be a lot of data. Isn't it?
              And what would make sense to you? I'm thinking to store it for a limited period of time, configurable of course.

              Well, this and other question (like which kind of metadata we should sequence) would be discussed in a dedicated 3ad on DNA or overlord forums. Or would you think is it fine here?





              • 4. Re: JCRMessageStoreImpl store only message body?
                maeste

                 

                "Kevin.Conner@jboss.com" wrote:
                Yes, storing the complete message makes sense.

                Burr is correct, this code was contributed by an external contributor and would have been done to fit their requirements. We should revisit this code.

                Ok, I can create an issue on Jira and take it, if you agree.

                My opinion is the best solution would be to store in ESB action whole message, without any manipulation (no extraction of properties, header and so on) and leave to DNA sequencer this kind of job.
                This approach would permit us to apply different sequencer to ESB message repository to extract metadata at different levels, depending on actually needing.
                Am I wrong?

                • 5. Re: JCRMessageStoreImpl store only message body?
                  marklittle

                  We should definitely be storing the entire message. I'd suggest adding some option (maybe on the message) that indicates the duration for which it can be stored and, when it gets deleted by the store, we should have a "grave stone" (marker) for it so that we don't break referential integrity.

                  If you delve back far enough in the design forums you'll find some of the original design ideas behind the message store, not all of which made it into the implementation. One of them was to store the entire message and have a generic search method that allowed you to send arbitrary rules in to the "engine" and get back a stream of results, c.f. dynamic SQL procedures. That way the store API isn't tied to a static and limited set of search criteria.

                  • 6. Re: JCRMessageStoreImpl store only message body?
                    maeste

                     

                    "mark.little@jboss.com" wrote:
                    We should definitely be storing the entire message. I'd suggest adding some option (maybe on the message) that indicates the duration for which it can be stored and, when it gets deleted by the store, we should have a "grave stone" (marker) for it so that we don't break referential integrity.


                    Is really it a responsibility you would leave totally in charge to message? Isn't it a governance responsibility?
                    I would prefer message is responsible to classify itself in a more abstract manner (maybe some properties?), and leave to repository the decision how longer messages have to be stored.
                    DNA could in fact classify messages, on properties base, and (maybe) refine this classification using metadata extracted by sequencer. This way drive to put configuration about repository at a different (higher) tier, where the decision of how long we keep them could depend also on which kind of use we would do of them (of course different kinds of analysis need different store duration).
                    Of course it's just my humble opinion.

                    • 7. Re: JCRMessageStoreImpl store only message body?
                      marklittle

                       

                      "maeste" wrote:
                      "mark.little@jboss.com" wrote:
                      We should definitely be storing the entire message. I'd suggest adding some option (maybe on the message) that indicates the duration for which it can be stored and, when it gets deleted by the store, we should have a "grave stone" (marker) for it so that we don't break referential integrity.


                      Is really it a responsibility you would leave totally in charge to message?
                      Isn't it a governance responsibility?


                      It's both. There will be some messages that users will refuse (for legal reasons) to allow to be stored. But until we have hooks for governance ...


                      I would prefer message is responsible to classify itself in a more abstract manner (maybe some properties?), and leave to repository the decision how longer messages have to be stored.


                      ok


                      DNA could in fact classify messages, on properties base, and (maybe) refine this classification using metadata extracted by sequencer. This way drive to put configuration about repository at a different (higher) tier, where the decision of how long we keep them could depend also on which kind of use we would do of them (of course different kinds of analysis need different store duration).
                      Of course it's just my humble opinion.


                      We definitely need some way of defining this as a policy across different "types" of messages, users and scenarios. Plus the capability to say "do not store" (I think we may have that already).

                      • 8. Re: JCRMessageStoreImpl store only message body?
                        maeste

                         

                        "mark.little@jboss.com" wrote:

                        We definitely need some way of defining this as a policy across different "types" of messages, users and scenarios. Plus the capability to say "do not store" (I think we may have that already).

                        Ok, and do you agree with me that we need to define a policy set limiting capabilities of message to force repository behavior for them selves? IOW I'm thinking to some policy enforcement refusing messages which haven't some well defined characteristic (i.e. storable, and type is for store duration > 1 h ecc).
                        Hmm..isn't it SLA policies (at lest one aspect of them) we have discussing in Overlord project?
                        Well I'll post my idea about them in Overlord's forum soon....

                        • 9. Re: JCRMessageStoreImpl store only message body?
                          michaelneale

                          Hmm... I would think JCR maybe not the right persistence mechanism for large amounts of transactional data - it certainly is suited for heterogenous message payloads, but it is generally more about content then billions of little messages.

                          Of course, with DNA implemeneting JCR, it can have "fast" modes where the versioning features and so on do not get in the way so it can cope with a deluge of data, but most other implementations I have seen would not cope that well with the number of messages. They cope well with large messages, however.

                          Just my 2c.

                          • 10. Re: JCRMessageStoreImpl store only message body?
                            maeste

                             

                            "michael.neale@jboss.com" wrote:
                            Hmm... I would think JCR maybe not the right persistence mechanism for large amounts of transactional data - it certainly is suited for heterogenous message payloads, but it is generally more about content then billions of little messages.

                            Ok, and what else do you have in mind? Relational DB? HDFS?
                            We can evaluate to leave data in these repository, write a DNA connector and view them with federation engine "as if it's in JCR".
                            For sure I'll make a first (easier) implementation of JCR message store, and I'll focus on sequencer, but other solution for storing and then federate them can be evaluated in next future.
                            So, what do you think is the best store for ESB message?
                            My 2c:
                            1) JCR could have limitation with large amounts of messages, not with big messages, it would be quite easy to configure and deploy. Moreover with DNA a 100% JBoss solution could be implemented ;)
                            2) Relational DB. No problem with large number of messages, maybe some problem with very very large messages. It is a well known technology, probably customers have at least one installation in their server farms. Heterogeneity of data may be an issue, even if we consider DNA federation the only (or at least the main) solution to intercat with data it would be a secondary problem (IOW brute storing data and leaving DNA the responsibility to make them able to be queried).
                            3) HDFS. A lot of benefits storing data (as far as I understood it, studing it too fewer to feel me confident): distributed, high available, fast(?), no problem for tons of file, no problem with huge files. What are the minus in my opinion? Not so easy to install and maintain (from customer point of view), another API to implement on both side (ESB and DNA federation). May be the best solution at long term, would be not so easy as starting point, it should be on our road map, but IMHO not at short terms.

                            But one more time my question is: does it make some sense to keep this large amount of data for a long time for all? I think, the best approach, would be to have more than one repository available for different purpose (and I think ESB already have both solution 1 and 2, even if they can be revisited as I'm saying in this 3ad).
                            A JCR implementation keeping data for a relative short period of time with governance goals in mind would be an useful kick start.



                            • 11. Re: JCRMessageStoreImpl store only message body?
                              marklittle

                              The message store should be implementation agnostic. It certainly started out that way. JCR is good as a specific type of implementation, but that should not be forced through to the front-end service interface.

                              • 12. Re: JCRMessageStoreImpl store only message body?
                                maeste

                                 

                                "mark.little@jboss.com" wrote:
                                The message store should be implementation agnostic. It certainly started out that way. JCR is good as a specific type of implementation, but that should not be forced through to the front-end service interface.

                                Agree 100%. I thought to don't change interface, but only the current JCR implementation. Are you saying it would be impossible, and I have to take care also to review the front-end interface (keeping of course in mind it have to be implementation agnostic)?

                                • 13. Re: JCRMessageStoreImpl store only message body?
                                  marklittle

                                  No, I'm just saying that we need to be careful.

                                  • 14. Re: JCRMessageStoreImpl store only message body?
                                    maeste

                                    Ok,
                                    just created http://jira.jboss.org/jira/browse/JBESB-1769
                                    If Kevin agree I can take care of this for 4.4 release.

                                    1 2 Previous Next