1 2 Previous Next 17 Replies Latest reply on Mar 24, 2012 11:20 PM by adamw

    Auditing with hierarchies #3 :)

    ahoehma

      User have n Contacts. A Contact can have a localized Comment (Comments are shared between Contacts). Java Beans:

      @Audited
      @Entity
      public class User {
         
      @OneToMany(fetch = FetchType.EAGER,
                     cascade
      = CascadeType.ALL,
                     orphanRemoval
      = true)
         
      Set<Contact> contacts;
      }

      @Audited
      @Entity
      public class Contact {
         
      @ManyToOne(fetch = FetchType.EAGER,
                     cascade
      = {
                               
      CascadeType.MERGE,
                               
      CascadeType.PERSIST,
                               
      CascadeType.REFRESH})
         
      Comment comment;
      }

      @Audited
      @Entity
      public class Comment {
         
      String de;
         
      String en;
         
      String fr;
      }

      If I change the german localization (Comment.de) of a contact (Contact.comment) then this will create a new revision but not for User. If I ask envers for User Revisions I will never see this "Level 2 change" because the relation between User and Contact was not change, only the german string in the Contact Comment was changed.

      But I want see in the User History a new Entry (Changed german comment for contact XYZ).

      How can I do this?

      http://stackoverflow.com/questions/9033275/envers-multi-level-entity-revision-howto

        • 1. Re: Auditing with hierarchies #3 :)
          vyacheslav86

          I have just the same question. For @oneToMany-@ManyToOne relationship owning side of relationship is the side with annotation @oneToMany and for @OneToOne owning side is that which has no mappedBy. So if i change owned entity I want this to count as change for ownign entity and owning of owning entity if present and so on. It would be cool if envers triggered owning entity change even if save method does not do so, for example:

           

          @Entity

          @Table(name = "PROPERTIES")

          @Inheritance(strategy=InheritanceType.SINGLE_TABLE)

          @DiscriminatorColumn(name="BASE_TYPE_ID", discriminatorType=DiscriminatorType.INTEGER)

          @Audited

          public class Property implements Auditable, Serializable {

          ...

              @OneToMany(fetch = FetchType.LAZY, mappedBy = "property", cascade = { CascadeType.ALL }, orphanRemoval = true)

              private List<PropertyFileLink> fileLinks = new ArrayList<PropertyFileLink>(0);

          ...

              @OneToOne(fetch=FetchType.LAZY, cascade = { CascadeType.ALL }, orphanRemoval = true)

              @JoinColumn(name="PROPERTY_SAFETY_ID")

              private PropertySafety propertySafety;

          ...

          }

           

          and this entities created and saved

          ...

          public PropertyFileLink addFile(Property property, FileType fileType, String desc, MultipartFile uploadedData) {

          ...

                  property.getFileLinks().add(fileLink); //even if i modify collection

                  persistenceManager.saveOrUpdate(fileLink); // but save element of collection.

          ...

          }

           

          only PropertyFileLink is present in revision created.

          Hibernate: insert into REVISIONS (TIMESTAMP, USERNAME, id) values (?, ?, ?)

          Hibernate: insert into PROPERTY_FILE_LINKS_AUD (REVTYPE, ID_OLD, FILE_TYPE_ID, ZONE_ID, FILE_ID, PROPERTY_ID, REV) values (?, ?, ?, ?, ?, ?, ?)

           

          as far as I see i can modify save methods to save object having collection and receive audit update for owning side of relationship(not tested yet), but the thing is i'm not always using it and such functionality should be implemented by envers event listener.

           

          Another interesting case is when many owning entities present in the hierachy - owned by many relationship. Oh gosh, at least dictionaries in my domain aren't audited. Imagine how much updates that would cost.

          • 2. Re: Auditing with hierarchies #3 :)
            adamw

            There's no good answer to this yet. And I don't know if there will be a generic solution. What are your "entity roots" is really domain-specific and hard to reason from code.

             

            Adam

            • 3. Re: Auditing with hierarchies #3 :)
              vyacheslav86

              Well speaking of "entity root" it is about owning-owner side of the realtion: we can guess that "user has contacts"/"user owns contacts". For comment it is harder to guess because comment is "owned by many" here . We can guess that from cascading information but there is no direct specification of such relation in hibernate mapping . Especially if reverse mapping is absert as in Andreas example mapping absent:

               

              @Audited
              @Entity
              public class Contact {
                 
              @ManyToOne(fetch = FetchType.EAGER,
                             cascade
              = {
                                       
              CascadeType.MERGE,
                                       
              CascadeType.PERSIST,
                                       
              CascadeType.REFRESH})
                 
              Comment comment;

                  @ManyToOne ...User ownerUser;//absent
              }

               

              also it seems that "Set<Context> contacts" is a typo for

              Set<Contact> contacts

               

              Entity root for Andreas example is User class.

              And for example for all social network sites User is also root-entity to cluster info by user.

               

              this class

              @Audited
              @Entity
              public class Comment {
                 
              String de;
                 
              String en;
                 
              String fr;
              }


              in his case is what i've called dictionary in my project: entity owned by many.

               

              I've called it "root entity" because for graph of objects serialization we need to choose root entity in the graph and build covering tree. In my project such serialization is needed for entity rendering to Json. And each root entity has a separate tab in a view.

               

              I'm implementing this functionality now but the only thing i'm worry about is perfomance and space required for dictionary update because it can result in "number of entities"/"number of dictionary enties" inserts in AUD tables, 100-10000. This definitely requires optimization in the future.


              • 4. Re: Auditing with hierarchies #3 :)
                adamw

                Exactly.

                 

                Determining what's the root and what's the child may be hard just from the JPA annotations. Maybe a separate set of annotations could help here.

                 

                And anyway, just storing parent audit whenever a child changes isn't good also - for storage reasons. You're just storing then lots of audit rows which have no changes. So another mechanism would be preferred, but what kind of mechanism, I don't yet know

                 

                Adam

                • 5. Re: Auditing with hierarchies #3 :)
                  ahoehma

                  Hi Adam,

                   

                  like the annotation approach.

                  @Audited
                  @Entity
                  public class User {
                     
                  @OneToMany
                     
                  Set<Contact> contacts;
                  }

                  @Audited(auditRoots={User.class})
                  @Entity
                  public class Contact {
                     
                  @ManyToOne
                     
                  Comment comment;
                  }

                  @Audited(auditRoots={User.class})
                  @Entity

                  public class Comment {
                     
                  String de;
                     
                  String en;
                     
                  String fr;
                  }

                   

                  My idea to minimize the space is to "store the affected root entities (if any) per revision":

                   

                       - envers can use this information to return "virtual" revisions for audited root enties if the user query for revisions of such a root entity

                       - a virtual revision have no direct change, only a "implizit" change: some sub entity changed a property

                       - so the last stored revision is the base for such a virtual revision - no extra space for unchanged properties

                       - at revision store time envers must analyze the entity tree, search changed entities, check for "auditRoots" and store those classes into revision-info table

                       - at revision query time envers can use the stored auditedRoots (if any) per revision to return the last know state of these entities

                   

                  I would be nice to have an feature like svn have ... a revision for each db change (some include/excludes are usefull to minimize the audit overhead)

                   

                  Regards

                  Andreas

                  • 6. Re: Auditing with hierarchies #3 :)
                    adamw

                    Ah, hmm, maybe a new revision type (CASCADE?), which would be used to store revisions of parent entities, with all fields null, to save disk space? Such revisions would have to be excluded when searching for data of a revision, but that should be possible, I think.

                     

                    Adam

                    • 7. Re: Auditing with hierarchies #3 :)
                      vyacheslav86

                      I've discovered that i need to CASCADE changes not only to parent entity but some other Hibernate Relations named in my model as "link" to both sides, so nobody ownes nobody but change is cascaded. So a change to a single entity resolves to change to subgraph of model entities.

                       

                      I doubt that nulling fields will save disk space, because it depends on inner implementation of db.

                       

                      Ability to search for only not cascaded changes is good to have.

                       

                      The only problem which becomes more important is that bottleneck related to number of inserts into ENTITY_CHANGED_IN_REVISION(analog to new envers table but with ids saved) table multiplies by 5-10 because of cascading change to subgraph.

                       

                      and also total audit overhead of modification of one entity in db rises up to 5-10 inserts in audit tables.

                      • 8. Re: Auditing with hierarchies #3 :)
                        adamw

                        Well, usually nulls are compacted and not saved. If the cascading would go deep, I think this can have a significant impact.

                         

                        I agree that the overhead could get large, but how to else implement this? (efficiently )

                         

                        Adam

                        • 9. Re: Auditing with hierarchies #3 :)
                          marx3

                          I have similair problem. Revision (which equals to transaction) has a flat structure (as I understand I can commit a few changes in one revision). But I need to present audited information in some high level report, and so I need some hierarchy of changes - revision is not enough.

                           

                          What I need to report is more application specific, one "ok/save" button click in application produces a few changes in a few transactions, and to show it correctly I need to create hierarchy of changes. Let's imagine application of user rights management - in one step I can create user, his description (many phones, many addresses) and give hime a few roles. It's one "save" button in application, hovewer it creates changes in 4 or more tables. Even if I manage to save changes in one transaction (which isn't always possible) I don't have hierarchy. I know hierarchy in the moment of saving, but how to save it?

                           

                          The simpliest would be adding custiom field "parent_revision" to CustomRevisionEntity and if null - it's root. Hovewer revision numbers are uknown in the moment of save, so how to fill them?

                           

                          And about virtual revisions, empty records seems logical and simple buit not quite beautiful. Maybe better would be additional table with virtual revisions pointing to real revisions, so one real revision could have many virtual revisions (and we doesn't need filtering, and doesn't create empty records)? And it should be driven by annotations. Similiar annotation @ContainedIn exists in Hibernate Search and is used to reindexing parent objects after change of his children.

                          • 10. Re: Auditing with hierarchies #3 :)
                            adamw

                            I think you are describing an entirerly different problem, one which I don't quite understand. What would such revision hierarchies do?

                             

                            Adam

                            • 11. Re: Auditing with hierarchies #3 :)
                              vyacheslav86

                              Adam Warski wrote:

                               

                              Well, usually nulls are compacted and not saved. If the cascading would go deep, I think this can have a significant impact.

                               

                              I agree that the overhead could get large, but how to else implement this? (efficiently )

                               

                              Adam

                              I've thought of that issue. First of all the information in db stored by current envers release is quite enough to load db state at any revision. But for entities with hierachy there can be space-time tradoff for query implementation:

                              1) current envers query example:

                              Property owns PropertyFileLinks

                              Property has revision entires with revision number 1,5.

                              PropertyFileLinks has revision numbers 1,2,3,4.

                              To query graph of entities for revision we must find maximal graph entities revision which is 1<=revisionNumber<5. which is 4 and query for that revision.

                              2) create revision entries without data fields changed for all changes of childs, so for example we have

                              Property at 1,5 revisions.

                              PropertyFileLinks has revision numbers 1,2,4.

                              and another childs PropertySafety has 1,3,4 revisions.

                              So we add entries at creation time of revisions 2,3,4 for Property and as a result we have 1,2,3,4,5 revisions for Property, lowering required quiries number to get 4 revision of the whole hierachy by spending more time and space inserting revisions for property.

                              3) we can use both ways of saving data for revision information storage: for example child entities updates create root entity updates, but root entities updates don't create child entities updates, so queries from root to child are faster that queries from child to root, because there is no need to search for revision number in the first case.

                               

                              4) and for hierachy of entities quering the bottleneck in this operation is definitely a number of id lookup queries and network access time, which is why i'm thinking of implementing stored procedure to do revision info update.

                              • 12. Re: Auditing with hierarchies #3 :)
                                vyacheslav86

                                Marx3 wrote:

                                 

                                I have similair problem. Revision (which equals to transaction) has a flat structure (as I understand I can commit a few changes in one revision). But I need to present audited information in some high level report, and so I need some hierarchy of changes - revision is not enough.

                                 

                                What I need to report is more application specific, one "ok/save" button click in application produces a few changes in a few transactions, and to show it correctly I need to create hierarchy of changes. Let's imagine application of user rights management - in one step I can create user, his description (many phones, many addresses) and give hime a few roles. It's one "save" button in application, hovewer it creates changes in 4 or more tables. Even if I manage to save changes in one transaction (which isn't always possible) I don't have hierarchy. I know hierarchy in the moment of saving, but how to save it?

                                 

                                The simpliest would be adding custiom field "parent_revision" to CustomRevisionEntity and if null - it's root. Hovewer revision numbers are uknown in the moment of save,(RevisionListener passes revisionEntity as parameter of void newRevision(Object revisionEntity); method, revisionEntity has revisionId.

                                so how to fill them?

                                 

                                And about virtual revisions, empty records seems logical and simple buit not quite beautiful. Maybe better would be additional table with virtual revisions pointing to real revisions, so one real revision could have many virtual revisions (and we doesn't need filtering, and doesn't create empty records)? And it should be driven by annotations. Similiar annotation @ContainedIn exists in Hibernate Search and is used to reindexing parent objects after change of his children.

                                Do i understand correctly that you want to introduce happens-before relationship between change of entity data and entity relations even if the occur in one transaction?

                                If so you can always think of such changes as happening in a sequence of objects graph traversing order specified by you(for example with annotations) for classes structure and build that changes sequence in app-server after all entities at revision returned by query.

                                I mean if changes occur in one transaction there could be any order of updates possible inside transaction. Why would you need to save that exact order? If so i suggest to use Spring Transaction Propagation REQUIRES_NEW or NESTED for saving hierachy of data and introduce counter(virtual revision) in saving procedure or just create new revision for every update.

                                • 13. Re: Auditing with hierarchies #3 :)
                                  marx3

                                  I'm sorry for the mess I did here. We have two different problem. One is how to mark/find virtual change of object if one of his properties (children) in other datatable (and lower, so it creates hierarchy) changes and I find this original problem similair to @ContainedIn annotation in Hibernate Search.

                                   

                                  The second problem is mine and yes, it shouldn't depend on scope of transaction, because it's rather logical hierarchy based on application functionality. I need it for reporting. Report based on auditing should show who when and what has modified, and it should be shown in high level way. For example: Administrator has added user and has given him rights for different places of application, and a few roles. This operation is one "save" in application, so it's shown as one row in report, hovewer it's hierarchical. So this row is divided into two sub-operation: creation of user and his attributes, and adding him rights and roles. It creates hierarchy - you can't add roles whithout adding first a user, so it creates hierarchy of sub-operations.

                                  It's rather simple example, but of course there are much more complicated issues

                                   

                                  I'm interested in both problems of course.

                                  • 14. Re: Auditing with hierarchies #3 :)
                                    vyacheslav86

                                    Well, both of them aren't supported by envers. First one i'm solving now in my project and is discussed in this thread. The second one is related to auditing and separating business operations, not db operations. You can create separate revision for each business operation by envers, to later build the hierachy of business operations from db data.

                                    1 2 Previous Next