1 2 Previous Next 25 Replies Latest reply on Feb 24, 2012 10:56 AM by sannegrinovero Go to original post
      • 15. Re: Expiration and Lucene Directory
        sannegrinovero

        Hi Horacio,

        I've run some tests on the versions you mentioned, and am going to add some automated to trunk soon to cover for such situations in future, but so far I've seen no issue.

        I'm testing that removing from or resetting the index correctly resets the index state, or shrinks it.

         

        I think this issue might be related to LUCENE-3147 : if Lucene fails to close a segment or chunk reference, we will consider it still in use, and so we'll prevent it from being deleted as some open Searcher instances might need the next chunk. Could you try keeping the same version of Hibernate Search but update Lucene to at least 3.2 ?

         

        Luckily while Lucene changes API very frequently, 3.2 and 3.1 are both compatible with Search 3.4.1

        1 of 1 people found this helpful
        • 16. Re: Expiration and Lucene Directory
          hvico

          I'll try that tomorrow, that upgrade should be simple!

           

          Thanks,

          • 17. Re: Expiration and Lucene Directory
            hvico

            I'm upgrading Lucene rigth now.

             

            About LUCENE-3147 I understand it only affects a test component "MockDirectoryWrapper", doesn't it?

            • 18. Re: Expiration and Lucene Directory
              sannegrinovero

              That's correct, but while improving the test utility as you can see in the comments they have found several unclosed file handles, and fixed them as part of the same issue.

              • 19. Re: Expiration and Lucene Directory
                hvico

                Sanne,

                 

                While I was upgrading Lucene I've found an improperly closed IndexReader in the application code. Maybe that has something to do with those file handles. I'll fix that and test my application again.

                • 20. Re: Expiration and Lucene Directory
                  sannegrinovero

                  great, that would explain a lot

                   

                  I hope it's clear on why it avoids deleting index chunks while they are open; this would not be required if you could live without chunking: I'm considering for the next release of Hibernate Search to configure (by default) the IndexWriter to not create big segments anyway, and make sure the chunking threshold is higher than the expected segment size. One should still avoid defining a too high chunksize buffer as it also affects how big the buffers are allocated internally.

                  • 21. Re: Expiration and Lucene Directory
                    hvico

                     

                     

                    Those prx "files" are still there in my cache after fixing that IndexReader leak.

                     

                    I've just finished the Lucene upgrade, now I am running my app with Lucene 3.2.0.

                     

                    Tomorrow will see how it works.

                    • 22. Re: Expiration and Lucene Directory
                      sannegrinovero

                      Also check to make sure you have no other IndexReader leaks, make sure you always close them in a finally block right after opening them.

                       

                      In worst case, we can see how you could reset the index state periodically, if that's an option.

                      • 23. Re: Expiration and Lucene Directory
                        hvico

                        Unfortunatelly the cachestore row count keeps growing after the upgrade to 3.2.0.

                         

                         

                        When I run an index optimization the size grows even more!

                         

                         

                        Is there a way to find out what could be locking the indexes? Or maybe a way to force the release of the locks and optimize after that.

                         

                         

                        Thanks,

                        • 24. Re: Expiration and Lucene Directory
                          hvico

                          After some days trying to fix this without any success, I've found that disabling all the Hibernate/Lucene automatic optimization prevents the crazy index growth.

                           

                          I removed this entries from my persistence.xml file:

                           

                          hibernate.search.default.optimizer.operation_limit.max = 1000

                          hibernate.search.default.optimizer.transaction_limit.max = 100

                           

                          Now my index grows slowly, as it should.

                           

                          Maybe I'll have performance issues later because of the lack of index optimization, and would be forced to do a full reindex from time to time.

                          • 25. Re: Expiration and Lucene Directory
                            sannegrinovero

                            Hi Horacio,

                            thanks for coming back with this information. It seems likely there is a file leak triggered by the IndexWriter optimize() .. you might want to look there or at least let the Lucene mailing list know about your findings.

                             

                            Your workaround looks fine: optimizing an interactive index (one on which the application writes frequently) is quite pointless and Lucene is pretty good nowadays even on non-optimized indexes making it less of a requirement to actually do optimization. It's entirely possible that for your use case it was a better option anyway to disable this.

                            1 2 Previous Next