8 Replies Latest reply on Mar 4, 2011 4:43 PM by nfilotto

    Clustering design wrt Indexing

    acoliver

      Hi All,

       

      I was wondering what the thought was on the current design of the JCR index and clustering was.  The topology of one writer and multiple readers on a shared GFS is not a very standard Lucene topology and I suspect performance is suboptimal.  More nominal are each node having a separate copy of the index and writes being passed via messaging and/or replicated via rsync.  Are there plans to support a more standard lucene topology?  If not, why not?  If so, when?

       

      Thanks,

       

      Andy

        • 1. Clustering design wrt Indexing
          theute

          Andy ! You are everywhere !

           

          To answer your question I agree that this would be a good configurable alternative to have.

           

          It depends if you care about "consistency" vs "eventual consitency", most of the time eventual consitency should be enough for searching but so far eXo JCR has been focused on being to find documents immediately after they are created. Getting rid of shared network requirement would definitely remove one hurdle for clustering setup.

           

          I didn't hear such a plan in eXo JCR roadmap and we haven't pushed for this feature, but contributions are always welcome And a JIRA in https://issues.jboss.org/browse/EXOJCR would be a good start.

          • 2. Clustering design wrt Indexing
            acoliver

            This surprises you? 

             

            With a transactional locking you could have actual consistancy or darn near close. 

             

            The shared FS bit doesn't scale to the same scale as other people using Lucene who have similar requirements:

             

            http://2010.lucenerevolution.org/slides/Lucene%20Rev%20Preso%20Wang%20LinkedIn%20Search.pdf

             

            while Salesforce does it more like Exo's setup, it does it with I imagine a lot more beef per system than is common and a lot of segmentation "pod"s:

             

            http://www.lucidimagination.com/files/LuceneRev_BPress_SalesForceCloudSearch.pdf

             

            I'll post the feature request.  I'll note your response to the team I'm working with.

            • 3. Clustering design wrt Indexing
              acoliver
              • 4. Clustering design wrt Indexing
                nfilotto

                Hi Andrew,

                 

                Thank for sharing ideas, it is always priceless especially when it comes from the community, I will read carefully all your documents and JIRAs. FYI we are aware of the limitations of our clustering design as you can see here https://issues.jboss.org/browse/EXOJCR-1080 and we work hard on this task for several months.

                 

                The main reason why we decided to use a shared file system instead of having dedicated indexes for each node, is because we wanted to be able to add new nodes to the cluster dynamically which is not really easy when your new node doesn't have its own copy of indexes. And the reason why we have one node in RW mode (the coordinator) and the other nodes in RO mode, it is mainly to avoid relying on lock mechanism proposed by shared file system which limits the risk of corrupting the indexes and to keep a consistent state of the indexes.

                 

                In JCR 1.14.0 thanks to EXOJCR-1080, we will allow all nodes to have their own copy of indexes since we improved the indexes recovery which is mainly needed when you add a new node to the cluster or a node has left the cluster during a certain period of time so its indexes are not up to date anymore and should be recreated. So now you have the choice between two options: either the node re-indexes the database if it the db is not too big of course (which can be done in a reasonable period of time since we improved the indexing a lot, as you can see here https://issues.jboss.org/browse/EXOJCR-1104, it is now multi-threaded and optimized for RDBMS) or it gets the indexes content from the coordinator.

                 

                I hope it helps,

                BR,

                Nicolas

                • 5. Re: Clustering design wrt Indexing
                  acoliver

                  apparently the EXOJCR-1080 JIRAs you refer to is private.  I do not have permission to view it.  So I cannot see it :-)

                  • 6. Clustering design wrt Indexing
                    nfilotto

                    Indeed the access was limited to eXo and JBoss employees, it should be plubic now

                    • 7. Re: Clustering design wrt Indexing
                      acoliver

                      How close is this to being ready?  Is it possible to use a build of this JCR with the released EPP 5.1?

                       

                      THANKS!

                       

                      -Andy

                      • 8. Clustering design wrt Indexing
                        nfilotto

                        Yes very close, 1.14.0-CR1 is expected before the end of the next week.