1 2 Previous Next 26 Replies Latest reply on Sep 14, 2011 12:36 PM by penkween

    Querying Repository (Filesystem Connector) return only Root node

    penkween

      Hi,

       

             I am using Modeshape-2.6.0-b2 on a File System Repository which having multiple level of folder (nt:folder) an files (nt:file) and all these nodes can indeed be dump out correctly using getNode().

       

            Currently I am testing "Query" using JCR-SQL2 and  I am having problem with query like "SELECT * FROM [nt:base]" where it doesn't return all the nodes and query like "SELECT * FROM [nt:file]" even return empty ResultSet . Belows showing the codes and errors:

       

      Thanks a lot for your help.

       

       

      Note:

      - This output behavior is identical if I use xpath

       

       

       

      Codes

      =============================================================================================

      InitialContext initCtx = new InitialContext();

      Context envCtx = (Context) initCtx.lookup("java:comp/env");

      Repository repository = (Repository) envCtx.lookup("jcr/local");

      System.out.println("Repository = " + repository);   //For Debug

       

      Session session = repository.login(new ServletCredentials(request)); //This has indeed login OK since I can dump node info

       

      QueryManager queryManager = session.getWorkspace().getQueryManager();

      String language = null;

      String expression = null;

      Query query = null;

      QueryResult result = null;

       


      language = "JCR-SQL2";

      expression = "SELECT * FROM [nt:base]";

      query = queryManager.createQuery(expression, language);

      result = query.execute();

      System.out.println("########################################################################################################");

      System.out.println("Language = " + language + "\tExpression = " + expression);

      System.out.println("########################################################################################################");

      System.out.println(result.toString());

       

       


      language = "JCR-SQL2";

      expression = "SELECT * FROM [nt:file]";

      query = queryManager.createQuery(expression, language);

      result = query.execute();

      System.out.println("########################################################################################################");

      System.out.println("Language = " + language + "\tExpression = " + expression);

      System.out.println("########################################################################################################");

      System.out.println(result.toString());

       

      session.logout();

       

       

      Errors

      =============================================================================================

      INFO: JcrEngine starting...

      INFO: JcrEngine started in 2768 ms

      INFO: Completed starting the "MyFileSystemRepository" repository

      INFO: Repository = org.modeshape.jcr.JcrRepository@20d219

      INFO: ########################################################################################################

      INFO: Language = JCR-SQL2    Expression = SELECT * FROM [nt:base]

      INFO: ########################################################################################################

      INFO: +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

      | # | jcr:primaryType | jcr:path | jcr:name | jcr:score | mode:localName | mode:depth | Location(nt:base)                                                                  | Score(nt:base) |

      +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

      | 1 | mode:root       | /        |          | 1.0       |                | 0          | </ && [{http://www.modeshape.org/1.0}uuid = fd129c12-81a8-42ed-aa4b-820dba49e6f1]> | 1.0            |

      +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

      INFO: ########################################################################################################

      INFO: Language = JCR-SQL2    Expression = SELECT * FROM [nt:file]

      INFO: ########################################################################################################

      INFO: +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

      | # | jcr:primaryType | jcr:created | jcr:createdBy | jcr:path | jcr:name | jcr:score | mode:localName | mode:depth | Location(nt:file) | Score(nt:file) |

      +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

      +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

        • 1. Re: Querying Repository (File system) return unexpected results
          penkween

          I think this issue might be similar and related to:

           

          http://community.jboss.org/message/615297#615297

          https://issues.jboss.org/browse/MODE-1221

           

          Perhaps I need to download the latest source and rebuild Modeshape binary.

          • 2. Re: Querying Repository (File system) return unexpected results
            rhauch

            It's hard to tell whether that situation is the same. That particular issue was marked as not reproducible, since our investigation couldn't replicate the behavior. Maybe it is still an issue and we just need to figure out how to replicate it.

             

            Where do you have ModeShape deployed? In an app server, or is this a standalone application? Can you post your ModeShape configuration?

            • 3. Re: Querying Repository (File system) return unexpected results
              penkween

              Hi Randall,

               

              The ModeShape (2.6.0-b2) is currently deployed at a Glassfish 3.1.1 App Server for testing.

               

              configRepository.xml

              ==========================================================================================================

              <?xml version="1.0" encoding="UTF-8"?>

              <configuration xmlns:mode="http://www.modeshape.org/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0">

                  <mode:sources jcr:primaryType="nt:unstructured">

               

                      <mode:source jcr:name="MEMORY Store" mode:classname="org.modeshape.graph.connector.inmemory.InMemoryRepositorySource"                     mode:retryLimit="3" mode:defaultWorkspaceName="workspace1"/>

               

                      <mode:source jcr:name="FILESYSTEM Store"

                          mode:classname="org.modeshape.connector.filesystem.FileSystemSource"

                          mode:description="The repository for our content"

                          mode:workspaceRootPath="./repository/repofs"

                          mode:defaultWorkspaceName="workspace1"

                          mode:creatingWorkspacesAllowed="false"

                          mode:extraPropertiesBehavior="store"

                          mode:exclusionPattern="^.*\.modeshape$"

                          mode:rootNodeUuid="fd129c12-81a8-42ed-aa4b-820dba49e6f1"

                          mode:updatesAllowed="true" >

               

                          <mode:cachePolicy jcr:name="nodeCachePolicy"

                            mode:classname="org.modeshape.graph.connector.base.cache.InMemoryNodeCache$PathCachePolicy"

                            mode:timeToLive="300" />

                      </mode:source>

               

                  </mode:sources>

               

                      <mode:repositories>

               

                      <mode:repository jcr:name="Cars">

                          <mode:source>MEMORY Store</mode:source>

                          <mode:options jcr:primaryType="mode:options">

                              <jaasLoginConfigName jcr:primaryType="mode:option" mode:value="modeshape-jcr"/>

                          </mode:options>

                          <jcr:nodeTypes mode:resource="/cars.cnd" />

                          <namespaces jcr:primaryType="mode:namespaces">

                              <car jcr:primaryType="mode:namespace" mode:uri="http://jboss.org/dna/examples/cars/1.0"/>

                          </namespaces>

                      </mode:repository>

               

                      <mode:repository jcr:name="MyFileSystemRepository">

                          <mode:source>FILESYSTEM Store</mode:source>

                          <mode:options jcr:primaryType="mode:options">

                              <jaasLoginConfigName jcr:primaryType="mode:option" mode:value="modeshape-jcr"/>

                          </mode:options>           

                        </mode:repository>

                  </mode:repositories>

               

              </configuration>

               

               

               

                     Just now afternoon, I git clone the latest Modeshape source and have to build using "mvn clean install -fae -Dmaven.test.skip=true" , after skipping test phase, yet still quite a few Modeshape Maven reactor failed to build. However,  the major one eg. modeshape-common, modeshape-connector-filesystem, modeshape-connector-jcr, modeshape-connector-jcr-api, modeshape-repository successfully produce jar target. Currently I am testing using the newly build Modehshape jar.

              • 4. Re: Querying Repository (File system) return unexpected results
                penkween

                       I am now using the latest Modeshape 2.6 latest git source build and using Jcr 2.0 Style of looking up repository. Below is a slightly different program output, although the "SELECT * FROM [nt:base]"  still doesn't return all the nodes. The query basically still doen't work.

                 

                 

                INFO: Trying to load Repository #1 using: org.modeshape.jcr.JcrRepositoryFactory@1876c63

                INFO: JcrEngine starting...

                INFO: JcrEngine started in 6484 ms

                INFO: Started rebuilding indexes for repository 'MyFileSystemRepository'

                INFO: Completed rebuilding indexes for repository 'MyFileSystemRepository'

                INFO: Completed starting the "MyFileSystemRepository" repository

                INFO: Repository = org.modeshape.jcr.JcrRepository@1e70b07

                INFO: ########################################################################################################

                INFO: Language = JCR-SQL2    Expression = SELECT * FROM [nt:base]

                INFO: ########################################################################################################

                INFO: +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

                | # | jcr:primaryType | jcr:path | jcr:name | jcr:score | mode:localName | mode:depth | Location(nt:base)                                                                  | Score(nt:base) |

                +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

                | 1 | mode:root       | /        |          | 1.0       |                | 0          | </ && [{http://www.modeshape.org/1.0}uuid = fd129c12-81a8-42ed-aa4b-820dba49e6f1]> | 1.0            |

                +---+-----------------+----------+----------+-----------+----------------+------------+------------------------------------------------------------------------------------+----------------+

                INFO: ########################################################################################################

                INFO: Language = JCR-SQL2    Expression = SELECT * FROM [nt:file]

                INFO: ########################################################################################################

                INFO: +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

                | # | jcr:primaryType | jcr:created | jcr:createdBy | jcr:path | jcr:name | jcr:score | mode:localName | mode:depth | Location(nt:file) | Score(nt:file) |

                +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

                +---+-----------------+-------------+---------------+----------+----------+-----------+----------------+------------+-------------------+----------------+

                 

                 

                    Although I still can integrate Modeshape inside my project and iterate all Node using getNode() but using JCR-SQL2 query is very much faster and very essential in the searching through all Nodes and its Metadata Properties.

                 

                   Pls help . Thanks.

                • 5. Re: Querying Repository (File system) return unexpected results
                  rhauch

                  You mentioned in another thread that you recently included the Lucene libraries. I wonder if that (or some other libraries may be missing) is causing the indexing to fail.

                   

                  If your ModeShape configuration includes sequencers, text extractors, and MIME type detectors, I'd suggest temporarily commenting out all of them, since they typically require additional JARs on the classpath. I'd also suggest trying to temporarily use only the InMemory connector, again minimizing the number of JARs that are required. Perhaps look at what our JBoss AS 5.x and 6.x kits are including and see if that will help identify any libraries you're missing.

                  • 6. Re: Querying Repository (File system) return unexpected results
                    penkween

                    Hi Randall,

                     

                    The above test result obtained after including all lucene and lucene-search Jars in to the project, if not I think I don't even able to obtain the repository instance from the factory.

                     

                     

                    I will do the test again following your suggestions as below:

                     

                    1. Exclude any squencers, text extractors and MIME (The current project already excluded these entry inside the configuration)

                    2. Use only InMemory connector (The current project is using Cars which is already the InMemory repository, if using getNode, all nodes can be dump out properly)

                    3. Include Libraries as in JBoss AS 5.x and 6.X kits

                     

                           Really running out of testing option. Btw, is there any way that I can turn on any comprehensive Debuging or Logging for Modeshape querying the repository because it seem that there is no any Errors or Exception raised by the QueryManager, it just return a resultSet with a single row. Weird !

                    • 7. Re: Querying Repository (File system) return unexpected results
                      rhauch

                      Btw, is there any way that I can turn on any comprehensive Debuging or Logging for Modeshape querying the repository because it seem that there is no any Errors or Exception raised by the QueryManager, it just return a resultSet with a single row. Weird !

                       

                      Yes, you can turn up logging for ModeShape using whatever mechamism your logger expects.

                       

                      For example, if you're using Log4J, the Log4J logging configuration should contain a line to turn on DEBUG (or TRACE) logging for all of ModeShape. Here is an example Log4J configuration file we use in some of our integration tests, and line 19 is set to only INFO-level logging:

                       

                      log4j.logger.org.modeshape=WARN
                      

                       

                      You can change "WARN" to "DEBUG" or "TRACE" to get very verbose logging. Note that second part of the property name (e.g., "org.modeshape") specifies the package(s) that the rule applies to. You can add other rules with other ModeShape packages. For example, adding this line will turn on DEBUG logging for all of the JCR-level classes:

                       

                      log4j.logger.org.modeshape.jcr=DEBUG
                      

                       

                      Let us know what happens with the earlier suggestions.

                      • 8. Re: Querying Repository (File system) return unexpected results
                        penkween

                        Hi Randall,

                         

                               I think I found out something regarding the Query. First of all, it is not due to missing library and I am still using the same configRepository.xml having 2 repository ie: Cars (In memory repo) and MyFileSystemRepository (File System Repo).

                         

                               This is how it goes:

                               ===============================================================================

                               //Session Login

                               Repository repository=null;

                               String configUrl="jndi:jcr/local?repositoryName=MyFileSystemRepository";  //Using Jcr 2.0 style

                               Map<String, String> parameters = Collections.singletonMap("org.modeshape.jcr.URL", configUrl);

                               for (RepositoryFactory factory : ServiceLoader.load(RepositoryFactory.class)) {

                                       repository = factory.getRepository(parameters);

                                        if (repository != null) break;

                               }

                               Session session = repository.login(new ServletCredentials(request));

                         

                              <----------------- This code area is between Session Login and Logout ------------------->

                         

                               //Querying the Repository

                               QueryManager queryManager = session.getWorkspace().getQueryManager();

                               Query query = queryManager.createQuery("SELECT * FROM [nt:base]", "JCR-SQL2");

                               QueryResult result = query.execute();

                         

                               //Session Logout

                               session.logout();

                             =================================================================================

                         

                              What I found out is only new Nodes created inside the Code area (as shown above) will get listed by the query.execute() . For example, let say I am using that "MyFileSystemRepository" and inside that workspace filesystem has a folder named "folder1".

                         

                        Cases:

                        1.) Then , if I run the codes aboves, the Query will only return root node (the existing "folder1" node is not listed by the Query)

                         

                        2.) Then, if I create a new Node named "folder2" (nt:folder) inside the code area above, session.save() it , the Query will return Root node and the new "folder2" node (the existing "folder1" node still not listed by the Query)

                         

                        3.) Then, since now the workspace filesystem having both "folder1" and "folder2" , if I run the codes above, then the Query will return only the Root node (Just like in Case 1). (Both "folder1" and "folder2" nodes are not listed by the Query)

                         

                         

                            It appear that the Session failed to index the existing nodes when it is started and the Query only care about what happen after the session started and only during that particular session. Any pre existing nodes (be it nt:folder or nt:file) will be ignored by the Query. This weird behavior happen to both InMemory and Filesystem repository.

                        • 9. Re: Querying Repository (File system) return unexpected results
                          rhauch

                          Maybe this might help explain the behavior:

                           

                          When a client logs into a session, that session will only see the state of the repository as it existed the time the session was established plus any changes that session has made. When the session is saved via "Session.save()", all transient changes made by that session are written to the repository and the session's state is reset to state of the repository at that instant. When a session is refreshed via "Session.refresh(true)", all transient changes made in the session (and not yet written to the repository) are cleared and the session's state is reset to the state of the repository at that instant.

                           

                          As for queries, they generally expose those nodes that are visible to the session and persisted in the repository. The primary reason is that a query is executed within the context of a session. The session provides the authorization context, meaning queries only expose nodes that are visible/accessible by the session. Additionally, even though the query results can be seen in result set (e.g., tuples) form, there is also the ability to get the node(s) that correspond to each row.

                           

                          So given two sessions, A and B, established at (effectively) the same time, session A will not see (via navigation or query) any nodes created by session B until:

                          • session B is saved; and
                          • session A is either saved or refreshed

                           

                          Of course, any new session established after B is saved will obviously see the changes made by B.

                           

                          This behavior is all specified by the JCR specification. There is some flexibility defined by the specification in terms of whether implementations allow one session to immediately see information persisted by other sessions, and the above behavior (implemented by ModeShape) provides the most isolated and (in our opinion) most repeatable behavior (since a session's non-modified state never changes until 'refresh'). However, we do understand the need of some applications to see less isolated behavior, and we'll be making this possible in the 3.0 timeframe (later this year).

                           

                          One thing to keep in mind is the following: JCR sessions are generally intended to be short-lived and therefore are expected to be inexpensive to create, used for a short period of time, and then terminated. About the only valid use case for long-lived sessions is a single sessions used to register event listeners; often a single such session will suffice for an application, and it is JCR best-practice that any code responding to an event use a new session to interact with the repository (e.g., get the node for which a creation event was seen). JCR sessions are also not thread safe, meaning they cannot be used concurrently by multiple threads; instead, each thread should establish its own session.

                           

                          Does this help explain the behavior you are seeing? Does this make sense? Does it help at all? If this is confusing, I'd be more than happy to talk about it and help you figure out how best to use JCR for your application. Again, all of this is defined by the specification, though the spec does not often make clear all of the best practices.

                           

                          Best regards

                          • 10. Re: Querying Repository (File system) return unexpected results
                            penkween

                            Hi Randall,

                             

                                Understand your points above but what really happen over here seem like "When a client logs into a session, that session WILL NOT see the state of the repository as it existed the time the session was established and WILL ONLY see changes that session has made"

                             

                                In fact, the codes above is packaged inside Servlet for every Test : 1,2,3, the Glassfish Server is restarted and the Servlet Web App is redeployed to ensure clean test for every test.

                             

                                So as mentioned in Test Case #3 in previous thread, we already have the following repository fileystem structure where "folder2" node is created from Test Case #2 previously:

                             

                            Filesystem Structure before any Session establishment

                            =============================================

                            [workspace1] /folder1

                                                 /folder2

                             

                            *Pls noted that Session A,B,C shown below basically just indicating different single session is established at different timespace after each App Server restart and Web App redeploy for each test. Session A,B,C Are Not  established at the same time by 3 servlets. For every test , we only have a single servlet, single session and the servlet is called from Browser one time only. From Server log, I can see the Jcr Engine started, Repository started and Index built every time the servlet is called.

                             

                            *Our current local time is 10:00am (GMT+8)*

                             

                            [Test #1 - Session A]

                            ========================

                            *Time =  10:00am

                                So, When I restart App Server and redeploy the Web App (containing the servlet for the modeshape codes). After new Session A is established and after the Query,  I suppose to see 3 rows which is "Root", "folder1" and "folder2" right? ,  but I don't , I ONLY see "Root" row. 

                             

                            [Test #2 - Session B]

                            ========================

                            *Time =  10:05am

                                So, again, if I restart App Server and redeploy the Web App. After new Session B is established, let say in the middle of a Session B, I create a new Node named "folder3", then if Query, it will list me "Root" and "folder3"  instead of just "Root".

                             

                            [Test #3 - Session C]

                            ========================

                            *Time =  10:10am

                                 So, again, if I restart App Server and redeploy the Web App. After Session C is established, It all back to square one. After Query, I ONLY see "Root" row. (eventhough I suppose to see "Root", "folder1", "folder2" and "folder3" which is really exist in the workspace at this point of time)

                             

                             

                             

                             

                               This behavior has render the Query totally unusable because I can't search what is existing in the repository when the Session is established. I don't think this is the expected behavior from Jcr.

                            • 11. Re: Querying Repository (File system) return unexpected results
                              penkween

                              Hi Randall,

                               

                              Aware that MODE-1238  is link to this issue. Just some further lnfo about the Query using others connector types:

                               

                               

                              1. Query using Disk Connector (OK)- I have done a test under the exact same environment using a Disk Connector store (Instead of the File System Connector store) and the Query ( SELECT * FROM [nt:base] ) just work fine. When I create a new node "folder1" (nt:folder) ,  after restart the Glassfish server and redeploy the web app, I can see "folder1" is listed by Query as follow:

                               

                              INFO: Trying to load Repository #1 using: org.modeshape.jcr.JcrRepositoryFactory@18c2043

                              INFO: Repository = org.modeshape.jcr.JcrRepository@c19dc4

                              INFO: ########################################################################################################

                              INFO: Language = JCR-SQL2    Expression = SELECT * FROM [nt:base]

                              INFO: ########################################################################################################

                              INFO: +---+-----------------+----------+----------+-----------+----------------+------------+---------------------------------------------------------------------------------------------+----------------+

                              | # | jcr:primaryType | jcr:path | jcr:name | jcr:score | mode:localName | mode:depth | Location(nt:base)                                                                           | Score(nt:base) |

                              +---+-----------------+----------+----------+-----------+----------------+------------+---------------------------------------------------------------------------------------------+----------------+

                              | 1 | mode:root       | /        |          | 1.0       |                | 0          | </ && [{http://www.modeshape.org/1.0}uuid = fd129c12-81a8-42ed-aa4b-820dba49e6f0]>          | 1.0            |

                              | 2 | nt:folder       | /folder1 | folder1  | 1.0       | folder1        | 1          | </{}folder1 && [{http://www.modeshape.org/1.0}uuid = f5c3b8af-5b83-40ef-ac3e-791ebe6aad85]> | 1.0            |

                              +---+-----------------+----------+----------+-----------+----------------+------------+---------------------------------------------------------------------------------------------+----------------+

                               

                               

                              2. Query using JpaSource Connector (OK) - Test using a Mysql5 datasource, the Query work fine too.

                               

                               

                              3. Query using File System Connector (Fail)- So, that is definitely something not right about the Query and indexing on File System store.

                              • 12. Re: Querying Repository (File system) return unexpected results
                                penkween

                                Hi Randall,

                                 

                                Really stuck here with Querying Filesystem repository, is there any workaround or something that I can do ?

                                 

                                Thanks.

                                • 13. Re: Querying Repository (File system) return unexpected results
                                  rhauch

                                  The thing that's making this difficult to verify/replicate is that we have several integration tests that verify this exact behavior, and these tests been running successfully for months.

                                   

                                  Is there any chance you can create a stand-alone test case that replicates this behavior? There's already a starting point with code that creates a standalone Maven-based project with an app that creates an embedded ModeShape instance.

                                  • 14. Re: Querying Repository (Filesystem Connector) return only Root node
                                    penkween

                                    Ok. Will proceed with the standalone app testing.

                                    1 2 Previous Next