Thanks for the help. At first I thought I just didn't know how to query, which I didn't. But also, as it turns out, I'm not doing the sequencing correct either. Because when I execute a query similar to what you posted I get something like this:

Exception in thread "main" javax.jcr.RepositoryException: Table 'text:column' does not exist

Table 'text:column' does not exist

          at org.modeshape.jcr.query.qom.JcrAbstractQuery.checkForProblems(JcrAbstractQuery.java:133)

          at org.modeshape.jcr.query.JcrQuery.execute(JcrQuery.java:104)

          at TestJCR.doExample(TestJCR.java:79)

          at TestJCR.main(TestJCR.java:27)

Which basically tells me I'm still not loading the file correctly. Here's the sequencer from my config and the code I'm using to load it:

Config:

   <mode:sequencers>

       <mode:sequencer jcr:name="Delimited Text Sequencer">

           <mode:classname>

            org.modeshape.sequencer.text.DelimitedTextSequencer

           </mode:classname>

           <mode:description>Sequences delimited files to extract values</mode:description>

           <mode:pathExpression>//(*.(csv)[*])/jcr:content[@jcr:data] => /csv/$1</mode:pathExpression>

           <mode:splitPattern>,</mode:splitPattern>

       </mode:sequencer>

   </mode:sequencers>

Java:

        final JcrConfiguration jcrConfig = new JcrConfiguration();       // org.modeshape.jcr.JcrConfiguration

        final JcrEngine engine = jcrConfig.loadFrom(configFile).build(); // org.modeshape.jcr.JcrEngine;

        engine.start();

        final JcrRepository repo = engine.getRepository(REPO_NAME);      // org.modeshape.jcr.JcrRepository

        final Session session = repo.login(WORKSPACE_NAME);              // javax.jcr.Session

        final Node rootNode = session.getRootNode();

        final Node testFolder = rootNode.getNode("test_folder");

        final Node fileNode = testFolder.addNode("people.csv", "nt:file");

        session.save();

        final String jql = "SELECT * FROM [text:column]";

        final QueryManager queryManager = session.getWorkspace().getQueryManager();

        final Query query = queryManager.createQuery(jql, Query.JCR_SQL2);

        final QueryResult queryResult = query.execute();

As I said in my first post, I'm just trying to modify the "beginner's guide" code in this forum to get comfortable using Modeshape. I'm just not sure how Modeshape distinguishes [nt:file] types from the delimited file types. I did look in the docs at the [text:column] and the [text:row] types, I'm just not sure where they go and how to use/specify them.

Any other pointers you (or anyone else) could provide would go a long way in helping me get more familiar with Modeshape.

Thanks!

Horia,

Thanks for the help so far. I think the sequencing is working now. Kinda.

But there are a couple questions I have.

1. I used the session.getNode('/csv/people.csv') != null. It works just fine and returns true. Then I try to query the file with this: "SELECT [text:column] FROM [nt:file] where [jcr:path] = '/csv/people.csv'". It returns this:

    /csv/people.csv {jcr:createdBy=<anonymous>, jcr:primaryType=nt:file, jcr:created=2012-05-10T00:17:36.038-05:00, }

        jcr:createdBy=<anonymous>

        jcr:primaryType=nt:file

        jcr:created=2012-05-10T00:17:36.038-05:00

But I still can't get to the rows and columns in the file. The output above was created by using the NodeIterator and PropertyIterator. So now that the sequencer seems to be working, am I querying correctly? What else do I need to do?

2. This relates to memory usage. Apparently after the file is sequenced, there seems to be a jump in memory consumption. I'm not sure if the sequencing is still going on in the background or what, but it grows to a GB then runs out of memory. I could just increase the heap size, but was curious as to what was causing it. The CSV is only 9MB with 500k rows and 5 cols. Any ideas on why this is happening?

Again, thanks so much.

Horia,

I'm still looking into the memory usage, but I'm attached what I've been able to gather so far. It's a snapshot before it crashed due to an OutOfMemoryError. I increased the heap to 2048m and it still crashed. I'm not sure how helpful the memory profile will be, but I posted it anyway.

As for querying, I still get the same error as before, Table 'text:column' does not exist, when using SELECT * FROM [text:column]. I'll try again tomorrow, if I still can't get it I might post my code. Then I'll probably start playing with v3 alpha4, maybe I'll have better luck with the new version.

Sorry for the headache,

eliquious

Sure, no problem. I've attached the code with the config file. The CSV is just a randomly generated file with 5 columns and 500k rows. Nothing special. I've also included the output of the program, which includes the query error.

I'll take that as good news. I was already planning on going with 3.x. I saw that error as well but wasn't sure on how to remidy it. I'll look into using the other connectors.

I'm much more knowledgable about modeshape now than I was before. Thanks for all the help and suggestions.

Hey Horia,

I've started playing with v3.0.0.Alpha4 and have succeeded in sequencing the test file included with modeshape. However, when I query for the file contents, no results are returned. I can iterate through the nodes, rows and columns, but querying doesn't return the data from the file. As you might notice, I've taken most of the code below from the text sequencing test code.

public static void main(String[] args) throws Exception {

    String REPO_NAME = "Test Repository";

    String filename = "multiLineCommaDelimitedFile.csv";

    String filePath = "delimited/" + filename;

    RepositoryConfiguration config = RepositoryConfiguration.read("config/repo-config.json");

    // Verify the configuration for the repository ...

    Problems problems = config.validate();

    if (problems.hasErrors()) {

        System.err.println("Problems starting the engine.");

        System.err.println(problems);

        System.exit(-1);

    }

 

    System.out.println("Starting");

    JcrEngine engine = new JcrEngine();

    engine.start();

    JcrRepository repository = engine.deploy(config);

    Node rootNode = session.getRootNode();

 

    System.out.println("Sequencing");

    Node parent = rootNode;

    String[] paths = filePath.split("/");

    for (int i = 0; i < paths.length - 2; i++) {

        parent = parent.addNode(paths[i], JcrConstants.NT_FOLDER);

    }

    parent = parent.addNode(paths[paths.length - 1], JcrConstants.NT_FILE);

    Node content = parent.addNode(JcrConstants.JCR_CONTENT,   JcrConstants.NT_RESOURCE);

    File file = new File(filePath);

    Binary binary = (((javax.jcr.Session) session).getValueFactory().createBinary(new FileInputStream(file)));

    content.setProperty(JcrConstants.JCR_DATA, binary);

    session.save();

    Node n = getSequencedNode(rootNode, filePath, 5);

    if(n == null)

        throw new Exception("Sequencing took longer than 5 seconds.");

    System.out.println(n);

 

    System.out.println("Querying");

    final String jql = "SELECT * from [text:column]";

    Workspace ws = session.getWorkspace();

    QueryManager queryManager = ws.getQueryManager(); // javax.jcr.QueryManager

    Query query = queryManager.createQuery(jql, Query.JCR_SQL2);

    QueryResult queryResult = query.execute();

    System.out.println(queryResult);

 

    System.out.println("Shutting down...");

    Future<Boolean> future = engine.shutdownRepository(REPO_NAME);

    future.get();

    engine.shutdown();

 

    System.out.println("Exiting...");

    System.exit(0);

}

public static Node getSequencedNode(Node parentNode, String path, int maxWaitTimeSeconds) throws Exception {

    //TODO author=Horia Chiorean date=12/14/11 description=Change this hack once there is a proper way (events) of retrieving the sequenced node

    long maxWaitTime = TimeUnit.SECONDS.toNanos(maxWaitTimeSeconds);

    long start = System.nanoTime();

    Node n = null;

    while (System.nanoTime() - start <= maxWaitTime) {

       try {

            n = parentNode.getNode(path);

      

            if (n != null) {

                return n;

            }

       } catch (Exception e) {

       }

    }

    return null;

}

I find it odd that I can iterate through the nodes, but nothing is found when querying. I've attached my configuration files. Do you have any ideas on what might be causing this behavior?