12 Replies Latest reply on Jan 22, 2014 1:35 PM by rareddy

    Accumulo Metadata

    pinkstondevin

      Hello,

       

      I keep hitting a metadata failed to load with accumulo.  I modified the RA with just two sysouts so i can see the roles and auths, thats why it is printing that out towards the bottom of the log.  I am not sure why it isn't connecting, any thoughts?  I pased the VDB and config for Accum as well.

       

      thanks!

       

       

      LOG:

      08:51:52,800 INFO  [org.teiid.RUNTIME] (ServerService Thread Pool -- 44)  TEIID50006 Teiid translator "accumulo" added.
      08:51:52,803 INFO  [org.teiid.RUNTIME] (ServerService Thread Pool -- 44)  TEIID50006 Teiid translator "solr" added.
      08:51:52,839 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1)  JBAS015876: Starting deployment of "accum-vdb.xml" (runtime-name: "accum-vdb.xml")
      08:51:52,843 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1)  JBAS015876: Starting deployment of "teiid-odata-8.7.0.Alpha1.war" (runtime-name: "teiid-odata-8.7.0.Alpha1.war")
      08:51:52,845 INFO  [org.jboss.as.remoting] (MSC service thread 1-3)  JBAS017100: Listening on #.#.#.#:9999
      08:51:52,846 INFO  [org.jboss.as.remoting] (MSC service thread 1-4)  JBAS017100: Listening on #.#.#.#:4447
      08:51:52,853 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-5)  JBAS015012: Started FileSystemDeploymentService for directory /home/funluser/jboss-eap-6.1/standalone/deployments
      08:51:53,303 INFO  [org.jboss.web] (ServerService Thread Pool -- 50)  JBAS018210: Register web context: /odata
      08:51:53,359 INFO  [org.teiid.RUNTIME] (MSC service thread 1-1)  TEIID50029 VDB accum.1 model "accumModel" metadata is currently being loaded. Start Time: 1/17/14 8:51 AM
      08:51:53,366 WARN  [org.teiid.RUNTIME] (teiid-async-threads - 1)  TEIID50036 VDB accum.1 model "accumModel" metadata failed to load. Reason:TEIID31097 Metadata load requires a connection factory: TEIID30481 Failed to find the Connection Factory with JNDI name java:/accumuloDS. Please check the name or deploy the Connection Factory with specified name.
      08:51:53,467 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 50)  HV000001: Hibernate Validator 4.3.1.Final-redhat-1
      08:51:53,590 INFO  [org.jboss.resteasy.spi.ResteasyDeployment] (ServerService Thread Pool -- 50)  Deploying javax.ws.rs.core.Application: class org.teiid.odata.TeiidODataApplication
      08:51:55,676 WARN  [org.apache.zookeeper.ClientCnxn] (MSC service thread 1-6-SendThread(#.#.#.#:2181))  SASL configuration failed: javax.security.auth.login.LoginException: unable to find LoginModule class: org.jboss.as.security.remoting.RemotingLoginModule from [Module "org.jboss.teiid.resource-adapter.accumulo:main" from local module loader @3b6f0be8 (finder: local module finder @7776cad3 (roots: /home/funluser/jboss-eap-6.1/modules,/home/funluser/jboss-eap-6.1/modules/system/layers/base))] Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
      08:51:55,678 INFO  [org.apache.zookeeper.ClientCnxn] (MSC service thread 1-6-SendThread(#.#.#.#:2181))  Opening socket connection to server #.#.#.#/#.#.#.#:2181
      08:51:55,683 INFO  [org.apache.zookeeper.ClientCnxn] (MSC service thread 1-6-SendThread(#.#.#.#:2181))  Socket connection established to #.#.#.#/#.#.#.#:2181, initiating session
      08:51:55,691 INFO  [org.apache.zookeeper.ClientCnxn] (MSC service thread 1-6-SendThread(#.#.#.#:2181))  Session establishment complete on server #.#.#.#/#.#.#.#:2181, sessionid = 0x143735d9e884d0f, negotiated timeout = 30000
      08:51:55,750 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-6)  JBAS010406: Registered connection factory java:/accumuloDS
      08:51:55,755 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (MSC service thread 1-6)  IJ020002: Deployed: file:/home/funluser/jboss-eap-6.1/modules/system/layers/base/org/jboss/teiid/resource-adapter/accumulo/main/./
      08:51:55,757 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-2)  JBAS010401: Bound JCA ConnectionFactory [java:/accumuloDS]
      08:51:55,786 INFO  [stdout] (teiid-async-threads - 2)  auth: [public]
      08:51:55,787 INFO  [stdout] (teiid-async-threads - 2)  roles: [Ljava.lang.String;@284c1da6
      08:51:55,816 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 27)  JBAS018559: Deployed "accum-vdb.xml" (runtime-name : "accum-vdb.xml")
      08:51:55,820 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 44)  JBAS018559: Deployed "teiid-odata-8.7.0.Alpha1.war" (runtime-name : "teiid-odata-8.7.0.Alpha1.war")
      08:51:55,998 INFO  [org.jboss.as] (Controller Boot Thread)  JBAS015961: Http management interface listening on http://#.#.#.#:9990/management
      08:51:55,999 INFO  [org.jboss.as] (Controller Boot Thread)  JBAS015951: Admin console listening on http://#.#.#.#:9990
      08:51:55,999 INFO  [org.jboss.as] (Controller Boot Thread)  JBAS015874: JBoss Teiid + Teiid 8.4 (AS 7.2.0.Final-redhat-8) started in 9086ms - Started 322 of 398 services (75 services are passive or on-demand)
      08:51:56,023 WARN  [org.teiid.RUNTIME] (teiid-async-threads - 2)  TEIID50036 VDB accum.1 model "accumModel" metadata failed to load. Reason:java.lang.NullPointerException: java.lang.NullPointerException
          at org.teiid.translator.accumulo.AccumuloMetadataProcessor.processMetadata(AccumuloMetadataProcessor.java:103)
          at org.teiid.translator.accumulo.AccumuloExecutionFactory.getMetadata(AccumuloExecutionFactory.java:76)
          at org.teiid.translator.accumulo.AccumuloExecutionFactory.getMetadata(AccumuloExecutionFactory.java:39)
          at org.teiid.query.metadata.NativeMetadataRepository.loadMetadata(NativeMetadataRepository.java:61) [teiid-engine-8.7.0.Alpha1.jar:8.7.0.Alpha1]
          at org.teiid.query.metadata.ChainingMetadataRepository.loadMetadata(ChainingMetadataRepository.java:55) [teiid-engine-8.7.0.Alpha1.jar:8.7.0.Alpha1]
          at org.teiid.jboss.VDBService$6.run(VDBService.java:403) [teiid-jboss-integration-8.7.0.Alpha1.jar:8.7.0.Alpha1]
          at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
          at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806)
          at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
          at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:826)
          at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
          at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

       

      VDB:

      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
        <vdb name="accum" version="1">
          <description>accum Test</description>
      
              <property name="UseConnectorMetaData" value="true"/>
           <model name="accumModel">
              <source name="accumulo-connector" translator-name="accumulo" connection-jndi-name="java:/accumuloDS"/>
          </model>
      </vdb>
      

       

       

      CONFIG:

      <resource-adapter id="accumulo">
                          <module slot="main" id="org.jboss.teiid.resource-adapter.accumulo"/>
                          <transaction-support>NoTransaction</transaction-support>
                          <connection-definitions>
                              <connection-definition class-name="org.teiid.resource.adapter.accumulo.AccumuloManagedConnectionFactory" jndi-name="java:/accumuloDS" enabled="true" use-java-context="true" pool-name="teiid-accumulo-ds">
                                  <config-property name="ZooKeeperServerList">
                                         #.#.#.#:2181
                                  </config-property>
                                  <config-property name="Roles">
                                      public
                                  </config-property>
                                  <config-property name="InstanceName">
                                         accumInstance
                                  </config-property>
                                  <config-property name="Username">
                                      root
                                  </config-property>
                                  <config-property name="Password">
                                         pwd
                                  </config-property>
                              </connection-definition>
                          </connection-definitions>
                      </resource-adapter>
      
        • 1. Re: Accumulo Metadata
          pinkstondevin

          It appears that the connection factory is not registering when the connector tries to scan for metadata, however I am making a successful connection to the Accumulo instance. 

           

          If i try to undeploy and redeploy the VDB, I keep receiving null point exceptions that the metadata failed to load.  Is it possible the connection factory isn't registering?

           

          Thanks

          • 2. Re: Accumulo Metadata
            rareddy

            Looking at the code, it looks like it did not find any scanners with connected user. Thus the issue. However it is bug, that causing the NPE. I can fix the NPE.

             

            Looks like I am doing the createScanner with empty Authorizations, which may be returning no rows, thus fail to create the table and NPE happens in the mean time. So, there are couple fixed needs to be done here. I will add comment to your previous JIRA about this.

             

            Ramesh..

            • 3. Re: Accumulo Metadata
              pinkstondevin

              Ramesh,

               

              Correct I fixed the scanner issue and it is working correctly now.  I will submit a patch via a pull request shortly.  Just after I got that working, i submitted a query against accumulo and received a data type conversion error:

               

              Any idea what that may be related to? 

               

              Executing using the TeiidDriver

              org.teiid.jdbc.TeiidSQLException: TEIID19004 Unsupported data type conversion

                at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135)

                at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71)

                at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:667)

                at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:63)

                at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:515)

                at org.teiid.client.util.ResultsFuture.done(ResultsFuture.java:135)

                at org.teiid.client.util.ResultsFuture.access$200(ResultsFuture.java:40)

                at org.teiid.client.util.ResultsFuture$1.receiveResults(ResultsFuture.java:79)

                at org.teiid.net.socket.SocketServerInstanceImpl.receivedMessage(SocketServerInstanceImpl.java:257)

                at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:295)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:102)

                at com.sun.proxy.$Proxy1.read(Unknown Source)

                at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:394)

                at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:524)

                at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:393)

                at org.teiid.jdbc.StatementImpl.executeQuery(StatementImpl.java:327)

                at JDBCClient.execute(JDBCClient.java:107)

                at JDBCClient.main(JDBCClient.java:67)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

              Caused by: org.teiid.core.TeiidException: TEIID19004 Unsupported data type conversion

                at org.teiid.client.ResultsMessage.setException(ResultsMessage.java:194)

                at org.teiid.dqp.internal.process.RequestWorkItem.sendError(RequestWorkItem.java:970)

                at org.teiid.dqp.internal.process.RequestWorkItem.close(RequestWorkItem.java:543)

                at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:346)

                at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:51)

                at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:248)

                at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:273)

                at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:119)

                at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:214)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

                at java.lang.Thread.run(Thread.java:662)

              Caused by: org.teiid.core.TeiidRuntimeException: TEIID19004 Unsupported data type conversion

                at org.teiid.translator.accumulo.AccumuloDataTypeManager.convertFromAccumuloType(AccumuloDataTypeManager.java:96)

                at org.teiid.translator.accumulo.AccumuloQueryExecution.nextRow(AccumuloQueryExecution.java:197)

                at org.teiid.translator.accumulo.AccumuloQueryExecution.next(AccumuloQueryExecution.java:175)

                at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.handleBatch(ConnectorWorkItem.java:376)

                at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.more(ConnectorWorkItem.java:200)

                at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:301)

                at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:110)

                at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:107)

                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                at org.teiid.dqp.internal.process.FutureWork.run(FutureWork.java:58)

                ... 6 more

              Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0

                at JDBCClient.main(JDBCClient.java:72)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:606)

                at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

              -----------------------------------

              Executing using the TeiidDataSource

               

               

              Process finished with exit code 1

              • 4. Re: Accumulo Metadata
                rareddy

                Devin,

                 

                According to the exception the ROWID column of your table is not been able to convert to the expected data type that has been defined on the metadata. What is type you defined on metadata?

                 

                Accumulo always considers the ROWID column to be string based to store in lexicographical sort order, so I made some assumptions  about data format. But that is not convertible to your defined data type (I am guessing you defined DDL for the table manually).

                 

                Ramesh..

                • 5. Re: Accumulo Metadata
                  pinkstondevin

                  Ramesh,

                   

                  We did not use DDL metadata actually, the MetaDataProcessor was used to import the data successfully. 

                  • 6. Re: Accumulo Metadata
                    rareddy

                    OK, Do you know the what is reported type? You can use web-console to see the schema of the model, or can use JDBC tool like SquirreL to inspect the schema structure exposed by the VDB

                     

                    Ramesh..

                    1 of 1 people found this helpful
                    • 7. Re: Accumulo Metadata
                      pinkstondevin

                      Yes, I have pasted the DDL from the web console here:

                       

                      rowid varbinary OPTIONS (UPDATABLE FALSE, SEARCHABLE 'All_Except_Like'),

                      "chkpt:user" varbinary OPTIONS (UPDATABLE FALSE, SEARCHABLE 'All_Except_Like', "teiid_accumulo:CF" 'chkpt', "teiid_accumulo:CQ" 'user', "teiid_accumulo:VALUE-IN" '{VALUE}'),

                      "user:" varbinary OPTIONS (UPDATABLE FALSE, SEARCHABLE 'All_Except_Like', "teiid_accumulo:CF" 'user', "teiid_accumulo:CQ" '', "teiid_accumulo:VALUE-IN" '{VALUE}'),

                      CONSTRAINT PK0 PRIMARY KEY(rowid)

                      ) OPTIONS (UPDATABLE TRUE);

                       

                      Basically imagine a username from some social media outlet, or an email address.  Sometimes it can contain numbers or other characters, I think that is the problem.

                       

                      Thanks

                      • 8. Re: Accumulo Metadata
                        rareddy

                        Devin,

                         

                        I believe the problem is how I am constructing metadata, in the case that DDL is not provided, I am assuming all the data types to VARBINARY and conversion to that type from BYTE[ ] seems to be the issue. The dis-advantage in using the native metadata from translator is, Accumulo does not define the data type, as it could be any serialized object, thus I used the most liberal type that can fit all the data types. May be I could map ROWID to string and keep others as BINARY type?

                         

                        Ramesh..

                        • 9. Re: Accumulo Metadata
                          pinkstondevin

                          Submitted Pull Request: https://github.com/teiid/teiid/pull/181

                           

                          This corrected the problem for me and I am able to query Accumulo successfully with the native Metadata.  I did not use DDL.  You were right, switching to STRING made it work!

                          • 10. Re: Accumulo Metadata
                            rareddy

                            Yeah, that looks good. Small problem is, if we were to use read as text they need to be adjusted for the character encoding. I will add a translator override property and set the default to "UTF-8"

                             

                            Ramesh..

                            • 11. Re: Accumulo Metadata
                              pinkstondevin

                              good point, didn't think about that.

                              • 12. Re: Accumulo Metadata
                                rareddy

                                I added the necessary code and resolved the issue. You can test let me know if you see any issues. [TEIID-2811] Accumulo RowID VARBINARY conversion - JBoss Issue Tracker