1 2 3 Previous Next 35 Replies Latest reply on Jan 21, 2016 9:17 AM by rareddy

    Hive integration

    jietao

      Hi,

      I want to create source model for Hadoop Hive with Teiid designer on Eclipse Luna (new installed). I followed Connect to a Hadoop Source using Hive2, but in the create data source dialog I did not find hive12.

       

      Help!

        • 1. Re: Hive integration
          shawkins

          It should appear if the driver has been successfully added to the server.  Did you modify the standalone.xml directly and perform a restart of the server?  Or did you add via CLI?

          • 2. Re: Hive integration
            rareddy
            • 3. Re: Hive integration
              jietao

              Thx for the rapid answer. I modified standalone-teiid.xml and restarted the server.

              • 4. Re: Hive integration
                jietao

                now I got it. I checked the .xml file and found that the modification disappeared. Do not know why this happened. Thanks again!

                 

                Jie

                • 5. Re: Hive integration
                  jietao

                  but I got this error: Data source type hive12 does not exist on server JBoss AS 7.1. I do not have a Hadoop source installed. Is this a reason?

                  • 6. Re: Hive integration
                    shawkins

                    Can you elaborate on the error?

                     

                    To just see the driver in Designer you do not need to have hadoop installed - just the driver needs to be installed and in the configuration.

                    • 7. Re: Hive integration
                      jietao

                      I got the problem. The driver was unpacked at the "dv" directory rather than "base". I copied it to the ...base/org/apache....  But I need a driver for hive 13.1. Where can I download this?

                       

                      Thanks again.

                      • 8. Re: Hive integration
                        shawkins

                        A community user pulled together the 0.13.1 jdbc driver jars in a maven pom - vchintal/hive-jdbc-jars-archive · GitHub

                        • 9. Re: Hive integration
                          jietao

                          thx. I got the driver jars and created a source model with importing metadata using JDBC. I can connect to our Hive (merging completed, completed difference analasis) but then I got this error:

                           

                          Error while obtaining quote string for New Generic JDBC Method not supported

                          java.sql.SQLException: Method not supported

                              at org.apache.hive.jdbc.HiveDatabaseMetaData.getIdentifierQuoteString(HiveDatabaseMetaData.java:342)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.getQuoteString(RelationalModelProcessorImpl.java:2675)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2635)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setNameAndNameInSource(RelationalModelProcessorImpl.java:2604)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.setColumnInfo(RelationalModelProcessorImpl.java:1317)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createColumns(RelationalModelProcessorImpl.java:1273)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.createNewObject(RelationalModelProcessorImpl.java:1174)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.performExecute(RelationalModelProcessorImpl.java:844)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl$1.run(RelationalModelProcessorImpl.java:342)

                              at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:567)

                              at org.teiid.designer.core.ModelEditorImpl.executeAsTransaction(ModelEditorImpl.java:611)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:349)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.executeWithinTransaction(RelationalModelProcessorImpl.java:324)

                              at org.teiid.designer.jdbc.relational.impl.RelationalModelProcessorImpl.execute(RelationalModelProcessorImpl.java:314)

                              at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard.runFinish(JdbcImportWizard.java:752)

                              at org.teiid.designer.jdbc.ui.wizards.JdbcImportWizard$1.run(JdbcImportWizard.java:292)

                              at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)

                           

                          Shall I create the Hive source model in other ways?

                          • 10. Re: Hive integration
                            shawkins

                            That is an issue with the hive driver that is not accounted for in the Designer import logic.  Teiid does account for that, so you should be able to use the Teiid connection importer to successfully import.

                            • 11. Re: Hive integration
                              jietao

                              sorry to trouble u again because I was not successful.

                               

                              I replaced all jars of modules/system/layers/base/org/apache/hadoop/hive12/main with the jars for hive 0.13.1 and adapted the names in modue.xml. I then followed the procedure based on the link in my first message to use Teiid connection. I got error by the last step: Get DDL for the import: error deploying importVDB. My server says that metadata is invald:

                               

                              09:24:17,512 INFO  [org.teiid.CONNECTOR] (teiid-async-threads - 2)  TEIID11002 Failed to report the JDBC driver and connection information

                              09:24:17,539 INFO  [org.teiid.RUNTIME] (teiid-async-threads - 2)  TEIID50030 VDB importVDB2.1 model "importVDB2SrcModel" metadata loaded. End Time: 15.10.15 09:24

                              09:24:17,541 WARN  [org.teiid.PLANNER.RESOLVER] (teiid-async-threads - 2)  TEIID31070 Empty model;There are no tables, procedures or functions defined in this model importVDB2SrcModel

                              09:24:17,541 INFO  [org.teiid.RUNTIME] (teiid-async-threads - 2)  TEIID40073 The metadata for the VDB importVDB2.1 is loaded, however it is not valid. Check models for errors. Correct the metadata and re-deploy.

                              09:24:17,541 INFO  [org.teiid.RUNTIME.VDBLifeCycleListener] (teiid-async-threads - 2)  TEIID40003 VDB importVDB2.1 is set to FAILED

                              09:24:17,621 INFO  [org.jboss.as.server] (management-handler-thread - 3)  JBAS015859: "importVDB2-vdb.xml" deployed (runtime-name: "importVDB2-vdb.xml")

                               

                              The following is the xml for deployment:

                               

                              <?xml version="1.0" encoding="UTF-8" standalone="yes"?>

                              <vdb name="importVDB2" version="1">

                                  <description>Importer VDB</description>

                                  <property name="UseConnectorMetadata" value="true" />

                                  <property name="deployment-name" value="importVDB2-vdb.xml" />

                                  <model name="importVDB2SrcModel" type="PHYSICAL" visible="true">

                                      <property name="trimColumnNames" value="true" />

                                      <source name="importVDB2SrcModel" translator-name="hive" connection-jndi-name="java:/HadoopHive" />

                                  </model>

                               

                              In hive12 driver I see a hive-metastore-0.12.0.2.0.6.0-76.jar but no such jar exist in 0.13.1. What I have are:

                               

                              commons-logging-1.1.3.jar         httpcore-4.2.5.jar
                              hadoop-common-2.4.0.jar           libfb303-0.9.0.jar
                              hive-exec-0.13.1.jar              libthrift-0.9.0.jar
                              hive-jdbc-0.13.1.jar              log4j-1.2.16.jar
                              hive-service-0.13.1.jar           slf4j-api-1.7.5.jar

                              httpclient-4.2.5.jar

                               

                              Something missing?

                              • 12. Re: Hive integration
                                shawkins

                                > Something missing?

                                 

                                In my local testing I needed to include hive-metastore-0.13.1.jar.  I'm not sure why that's excluded from Vijay's jar set.

                                • 13. Re: Hive integration
                                  jietao

                                  I downloaded this from maven repository, but the error is still there (is this a correct one?). Can you show me your modue.xml so that I can check if I included all jars? Thx

                                  • 14. Re: Hive integration
                                    shawkins

                                    > Can you show me your modue.xml so that I can check if I included all jars?

                                     

                                    I didn't have a pom from my earlier testing.  The jar set I was using included:

                                     

                                    commons-logging-1.1.3.jar

                                    httpclient-4.2.5.jar

                                    httpcore-4.2.5.jar

                                    slf4j-api-1.7.5.jar

                                    libfb303-0.9.0.jar

                                    hive-jdbc-0.13.1.jar

                                    hive-service-0.13.1.jar

                                    hive-metastore-0.13.1.jar

                                    hadoop-common-2.5.2.jar

                                    hive-exec-0.13.1.jar

                                     

                                    Try with that set and provide the error message you are getting if it doesn't work.

                                    1 2 3 Previous Next