13 Replies Latest reply on Sep 29, 2011 11:10 AM by rareddy

    SOLR  connector

    akhilkg1987

      Hi,

      Can somebody tell me how to connect to SOLR with teiid?the way we connect to sql or other platforms, I need to make a connection to SOLR.

       

      Thanks

      Akhil.

        • 1. Re: SOLR  connector
          rareddy

          Akhil,

           

          What is SOLR? I never heard of this data source, can you give more details? If this has source has JDBC driver then you can connect to it using Teiid, if not there are other ways, but all of them require custom coding for your source.

           

          Ramesh..

          • 2. Re: SOLR  connector
            akhilkg1987

            Hi ramesh,

            SOLR is actually an enterprise search platform which we are using to index the data.

            Solr is written in Java and runs as a standalone full-text search server within a servlet container such as tomcat and it uses the lucene java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs.

             

            And I was trying to download the connector development kit from http://community.jboss.org/wiki/TeiidConnectorDevelopmentEclipsePlugin but the link there isnt working with my SVN. seems like am getting stuck at every step

            • 3. Re: SOLR  connector
              akhilkg1987

              Hi Ramesh,

              Am back to this work..

               

              I have following queries:

              How is a connector different from translator in teiid?
              in examples I found a translator folder and a connector folder too(source\teiid\connectors\sandbox)

              How to use yahoo translator example?


              Is there any thread/documentation where it has been discussed

               

              I was going through the connector development guide,looking for what do we need to do once the coding part for connector

              is done.

              There they have mentioned following steps:

              -->Creating a Connector Type Definition file that defines the properties required to initialize your
              connector.
              • Identifying the Extension Modules (jars and resources) required for the Connector to run.
              • Creating the Connector Archive file to bundle the Connector Type Definition file and the
              Extension Modules.
              • Importing the Connector Archive file in the Teiid Console.
              • Creating a Connector Binding using your Connector Type.

               

              I wish to know do we have to create the above mentioned files for the examples such as yahoo?

               

              Thanks,

              Akhil

              • 4. Re: SOLR  connector
                akhilkg1987

                Hi Ramesh,

                 

                There is one more thing that I wish to know,when we were working with Metamatrix before teiid,we actually had a SOLR CAF file through which we made the connector(though I was not a part of that process).

                 

                We still have that connector for SOLR,but it is for Metamatrix.

                 

                Can we use any part of that connector ? and how to ?

                 

                thanks,

                Akhil

                • 5. Re: SOLR  connector
                  rareddy

                  Akhil,

                   

                  First of all the references you made about connector from MetaMatrix Connector (CAF, connector type, extension modules etc..) does NOT apply to Teiid. Please refer to the Teiid documents not MetaMatrix docs for the Translator development.

                   

                  Metamatrix Connector =~ Teiid Translator + Resource Adapter

                   

                  Resource Adapter provides the connection semantics only and is defined as JCA resource adapter. We provide base framework to implement a Resource Adapter. Check out example of File resource adapter. These are under projects with name "connector" in them.  Defined in -ds.xml file. at runtime.

                   

                  Translator = Strictly a translation layer between the query engine and resource adapter (source). Translator name is defined inside the VDB.

                   

                  Now coming to the programing

                  - Resource adapter is completely different, as I said above you can use File as reference, however you can use your previous code from MM Connector for connection semantics here (if you need a connection, connection pool etc)

                   

                  - Translator is some what similar to MM's connector, but implements different classes and packed in a simple jar, with jboss-beans.xml file.  Once you figure out, you can use lot of code with respect to SQL to lucene conversion, capabilities code as is with minor modifications.

                   

                  The Yahoo Translator does not have any resource adapter, as resource adpter part is optional, here it makes a direct connection to yahoo.com and does not require a connection pool etc. I suspect your usecase falls into this category.  "isSourceRequired" method on ExecutionFactory class determines if a resource adapter supplied or not.

                   

                  Hope this helps.

                   

                  Ramesh..

                  • 6. Re: SOLR  connector
                    akhilkg1987

                    Hi Ramesh,

                     

                    You are always helpful,no questions on that.

                     

                    Actually i was referring to

                    http://docs.jboss.org/teiid/6.0/connector-developer-guide/en-US/html/federate_info.html

                     

                    and it is there that I found the steps I mentioned above.

                     

                    if that doucment isnt the correct one,then am I supposed to follow the following ?

                     

                     

                    http://docs.jboss.org/teiid/7.0.0/developer-guide/en-US/html/introduction.html

                     

                    Apart from this,I wanted to know that are Connector-file and translator-file the basic projects showing the structure of a custom connector?

                     

                     

                    I have imported  Connector-ws and translator-ws   in a project in JAVA perspective and I dont know what to next..how to use these examples?

                     

                    I just hope I do not have to write a lot of fresh code for teiid

                     

                    Thanks

                    Akhil.

                    • 7. Re: SOLR  connector
                      rareddy

                      Akhil,

                       

                      Well, it depends on which version of the Teiid you are writing Translator to. Teiid 6.0 almost two years old, the tech is between 6.x line and 7.x line is much different. Read though docs, blogs if you need to understand what is all changed.

                       

                      The current version as of today is 7.5. For that developing a translator for that version, take a look at

                       

                      http://docs.jboss.org/teiid/7.5.0.Final/developer-guide/en-US/html_single/

                       

                      Apart from this,I wanted to know that are Connector-file and translator-file the basic projects showing the structure of a custom connector?

                      Yes, for file based translator. It is a very basic translator.

                       

                       

                      I have imported  Connector-ws and translator-ws   in a project in JAVA perspective and I dont know what to next..how to use these examples?

                       

                      Read, understand and write for the SOLR translator. Once you look though the code deep enough they are not much different from what you currently have. The class names and methods have been moved to different places, but essentially they are doing same functionality as before. If you have specific question I can answer, if there is some thing lacking in the documentation I can fix.

                       

                       

                      Ramesh..

                      • 8. Re: SOLR  connector
                        akhilkg1987

                        I have following doubts

                         

                        1)Do we need to define BASIC-MANAGED-CONNECTION-FACTORY CLASS (this class is inherited in Managedconnectionfactory class)or

                           I am missing some jar files

                         

                        2) I do not see connectionfactory class generated by default unlike managed connection factory and other classes.DO I need to

                        write that separately

                        I suspect i am missing some jars (may be teiid cdk or something)

                         

                        and is execution.java in teiid similar to syncQueryExecution in MMX?

                         

                        I have attached some screenshots of amateur work by me..please tell me if it looks correct..

                         

                         

                        solrconn.JPG

                         

                        cdk.JPG

                        in this screenshot is it required to give the path to the jar files??

                        and from where should i download the latest CDK??

                         

                        too many questions

                         

                         

                        Thanks

                        Akhil

                        • 9. Re: SOLR  connector
                          rareddy

                          Akhil,

                           

                          All the required classes are in

                           

                          1) teiid-common-core.jar

                          2) teiid-api.jar

                           

                          They are in the Teiid deployment (find them in <jboss-as>/server/<profile>/deployers/teiid directory). If you use Maven they are in jboss maven repository. Check out Teiid maven configuration for location. You can find information here http://community.jboss.org/docs/DOC-12955

                           

                          There is no more CDK. As I mentioned before the eclipse plugin you are using above with screen shots is dated, yet you continued to use it. Of course it does not generate the classes you are looking for and errors are expected.

                           

                          I advise that, copy of those projects I mentioned before, and start editing them to fit your needs rather than creating a new one.

                           

                          Ramesh..

                          • 10. Re: SOLR  connector
                            akhilkg1987

                            hi ramesh,

                             

                            I thought CDK is a requirement for developing a solr translator and connection adapter.

                             

                            If that isnt the case,what is the platform that is to be used.Should i use java perspective to edit the yahoo example and put in the code from MMX connector that I already have?

                             

                            I an unable to find similarity b/w the metamatrix connector abd yahoo,will look into a greater depth..

                             

                            and if I edit Yahoo example,in the last after all editing is done,how to club it with teiid designer?

                             

                            Thanks,

                            Akhil

                            • 11. Re: SOLR  connector
                              rareddy

                              I thought CDK is a requirement for developing a solr translator and connection adapter.

                               

                              If that isnt the case,what is the platform that is to be used.Should i use java perspective to edit the yahoo example and put in the code from MMX connector that I already have?

                              I am not sure where got that assumption from? It just a eclipse plugin. Yes, you can use java perspective, or even notepad if you are brave enough to use.

                               

                              and if I edit Yahoo example,in the last after all editing is done,how to club it with teiid designer?

                              Translator is to deployed into server, not in designer. You build a jar, and drop it in the deploy directory of jboss-as server for deploying.

                               

                              If you are current customer of MMX, may be you guys can get help from Redhat consulting services.

                               

                              Ramesh..

                              • 12. Re: SOLR  connector
                                akhilkg1987

                                Hi Ramesh,

                                 

                                Thanks a lot for reply.

                                 

                                Once we finish with writing the custom translator and pack it as a jar file and we will able to deploy the same in Teiid server.

                                 

                                But we would like to know how to integrate the custom translator in the Teiid Designer/JBDS, so that we can have the custom translator in the teiid designer for creating the CONNECTION PROFILE and DATA SOURCE(which will be required for designer/building the vdb). Or is there any way that we can deploy the vdb(which has source model connecting to the Solr) without having the CONNECTION PROFILE and DATA SOURCE for the custom translator of Solr.

                                 

                                We are stuck up in getting the custom translator in the teiid designer/JBDS. Please help us out.

                                 

                                Thanks & Regards,

                                Akhil.

                                • 13. Re: SOLR  connector
                                  rareddy

                                  Akhil,

                                   

                                  You would need to only care about JBDS integration, if only if you care to preview this stuff in the eclipse environment. If you do not care about that then there is no need for that integration.

                                   

                                  Once you deploy the translator in the server, translator is automatically available in the JBDS. A resource-adapter is not. A resource adapter is a JCA component, and it can not be deployed inside a eclipse environment as Eclipse does not support nor it has a container to manage JCA based connections. So, connection eclipse needs to be a custom connection-profile, which is way beyond scope of this thread. If you are not looking to import metadata from this source, then I do not think you need that either.

                                   

                                  Let's say once you model your vdb, in the vdb panel, it only asks you to enter the translator name and jndi-name of the source. Then create a source in jboss-as with the given JNDI name to the SOLR source with the resource adapter you developed and then deploy the VDB, and query.

                                   

                                  Ramesh..