13 Replies Latest reply: Sep 29, 2011 11:10 AM by Ramesh Reddy RSS

SOLR  connector

akhil gupta Newbie

Hi,

Can somebody tell me how to connect to SOLR with teiid?the way we connect to sql or other platforms, I need to make a connection to SOLR.

 

Thanks

Akhil.

  • 1. Re: SOLR  connector
    Ramesh Reddy Master

    Akhil,

     

    What is SOLR? I never heard of this data source, can you give more details? If this has source has JDBC driver then you can connect to it using Teiid, if not there are other ways, but all of them require custom coding for your source.

     

    Ramesh..

  • 2. Re: SOLR  connector
    akhil gupta Newbie

    Hi ramesh,

    SOLR is actually an enterprise search platform which we are using to index the data.

    Solr is written in Java and runs as a standalone full-text search server within a servlet container such as tomcat and it uses the lucene java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs.

     

    And I was trying to download the connector development kit from http://community.jboss.org/wiki/TeiidConnectorDevelopmentEclipsePlugin but the link there isnt working with my SVN. seems like am getting stuck at every step

  • 3. Re: SOLR  connector
    akhil gupta Newbie

    Hi Ramesh,

    Am back to this work..

     

    I have following queries:

    How is a connector different from translator in teiid?
    in examples I found a translator folder and a connector folder too(source\teiid\connectors\sandbox)

    How to use yahoo translator example?


    Is there any thread/documentation where it has been discussed

     

    I was going through the connector development guide,looking for what do we need to do once the coding part for connector

    is done.

    There they have mentioned following steps:

    -->Creating a Connector Type Definition file that defines the properties required to initialize your
    connector.
    • Identifying the Extension Modules (jars and resources) required for the Connector to run.
    • Creating the Connector Archive file to bundle the Connector Type Definition file and the
    Extension Modules.
    • Importing the Connector Archive file in the Teiid Console.
    • Creating a Connector Binding using your Connector Type.

     

    I wish to know do we have to create the above mentioned files for the examples such as yahoo?

     

    Thanks,

    Akhil

  • 4. Re: SOLR  connector
    akhil gupta Newbie

    Hi Ramesh,

     

    There is one more thing that I wish to know,when we were working with Metamatrix before teiid,we actually had a SOLR CAF file through which we made the connector(though I was not a part of that process).

     

    We still have that connector for SOLR,but it is for Metamatrix.

     

    Can we use any part of that connector ? and how to ?

     

    thanks,

    Akhil

  • 5. Re: SOLR  connector
    Ramesh Reddy Master

    Akhil,

     

    First of all the references you made about connector from MetaMatrix Connector (CAF, connector type, extension modules etc..) does NOT apply to Teiid. Please refer to the Teiid documents not MetaMatrix docs for the Translator development.

     

    Metamatrix Connector =~ Teiid Translator + Resource Adapter

     

    Resource Adapter provides the connection semantics only and is defined as JCA resource adapter. We provide base framework to implement a Resource Adapter. Check out example of File resource adapter. These are under projects with name "connector" in them.  Defined in -ds.xml file. at runtime.

     

    Translator = Strictly a translation layer between the query engine and resource adapter (source). Translator name is defined inside the VDB.

     

    Now coming to the programing

    - Resource adapter is completely different, as I said above you can use File as reference, however you can use your previous code from MM Connector for connection semantics here (if you need a connection, connection pool etc)

     

    - Translator is some what similar to MM's connector, but implements different classes and packed in a simple jar, with jboss-beans.xml file.  Once you figure out, you can use lot of code with respect to SQL to lucene conversion, capabilities code as is with minor modifications.

     

    The Yahoo Translator does not have any resource adapter, as resource adpter part is optional, here it makes a direct connection to yahoo.com and does not require a connection pool etc. I suspect your usecase falls into this category.  "isSourceRequired" method on ExecutionFactory class determines if a resource adapter supplied or not.

     

    Hope this helps.

     

    Ramesh..

  • 6. Re: SOLR  connector
    akhil gupta Newbie

    Hi Ramesh,

     

    You are always helpful,no questions on that.

     

    Actually i was referring to

    http://docs.jboss.org/teiid/6.0/connector-developer-guide/en-US/html/federate_info.html

     

    and it is there that I found the steps I mentioned above.

     

    if that doucment isnt the correct one,then am I supposed to follow the following ?

     

     

    http://docs.jboss.org/teiid/7.0.0/developer-guide/en-US/html/introduction.html

     

    Apart from this,I wanted to know that are Connector-file and translator-file the basic projects showing the structure of a custom connector?

     

     

    I have imported  Connector-ws and translator-ws   in a project in JAVA perspective and I dont know what to next..how to use these examples?

     

    I just hope I do not have to write a lot of fresh code for teiid

     

    Thanks

    Akhil.

  • 7. Re: SOLR  connector
    Ramesh Reddy Master

    Akhil,

     

    Well, it depends on which version of the Teiid you are writing Translator to. Teiid 6.0 almost two years old, the tech is between 6.x line and 7.x line is much different. Read though docs, blogs if you need to understand what is all changed.

     

    The current version as of today is 7.5. For that developing a translator for that version, take a look at

     

    http://docs.jboss.org/teiid/7.5.0.Final/developer-guide/en-US/html_single/

     

    Apart from this,I wanted to know that are Connector-file and translator-file the basic projects showing the structure of a custom connector?

    Yes, for file based translator. It is a very basic translator.

     

     

    I have imported  Connector-ws and translator-ws   in a project in JAVA perspective and I dont know what to next..how to use these examples?

     

    Read, understand and write for the SOLR translator. Once you look though the code deep enough they are not much different from what you currently have. The class names and methods have been moved to different places, but essentially they are doing same functionality as before. If you have specific question I can answer, if there is some thing lacking in the documentation I can fix.

     

     

    Ramesh..

  • 8. Re: SOLR  connector
    akhil gupta Newbie

    I have following doubts

     

    1)Do we need to define BASIC-MANAGED-CONNECTION-FACTORY CLASS (this class is inherited in Managedconnectionfactory class)or

       I am missing some jar files

     

    2) I do not see connectionfactory class generated by default unlike managed connection factory and other classes.DO I need to

    write that separately

    I suspect i am missing some jars (may be teiid cdk or something)

     

    and is execution.java in teiid similar to syncQueryExecution in MMX?

     

    I have attached some screenshots of amateur work by me..please tell me if it looks correct..

     

     

    solrconn.JPG

     

    cdk.JPG

    in this screenshot is it required to give the path to the jar files??

    and from where should i download the latest CDK??

     

    too many questions

     

     

    Thanks

    Akhil

  • 9. Re: SOLR  connector
    Ramesh Reddy Master

    Akhil,

     

    All the required classes are in

     

    1) teiid-common-core.jar

    2) teiid-api.jar

     

    They are in the Teiid deployment (find them in <jboss-as>/server/<profile>/deployers/teiid directory). If you use Maven they are in jboss maven repository. Check out Teiid maven configuration for location. You can find information here http://community.jboss.org/docs/DOC-12955

     

    There is no more CDK. As I mentioned before the eclipse plugin you are using above with screen shots is dated, yet you continued to use it. Of course it does not generate the classes you are looking for and errors are expected.

     

    I advise that, copy of those projects I mentioned before, and start editing them to fit your needs rather than creating a new one.

     

    Ramesh..

  • 10. Re: SOLR  connector
    akhil gupta Newbie

    hi ramesh,

     

    I thought CDK is a requirement for developing a solr translator and connection adapter.

     

    If that isnt the case,what is the platform that is to be used.Should i use java perspective to edit the yahoo example and put in the code from MMX connector that I already have?

     

    I an unable to find similarity b/w the metamatrix connector abd yahoo,will look into a greater depth..

     

    and if I edit Yahoo example,in the last after all editing is done,how to club it with teiid designer?

     

    Thanks,

    Akhil

  • 11. Re: SOLR  connector
    Ramesh Reddy Master

    I thought CDK is a requirement for developing a solr translator and connection adapter.

     

    If that isnt the case,what is the platform that is to be used.Should i use java perspective to edit the yahoo example and put in the code from MMX connector that I already have?

    I am not sure where got that assumption from? It just a eclipse plugin. Yes, you can use java perspective, or even notepad if you are brave enough to use.

     

    and if I edit Yahoo example,in the last after all editing is done,how to club it with teiid designer?

    Translator is to deployed into server, not in designer. You build a jar, and drop it in the deploy directory of jboss-as server for deploying.

     

    If you are current customer of MMX, may be you guys can get help from Redhat consulting services.

     

    Ramesh..

  • 12. Re: SOLR  connector
    akhil gupta Newbie

    Hi Ramesh,

     

    Thanks a lot for reply.

     

    Once we finish with writing the custom translator and pack it as a jar file and we will able to deploy the same in Teiid server.

     

    But we would like to know how to integrate the custom translator in the Teiid Designer/JBDS, so that we can have the custom translator in the teiid designer for creating the CONNECTION PROFILE and DATA SOURCE(which will be required for designer/building the vdb). Or is there any way that we can deploy the vdb(which has source model connecting to the Solr) without having the CONNECTION PROFILE and DATA SOURCE for the custom translator of Solr.

     

    We are stuck up in getting the custom translator in the teiid designer/JBDS. Please help us out.

     

    Thanks & Regards,

    Akhil.

  • 13. Re: SOLR  connector
    Ramesh Reddy Master

    Akhil,

     

    You would need to only care about JBDS integration, if only if you care to preview this stuff in the eclipse environment. If you do not care about that then there is no need for that integration.

     

    Once you deploy the translator in the server, translator is automatically available in the JBDS. A resource-adapter is not. A resource adapter is a JCA component, and it can not be deployed inside a eclipse environment as Eclipse does not support nor it has a container to manage JCA based connections. So, connection eclipse needs to be a custom connection-profile, which is way beyond scope of this thread. If you are not looking to import metadata from this source, then I do not think you need that either.

     

    Let's say once you model your vdb, in the vdb panel, it only asks you to enter the translator name and jndi-name of the source. Then create a source in jboss-as with the given JNDI name to the SOLR source with the resource adapter you developed and then deploy the VDB, and query.

     

    Ramesh..