6 Replies Latest reply: Apr 15, 2010 1:53 AM by derek lee RSS

how to hand large file in jbossesb FileGateway

derek lee Newbie

just as the title, by the way , i use jbossesb-4.5

  • 1. Re: how to hand large file in jbossesb FileGateway
    Edward Pilipczuk Newbie

    For that purpose I have my own solution based on smooks engine and ScheduledEventMessageComposer.

    Smooks engine and SAX parser allows submitting for processing to action pipeline only fragments of the huge input files.

    I can share my solution if it is interesting for you.

     

    Edward

  • 2. Re: how to hand large file in jbossesb FileGateway
    derek lee Newbie

    Hi, edward

     

       thank you for your generous. i am sure interested in it very much. i can`t wait to see it .

  • 3. Re: how to hand large file in jbossesb FileGateway
    Edward Pilipczuk Newbie

    My solution has been tested on JBossESB 4.7.

    The class FileFetchExecute (see attachment) is used as event processor in scheduled-listener.

    Example configuration for this listener is presented below:

    ...

    <listeners>

        <scheduled-listener

         event-processor="edx.jaz.schedule.FileFetchExecute" maxThreads="1"

         name="FileAFetchTrigger" scheduleidref="EveryMinute">

         <property name="charset" value="latin2"/>

         <property name="smooksConfig" value="smooks.config/ReportA.xml"/>

         <property name="sourceDir" value="/usr/local/jboss/run/input"/>

         <property name="suffix" value="A"/>

        </scheduled-listener>

    </listeners>

    ...

    The listener fetches file with 'A' suffix from the given directory, and submits it to smooks engine for processing.

    Smooks configuration file (relative path) is provided in smooksConfig parameter and might be as follows:

     

    <?xml version="1.0" encoding="UTF-8"?>
    <smooks-resource-list
      <params>
        <param name="stream.filter.type">SAX</param>
        <param name="default.serialization.on">false</param>
      </params>
      <csv:reader fields="lastname,name" quote="~" separator=";" skipLines="1"/>
      <resource-config selector="csv-record">
        <resource>org.milyn.delivery.DomModelCreator</resource>
      </resource-config>
      <calc:counter countOnElement="csv-record" beanId="counter" start="1"/>
      <jms:router beanId="csv_record_as_xml" destination="/queue/RecordQueueGW" executeBefore="false" routeOnElement="csv-record">
        <jms:message deliveryMode="persistent" priority="4" timeToLive="0" type="TextMessage">
          <jms:correlationIdPattern>PATTERN</jms:correlationIdPattern>
        </jms:message>
        <jms:connection factory="ConnectionFactory"/>
        <jms:session acknowledgeMode="AUTO_ACKNOWLEDGE" transacted="false"/>
        <jms:jndi contextFactory="org.jnp.interfaces.NamingContextFactory" namingFactory="org.jboss.naming:java.naming.factory.url.pkgs" providerUrl="jnp://localhost:1099"/>
        <jms:highWaterMark mark="200" pollFrequency="1000" timeout="60000"/>
      </jms:router>
      <ftl:freemarker applyOnElement="csv-record">
        <ftl:template>/smooks.config/record_A.ftl</ftl:template>
        <ftl:use>
          <ftl:bindTo id="csv_record_as_xml"/>
        </ftl:use>
      </ftl:freemarker>
    </smooks-resource-list>

     

    The smooks submits the fragments to JMS queue which is a gateway to other service deployed on ESB.

    In the above smooks configuration I used Freemarker to form XML fragments that is send to JMS for further processing.

    It can be as below example:

     

    <?xml version="1.0" encoding="utf-8"?>

    <#escape x as x?html>

    <#assign rec = .vars["csv-record"] /> <#-- special assignment because csv-record has a hyphen -->

    <datarecord xmlns="http://edx/jaz/DataModel" recno="${counter?string("0")}">

         <lastname>${rec.lastname}</lastname>

         <name>${rec.name}</name>

    </datarecord>

     

     

    The FileFetchExecute as a result of processing submits only statistic data to action pipeline of the service.

    All XML fragments are sent to other service via JMS gateway.

     

    Hope it will suitable for you.

    Regards,

    Edward

     

    Note: RecordType - is enumeration for 'A','B', ... and Utilities.objectName(-s) stands for set of unique strings you can define as you wish.

  • 4. Re: how to hand large file in jbossesb FileGateway
    derek lee Newbie

    Hi, Edward

     

        your reply is very helpful! it helped me sovle a big problem, but i have another problem in the large file processing. please image like this: just take the sample/quickstart/helloworld_file_action for example, i have a file bigger than 500MB, when i copy it to the input directory, it takes more than 30 seconds , during this process , the console throw a error:

    ----------------

     

     

    org.jboss.soa.esb.listeners.message.MessageDeliverException: Invalid File payload.  File 'F:\JAVA\ESB\jbossesb-server-4.5.GA\samples\quickstarts\helloworld_file_action\build\dirs\input\Fedora-12-i686-Live.iso.txt.esbInProcess' doesn't exist.
    at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:63)
    at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:44)
    at org.jboss.soa.esb.listeners.gateway.AbstractFileGateway.onSchedule(AbstractFileGateway.java:139)
    at org.jboss.soa.esb.listeners.lifecycle.AbstractScheduledManagedLifecycle$1.onSchedule(AbstractScheduledManagedLifecycle.java:68)
    at org.jboss.soa.esb.schedule.SchedulerJob$ESBScheduledJob.execute(SchedulerJob.java:289)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

     

     

     

    --------

     

     

    derek

  • 5. Re: how to hand large file in jbossesb FileGateway
    Edward Pilipczuk Newbie

    I have had the same problem when fetching remote files via FTP.

    I solved it with another FTPFileFetchMessageComposer which

    loads large file to the input directory but under temporary filename

    which is filtered out (not having "A" sufix). When the transfer is done

    the file is renamed to the pattern visible by the FileFetchExecute

    message composer.

     

    Edward

  • 6. Re: how to hand large file in jbossesb FileGateway
    derek lee Newbie

    clever!

     

    Thanks .