6 Replies Latest reply on Apr 15, 2010 1:53 AM by yushanyuan

    how to hand large file in jbossesb FileGateway

    yushanyuan

      just as the title, by the way , i use jbossesb-4.5

        • 1. Re: how to hand large file in jbossesb FileGateway
          edx

          For that purpose I have my own solution based on smooks engine and ScheduledEventMessageComposer.

          Smooks engine and SAX parser allows submitting for processing to action pipeline only fragments of the huge input files.

          I can share my solution if it is interesting for you.

           

          Edward

          • 2. Re: how to hand large file in jbossesb FileGateway
            yushanyuan

            Hi, edward

             

               thank you for your generous. i am sure interested in it very much. i can`t wait to see it .

            • 3. Re: how to hand large file in jbossesb FileGateway
              edx

              My solution has been tested on JBossESB 4.7.

              The class FileFetchExecute (see attachment) is used as event processor in scheduled-listener.

              Example configuration for this listener is presented below:

              ...

              <listeners>

                  <scheduled-listener

                   event-processor="edx.jaz.schedule.FileFetchExecute" maxThreads="1"

                   name="FileAFetchTrigger" scheduleidref="EveryMinute">

                   <property name="charset" value="latin2"/>

                   <property name="smooksConfig" value="smooks.config/ReportA.xml"/>

                   <property name="sourceDir" value="/usr/local/jboss/run/input"/>

                   <property name="suffix" value="A"/>

                  </scheduled-listener>

              </listeners>

              ...

              The listener fetches file with 'A' suffix from the given directory, and submits it to smooks engine for processing.

              Smooks configuration file (relative path) is provided in smooksConfig parameter and might be as follows:

               

              <?xml version="1.0" encoding="UTF-8"?>
              <smooks-resource-list
                <params>
                  <param name="stream.filter.type">SAX</param>
                  <param name="default.serialization.on">false</param>
                </params>
                <csv:reader fields="lastname,name" quote="~" separator=";" skipLines="1"/>
                <resource-config selector="csv-record">
                  <resource>org.milyn.delivery.DomModelCreator</resource>
                </resource-config>
                <calc:counter countOnElement="csv-record" beanId="counter" start="1"/>
                <jms:router beanId="csv_record_as_xml" destination="/queue/RecordQueueGW" executeBefore="false" routeOnElement="csv-record">
                  <jms:message deliveryMode="persistent" priority="4" timeToLive="0" type="TextMessage">
                    <jms:correlationIdPattern>PATTERN</jms:correlationIdPattern>
                  </jms:message>
                  <jms:connection factory="ConnectionFactory"/>
                  <jms:session acknowledgeMode="AUTO_ACKNOWLEDGE" transacted="false"/>
                  <jms:jndi contextFactory="org.jnp.interfaces.NamingContextFactory" namingFactory="org.jboss.naming:java.naming.factory.url.pkgs" providerUrl="jnp://localhost:1099"/>
                  <jms:highWaterMark mark="200" pollFrequency="1000" timeout="60000"/>
                </jms:router>
                <ftl:freemarker applyOnElement="csv-record">
                  <ftl:template>/smooks.config/record_A.ftl</ftl:template>
                  <ftl:use>
                    <ftl:bindTo id="csv_record_as_xml"/>
                  </ftl:use>
                </ftl:freemarker>
              </smooks-resource-list>

               

              The smooks submits the fragments to JMS queue which is a gateway to other service deployed on ESB.

              In the above smooks configuration I used Freemarker to form XML fragments that is send to JMS for further processing.

              It can be as below example:

               

              <?xml version="1.0" encoding="utf-8"?>

              <#escape x as x?html>

              <#assign rec = .vars["csv-record"] /> <#-- special assignment because csv-record has a hyphen -->

              <datarecord xmlns="http://edx/jaz/DataModel" recno="${counter?string("0")}">

                   <lastname>${rec.lastname}</lastname>

                   <name>${rec.name}</name>

              </datarecord>

               

               

              The FileFetchExecute as a result of processing submits only statistic data to action pipeline of the service.

              All XML fragments are sent to other service via JMS gateway.

               

              Hope it will suitable for you.

              Regards,

              Edward

               

              Note: RecordType - is enumeration for 'A','B', ... and Utilities.objectName(-s) stands for set of unique strings you can define as you wish.

              • 4. Re: how to hand large file in jbossesb FileGateway
                yushanyuan

                Hi, Edward

                 

                    your reply is very helpful! it helped me sovle a big problem, but i have another problem in the large file processing. please image like this: just take the sample/quickstart/helloworld_file_action for example, i have a file bigger than 500MB, when i copy it to the input directory, it takes more than 30 seconds , during this process , the console throw a error:

                ----------------

                 

                 

                org.jboss.soa.esb.listeners.message.MessageDeliverException: Invalid File payload.  File 'F:\JAVA\ESB\jbossesb-server-4.5.GA\samples\quickstarts\helloworld_file_action\build\dirs\input\Fedora-12-i686-Live.iso.txt.esbInProcess' doesn't exist.
                at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:63)
                at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:44)
                at org.jboss.soa.esb.listeners.gateway.AbstractFileGateway.onSchedule(AbstractFileGateway.java:139)
                at org.jboss.soa.esb.listeners.lifecycle.AbstractScheduledManagedLifecycle$1.onSchedule(AbstractScheduledManagedLifecycle.java:68)
                at org.jboss.soa.esb.schedule.SchedulerJob$ESBScheduledJob.execute(SchedulerJob.java:289)
                at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
                at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

                 

                 

                 

                --------

                 

                 

                derek

                • 5. Re: how to hand large file in jbossesb FileGateway
                  edx

                  I have had the same problem when fetching remote files via FTP.

                  I solved it with another FTPFileFetchMessageComposer which

                  loads large file to the input directory but under temporary filename

                  which is filtered out (not having "A" sufix). When the transfer is done

                  the file is renamed to the pattern visible by the FileFetchExecute

                  message composer.

                   

                  Edward

                  • 6. Re: how to hand large file in jbossesb FileGateway
                    yushanyuan

                    clever!

                     

                    Thanks .