-
1. Re: how to hand large file in jbossesb FileGateway
edx Apr 13, 2010 1:21 AM (in response to yushanyuan)For that purpose I have my own solution based on smooks engine and ScheduledEventMessageComposer.
Smooks engine and SAX parser allows submitting for processing to action pipeline only fragments of the huge input files.
I can share my solution if it is interesting for you.
Edward
-
2. Re: how to hand large file in jbossesb FileGateway
yushanyuan Apr 13, 2010 4:32 AM (in response to edx)Hi, edward
thank you for your generous. i am sure interested in it very much. i can`t wait to see it .
-
3. Re: how to hand large file in jbossesb FileGateway
edx Apr 13, 2010 5:44 AM (in response to yushanyuan)My solution has been tested on JBossESB 4.7.
The class FileFetchExecute (see attachment) is used as event processor in scheduled-listener.
Example configuration for this listener is presented below:
...
<listeners>
<scheduled-listener
event-processor="edx.jaz.schedule.FileFetchExecute" maxThreads="1"
name="FileAFetchTrigger" scheduleidref="EveryMinute">
<property name="charset" value="latin2"/>
<property name="smooksConfig" value="smooks.config/ReportA.xml"/>
<property name="sourceDir" value="/usr/local/jboss/run/input"/>
<property name="suffix" value="A"/>
</scheduled-listener>
</listeners>
...
The listener fetches file with 'A' suffix from the given directory, and submits it to smooks engine for processing.
Smooks configuration file (relative path) is provided in smooksConfig parameter and might be as follows:
<?xml version="1.0" encoding="UTF-8"?><smooks-resource-listxmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd"xmlns:calc="http://www.milyn.org/xsd/smooks/calc-1.1.xsd" ><params><param name="stream.filter.type">SAX</param><param name="default.serialization.on">false</param></params><csv:reader fields="lastname,name" quote="~" separator=";" skipLines="1"/><resource-config selector="csv-record"><resource>org.milyn.delivery.DomModelCreator</resource></resource-config><calc:counter countOnElement="csv-record" beanId="counter" start="1"/><jms:router beanId="csv_record_as_xml" destination="/queue/RecordQueueGW" executeBefore="false" routeOnElement="csv-record"><jms:message deliveryMode="persistent" priority="4" timeToLive="0" type="TextMessage"><jms:correlationIdPattern>PATTERN</jms:correlationIdPattern></jms:message><jms:connection factory="ConnectionFactory"/><jms:session acknowledgeMode="AUTO_ACKNOWLEDGE" transacted="false"/><jms:jndi contextFactory="org.jnp.interfaces.NamingContextFactory" namingFactory="org.jboss.naming:java.naming.factory.url.pkgs" providerUrl="jnp://localhost:1099"/><jms:highWaterMark mark="200" pollFrequency="1000" timeout="60000"/></jms:router><ftl:freemarker applyOnElement="csv-record"><ftl:template>/smooks.config/record_A.ftl</ftl:template><ftl:use><ftl:bindTo id="csv_record_as_xml"/></ftl:use></ftl:freemarker></smooks-resource-list>The smooks submits the fragments to JMS queue which is a gateway to other service deployed on ESB.
In the above smooks configuration I used Freemarker to form XML fragments that is send to JMS for further processing.
It can be as below example:
<?xml version="1.0" encoding="utf-8"?>
<#escape x as x?html>
<#assign rec = .vars["csv-record"] /> <#-- special assignment because csv-record has a hyphen -->
<datarecord xmlns="http://edx/jaz/DataModel" recno="${counter?string("0")}">
<lastname>${rec.lastname}</lastname>
<name>${rec.name}</name>
</datarecord>
The FileFetchExecute as a result of processing submits only statistic data to action pipeline of the service.
All XML fragments are sent to other service via JMS gateway.
Hope it will suitable for you.
Regards,
Edward
Note: RecordType - is enumeration for 'A','B', ... and Utilities.objectName(-s) stands for set of unique strings you can define as you wish.
-
FileFetchExecute.java.zip 3.4 KB
-
-
4. Re: how to hand large file in jbossesb FileGateway
yushanyuan Apr 14, 2010 9:30 PM (in response to edx)Hi, Edward
your reply is very helpful! it helped me sovle a big problem, but i have another problem in the large file processing. please image like this: just take the sample/quickstart/helloworld_file_action for example, i have a file bigger than 500MB, when i copy it to the input directory, it takes more than 30 seconds , during this process , the console throw a error:
----------------
org.jboss.soa.esb.listeners.message.MessageDeliverException: Invalid File payload. File 'F:\JAVA\ESB\jbossesb-server-4.5.GA\samples\quickstarts\helloworld_file_action\build\dirs\input\Fedora-12-i686-Live.iso.txt.esbInProcess' doesn't exist.at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:63)at org.jboss.soa.esb.listeners.gateway.LocalFileMessageComposer.compose(LocalFileMessageComposer.java:44)at org.jboss.soa.esb.listeners.gateway.AbstractFileGateway.onSchedule(AbstractFileGateway.java:139)at org.jboss.soa.esb.listeners.lifecycle.AbstractScheduledManagedLifecycle$1.onSchedule(AbstractScheduledManagedLifecycle.java:68)at org.jboss.soa.esb.schedule.SchedulerJob$ESBScheduledJob.execute(SchedulerJob.java:289)at org.quartz.core.JobRunShell.run(JobRunShell.java:203)at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)--------
derek
-
5. Re: how to hand large file in jbossesb FileGateway
edx Apr 15, 2010 1:48 AM (in response to yushanyuan)I have had the same problem when fetching remote files via FTP.
I solved it with another FTPFileFetchMessageComposer which
loads large file to the input directory but under temporary filename
which is filtered out (not having "A" sufix). When the transfer is done
the file is renamed to the pattern visible by the FileFetchExecute
message composer.
Edward
-
6. Re: how to hand large file in jbossesb FileGateway
yushanyuan Apr 15, 2010 1:53 AM (in response to edx)clever!
Thanks .