13 Replies Latest reply on Aug 7, 2011 9:39 PM by zhfeng

    NBF and all that.

    fhitchen

      I'm trying to understand how to use NBF to replace FML.

       

      The way FML is used is largely invisible to the developers in our implementation. Instead of an XSD a flat file describes the input and output buffers to a TUXEDO service...

       

      VIEW some-name

      $

      #type    cname                            fname                            count    flag     size     null   

      long     operator_id                      OPERATOR_ID                      1        -        -        -1061109568

      etc.

      END

       

      These can be converted to XSD without too much trouble, especially as all the buffers are flat.

       

      Now this bit I am not 100% sure about, but FML can then be used to take the layout files and produce code that does the marshalling and un-marshalling between the text based wire format and the actual C structure used in the TUXEDO service. This code is then called and the programmer works with a magically produced C struct with all the service paramters in it. On exit from the service the reverse is performed.

       

      How could this be acheived with NBF? The low level manipulaltion with btXattribute functions will be very tedious and error prone.

        • 1. Re: NBF and all that.
          zhfeng

          In actually, the NBF buffers is xml format on the wire.

          We use XSD to validate these buffers, and get type information of every element from schema which will been used in btXattribute functions.

           

          Yes, it must be very carefully to handle with the buffer string.

          Anyway, this xml buffer could be integrated with others, such as content based routing.

           

          Now, I'm working on using JBossESB to do CBR with NBF buffer.

          If you are interest , you can find a cbr example on blacktie trunk.

           

          Amos

          • 2. Re: NBF and all that.
            fhitchen

            I'm more interested in emulating legacy TUXEDO functions, i.e. XML to simple C struct with minimal overhead. Performance is key for my applications.

            • 3. Re: NBF and all that.
              tomjenkinson

              Hi Francis,

               

              Which tool do you use to convert the FML buffer into the C struct?

               

              Out of interest, can you explain why you needed to use FML and could not get away with using say X_C_TYPE as your buffer type instead? X_C_TYPE allows you to interact with structs by default.

               

              Tom

              • 4. Re: NBF and all that.
                fhitchen

                The FML library had a function Fvftos which takes the field view table and converts the FML buffer to a C struct.

                 

                We have Java and Sybase clients that need a neutral buffer format. Going forward we may drop Sybase but will need Java client support.

                • 5. Re: NBF and all that.
                  zhfeng

                  I'm trying to understand your application,

                   

                  in Java client you use FML buffer with JOLT, such as:

                  ...

                  fmlbuf.setString("NAME", "zhfeng");

                  fmlbuf.setINT("ID", "1234");

                  session.tpcall(...);

                   

                  in Server use C, such as:

                   

                  struct employee {

                       char name[32];

                       int id;

                  };

                  // this struct from view file such as

                  VIEW Employee

                  $

                  #type     cname          fbname     count     flag     size     null

                  string     name            -               1            -          32        -

                  long       id                  -               1            -          -          -

                  END

                   

                  void service(TPSVCINFO* rqst) {

                       FBFR* fml = (FBFR*) rqst->data;

                       struct employee buf;

                       Fvftos(fml, &buf, "employee");

                       ...

                       do something with buf

                  }

                   

                  you do not want to use fml function in server code as performance is key for your applications.

                  Is this clear ?

                  • 6. Re: NBF and all that.
                    fhitchen

                    In the Java code Jolt directly supports formatting the FML. We generate a jave fields mapping from the view that is also used by the C server side so you are mostly correct.

                     

                    As this is all mostly generated code driven off initial C struct's that are then converted to FML views which can easily be converted to XSD, what do you recommend as the most efficient way to replace the Jolt/FML layer in BlackTie?

                    • 7. Re: NBF and all that.
                      zhfeng

                      1. convert FML views to our buffer XSD

                      2. we have a tool to convert XSD schema to JavaBean, you can see example under

                          <BLACKTIE_SRC>/jatmibroker-xatmi/src/test/java/org/jboss/narayana/blacktie/jatmibroker/nbf/TestNBFToJavaBean.java

                          you can run mvn test -Dtest=TestNBFToJavaBean to see the generate JavaBean code in target/test-classes/EmployeeBuffer.java

                          you can consider this as a java fields mapping from the views

                      3. On the C server, you can only use btXattribute functions to access NBF buffer currently.

                       

                      In terms of performance, XML maybe not a good choice. I have some interest in google protoc buffer. If you think it is OK, we can work together to support this buffer in blacktie.

                      • 8. Re: NBF and all that.
                        tomjenkinson

                        Amos Feng wrote:

                         

                        3. On the C server, you can only use btXattribute functions to access NBF buffer currently.

                         

                         

                        It is our plan to support mapping structs to the XML wire format though. We plan to support this for X_COMMON, X_C_TYPE and NBF buffers.

                         

                        This means the programmer interacts with a struct, but on the wire it is XML so that it can be routed by ESB.

                        • 9. Re: NBF and all that.
                          zhfeng

                          Thanks Tom,

                           

                          I'm working on this issue Support transcoding of message payload to XML

                          • 10. Re: NBF and all that.
                            fhitchen

                            Tom, Amos,

                             

                            thanks for your inputs. I looked at the google protoc buffer, it does not support mapping to C structs, and I'm concerned that if we stick with XML on the wire, the parsing overhead will be too great. We call millions of TUX services a day and expect sub millisecond response times on some of them. The google doc quotes this statistic "When this message is encoded to the protocol buffer binary format (the text format above is just a convenient human-readable representation for debugging and editing), it would probably be 28 bytes long and take around 100-200 nanoseconds to parse. The XML version is at least 69 bytes if you remove whitespace, and would take around 5,000-10,000 nanoseconds to parse."

                            • 11. Re: NBF and all that.
                              tomjenkinson

                              Hi Francis,

                               

                              Thanks for reviewing the google buffer. It would be great to work with you on providing a suitably performant wire protocol implementation for BlackTie.

                               

                              Currently we support a binary wire format with zero compression (this could work faster than a compressed buffer format for certain classes of message as the compression algorithm is omitted).

                               

                              We also have the XML format which is more useful when you wish to take advantage of routing services for tpcall invocations as the ESB can easily parse the XML. This is what we are finishing off now.


                              We could introduce a third protocol which deals with larger buffers and compresses them for efficient transfer.

                               

                              If you would like to work with us on this and have some ideas on what you believe would be the most appropriate wire protocol, that would be excellent. BlackTie has a pluggable transport layer so the replacement of the buffer encoder is a relatively painless operation.

                               

                              Tom

                              • 12. Re: NBF and all that.
                                zhfeng

                                Hi Francis,

                                 

                                Have you take a look at protobuf-c (c binding for Google's Protocol Buffers) ?

                                 

                                Amos

                                • 13. Re: NBF and all that.
                                  zhfeng

                                  Hi Francis,

                                   

                                  We have raised a Jira for this https://issues.jboss.org/browse/BLACKTIE-379