Remote Protocol Specification

Table of Contents

Overview

The "remote" protocol is the protocol by which two Remoting 3 endpoints may be connected to communicate with one another.  The protocol requirements are:

1. Security: provide authentication and confidentiality support for clients and servers
2. Concurrency: allow multiple clients and requests to be active concurrently
3. Streaming: allow embedded streams
4. Efficiency: Make requests as compact as possible.
5. Performance: Provide equivalent or better latency characteristics than Remoting 2.

Message Format

Framing

Since the "remote" protocol is a message-oriented protocol, and TCP is a stream-oriented protocol, an additional level of framing is required.  The "remote" protocol uses a simple length-data framing scheme.

                      Bit #:
          7   6   5   4   3   2   1   0
  Byte# +---+---+---+---+---+---+---+---+
     0  |          Length (MSB)         |
        +-------------------------------+
     1  |          Length (LSB)         |
        +-------------------------------+
     2+ |            Data ...           |
        |                               |
        :                               :
        .                               .

Since every message has a predetermined length, the TCP_NODELAY socket option should always be employed to minimize message transmission latency on send.

Header

All regular messages in the "remote" protocol have a common structure.  They are all preceded by a message header, which associates the message with a specific request, client, or stream channel.

                      Bit #:
          7   6   5   4   3   2   1   0
  Byte# +---+---+---+---+---+---+---+---+
     0  | L | F | A | R | S.Ch.ID (MSB) |
        +---+---+---+---+---+---+---+---+
     1  |    Source Channel ID (LSB)    |
        +---+---+---+---+---+---+---+---+
     2  | X | X | D | E | D.Ch.ID (MSB) |
        +---+---+---+---+---+---+---+---+
     3  |  Destination Channel ID (LSB) |
        +-------------------------------+
     4  |            Data...            |
        :                               :
        .                               .

The meaning of each field is as follows:

• X - Reserved, set to 0 for forward compatibility.
• L - "Last", indicates that this message will be the last in the stream; once
• F - "First", indicates that this message is the first message in this direction for a channel pair.
• A - "Acknowledge", indicates that a previously received message should be acknowledged (see flow control for more information).
• R - "Refused", indicates that an incoming connect request was received for a channel ID that has no corresponding Listening or Accepting subsocket.
• D - "Data", a marker which indicates that the message contains a data message (with a length of zero or more).
• E - "End-of-data", a marker which indicates the end of a substream.
• Source Channel ID - the channel ID of the sender.
• Destination Channel ID - the channel ID of the recipient.

A subchannel is uniquely identified by a source/destination channel ID pair, not unlike how unique TCP connections are identified.  This means that more than one source channel ID can be connected to a destination channel ID, and there can be more than one connection with the same source ID, so long as the destination channel ID is different between each connection.

First and Last Markers

The First and Last markers on a message delineate the beginning and ending of a connection.  Once a Last marker is read on the remote side, no further data will be received on the channel.  When a Last marker is both sent and received, the associated channel ID pair becomes free for reuse.

End of Data Marker

The End-of-data marker does not have a significance to the overall protocol; it is used by the transport to mark the end of a message.  The bit will be set on the last packet of such a message.

Flow Control

Flow control is accomplished by two mechanisms.  The first is a negotiated limit on the number of allowed concurrent subchannels.  When this limit is reached, the peer interested in establishing a new subchannel is forced to wait.  If the peer ignores the subchannel limit, the connection may be terminated immediately.

The second mechanism is a fixed window size associated with each subchannel.  For a subchannel with a window size of N, each peer may send up to N messages on that subchannel without waiting for an "acknowledge" reply.  Further messages may not be sent until a message is received with the "A" bit set, at which time exactly one message is considered to be acknowledged.  If the window size for a subchannel is disregarded, the connection may be terminated immediately.

Such an acknowledgement may be sent in conjunction with a data message travelling in the opposite direction, or it may be sent as a standalone (dataless) message.

Subchannels and Subsockets

Subchannels are the heart of the protocol.  They allow the mulitplexing of multiple concurrent requests and client channels, as well as a means to tunnel bidirectional streams over the same connection.

Subchannels are identified by a unique channel ID pair.  This means that the maximum number of possible connected subchannels is equal to the number of channel IDs (4096) squared (16777216).

Subchannels work very similarly to TCP in that in order to establish a subchannel, you need a connecting side and an accepting side.  Each "side" of a subchannel (connected or unconnected) is called a subsocket.

Subsocket States

A subsocket may exist in one of several states:

• Established - A subsocket in this state has a source and destination channel ID.  Two way communication may occur on this subsocket until a Last message has been both sent and received.  This state may also be used to establish a connection with a remote subsocket which is in the Accepting or Listening state.
  
• Accepting - A subsocket in this state has a source channel ID.  Upon receiving a connect message (one with the First bit set) with this subsocket's channel ID as its destination ID, the source channel ID of the incoming message becomes the destination channel ID of the subsocket, and the subsocket transitions into the Established state.  The source channel ID then becomes available for other Accepting or Listening subsockets.
  
• Listening - A subsocket in this state works much like an Accepting subsocket, except that incoming connect message create new Established subsockets instead of replacing this subsocket, allowing many connections to be established to the same listener.

Due to the fact that Accepting and Listening subsockets only have a source channel ID, only 4096 such subsockets may exist on each side.

Security

Confidentiality

Confidentiality protection is implemented via TLS.  All TLS authentication options are supported, including but not limited to client authentication via TLS.  These connection parameters may be specified as options when the connection is created.

Authentication

Server authentication occurs naturally via the TLS server certificate validation process.  While clients may also be authenticated via this process, SASL-based authentication is also performed by the server.  If, for some reason, client authentication is not desired via SASL, then a NULL SASL mechanism should be selected.  If both client and server authentication is desired via SASL, then an appropriate mechanism supporting both should be chosen.  The authentication mechanism to be used is selected during Negotiation.

Negotiation

In order to reach agreement on what features should be available on the connection (selecting a marshaller, agreeing on parameters such as class tables and so forth), a negotiation phase is employed after authentication completes but before requests may be transferred.

Upon connection there is an Accepting subsocket dedicated to negotiation at channel ID 0 at each end of a connection.  Each side will send its negotiation parameters.  The negotiation parameters may be followed by zero or more SASL authentication messages.

Message Format

A negotiation message is formatted as follows:

                      Bit #:
          7   6   5   4   3   2   1   0
  Byte# +---+---+---+---+---+---+---+---+
     0  |            Command            |
        +---+---+---+---+---+---+---+---+
     1  |             Data...           |
        :                               :
        .                               .

The following commands are recognized:

• Negotiate - Contains negotiation parameters for this connection.  See next section for details.
• SASL Message - Message used to move SASL authentication phase ahead.
• Close - Initiate an orderly shutdown.

Once negotation is complete and authentication is accepted, each side will open the permanent Listening subsocket.  The Established 0-0 subchannel will remain open as a general control channel.