[pcap-ng-format] Proposed new Alternative Packet Block

[Serge Aleynikov]

Hi,

I would like to propose a couple of modifications to pcap-ng
specification based on the rationale stated below.

Regards,

Serge


*Rationale*
   When working with very chatty protocols (e.g. financial market data)
   it's important to reduce the overhead introduced by the file format
   in order to conserve the overall file size (which for some financial
   exchanges is measured in tens to hundreds of gigabytes per day).

   This proposal introduces the format that extends the option
   definition to be able to store values in place of the option length
   field, therefore reducing the size of the option code/value pair to
   four octets.

   Additionally all values except for the timestamp are moved from the
   fixed portion of the block into options.

   The Options are placed before the packet data, so that some
   characteristics of the packet's data can be learned from options
   before reading the packet data (e.g. compression type, captured
   length).  If there are no options defined, there will be just one
   'opt_endofopt' option entry.

*3.5 Options*
   Proposed extention to Options includes a special option value setting
   the highest bit of the "Option Code" to 1 and leaving remaining 7 bits
   of space for 127 possible option codes.  The remaining 24 bits will be
   used for storing the option's value:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |1| Option Code |                  Option Value                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   This layout saves 4 bytes for options with compact values, such as
   the ones described in the following packet block.


*5.1 Alternative Packet Block*
    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +---------------------------------------------------------------+
 0 |                    Block Type = 0x00000010                    |
   +---------------------------------------------------------------+
 4 |                      Block Total Length                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 8 |                        Timestamp (High)                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
12 |                        Timestamp (Low)                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 /                                                               /
   /                      Options (variable)                       /
   /                                                               /
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   /                                                               /
   /                          Packet Data                          /
   /              variable length, padded to 32 bits               /
   /                                                               /
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Block Total Length                       |
   +---------------------------------------------------------------+

   Captured Packet Length  -  Length of the packet


Options:
   Name          Code  Length  Multiple?  Description
   apb_opt_size  0x81  -       no         Total byte size of options
   apb_capt_len  0x82  -       no         Packet's Captured Length
   apb_orig_len  0x83  -       no         Packet's Original Length
   apt_flags     0x84  -       no         Alternative Packet Block Flags

Alternative Packet Block Flags:
   Bits     Description
   0-1      Direction
                00 = unspecified
                01 = inbound packet
                10 = outbound packet
   2-3    Reception type
                00 = unspecified
                01 = unicast
                10 = multicast
                11 = broadcast
   4-11     Compression type
                0  = uncompressed
                1  = lzw
                2  = gzip
                3  = bzip2
                4  = zip
                5  = 7z
                6  = lzo
                7  = ucl
                8  = snappy
                ...
   12-19    Reserved
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20160813/8cb8904f/attachment.html>