[pcap-ng-format] Proposed new Alternative Packet Block

[Guy Harris]

On Aug 13, 2016, at 6:43 PM, Serge Aleynikov <serge at aleynikov.org> wrote:

> Rationale
>    When working with very chatty protocols (e.g. financial market data)
>    it's important to reduce the overhead introduced by the file format
>    in order to conserve the overall file size (which for some financial
>    exchanges is measured in tens to hundreds of gigabytes per day).
>    
>    This proposal introduces the format that extends the option
>    definition to be able to store values in place of the option length
>    field, therefore reducing the size of the option code/value pair to
>    four octets.

So this different option format would apply *only* to this block, and this block would *not* support "local use" options, as the pcapng spec already says, in section 3.5 "Options":

	Option Type (2 octets): it contains the code that specifies the type of the current TLV record. Option types whose Most Significant Bit is equal to one are reserved for local use; therefore, there is no guarantee that the code used is unique among all capture files (generated by other applications), and is most certainly not portable.

so

	1) in other blocks, if the option type's high-order bit is set, it means "this is a local-use option", not "this is an option with a 24-bit value";

	2) in a block that uses these alternative options, if the high-order bit is set, it's a "short" option with the value in the remaining 24 bits, not a "local-use" option.

If you want to have a new option type that applies to *all* blocks, and to support local-use options in the Alternative Packet Block, you would have to pick another bit to indicate a "short" option.

The bit below the high-order bit isn't available, unless you treat the "do not copy this to an output file" custom options as special cases, as the option codes for those are 19372 and 19373, which have the bit below the high-order bit set.  No option code currently has the bit below the bit below the high-order bit (the 0x2000 bit) set.
      
> 3.5 Options
>    Proposed extention to Options includes a special option value setting
>    the highest bit of the "Option Code" to 1 and leaving remaining 7 bits
>    of space for 127 possible option codes.  The remaining 24 bits will be
>    used for storing the option's value:
> 
>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |1| Option Code |                  Option Value                 |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    
>    This layout saves 4 bytes for options with compact values, such as
>    the ones described in the following packet block.

The following packet block would be the *only* one that can use it, unless you pick a different bit.
> 
> 
> 5.1 Alternative Packet Block
>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +---------------------------------------------------------------+
>  0 |                    Block Type = 0x00000010                    |
>    +---------------------------------------------------------------+
>  4 |                      Block Total Length                       |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>  8 |                        Timestamp (High)                       |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 12 |                        Timestamp (Low)                        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 16 /                                                               /
>    /                      Options (variable)                       /
>    /                                                               /
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    /                                                               /
>    /                          Packet Data                          /
>    /              variable length, padded to 32 bits               /
>    /                                                               /
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                      Block Total Length                       |
>    +---------------------------------------------------------------+
>    
>    Captured Packet Length  -  Length of the packet 
>    
>    
> Options:
>    Name          Code  Length  Multiple?  Description
>    apb_opt_size  0x81  -       no         Total byte size of options

If that field is present, does the option list need to end with an opt_endofopt option?  There *shouldn't* be a need for it (in fact, there isn't a need for an opt_endofopt option in the current pcapng spec, as, for all the other blocks, the number of bytes of option can be calculated from the size of the block and the size - no need for a redundant terminator).

>    apb_capt_len  0x82  -       no         Packet's Captured Length

Wouldn't that be redundant, just as it is for the Simple Packet Block?  You have to figure out how many bytes of option there are before you can find the beginning of the packet data and, at that point, the original size of the packet is min(snapshot length, bytes left in the block).

BTW, presumably all these packets have arrived on interface 0, as there's no interface ID field.

>    apt_flags     0x84  -       no         Alternative Packet Block Flags
> 
> Alternative Packet Block Flags:

	...

>    4-11     Compression type
>                 0  = uncompressed
>                 1  = lzw
>                 2  = gzip
>                 3  = bzip2
>                 4  = zip
>                 5  = 7z
>                 6  = lzo
>                 7  = ucl
>                 8  = snappy
>                 ...

So that indicates the form of compression used on the packet data?

There's no FCS length field; do these packets have an FCS iff the "FCS length" field for interface 0's IDB is non-zero (so that all packets for that interface, whether they were transmitted by this host or received by this host, either have an FCS or don't have an FCS)?