[pcap-ng-format] Proposal for storing decryption secrets in a pcapng block

[Jasper Bongertz]

Friday, October 5, 2018, 11:01:58 AM, Anders Broman wrote:
> -----Original Message-----
> From: pcap-ng-format <pcap-ng-format-bounces at winpcap.org> On Behalf Of Guy Harris
> Sent: den 5 oktober 2018 08:11
> To: Pcap-ng file format <pcap-ng-format at winpcap.org>
> Cc: Developer support list for Wireshark <wireshark-dev at wireshark.org>
> Subject: Re: [pcap-ng-format] Proposal for storing decryption secrets in a pcapng block

> On Sep 30, 2018, at 10:47 AM, Peter Wu <peter at lekensteyn.nl> wrote:

>> Requirements for block placement:
>> - No requirement. Producers are allowed to write the block anywhere.
>>  Disadvantages for consumers: requires a two-pass scan to collect  
>> secrets before they are used.
>> - Place secrets before the packet blocks that require them. Consumers  
>> can read and decrypt in one pass. Disadvantage: producers cannot  
>> always guarantee availability of secrets while writing the capture.
>> - Place a single secret block before the first packet block. Consumers  
>> can read and decrypt in one pass. Disadvantage: requires producers to  
>> post-process (rewrite) the capture file to insert secrets.

>>The third of those appears to be a special case of the second of those.  I don't see any need to require the secrets to be before the *first* packet block if the first packet block doesn't require the secret; presumably >"before the packet blocks that require them" just means "*somewhere* before the packet blocks that require them", which is *allowed* to be "before all packet blocks in the file" but not *required* to be "before all >packet blocks in the file".
>>
>>If the secret isn't available by the time the first packet requiring the secret for decryption is ready to be written to the capture, *somebody* will have to do some form of two-pass processing.
>>
>>The first option says the consumer must do so; that's inconvenient for a consumer doing one-pass processing (tcpdump, TShark without the -2 option), and isn't even really good for at least some consumers doing two->pass processing (Wireshark, TShark with the -2 option), because dissection is done on the first pass.

>>The second and third option require either the producer, or some post-processor, to write a new version of the file putting the secrets before the packets that require them.  The producer isn't necessarily responsible >for doing so; one might have tcpdump, or dumpcap (or some program using dumpcap, such as TShark or Wireshark) write out a capture with no secrets, and then have another program (a utility, or Wireshark after >having read in the file and then given the secret in question) write out a new file with the secrets early enough in the file ("before all the packet blocks" is probably the simplest implementation).
>>
>>A producer that *does* happen to have the secret available before seeing any packets that require the secret *could* write it directly.

> For the sake of the file format as such would it make sense to say that the
> decryption block can occur at any place in the file but if that's the case all
> file readers may not be able to do decryption without first
> Rearranging the file.

I concur. My strategy would be like this:
- try to write the decryption block right before the first packet block so that
it's easily found when scanning the header blocks (SDB, IDB, etc.)
- if that's not possible, try to write it before any packet block that requires it
- if that's still not possible, write it whenever you have the information, even
if it's the last block of the file

Thinking of that - since pcap-ng offers the wonderful option of reading blocks
backwards we could put them all at the end. That way any tool could read them
easily and has the information ready when starting to read packet blocks from
the beginning. The downside is that if the capture crashes the blocks are never
written, so maybe it's not that cool...