[pcap-ng-format] comments/edits on pcapng

[Guy Harris]

On Nov 10, 2015, at 5:05 PM, Michael Richardson <mcr+ietf at sandelman.ca> wrote:

> I reviewed draft-tuexen-opsawg-pcapng, while it hasn't had an -01, some work
> has been occuring in the github copy, and so the effort isn't dead, just not
> as visible.

The official version of the spec is, indeed draft-tuexen-opsawg-pcapng.xml at the GitHub repository.

> Tcpdump has been able to read pcapng files for awhile now, but still writes
> pcap format files.

...as is the case with other programs using libpcap.

The current capabilities are what can be done without any additions to the API.  At some point I plan to add additional APIs to expose the full capabilities of pcapng when reading captures and to write pcapng files.  The APIs for reading files will also support pcap files, so if a program such as tcpdump is changed to use the new APIs when available, it'll still handle pcap files.

> section 4.1 says this about major/minor:
> 
>        Major Version: number of the current mayor version of the
>              format. Current value is 1. This value should change if the format changes in
>              such a way that code that reads the new format could not read the old format
>              (i.e., code to read both formats would have to check the version number and
>              use different code paths for the two formats) and code that reads the old
>              format could not read the new format.
> 
>        Minor Version: number of the current minor version of the
>              format. Current value is 0. This value should change if the format
>              changes in such a way that code that reads the new format could read
>              the old format without checking the version number but code that
>              reads the old format could not read all files in the new format.
> 
> and I think that I don't buy it.
> [First, "major version" should have been started at "2" :-) ]
> 
> But, more importantly, we have an easily, and almost always incremenally
> extendable file format.  It's hard to imagine a change so drastic that it
> would be worth incrementing the major version number that would still leave

> enough of the file format intact that there would be any value left.

Well, one change that would be that drastic would be changing it from a format in which each block type has its own name space for options to a format in which there's a single global name space for options, and thus changing the option numbers.  That change would not only mean that old code would no longer be able to read the new files, it would also mean that code that handles the new files would need a different code path to handle the old files.

It would not, however, change the format of the blocks, for example, so the rest of the code, with the exception of the option processing code, would need no change (and the option processing code would only need to handle the renumbering of the options).

> One would also wind up changing magic/header numbers.

The main magic number is the byte-order magic number, which would only need to change if the format of the SHB changed.

> So I think that we should simply turn this into a 4-byte int32 field.  Let's
> just put the highest *RFC* number that must be understood in order to comprehend
> all the blocks present.  At approx. 3500 RFCs/10 years, 32-bits will last a
> long long long time... We could even do this with just the Minor Version and
> be good until rfc65535, which might be by oh year 2265 assuming the IETF
> continues at it's present rate.

So are you suggesting that updates to the format, such as the addition of new blocks, new options, new link-layer types, etc. would be done with RFCs?

That would fit with the use of IANA registries for block types/option values/link-layer header types/etc..

> On the Interface Description Block, (section: 4.2), the table shown should be
> an IANA controlled registry.

I.e., replace http://www.tcpdump.org/linktypes.html with an IANA registry?

> The reference to section 3.5 is initially
> confusing when reading.  I think that there should be a single table (in
> section 3.5).  I see that the Section Header Block also reuses codes used in
> the IDB:  I think that this is a mistake, the codes in a comment block should
> not be context dependant.  My pull request takes care of all of them, I
> hope.

Does your pull request also bump the current major version number?  Renumbering the options means that all the code out there that reads the current file formats won't be able to read the new format *and* code that *does* read the new format would need to interpret the option values differently based on the version number, so that sure sounds like a major version change.

I don't see what the problem is with different block types having different option name spaces.  Common code to handle options in all block types wouldn't be that useful - that code wouldn't, for example, be able to do anything useful with the interface speed option in a Name Resolution Block.  (And what do you mean by a "comment block"; we don't have blocks whose sole purpose is to contain comments, we have a comment option that can be used in any block, the content of which is solely intended for display to humans.)

> On the subject of the Encryption Block.  In discussion on the list many asked
> if we needed it.  Let me suggest that we do, but that yes, we need more
> information in it.

Well, the first question is what sort of information would be put in it.  The current spec just says it "is used to store encrypted data" and says of the data in it that "Once decrypted, it originates other blocks.", but it doesn't say what sort of stuff is being encrypted!  Lists of credit card numbers? :-)

> Now, about LinkType: I suggest that having libpcap/tcpdump continue to
> maintain the table is not ideal.  Instead, let's create an IANA registry for
> it, populate it with existing values, and make the libpcap group the Expert
> Review.

So where would the actual formats be made available?

If we look at, for example, the snoop data link types registry:

	http://www.iana.org/assignments/snoop-datalink-types/snoop-datalink-types.xhtml#snoop-datalink-types-2

it just gives the code, a name for the type, and a reference, which links to RFCs as a reference.

If we were to publish an RFC for each linktype, or possibly incorporate a number of linktypes in a single RFC so as not to waste an entire RFC to say "IEEE 802.3 Ethernet" for type 1, that would work - in effect

	http://www.iana.org/assignments/pcap-datalink-types/pcap-datalink-types.xhtml

would replace the "LINKTYPE_ value" and a somewhat more human-oriented version of the "LINKTYPE_ name" column, and point to RFCs that provide information now in the "Description" column, of http://www.tcpdump.org/linktypes.html.

I don't think an RFC is required as a reference, so we wouldn't have to wait for an RFC to percolate through the process before making a new link-layer header type available, but that would leave open the question of what to use as a reference until the RFC comes out, and if you don't want www.tcpdump.org to store the description until it shows up in an RFC...

...well, actually I guess we could use the pcap-ng GitHub repository for that.