[pcap-ng-format] Cleanup of the PCAP-NG spec

[Guy Harris]

On Nov 15, 2016, at 6:11 PM, Michael Richardson <mcr at sandelman.ca> wrote:

> I think that it just hasn't been a priority for libpcap to move towards
> writing pcapng by default.  That's what we need to get to.  This means
> knowing that libpcap 1.x can read pcapng format well enough that nobody will
> be excessively annoyed by a libpcap 2.x that writes in a different format by
> default.

(Well, I'm not sure I'd want to have a "2.0" release of libpcap, if it ended up getting a shared-library version number of 2.x.  The goal should be to keep the releases source-compatible and binary-compatible with previous releases, so that a program written to the libpcap 0.4 API will still work with the libpcap that fully supports pcapng, even if full support of all of pcapng's capabilities *in the application* requires API changes, and so that a program compiled against the libpcap in an older release of $OPERATING_SYSTEM will still run in a newer release that has a newer $OPERATING_SYSTEM.)

But, yes, we do need reading-API changes to provide full support of pcapng's capabilities, and changes to support writing pcapng.  Tcpdump should probably start out defaulting to pcap, with a -P flag to get it to write pcapng, just as Apple's version of tcpdump has.  At some point we could switch to defaulting to pcapng, although it probably  shouldn't switch to using -P to mean "write pcap", as that'd be too confusing.

I'd go with Apple's API, except that their libpcap changes are covered under the APSL, and I suspect that there would be OSes that wouldn't want to go with a libpcap with APSL code just as there would be ones that wouldn't want to go with a libpcap with GPL code.

>> I haven't seen a single big endian formatted file except for the
>> broken one from the Wireshark online PCAPng library in my life (yes, I
>> know, PacketCache does it ;-)), so as long as Intel CPUs are used in
>> all the capture devices we'll mostly see little endian (I may be wrong
>> here and it has nothing to do with CPU designs)
> 
> Yes, and while ARM can come in BE or LE (as a hardware design choice), it's
> mostly being designed as LE, from what I can tell.  I think MIPS systems are
> mostly BE, and there are lots and lots of MIPS processors out there inside
> core BFRs.

I.e., it is not the case that Intel CPUs are used in all the capture devices; *desktop and notebook computers* (perhaps with tiny exceptions) have little-endian x86 CPUs, and smartphones/tables have (probably) little-endian ARM CPUs, but servers and network devices might have other CPUs, such as possibly big-endian MIPS in networking equipment and big-endian SPARC, Power ISA, and z/Architecture in servers.

Both pcap and pcapng support "reader makes it right", i.e. the writer writes the file in their byte order and the reader byte-swaps it as necessary.  When pcap format was developed, back in the early 1990's, there were Sun SPARC workstations, SGI MIPS workstations, HP PA-RISC workstations, and IBM RS/6000 workstations, all big-endian, as well as DEC MIPS and ALPHA workstations, all little-endian.

In any case, the spec says either byte order can be used, and it's a bit late to undo that.  Wireshark handles it, libpcap's current code to read pcapng files handles it, and any code I write to support pcapng's full capabilities in libpcap will handle it.