[pcap-ng-format] Cleanup of the PCAP-NG spec

[Erik Hjelmvik]

2016-11-16 4:43 GMT+01:00 Guy Harris <guy at alum.mit.edu>:

> On Nov 15, 2016, at 6:11 PM, Michael Richardson <mcr at sandelman.ca> wrote:
>
> I'd go with Apple's API, except that their libpcap changes are covered
> under the APSL, and I suspect that there would be OSes that wouldn't want
> to go with a libpcap with APSL code just as there would be ones that
> wouldn't want to go with a libpcap with GPL code.
>
> >> I haven't seen a single big endian formatted file except for the
> >> broken one from the Wireshark online PCAPng library in my life (yes, I
> >> know, PacketCache does it ;-)), so as long as Intel CPUs are used in
> >> all the capture devices we'll mostly see little endian (I may be wrong
> >> here and it has nothing to do with CPU designs)
> >
> > Yes, and while ARM can come in BE or LE (as a hardware design choice),
> it's
> > mostly being designed as LE, from what I can tell.  I think MIPS systems
> are
> > mostly BE, and there are lots and lots of MIPS processors out there
> inside
> > core BFRs.
>
> I.e., it is not the case that Intel CPUs are used in all the capture
> devices; *desktop and notebook computers* (perhaps with tiny exceptions)
> have little-endian x86 CPUs, and smartphones/tables have (probably)
> little-endian ARM CPUs, but servers and network devices might have other
> CPUs, such as possibly big-endian MIPS in networking equipment and
> big-endian SPARC, Power ISA, and z/Architecture in servers.
>

Not all code is written in C by old school developers; please read the
erratasec blog post I referred to earlier. The code for my sniffer tools
(RawCap and PacketCache) can write capture files in any endianness
regardless of what endian the CPU is.

I'm arguing that the pcapng spec should not have to "worry" about what CPU
the code is running on or what programming language that is used to
implement the spec. The goal should be to make a simple and clear spec with
no ambiguity. As I hinted before, this whole 32-bit alignment is one
example where the spec is written specifically to support parsing the files
by loading the data into packed structs. A modern protocol or file spec
should not assume that this is how the data will be parsed. As an example,
please read this note from the current pcapng implementation regarding
alignment of 64 bit values:

"Please note: 64-bit values are not aligned to 64-bit boundaries. This is
because the file is naturally aligned to 32-bit boundaries only. Special
care MUST be taken when reading and writing such values. (Note also that
some 64-bit values are represented as a 64-bit integer in the endianness of
the machine that wrote the file, and others are represented as 2 32-bit
values, one containing the upper 32 bits of the value and one containing
the lower 32 bits of the value, each written as 32-bit integers in the
endianness of the machine that wrote the file. Neither of these formats
guarantee 64-bit alignment.)"

This is definitely not KISS, I can see so many ways a developer could screw
this up when writing a pcapng parser!


>
> Both pcap and pcapng support "reader makes it right", i.e. the writer
> writes the file in their byte order and the reader byte-swaps it as
> necessary.  When pcap format was developed, back in the early 1990's, there
> were Sun SPARC workstations, SGI MIPS workstations, HP PA-RISC
> workstations, and IBM RS/6000 workstations, all big-endian, as well as DEC
> MIPS and ALPHA workstations, all little-endian.
>
> In any case, the spec says either byte order can be used, and it's a bit
> late to undo that.


I agree it's way to late to change the PCAP spec, but is it really to late
for pcapng? Why not just change the pcapng spec and force all code to write
in one endianness in the future? I'm arguing that allowing any endian
generates more problems than it solves.

How many pcapng "compatible" tools can read a pcapng file with multiple
Section Header Blocks of mixed endian? In fact, my experience is that
several tools that claim to support pcapng don't even support big-endian
files (I have quite a lot of big endian ones).

Allowing mutiple secion header blocks with interchanging endian creates a
complexity that will lead to buggy code in many implementations. And this
is just one example of where the pcapng spec isnt supporting the KISS
principle. A full implementation of the current pcang spec will be much
more prone to contain bugs compared to the old PCAP spec. Also most
developers will probably not implement the whole spec, in fact you can read
most pcapng files by just implementing SHB, IDB and EPB. Chances are that
other block types will either not be implemented or remain untested (i.e.
byggy).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20161116/f4053fc9/attachment.html>