[pcap-ng-format] Cleanup of the PCAP-NG spec

Tue Nov 15 23:26:26 UTC 2016

Tuesday, November 15, 2016, 10:46:51 AM, Erik Hjelmvik wrote:

> I have been a lurker on this list for a long time, but I feel it's
> time to speak up. I love the addition of support for multiple
> interface types and annotations in PCAP-NG. However apart from the
> Interface Description Block and opt_comment I find that most of the
> extra block types in PCAP-NG will prevent this new format from being
> as widely accepted/implemented as the legacy PCAP format is.

Acceptance is also slow because people can live with PCAP features in
many situations. Being able to capture on multiple interfaces and
comments are real cool, but my guess is the majority of captures is
still single interface and few people actually using comments. Wow, I
sound like a PCAP fanboy just now - and I'm really not :-)

> The legacy PCAP format adhered to the KISS principle (Keep it
> simple stupid), which was one of the reasons for why this file
> format prevailed and the others didn't. However, PCAP-NG
> unfortunately doesn't adhere to KISS. It has lots of block types and
> most of these block types will only be used by a very small clique
> of users/vendors. I also see requests for additional block types
> being discussed on this mailing list every now and then.
> Unfortunately there seems to be no discussion about removing unneeded block types!

There were discussions like that, but in the past it seemed difficult
to get to a point were everybody involved could agree on a common way
of proceeding. Which is why I asked for and would loved to have a
PCAPng working group gathering at Sharkfest US to "go whiteboard"
together and start solving issues one by one until we have a final 1.0
spec. This didn't work (maybe it was too short notice) but I'd love to
do that next time. The problem I see is getting all the relevant
people into the same place at the same time to do efficient work.

It is also quite possible that I'm seeing a problem where there isn't
one by doing this via email. It's just that nobody seems to drive the
specification work as a team lead - I would offer to do this, but have
kept myself back because Guy is so much more skilled in all things
PCAP/PCAPng :-)

> I would love to see a great cleanup of the PCAN-NG specification,
> if possible. I'd love to see those rarely used block types removed,
> including all the optional block types.

Agreed.

> In fact declaring a block type as optional doesn't solve anything.
> It's better to have a simple standard that can be fully implemented
> by all, you otherwise risk running into compatibility and
> interoperability problems later on. In fact designing a file type
> specification is very much like defining a protocol, which is why I
> recommend that you watch a presentation by Daniel Stenberg
> (developer of cURL) about the process behind designing the HTTP2
> specification. In this presentation Daniel says “you should never
> ever have optional parts”: https://youtu.be/3Bg5NZGsgTg?t=17m06s

This is something I'll have to think about for a bit - I see the point
but I'm not sure if that may reduce PCAPng's versatility too much. It
needs to be able to do more than PCAP and be flexible, but I agree
that that can make writing read/write code harder.

> I am also not very fond of the optional little/big endian part of
> the specification. Converting fields from one endlessness to the
> other is not a lot of work for the sniffing application, the
> bottlenecks of the sniffers are in completely different areas. I
> claim that forcing sniffers to write field values in well-defined
> endian will not affect their performance. But it will, on the other
> hand, make the code in the parsers much more readable. In the
> current PCAP-NG spec the Section Header Block actually has a 4 byte
> length field, which the parser doesn't know how to interpret since
> the endianness indicator (Byte-Order Magic) hasn't been read yet.

I haven't seen a single big endian formatted file except for the
broken one from the Wireshark online PCAPng library in my life (yes, I
know, PacketCache does it ;-)), so as long as Intel CPUs are used in
all the capture devices we'll mostly see little endian (I may be wrong
here and it has nothing to do with CPU designs)

> Robert Graham actually wrote a very good blog post about handling endianness in software yesterday:
> http://blog.erratasec.com/2016/11/how-to-teach-endian.html

> Robert also mentions that aligning values isn't really important
> for performance. My take on this is that it would have been better
> to try get a simple and clear PCAP-NG file specification than to
> force everything to be aligned to a 32-bit boundary.

> Sorry for the rant, but I'd really like to see a discussion about
> cleaning up the PCAP-NG spec rather than adding even more block types!

It's not really a rant though, Erik ;-)

So my proposal would be to find a process that leads to a cleaning up
of the specs to be useful. I have no idea how other software projects
do this (well, Linus seems to say "yarp" or "narp" on all things Linux
kernel), but I think we need something that allows us to move forward
even if not every single person agrees with the course of action.
Because I think that's what's holding us up right now - everybody
looking at everybody else to see if we all nod (and yes, I admit I'm
doing that myself)

Thoughts?

Cheers,
Jasper

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3283 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20161116/0b6f4414/attachment.bin>