[pcap-ng-format] TODO in pcap-ng specifications

[Jasper Bongertz]

Hi all,

I've just spent a little time in the specs and searched for all TODOs to
see what can be done about them. I have created a text document with my
thoughts, and maybe some of you can take a look at it and we can start a
discussion about it to get things going.

If nobody disagrees I will replace the simple TODO items (for the
examples mostly) in the SVN sometime end of this week.

Cheers,
Jasper
-------------- next part --------------
PCAPng TODOs
--------------------

This file lists all places that are marked with a "TODO" and my thoughts about them. I think a version 1.0 of the specs should have all of these TODOs worked out (or removed) and replaced with definitive values or definitions.


Section 2.5

Text: "TODO: mention for each option, if it can/shouldn't appear more than one time. The option list is terminated by a Option which uses the special 'End of Option' code (opt_endofopt)."

Options appearing more than once make the file parsing process more difficult since you can't assume that there will be only one value (or none at all) for each option, but on the other hand it gives greater flexibility, e.g. to store multiple comments in case a file was processed etc. This may be a way to get around adding more option fields each time someone needs to store additional information by just adding them. 
A problem might be that we will have to force writing options in the same order as they were read to prevent something like sequential comments getting out of order after writing the file again.

The TODO asks for each option if they can appear multiple times, so here are my suggestions:

opt_endofopt			only once, if at all
opt_comment				multiple
shb_hardware			multiple  
shb_os						multiple
shb_userappl			multiple
if_name						once
if_description		once
if_IPv4addr				multiple
if_IPv6addr				multiple
if_MACaddr				once
if_EUIaddr				once
if_speed					once
if_tsresol				once
if_tzone					once
if_filter					once
if_os							once
if_fcslen					once
if_tsoffset 			once
epb_flags					once
epb_hash					once
epb_dropcount			once
pack_flags			  once
pack_hash					once
ns_dnsname				multiple
ns_dnsIP4addr			multiple
ns_dnsIP6addr			multiple
isb_starttime			once
isb_endtime				once
isb_ifrecv				once
isb_ifdrop				once
isb_filteraccept	once
isb_osdrop				once
isb_usrdeliv			once

Regarding the shb options: I think multiple values can be used to allow reading and writing the file while setting new values without losing original values, e.g. allowing a trace file utility to keep the "dumpcap..." value for shb_userappl while adding a new one, documenting that the file had been processed further.


2.6 Data Format

Text: "The alignment bytes (marked in this document e.g. with "aligned to 32 bits") should be filled with zero bytes (TODO: is this requirement a good idea for the sake of performance / do we want to allow bogus bytes here?)."

I think asking for zero byte padding is good to make it clear that this is padding and avoiding security risks. If we allow arbitrary bytes here it can lead to information leakage in worst case scenarios because we don't know where these bytes where taken from. I don't think performance is that much of an issue here anyway (but I might be wrong).


Text: "TODO: the spec is not too consistent wrt how 64-bit values are saved. in the Packet blocks we clearly specify where the low and high 32-bits of a 64-bit timestamp should be saved. In the SHB we do use the endianess of the machine when we save the section length."

This has probably to stay how it is right now since the implementations out there use it already, unless we want to split the section length into two 32 bit values as well - I haven't seen any trace file yet that has anything else than the "not specified" value in it. If we leave it like it is we should remove the TODO text here.


Text: "TODO - Maybe we have to specify something more here. Is what we're saying enough to avoid any kind of ambiguity?."

I think it is good enough.


3.2 Interface Description Block

Text: "SnapLen: maximum number of bytes dumped from each packet. The portion of each packet that exceeds this value will not be stored in the file. (TODO: Is there a need to signal "no limit"?)"

If we need a signal for "no limit" I think it can be added in a later version of the specifications, so I would remove that TODO for now.


Text: "(TODO - It would be nice, to have a "invalid Interface ID" defined, e.g. 0xFFFFFFFF"

I guess we can just say "Use 0xFFFFFFFF to signal an invalid Interface ID" here. 


Text: "Interface Hardware EUI address (64 bits), if available. 	TODO: give a good example"

How about "0234:56FF:FE78:9ABC"?


Text: "Time zone for GMT support (TODO: specify better). 	TODO: give a good example"

I'm not sure what the time zone is for, but I guess it can be used to track in which time zone a trace was taken. That would allow to adjust the UTC saved in the time stamps to the local time of when the capture was made. I think the best way to specify this would be to just note the offset from UTC in minutes, like "UTC+60" (for a trace taking in Germany) or just "60". Storing it in minutes instead of hours takes care of offsets that are fractional, (e.g. Iran has a time zone of UTC+4.5)

For daylight saving time I'd just write the UTC offset including the additional minutes.

By the way, I just scanned the whole document to find the specification that all timestamps have to be written in UTC and didn't find it. Did I miss it, or is it really not specified at the moment? If so, this is probably a critical topic to add, isn't it?


Text: "The filter (e.g. "capture only TCP traffic") used to capture traffic. The first byte of the Option Data keeps a code of the filter used (e.g. if this is a libpcap string, or BPF bytecode, and more). More details about this format will be presented in Appendix XXX (TODO). (TODO: better use different options for different fields? e.g. if_filter_pcap, if_filter_bpf, ...)"

Maybe this is something for someone who is more specialized in the capture filter business. I'm not sure if we need different fields for this.


Text: "TODO: won't a if_tsoffset_low for fractional second offsets be useful for highly syncronized capture systems?"

Not sure about this either, but if_tsoffset is specified as having a length of 8 bytes. Can't we just change the spec for this to be an offset in nanoseconds? I doubt anyone needs to adjust timestamps beyond the range of what we have with a count of 64 bits worth of nanoseconds, and it would allow adjustements as finely as possible. By the way, is this just a value to be added, or can it be negative as well? In that case we need to specify this as an signed 64bit value...


3.3 Enhanced Packet Block

Text: "(TODO: the text above uses "first bit", but shouldn't this be "first byte"?!?)"

I think it is already fixed, so this TODO is deprecated. Or am I missing something here? 

An example for a hash could be "2 EC1D8797 / 3 456ec2177c101e3c2e996ec29a3d508e"

By the way, the text "In addition to the options defined in Section 2.5 and in the Packet Block, the following options are valid within this block:" should be replaced by "In addition to the options defined in Section 2.5, the following options are valid within this block:", because the PB specifies the same options as the EPB with a reference to the EPB (we got a loop here :-))

3.5 Packet Block

Text: "TODO: give a good example" - see 3.3.


3.6 Name Resolution Block

Text: "TODO: give a good example" (for IPv6 records)

How about: 20 01 0d b8 00 00 00 00 00 00 00 00 12 34 56 78. Yes, RFC 5952 forces shortening of zeros etc, but this is a byte representation inside a file format, so we write 16 bytes with zeros. This right here is in Hex, as usual for IPv6, so I think we should give an example like that, especially since people will recognize the leading 2001 easier than if we'd write it in decimal.

Text: "TODO: give a good example" (for optional DNS server IPv6 records)

How about the public google DNS: 20 01 48 60 48 60 00 00 00 00 00 00 00 00 88 88

Text : "TODO: Add an "Interface ID" option, if the name resolution is only valid for a specific interface?"

I don't think an option will help here, because a NRB keeps all name resolutions available on the system. If we want to write NRBs specific to interfaces we might need to extend the NRB header with a 32bit Interface ID field (which, in my opinion, isn't something for spec 1.0). By doing that we could write one NRB per interface in later versions, but then again we'd need an "all interfaces" ID to be able to write an NRB that is valid for all of them. Maybe not so good.

The other way to do it would be to have the NS records written with the interface ID as part of the record value, like "0 127 0 0 1 'localhost'" for a record belonging to interface ID 0. Still something for a later version of the spec.

Text: "TODO: Does it make sense to have two "optional mechanisms" (records vs. options) here?"

Yes, I think so. Records are the necessary information and will have at least one record (why bother writing an optional NRB otherwise?), options are nice to have and can be empty.


3.7.  Interface Statistics Block

Text: "TODO: give a good example" (for isb_starttime)

How about "97 c3 04 00 aa 47 ca 64 (Little Endian, decodes to 06/29/2012 06:16:50 UTC)"

Text: "TODO: give a good example" (for isb_endtime)

How about	"96 c3 04 00 73 89 6a 65 (Little Endian, decodes to 06/29/2012 06:17:00 UTC)" 



Remaining TODOs: 

- 4.3.  Encryption Block (experimental)
- 6.  How to add Vendor / Domain specific extensions
- Appendix D.  Link Layer Headers

Thx for reading all of this!
Jasper


jasper.bongertz at flane.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4484 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20120724/bbe4183d/attachment.bin>