[pcap-ng-format] Issue #24: Need to specify the IDB if_tzone option format

[Guy Harris]

On Aug 25, 2015, at 7:38 PM, Stephen Donnelly <stephen.donnelly at avagotech.com> wrote:

> On Wed, Aug 26, 2015 at 11:41 AM, Guy Harris <guy at alum.mit.edu> wrote:
> 
>> On Aug 25, 2015, at 3:50 PM, Stephen Donnelly <stephen.donnelly at avagotech.com> wrote:
>> 
>> The current spec says
>> 
>>         The timestamp is a single 64-bit unsigned integer representing the number of units since 1/1/1970 00:00:00 UTC.
>> 
>> It doesn't say "units *except for those evil leap seconds!!!!!111ONE!!!!!!*", so I'd say "it counts up once per unit, every unit" - i.e., *not* POSIX/Windows-style - is a valid interpretation.
> 
> You know the pcap code base better than I, how many of the input layers currently 'convert' the system/POSIX time into a monotonic counter? I thought they just shoved the packet 'system' time stamp in there as-is. This means the defacto implementation is already POSIX style.

Yes, but the spec doesn't say that, it speaks of a "number of units", which sounds more like a monotonic counter than like POSIX time.

>>> A monotonic count of seconds since an epoch is in practice what both TAI and GPS provide (there is a fixed offset between them due to different epochs). TAI is the default time base for PTP and IRIG-B clocks, so is often available where people really care about time stamp quality.
>> 
>> There's "time stamps" in the sense of "real-number coordinates on an axis with a given origin", and there's "time stamps" in the sense of "stuff a normal human would recognize as a date and time".
>> 
>> Most if not all capture files, including pcap and pcapng, provide time stamps that are approximations of the first of those senses; it's the capture file readers that turn them into values of the second type.
>> 
>> If both TAI and UTC were represented as seconds that have elapsed since their individual epochs, the difference between those representations would be a constant (the difference between their epochs, if any), unaffected by leap seconds, and either one would work as a monotonic count of seconds; it's attempting to tie a monotonic counter to the rotation of the earth that causes the problem.
>> 
>>> I was not suggesting changing the 'default' time stamp definition of pcapng, but rather adding an option for the interpretation to be e.g. 'TAI' rather than 'POSIX/Windows UTC' encoding for clarity.
>> 
>> Unfortunately, given the hardware and software on which most captures are done, it's probably not an interpretation that would allow a valid interpretation of the time stamp values.
>  
> Not sure what this means? I suggested allowing the capture system to indicate what interpretation to use, e.g. POSIX, Windows, 'monotonic UTC', TAI, GPS etc.

Presumably by "interpretation" you mean "time stamp format".

The characteristics of a time stamp format that represents absolute time as a count of units are:

	1) the origin of the time stamp - 1970-01-01 00:00:00 UTC for POSIX, 1601-01-01 00:00:00 "UTC" (proleptic Gregorian calendar, UTC extended back as well) for Windows, ??? for "monotonic UTC", 1958-01-01 00:00:00 (?) for TAI, 1957-12-31 23:59:41 (?) for GPS;

	2) whether the count of units counts positive leap seconds and *doesn't* count seconds removed by negative leap seconds - "no" for POSIX, "no" for Windows, "yes" for "monotonic UTC", "yes" for TAI, "yes" for GPS.

1) really isn't that interesting - we can just pick 1970-01-01 00:00:00 UTC and, if the time stamp ultimately comes from another source, just have it add or subtract the appropriate time offset before putting it into the file.

So that leaves 2), in which case there are only two choices:

	"no", which covers POSIX (and would *be* POSIX, with 1970-01-01 00:00:00 UTC as the epoch) and Windows (with the appropriate time shift of 11644473600 seconds);

	"yes", which would be "POSIX as it would have been had they done it the way I tried to get them to do it", and would also cover "monotonic UTC", TAI, and GPS, with the appropriate time shifts for the epoch.

> Only the display tool needs to be able to interpret into a human readable UTC or local timezone for display. Wireshark must already use on leap-second tables/timezone files for this if pcap/pcapng time stamps are monotonic as you suggest and not POSIX.

Wireshark just shoves the time stamps into localtime() or gmtime() and hopes for the best. :-)  What happens there depends on whether

	1) the time stamp counts leap seconds or not

and

	2) localtime() and gmtime() use the TZDB leap seconds information (which it might, on some systems configured to do so even though it breaks POSIX compatibility).

>> So, if we're to allow packet captures on UN*X and Windows systems, with OS-supplied time stamps, we'd have to find *some* way to describe POSIX-style time stamps.
>> 
>> If a "unit" is 1 second, then we could go with POSIX.
>> 
>> However, if it's less than 1 second, what happens during a leap second?  Does the counter keep ticking milliseconds or microseconds or nanoseconds or whatever, and then reset back to the beginning of the previous second?
> 
> That's precisely the problem I am trying to address by allowing the specification of a TAI time base instead of POSIX/System, so we could have a monotonic standard. The POSIX behavior is under-specified during leap seconds, implementations dependent (1). Some systems jump the clock back after the leap second, causing duplicates, while some 'pause' the clock during the leap second, messing up latency/bandwidth calculations. Some systems (Google) reportedly adjust their clock rate slightly over the next several hours/day, also messing up statistics and leading to loss of global synchronization.

I think calling it TAI rather than, say, "seconds since 1970-01-01 00:00:00 UTC, for realz, not those bogus POSIX so-called "seconds-since-the-epoch" might be a bit confusing.  I think we should just go with 1970-01-01 00:00:00 UTC, and allow an indication of whether leap seconds are counted or not, with a note that:

	if leap seconds *aren't* counted:

		you may not get the right answer if you subtract two time stamps;

		if you do POSIX-style conversion of time stamps to YYYY-MM-DD HH:MM:SS, you will not get the correct UTC or local time stamp for the packet if the packet was captured during a leap second, but you will get the same YYYY-MM-DD HH:MM:SS value that a POSIX or Windows system would have given (so it'll match time stamps in log messages if they're time stamped with POSIX or Windows time);

	if leap seconds *are* counted:

		you will get the right answer if you subtract two time stamps;

		if you do POSIX-style conversion of time stamps to YYYY-MM-DD HH:MM:SS, you will not get the right answer, although if you use the TZDB database with leap seconds and code that uses the leap second information you will get the right answer *even if the packet was captured during a leap second*.

If you have a time stamp source that doesn't use 1970-01-01 00:00:00 UTC as the epoch, and it's too costly to adjust each time stamp before writing it to the file, you can use the if_tsoffset option in the IDB to let the application that reads the file adjust time stamps to a 1970-01-01 00:00:00 UTC epoch.

And, as long as we're adding flags to describe the time stamp, we might want to add flags to indicate whether the time stamp is generated by the capturing host OS or by an adapter timer (or *something* to indicate how badly time stamps will be shifted by interrupt batching, interrupt response delays, networking stack delays, etc.) and perhaps whether that timer is synced with the host system timer (so you can compare packet time stamps with time stamps from event logging mechanisms).