[ntar-workers] Seekable file layouts etc

[Gianluca Varenni]

Hello Alex.


----- Original Message ----- 
From: "Alexander Dupuy" <alex.dupuy at counterstorm.com>
To: <ntar-workers at winpcap.org>
Sent: Wednesday, July 06, 2005 5:54 AM
Subject: Re: [ntar-workers] Seekable file layouts etc


> Gianluca Varenni writes:
>> Well, in any case if a trace file is truncated (for any reason, download
>> error, dumper app crashing....), it's not a valid pcap-ng file. If you
>> concatenate such a file with another (even correct) one, you end having
>> garbage.
>
> Unfortunately, truncated trace files are quite common, generally as a 
> result of leaving packet capture running until the filesystem fills up... 
> And, inevitably, those trace files get concatenated.
>
> I actually modified libpcap's savefile support to allow it to 
> resynchronize in cases like this; basically, when a packet header seemed 
> to be garbage (capture length > packet length, non-zero timezone offset 
> field [never used by any version of libpcap/tcpdump], or some other failed 
> sanity check) it would look backwards in the (saved) previous packet data 
> searching for a file header.  (I also modified it to handle the special 
> case of concatenated non-truncated capture files, where a file header 
> appeared in place of a packet header, but it sounds like NTAR already 
> deals well with this.)

Yes. This is one of the features of the pcap-ng file format.

>
> This did not require a seek, or even special stdio buffering, it just used 
> the library buffers for packet header and data, so that the capture data 
> from the last packet was still available after reading the next packet 
> header.  This approach was not 100% reliable, but did allow recovery from 
> most cases of concatenated trace files.
>
> It would be very nice if there was some similar sanity checking that could 
> be performed on NTAR packet block headers (I suspect the best bet would be 
> a valid option field - either four bytes of zeros or a complete chain 
> ending in EOOB). With the special magic for SPB headers, resynchronization 
> in this case should not be too difficult.

Uhm, can you elaborate on that? Do you mean adding something at the end of a 
block? At the moment every block has a trailer containing the block size; 
this trailer can be used for both backward seeks in the file, and for sanity 
check.

What do you mean by "special magic for SPB headers?" SPB is the simple 
packet block (so a "normal" block), and it should have nothing to do with 
synchronization.

In any case, it would be interesting to have some mechanism to resynchronize 
a tracefile obtained out of truncated captures, I don't know if it's so easy 
(basically you need to find a new SHB where you were expecting the data of a 
block). Basically, apart from detecting the "jamming" in the tracefile, the 
library should be able to continue its work flawlessy; I'm thinking for 
example of seeks and random accesses to the file (which NTAR will support as 
soon as I find some time to work on that). Maybe the best idea in this case 
would be to have some sort of "recovery mode" in ntar, where all "cool" 
features (backward seeks, random access to blocks, indexes/markers) are 
disabled. You basically use this mode to take the corrupted trace file and 
regenerate a good trace file (cutting out all the garbage).

GV


>
> @alex
> -- 
> mailto:dupuy at counterstorm.com
> _______________________________________________
> ntar-workers mailing list
> ntar-workers at winpcap.org
> https://www.winpcap.org/mailman/listinfo/ntar-workers