[ntar-workers] Seekable file layouts etc

[Alexander Dupuy]

I wrote:
>> when a packet header seemed to be garbage ... it would look backwards in 
>> the (saved) previous packet data searching for a file header.

>> It would be very nice if there was some similar sanity checking that could
>> be performed on NTAR packet block headers ... With the special magic for
>> SPB headers, resynchronization in this case should not be too difficult.

Gianluca Varenni replied:
> Uhm, can you elaborate on that? Do you mean adding something at the end of a
> block? At the moment every block has a trailer containing the block size; 
> this trailer can be used for both backward seeks in the file, and for sanity
> check.

I was thinking of something at the start of the block, so that you can 
determine sanity before you read the block data itself (as it may be long, 1-2K 
or even more with jumbo frames).  However, the trailing block size would be 
usable for this purpose, as you would use a valid length trailer from one block 
to validate the next one (for truncation detection).

> What do you mean by "special magic for SPB headers?" SPB is the simple 
> packet block (so a "normal" block), and it should have nothing to do with 
> synchronization.

Sorry, I meant SHB (section header block).

> In any case, it would be interesting to have some mechanism to resynchronize
> a tracefile obtained out of truncated captures, I don't know if it's so 
> easy (basically you need to find a new SHB where you were expecting the data
> of a block).

> Maybe the best idea in this case would be to have some sort of "recovery 
> mode" in ntar, where all "cool" features (backward seeks, random access to 
> blocks, indexes/markers) are disabled. You basically use this mode to take 
> the corrupted trace file and regenerate a good trace file (cutting out all 
> the garbage).

Christian Kreibich then responded.
> I think once you detect corruption you simply have to start from the last 
> valid block and do a byte-by-byte scan and test whether the parseable 
> sequence of blocks looks decent, per the above. From that you should be able
> to fix the block size fields in situ to restore correct sequencing (if you 
> don't want to duplicate a 5GB trace), or ...

An internal recovery mode would be useful.  This might best be handled by a 
state variable with three possible values: Unknown, Read, and Recovering.  The 
initial state would be Unknown, and it would be reset to Unknown after any API 
seek is performed, and would disable any attempt at recovery.  When in Unknown 
state, any successful block read would set the state to Read.  The Recovering 
state would be set when truncation is detected, and would disable all 
operations that perform file i/o, except for block/section/file close, and 
get_next_section/get_next_block.

Detection and recovery from an invalid (truncated) block, could be implemented 
using something like the following logic, without any need for seeks on the 
ntar file (i.e. this should work even when reading from a pipe):

Before block.c:read_raw_block_data returns success, it verifies non-truncation 
by checking that the trailing length is present and matches the leading length.
(This is an API change, currently trailer is verified only in ntar_read_section 
and ntar_close_block [close_block_read_mode]).

In Read state (only), if an invalid trailing length is present, its four bytes, 
and 11 additional bytes read from the file are appended to the block's raw data 
(the 11 bytes are needed in case the SHB has only been partially read, and 
represent 3 bytes of block type, 4 bytes block length, and 4 bytes byte-order 
magic).

The raw data is scanned forwards, looking for the SHB "\r\n\n\r" palindrome, 
and confirming that one of the possible byte-order magic numbers follows four 
bytes later.  If an SHB is detected in the raw data, state is set to 
Recovering, the data from the SHB palindrome to the end of the raw data is 
copied to a recovery buffer, and a special RECOVERABLE_TRUNCATION error is 
returned.  If no SHB is detected, this is an unrecoverable error, and a 
MISMATCH error is returned.

In Recovering mode, get_next_section and get_next_block read data from the 
recovery buffer instead of doing freads, until the recovery buffer has been 
exhausted, at which point the state is set to Unknown.

@alex
-- 
mailto:dupuy at counterstorm.com