[ntar-workers] On Markers and Bookmarks

Gianluca Varenni gianluca.varenni at gmail.com
Tue Jul 12 05:43:54 GMT 2005 

----- Original Message ----- 
From: "Christian Kreibich" <christian at whoop.org>
To: "NTAR workers" <ntar-workers at winpcap.org>
Sent: Monday, July 11, 2005 6:14 PM
Subject: Re: [ntar-workers] On Markers and Bookmarks


> On Fri, 2005-07-08 at 08:57 -0700, Gianluca Varenni wrote:
>>
>> > mhmm unfortunately I cannot recall that discussion any more. Note 
>> > though
>> > that I don't mean arbitrary recursive nesting -- just packet blocks
>> > contained in a grouping block. This was on the assumption that it might
>> > be useful to have a grouping mechanism for a bunch of packets on a 
>> > finer
>> > granularity than sections. So in a sense a GoPBlock is similar to a
>> > marker block that is valid until the next marker, with GoPBlocks having
>> > the added benefit that you can potentially have end-of-GoP content more
>> > naturally.
>>
>> ...and the disadvantage that if the capture file is truncated, the entire
>> GoPBlock can be almost unusable (since you don't have the GoPBlock 
>> trailer).
>
> I think you would be in no worse a situation than without the GoP block
> trailer. Corruption yields a broken final block, whatever it is, likely
> a packet block. Corrupted blocks will have to be detected at a "low"
> level in the library anyway. I wouldn't put vital information in the
> trailer, only stats or whatever your app might willingly put there. Same
> for the Section End Block, btw. If it has stats and indexing structures,
> yay; if not, too bad, you'll have to seek.
>
> Reconstruction of corrupted files will involve a re-scan of the last
> section (as the baseline granularity of the file) anyway. This will be
> sped up while valid GoP blocks exist because they can be skipped as a
> whole, and slower for the last GoP. Packet counters could be re-
> established on the fly there.
>
> Would you feel better about optional end-of-GoP bocks? That might be an
> option, though I think Section End Blocks should be required.

Actually, I'm realizing now that maybe we have different concepts of 
grouping. I imagined a *strict* grouping of packets into GoP (i.e. a packet 
block cannot live outside a GoP, thus strict nesting). Instead maybe your 
vision is a more "relaxed" one: a section can groups packets using markers, 
using a Begin_GoP block and an End_GoP block. A good (i.e. easy to navigate) 
file will have packets surrounded by Begin/End_GoP blocks every now and 
then, but a file without Begin/End_GoP blocks, or with packets "floating" 
outside those blocks, is still well-formed.

Example of a well formed file:

SHB
---
IDB
---
PacketBlock
---
PacketBlock
---
...
EndOfFile


Example of a "good" file:

SHB
---
IDB
---
Begin_GoP
---
Packet
---
...
---
End_GoP
---
Begin_GoP
---
Packet
---
...
---
End_GoP
---
EndOfFile

Is this what you mean? In any case, I really love this idea.

>
> I do agree btw that in-situ file restoration is made harder by end-of-
> {GoP,Section} blocks. It's still doable though -- you overwrite a bit
> more at the end with new valid blocks (needn't be more than 12 bytes
> each -- a type and two null length fields) and mark the last block
> before that as unusable.

The only real issue I see in in-situ restoration is alignment. The spec 
clearly states that every block should be aligned to 32bit boundaries. We 
can have ntar (or any other implementation of pcap-ng) ignore this detail, 
create non-aligned "garbage" blocks to fix the truncated captures, and tweak 
the code to deal with "garbage" blocks differently (by spec, the block 
header contains the "useful" block length, that can be unaligned to 32 bits, 
but the actual block size is ALIGN_TO_32BITS(block_unaligned_size)), but in 
any case you have a file that cannot be made well-formed.


>
>> > Isn't that just the indexing we're discussing here (including Chema's
>> > navigation blocks proposal)?
>>
>> Not exactly. It's a (missing) feature by which you can
>> - ask the offset of a block relative to the section/file
>> - reuse such offset to open a specified block (something like
>> "ntar_open_block_at_offset()")
>
> Ah, I see. Thanks.
>
>> I'm still thinking on how to implement it with a clean and yet powerful
>> inteface (one of the problems is trying to open a block in a section that
>> it's different from the one you are actually in. Is it allowed? What are 
>> the
>> implications?).
>
> You could wrap the concept of an offset in an opaque handle. In the
> handle implementation, include a pointer to the section it belongs to,
> and in the function abort if an offset handle is passed in that doesn't
> match the session handle. Would that work? Kind of a poor-man's type
> checking?

It could work. The advantage of offsets is that they are integral values, 
usually returned by value (so no dynamic memory allocation is involved), or 
however allocated by the caller (i.e. you have something like 
offset=ntar_get_block_offset() or result=ntar_get_block_offset(..., 
&offset,...)), while an opaque handle (being opaque) is allocated by the 
callee, so you have something like offset_handle = 
ntar_get_block_offset_handle() and then 
ntar_close_offset_handle(offset_handle). If your app maintains several (= 
possibly thousands) handles, it can be troublesome (lots of mallocs or 
handle pools, possible leaks because the app forgets to close the handles).

In general, even a simple offset can be tested: when you open a section, you 
know its offset in the file and the length (well, the section length is 
optional), so you can easily check if
section_start < requested_block_offset < section_start + section_length.

Have a nice day
GV






>
>> > I think it doesn't matter too much as long as comparison operations 
>> > work
>> > universally.
>>
>> Yes, provided that the user does not try to compare indexing timestamps 
>> with
>> the packet timestamps directly...
>
> Yeah, that's true.
>
> Cheers,
> Christian.
> -- 
> ________________________________________________________________________
>                                          http://www.cl.cam.ac.uk/~cpk25
>                                                    http://www.whoop.org
>
>
> _______________________________________________
> ntar-workers mailing list
> ntar-workers at winpcap.org
> https://www.winpcap.org/mailman/listinfo/ntar-workers

More information about the ntar-workers mailing list