[ntar-workers] Seekable file layouts etc

[Christian Kreibich]

On Thu, 2005-06-30 at 10:14 -0700, Loris Degioanni wrote:
>
> > - I don't think timestamps are the only search criterion that's handy. I
> > wouldn't be surprised if some applications would like "bookmarks" for
> > all kinds of things (and a timeframe of packets really is just
> > equivalent to two such bookmarks). I work a lot with IDSs and I know for
> > a fact that we'd often love to label the first packet of individual
> > flows as "suspicious", "scanner", "worm", etc. Being able to index those
> > as a component of the file format would rock.
> 
> This could be achieved easily through a "marker" block before the 
> suspiciuos.

Yes, that's true. My point was mainly that I'd like an indexing
structure to somehow point me quickly to those marked packets. In that
case, it'd be those marker blocks. 

> This is really cool, but opens a problem: what happens if the capture is 
> truncated before writing the SEB? One of the goals of pcap-NG is to 
> support concatenating trace files: What happens if a file is without the 
> SEB?

Note that if you have truncation you're screwed at the moment anyway
because per Figure 1 *all* blocks contain the block size (twice even,
once at the end -- I very much like that idea). So if you have
truncation, you won't be able to concatenate either, much like with
current pcap traces.

The last block will be unusable no matter what -- the only difference is
that with SEBs the significance of a garbled last block is greater.
However I presume the code could be made smart enough to discover
invalid SEB structures without much difficulty:

- seek to end of file
- use last 4 bytes as presumably correct block size
- seek backward by size
- check if first block size field is consistent with last one
- AND in my proposal, if it's an SEB-type block.

I always assume seeks in both directions are feasible -- but if they're
not then on such systems the whole idea of efficient lookups is somewhat
moot in my opinion, plus the traces can still be used in the normal
fashion.

> > At the very least, this proposal clearly gets the Best ASCII Art Award
> > so far.
> 
> I totally agree with that. My complments. :-)

Thank you. :^)

Cheers,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org