[ntar-workers] [Ethereal-dev] Re: NTAR - PCAP next generation dump file format implementation

[Loris Degioanni]

If I understand well, what you ask is a way to "group" a set of packets 
to quickly navigate among them. In this case, ntar already has a 
solution to the problem: the file format subdivides the capture in 
sections, whose number and size is controlled by the calling program (or 
library, in case of libpcap) using the ntar_create_section() and 
ntar_close_section(). It is possible, for example, to close the current 
sections and create a new one after 10.000 packets. When reading the 
file ntar_get_next_section() or ntar_get_previous_section() can be used 
to navigate through the sections and reach the desired one.

Regarding advanced ways to speed-up Ethereal capture handling, this is 
the precise goal of the directory block 
(http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html#anchor15), 
which however is not been implemented yet.

Loris


ronnie sahlberg wrote:
> The draft describes two different blocks for holding a captured packet:
> 
> Packet Block: it contains a single captured packet, or a portion of it. 
> Simple Packet Block: it contains a single captured packet, or a
> portion of it, with only a minimal set of information about it.
> 
> 
> I often work with very very large capture files and often want to only
> extract a very small subset (packets captured between time X and time
> Y).
> This is very very slow with the current fileformats doe to the massive
> amount of data that has to be processed.
> 
> 
> I would propose to introduce a new type of block  and encapsulate both
> PacketBlock and SimplePacketBlock in it :
> 
> 
> 
> Packet
> ======
> This the block you refer to in the draft as PacketBlock. A Packet MUST
> be encapsulated inside a PacketBlock.
> 
> 
> SimplePacket
> ============
> This the block you refer to in the draft as SimplePacketBlock. A
> SimplePacket MUST be encapsulated inside a PacketBlock.
> 
> 
> Then introduce a new PacketBlock :
> 
> PacketBlock
> ===========
> A block that contains one or (preferrably) more Packet or SimplePacket
> structures.
> Format of PacketBlock (in units of 4 bytes):
> 
> Version: 1 byte,  version of packetblock structure.
> 
> Num packets: 3 bytes,  number of Packet or SimplePacket structures in
> this PacketBlock, or zero (==use offline tool to calculate this and
> populate the field)
> 
> Length : 8 bytes,  total length of this block in bytes.
> 
> TSofFirstPacket: timestamp of the first packet or simplepacket in this
> PacketBlock or zero (==run some offline tool to populate this field if
> you need it)
> 
> TSofLastPacket: timestamp of last packet or simplepacket in this
> PaketBlock or zero.
> 
> 
> 
> A PCAP writer should aggregate PacketBlocks/SimplePacketBlocks into
> blocks of approximately 10MByte.
> This default approximate block-size should be controllable by the user
> but i think 10MByte should be reasonable.
> 
> 
> 
> Assuming we have a 10GByte capture file, then this would make it possible to :
> 1, parse the cap file and collect all PacketBlock headers, timestamps
> and number of packets in each block.  fseek() to the next block
> instead of parsing it packet by packet.
> 2, present to the user a list :
> Block 1: start:x:y stop:x:y  packets:z
> Block 2: ...
> ...
> 3, Allow the user to ONLY read/extract a subset of the PacketBlocks in
> teh capture file.
> 
> 
> Having this kind of encapsulation structure would allow users to use
> large captures (and skip the portions that are of no interest) while
> the current design would require the user to read the entire 10GB
> file.
> 
> 1GB and larger capture files are becoming more and more standard to
> many users and reading through this massive amount of data is slow and
> painful.
> 
> 
> 
> On 6/25/05, Gianluca Varenni <gianluca.varenni at gmail.com> wrote:
> 
>>Hi all.
>>
>>This mail is to announce the birth of the NTAR project. NTAR stands for
>>Network Trace Archival and Retrieval library, and is an implementation of
>>the PCAP next generation dump file format, that was proposed and discussed
>>last year by several folks on the libpcap/tcpdump and WinPcap mailing
>>lists.
>>The library is released under the 3-clause/BSD license.
>>
>>The URL of the project is:
>>
>>http://www.winpcap.org/ntar
>>
>>On this website you can find
>>- the source file of the library (both for windows and *nix)
>>- the HTML documentation of the API (generated with doxygen from the
>>  commented source files), both for the user  and for someone wanting to
>>  extend it. The docs contain some tests/examples that you can look to
>>  get an idea of how the library works.
>>- An updated version of the PCAP draft specifying the file format.
>>  The original draft of the file format is available at
>>     http://www.tcpdump.org/pcap/pcap.html
>>
>>
>>
>>I'm seeking contributors to improve the library in terms of
>>- testing the library on different platforms and operating systems
>>- reviewing the API
>>- implementing new extensions to the library.
>>
>>A mailing list, ntar-workers at winpcap.org, has been created for NTAR-related
>>discussions. People interested in this project are welcome to join it, the
>>mailman web interface to subscribe is available at
>>
>>https://www.winpcap.org/mailman/listinfo/ntar-workers
>>
>>Have a nice day
>>Gianluca Varenni
>>
>>_______________________________________________
>>Ethereal-dev mailing list
>>Ethereal-dev at ethereal.com
>>http://www.ethereal.com/mailman/listinfo/ethereal-dev
>>
> 
> 
> _______________________________________________
> Ethereal-dev mailing list
> Ethereal-dev at ethereal.com
> http://www.ethereal.com/mailman/listinfo/ethereal-dev
>