[pcap-ng-format] [tcpdump-workers] sane maximum length for block_length

[Michael Richardson]

Guy Harris <gharris at sonic.net> wrote:
    > Currently, Wireshark's pcapng reading code imposes a block size limit
    > for all blocks:

    > enough that * the resulting block size would be less than the previous
    > 16 MiB limit.  */ #define MAX_BLOCK_SIZE (MIN_EPB_SIZE +
    > WTAP_MAX_PACKET_SIZE_DBUS + 131072)

    > WTAP_MAX_PACKET_SIZE_DBUS is 16 MiB.

So, MIN_EPB_SIZE (28?) + 16MiB + 128KiB.
I think that this is a fine maximum for quite a number of block types.
I propose to introduce sane maximums for each block type, on a block type basis.

    >> I'll put them clearly in a header, and they ought to be per
    >> block-type.  Is it reasonable for the internet-draft to say something
    >> about each type?

    > Given that we may get complaints in the future that GigantoBus messages
    > can be up to 1 GiB, so we need to increase the maximum packet size for
    > LINKTYPE_GIGANTOBUS, we can't provide a maximum EPB/PB/SPB size that
    > won't get in somebody's way in the future (other than the obvious
    > 2^31-1 - if *that's* a problem, we'd have to introduce pcapng 2.0, with
    > 64-bit block sizes and packet lengths, add new APIs, and, on 32-bit
    > platforms, just say "we don't support reading RidiculoBus captures")

Agreed.

    > Perhaps we could, as per the above arbitrarily say "don't have more
    > than 128 KiB of options" and "for now, don't have packets bigger than
    > 16 GiB", with a warning that the maximum packet size is subject to
    > change, and have recommended maxima based on the minimum block sizes
    > and that.  _______________________________________________

I can live with 16GiB as the *maximum* that we will allocate.
I'd like to put this in the draft: every block should have a *reasonable*
maximum.  I plan to work on a mmap() based reading API, and I that shouldn't
have a problem with block size on 64-bit systems.  But maybe on 32-bit
systems, it should use mmap() in some more creative way. I'm not sure
here. Are there any good libraries to outsource this problem?

I'd like to do an AIO (libuio) based writing API; apparently many have
problems doing >1GB/s captures with fwrite() and only one thread, but it
works great if they do AIO.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr at sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20190920/4cc0db75/attachment.sig>