[pcap-ng-format] Proposal for new block type: Flow Block (2 of 4)
Michael Haney
michael-haney at utulsa.edu
Thu Aug 27 06:19:41 UTC 2015
I'd like to propose the following new block structure for PcapNG:
<artwork>
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Block Type = 0xD5555555 |
+---------------------------------------------------------------+
| Block Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flow Type | IPFIX options |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
/ /
/ Src IP 4 bytes for IPv4; 16 bytes (4 words) for IPv6 /
/ /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Dst IP 4 bytes for IPv4; 16 bytes (4 words) for IPv6 /
/ /
/ /
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Src Port or ICMP Type | Dst Port or ICMP Code |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flow Start Time (high order) |
+---------------------------------------------------------------+
| Flow Start Time (low order) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flow Stop Time (high order) |
+---------------------------------------------------------------+
| Flow Stop Time (low order) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Flow ID = 20 bytes |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
/ /
/ Other Options (variable) /
/ /
+-------------------------------+-------------------------------+
| total_pkts_option = 0x0002 | total_pkts_length = 0x0004 |
+-------------------------------+-------------------------------+
| Number of Packets seen in this flow |
+-------------------------------+-------------------------------+
| total_bytes_option = 0x0003 | total_bytes_length = 0x0004 |
+-------------------------------+-------------------------------+
| Number of Bytes seen in this flow |
+-------------------------------+-------------------------------+
| end_of_options = 0x0000 | options_length = 0x0000 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Block Total Length |
+---------------------------------------------------------------+
</artwork>
Ideally, libpcap can be built to support IPFIX flow information collection,
and
the PcapNG file standard will be extended to fully support IPFIX-based
blocks in
the file. IPFIX templates, along with Cisco NetFlow v9, are widely
available
from a number of devices and there are tools designed specifically to
consume
this data. The full IPFIX model includes sensors, collectors, and many
extensions and specifications. IPFIX data is intended to be passed as the
payload of a network packet from a sensor to a collector. However, it could
also
easily be stored as block data in the binary PcapNG file format.
At this early stage, a rudimentary Flow Block structure is proposed,
specifically to support management of Encrypted Blocks. A Flow Block is
designed
to provide a very simple means to track the initial characteristics of a
network
flow and record these metadata and addressing elements along with a flow
identifier. At least ne of these blocks will be recorded for each newly
identified flow. This block type will contain the various elements of the
packet
used to uniquely identify the flow (e.g. the source and destination IP
addresses) and will provide a searchable way to identify the associated
encrypted packets in the recording. The Flow Blocks will provide the
appropriate
means of searching for network characteristics (e.g. by IP address) and
will
provide the flow ID or Key Serial Number, which can then be searched from
all
Encrypted Blocks or EPB FlowID Option in the file as an index.
Block Type: The block type proposed for the Flow Block is 0xD5555555 or as
it
appears in binary: 0b11010101010101010101010101010101. This is a number
chosen
that is above 0x80000000 indicating that it is for experimental use. This
number
is distinctively non-random-looking when examining the binary file, but is
an
otherwise arbitrary selection.
Flow Type and IPFIX Options: Flow type identifies what type of flow this
block
contains. IPFIX Options is a reserved field for future development,
intended to
allow the full support of IPFIX formatted blocks. This field could contain
IPFIX-compatible template options that determine what fields are available
in
this block, or these fields may all be implemented as block options. The
system
design presented here relies only on a simple method of determining flow
type:
summing the protocol numbers (e.g IP version + embedded layer-4 protocol).
This is an over-simplified approach to determining a flow type. In
particular,
there is a possibility of flow-type collisions. But it serves my current
purposes:
Flow Types (ip version + protocol):
ARP = 1
IPv4 = 4
IPv6 = 6
IPv4 + ICMP = 5
IPv6 + ICMPv6 = 7
IPv4 + TCP = 4 + 6 = 10
IPv6 + TCP = 6 + 6 = 12
IPv4 + UDP = 4 + 17 = 21
IPv6 + UDP = 6 + 17 = 23
IPv4 + SCTP = 4 + 132 = 136
IPv6 + SCTP = 6 + 132 = 138
Flow Options:
code 2 = "Total Packets"
code 3 = "Total Bytes"
Flow ID: The Flow Identifier is a unique number determined in the same way
the
Key Serial Number is chosen in DUKPT. There is a portion of this identifier
that
uniquely identifies the monitor station, the organization that is doing the
monitoring and the device being used. For example, this could include an
organizational identifier concatenated to the Ethernet MAC address of the
sniffing interface. The least-significant bits of the identifier should be
a
unique counter that will be incremented each time a new flow is identified.
As
mentioned, the DUKPT standard for credit card transaction hardware has a
scheme
with the most significant 59 bits associated with the specific terminal,
and the
least significant 21 bits associated with the transaction counter.
While the flow ID could be an arbitrary length, it is first helpful to
define it
as a multiple of 32 bits. This allows it to be used in multiple ways within
the
PcapNG file. In the method presented here, it appears as a field in the
body of
two different block types and as an option in a third. Secondly, it may be
a
requirement that the flow IDs are globally unique. There should be enough
bits
to uniquely identify a recording session by device within a given
organization
and perhaps across all organizations. There should likewise be enough bits
to
uniquely count each flow that is observed during that recording session or
by
that device. In the proposed method, the Flow ID is 20 bytes, and the first
two
are always 0x01A6 or 0x01A7 to provide the appropriate options for this
system.
The last 8 bytes are a unique counter, and the middle 10 bytes should
contain
the 6-byte MAC address of the sniffer, and a 4-byte unique organizational
identifier. While 20 bytes may seem like significant additional overhead
per
packet, this is still more efficient than keeping track of all the
addressing
information used to uniquely identify a flow. The first 12 bytes will
remain
static. A TCP flow over IPv6 would have 37 or more bytes to track it
uniquely:
16 for each IP address and 2 for each port number, plus 1 byte to
distinguish if
it is UDP, TCP, or SCTP ports or ICMP type and code that was tracked. 8
bytes
for a counter is the same space required to keep track of only 2 IPv4
addresses.
And as a counter, it is sufficiently large enough to account for the same
number
of flows as if every unique possible IPv4 address had a conversation with
every
other unique possible IPv4 address, including itself. This should therefore
be
considered an appropriate length for a flow identifier.
Flow Start and End Times: One of the most important metadata fields to
capture
when performing flow analysis is the time at which a flow occurred. The
start
time can easily be captured. As a packet is identified as belonging to a
unique
new flow, the time of its arrival marks the start of the flow. Start time
is set
once when the flow is created. The end time is always updated with every
packet
added to the flow tally, initially set the same time as the start time.
This
serves as a “last seen” timestamp which will be accurate if a flow is
stopped
for other reasons (e.g. timeout, memory release).
Flow Block Options: There are a number of different ways to track flows and
to
help flows actively or passively time out to completion. When a flow is
initially identified, it should immediately be written to the PcapNG file.
Any
block that follows it will be associated with the flow ID and should have
the
proper ID stored in the file ahead of these packets. However, as the flow
stats
are accumulating and these elements are being tracked in memory, either
when the
flow ends, times out, or the recording session ends, this accumulated data
should also be written to the PcapNG file. In order to take advantage of
the
extensible design of PcapNG, these statistics are implemented as options to
the
block, so that both the initial and final flow blocks can be defined by the
same
block type. Initial and final flow blocks are distinguished by the presence
of
options for various flow statistics or IPFIX fields.
Two metadata elements that are relatively important and relatively easy to
track
are packet count and byte count. The packet count can exist as a variable
of the
flow state that is incremented for each new packet identified with that
flow.
The byte count can be a variable increased by the size of each packet as it
is
captured. Combined with time, an analyst can later calculate the rates of
packets per second, bytes per second, and average bytes per packet, each of
which portray important characteristics of that particular network flow.
Regards,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winpcap.org/pipermail/pcap-ng-format/attachments/20150827/40e46240/attachment-0001.html>
More information about the pcap-ng-format
mailing list