A "Distributed Pcap" for Remote Monitoring LANs & WANs (Design Notes for the SITA ACN device) | Fulko Hew SITA INC Canada, Inc. Revised: October 2, 2007

### SUMMARY

Note: This document is part of the libpcap CVS and was derived from 'pcap.3' (circa Aug/07).

The ACN provides a customized/distributed version of this library that alows SMPs to interact with the various IOPs within the site providing a standard mechanism to capture LAN and WAN message traffic.

SMP | The Supervisory Management Processor where Wireshark (or equivalent) runs in conjuction with a libpcap front-end.
IOP | I/O Processors where the monitored ports exist in conjunction with a custom device driver/libpcap back-end.

Each IOP will be capable of supporting multiple connections from an SMP enabling monitoring of more than one interface at a time, each through its own seperate connection. The IOP is responsible to ensure and report an error if any attempt is made to monitor the same interface more than once.

There are three applications that will be supported by the ACN version of libpcap. They each use a slightly different mode for looping/capturing and termination as summarized in the following table:

Application | Capture | Termination
--- | --- | ---
wireshark | pcap_dispatch(all packets in one buffer of capture only) | pcap_breakloop()
tshark | pcap_dispatch(one buffer of capture only) | Since a CTRL-C was used to terminate the application, pcap_breakloop() is never called.
tcpdump | pcap_loop(all packets in the next buffer, and loop forever) | pcap_breakloop()

Note: In all cases, the termination of capturing is always (apparently) followed by pcap_close(). Pcap_breakloop() is only used to stop/suspend looping/processing, and upon close interpretation of the function definitions, it is possible to resume capturing following a pcap_breakloop() without any re-initialization.

#### ACN Limitations

1. Monitoring of backup IOPs is not currently supported.
2. Ethernet interfaces cannot be monitored in promiscuous mode.

### ROUTINES

The following list of functions is the sub-set of Pcap functions that have been altered/enhanced to support the ACN remote monitoring facility. The remainder of the Pcap functions continue to perform their duties un-altered. Libpcap only supports this mode of operation if it has been configured/compiled for SITA/ACN support.

pcap_findalldevs
pcap_freealldevs
pcap_open_live
pcap_close
pcap_setfilter
pcap_dispatch
pcap_loop
pcap_next
pcap_next_ex
pcap_stats

These subroutines have been modified for the ACN specific distributed and remote monitoring ability perform the following basic functions. More detail is provided in the "SMP/IOP Inter-Process Communication Protocol" section.

pcap_open_live() | Used to obtain a packet capture descriptor to look at packets on the network.
SMP -> IOP | The SMP will open a connection to the selected IOP on its 'sniffer' port to ensure it is available. It sends a null terminated string identifying the interface to be monitored.
IOP -> SMP | After any required processing is complete, the IOP will return a null terminated string containing an error message if one occured. If no error occured, a empty string is still returned. Errors are: "Interface (xxx) does not exist." "Interface (xxx) not configured." "Interface (xxx) already being monitored."
pcap_findalldevs() | It constructs a list of network devices that can be opened with pcap_open_live().
SMP | It obtains a list of IOPs currently available (via /etc/hosts).
SMP -> IOP | The SMP will sequentially open a connection to each IOP on its 'sniffer' port to ensure the IOP is available. It sends a null terminated empty interface ID followed by the query request command.
IOP -> SMP | The IOP returns an error response and its list of devices.
SMP -> IOP | The SMP closes the TCP connection with each IOP.
SMP | The SMP adds the received information to its internal structure.
pcap_freealldevs() | Used to free a list allocated by pcap_findalldevs().
SMP | The SMP frees the structure it built as a result of the previous invocation of pcap_findalldevs().
pcap_dispatch() | Used to collect and process packets.
SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use.
IOP -> SMP | The IOP now sends a stream of captured data.
SMP | The SMP will read the reverse channel of the connection between the SMP and the IOP that provides the captured data (via 'p->read_op' which is 'pcap_read_linux()' until the select() call returns a 'no more data' indication. It will the process (at most) the next 'cnt' packets and invoke the specified callback function for each packet processed.
IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP.
pcap_loop() | Is similar to pcap_dispatch() except it keeps reading packets until the requested number of packets are processed or an error occurs.
SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use.
IOP -> SMP | The IOP now sends a stream of captured data.
SMP | The SMP continuously reads the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via 'p->read_op' which is 'pcap_read_linux()' until 'cnt' packets have been received. The specified callback function will be invoked for each packet received.
IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP.
pcap_next() | It reads the next packet (by calling pcap_dispatch() with a count of 1) and returns a pointer to the data in that packet.
SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use.
IOP -> SMP | The IOP now sends a stream of captured data.
SMP | The SMP reads only the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via calling pcap_dispatch() with a count of 1) and returns a pointer to that data by invoking an internal callback.
IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP.
pcap_next_ex() | Reads the next packet and returns a success/failure indication.
SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use.
IOP -> SMP | The IOP now sends a stream of captured data.
SMP | The SMP reads only the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via calling pcap_dispatch() with a count of 1) and returns seperate pointers to both the packet header and packet data by invoking an internal callback.
IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP.
pcap_setfilter() | Used to specify a filter program.
SMP -> IOP | The SMP sends a 'set filter' command followed by the BPF commands.
IOP -> SMP | The IOP returns a null terminated error string if it failed to accept the filter. If no error occured, then a NULL terminated empty string is returned instead. Errors are: "Invalid BPF." "Insufficient resources for BPF."
pcap_stats() | Fills in a pcap_stat struct with packet statistics.
SMP -> IOP | The SMP sends a message to the IOP requesting its statistics.
IOP -> SMP | The IOP returns the statistics.
SMP | The SMP fills in the structure provided with the information retrieved from the IOP.
pcap_close() | Closes the file and deallocates resources.
SMP -> IOP | The SMP closes the file descriptor, and if the descriptor is that of the comminucation session with an IOP, it too is terminated.
IOP | If the IOP detects that its communication session with an SMP has closed, it will terminate any monitoring in progress, release any resources and close its end of the session. It will not maintain persistance of any information or prior mode of operation.

### SMP/IOP Inter-Process Communication Protocol

- Communications between an SMP and an IOP consists of a TCP session between an ephemeral port on the SMP and the well known port of 49152 (which is the first available port in the 'dynamic and/or private port' range) on an IOP.

- Following a TCP open operation the IOP receives a null terminated 'interface ID' string to determine the type of operation that follows:

- Every command received by an IOP implies a 'stop trace/stop forwarding' operation must occur before executing the received command.

- A session is closed when the SMP closes the TCP session with the IOP. Obviously monitoring and forwarding is also stopped at that time. Note: All multi-octet entities are sent in network neutral order.

  pcap_findalldevs() | SMP -> IOP | Open socket (to each IOP), and sends:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  Interface ID | 1 | A NULL to indicate an an empty 'interface ID'.
  IOP -> SMP | Send its (possibly empty) NULL terminated error response string.
  SMP -> IOP | Sends the 'interface query request':
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  Interface ID | 1 | A 'Q' (indicating 'interface query request').
  IOP -> SMP | The IOP returns a list of sequences of information as defined by the return parameter of this function call (as shown in the following table). Elements are specified by providing an unsigned byte preceeding the actual data that contains length information.
  Notes: | Name/ Purpose | Size (in bytes) | Description
  --- | --- | --- | ---
  | length | 1 | The number of octets in the name field that follows.
  Name | 1-255 | The name of the interface. The format of the name is an alphabetic string (indicating the type of interface) followed by an optional numeric string (indicating the interface's sequence number). Sequence numbers (if needed) will begin at zero and progress monotonically upwards. (i.e. 'eth0', 'lo', 'wan0', etc.) For an IOP, the alphabetic string will be one of: 'eth', 'wan', and 'lo' for Ethernet, WAN ports and the IP loopback device respectively. An IOP currently supports: 'eth0', 'eth1', 'lo', 'wan0' ... 'wan7'. Note: IOPs and ACNs will not currently support the concept of 'any' interface.
  length | 1 | The number of octets in the interface description field that follows.
  Interface Description | 0-255 | A description of the interface or it may be an empty string. (i.e. 'ALC')
  Interface Type | 4 | The type of interface as defined in the description for pcap_datalink() (in network neutral order).
  Loopback Flag | 1 | 1 = if the interface is a loopback interface, zero = otherwise.
  count | 1 | # of address entries that follow. Each entry is a series of bytes in network neutral order. See the parameter definition above for more details.
  Repeated 'count' number of times. | length | 1 | The number of octets in the address field that follows.
  Address | 1-255 | The address of this interface (in network neutral order).
  length | 1 | The number of octets in the netmask field that follows.
  Network Mask | 0-255 | The network mask used on this interface (if applicable) (in network neutral order).
  length | 1 | The number of octets in the broadcast address field that follows.
  Broadcast Address | 0-255 | The broadcast address of this interface (if applicable) (in network neutral order).
  length | 1 | The number of octets in the destination address field that follows.
  Destination Address | 0-255 | The destination address of this interface (if applicable) (in network neutral order).
  SMP -> IOP | Close the socket.
  IOP -> SMP | Close the socket.
  pcap_open_live() | SMP -> IOP | Open socket, and sends:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  Interface ID | 'n' | 'n' octets containing a NULL terminated interface name string.
  IOP -> SMP | Send its NULL terminated error response string.
  pcap_dispatch() pcap_loop() pcap_next() pcap_next_ex() | SMP -> IOP | On the first invocation following a pcap_open_live() or pcap_breakloop() additional information is sent:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  command | 1 | 'M' (indicating 'monitor start')
  snaplen | 4 | snaplen
  timeout | 1 | timeout value (in milliseconds)
  promiscuous | 1 | A flag indicating that the interface being monitored show operate in promiscuous mode. [off(0) / on(NZ)]
  direction | 1 | A flag indicating the direction of traffic that should be captuted [both(0) / in(1) / out(2)]
  IOP -> SMP | Sends captured packets.
  pcap_setfilter() | SMP -> IOP | At any time, the SMP can issue a set filter command which contains an indicator, a count of the number of statements in the filter, followed by the sequence of filter commands represented as a sequence of C-style structures.
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  command | 1 | 'F' (indicating 'filter')
  count | 4 | The number of command in the Berkeley Packet Filter that follow.
  BPF program | 'n' | 8 bytes of each command (repeated 'n' times). Each command consists of that C-style structure which contains:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  opcode | 2 | The command's opcode.
  'jt' | 1 | The 'jump if true' program counter offset.
  'jf' | 1 | The 'jump if false' program counter offset.
  'k' | 4 | The 'other' data field.
  Refer to the bpf(4) man page for more details.
  IOP -> SMP | In return the IOP will send its (possibly empty) NULL terminated error response string.
  pcap_stats() | SMP -> IOP | At any time, the SMP can issue a 'retrieve statistics' command which contains:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  command | 1 | 'S' (indicating 'request statistics')
  IOP -> SMP | In return the IOP will send:
  Name/ Purpose | Size (in bytes) | Description
  --- | --- | ---
  ps_recv | 4 | The number of packets that passed the filter.
  ps_drop | 4 | The number of packets that were dropped because the input queue was full, regardless of whether they passed the filter.
  ps_ifdrop | 4 | The number of packets dropped by the network inteface (regardless of whether they would have passed the input filter).
  pcap_close() | SMP -> IOP | At any time, the SMP can close the TCP session with the IOP.

### Interface ID Naming Convention

Each interface within an IOP will be referred to uniquely. Since an currently contains 8 monitorable WAN ports and a monitorable Ethernet port, the naming convention is:

Interface # | Type | Name
--- | --- | ---
1 | WAN | wan0
2 | WAN | wan1
3 | WAN | wan2
4 | WAN | wan3
5 | WAN | wan4
6 | WAN | wan5
7 | WAN | wan6
8 | WAN | wan7
9 | Ethernet | eth0
10 | Ethernet | eth1

### Packet Trace Data Format

The format of the trace data that is sent to the SMP follows a portion of the libpcap file format and is summarized here. This format specifies the generic requirements needed to be able to decode packets, but does not cover ACN specifics such as custom MAC addressing and WAN protocol support.

Although a libpcap file begins with a global header followed by zero or more records for each captured packet, trace data sent to the SMP does NOT begin with a global header. A trace sequence looks like this:

[Packet Header] | [Packet Data] | [Packet Header] | [Packet Data] | [Packet Header] | [Packet Data] | ...

#### Packet Header

Each captured packet starts with a header that contains the following values (in network neutral order):

     uint32 tv_sec;  /* timestamp seconds */
     uint32 tv_usec; /* timestamp microseconds */
     uint32 caplen;  /* number of octets in the following packet */
     uint32 len;     /* original length of packet on the wire */

tv_sec | The date and time when this packet was captured. This value is in seconds since January 1, 1970 00:00:00 GMT; this is also known as a UN*X time_t. You can use the ANSI C time() function from time.h to get this value, but you might use a more optimized way to get this timestamp value. If this timestamp isn't based on GMT (UTC), use thiszone from the global header for adjustments.
tv_usec | The microseconds when this packet was captured, as an offset to ts_sec. Beware: this value must never reach 1 second (1,000,000), in this case ts_sec must be increased instead!
caplen | The number of bytes actually provided in the capture record. This value should never become larger than len or the snaplen value specified during the capture.
len | The length of the packet "on the wire" when it was captured. If caplen and len differ, the actually saved packet size was limited by the value of snaplen specified during one of the capture directives such as pcap_dispatch().

#### Packet Data

The actual packet data will immediately follow the packet header as a sequence of caplen octets. Depending on the DLT encoding number assigned to the interface, the packet data will contain an additional custom header used to convey WAN port related information.

#### ACN Custom Packet Header

PCAP, Wireshark and Tcpdump enhancements have been added to the ACN to support monitoring of its ports, however each of these facilities were focused on capturing and displaying traffic from LAN interfaces. The SITA extentions to these facilities are used to also provide the ability to capture, filter, and display information from an ACN's WAN ports.

Although each packet follows the standard libpcap format, since there are two types of interfaces that can be monitored, the format of the data packet varies slightly.

- For Ethernet (like) devices, the packet format is unchanged from the standard Pcap format.
- For WAN devices, the packet contains a 5 byte header that preceeds the actual captured data described by the following table:

Octet | Name | Mask/Value | Definition
--- | --- | --- | ---
0 | Control / Status | xxxxxxx0 | Transmitted by capture device | (see 'Errors' octets)
xxxxxxx1 | Received by capture device
1xxxxxxx | No buffer was available during capture of previous packet.
1 | Signals | xxxxxxx1 | DSR asserted
xxxxxx1x | DTR asserted
xxxxx1xx | CTS asserted
xxxx1xxx | RTS asserted
xxx1xxxx | DCD asserted
xx1xxxxx | Undefined
x1xxxxxx | Undefined
1xxxxxxx | Undefined
2 | Errors (octet 1) |  | Tx | Rx
xxxxxxx1 | Underrun | Framing
xxxxxx1x | CTS Lost | Parity
xxxxx1xx | UART Error | Collision
xxxx1xxx | Re-Tx Limit Reached | Long Frame
xxx1xxxx | Undefined | Short Frame
xx1xxxxx | Undefined | Undefined
x1xxxxxx | Undefined | Undefined
1xxxxxxx | Undefined | Undefined
3 | Errors (octet 2) |  | Tx | Rx
xxxxxxx1 | Undefined | Non-Octet Aligned
xxxxxx1x | Undefined | Abort Received
xxxxx1xx | Undefined | CD Lost
xxxx1xxx | Undefined | Digital PLL Error
xxx1xxxx | Undefined | Overrun
xx1xxxxx | Undefined | Frame Length Violation
x1xxxxxx | Undefined | CRC Error
1xxxxxxx | Undefined | Break Received
4 | Protocol |
0x01 | - | LAPB (BOP)
0x02 | - | Ethernet 1
0x03 | - | Async (Interrupt IO)
0x04 | - | Async (Block IO)
0x05 | - | IPARS
0x06 | - | UTS
0x07 | - | PPP (HDLC)
0x08 | - | SDLC
0x09 | - | Token Ring 1
0x10 | - | I2C
0x11 | - | DPM Link
0x12 | - | Frame Relay (BOP)
Note 1: Ethernet and Token Ring frames will never be sent as DLT_SITA (with the 5 octet header), but will be sent as their corresponding DLT types instead.