gnat_flow

Under Development

gnat_flow is currently under active development and is not ready for production use. The API, command-line options, and output format may change without notice. Use at your own risk.

Synopsis

A high-performance YAF-like flow meter for generating network flow records from packet captures.

Description

The gnat_flow tool is a network flow meter written in Rust, based on CERT NetSA YAF (Yet Another Flowmeter). It captures network packets and generates bidirectional flow records in IPFIX-compliant JSON format.

Key features:

• Packet capture via libpcap - Live interface capture or pcap file reading
• High-performance AF_PACKET capture - Linux TPACKET_V3 with mmap ring buffer for 10Gbps+ capture
• Fragment reassembly - Reassembles fragmented IP packets before flow processing
• Flow generation - 5-tuple + VLAN flow key (matching YAF's default behavior)
• Bidirectional flow assembly - Forward and reverse direction tracking
• Configurable timeouts - Idle and active timeout support
• nDPI integration - Application protocol detection (optional)
• IPFIX-compliant JSON export - Field names follow RFC 5102 naming conventions

Usage

Basic Usage

# Read from pcap file
gnat_flow -i capture.pcap -o flows.json

# Live capture with libpcap (requires root or CAP_NET_RAW)
sudo gnat_flow -i eth0 --live pcap -o flows.json

# High-performance capture with AF_PACKET (Linux only)
sudo gnat_flow -i eth0 --live afpacket -o flows.json

# With nDPI application labeling
gnat_flow -i capture.pcap --ndpi -o flows.json

# Pretty-printed JSON output
gnat_flow -i capture.pcap --pretty -o flows.json

Command Line Options

Usage: gnat_flow [OPTIONS] --in <INPUT>

Options:
  -i, --in <INPUT>              Input source: interface name or pcap file
  -o, --out <OUTPUT>            Output file (default: stdout)
      --live <TYPE>             Live capture type: pcap, afpacket
      --filter <FILTER>         BPF filter expression (pcap only)
      --idle-timeout <SECS>     Idle timeout in seconds [default: 300]
      --active-timeout <SECS>   Active timeout in seconds [default: 1800]
      --max-flows <COUNT>       Maximum flow table size (0 = unlimited)
      --max-payload <BYTES>     Maximum payload capture per direction
      --ndpi                    Enable nDPI application labeling
      --udp-uniflow <PORT>      UDP uniflow port (0=disabled, 1=all)
      --promisc-off             Disable promiscuous mode

Fragment Reassembly Options

Fragment reassembly is enabled by default. gnat_flow reassembles fragmented IP packets before processing them into flows.

      --no-frag                 Disable fragment reassembly
      --max-frag-tables <N>     Maximum fragment tables [default: 1024]
      --frag-timeout <SECS>     Fragment timeout [default: 30]

AF_PACKET Options (Linux only)

AF_PACKET with TPACKET_V3 provides significantly better performance than libpcap for high-speed capture scenarios.

      --afp-block-size <BYTES>  Ring buffer block size [default: 2097152]
      --afp-block-count <N>     Ring buffer block count [default: 64]
      --afp-fanout-group <ID>   Fanout group ID (0 = disabled)
      --afp-fanout-mode <MODE>  Fanout mode [default: hash]

Output Options

      --format <FORMAT>         Output format: json-lines, json-array
      --pretty                  Pretty-print JSON output
      --stats                   Print statistics at end
      --show-flows              Show exported flows on stderr
  -v, --verbose                 Verbose output (-vv enables --show-flows)
  -q, --quiet                   Quiet mode
      --list-interfaces         List available interfaces

Output Format

Field names follow IPFIX Information Element naming conventions from RFC 5102 and RFC 5103 (biflow export). Each flow is exported as a JSON object:

{
  "flowStartMilliseconds": "2024-01-15T10:30:00.000Z",
  "flowEndMilliseconds": "2024-01-15T10:30:05.123Z",
  "flowDurationMilliseconds": 5123,
  "ipVersion": 4,
  "protocolIdentifier": 6,
  "sourceIPAddress": "192.168.1.100",
  "sourceTransportPort": 54321,
  "destinationIPAddress": "93.184.216.34",
  "destinationTransportPort": 443,
  "vlanId": 100,
  "octetTotalCount": 1234,
  "packetTotalCount": 15,
  "reverseOctetTotalCount": 5678,
  "reversePacketTotalCount": 12,
  "initialTcpControlBits": "Ssa",
  "unionTcpControlBits": "Aa",
  "firstEightNonEmptyPacketDirections": "01100010",
  "producerConsumerRatio": -0.64,
  "payloadEntropy": 235,
  "reversePayloadEntropy": 228,
  "flowEndReason": "endOfFlow",
  "applicationName": "TLS"
}

IPFIX Information Elements

Field	IPFIX IE	Description
flowStartMilliseconds	IE #152	Flow start time
flowEndMilliseconds	IE #153	Flow end time
flowDurationMilliseconds	IE #161	Flow duration
ipVersion	IE #60	IP version (4 or 6)
protocolIdentifier	IE #4	IP protocol number
sourceIPAddress	IE #8/27	Source IP (v4/v6)
sourceTransportPort	IE #7	Source port
destinationIPAddress	IE #12/28	Destination IP (v4/v6)
destinationTransportPort	IE #11	Destination port
vlanId	IE #58	VLAN identifier
octetTotalCount	IE #85	Total bytes (forward)
packetTotalCount	IE #86	Total packets (forward)
tcpControlBits	IE #6	TCP flags
flowEndReason	IE #136	Why flow ended
payloadEntropy	CERT IE 35	Shannon entropy of forward payload (0-255)
reversePayloadEntropy	CERT IE 35	Shannon entropy of reverse payload (0-255)

Entropy Values

The entropy fields are Shannon entropy calculations on payload data (excluding headers), scaled from 0.0-8.0 bits/byte to 0-255:

Value	Meaning
>230	Likely compressed or encrypted data
~140	English text
Low values	Less information content, or zero-padded (SSL/TLS)

Examples

Fragment Reassembly Configuration

# Disable fragment reassembly
gnat_flow -i capture.pcap --no-frag -o flows.json

# Configure fragment timeout (default: 30 seconds)
gnat_flow -i capture.pcap --frag-timeout 60 -o flows.json

# Configure maximum fragment tables (default: 1024)
gnat_flow -i capture.pcap --max-frag-tables 2048 -o flows.json

High-Performance AF_PACKET Capture

# Basic AF_PACKET capture
sudo gnat_flow -i eth0 --live afpacket -o flows.json

# Configure ring buffer size (128MB total)
sudo gnat_flow -i eth0 --live afpacket \
    --afp-block-size 2097152 \
    --afp-block-count 64 \
    -o flows.json

# Enable fanout for multi-queue NICs
sudo gnat_flow -i eth0 --live afpacket \
    --afp-fanout-group 1 \
    --afp-fanout-mode hash \
    -o flows.json

Performance Tuning

For high-speed capture (10Gbps+):

1. Increase ring buffer size:

   --afp-block-size 4194304 --afp-block-count 128  # 512MB ring

2. Use fanout with multiple processes:

   # Run in separate terminals with same fanout group
   gnat_flow -i eth0 --live afpacket --afp-fanout-group 1 -o flows1.json
   gnat_flow -i eth0 --live afpacket --afp-fanout-group 1 -o flows2.json

3. System tuning:

   # Increase socket buffer sizes
   sudo sysctl -w net.core.rmem_max=134217728
   sudo sysctl -w net.core.rmem_default=134217728
   
   # Increase netdev budget
   sudo sysctl -w net.core.netdev_budget=600

Requirements

• Rust 1.70+ for building
• libpcap development headers (libpcap-dev)
• Optional: libndpi 4.0+ for application labeling

See Also

• gnat_import