Schema

The table details the IPFIX information elements generated by gnat_sensor and the corresponding Apache Parquet schema. The schema includes a wide range of features which enable advanced insights into network traffic patterns, anomalies, and security risks. More specifically, the schema includes fields for basic flow information, protocol details, traffic statistics, entropy and timing analysis, packet size analysis, flow classification, hardware addresses, geographical and ASN information, histograms, and deep packet inspection.

Field Definitions

Basic Flow Information

---

Field	Type	Description
version	UINTEGER	Schema version number for the flow record
id	UUID	Unique identifier for the flow record
observe	VARCHAR	Observer or sensor identifier that captured the flow
stime	TIMESTAMP	Start time of the flow
etime	TIMESTAMP	End time of the flow
dur	UINTEGER	Duration of the flow in microseconds
rtt	UINTEGER	Round-trip time in microseconds
pcr	INTEGER	Packet capture rate or sampling rate

Protocol and Network Information

---

Field	Type	Description
proto	VARCHAR	Protocol type (TCP, UDP, ICMP, etc.)
saddr	VARCHAR	Source IP address
daddr	VARCHAR	Destination IP address
sport	USMALLINT	Source port number
dport	USMALLINT	Destination port number

TCP-Specific Fields

---

Field	Type	Description
iflags	VARCHAR	Initial TCP flags observed in the flow
uflags	VARCHAR	Union of all TCP flags seen during the flow
stcpseq	UINTEGER	Source TCP sequence number
dtcpseq	UINTEGER	Destination TCP sequence number
stcpurg	UINTEGER	Source TCP urgent pointer count
dtcpurg	UINTEGER	Destination TCP urgent pointer count

VLAN Information

---

Field	Type	Description
svlan	USMALLINT	Source VLAN ID
dvlan	USMALLINT	Destination VLAN ID

Traffic Statistics

---

Field	Type	Description
spkts	UBIGINT	Number of packets from source to destination
dpkts	UBIGINT	Number of packets from destination to source
sbytes	UBIGINT	Number of bytes from source to destination
dbytes	UBIGINT	Number of bytes from destination to source

Entropy and Timing Analysis

---

Field	Type	Description
sentropy	UTINYINT	Source payload entropy (randomness measure)
dentropy	UTINYINT	Destination payload entropy (randomness measure)
siat	UBIGINT	Source inter-arrival time statistics
diat	UBIGINT	Destination inter-arrival time statistics
sstdev	UBIGINT	Source standard deviation of inter-arrival times
dstdev	UBIGINT	Destination standard deviation of inter-arrival times

Packet Size Analysis

---

Field	Type	Description
ssmallpktcnt	UINTEGER	Count of small packets from source
dsmallpktcnt	UINTEGER	Count of small packets from destination
slargepktcnt	UINTEGER	Count of large packets from source
dlargepktcnt	UINTEGER	Count of large packets from destination
snonemptypktcnt	UINTEGER	Count of non-empty packets from source
dnonemptypktcnt	UINTEGER	Count of non-empty packets from destination
sfirstnonemptycnt	USMALLINT	First non-empty packet count from source
dfirstnonemptycnt	USMALLINT	First non-empty packet count from destination
smaxpktsize	USMALLINT	Maximum packet size from source
dmaxpktsize	USMALLINT	Maximum packet size from destination
sstdevpayload	USMALLINT	Standard deviation of payload sizes from source
dstdevpayload	USMALLINT	Standard deviation of payload sizes from destination

Flow Classification

---

Field	Type	Description
spd	VARCHAR	Speed or rate classification
reason	VARCHAR	Reason for flow termination or classification
orient	VARCHAR	Flow orientation or direction classification
tag	VARCHAR[]	Array of tags or labels associated with the flow

Hardware Addresses

---

Field	Type	Description
smac	VARCHAR	Source MAC address
dmac	VARCHAR	Destination MAC address

Geographical and ASN Information

Field	Type	Description
scountry	VARCHAR	Source IP country code
dcountry	VARCHAR	Destination IP country code
sasn	UINTEGER	Source Autonomous System Number
dasn	UINTEGER	Destination Autonomous System Number
sasnorg	VARCHAR	Source ASN organization name
dasnorg	VARCHAR	Destination ASN organization name

Histogrambased Outlier Score (HBOS)

---

Field	Type	Description
hbos_score	DOUBLE	Histogram-Based Outlier Score for anomaly detection
hbos_severity	UTINYINT	Severity level based on HBOS analysis (0-255)
hbos_map	MAP(VARCHAR, FLOAT)	Detailed HBOS feature scores as key-value pairs

Deep Packet Inspection (nDPI)

---

Field	Type	Description
ndpi_appid	VARCHAR	Application ID identified by nDPI
ndpi_category	VARCHAR	Application category from nDPI classification
ndpi_risk_bits	UBIGINT	Bit field representing various risk factors
ndpi_risk_score	UINTEGER	Numerical risk score calculated by nDPI
ndpi_risk_severity	UTINYINT	Risk severity level (0-255)
ndpi_risk_list	VARCHAR[]	Array of specific risk descriptions

Flow Processing

---

Field	Type	Description
trigger	TINYINT	Trigger condition or event that caused flow processing

Data Types Reference

---

• UINTEGER: Unsigned 32-bit integer
• UBIGINT: Unsigned 64-bit integer
• USMALLINT: Unsigned 16-bit integer
• UTINYINT: Unsigned 8-bit integer
• TINYINT: Signed 8-bit integer
• INTEGER: Signed 32-bit integer
• DOUBLE: Double-precision floating-point
• FLOAT: Single-precision floating-point
• VARCHAR: Variable-length character string
• VARCHAR[]: Array of variable-length character strings
• UUID: Universally Unique Identifier
• TIMESTAMP: Date and time value
• MAP(VARCHAR, FLOAT): Key-value mapping with string keys and float values