Skip to main content

MotherDuck

Description​

This guide describes how to set up GNAT pipeline to work with Motherduck. It includes a simple docker-compose configuration file with three services:

  • gnat-sensor: The Galileo sensor that captures and generates flow data.
  • gnat-import: The Galileo import service that processes and imports data into Parquet format.
  • gnat-store: The Galileo storate service that uploads flow records to motherduck.

You can run this configuration on your local machine or on dedicated hardware connected to a SPAN port or TAP, and it will automatically capture network traffic, process it, and store it in MotherDuck for further analysis.

Concept of Operation​

pipeline

Prerequisites​

  • Basic knowledge of Linx command-line interface (CLI).
  • Basic knowledge of Docker and Docker Compose.
  • First, follow the Quick Start Instructions to set up the basic GNAT pipeline on your machine.

Create a MotherDuck Account​

Sign up for a free (or paid) account at MotherDuck. MotherDuck offers generous free tier limits that are perfect for getting started with network flow analysis.

Generate a Security Token​

Once logged into MotherDuck:

  • Navigate to your account settings
  • Generate a new API token
  • Copy and save this token securely — you'll need it for the configuration

Adjust Query Performance​

To optimize query performance in MotherDuck, consider adjusting the instance type on the upper right corner of the MotherDuck console:

  • For cost efficiency, the pulse instance type is recommended for most use cases.
  • For high-performance queries, consider using the standard instance type.

Create a Database​

Create a new database in MotherDuck where the GNAT data will be stored.

  • Navigate to the "Attached databases" section in the MotherDuck console.
  • Create a new database by clicking the +, naming it galileo (or any name you prefer).

Docker Compose Configuration​

Create a docker-compose.yml file in your project directory with the following content:

services:
  gnat_sensor:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_sensor
      restart: unless-stopped
      network_mode: host
      cap_add:
        - net_admin
        - net_raw
        - sys_nice
      environment:
        GNAT_INTERFACE: ${GNAT_INTERFACE}
        GNAT_OBSERVATION_TAG: ${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_EXPORT_INTERVAL: 20
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_sensor.sh

    gnat_import:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_import
      restart: unless-stopped
      environment:  
        GNAT_INPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/import
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_import.sh

    gnat_store:
        image: fidelismachine/galileo_toolkit:latest
        container_name: gnat_store
        restart: unless-stopped
        environment:
        GNAT_INPUT: /var/spool/import
        GNAT_OUTPUT: md:galileo
        GNAT_INTERVAL: minute
        motherduck_token: ${motherduck_token}
        TZ: UTC
        volumes:
        - /var/spool/gnat:/var/spool
        command: /opt/gnat/scripts/entrypoint-gnat_store.sh

Configuration File​

Create a .env file in the same directory with the following content:

GNAT_INTERFACE=eth0
GNAT_OBSERVATION_TAG=gnat
motherduck_token=<your_motherduck_token>

Replace eth0 with the name of the network interface you want to monitor. You can find the name of your network interface by running the command ip link show. Additionally, you can change the GNAT_OBSERVATION_TAG to a custom string value that will be used to tag the observations.

To create a motherduck_token follow the instructions in the MotherDuck documentation.

Data Directories​

The data directories are mounted to /var/spool/gnat on the host machine. Therefore, make sure this directories exists and are writable by the Docker containers. You can create this directory with the following command:

sudo mkdir -p /var/spool/gnat
sudo chown $USER:$USER /var/spool/gnat

This directory will contain the following subdirectories:

  • /var/spool/gnat/{GNAT_OBSERVATION_TAG}: Contains the raw flow data captured by the gnat_sensor.
  • /var/spool/gnat/import: Contains the imported data from the gnat_sensor.

Starting the Services​

Starts all the services defined in the background with the following command:

docker-compose up -d

Within minutes:

  • Galileo will generate, collect, and process network flow data
  • Automatically ship enriched network data to MotherDuck for analysis
  • Provide threat hunting environment ready for SQL-based investigations

Monitoring the Services​

Monitor the services with the following command:

docker-compose logs -f

Stopping the Services​

Stop all the services with the folling command:

docker-compose down

See Also​