Skip to main content

Local Storage

Description​

This guide describes how to set up a GNAT pipeline with local storage. It includes a docker-compose configuration file with three services:

  • gnat-sensor: The Galileo sensor that captures and generates flow data.
  • gnat-import: The Galileo import service that processes and imports data into Parquet format.
  • gnat-store: The Galileo storage service that write the flow data in partitions of Parquet files.

Concept of Operation​

pipeline

Prerequisites​

  • Basic knowledge of Linx command-line interface (CLI).
  • Basic knowledge of Docker and Docker Compose.
  • First, follow the Quick Start Instructions to set up the basic GNAT pipeline on your machine.

Docker Compose Configuration​

services:
  gnat_sensor:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_sensor
      restart: unless-stopped
      network_mode: host
      cap_add:
        - net_admin
        - net_raw
        - sys_nice
      environment:
        GNAT_INTERFACE: ${GNAT_INTERFACE}
        GNAT_OBSERVATION_TAG: ${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_EXPORT_INTERVAL: 20
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_sensor.sh

    gnat_import:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_import
      restart: unless-stopped
      environment:  
        GNAT_INPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/import
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_import.sh

    gnat_store:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_store
      restart: unless-stopped
      environment:
        GNAT_INPUT: /var/spool/import
        GNAT_OUTPUT: /var/flow
        TZ: UTC
      volumes:
        - /var/flow:/var/flow
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_store.sh

Configuration File​

Create a .env file in the same directory with the following content:

GNAT_INTERFACE=eth0
GNAT_OBSERVATION_TAG=gnat

Replace eth0 with the name of the network interface you want to monitor. You can find the name of your network interface by running the command ip link show. Additionally, you can change the GNAT_OBSERVATION_TAG to a custom string value that will be used to tag the observations.

Starting the Services​

Starts all the services defined in the background with the following command:

docker-compose up -d

Monitoring the Services​

Monitor the services with the following command:

docker-compose logs -f

Stopping the Services​

Stop all the services with the folling command:

docker-compose down

See Also​