Skip to main content

S3 Storage

Coming soon. This recipe has not be verified yet. Please subscribe to the Galileo Blog to be notified when this recipe is available.

Description​

This guide describes how to set up GNAT pipeline to work with an S3 compatible service. It includes a docker-compose configuration file with three services:

  • gnat-sensor: The Galileo sensor that captures and generates flow data.
  • gnat-import: The Galileo import service that processes and imports data into Parquet format.
  • gnat-store: The Galileo storate service that uploads flow records to motherduck.

Concept of Operation​

pipeline

Prerequisites​

  • Basic knowledge of Linx command-line interface (CLI).
  • Basic knowledge of Docker and Docker Compose.
  • First, follow the Quick Start Instructions to set up the basic GNAT pipeline on your machine.

Docker Compose Configuration​

services:
  gnat_sensor:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_sensor
      restart: unless-stopped
      network_mode: host
      cap_add:
        - net_admin
        - net_raw
        - sys_nice
      environment:
        GNAT_INTERFACE: ${GNAT_INTERFACE}
        GNAT_OBSERVATION_TAG: ${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_EXPORT_INTERVAL: 20
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_sensor.sh

    gnat_import:
      image: fidelismachine/galileo_toolkit:latest
      container_name: gnat_import
      restart: unless-stopped
      environment:  
        GNAT_INPUT: /var/spool/${GNAT_OBSERVATION_TAG}
        GNAT_OUTPUT: /var/spool/import
      volumes:
        - /var/spool/gnat:/var/spool
      command: /opt/gnat/scripts/entrypoint-gnat_import.sh

    gnat_store:
        image: fidelismachine/galileo_toolkit:latest
        container_name: gnat_store
        restart: unless-stopped
        environment:
          GNAT_INPUT: /var/spool/import
          GNAT_OUTPUT: s3://<bucket>/<folder>
          # Replace with your S3 compatible service details
          GNAT_INTERVAL: minute
          s3_region: ${s3_region}
          s3_endpoint: ${s3_endpoint}
          s3_access_key_id: ${s3_access_key_id}
          s3_secret_access_key: ${s3_secret_access_key}
          s3_url_style: ${s3_url_style}
        TZ: UTC
        volumes:
        - /var/spool/gnat:/var/spool
        command: /opt/gnat/scripts/entrypoint-gnat_store.sh

Configuration File​

Create a .env file in the same directory with the following content:

GNAT_INTERFACE=
GNAT_OBSERVATION_TAG=<gnat_observation_tag>
motherduck_token=<your_motherduck_token>
s3_region=<your_s3_region>
s3_endpoint=<your_s3_endpoint>
s3_access_key_id=<your_s3_access_key_id>
s3_secret_access_key=<your_s3_secret_access_key>
s3_url_style=<your_s3_url_style>

Replace eth0 with the name of the network interface you want to monitor. You can find the name of your network interface by running the command ip link show. Additionally, you can change the GNAT_OBSERVATION_TAG to a custom string value that will be used to tag the observations.

Refer to your cloud provider for the following S3 compatible service details:

  • s3_region
  • s3_endpoint
  • s3_access_key_id
  • s3_secret_access_key
  • s3_url_style

Starting the Services​

Starts all the services defined in the background with the following command:

docker-compose up -d

Monitoring the Services​

Monitor the services with the following command:

docker-compose logs -f

Stopping the Services​

Stop all the services with the folling command:

docker-compose down

See Also​