Configuration

​

Overview

Arroyo has a flexible and powerful configuration system that allows options to be set via files (in TOML or Yaml format) and environment variables.

The system will look for configuration in the following places, from highest to lowest priority:

1. ARROYO__* environment variables
2. Config file specified via the --config option
3. Any *.toml or *.yaml files in the --config-dir directory
4. arroyo.toml in the current directory
5. $(user conf dir)/arroyo/config.{toml,yaml} — (this is ~/.config/arroyo on Linux and ~/Library/Application Support/arroyo on MacOS)
6. Default configuration

​

Config files

In TOML or YAML, nested configurations are specified as tables under the given key name, for example:

checkpoint-url = 's3://my-bucket/checkpoints'

[controller]
scheduler = 'node'

[database]
type = "postgres"

​

Environment variables

All configuration options can be set as environment variables as well. To convert a config name into an environment variable, the following rule are applied:

1. Start with ARROYO__
2. Replace all dots (i.e., layers of nesting) with __ (double underscore)
3. Replace all - with _ (single underscore)
4. Uppercase all letters

Some examples:

• checkpoint-url => ARROYO__CHECKPOINT_URL
• pipeline.compaction.enabled => ARROYO__PIPELINE__COMPACTION__ENABLED
• api.bind-address => API__BIND_ADDRESS

Reasonable type conversions will be applied for values specified as environment variable, for example numbers and booleans will be parsed into the correct type.

​

Options

Here we list all of the available configuration options by the key they are nested under. So for example, the option in the Pipeline section listed as source-batch-size would be specified in the config file as pipeline.source-batch-size or as a table

[pipeline]
source-batch-size = 128

Top-level options:

​

Pipeline

Configuration that applies to individual pipelines.

Key: pipeline

​

Run (pipeline clusters)

Configuration for pipeline clusters

Key: run

​

API

Configuration for the API service

Key: api

​

Controller

Configuration for the controller service

Key: controller

​

Compiler

Configuration for the UDF compiler service.

Key: compiler

​

Admin

Configuration for the Admin service

Key: admin

​

Node

Configuration for the Node service

Key: node

​

Worker

Configuration for pipeline workers

Key: worker

​

Schedulers

Configuration for the various schedulers

​

Process Scheduler

Key: process-scheduler

​

Kubernetes Scheduler

Key: kubernetes-scheduler

Some values for the kubernetes scheduler are complete Kubernetes object, for example, the worker.resources object can be specified as a Kubernetes resource object.

When specifying these via environment variables they should be encoded as Yaml.

See the Kubernetes deployment docs for more details.

There are two modes for allocating resources for Kubernetes, specified as the kubernetes-scheduler.resource-mode:

• In per-slot mode, tasks are packed onto workers up to the task-slots config, and for each slot the amount of resources specified in resources is provided. This can be much more efficient for diversely-sized pipelines
• In per-pod mode, every pod has exactly task-slots slots, and exactly the resources in resources, even if it is scheduled for fewer slots. This is the behavior from before 0.11.

​

Database

Key: database

​

Logging

Key: logging