Skip to content

Configuration

Customizing the system behavior

Overview

Arroyo has a flexible and powerful configuration system that allows options to be set via files (in TOML or Yaml format) and environment variables.

The system will look for configuration in the following places, from highest to lowest priority:

  1. ARROYO__* environment variables
  2. Config file specified via the --config option
  3. Any *.toml or *.yaml files in the --config-dir directory
  4. arroyo.toml in the current directory
  5. $(user conf dir)/arroyo/config.{toml,yaml} — (this is ~/.config/arroyo on Linux and ~/Library/Application Support/arroyo on MacOS)
  6. Default configuration

Config files

In TOML or YAML, nested configurations are specified as tables under the given key name, for example:

checkpoint-url = 's3://my-bucket/checkpoints'

[controller]
scheduler = 'node'

[database]
type = "postgres"

Environment variables

All configuration options can be set as environment variables as well. To convert a config name into an environment variable, the following rule are applied:

  1. Start with ARROYO__
  2. Replace all dots (i.e., layers of nesting) with __ (double underscore)
  3. Replace all - with _ (single underscore)
  4. Uppercase all letters

Some examples:

  • checkpoint-url => ARROYO__CHECKPOINT_URL
  • pipeline.compaction.enabled => ARROYO__PIPELINE__COMPACTION__ENABLED
  • api.bind-address => ARROYO__API__BIND_ADDRESS

Reasonable type conversions will be applied for values specified as environment variable, for example numbers and booleans will be parsed into the correct type.

Options

Here we list all of the available configuration options by the key they are nested under. So for example, the option in the Pipeline section listed as source-batch-size would be specified in the config file as pipeline.source-batch-size or as a table

[pipeline]
source-batch-size = 128

Top-level options:

Name Description Default Value
checkpoint-url URL of an object store or filesystem for storing checkpoints; in a distributed cluster this must be a location available to all nodes /tmp/arroyo/checkpoints
default-checkpoint-interval Default checkpointing interval 10s
api-endpoint Endpoint of the API, used by other services to connect to it inferred
controller-endpoint Endpoint of the controller, used by other services to connect to it inferred
compiler-endpoint Endpoint of the compiler, used by other services to connect to it inferred
job-controller Where each pipeline’s job controller runs: controller runs it inside the Arroyo controller, while worker (beta) runs it on one of the pipeline’s workers (the leader). See Job controller modes controller
disable-telemetry Disable open-source telemetry false

Pipeline

Configuration that applies to individual pipelines.

Key: pipeline

Name Description Default Value
source-batch-size Max size of source batches 512
source-batch-linger Batch linger time (how long to wait before flushing) 100ms
source-batch-max-bytes Flush a source batch once it has buffered this many bytes, measured as the raw size of the input data. Specified as an integer number of bytes; unset disables size-based flushing none
update-aggregate-flush-interval How often to flush aggregates 1s
allowed-restarts How many restarts to allow before moving to failed (-1 for infinite) 20
worker-heartbeat-timeout Number of seconds to wait for a worker heartbeat before considering it dead 30s
healthy-duration After this amount of time, we consider the job to be healthy and reset the restarts counter 2m
worker-startup-time Amount of time to wait for workers to start up before considering them failed 10m
task-startup-time Amount of time to wait for tasks to startup before considering it failed 2m
state-initial-backoff Initial delay before the controller retries a pipeline after a retryable error; the delay doubles (with jitter) on each subsequent attempt 500ms
state-max-backoff Maximum delay between controller retries after retryable errors 1m
compaction.enabled Whether to enable compaction for checkpoints; not supported in worker mode false
compaction.checkpoints-to-compact The number of outstanding checkpoints that will trigger compaction 4
checkpoint.timeout Worker mode only: maximum time to wait for the final checkpoint when stopping a pipeline with a checkpoint or rescaling it. If the checkpoint doesn’t finish in time, the job fails and goes through normal recovery none
chaining.enabled Whether to enable operator chaining, which reduces the number of operators in the pipeline true
store-deserialization-errors Whether to persist deserialization errors to the database true

Run (pipeline clusters)

Configuration for pipeline clusters

Key: run

Name Description Default Value
query The query to run for this pipeline cluster (equivalent to the query command-line parameter none
state-dir Sets the directory that state will be written to and read from none

TLS Configuration

All services support TLS encryption, which can be configured globally for all services or per-service by prefixing the key by the service name (for example, api.tls).

Key: tls

Name Description Required
cert-file Path to the TLS certificate file (PEM format) Yes
key-file Path to the TLS private key file (PEM format) Yes
mtls-ca-file Path to the CA certificate for mutual TLS (mTLS) authentication No

gRPC

Configuration for the gRPC connections that Arroyo services use to communicate with each other.

Key: grpc

Name Description Default Value
max-message-size Maximum size of a gRPC message that a process will accept, specified as a data size like 32mb or 512kb (units are powers of 1024); must be less than 4gb. Each process enforces this limit on the messages it receives; very large pipelines may need a higher limit 32mb

API

Configuration for the API service

Key: api

Name Description Default Value
bind-address The host the API service should bind to 0.0.0.0
http-port The HTTP port for the API service 5115
run-http-port The HTTP port for the API service in run mode; defaults to a random port 0

Authentication

The API service supports multiple authentication modes, configured via the auth-mode option:

None (default) No authentication required.

[api]
auth-mode = { type = "none" }

Static API Key Require a bearer token for all API requests.

[api]
auth-mode = { type = "static-api-key", api-key = "your-secret-key" }

Clients must include the API key in the Authorization header:

Authorization: Bearer your-secret-key

Mutual TLS (mTLS) Require client certificates for authentication.

[tls]
cert-file = "/path/to/server-cert.pem"
key-file = "/path/to/server-key.pem"
mtls-ca-file = "/path/to/ca-cert.pem"

[api]
auth-mode = { type = "mtls", ca-cert-file = "/path/to/ca-cert.pem" }

With mTLS, clients must present a valid certificate signed by the specified CA.

CORS

By default, the API allows cross-origin requests from any origin. To restrict which origins can call the API from a browser, switch to an allow-list:

[api.cors]
origin-policy = "allow-list"
allowed-origins = ["https://arroyo.example.com"]
allow-credentials = true

Key: api.cors

Name Description Default Value
origin-policy Either any, which allows requests from all origins, or allow-list, which only allows the origins in allowed-origins any
allowed-origins Origins allowed when origin-policy is allow-list. Each entry must be a bare origin (scheme, host, and optional port, such as https://example.com:8443) with no path or trailing slash; the wildcard * is not allowed. Must be empty when origin-policy is any []
allow-credentials Whether browsers may send credentials (cookies or Authorization headers) with cross-origin requests. Can only be enabled with the allow-list policy false

The API service will fail to start if the CORS configuration is invalid.

Controller

Configuration for the controller service

Key: controller

Name Description Default Value
bind-address The host the controller should bind to 0.0.0.0
rpc-port The RPC port for the controller 5116
scheduler The scheduler to use; one of process, kubernetes, node, embedded, or manual process
metrics.enabled Whether to collect operator metrics for running pipelines, which power the metrics graphs in the Web UI and the operator metrics API true
leader-poll-interval Worker mode only: how often the controller polls each pipeline’s leader for its status 500ms
connect-timeout Worker mode only: timeout for the controller to establish a connection to a pipeline’s leader none

Compiler

Configuration for the UDF compiler service.

Key: compiler

Name Description Default Value
bind-address Bind address for the compiler 0.0.0.0
rpc-port Port for the Compiler RPC service 5117
install-clang Whether the compiler should attempt to install clang if it’s not already installed true
install-rustc Whether the compiler should attempt to install rustc if it’s not already installed true
artifact-url Where to store compilation artifacts, in a distributed cluster this must be a location available to all nodes /tmp/arroyo/artifacts
build-dir Directory for build files /tmp/arroyo/build-dir
use-local-udf-crate Whether to use a local version of the UDF library or the published crate (only enable in development environments) false

Admin

Configuration for the Admin service

Key: admin

Name Description Default Value
bind-address Address to bind the Admin service 0.0.0.0
http-port Port for the Admin HTTP service 5114
allow-unauthenticated-metrics Serve the Prometheus /metrics endpoint without requiring an API key, even when static-api-key authentication is enabled; other admin endpoints still require the key false

Authentication

Like the API service, the Admin service supports authentication modes. See the API Authentication section for details.

The /status endpoint never requires an API key, so it can be used for liveness and readiness probes.

Node

Configuration for the Node service

Key: node

Name Description Default Value
bind-address Address to bind the Node service 0.0.0.0
rpc-port Port for the Node RPC service 5118
task-slots Number of task slots for the Node 16

Worker

Configuration for pipeline workers

Key: worker

Name Description Default Value
bind-address Address to bind the Worker service 0.0.0.0
rpc-port RPC port for the worker to listen on; set to 0 to use a random available port 0
data-port Data port for the worker to listen on; set to 0 to use a random available port 0
task-slots Number of task slots for the Worker 16
queue-size Size of the queues between nodes in the dataflow graph 8192
checkpoint-details-to-keep Worker mode only: number of recent checkpoints whose details the leader keeps for the Web UI and checkpoints API 10

Schedulers

Configuration for the various schedulers

Process Scheduler

Key: process-scheduler

Name Description Default Value
slots-per-process Number of slots per process in the scheduler 16
shutdown-with-controller Whether worker processes are killed when the controller shuts down. Disabling this can leave orphaned worker processes, and is mostly useful for testing controller recovery true

Manual Scheduler

The manual scheduler does not start workers itself. Instead, when a pipeline is scheduled it prints the commands needed to start each worker, which you then run yourself. This is mostly useful for testing and debugging scheduling behavior.

Key: manual-scheduler

Name Description Default Value
slots-per-process Number of task slots assigned to each worker; the scheduler prints a command for enough workers to cover all of the pipeline’s slots 16

Kubernetes Scheduler

Key: kubernetes-scheduler

Some values for the kubernetes scheduler are complete Kubernetes object, for example, the worker.resources object can be specified as a Kubernetes resource object.

When specifying these via environment variables they should be encoded as Yaml.

See the Kubernetes deployment docs for more details.

There are two modes for allocating resources for Kubernetes, specified as the kubernetes-scheduler.resource-mode:

  • In per-slot mode, tasks are packed onto workers up to the task-slots config, and for each slot the amount of resources specified in resources is provided. This can be much more efficient for diversely-sized pipelines
  • In per-pod mode, every pod has exactly task-slots slots, and exactly the resources in resources, even if it is scheduled for fewer slots. This is the behavior from before 0.11.
Name Description Default Value
namespace Kubernetes namespace for the scheduler default
resource-mode Resource allocation mode; per-slot or per-pod per-slot
worker.name-prefix Prefix for worker names arroyo
worker.image Docker image for workers ghcr.io/arroyosystems/arroyo:latest
worker.image-pull-policy Image pull policy for worker containers IfNotPresent
worker.service-account-name Service account name for worker containers default
worker.resources.requests Kubernetes resource object representing the requests for the worker pods {cpu: "900m", memory: "500Mi"}
worker.resources.limits Kubernetes resource object representing the limits for the worker pods none
worker.task-slots Number of task slots per worker 16
worker.command Command to start worker containers /app/arroyo worker
worker.env List of environment variables for worker containers, each a k8s-style map with name and value keys none

Database

Key: database

Name Description Default Value
type Type of the database (either sqlite or postgres) sqlite
sqlite.path Path to the database file $(user config dir)/arroyo/config.sqlite
postgres.database-name Name of the Postgres database arroyo
postgres.host Host of the Postgres database localhost
postgres.port Port of the Postgres database 5432
postgres.user User for the Postgres database arroyo
postgres.password Password for the Postgres database arroyo
postgres.schema Postgres schema that Arroyo’s tables live in. The schema must already exist, and may only contain ASCII letters, numbers, and underscores. Useful for sharing a Postgres database with other services public
postgres.tls.mode TLS mode for connections to Postgres; one of no-tls, skip-verification, system-roots, or ca-cert (see below) no-tls
postgres.tls.path Path to a PEM-encoded CA certificate; required when postgres.tls.mode is ca-cert none

Postgres TLS

By default, Arroyo connects to Postgres without TLS. The postgres.tls.mode option supports the following modes:

  • no-tls: Connect over plaintext
  • skip-verification: Encrypt the connection, but do not verify the server’s certificate
  • system-roots: Encrypt the connection and verify the server’s certificate against the system trust store; suitable for managed Postgres providers with publicly-signed certificates
  • ca-cert: Encrypt the connection and verify the server’s certificate against the system trust store plus the CA certificate at postgres.tls.path

For example:

[database.postgres]
host = "my-db.example.com"
tls = { mode = "ca-cert", path = "/etc/arroyo/postgres-ca.pem" }

or via environment variables:

ARROYO__DATABASE__POSTGRES__TLS__MODE=ca-cert
ARROYO__DATABASE__POSTGRES__TLS__PATH=/etc/arroyo/postgres-ca.pem

Logging

Key: logging

Name Description Default Value
format Set the log format (one of json, logfmt, or plaintext) plaintext
nonblocking Whether to use nonblocking logging; this uses more memory but ensures processing is not blocked by a high rate of logging false
buffered-lines-limit Number of lines to buffer before dropping logs or exerting backpressure on senders; only valid when nonblocking is set to true 4096
enable-file-line Whether to record the source file line in the log false
enable-file-name Whether to record the source file name in the log false
static-fields Map of additional fields to include in every log line; only applies to the json format none

In the json format, every log line also includes an arroyo_service field with the name of the service that emitted it and, for pipeline workers, a pipeline_id field. Additional fields can be added with static-fields:

[logging]
format = "json"
static-fields = { cluster = "prod-us-east-1", team = "data-platform" }

Type to search…

↑↓ navigate↵ selectEsc close