Overview
Arroyo has a flexible and powerful configuration system that allows options to be set via files (in TOML or Yaml format) and environment variables.
The system will look for configuration in the following places, from highest to lowest priority:
ARROYO__*environment variables- Config file specified via the
--configoption - Any *.toml or *.yaml files in the
--config-dirdirectory arroyo.tomlin the current directory$(user conf dir)/arroyo/config.{toml,yaml}— (this is~/.config/arroyoon Linux and~/Library/Application Support/arroyoon MacOS)- Default configuration
Config files
In TOML or YAML, nested configurations are specified as tables under the given key name, for example:
checkpoint-url = 's3://my-bucket/checkpoints'
[controller]
scheduler = 'node'
[database]
type = "postgres"Environment variables
All configuration options can be set as environment variables as well. To convert a config name into an environment variable, the following rule are applied:
- Start with
ARROYO__ - Replace all dots (i.e., layers of nesting) with
__(double underscore) - Replace all
-with_(single underscore) - Uppercase all letters
Some examples:
checkpoint-url=>ARROYO__CHECKPOINT_URLpipeline.compaction.enabled=>ARROYO__PIPELINE__COMPACTION__ENABLEDapi.bind-address=>ARROYO__API__BIND_ADDRESS
Reasonable type conversions will be applied for values specified as environment variable, for example numbers and booleans will be parsed into the correct type.
Options
Here we list all of the available configuration options by the key they are nested under. So for example,
the option in the Pipeline section listed as source-batch-size would be specified in the config file as
pipeline.source-batch-size or as a table
[pipeline]
source-batch-size = 128Top-level options:
| Name | Description | Default Value |
|---|---|---|
checkpoint-url |
URL of an object store or filesystem for storing checkpoints; in a distributed cluster this must be a location available to all nodes | /tmp/arroyo/checkpoints |
default-checkpoint-interval |
Default checkpointing interval | 10s |
api-endpoint |
Endpoint of the API, used by other services to connect to it | inferred |
controller-endpoint |
Endpoint of the controller, used by other services to connect to it | inferred |
compiler-endpoint |
Endpoint of the compiler, used by other services to connect to it | inferred |
job-controller |
Where each pipeline’s job controller runs: controller runs it inside the Arroyo controller, while worker (beta) runs it on one of the pipeline’s workers (the leader). See Job controller modes |
controller |
disable-telemetry |
Disable open-source telemetry | false |
Pipeline
Configuration that applies to individual pipelines.
Key: pipeline
| Name | Description | Default Value |
|---|---|---|
source-batch-size |
Max size of source batches | 512 |
source-batch-linger |
Batch linger time (how long to wait before flushing) | 100ms |
source-batch-max-bytes |
Flush a source batch once it has buffered this many bytes, measured as the raw size of the input data. Specified as an integer number of bytes; unset disables size-based flushing | none |
update-aggregate-flush-interval |
How often to flush aggregates | 1s |
allowed-restarts |
How many restarts to allow before moving to failed (-1 for infinite) | 20 |
worker-heartbeat-timeout |
Number of seconds to wait for a worker heartbeat before considering it dead | 30s |
healthy-duration |
After this amount of time, we consider the job to be healthy and reset the restarts counter | 2m |
worker-startup-time |
Amount of time to wait for workers to start up before considering them failed | 10m |
task-startup-time |
Amount of time to wait for tasks to startup before considering it failed | 2m |
state-initial-backoff |
Initial delay before the controller retries a pipeline after a retryable error; the delay doubles (with jitter) on each subsequent attempt | 500ms |
state-max-backoff |
Maximum delay between controller retries after retryable errors | 1m |
compaction.enabled |
Whether to enable compaction for checkpoints; not supported in worker mode | false |
compaction.checkpoints-to-compact |
The number of outstanding checkpoints that will trigger compaction | 4 |
checkpoint.timeout |
Worker mode only: maximum time to wait for the final checkpoint when stopping a pipeline with a checkpoint or rescaling it. If the checkpoint doesn’t finish in time, the job fails and goes through normal recovery | none |
chaining.enabled |
Whether to enable operator chaining, which reduces the number of operators in the pipeline | true |
store-deserialization-errors |
Whether to persist deserialization errors to the database | true |
Run (pipeline clusters)
Configuration for pipeline clusters
Key: run
| Name | Description | Default Value |
|---|---|---|
query |
The query to run for this pipeline cluster (equivalent to the query command-line parameter | none |
state-dir |
Sets the directory that state will be written to and read from | none |
TLS Configuration
All services support TLS encryption, which can be configured globally for all services or per-service
by prefixing the key by the service name (for example, api.tls).
Key: tls
| Name | Description | Required |
|---|---|---|
cert-file |
Path to the TLS certificate file (PEM format) | Yes |
key-file |
Path to the TLS private key file (PEM format) | Yes |
mtls-ca-file |
Path to the CA certificate for mutual TLS (mTLS) authentication | No |
gRPC
Configuration for the gRPC connections that Arroyo services use to communicate with each other.
Key: grpc
| Name | Description | Default Value |
|---|---|---|
max-message-size |
Maximum size of a gRPC message that a process will accept, specified as a data size like 32mb or 512kb (units are powers of 1024); must be less than 4gb. Each process enforces this limit on the messages it receives; very large pipelines may need a higher limit |
32mb |
API
Configuration for the API service
Key: api
| Name | Description | Default Value |
|---|---|---|
bind-address |
The host the API service should bind to | 0.0.0.0 |
http-port |
The HTTP port for the API service | 5115 |
run-http-port |
The HTTP port for the API service in run mode; defaults to a random port | 0 |
Authentication
The API service supports multiple authentication modes, configured via the auth-mode option:
None (default) No authentication required.
[api]
auth-mode = { type = "none" }Static API Key Require a bearer token for all API requests.
[api]
auth-mode = { type = "static-api-key", api-key = "your-secret-key" }Clients must include the API key in the Authorization header:
Authorization: Bearer your-secret-keyMutual TLS (mTLS) Require client certificates for authentication.
[tls]
cert-file = "/path/to/server-cert.pem"
key-file = "/path/to/server-key.pem"
mtls-ca-file = "/path/to/ca-cert.pem"
[api]
auth-mode = { type = "mtls", ca-cert-file = "/path/to/ca-cert.pem" }With mTLS, clients must present a valid certificate signed by the specified CA.
CORS
By default, the API allows cross-origin requests from any origin. To restrict which origins can call the API from a browser, switch to an allow-list:
[api.cors]
origin-policy = "allow-list"
allowed-origins = ["https://arroyo.example.com"]
allow-credentials = trueKey: api.cors
| Name | Description | Default Value |
|---|---|---|
origin-policy |
Either any, which allows requests from all origins, or allow-list, which only allows the origins in allowed-origins |
any |
allowed-origins |
Origins allowed when origin-policy is allow-list. Each entry must be a bare origin (scheme, host, and optional port, such as https://example.com:8443) with no path or trailing slash; the wildcard * is not allowed. Must be empty when origin-policy is any |
[] |
allow-credentials |
Whether browsers may send credentials (cookies or Authorization headers) with cross-origin requests. Can only be enabled with the allow-list policy |
false |
The API service will fail to start if the CORS configuration is invalid.
Controller
Configuration for the controller service
Key: controller
| Name | Description | Default Value |
|---|---|---|
bind-address |
The host the controller should bind to | 0.0.0.0 |
rpc-port |
The RPC port for the controller | 5116 |
scheduler |
The scheduler to use; one of process, kubernetes, node, embedded, or manual |
process |
metrics.enabled |
Whether to collect operator metrics for running pipelines, which power the metrics graphs in the Web UI and the operator metrics API | true |
leader-poll-interval |
Worker mode only: how often the controller polls each pipeline’s leader for its status | 500ms |
connect-timeout |
Worker mode only: timeout for the controller to establish a connection to a pipeline’s leader | none |
Compiler
Configuration for the UDF compiler service.
Key: compiler
| Name | Description | Default Value |
|---|---|---|
bind-address |
Bind address for the compiler | 0.0.0.0 |
rpc-port |
Port for the Compiler RPC service | 5117 |
install-clang |
Whether the compiler should attempt to install clang if it’s not already installed | true |
install-rustc |
Whether the compiler should attempt to install rustc if it’s not already installed | true |
artifact-url |
Where to store compilation artifacts, in a distributed cluster this must be a location available to all nodes | /tmp/arroyo/artifacts |
build-dir |
Directory for build files | /tmp/arroyo/build-dir |
use-local-udf-crate |
Whether to use a local version of the UDF library or the published crate (only enable in development environments) | false |
Admin
Configuration for the Admin service
Key: admin
| Name | Description | Default Value |
|---|---|---|
bind-address |
Address to bind the Admin service | 0.0.0.0 |
http-port |
Port for the Admin HTTP service | 5114 |
allow-unauthenticated-metrics |
Serve the Prometheus /metrics endpoint without requiring an API key, even when static-api-key authentication is enabled; other admin endpoints still require the key |
false |
Authentication
Like the API service, the Admin service supports authentication modes. See the API Authentication section for details.
The /status endpoint never requires an API key, so it can be used for liveness and readiness probes.
Node
Configuration for the Node service
Key: node
| Name | Description | Default Value |
|---|---|---|
bind-address |
Address to bind the Node service | 0.0.0.0 |
rpc-port |
Port for the Node RPC service | 5118 |
task-slots |
Number of task slots for the Node | 16 |
Worker
Configuration for pipeline workers
Key: worker
| Name | Description | Default Value |
|---|---|---|
bind-address |
Address to bind the Worker service | 0.0.0.0 |
rpc-port |
RPC port for the worker to listen on; set to 0 to use a random available port | 0 |
data-port |
Data port for the worker to listen on; set to 0 to use a random available port | 0 |
task-slots |
Number of task slots for the Worker | 16 |
queue-size |
Size of the queues between nodes in the dataflow graph | 8192 |
checkpoint-details-to-keep |
Worker mode only: number of recent checkpoints whose details the leader keeps for the Web UI and checkpoints API | 10 |
Schedulers
Configuration for the various schedulers
Process Scheduler
Key: process-scheduler
| Name | Description | Default Value |
|---|---|---|
slots-per-process |
Number of slots per process in the scheduler | 16 |
shutdown-with-controller |
Whether worker processes are killed when the controller shuts down. Disabling this can leave orphaned worker processes, and is mostly useful for testing controller recovery | true |
Manual Scheduler
The manual scheduler does not start workers itself. Instead, when a pipeline is scheduled it prints the commands needed to start each worker, which you then run yourself. This is mostly useful for testing and debugging scheduling behavior.
Key: manual-scheduler
| Name | Description | Default Value |
|---|---|---|
slots-per-process |
Number of task slots assigned to each worker; the scheduler prints a command for enough workers to cover all of the pipeline’s slots | 16 |
Kubernetes Scheduler
Key: kubernetes-scheduler
Some values for the kubernetes scheduler are complete Kubernetes object, for
example, the worker.resources object can be specified as a
Kubernetes resource object.
When specifying these via environment variables they should be encoded as Yaml.
See the Kubernetes deployment docs for more details.
There are two modes for allocating resources for Kubernetes, specified as the kubernetes-scheduler.resource-mode:
- In
per-slotmode, tasks are packed onto workers up to thetask-slotsconfig, and for each slot the amount of resources specified inresourcesis provided. This can be much more efficient for diversely-sized pipelines - In
per-podmode, every pod has exactlytask-slotsslots, and exactly the resources inresources, even if it is scheduled for fewer slots. This is the behavior from before 0.11.
| Name | Description | Default Value |
|---|---|---|
namespace |
Kubernetes namespace for the scheduler | default |
resource-mode |
Resource allocation mode; per-slot or per-pod |
per-slot |
worker.name-prefix |
Prefix for worker names | arroyo |
worker.image |
Docker image for workers | ghcr.io/arroyosystems/arroyo:latest |
worker.image-pull-policy |
Image pull policy for worker containers | IfNotPresent |
worker.service-account-name |
Service account name for worker containers | default |
worker.resources.requests |
Kubernetes resource object representing the requests for the worker pods | {cpu: "900m", memory: "500Mi"} |
worker.resources.limits |
Kubernetes resource object representing the limits for the worker pods | none |
worker.task-slots |
Number of task slots per worker | 16 |
worker.command |
Command to start worker containers | /app/arroyo worker |
worker.env |
List of environment variables for worker containers, each a k8s-style map with name and value keys | none |
Database
Key: database
| Name | Description | Default Value |
|---|---|---|
type |
Type of the database (either sqlite or postgres) |
sqlite |
sqlite.path |
Path to the database file | $(user config dir)/arroyo/config.sqlite |
postgres.database-name |
Name of the Postgres database | arroyo |
postgres.host |
Host of the Postgres database | localhost |
postgres.port |
Port of the Postgres database | 5432 |
postgres.user |
User for the Postgres database | arroyo |
postgres.password |
Password for the Postgres database | arroyo |
postgres.schema |
Postgres schema that Arroyo’s tables live in. The schema must already exist, and may only contain ASCII letters, numbers, and underscores. Useful for sharing a Postgres database with other services | public |
postgres.tls.mode |
TLS mode for connections to Postgres; one of no-tls, skip-verification, system-roots, or ca-cert (see below) |
no-tls |
postgres.tls.path |
Path to a PEM-encoded CA certificate; required when postgres.tls.mode is ca-cert |
none |
Postgres TLS
By default, Arroyo connects to Postgres without TLS. The postgres.tls.mode option supports the following modes:
no-tls: Connect over plaintextskip-verification: Encrypt the connection, but do not verify the server’s certificatesystem-roots: Encrypt the connection and verify the server’s certificate against the system trust store; suitable for managed Postgres providers with publicly-signed certificatesca-cert: Encrypt the connection and verify the server’s certificate against the system trust store plus the CA certificate atpostgres.tls.path
For example:
[database.postgres]
host = "my-db.example.com"
tls = { mode = "ca-cert", path = "/etc/arroyo/postgres-ca.pem" }or via environment variables:
ARROYO__DATABASE__POSTGRES__TLS__MODE=ca-cert
ARROYO__DATABASE__POSTGRES__TLS__PATH=/etc/arroyo/postgres-ca.pemLogging
Key: logging
| Name | Description | Default Value |
|---|---|---|
format |
Set the log format (one of json, logfmt, or plaintext) |
plaintext |
nonblocking |
Whether to use nonblocking logging; this uses more memory but ensures processing is not blocked by a high rate of logging | false |
buffered-lines-limit |
Number of lines to buffer before dropping logs or exerting backpressure on senders; only valid when nonblocking is set to true |
4096 |
enable-file-line |
Whether to record the source file line in the log | false |
enable-file-name |
Whether to record the source file name in the log | false |
static-fields |
Map of additional fields to include in every log line; only applies to the json format |
none |
In the json format, every log line also includes an arroyo_service field with the name of the
service that emitted it and, for pipeline workers, a pipeline_id field. Additional fields can be
added with static-fields:
[logging]
format = "json"
static-fields = { cluster = "prod-us-east-1", team = "data-platform" }