---
title: "Configuration"
description: "Customizing the system behavior"
---

> Documentation Index
> Fetch the complete documentation index at: https://www.arroyo.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configuration

## Overview

Arroyo has a flexible and powerful configuration system that allows options to
be set via files (in TOML or Yaml format) and environment variables.

The system will look for configuration in the following places, from highest to lowest priority:

1. `ARROYO__*` environment variables
2. Config file specified via the `--config` option
3. Any *.toml or *.yaml files in the `--config-dir` directory
4. `arroyo.toml` in the current directory
5. `$(user conf dir)/arroyo/config.{toml,yaml}` — (this is `~/.config/arroyo` on Linux and `~/Library/Application Support/arroyo` on MacOS)
6. [Default configuration](https://github.com/ArroyoSystems/arroyo/blob/master/crates/arroyo-rpc/default.toml)

### Config files

In TOML or YAML, nested configurations are specified as tables under the given key name, for example:

```toml
checkpoint-url = 's3://my-bucket/checkpoints'

[controller]
scheduler = 'node'

[database]
type = "postgres"
```

### Environment variables

All configuration options can be set as environment variables as well. To convert a config name into an
environment variable, the following rule are applied:

1. Start with `ARROYO__`
2. Replace all dots (i.e., layers of nesting) with `__` (double underscore)
3. Replace all `-` with `_` (single underscore)
4. Uppercase all letters

Some examples:

* `checkpoint-url` => `ARROYO__CHECKPOINT_URL`
* `pipeline.compaction.enabled` => `ARROYO__PIPELINE__COMPACTION__ENABLED`
* `api.bind-address` => `ARROYO__API__BIND_ADDRESS`

Reasonable type conversions will be applied for values specified as environment variable, for example
numbers and booleans will be parsed into the correct type.

## Options

Here we list all of the available configuration options by the key they are nested under. So for example,
the option in the Pipeline section listed as `source-batch-size` would be specified in the config file as
`pipeline.source-batch-size` or as a table

```
[pipeline]
source-batch-size = 128
```

Top-level options:

| Name                        | Description                                                        | Default Value              |
|-----------------------------|--------------------------------------------------------------------|----------------------------|
| `checkpoint-url`            | URL of an object store or filesystem for storing checkpoints; in a distributed cluster this must be a location available to all nodes       | `/tmp/arroyo/checkpoints`  |
| `default-checkpoint-interval` | Default checkpointing interval                                 | `10s`                      |
| `api-endpoint`              | Endpoint of the API, used by other services to connect to it |   _inferred_    |
| `controller-endpoint`       | Endpoint of the controller, used by other services to connect to it | _inferred_ |
| `compiler-endpoint`       | Endpoint of the compiler, used by other services to connect to it | _inferred_ |
| `job-controller`            | Where each pipeline's job controller runs: `controller` runs it inside the Arroyo controller, while `worker` (beta) runs it on one of the pipeline's workers (the leader). See [Job controller modes](https://www.arroyo.dev/docs/architecture#job-controller-modes) | `controller` |
| `disable-telemetry`         | Disable open-source telemetry                                       |   false   |

### Pipeline

Configuration that applies to individual pipelines.

Key: `pipeline`

| Name                            | Description                                    | Default Value   |
|---------------------------------|------------------------------------------------|-----------------|
| `source-batch-size`             | Max size of source batches                         | `512`           |
| `source-batch-linger`           | Batch linger time (how long to wait before flushing)                 | `100ms`         |
| `source-batch-max-bytes`        | Flush a source batch once it has buffered this many bytes, measured as the raw size of the input data. Specified as an integer number of bytes; unset disables size-based flushing | _none_ |
| `update-aggregate-flush-interval`| How often to flush aggregates           | `1s`            |
| `allowed-restarts`              | How many restarts to allow before moving to failed (-1 for infinite)       | `20`            |
| `worker-heartbeat-timeout`      | Number of seconds to wait for a worker heartbeat before considering it dead                   | `30s`           |
| `healthy-duration`              | After this amount of time, we consider the job to be healthy and reset the restarts counter                     | `2m`            |
| `worker-startup-time`           | Amount of time to wait for workers to start up before considering them failed                       | `10m`           |
| `task-startup-time`             | Amount of time to wait for tasks to startup before considering it failed                         | `2m`            |
| `state-initial-backoff`         | Initial delay before the controller retries a pipeline after a retryable error; the delay doubles (with jitter) on each subsequent attempt | `500ms` |
| `state-max-backoff`             | Maximum delay between controller retries after retryable errors | `1m` |
| `compaction.enabled`            | Whether to enable compaction for checkpoints; not supported in [worker mode](https://www.arroyo.dev/docs/architecture#worker-mode)                  | `false`         |
| `compaction.checkpoints-to-compact`        | The number of outstanding checkpoints that will trigger compaction               | `4`             |
| `checkpoint.timeout` | [Worker mode](https://www.arroyo.dev/docs/architecture#worker-mode) only: maximum time to wait for the final checkpoint when stopping a pipeline with a checkpoint or rescaling it. If the checkpoint doesn't finish in time, the job fails and goes through normal recovery | _none_ |
| `chaining.enabled` | Whether to enable operator chaining, which reduces the number of operators in the pipeline | `true` |
| `store-deserialization-errors` | Whether to persist deserialization errors to the database | `true` |


### Run (pipeline clusters)

Configuration for pipeline clusters

Key: `run`


| Name                           | Description                                   | Default Value    |
|--------------------------------|-----------------------------------------------|------------------|
| `query`                        | The query to run for this pipeline cluster (equivalent to the query command-line parameter | _none_ |
| `state-dir`                    | Sets the directory that state will be written to and read from | _none_ |


### TLS Configuration

All services support TLS encryption, which can be configured globally for all services or per-service
by prefixing the key by the service name (for example, `api.tls`).

Key: `tls`

| Name            | Description                       | Required |
|-----------------|-----------------------------------|----------|
| `cert-file`     | Path to the TLS certificate file (PEM format) | Yes |
| `key-file`      | Path to the TLS private key file (PEM format) | Yes |
| `mtls-ca-file`  | Path to the CA certificate for mutual TLS (mTLS) authentication | No |


### gRPC

Configuration for the gRPC connections that Arroyo services use to communicate with each other.

Key: `grpc`

| Name               | Description                       | Default Value |
|--------------------|-----------------------------------|---------------|
| `max-message-size` | Maximum size of a gRPC message that a process will accept, specified as a data size like `32mb` or `512kb` (units are powers of 1024); must be less than `4gb`. Each process enforces this limit on the messages it receives; very large pipelines may need a higher limit | `32mb` |


### API

Configuration for the API service

Key: `api`

| Name            | Description                       | Default Value |
|-----------------|-----------------------------------|---------------|
| `bind-address`  | The host the API service should bind to   | `0.0.0.0`     |
| `http-port`     | The HTTP port for the API service     | `5115`        |
| `run-http-port`     | The HTTP port for the API service in run mode; defaults to a random port     | `0`        |

#### Authentication

The API service supports multiple authentication modes, configured via the `auth-mode` option:

**None (default)**
No authentication required.

```toml
[api]
auth-mode = { type = "none" }
```

**Static API Key**
Require a bearer token for all API requests.

```toml
[api]
auth-mode = { type = "static-api-key", api-key = "your-secret-key" }
```

Clients must include the API key in the `Authorization` header:

```
Authorization: Bearer your-secret-key

```

**Mutual TLS (mTLS)**
Require client certificates for authentication.

```toml
[tls]
cert-file = "/path/to/server-cert.pem"
key-file = "/path/to/server-key.pem"
mtls-ca-file = "/path/to/ca-cert.pem"

[api]
auth-mode = { type = "mtls", ca-cert-file = "/path/to/ca-cert.pem" }
```

With mTLS, clients must present a valid certificate signed by the specified CA.

#### CORS

By default, the API allows cross-origin requests from any origin. To restrict which
origins can call the API from a browser, switch to an allow-list:

```toml
[api.cors]
origin-policy = "allow-list"
allowed-origins = ["https://arroyo.example.com"]
allow-credentials = true
```

Key: `api.cors`

| Name                | Description                                    | Default Value |
|---------------------|------------------------------------------------|---------------|
| `origin-policy`     | Either `any`, which allows requests from all origins, or `allow-list`, which only allows the origins in `allowed-origins` | `any` |
| `allowed-origins`   | Origins allowed when `origin-policy` is `allow-list`. Each entry must be a bare origin (scheme, host, and optional port, such as `https://example.com:8443`) with no path or trailing slash; the wildcard `*` is not allowed. Must be empty when `origin-policy` is `any` | `[]` |
| `allow-credentials` | Whether browsers may send credentials (cookies or `Authorization` headers) with cross-origin requests. Can only be enabled with the `allow-list` policy | `false` |

The API service will fail to start if the CORS configuration is invalid.

### Controller

Configuration for the controller service

Key: `controller`

| Name            | Description                              | Default Value |
|-----------------|------------------------------------------|---------------|
| `bind-address`  | The host the controller should bind to   | `0.0.0.0`     |
| `rpc-port`      | The RPC port for the controller      | `5116`        |
| `scheduler`     | The scheduler to use; one of `process`, `kubernetes`, `node`, `embedded`, or `manual`    | `process`     |
| `metrics.enabled` | Whether to collect operator metrics for running pipelines, which power the metrics graphs in the Web UI and the operator metrics API | `true` |
| `leader-poll-interval` | [Worker mode](https://www.arroyo.dev/docs/architecture#worker-mode) only: how often the controller polls each pipeline's leader for its status | `500ms` |
| `connect-timeout` | [Worker mode](https://www.arroyo.dev/docs/architecture#worker-mode) only: timeout for the controller to establish a connection to a pipeline's leader | _none_ |


### Compiler

Configuration for the UDF compiler service.

Key: `compiler`

| Name              | Description                               | Default Value                 |
|-------------------|-------------------------------------------|-------------------------------|
| `bind-address`    | Bind address for the compiler      | `0.0.0.0`                     |
| `rpc-port`        | Port for the Compiler RPC service         | `5117`                        |
| `install-clang`   | Whether the compiler should attempt to install clang if it's not already installed                  | `true`                        |
| `install-rustc`   | Whether the compiler should attempt to install rustc if it's not already installed          | `true`                        |
| `artifact-url`    | Where to store compilation artifacts, in a distributed cluster this must be a location available to all nodes        | `/tmp/arroyo/artifacts`       |
| `build-dir`       | Directory for build files                 | `/tmp/arroyo/build-dir`       |
| `use-local-udf-crate` | Whether to use a local version of the UDF library or the published crate (only enable in development environments) | `false`                       |

### Admin

Configuration for the Admin service

Key: `admin`

| Name            | Description                          | Default Value |
|-----------------|--------------------------------------|---------------|
| `bind-address`  | Address to bind the Admin service    | `0.0.0.0`     |
| `http-port`     | Port for the Admin HTTP service      | `5114`        |
| `allow-unauthenticated-metrics` | Serve the Prometheus `/metrics` endpoint without requiring an API key, even when `static-api-key` authentication is enabled; other admin endpoints still require the key | `false` |

#### Authentication

Like the API service, the Admin service supports authentication modes. See the [API Authentication](#authentication) section for details.

The `/status` endpoint never requires an API key, so it can be used for liveness and readiness probes.


### Node

Configuration for the Node service

Key: `node`

| Name            | Description                          | Default Value |
|-----------------|--------------------------------------|---------------|
| `bind-address`  | Address to bind the Node service     | `0.0.0.0`     |
| `rpc-port`      | Port for the Node RPC service        | `5118`        |
| `task-slots`    | Number of task slots for the Node    | `16`          |


### Worker

Configuration for pipeline workers

Key: `worker`

| Name            | Description                              | Default Value |
|-----------------|------------------------------------------|---------------|
| `bind-address`  | Address to bind the Worker service       | `0.0.0.0`     |
| `rpc-port`      | RPC port for the worker to listen on; set to 0 to use a random available port          | `0`           |
| `data-port`     | Data port for the worker to listen on; set to 0 to use a random available port         | `0`           |
| `task-slots`    | Number of task slots for the Worker      | `16`          |
| `queue-size`    | Size of the queues between nodes in the dataflow graph                 | `8192`        |
| `checkpoint-details-to-keep` | [Worker mode](https://www.arroyo.dev/docs/architecture#worker-mode) only: number of recent checkpoints whose details the leader keeps for the Web UI and checkpoints API | `10` |


### Schedulers

Configuration for the various [schedulers](https://www.arroyo.dev/docs/architecture#schedulers)

#### Process Scheduler

Key: `process-scheduler`

| Name                | Description                                | Default Value |
|---------------------|--------------------------------------------|---------------|
| `slots-per-process` | Number of slots per process in the scheduler | `16`          |
| `shutdown-with-controller` | Whether worker processes are killed when the controller shuts down. Disabling this can leave orphaned worker processes, and is mostly useful for testing controller recovery | `true` |

#### Manual Scheduler

The manual scheduler does not start workers itself. Instead, when a pipeline is scheduled
it prints the commands needed to start each worker, which you then run yourself. This is mostly
useful for testing and debugging scheduling behavior.

Key: `manual-scheduler`

| Name                | Description                                | Default Value |
|---------------------|--------------------------------------------|---------------|
| `slots-per-process` | Number of task slots assigned to each worker; the scheduler prints a command for enough workers to cover all of the pipeline's slots | `16`          |

#### Kubernetes Scheduler

Key: `kubernetes-scheduler`

Some values for the kubernetes scheduler are complete Kubernetes object, for
example, the `worker.resources` object can be specified as a
[Kubernetes resource object](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#resources).

When specifying these via environment variables they should be encoded as Yaml.

See the [Kubernetes deployment docs](https://www.arroyo.dev/docs/deployment/kubernetes) for more details.

There are two modes for allocating resources for Kubernetes, specified as the `kubernetes-scheduler.resource-mode`:

* In `per-slot` mode, tasks are packed onto workers up to the `task-slots`
  config, and for each slot the amount of resources specified in `resources` is
  provided. This can be much more efficient for diversely-sized pipelines
* In `per-pod` mode, every pod has exactly `task-slots` slots, and exactly the
  resources in `resources`, even if it is scheduled for fewer slots. This
  is the behavior from before 0.11.



| Name                  | Description                                      | Default Value             |
|-----------------------|--------------------------------------------------|---------------------------|
| `namespace`           | Kubernetes namespace for the scheduler           | `default`                 |
| `resource-mode`       | Resource allocation mode; `per-slot` or `per-pod`                         | `per-slot`                |
| `worker.name-prefix`  | Prefix for worker names                          | `arroyo`                  |
| `worker.image`        | Docker image for workers                         | `ghcr.io/arroyosystems/arroyo:latest` |
| `worker.image-pull-policy` | Image pull policy for worker containers     | `IfNotPresent`            |
| `worker.service-account-name` | Service account name for worker containers | `default`                 |
| `worker.resources.requests` | Kubernetes resource object representing the requests for the worker pods    | `{cpu: "900m", memory: "500Mi"}`                 |
| `worker.resources.limits` | Kubernetes resource object representing the limits for the worker pods | none                 |
| `worker.task-slots`   | Number of task slots per worker                  | `16`                      |
| `worker.command`      | Command to start worker containers               | `/app/arroyo worker`      |
| `worker.env`          | List of environment variables for worker containers, each a k8s-style map with [name and value keys](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/#envvar-v1-core) | none                      |


### Database

Key: `database`

| Name                      | Description                                | Default Value             |
|---------------------------|--------------------------------------------|---------------------------|
| `type`                    | Type of the database (either `sqlite` or `postgres`)   | `sqlite`                  |
| `sqlite.path`             | Path to the database file                    | $(user config dir)/arroyo/config.sqlite |
| `postgres.database-name`  | Name of the Postgres database              | `arroyo`                  |
| `postgres.host`           | Host of the Postgres database              | `localhost`               |
| `postgres.port`           | Port of the Postgres database              | `5432`                    |
| `postgres.user`           | User for the Postgres database             | `arroyo`                  |
| `postgres.password`       | Password for the Postgres database         | `arroyo`                  |
| `postgres.schema`         | Postgres schema that Arroyo's tables live in. The schema must already exist, and may only contain ASCII letters, numbers, and underscores. Useful for sharing a Postgres database with other services | `public` |
| `postgres.tls.mode`       | TLS mode for connections to Postgres; one of `no-tls`, `skip-verification`, `system-roots`, or `ca-cert` (see below) | `no-tls` |
| `postgres.tls.path`       | Path to a PEM-encoded CA certificate; required when `postgres.tls.mode` is `ca-cert` | _none_ |

#### Postgres TLS

By default, Arroyo connects to Postgres without TLS. The `postgres.tls.mode` option supports the following modes:

* `no-tls`: Connect over plaintext
* `skip-verification`: Encrypt the connection, but do not verify the server's certificate
* `system-roots`: Encrypt the connection and verify the server's certificate against the system trust store;
  suitable for managed Postgres providers with publicly-signed certificates
* `ca-cert`: Encrypt the connection and verify the server's certificate against the system trust store
  plus the CA certificate at `postgres.tls.path`

For example:

```toml
[database.postgres]
host = "my-db.example.com"
tls = { mode = "ca-cert", path = "/etc/arroyo/postgres-ca.pem" }
```

or via environment variables:

```
ARROYO__DATABASE__POSTGRES__TLS__MODE=ca-cert
ARROYO__DATABASE__POSTGRES__TLS__PATH=/etc/arroyo/postgres-ca.pem
```

### Logging

Key: `logging`

| Name       | Description                  | Default Value |
|------------|------------------------------|---------------|
| `format`   | Set the log format (one of `json`, `logfmt`, or `plaintext`)  | `plaintext`          |
| `nonblocking` | Whether to use nonblocking logging; this uses more memory but ensures processing is not blocked by a high rate of logging | `false` |
| `buffered-lines-limit` | Number of lines to buffer before dropping logs or exerting backpressure on senders; only valid when `nonblocking` is set to true | `4096` |
| `enable-file-line` | Whether to record the source file line in the log | `false` |
| `enable-file-name` | Whether to record the source file name in the log | `false` |
| `static-fields` | Map of additional fields to include in every log line; only applies to the `json` format | _none_ |

In the `json` format, every log line also includes an `arroyo_service` field with the name of the
service that emitted it and, for pipeline workers, a `pipeline_id` field. Additional fields can be
added with `static-fields`:

```toml
[logging]
format = "json"
static-fields = { cluster = "prod-us-east-1", team = "data-platform" }
```

Source: https://www.arroyo.dev/docs/configuration/index.mdx
