---
title: "File System"
description: "Write Parquet and JSON to S3 and local filesystems"
---

> Documentation Index
> Fetch the complete documentation index at: https://www.arroyo.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# File System

Arroyo provides the capability to read and write Parquet and JSON files to/from object stores and local filesystems.
When used as a sink, Arroyo will produce complete files in line with the checkpointing system.
As such, the file system sinks write all data exactly once.
This is done against S3 by tracking multi-part uploads within the state store,
allowing Arroyo to resume an in-progress upload in the event of a failure.

The FileSystem connector supports local filesystem, S3 (including S3-compatible stores like MinIO), GCS, Cloudflare R2, and Azure Blob Storage/ADLS Gen2.

As a source Arroyo reads all files to completion, at which point the job will finish.

## Common Configuration
Both the source and sink versions of the connector make use of Arroyo's StorageBackend,
which is a generalization of an object store.
The location within the StorageBackend is configured via the `path` variable in the `WITH` clause of the `CREATE TABLE` statement.
The value is a URL pointing to the destination directory. The most common examples are shown below.
 |  Description | Example |
 |  ----------- | ------- |
 | Local file | `file:///test-data/my-cool-arroyo-pipeline`|
 | S3 Path | `s3://awesome-arroyo-bucket/amazing-arroyo-dir` |
 | S3 HTTP Endpoint| `https://s3.us-west-2.amazonaws.com/awesome-arroyo-bucket/amazing-arroyo-dir`|
 | Local MinIO installation | `s3::http://localhost:9123/local_bucket/sweet-dir` |
 | Cloudflare R2 | `r2://my-bucket/path` or `r2://account-id@my-bucket/path` |
 | Azure Blob Storage | `abfs://container@account.dfs.core.windows.net/path` |
 | Azure HTTPS | `https://account.blob.core.windows.net/container/path` |
 | GCS | `gs://my-bucket/path` |

### Additional Backend Configuration

The StorageBackend can be passed additional configuration options, which are namespaced with "storage." at the beginning.
This allows you to pass in custom endpoints, credentials, and regions.

#### S3 and S3-Compatible Storage (including MinIO and R2)

| Field | Description | Example |
| ----- | ----------- | ------- |
| `storage.aws_region` | Manually set the AWS region | `us-east-1` |
| `storage.aws_endpoint` | Manually set the AWS endpoint | `https://s3-custom-endpoint.com` |
| `storage.aws_secret_access_key` | Manually set the AWS secret access key | `your-secret-key` |
| `storage.aws_access_key_id` | Manually set the AWS access key ID | `your-access-key-id` |

#### Cloudflare R2

R2 can be configured using the `r2://` URL scheme. Authentication can be provided via environment variables or storage options:

**Environment Variables:**
- `CLOUDFLARE_ACCOUNT_ID` or set as part of the URL
- `R2_ACCESS_KEY_ID` or `AWS_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY` or `AWS_SECRET_ACCESS_KEY`

**Storage Options:**
- `storage.r2_access_key_id` or `storage.aws_access_key_id`
- `storage.r2_secret_access_key` or `storage.aws_secret_access_key`

**URL Formats:**
- `r2://bucket/path` - Account ID from environment
- `r2://account-id@bucket/path` - Account ID in URL
- `https://account-id.r2.cloudflarestorage.com/bucket/path` - Full endpoint URL
- `https://account-id.eu.r2.cloudflarestorage.com/bucket/path` - With jurisdiction (e.g., EU)

#### Azure Blob Storage / ADLS Gen2

Azure storage uses standard Azure authentication via environment variables. The following URL formats are supported:

**URL Formats:**
- `abfs://container@account.dfs.core.windows.net/path` - ADLS Gen2 format
- `abfss://container@account.dfs.core.windows.net/path` - ADLS Gen2 with SSL
- `https://account.blob.core.windows.net/container/path` - Blob storage HTTPS
- `https://account.dfs.core.windows.net/container/path` - ADLS Gen2 HTTPS

**Authentication:**
Azure authentication is configured via standard Azure environment variables. Refer to the
[Azure SDK documentation](https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string)
for details on authentication options.

### Format
Both sources and sinks require a format, and support `parquet` and `json`. JSON files written by the sink can be
gzip-compressed (see [JSON options](#json-options)), and the source can read compressed JSON files (see
[source configuration](#source-specific-configuration)).

## Sink Specific Configuration

### Sink version

Arroyo 0.16 introduced a new implementation of the sink, `v2`, which is now the default. The original
implementation, `v1`, is still available and can be selected with the `sink.version` option.

| Field | Description | Default |
| ----- | ----------- | -------- |
| `sink.version` | Which sink implementation to use: `v1` or `v2` | `'v2'` |


### File rollover configs

These configs control when files are rolled over, creating new files. Rolling over
more frequently allows lower latency for readers, but at the cost of more files, which
can slow down queries.

| Field | Description | Default | Example |
| ----- | ----------- | -------- | ------- |
| `rolling_policy.file_size` | Target size in a file before it is rolled; at most 5TB | None | `'128MB'` |
| `rolling_policy.interval` | Amount of time a file after creation that a file will be rolled, expressed as a SQL interval | None | `interval '30 minutes'` |
| `rolling_policy.inactivity_interval` | Amount of time a file will be open without any new data before it will be rolled, expressed as a SQL interval | None | `interval '1 hour'` |
| `rolling_policy.watermark_expiration` | Roll time-partitioned files once the event-time watermark moves past their partition's time bucket. Only applies when `partitioning.time_pattern` is set, so it has no effect on the Iceberg sink | `false` | `true` |


#### File Naming Options

By default Arroyo names files using [UUIDv7](https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_7_(timestamp_and_random)),
which ensures that they sort in time-order. This behavior can be configured via the following options:

| Field | Description | Default |
| ----- | ----------- | -------- |
| `filename.prefix` | Prefix that will be appended to the beginning of the file name, followed by a `-` | None |
| `filename.suffix` | Suffix (extension) that will be appended to the end of the file name, preceded by a `.` | `'parquet'`, `'json'`, or `'json.gz'` (for gzip-compressed JSON) |
| `filename.strategy` | Filenaming strategy to use. Supported values: `serial`, `uuid`, `ulid`, `uuid_v7` | `'uuid_v7'` |

### Parquet Options

| Field | Description | Default |
| ----- | ----------- | -------- | 
| `parquet.compression` | The compression codec to use for Parquet files. Supported values: `none`, `snappy`, `gzip`, `zstd`, `lz4`, `lz4_raw`. | `'zstd'` |
| `parquet.row_group_size` | Target size for each Parquet row group; at most 1GB | `'128MB'` |

### Multipart upload settings

These options allow configuring the behavior of how writes against object storage are split into parts:

| Field | Description | Default |
| ----- | ----------- | -------- |
| `multipart.target_part_size` | The target size of each multipart upload. Must be between 5MB and 5GB. | `'32MB'` |
| `multipart.max_parts` | Maximum number of multipart uploads before a multipart will be completed | `1000` |
| `multipart.minimum_multipart_size` | (v2 only) Files that are closed before reaching this size are uploaded with a single request instead of a multipart upload. Must be at most 5GB. The default of `0` uses a multipart upload for every file | `0` |

For local filesystem paths, the sink always uses a separate local writer, so `sink.version` and the multipart
upload settings have no effect.

### JSON Options

| Field | Description | Default |
| ----- | ----------- | -------- |
| `json.compression` | Compression for JSON files written by the sink: `uncompressed` or `gzip`. Gzip-compressed files use the `json.gz` suffix by default. Only supported by the filesystem connector | `'uncompressed'` |

The other JSON format options are described in the [formats docs](https://www.arroyo.dev/docs/connectors/formats#json).

### Partitioning Options

Arroyo supports partitioning of outputs. There are two types of partitioning: event time-based and field-based.
You can use either or both of these types of partitioning.
If both are used, the time-based partitioning is placed prior to the field-based partitioning.

#### Event Time-based Partitioning
Event time partitioning uses each record's event_time, formatting it using a strftime-style formatting string.
You can set the `partitioning.time_pattern` key in the sink to define the pattern.

Example:
`partitioning.time_pattern = '%Y/%m/%d/%H'`

By default, time-partitioned files are rolled according to the [file rollover configs](#file-rollover-configs)
like any other file. To also close a file as soon as the event-time watermark moves past its time bucket (for example,
closing the files for `2026/01/15/09` once the watermark passes 10:00), set `rolling_policy.watermark_expiration = true`.
This lets readers see each partition's files sooner after its time window ends.

> **Changed in 0.16**
>
> Before Arroyo 0.16, time-partitioned files were always rolled when the watermark passed their time bucket. To keep
> that behavior after upgrading, set `rolling_policy.watermark_expiration = true`.

#### Field-based Partitioning
Field-based formatting produces a string mirroring the Hive-style partition directories,
so partitioning on field_1, field_2 will result in folders like `field_1=X/field_2=Y`.
 You can set the `partitioning.fields` key in the sink to define the partition fields.

Example:
`partitioning.fields = 'field_1,field_2'`

#### Shuffle by partition

When using field-based partitioning and high parallelism, you may end up with many files;
typically each sink subtask will write a file for every partition key. To avoid this,
you can configure the dataflow to insert a shuffle step before the sink, which will ensure
that all records for a particular partition key end up on the same sink node:

```
'shuffle_by_partition.enabled' = true
```

For example, if our partition key is `event_type` and we have 100 distinct
types, at parallelism 32 we'd end up with 3,200 files being written for each
flush interval. By enabling shuffle_by_partition, we reduce that 100.

Note that this may lead to performance problems if your data is highly skewed
across your partition keys; for example, if 90% of your data is in the same
partition, those events will all end up on the same sink subtask which may
not be able to keep up with the volume.

## Source Specific Configuration
When using the file system source, the following options are available
| Field | Description | Default | Example |
| ----- | ----------- | -------- | ------- |
| `source.compression` | The compression format of the files to read. Supported values: `none`, `zstd`, `gzip`. Only used for JSON input | `none` | `gzip` |
| `source.regex_pattern` | A regex pattern to match files to read. If specified all files within the path will be evaluated against pattern. If not specified only files directly under the path will be read.  | None | `.*\.json` |


## File System Sink DDL

Here's an example for how to create a table to write parquet to S3 with partitioning:
```sql
CREATE TABLE bids (
  auction bigint,
  bidder bigint,
  price bigint,
  datetime timestamp,
  region text,
  account_id text
) WITH (
  connector = 'filesystem',
  type = 'sink',
  path = 'https://s3.us-west-2.amazonaws.com/demo/s3-uri',
  format = 'parquet',
  'parquet.compression' = 'zstd',
  'rolling_policy.interval' = interval '60 seconds',
  'partitioning.time_pattern' = '%Y/%m/%d/%H',
  'partitioning.fields' = 'region,account_id'
);
```

## File System Source DDL

Here's an example for how to create a table to read parquet from S3:
```sql
CREATE TABLE bids (
  auction bigint,
  bidder bigint,
  price bigint,
  datetime timestamp,
  region text,
  account_id text
) WITH (
  connector = 'filesystem',
  type = 'source',
  path = 'https://s3.us-west-2.amazonaws.com/demo/s3-uri',
  format = 'parquet',
  'source.regex_pattern' = '.*\.parquet$'
);
```

Source: https://www.arroyo.dev/docs/connectors/filesystem/index.mdx
