Arroyo 0.8 is now available, with a new FileSystem source, Delta Lake sink, Redis sink, Avro support, global UDFs, and more.

What does it mean to apply SQL—a batch-oriented query language—to streams of data that are never complete? Read on for a deep dive into streaming SQL in Arroyo and other engines.

The easiest way to run a highly-scaled production Arroyo cluster is on Kubernetes. Setting up a Kubernetes cluster used to be a daunting task, but services like Amazon EKS have made it much easier. This post will walk through how to set up an EKS cluster and deploy Arroyo to it.

Recent versions of Arroyo have added support for HTTP sources, and treating individual lines of a response as streaming messages. So I wondered: could we use Arroyo to directly process metrics?

Arroyo 0.7.0 is now available, with custom partitioning for s3 writes, message framing, unnest, union, state compaction, and more.

Arroyo is written in Rust, a modern systems language. We think it's become the best choice for writing high-performance systems like databases and stream processing engines. Read on for why we chose Rust, and what we've learned along the way.

Arroyo 0.6 brings support for Google Cloud Storage, user-defined aggregate functions, SQL correctness tests, and more

Arroyo 0.5 added the FileSystem connector, a high-performance, transactional sink that lets you write pipeline outputs to file systems and object stores like S3—and makes Arroyo a great tool for performing real-time ETL. This turns out to be surprisingly tricky to do well. Read on for a deep dive into how Arroyo solved this with a new checkpointing strategy and some clever Parquet tricks.

Working with real-time data can be daunting. We're working to solve that by building a new stream processing engine that's easy enough for anyone to use. So how easy is it to solve real-world streaming problems with Arroyo today? I decided to find out.

At Arroyo we're building a new stream processing engine to replace legacy Java systems like Flink and KSQL. So we were excited to see a project that's doing the same thing for Kafka. It's called WarpStream, and they're building a replacement for Kafka that's backed directly by S3.

Release 0.5 of Arroyo is all about connectors. We've added a high-performance transactional FileSystem sink, exactly-once Kafka support, a Kinesis connector, and more.

Flink is a mature and powerful streaming engine. So why didn't we build Arroyo on top of it?

With the 0.4.0 release we've added Debezium support, a new REST API, and made the process of contributing connectors much easier

The Arroyo 0.3.0 release adds UDFs, DDL statements, custom event time and watermarks, web UI improvements, and more.

Testing a complex system like Arroyo is hard. But with Rust's powerful proc macros, we're able to easily produce end-to-end tests of our SQL features.

Arroyo 0.2.0 brings a number of improvements including native Kubernetes support, new SQL features, and many other fixes and improvements.

After launching our state-of-the-art cloud real-time data processor, we're opening up the technology that powers it: the Arroyo streaming engine

Arroyo's Rust-based stream processing engine outperforms Apache Flink in sliding window queries due to its efficient algorithms that maintain near-constant throughput even with smaller slides and larger windows