v0.12.0

Arroyo Documentation

Introduction

Getting started

Concepts

Architecture

Configuration

First pipeline

Real-time analytics on the Mastodon firehose

Mastodon trends

Kafka

Ingesting from Postgres tables with Debezium

Debezium

Use user-defined functions to parse custom formats

Parsing with UDFs

Overview

How to configure the format for sources and sinks

Formats

SQL Data Types

SELECT Statements

DDL Statements

Streaming Windows

Joins

Semantics for updating tables in Arroyo SQL

Updating Tables

Supported Aggregate Functions in Arroyo SQL

Aggregate Functions

Window Functions

Extending Arroyo with user-defined functions

UDF Overview

Pipeline clusters are a lightweight mode for running individual jobs

Pipeline clusters

Kubernetes

Setting up an Arroyo cluster on VMs or bare-metal servers

VMs and bare-metal

Set up your environment to develop Arroyo

Developer setup

API Overview

v0.14.0

v0.13.0

v0.11.0

v0.10.0

v0.9.0

v0.8.0

v0.7.0

v0.6.0

v0.5.0

v0.4.0

v0.3.0

v0.2.0

Github

Discord

API Reference

Releases

Arroyo Cloud

Blackhole Sink

Connect Arroyo to a Confluent Kafka topic

Confluent

Write Parquet files to a Delta Lake Table

Delta Lake Sink

Write Parquet and JSON to S3 and local filesystems

File System

Fluvio

Impulse

Use Amazon Kinesis streams as sources and sinks

Kinesis

MQTT

MySQL

Publish and consume from Core NATS and Jetstream

NATS

Polling HTTP

Postgres

RabbitMQ

Redis

Redpanda

Server-Sent Events

Webhook

WebSocket

Scalar functions for mathematical operations

Math functions

Scalar functions to implement conditional logic

Conditional functions

Scalar functions for manipulating strings

String functions

Scalar functions for manipulating binary strings

Binary string functions

Regex functions

JSON functions

Scalar functions for handling times and dates

Time and date functions

Array functions

Scalar functions for manipulating structs

Struct functions

Hashing functions

Other functions

Rust UDFs

Rust UDAFs

Async Rust UDFs

Python UDFs

List all connectors

List all connection profiles

Create connection profile

Test connection profile

Delete a Connection Profile

Get autocomplete suggestions for a connection profile

List all connection tables

Create a new connection table

Test a Connection Schema

Test a Connection Table

Delete a Connection Table

Get all jobs

List a job's checkpoints

Get a checkpoint's details

List a job's error messages

Get a job's metrics

Subscribe to a job's output

List all pipelines

The API will create a single job for the pipeline.

Create a new pipeline

Create a new preview pipeline

Validate a query and return pipeline graph

Get a single pipeline

Delete a pipeline

Update a pipeline

List a pipeline's jobs

Restart a pipeline

Get Global UDFs

Create a global UDF

Validate UDFs

Delete UDF