Arroyo includes two types of Redis connectors: a sink and a lookup connector, for lookup joins.

​
Sink

The Redis sink allows you to write data to a Redis instance or cluster. Redis is a great choice for use cases where you need low-latency access to the outputs of your Arroyo pipeline.

The Redis sink uses pipelining to achieve high throughput and is capable of writing millions of records per second to a single Redis instance.

Redis supports a variety of data structures. Currently, the Redis sink supports writing to the following data structures:

Each value in Redis is identified by a key. The configuration of the Redis sink allows you to construct the key in various ways, including a fixed prefix and a suffix constructed from the columns of the result.

The value for each record will be the serialized representation of the row, using the configured format.

​
Configuring the Connection

Redis sinks can be created in the Web UI or directly in SQL.

​
General configuration

FieldDescriptionRequiredExample
addressThe address of a single, non-clustered Redis instanceNo*redis://localhost:6379
cluster.addressesA comma-separated list of addresses of Redis instances in a clusterNo*redis://localhost:6379,redis://localhost:6380
typeThe type of table (currently only sink is supported)Yessink
targetThe data type to write to. Currently, string, hash, and list are supportedYesstring
target.key_prefixA prefix that will be prepended to all keysYesoutputs.
target.key_columnsA column whose value will be appended to the key for each recordNoid

​
string target

FieldDescriptionRequiredExample
target.ttl_secsThe time-to-live for each key in secondsNo3600

​
list target

FieldDescriptionRequiredExample
target.max_lengthThe max length for this list; once the list exceeds this, the oldest elements will be droppedNo100
target.operationThe operation to perform on the list; one of ‘append’ (default) or ‘prepend’Noappend

​
hash target

FieldDescriptionRequiredExample
target.field_columnThis column will be used as the field name in the hashYeskey

​
Example

CREATE TABLE redis (
  user_id TEXT NOT NULL,
  event_count BIGINT NOT NULL,
) WITH (
    connector = 'redis',
    type = 'sink',
    format = 'json',
    'address' = 'redis://localhost:6379',
    target = 'string',
    'target.key_prefix' = 'outputs.',
    'target.key_column' = 'user_id'
);

INSERT INTO redis
SELECT user_id, count(*)
FROM events
GROUP BY
    user_id, hop(interval '5 seconds', interval '1 hour');

Given the above query, we can see the following output in Redis:

$ redis-cli
127.0.0.1:6379> GET outputs.fred
"{\"user_id\":\"fred\",\"event_count\":10}"

​
Lookup

Redis can also be used as a source for lookup joins. When used in a lookup join, queries are made via a Redis GET command, supporting key-value records.

Lookup tables must currently be created via SQL. A Redis lookup table looks like this:

CREATE TEMPORARY TABLE customers (
    -- For Redis lookup tables, it's required that there be a single
    -- METADATA FROM 'key' marked as PRIMARY KEY, as Redis only supports
    -- efficient lookups by key
    customer_id TEXT METADATA FROM 'key' PRIMARY KEY,
    name TEXT,
    plan TEXT
) with (
    connector = 'redis',
    address = 'redis://localhost:6379',
    format = 'json',
    'lookup.cache.max_bytes' = 1000000,
    'lookup.cache.ttl' = interval '5 seconds'
);

The address and format arguments are required.