Ingesting from Postgres tables with Debezium
checkpoints
table in the Arroyo postgres database,
which stores information on checkpoint progress and history for Arroyo
pipelines.
postgresql.conf
:
/opt/homebrew/var/postgresql@15/postgresql.conf
,
while on Ubuntu something like /etc/postgresql/14/main/postgresql.conf
.
Then restart Postgres to apply the changes:
checkpoints
table, we need to enable the REPLICA IDENTITY
to FULL
:
arroyo
user that we created for the Arroyo services,
but in production it’s recommended to create a separate user for Debezium.
docker-compose.yml
) and run it with docker-compose up
.
Confirm that the three docker images are running via docker ps
and tailing their logs.
curl
to create a debezium connection using the Debezium REST API.
First, create a file connector-config.json with contents
checkpoints
table to a topic named arroyo.public.checkpoints
.
"before"
, "after"
, and
"op"
, in line with the Debezium schema. However, we can also write more
complex queries over the source. For instance, consider the following query.