How to configure the format for sources and sinks
format
option in SQL.
json
- JSON data in any formatdebezium_json
- JSON data in the format produced by
Debezium for reading and writing from relational databases like PostgresOption | Description | Default |
---|---|---|
format | The format of the data. Must be one of json or debezium_json . | json |
json.confluent_schema_registry | Set to true if data was produced by (or will be consumed via) a Confluent Schema Registry connected source | false |
json.include_schema | Set to true to include the schema in the output, allowing it to be used with Kafka Connect connectors that require a schema | false |
json.unstructured | Set to true to treat the data as unstructured JSON, which will be parsed as a single column of type TEXT | false |
json.timestamp_format | The format of timestamps in the data. May be one of rfc3339 or unix_millis | rfc3339 |
TEXT
with JSON-serialized data in it.
Unstructured data is treated as a single column named value
with type TEXT
, which can be operated on using
SQL json functions or UDFs.
Option | Description | Default |
---|---|---|
format | The format of the data. Must be avro | avro |
avro.confluent_schema_registry | Set to true if data was produced by (or will be consumed via) a Confluent Schema Registry connected source | false |
avro.raw_datums | Set to true to serialize and deserialize as raw Avro datums instead of complete Avro records | false |
avro.into_unstructured_json | Convert the avro record to JSON | false |
avro.into_unstructured_json
option, if set, will cause the Avro data to be deserialized and re-serialized
to JSON, which can then be operated on using SQL json functions or UDFs.
This can be useful if the Avro schema for the data may change, and offers flexibility in how the data is
processed.
avro.confluent_schema_registry
option is set to true
. This allows the schema to be used by
other applications that read from the same topic.
Schema registry
.
raw_string
format. Raw string tables have a single column named value
with
type TEXT
, for example:
value
with type BYTEA
:
&[u8]
, for example: