Connectors
Polling HTTP
Read from streaming HTTP sources
The Polling HTTP source periodically polls an HTTP endpoint and emits the response as a record. This can be used to turn any HTTP API into a streaming data source.
Configuring the Connection
Polling HTTP sources can be created via the Web UI or directly in SQL.
An polling http connection has several required and optional fields:
Field | Description | Required | Example |
---|---|---|---|
endpoint | The endpoint for the HTTP server | Yes | https://api.example.com/v1/events |
headers | A comma-separated list of colon separated key-value pairs of headers that will be sent to the server | No | Content-Type:application/json,Authorization: Bearer mytoken |
method | The HTTP method to use when polling the endpoint | No | GET |
body | The body to send with the request | No | {"foo": "bar"} |
poll_interval_ms | The interval at which to poll the endpoint, in milliseconds | No | 5000 |
emit_behavior | Controls when records are emitted; can be all or changed | No | changed |
For example, in SQL:
CREATE TABLE prs (
value TEXT
) WITH (
connector = 'polling_http',
endpoint = 'https://api.github.com/repos/ArroyoSystems/arroyo/pulls?per_page=1&state=all',
poll_interval_ms = '5000',
emit_behavior = 'changed',
headers = 'User-Agent:arroyo/0.6',
format = 'json',
'json.unstructured' = 'true'
);