Polling HTTP

The Polling HTTP source periodically polls an HTTP endpoint and emits the response as a record. This can be used to turn any HTTP API into a streaming data source.

Configuring the Connection

Polling HTTP sources can be created via the Web UI or directly in SQL.

An polling http connection has several required and optional fields:

Field	Description	Required	Example
endpoint	The endpoint for the HTTP server	Yes	https://api.example.com/v1/events
headers	A comma-separated list of colon separated key-value pairs of headers that will be sent to the server	No	Content-Type:application/json,Authorization: Bearer mytoken
method	The HTTP method to use when polling the endpoint	No	GET
body	The body to send with the request	No	{"foo": "bar"}
poll_interval_ms	The interval at which to poll the endpoint, in milliseconds	No	5000
emit_behavior	Controls when records are emitted; can be all or changed	No	changed

For example, in SQL:

CREATE TABLE prs (
    value TEXT
) WITH (
    connector = 'polling_http',
    endpoint = 'https://api.github.com/repos/ArroyoSystems/arroyo/pulls?per_page=1&state=all',
    poll_interval_ms = 5000,
    emit_behavior = 'changed',
    headers = 'User-Agent:arroyo/0.6',
    format = 'json',
    'json.unstructured' = true
);