The Polling HTTP source periodically polls an HTTP endpoint and emits the response as a record. This can be used to turn any HTTP API into a streaming data source.

​
Configuring the Connection

Polling HTTP sources can be created via the Web UI or directly in SQL.

An polling http connection has several required and optional fields:

FieldDescriptionRequiredExample
endpointThe endpoint for the HTTP serverYeshttps://api.example.com/v1/events
headersA comma-separated list of colon separated key-value pairs of headers that will be sent to the serverNoContent-Type:application/json,Authorization: Bearer mytoken
methodThe HTTP method to use when polling the endpointNoGET
bodyThe body to send with the requestNo{"foo": "bar"}
poll_interval_msThe interval at which to poll the endpoint, in millisecondsNo5000
emit_behaviorControls when records are emitted; can be all or changedNochanged

For example, in SQL:

CREATE TABLE prs (
    value TEXT
) WITH (
    connector = 'polling_http',
    endpoint = 'https://api.github.com/repos/ArroyoSystems/arroyo/pulls?per_page=1&state=all',
    poll_interval_ms = '5000',
    emit_behavior = 'changed',
    headers = 'User-Agent:arroyo/0.6',
    format = 'json',
    'json.unstructured' = 'true'
);
NATSPostgres