The Polling HTTP source periodically polls an HTTP endpoint and emits the response as a record. This can be used to turn any HTTP API into a streaming data source.

​
Configuring the Connection

Polling HTTP sources can be created via the Web UI or directly in SQL.

polling http source creationg flow

An polling http connection has several required and optional fields:

FieldDescriptionRequiredExample
endpointThe endpoint for the HTTP serverYeshttps://api.example.com/v1/events
headersA comma-separated list of colon separated key-value pairs of headers that will be sent to the serverNoContent-Type:application/json,Authorization: Bearer mytoken
methodThe HTTP method to use when polling the endpointNoGET
bodyThe body to send with the requestNo{"foo": "bar"}
poll_interval_msThe interval at which to poll the endpoint, in millisecondsNo5000
emit_behaviorControls when records are emitted; can be all or changedNochanged

For example, in SQL:

CREATE TABLE prs (
    value TEXT
) WITH (
    connector = 'polling_http',
    endpoint = 'https://api.github.com/repos/ArroyoSystems/arroyo/pulls?per_page=1&state=all',
    poll_interval_ms = '5000',
    emit_behavior = 'changed',
    headers = 'User-Agent:arroyo/0.6',
    format = 'json',
    'json.unstructured' = 'true'
);
MySQLPostgres