Skip to content

Polling HTTP

Read from streaming HTTP sources

The Polling HTTP source periodically polls an HTTP endpoint and emits the response as a record. This can be used to turn any HTTP API into a streaming data source.

Configuring the Connection

Polling HTTP sources can be created via the Web UI or directly in SQL.

polling http source creationg flow

An polling http connection has several required and optional fields:

Field Description Required Example
endpoint The endpoint for the HTTP server Yes https://api.example.com/v1/events
headers A comma-separated list of colon separated key-value pairs of headers that will be sent to the server No Content-Type:application/json,Authorization: Bearer mytoken
method The HTTP method to use when polling the endpoint No GET
body The body to send with the request No {"foo": "bar"}
poll_interval_ms The interval at which to poll the endpoint, in milliseconds No 5000
emit_behavior Controls when records are emitted; can be all or changed No changed

For example, in SQL:

CREATE TABLE prs (
    value TEXT
) WITH (
    connector = 'polling_http',
    endpoint = 'https://api.github.com/repos/ArroyoSystems/arroyo/pulls?per_page=1&state=all',
    poll_interval_ms = 5000,
    emit_behavior = 'changed',
    headers = 'User-Agent:arroyo/0.6',
    format = 'json',
    'json.unstructured' = true
);

Type to search…

↑↓ navigate↵ selectEsc close