Stream URL

  • To stream updates for all entities in the namespace:

    {ApiServerUrl}/${namespace}/stream?...
    
  • To stream updates for the ${entity} entity only:

    {ApiServerUrl}/${namespace}/${entity}/stream?...
    

The response to a request to the stream URL is a JSON document similar to the following:

Message typeExample
UPDATE<br>{<br> "cursor": 12125764,<br> "at": "2017-01-01T00:00:00Z",<br> "messageType": "UPDATE",<br> "entityType": "${entity}",<br> "data": [...],<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "12125764,123"<br>}<br>
DELETE<br>{<br> "cursor":89589,<br> "at": "2017-01-01T00:00:00Z",<br> "messageType":"DELETE",<br> "entityType":"${entity}",<br> "ids":["012345678911"],<br> "from":"2017-01-01T23:00:00Z",<br> "to":"2017-01-02T23:00:00Z",<br> "dateTimeField":"asOfDateTime",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "89589"<br>}<br>
EMPTY<br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"EMPTY",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br>
DICTIONARY_UPDATE<br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"DICTIONARY_UPDATE",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br>
PURGE<br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"PURGE",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br>
REFRESH_CACHE<br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"REFRESH_CACHE",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br>
CLEAR_DATA<br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"CLEAR_DATA",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br>

Where:

  • cursor is the current position in the stream.

    Note: The cursor value increases with each subsequent message in the stream, but it is not guaranteed that the values are consecutive. Furthermore, the cursor applies to the whole namespace, so even when streaming /${namespace}/${entity}/stream, the cursor values are a subset of those provided by the whole-namespace /${namespace}/stream URL.

    If a particular message does not contain ${entity} data, then an EMPTY response is returned for that cursor on the /${namespace}/${entity}/stream URL.

  • at is the time the message entered the stream.

    Note: This is a technical field with no business meaning.

  • messageType is the type of message:

    • UPDATE messages have a data field that is an array of updated or new instances. For example, if entityType is ${entity}, then data is an array of ${entity} instances.

    • DELETE messages have an ids field that is an array of the IDs of entities to be deleted.

      DELETE messages may also define a from and to time range in which records are to be deleted, otherwise DELETE indicates that all records with that entityType and ID are deleted from the namespace.

    • PURGE messages indicate data was completely removed from the platform, including from the /stream service. The ids of the deleted records are not provided.

    • DICTIONARY_UPDATE indicates that the namespace dictionary has been updated, potentially adding new fields or entities to the data model.

Applies to: batch

  • BATCH_RUN_EVENT messages can be returned when the data is sent in discrete batches. BATCH_RUN_EVENT marks the start and end of each batch run. However, both start and end events are only published when the entire batch run is completed.

  • EMPTY messages have no data and are returned in the following circumstances:

    • EMPTY messages can be returned if the head of the stream is reached and there are no new updates. These EMPTY messages return an after value equal to the value submitted in the request's after parameter.
    • EMPTY messages can be returned when no data matching the query and entitlements can be found; the REST API service iterates through a number of messages for matching data but returns EMPTY (with an updated after value) if no matching data is found in a reasonable time. These EMPTY messages return an after value different to the value submitted in the request's after parameter.

    Note: Regardless of the type of EMPTY message that has been returned, the correct response to any EMPTY message is to immediately poll the REST API server with the after value in the response.

  • Other message types are occasionally sent. These include:

    • CLEAR_DATA: indicating the REST API will drop data for a given entity.
    • REFRESH_CACHE: indicating the REST API will refresh its in-memory caches.
  • The range of message types occurring in the /stream service evolves over time.

    Important: A /stream service client application must handle new message types gracefully, for example by logging unknown message types, rather than failing on encountering an unknown message type.

  • entityType is the type of data to which the message applies.

  • timeToFirstWrite gives the processing time that elapsed on the REST API server before the server started sending the response to the client.

  • after is the value to use for the after parameter in the next request to obtain the next message in the stream.

    When the /stream service is filtered by from and to parameters, it is possible that no stream messages match the applied filters. In these cases, the /stream service returns an EMPTY message with after=-1, indicating that no message could be returned and therefore no meaningful after value can be provided. An after value of -1 should not be used in a subsequent request; instead consider if the request parameters or timing of the request should be adjusted before retrying.

Important: Note that new fields can be added to the /stream envelope in the future. The Consumer API can add new fields to the message envelope or message body at any time, and client applications must handle new fields gracefully.

On subsequent calls to the stream URL, you must include the after parameter to ensure you get the next message from the input stream. You must set after equal to the after response field value:

{ApiServerUrl}/${namespace}/stream?after=AFTER_VALUE

Important: Poll for the next message in the stream immediately after receiving the response. You should process the message in a separate thread to avoid blocking or delaying the polling process.

Note: If the /stream request contains no after parameter or after=head, the stream URL always returns the most recent message in the stream. If an after value is specified, but there is no /stream message after the supplied after value, then a message of type EMPTY is returned.

Note: If after=scan is specified in the request, the /stream service returns the most recent message with data from the stream. If no data can be found by scanning backward in the stream within a given period, then a message of type EMPTY is returned.

It is possible to limit the size of the response by setting the limit parameter on the request. For example, if an UPDATE message on the REST API server contains 1000 records and the request limit is set to 200, then the message is split over five requests, each containing 200 records. The after field of the response is constructed to enable the request of the next part of the overall server message.

It is also possible to set limit to a high number up to 10000, and set aggregate=true to return data from multiple stream messages in one response with the total number of records returned not exceeding the limit set.

.../stream?aggregate=true&limit=10000

Less data will be returned if there is no further data in the stream. The response message format is changed when aggregate=true is set. Instead of a data field containing the data, there is a messages field containing an array of messages.

Using aggregate=true is particularly useful when processing streams with a high update rate, as the aggregate responses reduce the impact of network latency.

Filtering the stream by message type and other meta data

It is possible to filter the stream based on message meta data such as the message type or publication mode. these meta data filters are indexes so can be used in some circumstances to quickly reduce the volume of the stream. Common meta data filters include:

  • messageType: filter for specific message types such as UPDATE.
  • mode: filter by publication mode, including NORMAL, BACKFILL, RESTATEMENT and ADJUSTMENT.

Applies to: batch

  • batchRun: filter the stream for a specific batch run.
  • receivedSequence: enables linking from the /latest and /timeseries services to the /stream.

Back to help overview