Stream URL
To stream updates for all entities in the namespace:
{ApiServerUrl}/${namespace}/stream?...To stream updates for the ${entity} entity only:
{ApiServerUrl}/${namespace}/${entity}/stream?...
The response to a request to the stream URL is a JSON document similar to the following:
| Message type | Example |
|---|---|
| UPDATE | <br>{<br> "cursor": 12125764,<br> "at": "2017-01-01T00:00:00Z",<br> "messageType": "UPDATE",<br> "entityType": "${entity}",<br> "data": [...],<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "12125764,123"<br>}<br> |
| DELETE | <br>{<br> "cursor":89589,<br> "at": "2017-01-01T00:00:00Z",<br> "messageType":"DELETE",<br> "entityType":"${entity}",<br> "ids":["012345678911"],<br> "from":"2017-01-01T23:00:00Z",<br> "to":"2017-01-02T23:00:00Z",<br> "dateTimeField":"asOfDateTime",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "89589"<br>}<br> |
| EMPTY | <br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"EMPTY",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br> |
| DICTIONARY_UPDATE | <br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"DICTIONARY_UPDATE",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br> |
| PURGE | <br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"PURGE",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br> |
| REFRESH_CACHE | <br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"REFRESH_CACHE",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br> |
| CLEAR_DATA | <br>{<br> "cursor":19101427,<br> "at":"2017-06-29T08:27:09.991Z",<br> "messageType":"CLEAR_DATA",<br> "entityType":"${entity}",<br> "publishRequestId":"X0.1606987416.1",<br> "timeToFirstWrite":1,<br> "after": "19101427"<br>}<br> |
Where:
cursoris the current position in the stream.Note: The cursor value increases with each subsequent message in the stream, but it is not guaranteed that the values are consecutive. Furthermore, the cursor applies to the whole namespace, so even when streaming
/${namespace}/${entity}/stream, the cursor values are a subset of those provided by the whole-namespace/${namespace}/streamURL.If a particular message does not contain ${entity} data, then an EMPTY response is returned for that cursor on the
/${namespace}/${entity}/streamURL.atis the time the message entered the stream.Note: This is a technical field with no business meaning.
messageTypeis the type of message:UPDATEmessages have adatafield that is an array of updated or new instances. For example, ifentityTypeis ${entity}, thendatais an array of ${entity} instances.DELETEmessages have anidsfield that is an array of the IDs of entities to be deleted.DELETEmessages may also define afromandtotime range in which records are to be deleted, otherwiseDELETEindicates that all records with thatentityTypeand ID are deleted from the namespace.PURGEmessages indicate data was completely removed from the platform, including from the /stream service. The ids of the deleted records are not provided.DICTIONARY_UPDATEindicates that the namespace dictionary has been updated, potentially adding new fields or entities to the data model.
Applies to: batch
BATCH_RUN_EVENTmessages can be returned when the data is sent in discrete batches.BATCH_RUN_EVENTmarks the start and end of each batch run. However, both start and end events are only published when the entire batch run is completed.EMPTYmessages have no data and are returned in the following circumstances:EMPTYmessages can be returned if the head of the stream is reached and there are no new updates. TheseEMPTYmessages return anaftervalue equal to the value submitted in the request'safterparameter.EMPTYmessages can be returned when no data matching the query and entitlements can be found; the REST API service iterates through a number of messages for matching data but returnsEMPTY(with an updatedaftervalue) if no matching data is found in a reasonable time. TheseEMPTYmessages return anaftervalue different to the value submitted in the request'safterparameter.
Note: Regardless of the type of
EMPTYmessage that has been returned, the correct response to anyEMPTYmessage is to immediately poll the REST API server with theaftervalue in the response.Other message types are occasionally sent. These include:
CLEAR_DATA: indicating the REST API will drop data for a given entity.REFRESH_CACHE: indicating the REST API will refresh its in-memory caches.
The range of message types occurring in the /stream service evolves over time.
Important: A /stream service client application must handle new message types gracefully, for example by logging unknown message types, rather than failing on encountering an unknown message type.
entityTypeis the type of data to which the message applies.timeToFirstWritegives the processing time that elapsed on the REST API server before the server started sending the response to the client.afteris the value to use for theafterparameter in the next request to obtain the next message in the stream.When the /stream service is filtered by from and to parameters, it is possible that no stream messages match the applied filters. In these cases, the /stream service returns an EMPTY message with
after=-1, indicating that no message could be returned and therefore no meaningfulaftervalue can be provided. Anaftervalue of -1 should not be used in a subsequent request; instead consider if the request parameters or timing of the request should be adjusted before retrying.
Important: Note that new fields can be added to the /stream envelope in the future. The Consumer API can add new fields to the message envelope or message body at any time, and client applications must handle new fields gracefully.
On subsequent calls to the stream URL, you must include the after parameter to ensure you get the next message from the input stream. You must set after equal to the after response field value:
{ApiServerUrl}/${namespace}/stream?after=AFTER_VALUE
Important: Poll for the next message in the stream immediately after receiving the response. You should process the message in a separate thread to avoid blocking or delaying the polling process.
Note: If the /stream request contains no
afterparameter orafter=head, the stream URL always returns the most recent message in the stream. If anaftervalue is specified, but there is no /stream message after the suppliedaftervalue, then a message of typeEMPTYis returned.
Note: If
after=scanis specified in the request, the /stream service returns the most recent message with data from the stream. If no data can be found by scanning backward in the stream within a given period, then a message of typeEMPTYis returned.
It is possible to limit the size of the response by setting the limit parameter on the request. For example, if an UPDATE message on the REST API server contains 1000 records and the request limit is set to 200, then the message is split over five requests, each containing 200 records. The after field of the response is constructed to enable the request of the next part of the overall server message.
It is also possible to set limit to a high number up to 10000, and set aggregate=true to return data from multiple stream messages in one response with the total number of records returned not exceeding the limit set.
.../stream?aggregate=true&limit=10000
Less data will be returned if there is no further data in the stream. The response message format is changed when aggregate=true is set. Instead of a data field containing the data, there is a messages field containing an array of messages.
Using aggregate=true is particularly useful when processing streams with a high update rate, as the aggregate responses reduce the impact of network latency.
Filtering the stream by message type and other meta data
It is possible to filter the stream based on message meta data such as the message type or publication mode. these meta data filters are indexes so can be used in some circumstances to quickly reduce the volume of the stream. Common meta data filters include:
messageType: filter for specific message types such as UPDATE.mode: filter by publication mode, including NORMAL, BACKFILL, RESTATEMENT and ADJUSTMENT.
Applies to: batch
batchRun: filter the stream for a specific batch run.receivedSequence: enables linking from the /latest and /timeseries services to the /stream.