Skip to main content

How to read the collector

Concept of the collector

When you read a collector, in addition to the RESULT chunks, you will receive CONTROL chunks where you need to store in your system the offset ID: streaming_collector_resume_offset.png

This ID will allow you to read the collector starting from the last known position in the collector.

To restart from the last known article in the collector, store the offset ID and use it in the parameter “resume_offset.”

note

To start from the first article in the collector, the offset ID is earliest.

Read and listen to the collector as a stream

To read a collector as a stream, you should set the parameter end_behaviour to the value wait. With this parameter, once you reach the end of the collector (the queue), you will receive a last CONTROL chunk with the offset ID, but the connection remains open and new articles or mentions will be pushed to the collector and to the listening system:

To restart from the last known article in the collector, store the offset ID and use it in the parameter resume_offset.

Commanf
curl -X GET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?resume_offset=<offset_id>&end_behaviour=wait&access_token=<access_token>'

Read the collector as a queue

To read a collector as a queue, you should set the parameter end_behaviour to the value stop. With this parameter, once you reach the end of the collector (the queue), you will receive a last CONTROL chunk with the offset ID, and the connection will be closed.

Even if you are not reading the collector, new articles and mentions are still pushed to the collector and will consume credits.

To restart from the last known article in the collector, store the offset ID and use it in the parameter resume_offset.

Command
curl -X GET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?resume_offset=<offset_id>&end_behaviour=stop&access_token=<access_token>'