How to read the collector
Concept of the collector
When you read a collector, in addition to the RESULT chunks, you will receive CONTROL chunks where you need to store in your system the offset ID:
This ID will allow you to read the collector starting from the last known position in the collector.
To restart from the last known article in the collector, store the offset ID and use it in the parameter “resume_offset.”
To start from the first article in the collector, the offset ID is earliest.
Read and listen to the collector as a stream
To read a collector as a stream, you should set the parameter end_behaviour to the value wait. With this parameter, once you reach the end of the collector (the queue), you will receive a last CONTROL chunk with the offset ID, but the connection remains open and new articles or mentions will be pushed to the collector and to the listening system:
To restart from the last known article in the collector, store the offset ID and use it in the parameter resume_offset.
curl -X GET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?resume_offset=<offset_id>&end_behaviour=wait&access_token=<access_token>'
Read the collector as a queue
To read a collector as a queue, you should set the parameter end_behaviour to the value stop. With this parameter, once you reach the end of the collector (the queue), you will receive a last CONTROL chunk with the offset ID, and the connection will be closed.
Even if you are not reading the collector, new articles and mentions are still pushed to the collector and will consume credits.
To restart from the last known article in the collector, store the offset ID and use it in the parameter resume_offset.
curl -X GET 'https://api.talkwalker.com/api/v3/stream/c/<collector_id>/results?resume_offset=<offset_id>&end_behaviour=stop&access_token=<access_token>'