Export task principles
Overview on export tasks
Exports allow to create asynchronous tasks that copy a selection of past data into a collector, which can then be accessed through collectors.
There are three options to define the source of past data:
- Existing stream definitions (via a stream ID)
- Talkwalker projects (via project ID)
- Specifying a query in the request body
An ongoing task can be checked or aborted by using its task ID, included in the response.
There are 3 POST endpoints which can execute an export task. These 3 endpoints share the following parameters, set inside the body:
| parameter | description | required? | default |
|---|---|---|---|
| start | timestamp (milliseconds since 1.1.1970, e.g. 1539302400000) or date of the timeframe's start (2018-10-12). Day is included. | required | |
| stop | timestamp or date of the timeframe's end. Day is excluded from the export. | optional | |
| target | ID of the collector | required | |
| query | the query to search for (conjunctive to existing queries, i.e. matching all) | optional | |
| limit | the maximum number of results to export before interrupting | optional | 1.000.000 |
- The maximum number of exported results per task is 1.000.000
- Each exported result consumes 1 credit. Exporting the same result multiple times due to overlapping export tasks therefore requires multiple credits.
The 3 different endpoints to create export tasks
Creation of export tasks for Talkwalker projects
POST 'https://api.talkwalker.com/api/v3/stream/p/<project_id>/export?access_token=<access_token>'
Creation of export tasks for existing streams
POST 'https://api.talkwalker.com/api/v3/stream/s/<stream_id>/export?access_token=<access_token>'
Creation of export tasks based on query parameter
POST 'https://api.talkwalker.com/api/v3/stream/export?access_token=<access_token>'
Steps to create an export task
- Create an empty collector which will act as a queue where the results from the export task will be pushed. This step is optional as we can use an existing running queue.
- Create the export task
- Wait the export task to complete
- Read the collector as a queue
In the coming page, we will present the export from a project. The steps remain the same for the other types of export tasks, only the endpoint to create export task will change.
Best practice
If results for longer time periods shall be exported, it makes sense to split the export task into multiple smaller export tasks (e.g. one month when exporting results for half a year).
This allows for a better estimation of the credit cost and the amount of results for the remaining time frame.