Skip to main content

Export task principles

Overview on export tasks

Exports allow to create asynchronous tasks that copy a selection of past data into a collector, which can then be accessed through collectors.

There are three options to define the source of past data:

  • Existing stream definitions (via a stream ID)
  • Talkwalker projects (via project ID)
  • Specifying a query in the request body

An ongoing task can be checked or aborted by using its task ID, included in the response.

There are 3 POST endpoints which can execute an export task. These 3 endpoints share the following parameters, set inside the body:

starttimestamp (milliseconds since 1.1.1970, e.g. 1539302400000) or date of the timeframe's start (2018-10-12). Day is included.required
stoptimestamp or date of the timeframe's end. Day is excluded from the export.optional
targetID of the collectorrequired
querythe query to search for (conjunctive to existing queries, i.e. matching all)optional
limitthe maximum number of results to export before interruptingoptional1.000.000
  • The maximum number of exported results per task is 1.000.000
  • Each exported result consumes 1 credit. Exporting the same result multiple times due to overlapping export tasks therefore requires multiple credits.

The 3 different endpoints to create export tasks

Creation of export tasks for Talkwalker projects

POST '<project_id>/export?access_token=<access_token>'

Creation of export tasks for existing streams

POST '<stream_id>/export?access_token=<access_token>'

Creation of export tasks based on query parameter

POST '<access_token>'

Steps to create an export task

  1. Create an empty collector which will act as a queue where the results from the export task will be pushed. This step is optional as we can use an existing running queue.
  2. Create the export task
  3. Wait the export task to complete
  4. Read the collector as a queue

In the coming page, we will present the export from a project. The steps remain the same for the other types of export tasks, only the endpoint to create export task will change.

Best practice

If results for longer time periods shall be exported, it makes sense to split the export task into multiple smaller export tasks (e.g. one month when exporting results for half a year).

This allows for a better estimation of the credit cost and the amount of results for the remaining time frame.