Skip to content

Getting Started

Before You Begin

API Registration

Before using the Datarock Core API, you'll need to register for API access and generate authentication keys. This process involves:

  1. Confirming permissions: Only project managers can register for API access
  2. Generating a keypair: Create a public-private key pair for authentication
  3. Registering your public key: Upload your public key through the Datarock Platform UI

For detailed instructions on API registration and authentication setup, see our API Registration Guide.

API Version

This Getting Started guide covers version 3 (v3) of the Datarock Core API. If you're using a different version, please refer to the appropriate documentation for your API version.

How to use this API

The Datarock Core API provides a complete workflow for discovering and exporting product data. Here's the typical flow:

1. Fetch All Projects

Start by retrieving all projects you have access to:

GET /projects
curl -X GET "https://api.prod.datarock.com.au/v3/projects" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature"

Response: 

[
  {
    "uuid": "project-uuid-1",
    "name": "Project Name 1"
  },
  {
    "uuid": "project-uuid-2",
    "name": "Project Name 2"
  }
]

2. Fetch Available Product Data for a Project

Once you have a project UUID, discover what product data are available:

GET /projects/{project-uuid}/artefacts
curl -X GET "https://api.prod.datarock.com.au/v3/projects/{project-uuid}/artefacts" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature"

Response: 

{
  "artefacts": [
    {
      "name": "Artefact Name",
      "key": "artefact_key",
      "description": "Artefact description",
      "versions": [
        {
          "version": "v1"
        }
      ],
      "product": {
        "name": "Product Name"
      },
      "productCategory": {
        "name": "Product Category Name"
      }
    }
  ]
}

3. Get Holes for the Project (Optional)

If you need to know which holes are available in a project:

GET /projects/{project-uuid}/holes
curl -X GET "https://api.prod.datarock.com.au/v3/projects/{project-uuid}/holes" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature"

Response: 

[
  {
    "id": "hole-id-1"
  },
  {
    "id": "hole-id-2"
  }
]

4. Create a New Export

Request an export for a specific product data type and holes. You must provide the key of the product data type you want to export, as well as the version. Note - When creating an export, the API will always use the latest pipeline run it can find for that product. If you specify a time filter, the pipeline must have started within the time range that you specify.

POST /export/artefacts/{artefact-key}/v1
curl -X POST "https://api.prod.datarock.com.au/v3/export/artefacts/{artefact-key}/v1" \
  -H "Content-Type: application/json" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature" \
  -d '{
    "projectUuid": "project-uuid",
    "holes": ["hole-id-1", "hole-id-2"],
    lastUpdatedSince": 9876543210,
    lastUpdatedSince": 1234567890
  }'

Response: 

{
  "status": "Success",
  "exportId": "export-uuid"
}

Example: How do I export all my project data for the last 24 hours?

To export data that was updated in the last 24 hours, you can use the lastUpdatedSince parameter. First, calculate the Unix timestamp for 24 hours ago, then include it in your export request:

POST /export/artefacts/{artefact-key}/v1 (24hr example)
curl -X POST "https://api.prod.datarock.com.au/v3/export/artefacts/{artefact-key}/v1" \
  -H "Content-Type: application/json" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature" \
  -d '{
    "projectUuid": "project-uuid",
    "holes": ["hole-id-1", "hole-id-2"],
    "lastUpdatedSince": 1234567890
  }'

Note: The lastUpdatedSince value should be a Unix timestamp representing 24 hours ago. You can calculate this programmatically: - JavaScript: Math.floor((Date.now() - (24 * 60 * 60 * 1000)) / 1000) - Python: int((datetime.now() - timedelta(days=1)).timestamp())

5. Check Export Status

Poll the export status until it's completed. Note: Large exports can take a while to complete.

GET /export/artefacts/{export-uuid}
curl -X GET "https://api.prod.datarock.com.au/v3/export/artefacts/{export-uuid}" \
  -H "X-API-User: your-email@example.com" \
  -H "Signature: your-jwt-signature"

Response (In Progress): 

{
  "uuid": "export-uuid",
  "status": "in_progress",
  "projectId": "project-uuid",
  "type": {
    "name": "Artefact Name",
    "key": "artefact_key"
  },
  "expiryDate": 1234567890,
  "artefacts": {
    "data": [
      {
        "holeId": "hole-id-1",
        "status": "in_progress",
        "exportType": { /* artefact details */ }
      }
    ],
    "count": 2,
    "limit": 500,
    "offset": 0,
    "total": 2
  }
}

Response (Completed): 

{
  "uuid": "export-uuid",
  "status": "completed",
  "projectId": "project-uuid",
  "type": {
    "name": "Artefact Name",
    "key": "artefact_key"
  },
  "expiryDate": 1234567890,
  "artefacts": {
    "data": [
      {
        "holeId": "hole-id-1",
        "status": "completed",
        "exportType": { /* artefact details */ },
        "url": "https://example.com/export-file-1.csv",
        "linkExpiryTime": 1234567890,
        "runAt": 1234567890
      },
      {
        "holeId": "hole-id-2",
        "status": "completed",
        "exportType": { /* artefact details */ },
        "url": "https://example.com/export-file-2.csv",
        "linkExpiryTime": 1234567890,
        "runAt": 1234567890
      }
    ],
    "count": 2,
    "limit": 500,
    "offset": 0,
    "total": 2
  }
}

6. Download Your Data

Once the export is completed, use the provided URLs to download your data. The download links are valid for 1 hour, and the export itself expires after 7 days.

Key Points to Remember

  • Asynchronous Processing: Exports are processed asynchronously - you must poll the status endpoint to fetch the state of an export.
  • Product Keys: When requesting product data from the API, you should always use the key, and not the description or name of the product data.
  • Rate Limits: The API has rate limits (1000 requests per 5 minutes for most endpoints, 300 for exports)
  • Authentication: All requests require proper JWT signature authentication
  • Expiry: Export URLs expire after 1 hour, but you can re-fetch them by calling the status endpoint again
  • Export Lifecycle: Exports are available for 7 days after creation

Next Steps