Batches

Submit up to 10,000 URLs with shared options in one call, and track them as one batch.

Create a batch

A batch is a convenience over jobs: one call creates one job per URL (each with its own id, result and webhook) plus a batch id for aggregate progress. Options apply to every URL.

cURL

curl -X POST "https://api.scrapely.dev/v1/batches" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/p/1", "https://example.com/p/2"],
    "country_code": "us",
    "output_format": "markdown",
    "callback_url": "https://your.app/hooks/scrapely"
  }'

Python

import requests

urls = [line.strip() for line in open("urls.txt") if line.strip()]
batch = requests.post(
    "https://api.scrapely.dev/v1/batches",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"urls": urls, "output_format": "markdown"},
).json()
print(batch["id"], batch["total"], batch["credits_reserved"])

Response

{
  "id": "batch_9a8b7c6d5e4f3a2b",
  "object": "batch",
  "status": "queued",
  "total": 2,
  "counts": { "queued": 2, "running": 0, "finished": 0, "failed": 0, "cancelled": 0 },
  "progress": 0,
  "credits_reserved": 2,
  "credits_charged": 0,
  "job_ids": ["job_1f…", "job_2a…"]
}

All or nothing: The whole batch is reserved up front: URLs × cost per URL. If your balance cannot cover it, the request is a 402 and nothing is created. Failed jobs are refunded one by one as they fail.

Validation

urls is an array (a newline-separated string also works). Every URL is validated before anything is charged; the first bad one is reported by index, e.g. param: "urls[1834]".

Track progress

GET /v1/batches/{id} returns counts per status, progress (0–1), and credits_charged so far. Add include_jobs=true for a compact per-job list; fetch each job for its content, or let webhooks deliver them.