Skip to content

Async Scraper

Big crawls. Zero queues to run.

Submit jobs one at a time or 10,000 per batch. We schedule them against your concurrency, retry what fails, and call your webhook with a signed payload the moment each one is done.

Get 5,000 free credits Read the docs
URLs per batch request
10,000URLs per batch request
results retained for polling
72hresults retained for polling
signed webhook deliveries
HMACsigned webhook deliveries

In practice

Submit a batch with a webhook

request
const res = await fetch("https://scrapely.15.204.158.166.sslip.io/v1/batches", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCRAPELY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    urls: productUrls, // up to 10,000
    render: true,
    callback_url: "https://hooks.yourapp.com/scrapely",
  }),
});

const { batch_id, jobs } = await res.json();
response
{
  "event": "job.completed",
  "job_id": "job_8KQ2v7mXcN",
  "batch_id": "bat_3Fz91LpQ",
  "status_code": 200,
  "credits": 10,
  "body_url": "https://scrapely.15.204.158.166.sslip.io/v1/jobs/job_8KQ2v7mXcN/body"
}

Capabilities

What you get with Async Scraper

How it works

Three steps, no infrastructure

  1. Submit

    POST one job or a batch. You get ids back immediately.

  2. 02

    We work through it

    Jobs run at your concurrency limit, with retries and refunds handled.

  3. 03

    Results arrive

    Signed webhook per job, or poll whenever suits your pipeline.

Use cases

What it is built for

The workloads it is designed around.

  • Nightly catalogue crawls

    Hundreds of thousands of product pages before the morning stand-up.

  • Backfills

    Re-fetch a year of archived URLs without building a queue.

  • Event-driven pipelines

    Trigger downstream ETL from the webhook.

  • Spiky workloads

    Burst to millions, then go quiet — credits do not expire.

FAQ

Async Scraper questions

How do I verify a webhook?

Compute HMAC-SHA256 over the timestamp and raw body with your webhook secret and compare it to the signature header in constant time. The docs include examples for five languages.

What happens if my endpoint is down?

Deliveries are retried with exponential backoff for 24 hours. Results remain available to poll for 72 hours either way.

Do batches cost more?

No. A job costs exactly what the same synchronous request would cost.

Retailers on the same credits

Your first 5,000 credits are on us

Create an account, copy your key, fetch your first products from any live retailer. No card, no subscription, no expiry date on what you buy later.

Get 5,000 free creditsTalk to us about volume

POST scrapely.15.204.158.166.sslip.io/v1/products { "platform", "marketplace", "ids": […] }