Webhooks

Receive finished jobs as signed HTTPS callbacks, and verify that each one really came from Scrapely.

Delivery

When a job ends we POST it to the job's callback_url, or, if it has none, to your account webhook (Dashboard → Webhooks). One event per job, batches included.

{
  "id": "evt_1a2b3c4d5e6f7a8b9c0d",
  "type": "job.finished",
  "created_at": "2026-10-15T09:12:05.512Z",
  "data": {
    "id": "job_5f0c8a1b2d3e4f56",
    "object": "job",
    "status": "finished",
    "url": "https://example.com/products/42",
    "credits": 10,
    "status_code": 200,
    "response": {
      "status_code": 200,
      "content_type": "text/html; charset=utf-8",
      "content": "<!doctype html>…"
    }
  }
}

Responding and retries

Answer with any 2xx within 10 seconds; do the real work after responding. Anything else — a 4xx/5xx, a timeout, a redirect — is retried after 10 s, 1 min, 5 min, 30 min and 2 h. Redirects are never followed.

Make handlers idempotent: Delivery is at least once: a retry can arrive after a response we did not see. Record Scrapely-Delivery ids you have processed and skip repeats.

Verifying signatures

Compute HMAC-SHA256 over "<t>.<raw body>" with your webhook signing secret (whsec_…, Dashboard → Webhooks), and compare it to v1 in constant time. Refuse timestamps more than 5 minutes old: that is what stops a captured delivery being replayed.

Use the raw body: Verify the exact bytes you received, before JSON parsing. Re-serialising changes whitespace and the signature will not match.

Node.js

import { createHmac, timingSafeEqual } from 'node:crypto';

export function verifyScrapely(rawBody, header, secret, toleranceSec = 300) {
  const parts = Object.fromEntries(header.split(',').map((p) => p.split('=')));
  const t = Number(parts.t);
  if (!t || Math.abs(Date.now() / 1000 - t) > toleranceSec) return false;
  const expected = createHmac('sha256', secret).update(`${t}.${rawBody}`).digest();
  const given = Buffer.from(parts.v1 ?? '', 'hex');
  return given.length === expected.length && timingSafeEqual(given, expected);
}

// Express: keep the raw body for this route.
app.post('/hooks/scrapely', express.raw({ type: 'application/json' }), (req, res) => {
  const raw = req.body.toString('utf8');
  if (!verifyScrapely(raw, req.get('Scrapely-Signature') ?? '', process.env.SCRAPELY_WEBHOOK_SECRET)) {
    return res.sendStatus(400);
  }
  res.sendStatus(200);
  queue.add(JSON.parse(raw)); // work after acknowledging
});

Python

import hmac, hashlib, time

def verify_scrapely(raw_body: bytes, header: str, secret: str, tolerance: int = 300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    t = int(parts.get("t", 0))
    if not t or abs(time.time() - t) > tolerance:
        return False
    signed = f"{t}.".encode() + raw_body
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts.get("v1", ""))

# Flask
@app.post("/hooks/scrapely")
def scrapely_hook():
    if not verify_scrapely(request.get_data(), request.headers.get("Scrapely-Signature", ""), SECRET):
        abort(400)
    enqueue(request.get_json())
    return "", 200

Go

func VerifyScrapely(rawBody []byte, header, secret string, tolerance time.Duration) bool {
	var ts int64
	var sig string
	for _, p := range strings.Split(header, ",") {
		k, v, _ := strings.Cut(p, "=")
		switch k {
		case "t":
			ts, _ = strconv.ParseInt(v, 10, 64)
		case "v1":
			sig = v
		}
	}
	if ts == 0 || time.Since(time.Unix(ts, 0)).Abs() > tolerance {
		return false
	}
	mac := hmac.New(sha256.New, []byte(secret))
	fmt.Fprintf(mac, "%d.", ts)
	mac.Write(rawBody)
	given, err := hex.DecodeString(sig)
	return err == nil && hmac.Equal(given, mac.Sum(nil))
}

PHP

function verify_scrapely(string $rawBody, string $header, string $secret, int $tolerance = 300): bool {
    parse_str(str_replace(',', '&', $header), $parts);
    $t = (int) ($parts['t'] ?? 0);
    if (!$t || abs(time() - $t) > $tolerance) return false;
    $expected = hash_hmac('sha256', $t . '.' . $rawBody, $secret);
    return hash_equals($expected, $parts['v1'] ?? '');
}

$ok = verify_scrapely(file_get_contents('php://input'), $_SERVER['HTTP_SCRAPELY_SIGNATURE'] ?? '', $secret);

During a secret rotation the header may carry two v1= values; accept the request if either matches.

Allowed destinations

callback_url must be https://, on port 443 or 1024–65535, and resolve to a public address. Loopback, private (RFC 1918), link-local and cloud metadata addresses are refused at creation and again at delivery. For local development, use a tunnel (ngrok, Cloudflare Tunnel) or webhook.site.