Fetch pages

Overview

Takes up to ten URLs in one request and returns readable page content as markdown, HTML, or structured JSON, along with page metadata such as title, description, language, author, and publication date. Extraction can be scoped with CSS selectors that either restrict it to part of a page or strip unwanted elements, and a plain-language statement of what the caller is after steers what gets prioritised. Results and per-URL errors come back separately, so one bad URL does not fail the batch.

One entry per URL with its title, metadata, and extracted content, and a separate list of URLs that failed.

Input

These go in the input object of the run request.

Requests up to 1 MB.

Example

Use your own API key and a saved idempotency key for each new job. Replace image or media placeholders with your own publicly reachable HTTPS URLs. Keep the same key and request body when recovering a timeout.

# Set once per job; reuse this key and the identical body after a timeout.
: "${AGENTSKY_IDEMPOTENCY_KEY:?Set a unique, saved key for this job}"
curl --fail-with-body https://gateway.agentsky.dev/v1/run \
  -H "authorization: Bearer $AGENTSKY_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: $AGENTSKY_IDEMPOTENCY_KEY" \
  -d '{"provider":"tinyfish","endpoint":"fetch","input":{"urls":["https://example.com"],"format":"markdown"}}'
# If status is READY/RUNNING, poll GET /v1/runs/{runId} with the same API key.

$0 per call using your plan when signed in. Read the final customer charge from price.amount.value after the run finishes. A RUNNING response is not the final bill.

Questions

More from TinyFish