# Fetch pages

Canonical page: https://agentsky.dev/tools/tinyfish/fetch
JSON record: https://gateway.agentsky.dev/v1/endpoints/tinyfish/fetch

Fetch pages is a tool from TinyFish, callable through the AgentSky API as tinyfish/fetch and billed per call ($0).

Takes up to ten URLs in one request and returns readable page content as markdown, HTML, or structured JSON, along with page metadata such as title, description, language, author, and publication date. Extraction can be scoped with CSS selectors that either restrict it to part of a page or strip unwanted elements, and a plain-language statement of what the caller is after steers what gets prioritised. Results and per-URL errors come back separately, so one bad URL does not fail the batch.

One entry per URL with its title, metadata, and extracted content, and a separate list of URLs that failed.

## Facts

- **Id:** `tinyfish/fetch`
- **Provider:** TinyFish
- **Kind:** tool
- **Billed per:** call
- **Timeout:** 5 min
- **Max request:** 1 MB

## Call it

`POST https://gateway.agentsky.dev/v1/run` with `{"provider": "tinyfish", "endpoint": "fetch", "input": {…}}` and `Authorization: Bearer $AGENTSKY_API_KEY`.

### Input fields

| Field | Type | Required | Description |
|---|---|---|---|
| `urls` | array | yes | The pages to fetch, up to 10. |
| `format` | string | no | How to return the content. Allowed: html, markdown, json. Default: "markdown". |
| `purpose` | string | no | What the content is for, in plain language, to steer extraction. |
| `links` | boolean | no | Include every link found on the page. Default: false. |
| `ttl` | integer | no | How stale a cached copy may be, in seconds. |

## Example

### curl

```bash
# Set once per job; reuse this key and the identical body after a timeout.
: "${AGENTSKY_IDEMPOTENCY_KEY:?Set a unique, saved key for this job}"
curl --fail-with-body https://gateway.agentsky.dev/v1/run \
  -H "authorization: Bearer $AGENTSKY_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: $AGENTSKY_IDEMPOTENCY_KEY" \
  -d '{"provider":"tinyfish","endpoint":"fetch","input":{"urls":["https://example.com"],"format":"markdown"}}'
# If status is READY/RUNNING, poll GET /v1/runs/{runId} with the same API key.
```

### JavaScript

```js
// Node.js 20+. Save one unique key per job; reuse it with the identical body.
const key = process.env.AGENTSKY_IDEMPOTENCY_KEY;
if (!key || !process.env.AGENTSKY_API_KEY) {
  throw new Error("Set AGENTSKY_API_KEY and a saved AGENTSKY_IDEMPOTENCY_KEY first");
}
const base = "https://gateway.agentsky.dev";
const headers = {
  authorization: `Bearer ${process.env.AGENTSKY_API_KEY}`,
  "content-type": "application/json",
  "Idempotency-Key": key,
};
const body = JSON.stringify({
  "provider": "tinyfish",
  "endpoint": "fetch",
  "input": {
    "urls": [
      "https://example.com"
    ],
    "format": "markdown"
  }
});
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function request(path, options = {}) {
  for (let attempt = 0; attempt < 3; attempt++) {
    let response;
    let data;
    try {
      response = await fetch(base + path, {
        ...options, headers, signal: AbortSignal.timeout(180_000),
      });
      const text = await response.text();
      try { data = JSON.parse(text); } catch { /* Proxy timeouts may return HTML. */ }
    } catch (error) {
      if (attempt === 2) throw error; // Resume with the same key and body.
      await sleep(2000);
      continue;
    }
    // Failed runs can have non-2xx HTTP status. Never regenerate them automatically.
    if (data?.runId && data?.status) return data;
    if (response.ok && data) throw new Error("Unexpected response: missing runId/status");
    if (attempt < 2 && (response.status >= 500 || response.status === 408 || response.status === 429)) {
      await sleep(2000);
      continue; // POST retries keep the original key and body.
    }
    throw new Error(`HTTP ${response.status}: ${JSON.stringify(data ?? "Non-JSON response; retry with the same key and body")}`);
  }
}

let run = await request("/v1/run", { method: "POST", body });
console.log("Save runId:", run.runId);
const deadline = Date.now() + 30 * 60_000;
while (["READY", "RUNNING"].includes(run.status)) {
  if (Date.now() >= deadline) {
    throw new Error(`Still pending: poll /v1/runs/${run.runId} later; do not create a new job`);
  }
  await sleep(5000);
  run = await request(`/v1/runs/${encodeURIComponent(run.runId)}`);
}
console.log("Status:", run.status, "Charge:", run.price?.amount);
if (run.status !== "COMPLETED") {
  throw new Error(`Run ${run.runId}: ${JSON.stringify(run.error ?? run.status)}`);
}
console.log(run.output); // Output shape depends on the endpoint.
// READY/RUNNING price is provisional. Use the terminal receipt for reconciliation.
```

## Pricing

$0 per call

Read the final customer charge from price.amount.value after the run finishes. A RUNNING response is not the final bill.

## Questions

### What does Fetch pages do?

Takes up to ten URLs in one request and returns readable page content as markdown, HTML, or structured JSON, along with page metadata such as title, description, language, author, and publication date. Extraction can be scoped with CSS selectors that either restrict it to part of a page or strip unwanted elements, and a plain-language statement of what the caller is after steers what gets prioritised. Results and per-URL errors come back separately, so one bad URL does not fail the batch.

### How much does Fetch pages cost?

$0 per call, using your plan when signed in. The customer charge is returned as price.amount.value with its currency; READY/RUNNING values are provisional.

### What does Fetch pages take?

urls is required; the rest of the fields are optional. Send them in the input object of a POST to /v1/run.

### What are the limits on Fetch pages?

The catalog lists an execution timeout of 5 min. A request can return READY or RUNNING before completion; poll GET /v1/runs/{runId} with the same API key until it reaches a terminal status. An HTTP timeout does not prove the generation stopped.

### How do I retry safely and check the final charge?

Save a unique Idempotency-Key for each job. After a network error or HTML timeout, resend the identical POST body with the same key and API key to recover the existing run, then poll its runId. A new key can start another billable generation. An HTTP error or timeout does not establish the run outcome or final charge. Pending runs can still settle; check the terminal price.amount and billing state, and retain the runId for reconciliation. Do not automatically create a new job after a terminal failure.