Web crawl API

Breadth-first crawl from a seed URL that reads robots.txt and reports each page it visited.

Ask your agent

“Call web/crawl with url https://example.com and summarize what comes back.”

Works in any MCP client connected to https://data.openkrill.app/mcp. Install in one line.

Path: web/crawl
MCP tool: call_endpoint
Credits: 5
Cache: 10 min

Parameters

urlstring, required	Seed URL
limitinteger, optional	1-25 pages, default 10
depthinteger, optional	1-3 link hops, default 2
same_hostboolean, optional	Stay on the seed host, default true

Response

data.resources[] uses the same keys as every other platform: author, engagement, metadata, with Web-only fields under extras. A failed upstream call is never billed.

More Web tools

MCP tool call

{
  "name": "call_endpoint",
  "arguments": {
    "path": "web/crawl",
    "params": {
      "url": "https://example.com"
    }
  }
}

curl

curl "https://data.openkrill.app/v1/web/crawl?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: YOUR_API_KEY"

Response

{
  "success": true,
  "platform": "web",
  "endpoint": "web/crawl",
  "data": {
    "resources": [
      { "id": …, "title": …, "text": …, "url": …,
        "author": {…}, "engagement": {…},
        "metadata": {…}, "extras": {…} }
    ]
  },
  "credits_used": 5,
  "cached": false
}