Web robots API

Parsed robots.txt groups, sitemaps and crawl delay for a site.

Ask your agent

“Call web/robots with url https://example.com and summarize what comes back.”

Works in any MCP client connected to https://data.openkrill.app/mcp. Install in one line.

Path: web/robots
MCP tool: call_endpoint
Credits: 1
Cache: 2 min

Parameters

urlstring, required	Any URL on the site to inspect

Response

data.resources[] uses the same keys as every other platform: author, engagement, metadata, with Web-only fields under extras. A failed upstream call is never billed.

More Web tools

MCP tool call

{
  "name": "call_endpoint",
  "arguments": {
    "path": "web/robots",
    "params": {
      "url": "https://example.com"
    }
  }
}

curl

curl "https://data.openkrill.app/v1/web/robots?url=https%3A%2F%2Fexample.com" \
  -H "x-api-key: YOUR_API_KEY"

Response

{
  "success": true,
  "platform": "web",
  "endpoint": "web/robots",
  "data": {
    "resources": [
      { "id": …, "title": …, "text": …, "url": …,
        "author": {…}, "engagement": {…},
        "metadata": {…}, "extras": {…} }
    ]
  },
  "credits_used": 1,
  "cached": false
}