# Space Crawls to Avoid False Outage Reports

> Pace requests to each host before treating burst-induced timeouts as source downtime.

Page: https://videotomd.com/b/text-local-diagnose-poor-web-page-crawls-bf6fbf443551/
Tool: Web research · Tags: web-crawling, request-pacing, diagnostics · 170 words, about 221 tokens
Pills: free · checked 2026-10-09 · Any AI · file tools · shell · web · on demand · frontier · research

## Recipe

- Kind: workflow
- Stage: not set
- Loads: on demand
- Works in: Any AI
- Needs: file tools, shell, web
- Bring: nothing
- Cost: free
- Model: frontier
- Tested in: not yet
- Checked: 2026-10-09
- For: research

## Put it here

Loads on demand: paste or attach it when the task comes up. To have it every session, put it where your tool reads instructions:

- Any AI: Coding agent: save as AGENTS.md in the project root (read natively by Codex, Cursor, Copilot, Gemini CLI, Kiro, Devin, Cline, Roo, OpenHands, OpenCode, Goose, Amp, Zed, Warp, Junie, Factory and Jules; Claude Code when no CLAUDE.md exists; Aider with --read). Chat: paste it first. Reload: new session. Check it loaded: ask it to repeat one rule from the file. Source: https://agents.md/ (checked 2026-10-12).

## Access

Membership: plans at https://videotomd.com/pricing/, or email then a plan at https://videotomd.com/b/text-local-diagnose-poor-web-page-crawls-bf6fbf443551/. A member's browser downloads the file there; an AI without that session needs the member to attach it.