Pimp My IDE / garage dispatchBack to the garage
13 Oct 2026 / site operations / crawler traffic

A request is load before it is audience.

Robots rules state a preference. Logs show behavior. Rate limits protect capacity. Keep those jobs on separate controls.

Measure pathsSignal policyEnforce capacity

The spike may not be readers.

Sammy Straus reported that a Hacker News post brought attention to a feed reader, then the site stayed slow after the discussion cooled. Endpoint counts showed that most requests did not fetch news articles. Aggregate server logs showed almost half of traffic came from one user that described itself as a website-monitoring and summarization service. Straus then added crawler rules, rate limits, and infrastructure changes.[1]

That is one operator's account, not a controlled traffic study. The useful move is smaller. Count requests by route and client signal before treating a traffic spike as audience growth. A homepage hit, an application data request, a feed fetch, and a scan for /wp-content/phpinfo.php do not describe the same work.

Traffic tells you that something knocked. Logs tell you which door moved.

Robots rules are a request.

RFC 9309 standardizes /robots.txt. It defines user-agent groups, allow and disallow rules, matching, caching, and error handling. The same standard says those rules are not access authorization. Its security section warns that listed paths become public and recommends real application-layer controls for restricted resources.[2]

Use robots rules to tell conforming crawlers where they may go. Do not use them to hide an admin route, stop a hostile client, or protect private content. A crawler can ignore the file. Any reader can open it.

Errors change crawler behavior.

Google's crawler documentation says most 4xx responses for robots.txt are treated as if no crawl restrictions exist. Status 429 is the exception. Google also says not to use 401 or 403 to control crawl rate.[3]

That distinction matters during an incident. Breaking the policy file can remove the preference you meant to preserve. Protect /robots.txt as a tiny, cacheable control file. Handle excess request volume at the request path or edge where the load occurs.

Throttle with a receipt.

HTTP 429 Too Many Requests tells a client that it exceeded a request limit. A Retry-After header can state when the client may try again. MDN notes that implementations vary. Limits may apply to the server, a resource, an IP address, or an authenticated client.[4]

Record the key, window, threshold, affected route, response, and recovery test. User-agent strings help classify polite crawlers, but they are supplied by the client. Do not treat a familiar name as authenticated identity.

Interactive makeover / raw counts replaced

Set the pressure line.

This teaching gauge combines two native range controls, a shared scale, a threshold verdict, and a copyable test receipt. It illustrates arithmetic. It does not read server traffic or recommend a production limit.

Count requestsCompare capacityChoose responseReplay recovery

Illustrative inputs

Set one observed request rate and one tested capacity. The gauge shows their ratio. Real limits also need latency, concurrency, route cost, cache state, and user impact.

120 req/min
0 req/min600 req/min
300 req/min
50 req/min600 req/min
Why this is better: two disconnected counters leave the decision in someone's head. This manifold puts both values on one gauge and prints the assumptions that still need proof.

Capacity teaching gauge

Teaching verdictHEADROOM SHOWN
CRAWLER_PRESSURE_TEST=ILLUSTRATIVE
OBSERVED_RATE=120_REQ_PER_MIN
TESTED_CAPACITY=300_REQ_PER_MIN
RATIO=40_PERCENT
POLICY=ROBOTS_RULES_ARE_NOT_AUTHORIZATION
LIMIT_KEY=[REQUIRED]
WINDOW=[REQUIRED]
RETRY_AFTER=[REQUIRED]
RECOVERY_REPLAY=[REQUIRED]

Amber begins at 70 percent. Red begins at 100 percent. Those thresholds belong to this teaching demo, not your server.

Sources read / 13 Oct 2026

Source log

Open the evidence boundary
  1. Sammy Straus, "I posted to HN, and then my website was flooded with AI BOT traffic", read October 13, 2026. This first-person report supplies the site's symptoms, endpoint logging, aggregate traffic finding, crawler mix, robots change, rate limits, and performance response. We did not inspect the site's private logs or reproduce the incident.
  2. RFC 9309, Robots Exclusion Protocol, published September 2022 and read October 13, 2026. The standard defines crawler groups, rules, matching, caching, errors, and the explicit boundary between crawler preferences and access authorization.
  3. Google Crawling Infrastructure, "How Google interprets the robots.txt specification", updated August 31, 2026 and read October 13, 2026. It documents file scope, status handling, caching, syntax, and Google's treatment of 429 versus other 4xx responses.
  4. MDN, "429 Too Many Requests", modified June 22, 2026 and read October 13, 2026. It defines the status, optional Retry-After header, and common limit scopes.
  5. Hacker News item 50073221, resolved through the Hacker News API and read October 13, 2026. It maps the current discussion to Straus's exact article URL. Claims about the incident come from the article.

Evidence boundary: the manifold is a browser-local teaching proxy. It calculates an observed-rate-to-capacity ratio and applies illustrative 70 percent and 100 percent thresholds. It does not inspect logs, identify bots, authenticate a client, configure a server, measure latency, or prove that a limit protects real users.