Endpoint liveness heuristic

What it measures

The share of mapper endpoints classified as live, against those classified as dead or unknown, under the dashboard's current definition: status IN (200,402) is live; every other value, including NULL, is dead or unknown.

How it is produced

The dashboard definition and mapper update code are inspected first. The canonical query then classifies every stored endpoint status. Supporting queries decompose the result by status and parsed host.

Dashboard definition inspection

sed -n '1,90p' generate-dashboard.sh

Canonical query

SELECT
  COUNT(*) AS endpoints,
  SUM(CASE WHEN status IN (200,402) THEN 1 ELSE 0 END) AS live_status_200_402,
  SUM(CASE WHEN status NOT IN (200,402) OR status IS NULL THEN 1 ELSE 0 END) AS dead_or_unknown,
  ROUND(100.0 * SUM(CASE WHEN status IN (200,402) THEN 1 ELSE 0 END) / COUNT(*), 6) AS live_pct,
  ROUND(100.0 * SUM(CASE WHEN status NOT IN (200,402) OR status IS NULL THEN 1 ELSE 0 END) / COUNT(*), 6) AS dead_or_unknown_pct,
  SUM(CASE WHEN payment_required_valid=1 THEN 1 ELSE 0 END) AS payment_valid
FROM endpoints;

Status decomposition

SELECT
  COALESCE(status,'NULL') AS status,
  COUNT(*) AS n,
  ROUND(100.0 * COUNT(*) / (SELECT COUNT(*) FROM endpoints), 6) AS pct
FROM endpoints
GROUP BY status
ORDER BY n DESC
LIMIT 20;

Host decomposition

WITH parsed AS (
  SELECT
    id,
    url,
    status,
    CASE WHEN instr(url,'://')>0 THEN substr(url,instr(url,'://')+3) ELSE url END AS rest
  FROM endpoints
),
hosts AS (
  SELECT
    lower(CASE WHEN instr(rest,'/')>0 THEN substr(rest,1,instr(rest,'/')-1) ELSE rest END) AS host,
    status
  FROM parsed
)
SELECT
  COALESCE(NULLIF(host,''),'(blank)') AS host,
  COUNT(*) AS endpoints,
  SUM(CASE WHEN status IN (200,402) THEN 1 ELSE 0 END) AS live,
  ROUND(100.0 * COUNT(*) / (SELECT COUNT(*) FROM endpoints), 6) AS pct_of_total,
  ROUND(100.0 * SUM(CASE WHEN status IN (200,402) THEN 1 ELSE 0 END) / (
    SELECT SUM(CASE WHEN status IN (200,402) THEN 1 ELSE 0 END) FROM endpoints
  ), 6) AS pct_of_live
FROM hosts
GROUP BY host
ORDER BY endpoints DESC
LIMIT 20;

Mapper status update inspection

sed -n '510,600p' mapper.py

Limits and assumptions

• The definition is a dashboard heuristic over the latest stored status, not a clean, recency-gated liveness model.
• The mapper preserves a previous 402 when a later probe times out, so a current timeout may still display as 402.
• NULL can mean unscanned or unknown rather than dead. The source note therefore marks this metric as unsuitable for a publication-grade live-versus-dead claim.
• The deterministic replacement uses the latest endpoint_history probe for each endpoint, requires a recency TTL, separates payment-required, general HTTP-live, dead, and unknown states, and requires repeated failures before calling an endpoint dead.

How to refute this

1. Re-run the canonical SQL and get different live/dead counts.
2. Recompute from the latest endpoint_history rows and show materially different liveness.
3. Freshly probe a sample of NULL or preserved-402 endpoints and show they are not accurately represented by the dashboard heuristic.

Last validated

2026-06-08