The client-proof workflow → Sampling methodology
Sampling vs Full Checks: The Index Audit Methodology
Checking all 80,000 URLs every week is a budget decision pretending to be a diligence decision. Checking 40 random ones and calling it monitoring is the opposite mistake. The working answer sits in between, and it has rules.
The core idea: templates fail together
Pages built from one template share their indexing fate — same internal linking pattern, same content skeleton, same directives. When deindexation happens at scale, it happens to a template, not to random URLs. A deploy adds noindex to pagination; a variant rollout thins out product descriptions; a category redesign orphans a level. Which means a modest sample per template detects the failure almost as fast as a full check would.
The sampling rules
- Stratify by template, always. A mixed sample hides exactly the signal you're paying to see. Keep category, product, pagination, and editorial samples separate.
- Pick a fixed sample and keep it stable. Re-checking the same URLs each cycle turns noise into a clean trend line. Add a small rotating slice for coverage of fresh pages.
- Size by consequence, not by formula. A few hundred URLs per money-template is a workable band for weekly cycles; niche templates can run leaner. The point is drop detection, not a confidence interval for a paper.
- Alert on deltas, not on absolutes. A pagination sample that always sits low is a known condition. The same sample dropping by half in one cycle is an incident.
When only a full check will do
- Baselines. The first audit of a new client runs the whole scope — samples need a reference point.
- Milestones. Migration, replatforming, template rollout, domain consolidation. Google's own sitemap documentation is the reminder that discovery and indexing are separate steps — after big changes you verify both, URL by URL.
- Disputes. Vendor and client arguments are settled with complete files, never with samples.
Batch capacity makes the full runs painless — up to 100,000 links per task through the bulk index checker, with a per-URL file back. The economics of the audit come from spending that capacity where it changes decisions, and sampling everywhere else. For collecting the URL lists themselves, the free sitemap extractor covers the sitemap side.