Heuristic Research Systems
Methodology
Ordinary in its mechanics, unusual in one respect: it refuses to resolve disagreement between sources.
- 01
Define the question before pointing anything at it
The criteria are decomposed and written down first. Each with a definition and a scale, and the resulting schema is versioned. Research that starts by gathering and decides afterwards what it was looking for cannot be repeated, and cannot be argued with, because there is no fixed thing to disagree about.
- 02
Route the work across agents, not through one model
A coordinated set of agents works the brief, each routed to the model that suits its task: retrieval, extraction, comparison, scoring. The division of labour matters more than the model choice: it is what makes the work reviewable step by step rather than as one opaque answer.
- 03
Keep evidence separated by origin
What a subject publishes about itself and what independent sources record about it are gathered and scored as distinct passes that never see each other. Combining them at collection time is irreversible, once merged, no reader can recover which claim came from where.
- 04
Publish the disagreement rather than resolving it
The passes are presented side by side. Convergence is a confidence signal; divergence is the finding. Averaging them produces something easier to read and strictly less informative than either input.
- 05
Gate the release on a person
Gathering runs on a cadence. Publication does not. A finding reaches a reader only when someone releases it, because a system that publishes its own conclusions has no point at which anyone is accountable for them.
Where the two passes disagree
An illustration, not an assessment. What a subject publishes about itself and what independent sources record are gathered and scored separately, and never see each other. Plotting them together is the first time they meet.
- Data handling42 apart
- Access control14 apart
- Incident response7 apart
- Third-party oversight48 apart
- Retention and deletion23 apart
The bar between them is the finding. Convergence is a confidence signal. Divergence is the result worth reporting, and it only survives because the passes were never merged.
What an average would leave you with
One dot per dimension, every gap above gone, and no way to recover which reading came from where. Easier to read, and strictly less informative than either input.
| Dimension | Claimed | Independently observed | Gap |
|---|---|---|---|
| Data handling | 92 | 50 | 42 |
| Access control | 61 | 75 | 14 |
| Incident response | 74 | 81 | 7 |
| Third-party oversight | 88 | 40 | 48 |
| Retention and deletion | 70 | 47 | 23 |
Why not just average them?
Because the average of a self-reported claim and an independent observation is neither of them. If a subject rates itself strongly on something the independent record does not support, that gap is the most useful thing the research can tell you, and a blended score is precisely the operation that removes it.
Keeping the passes apart costs readability. It buys the ability to answer the only question that matters when a decision is challenged: on what evidence.
Domain-agnostic
The method does not care what is being assessed. It needs a question worth defining precisely, and a decision someone will later have to justify.