Reading tools and contents
4 min
Ontology & Knowledge Graph Engineering

Chapter 7 of 10

Entity resolution with reversible identity

Identity engineering

About 4 minutes · includes examples, an exercise, and references

Chapter at a glance

  • •Preserve source records and originals before normalization or clustering.
  • •Measure blocking recall separately from match-model quality.
  • •Use local, evidence-bearing resolution relations instead of probabilistic owl:sameAs.

Entity resolution decides which source records refer to the same real-world entity. It is not ordinary classification because one decision changes the graph’s topology: a bad merge combines histories, permissions, relationships, and inferred consequences, while a missed merge fragments evidence. The safest mental model is a governed evidence process that produces revisable links between source records and canonical entities.

Begin with source-record identity. Preserve every upstream key under its source namespace and mint an immutable record IRI. Normalize fields into comparison features without discarding originals. Case folding, Unicode normalization, address parsing, phone formatting, and legal-suffix handling improve comparison, but normalized equality is evidence rather than proof. The original value, normalization code version, locale, and parsing warnings must remain available.

Blocking generates plausible candidate pairs without comparing every record to every other record. Blocking keys might use tax-identifier fragments, phonetic surname keys, postal regions, email domains, or learned approximate-neighbor indexes. Use several complementary blocks because a single corrupt field can hide a true pair. Measure blocking recall against reviewed pairs before tuning the match model; a perfect classifier cannot recover a pair it never sees.

Fellegi and Sunter formalized linkage through comparison outcomes. For a comparison vector gamma, each feature contributes evidence according to how likely that outcome is among matches versus non-matches. The log likelihood ratio sums those contributions. Two thresholds divide candidates into link, possible-link review, and non-link regions. Modern systems may use supervised models, but the discipline remains valuable: calibrate probabilities, inspect feature contributions, and reserve an abstention region.

Identifiers require type-specific rules. A verified tax identifier may strongly support an organization match, while a shared telephone number may indicate a household, call center, or stale contact. Missingness is informative only if its generation process is understood. Negative evidence matters: incompatible dates, mutually exclusive jurisdictions, or concurrent appearances in different roles can veto an otherwise high string score. Encode hard constraints separately from statistical evidence so reviewers can see why a pair was blocked.

Pairwise decisions must produce coherent clusters. If A matches B and B matches C, transitive closure would merge A and C even when their evidence conflicts. Use constrained clustering that checks cluster-level invariants, or require a representative-based decision. Store membership links with model version, score, evidence features, reviewer, and time. Do not emit owl:sameAs from a probabilistic match. It asserts semantic identity with substitutive consequences. A local relation such as resolvesTo or possibleMatch can carry the intended operational meaning.

Canonical entities need survivorship rules. Decide which source supplies a preferred label, how conflicting attributes remain visible, and when a value becomes current. Canonicalization must not erase record lineage. A golden record is a view over assertions, not necessarily a new unquestionable truth. Where source authority varies by attribute, select values with field-specific policies.

Make merges reversible. A merge operation should create a versioned cluster change, not rewrite every edge irreversibly. Keep source records as nodes and route canonical views through membership. A split can then assign records to new canonical nodes and recompute affected derived edges. Track blast radius: reports, embeddings, permissions, cached answers, and downstream exports created under the old cluster version may need invalidation.

Evaluation needs pair and cluster metrics. Pair precision and recall expose false and missed links. B-cubed or other cluster-aware measures reveal over-merged and fragmented entities. Slice results by source pair, language, geography, entity size, missingness, and protected or risk-relevant groups. Review queues should sample near thresholds, high-impact merges, and random automatic decisions, not merely easy confirmed pairs.

Resolution is never “done.” New sources, renamed organizations, recycled identifiers, and model changes alter evidence. Monitor candidate rates, score drift, merge reversals, review agreement, and downstream incidents. Version the full pipeline so any canonical view can be reproduced. Identity quality is the foundation on which every graph traversal and agent action depends.

Key points

  • Preserve source records and originals before normalization or clustering.
  • Measure blocking recall separately from match-model quality.
  • Use local, evidence-bearing resolution relations instead of probabilistic owl:sameAs.
  • Represent merges as reversible cluster versions with known invalidation scope.

A transparent linkage score

Read the expected behavior in the surrounding walkthrough, then copy and run this reference implementation.

A transparent linkage scorepython
from math import log

def linkage_weight(comparisons, rates):
    total = 0.0
    evidence = []
    for feature, outcome in comparisons.items():
        m = rates[feature][outcome]["match"]
        u = rates[feature][outcome]["non_match"]
        weight = log(m / u)
        total += weight
        evidence.append((feature, outcome, weight))
    return total, sorted(evidence, key=lambda row: abs(row[2]), reverse=True)

# Thresholds and hard-veto rules are calibrated on reviewed source pairs.

Worked examples

Toy

Three similar authors

Records A and B share an email; B and C share a name, but A and C have incompatible birth years.

The cluster constraint prevents transitive over-merging. A and B can resolve together while C remains separate or in review, with the birth-year veto visible.

  • Pair evidence
  • Cluster constraint
  • Abstention decision

System

Supplier master split

An enterprise discovers that one canonical supplier combines two legal organizations.

Reassign immutable source records to two new cluster versions, recompute derived ownership and risk relations, and invalidate answers and exports that used the old cluster.

  • Reversible membership
  • Affected artifacts
  • Audit trail

Exercise

Specify an entity-resolution decision system

Design resolution for two sources that contain people or organizations with incomplete identifiers.

  1. Define source identities and normalization lineage.
  2. Propose at least three blocking strategies.
  3. Separate evidence features, vetoes, and thresholds.
  4. Design review and reversible split operations.

Success criteria

  • Blocking recall has a test set.
  • Automatic links have calibrated evidence.
  • Cluster invariants prevent known over-merges.
  • Canonical output retains every source record and decision version.

Reflect: Which false merge would have the highest downstream cost, and how should that change the threshold?

References and further reading