Skip to main content
Technology & EngineeringData Engineering35 lines

Entity Resolution and Record Linkage

Link duplicate records with explainable matching, candidate blocking, measured thresholds and reversible merge decisions.

Quick Summary7 lines
Use this skill when records from one or more datasets may refer to the same real-world entity. Define the entity type, permitted use, source permissions, cost of a false merge and cost of a missed match. Do not equate a matching name with an identity. Choose review thresholds according to consequences rather than a convenient round score.

## Key Points

- [Splink record linkage](https://moj-analytical-services.github.io/splink/)
skilldb get data-engineering-skills/entity-resolution-record-linkageFull skill: 35 lines
Paste into your CLAUDE.md or agent config

Entity Resolution and Record Linkage

Use this skill when records from one or more datasets may refer to the same real-world entity. Define the entity type, permitted use, source permissions, cost of a false merge and cost of a missed match. Do not equate a matching name with an identity. Choose review thresholds according to consequences rather than a convenient round score.

Prepare comparable evidence

Keep the original records and source IDs immutable. Create derived comparison fields for normalization, preserving original spelling and values. Treat absent fields as missing evidence, not automatically as agreement. Distinguish a standardized date from an uncertain or partial date. Document transliteration and address normalization because they can erase meaningful differences.

Examine field completeness, common values, duplicate IDs and time coverage. Names and addresses may change; an apparent conflict can reflect chronology. Do not collect additional sensitive attributes merely to improve an unspecified matching score.

Build candidates and comparisons

Use blocking rules to avoid comparing every possible pair. Combine complementary routes, such as normalized contact information and separate name-plus-location candidates. Measure blocking recall against labeled examples before tuning a classifier: a true pair excluded during blocking cannot be recovered by a better score.

Start with transparent deterministic rules when reliable identifiers exist. For probabilistic linkage, define comparisons, estimate model parameters on representative data and inspect common-value effects. A high pair score is not automatically a calibrated probability. Keep training, threshold selection and final evaluation separate.

Create reviewed positive pairs, hard negatives and uncertain cases. Report precision and recall with sample sizes, subgroup limitations and sampling method. Do not evaluate only the easy exact matches.

Review, cluster and release

Choose automatic-match, manual-review and non-match ranges. Show reviewers the evidence contributing to each decision without exposing unnecessary personal data. Clustering adds a separate risk: A matching B and B matching C does not prove A and C are compatible. Check clusters for contradictory identifiers, implausible size and temporal conflicts.

Write a reversible crosswalk from source records to canonical entity IDs. Preserve decision version, evidence, reviewer and superseded links. Keep destructive operational merges separate from analytical linkage and require the relevant authority before applying them.

Deliver linkage rules, evaluation data, error analysis, a review queue and rollback mapping. Re-evaluate after a new source, normalization change or population shift; yesterday's threshold is not guaranteed to retain today's error rate.

References

Install this skill directly: skilldb add data-engineering-skills

Get CLI access →

Related Skills

Real-Time Analytics

Triggers when users need help with real-time analytics, real-time dashboards, OLAP engines, ClickHouse, Apache Druid, Apache Pinot, StarRocks, pre-aggregation vs on-the-fly computation, materialized views for real-time, approximate query processing, HyperLogLog, Count-Min Sketch, lambda architecture, or kappa architecture. Activate for questions about building real-time analytical systems, choosing OLAP databases, sub-second query performance, or real-time aggregation strategies.

Data Engineering•153L

Stream Processing

Triggers when users need help with stream processing, Apache Kafka architecture, topics, partitions, consumer groups, exactly-once processing, Kafka Streams, Apache Flink, Spark Streaming, event sourcing, CQRS, windowing strategies, watermarks, late data handling, stream-table duality, or real-time aggregation. Activate for questions about building streaming pipelines, choosing streaming frameworks, event-driven architectures, or real-time data processing.

Data Engineering•126L

Analytics Engineering

Triggers when users need help with analytics engineering, dbt, dbt models, tests, macros, packages, incremental models, semantic layer, metrics definitions, documentation-as-code, CI/CD for data transformations, data mesh, self-serve analytics, or reverse ETL. Activate for questions about building dbt projects, designing transformation layers, implementing metrics layers, or enabling self-service data access.

Data Engineering•130L

Batch Processing

Triggers when users need help with Apache Spark, batch data processing, RDDs, DataFrames, Catalyst optimizer, partitioning strategies, shuffle optimization, memory tuning, Spark SQL, data skew handling, broadcast joins, adaptive query execution, PySpark best practices, or cluster sizing. Activate for questions about Spark job optimization, batch ETL performance, distributed data processing, or large-scale data transformation.

Data Engineering•123L

Data Governance

Triggers when users need help with data governance, data cataloging, DataHub, Amundsen, OpenMetadata, data lineage tracking, access control policies, PII detection, data classification, retention policies, regulatory compliance (GDPR, CCPA, HIPAA), data stewardship, or metadata management. Activate for questions about organizing data assets, tracking data lineage, managing data access, handling sensitive data, or meeting regulatory requirements.

Data Engineering•155L

Data Integration

Triggers when users need help with data integration, Change Data Capture (CDC), Debezium, Fivetran, Airbyte, API data extraction, file-based ingestion, CSV, JSON, Parquet, Avro, webhook processing, data replication, schema registry, Confluent Schema Registry, AWS Glue Schema Registry, or format selection. Activate for questions about ingesting data from external sources, choosing ingestion tools, implementing CDC, or managing data formats and schemas.

Data Engineering•143L