Entity Resolution and Record Linkage
Link duplicate records with explainable matching, candidate blocking, measured thresholds and reversible merge decisions.
Use this skill when records from one or more datasets may refer to the same real-world entity. Define the entity type, permitted use, source permissions, cost of a false merge and cost of a missed match. Do not equate a matching name with an identity. Choose review thresholds according to consequences rather than a convenient round score. ## Key Points - [Splink record linkage](https://moj-analytical-services.github.io/splink/)
skilldb get data-engineering-skills/entity-resolution-record-linkageFull skill: 35 linesEntity Resolution and Record Linkage
Use this skill when records from one or more datasets may refer to the same real-world entity. Define the entity type, permitted use, source permissions, cost of a false merge and cost of a missed match. Do not equate a matching name with an identity. Choose review thresholds according to consequences rather than a convenient round score.
Prepare comparable evidence
Keep the original records and source IDs immutable. Create derived comparison fields for normalization, preserving original spelling and values. Treat absent fields as missing evidence, not automatically as agreement. Distinguish a standardized date from an uncertain or partial date. Document transliteration and address normalization because they can erase meaningful differences.
Examine field completeness, common values, duplicate IDs and time coverage. Names and addresses may change; an apparent conflict can reflect chronology. Do not collect additional sensitive attributes merely to improve an unspecified matching score.
Build candidates and comparisons
Use blocking rules to avoid comparing every possible pair. Combine complementary routes, such as normalized contact information and separate name-plus-location candidates. Measure blocking recall against labeled examples before tuning a classifier: a true pair excluded during blocking cannot be recovered by a better score.
Start with transparent deterministic rules when reliable identifiers exist. For probabilistic linkage, define comparisons, estimate model parameters on representative data and inspect common-value effects. A high pair score is not automatically a calibrated probability. Keep training, threshold selection and final evaluation separate.
Create reviewed positive pairs, hard negatives and uncertain cases. Report precision and recall with sample sizes, subgroup limitations and sampling method. Do not evaluate only the easy exact matches.
Review, cluster and release
Choose automatic-match, manual-review and non-match ranges. Show reviewers the evidence contributing to each decision without exposing unnecessary personal data. Clustering adds a separate risk: A matching B and B matching C does not prove A and C are compatible. Check clusters for contradictory identifiers, implausible size and temporal conflicts.
Write a reversible crosswalk from source records to canonical entity IDs. Preserve decision version, evidence, reviewer and superseded links. Keep destructive operational merges separate from analytical linkage and require the relevant authority before applying them.
Deliver linkage rules, evaluation data, error analysis, a review queue and rollback mapping. Re-evaluate after a new source, normalization change or population shift; yesterday's threshold is not guaranteed to retain today's error rate.
References
Install this skill directly: skilldb add data-engineering-skills
Related Skills
Real-Time Analytics
Triggers when users need help with real-time analytics, real-time dashboards, OLAP engines, ClickHouse, Apache Druid, Apache Pinot, StarRocks, pre-aggregation vs on-the-fly computation, materialized views for real-time, approximate query processing, HyperLogLog, Count-Min Sketch, lambda architecture, or kappa architecture. Activate for questions about building real-time analytical systems, choosing OLAP databases, sub-second query performance, or real-time aggregation strategies.
Stream Processing
Triggers when users need help with stream processing, Apache Kafka architecture, topics, partitions, consumer groups, exactly-once processing, Kafka Streams, Apache Flink, Spark Streaming, event sourcing, CQRS, windowing strategies, watermarks, late data handling, stream-table duality, or real-time aggregation. Activate for questions about building streaming pipelines, choosing streaming frameworks, event-driven architectures, or real-time data processing.
Analytics Engineering
Triggers when users need help with analytics engineering, dbt, dbt models, tests, macros, packages, incremental models, semantic layer, metrics definitions, documentation-as-code, CI/CD for data transformations, data mesh, self-serve analytics, or reverse ETL. Activate for questions about building dbt projects, designing transformation layers, implementing metrics layers, or enabling self-service data access.
Batch Processing
Triggers when users need help with Apache Spark, batch data processing, RDDs, DataFrames, Catalyst optimizer, partitioning strategies, shuffle optimization, memory tuning, Spark SQL, data skew handling, broadcast joins, adaptive query execution, PySpark best practices, or cluster sizing. Activate for questions about Spark job optimization, batch ETL performance, distributed data processing, or large-scale data transformation.
Data Governance
Triggers when users need help with data governance, data cataloging, DataHub, Amundsen, OpenMetadata, data lineage tracking, access control policies, PII detection, data classification, retention policies, regulatory compliance (GDPR, CCPA, HIPAA), data stewardship, or metadata management. Activate for questions about organizing data assets, tracking data lineage, managing data access, handling sensitive data, or meeting regulatory requirements.
Data Integration
Triggers when users need help with data integration, Change Data Capture (CDC), Debezium, Fivetran, Airbyte, API data extraction, file-based ingestion, CSV, JSON, Parquet, Avro, webhook processing, data replication, schema registry, Confluent Schema Registry, AWS Glue Schema Registry, or format selection. Activate for questions about ingesting data from external sources, choosing ingestion tools, implementing CDC, or managing data formats and schemas.