You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have been asked by an auditor to reproduce a figure from a report dated eight months earlier and discovered that the table it came from had been rewritten twice, the code had no tag, and the parameters lived in a notebook someone had since deleted. You reproduced it, eventually, from raw files and git archaeology, and it took eleven days. You treat every series as evidence with a chain of custody now, and reproducibility is the part of that chain you can test before anyone asks.

## Key Points

1. Canonicalise: fixed column order, rows sorted by the natural key, timestamps as UTC epoch at contract precision, decimals as strings at contract scale, nulls as a fixed token.
2. Hash each partition with SHA-256 over the canonical rows.
3. Hash the dataset version as SHA-256 over the sorted list of `partition_key:partition_hash` pairs. This is a two-level Merkle structure; a changed partition changes exactly one leaf and the root.
- dataset: readings
- dataset: calibration
2. **Restore the code**: `git checkout <commit>`; confirm `dirty: false`. Rebuild the environment from the lockfile hash, not from "latest."
4. **Pin the environment's time**: set the runner timezone and tz database to the manifest values. Timezone rule changes in the intervening six months can alter local-time aggregations.
7. **Write the reproduction record**: who, when, what matched, what tolerance was needed, what had to be rebuilt. Attach it to the original artifact's lineage as a new activity.
- Every dataset version has a content hash computed over canonical rows, not file bytes.
- Partitions are write-once; corrections create new paths and new manifests.
- Table-format retention windows are known, and anything a report depends on is tagged or cloned beyond them.
- Every artifact has a manifest naming data refs with hashes, code commit, environment lockfile, parameters, seed and watermark.

## Quick Example

```text
s3://ts-lake/readings/series_id=P12.pressure/date=2026-03-14/run_id=3f1c.../part-0.parquet
s3://ts-lake/readings/series_id=P12.pressure/date=2026-03-14/run_id=7b09.../part-0.parquet
```

```bash
lakectl branch create lakefs://ts-lake/backtest-2026-04 --source lakefs://ts-lake/main
# write to s3://ts-lake/backtest-2026-04/... via the S3 gateway
lakectl commit lakefs://ts-lake/backtest-2026-04 -m "readings through 2026-03-31, calibration CAL-2026-031"
lakectl tag create lakefs://ts-lake/report-2026-04-q1 lakefs://ts-lake/backtest-2026-04
```

Dataset Versioning and Reproducibility

You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have been asked by an auditor to reproduce a figure from a report dated eight months earlier and discovered that the table it came from had been rewritten twice, the code had no tag, and the parameters lived in a notebook someone had since deleted. You reproduced it, eventually, from raw files and git archaeology, and it took eleven days. You treat every series as evidence with a chain of custody now, and reproducibility is the part of that chain you can test before anyone asks.

Core Principles

A number is reproducible when three things are pinned: the data version, the code version and the parameters. Miss any one and you have a number that was true once. Environment (library versions, locale, timezone of the runner) is a fourth that bites less often but harder.

Immutability is cheaper than reconstruction. Storage is cheap; eleven days of archaeology is not. Write once, version forward, never rewrite a partition that anything has read.

Version the dataset, not the file. A file has a name; a dataset version has a content identity that is the same wherever the bytes live and different whenever a value changes.

Time travel has an expiry date. Every table format that offers "as of version N" also has a retention job that deletes the files version N needs. Reproducibility beyond the retention window requires a tag, a snapshot copy, or both.

The manifest is the report's birth certificate. A report without a manifest naming its inputs by hash and its code by commit is an opinion with charts.

Techniques

Content hashing

Hash the logical content, not the bytes of the file: Parquet files embed writer metadata and row-group boundaries that change without any value changing.

1. Canonicalise: fixed column order, rows sorted by the natural key, timestamps as UTC epoch at contract precision, decimals as strings at contract scale, nulls as a fixed token.
2. Hash each partition with SHA-256 over the canonical rows.
3. Hash the dataset version as SHA-256 over the sorted list of partition_key:partition_hash pairs. This is a two-level Merkle structure; a changed partition changes exactly one leaf and the root.

import hashlib, pandas as pd

def partition_hash(df: pd.DataFrame, key: list[str], scale: dict[str, int]) -> str:
    canon = df.sort_values(key).reset_index(drop=True)
    for col, s in scale.items():
        canon[col] = canon[col].map(lambda v: "NULL" if pd.isna(v) else f"{v:.{s}f}")
    for col in canon.select_dtypes("datetimetz").columns:
        canon[col] = canon[col].dt.tz_convert("UTC").astype("int64")
    payload = canon.to_csv(index=False, lineterminator="\n").encode()
    return hashlib.sha256(payload).hexdigest()

def dataset_hash(partition_hashes: dict[str, str]) -> str:
    lines = "\n".join(f"{k}:{v}" for k, v in sorted(partition_hashes.items()))
    return hashlib.sha256(lines.encode()).hexdigest()

Store partition_hash in the partition's manifest and dataset_hash in the version record. Recompute on read when the stakes justify it.

Immutable partitions

Lay out storage so that a write never touches an existing path:

s3://ts-lake/readings/series_id=P12.pressure/date=2026-03-14/run_id=3f1c.../part-0.parquet
s3://ts-lake/readings/series_id=P12.pressure/date=2026-03-14/run_id=7b09.../part-0.parquet

A manifest per dataset version names which run_id path is current for each partition. Readers go through the manifest, never through directory listing. Corrections write a new run_id path and a new manifest; the old path stays until a retention policy older than the longest reproducibility commitment removes it.

DVC

DVC versions data files alongside git by storing a small pointer file with a content hash and pushing the bytes to a remote.

dvc init
dvc remote add -d store s3://ts-lake/dvc
dvc add data/readings_2026q1.parquet        # writes data/readings_2026q1.parquet.dvc
git add data/readings_2026q1.parquet.dvc .gitignore
git commit -m "readings 2026Q1 as delivered"
dvc push
git tag -a data/readings-2026q1-v1 -m "as delivered 2026-04-02"

To reproduce: git checkout data/readings-2026q1-v1 && dvc checkout. dvc.yaml stages declare deps, params and outs; dvc repro reruns only what changed, and dvc.lock records the hashes of every input and output of the last run, which is a manifest you get for free. DVC is a good fit for research datasets and model artifacts; it is a poor fit for high-frequency appends because each version is a whole-file identity.

lakeFS

lakeFS puts git semantics on an object store: branches, commits, tags and merges over S3-compatible paths, with the objects stored once and referenced by many versions.

lakectl branch create lakefs://ts-lake/backtest-2026-04 --source lakefs://ts-lake/main
# write to s3://ts-lake/backtest-2026-04/... via the S3 gateway
lakectl commit lakefs://ts-lake/backtest-2026-04 -m "readings through 2026-03-31, calibration CAL-2026-031"
lakectl tag create lakefs://ts-lake/report-2026-04-q1 lakefs://ts-lake/backtest-2026-04

A tag is a permanent pointer to a commit; a report pins lakefs://ts-lake/report-2026-04-q1 and every path under it resolves the same way forever, subject only to garbage collection rules you control. Pre-commit hooks can run validation so that a commit to main is impossible with failing checks.

Delta Lake time travel

DESCRIBE HISTORY readings;
SELECT * FROM readings VERSION AS OF 412;
SELECT * FROM readings TIMESTAMP AS OF '2026-03-15T03:00:00Z';
RESTORE TABLE readings TO VERSION AS OF 412;    -- creates a new version, does not delete history

The trap: VACUUM removes data files no longer referenced by the current version once they are older than the retention threshold (seven days by default), and the transaction log itself is trimmed after delta.logRetentionDuration (thirty days by default). After either, VERSION AS OF fails for old versions. For reproducibility beyond the window, CLONE the table version to a pinned location (a deep clone copies the files; a shallow clone shares them and dies with the same vacuum) or export a snapshot.

Apache Iceberg snapshots, tags and branches

Every Iceberg commit creates a snapshot with an id. Snapshots can be named and retained explicitly:

SELECT * FROM lake.readings VERSION AS OF 8123456789012345678;
SELECT * FROM lake.readings TIMESTAMP AS OF '2026-03-15 03:00:00';
ALTER TABLE lake.readings CREATE TAG report_2026_q1 AS OF VERSION 8123456789012345678 RETAIN 1825 DAYS;
ALTER TABLE lake.readings CREATE BRANCH audit_replay AS OF VERSION 8123456789012345678;
SELECT * FROM lake.readings VERSION AS OF 'report_2026_q1';

This is the Spark SQL spelling; Trino and Flink expose the same tags and branches with their own syntax, so check the engine's reference before pasting. expire_snapshots will not remove a snapshot a tag references while its RETAIN period holds. That makes an Iceberg tag with a long retention the closest thing to a git tag for tables, and it is the mechanism to prefer when the platform supports it. Inspect lake.readings.snapshots and lake.readings.refs metadata tables to see what is pinned.

The manifest

Every produced artifact (report, backtest result, model, regulatory figure) writes a manifest next to it:

artifact: risk_report_2026-03-31.pdf
produced_at: 2026-04-02T07:14:22Z
produced_by: svc-reporting on behalf of risk-team
code:
  repo: git@example:ts/reporting.git
  commit: a91e2d4f1c0b7e3d
  dirty: false
environment:
  lockfile_sha256: 6c1f...
  runner_timezone: UTC
  tz_database: 2026a
inputs:
  - dataset: readings
    ref: iceberg tag report_2026_q1
    snapshot_id: 8123456789012345678
    dataset_hash: 3d9a...
  - dataset: calibration
    ref: lakefs://ts-lake/report-2026-04-q1/calibration/
    commit: 5e77...
parameters:
  window_days: 90
  confidence: 0.99
  seed: 20260331
watermark: 2026-04-01T00:00:00Z

The manifest is itself hashed and its hash is printed in the artifact's footer. That is the only link between a PDF on someone's desk and the bytes that produced it.

Procedure: Reproduce a Six-Month-Old Number

1. Locate the manifest from the artifact's footer hash. If there is no manifest, stop and record that the figure is not reproducible by construction; do not "approximately" reproduce it and call it reproduced.
2. Restore the code: git checkout <commit>; confirm dirty: false. Rebuild the environment from the lockfile hash, not from "latest."
3. Resolve every input ref: Iceberg tag, lakeFS tag, DVC tag or manifest path. For each, recompute dataset_hash and compare with the manifest. A mismatch means retention deleted files or someone rewrote a partition; record which, and go to the raw-source recovery below.
4. Pin the environment's time: set the runner timezone and tz database to the manifest values. Timezone rule changes in the intervening six months can alter local-time aggregations.
5. Run with the manifest's parameters and seed. Compare the output to the artifact at the precision the contract specifies. Bit-for-bit equality is the goal for deterministic pipelines; for floating-point reductions on parallel engines, equality within contract precision is the realistic target, and the tolerance must be stated.
6. If the hash mismatched in step 3, rebuild the input from raw: raw landing files are immutable and retained longest; replay the ingest and transform at the manifest's transform_version up to the manifest's watermark, recompute the hash, and compare. If the raw files are gone, the number is unreproducible; say so.
7. Write the reproduction record: who, when, what matched, what tolerance was needed, what had to be rebuilt. Attach it to the original artifact's lineage as a new activity.

Checklist

• Every dataset version has a content hash computed over canonical rows, not file bytes.
• Partitions are write-once; corrections create new paths and new manifests.
• Table-format retention windows are known, and anything a report depends on is tagged or cloned beyond them.
• Every artifact has a manifest naming data refs with hashes, code commit, environment lockfile, parameters, seed and watermark.
• Raw landing files are retained at least as long as the longest reproducibility commitment.
• Reproduction is rehearsed: at least one old artifact per quarter is rebuilt from its manifest and the result recorded.
• Tags are named for the artifact that depends on them, so nobody deletes report_2026_q1 thinking it is a scratch branch.

Common Mistakes

• Pinning "the table" instead of a version of the table.
• Trusting TIMESTAMP AS OF to work forever and finding out at audit time that vacuum ran.
• Hashing Parquet bytes and getting a new hash on every rewrite of identical data.
• Storing parameters in the notebook that ran the job.
• A manifest that names inputs by path with no hash, so a silent rewrite passes.
• Rebuilding the environment from current package versions and attributing the difference to "floating point."
• Deleting raw files on a retention schedule shorter than the audit horizon because they were "already processed."

Limits

This skill makes a result reproducible; it does not make it correct. Point-in-time semantics (what was known when) are a separate concern handled by the bitemporal skill, though a versioned dataset is the substrate for both. The techniques scale poorly for datasets that must be mutable at high frequency with sub-second reads; there, version the immutable raw layer and treat the serving layer as a cache. For a one-off exploratory analysis, a git commit and a hashed copy of the input file are enough; the full manifest discipline is for anything that leaves the building.