You are an expert in build artifact management and caching strategies for CI/CD pipelines.

## Key Points

- uses: actions/cache@v4
- run: npm ci
- name: Build and push
- name: Docker meta
- name: Build and push
- label: ":mag: Detect Changes"
- Use content-addressable cache keys (hash of lockfiles) so caches automatically invalidate when dependencies change.
- Set artifact retention policies; CI artifacts rarely need to persist beyond 7 days. Release artifacts go to a permanent registry.
- Tag container images with the immutable commit SHA as the primary identifier; use mutable tags (`latest`, `v1`) only as convenience aliases.
- Use multi-stage Docker builds to keep final images small and separate build-time from runtime dependencies.
- Leverage registry-based Docker layer caching (`cache-from/cache-to`) for faster image builds in CI.
- Sign and verify artifacts (cosign, GPG, npm provenance) to ensure supply chain integrity.

Artifact Management — CI/CD

You are an expert in build artifact management and caching strategies for CI/CD pipelines.

Overview

Artifact management covers the creation, storage, versioning, and distribution of build outputs across CI/CD pipelines. This includes compiled binaries, container images, packages, test results, and intermediate build products. Effective caching and artifact strategies directly impact pipeline speed, reliability, and cost. Key concerns are cache invalidation, artifact immutability, retention policies, and secure distribution.

Setup & Configuration

Artifact systems span multiple layers: CI platform artifacts (ephemeral, per-build), artifact registries (permanent, versioned), and caches (best-effort, performance optimization).

Common artifact types and their storage:

Artifact Type	Storage	Examples
Build outputs	CI artifacts / S3	Binaries, JARs, dist folders
Container images	Container registry	Docker images, OCI artifacts
Packages	Package registry	npm, PyPI, Maven, NuGet
Test results	CI platform	JUnit XML, coverage reports
Dependencies	CI cache	node_modules, .gradle, .pip

Core Patterns

Dependency Caching

GitHub Actions caching:

- uses: actions/cache@v4
  id: npm-cache
  with:
    path: |
      ~/.npm
      node_modules
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

- run: npm ci
  if: steps.npm-cache.outputs.cache-hit != 'true'

GitLab CI caching with fallback keys:

default:
  cache:
    - key:
        files:
          - Gemfile.lock
        prefix: ruby
      paths:
        - vendor/ruby
      policy: pull-push
    - key:
        files:
          - package-lock.json
        prefix: node
      paths:
        - node_modules
      policy: pull-push

build:
  stage: build
  cache:
    - key:
        files:
          - Gemfile.lock
        prefix: ruby
      paths:
        - vendor/ruby
      policy: pull  # Only read cache, don't update
  script:
    - bundle install --path vendor/ruby
    - bundle exec rake build

Docker Layer Caching

Multi-stage Dockerfile with cache optimization:

# syntax=docker/dockerfile:1.7
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --production

FROM node:20-alpine AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
RUN npm run build

FROM node:20-alpine AS runtime
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY package.json ./
USER node
EXPOSE 3000
CMD ["node", "dist/server.js"]

CI pipeline with registry-based Docker cache:

# GitHub Actions
- name: Build and push
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: |
      ghcr.io/org/app:${{ github.sha }}
      ghcr.io/org/app:latest
    cache-from: type=registry,ref=ghcr.io/org/app:buildcache
    cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max

Artifact Passing Between Jobs

GitHub Actions with upload/download:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
          retention-days: 1
          if-no-files-found: error

  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
      - run: ./deploy.sh staging

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
      - run: ./deploy.sh production

Container Image Tagging Strategy

# Build and tag with multiple strategies
- name: Docker meta
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: ghcr.io/org/app
    tags: |
      type=sha,prefix=
      type=ref,event=branch
      type=semver,pattern={{version}}
      type=semver,pattern={{major}}.{{minor}}
      type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}

- name: Build and push
  uses: docker/build-push-action@v5
  with:
    push: true
    tags: ${{ steps.meta.outputs.tags }}
    labels: ${{ steps.meta.outputs.labels }}

Package Publishing

npm package publishing with provenance:

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          registry-url: 'https://registry.npmjs.org'
      - run: npm ci
      - run: npm publish --provenance --access public
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Monorepo Artifact Isolation

# Buildkite: Build only changed packages
steps:
  - label: ":mag: Detect Changes"
    command: |
      CHANGED=$(git diff --name-only HEAD~1 | cut -d/ -f1-2 | sort -u)
      for pkg in packages/*; do
        if echo "$CHANGED" | grep -q "$pkg"; then
          cat <<YAML | buildkite-agent pipeline upload
        steps:
          - label: ":package: Build $(basename $pkg)"
            command: cd $pkg && npm ci && npm run build
            artifact_paths:
              - "$pkg/dist/**/*"
      YAML
        fi
      done

Artifact Integrity and Signing

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build

      - name: Generate checksums
        run: |
          cd dist
          sha256sum * > ../checksums.txt

      - name: Sign with cosign
        uses: sigstore/cosign-installer@v3
      - run: cosign sign-blob --yes checksums.txt --output-signature checksums.sig

      - uses: actions/upload-artifact@v4
        with:
          name: release
          path: |
            dist/
            checksums.txt
            checksums.sig

Core Philosophy

Artifacts are the connective tissue of a CI/CD pipeline. Every handoff between stages — build to test, test to deploy, deploy to production — should exchange immutable, versioned artifacts rather than rebuilding from source. When you treat artifacts as first-class citizens with their own lifecycle (creation, verification, storage, distribution, expiration), you eliminate an entire class of "works on my machine" and "what exactly did we deploy?" problems. The build should happen once and only once; everything downstream consumes that exact output.

Caching is a performance optimization, not a correctness mechanism. A pipeline must produce correct results with an empty cache; caching only makes it faster. This means cache keys must be deterministic (derived from lockfiles, not timestamps), cache misses must be graceful (fall back to a full build, not a failure), and cache corruption must be recoverable (version your cache keys so you can bust them). The moment a pipeline depends on cache contents for correctness, you have introduced a silent, intermittent failure mode that will surface at the worst possible time.

Supply chain integrity starts at the artifact layer. If you cannot answer "who built this artifact, from what source, and has it been tampered with?" then your deployment pipeline has a trust gap. Signing artifacts, generating provenance attestations, and verifying checksums before deployment are not optional hardening steps — they are the baseline for any pipeline that deploys to production. An unsigned artifact is an unsigned contract: it means nothing.

Anti-Patterns

• Rebuild-per-stage. Building the application separately in the test job and again in the deploy job means you are testing one binary and deploying a different one. Build once, then pass the artifact downstream via upload/download or workspace persistence so every stage operates on the identical output.

• Cache-as-source-of-truth. Designing a pipeline where a missing cache causes a build failure (e.g., skipping npm ci entirely when cache hits) means a cache eviction turns into a broken pipeline. Caches must be best-effort acceleration, never a prerequisite for correctness.

• Mutable production tags. Tagging container images only as latest or stable and deploying those tags to production means you cannot determine which commit is running, reproduce the exact image, or roll back to a known version. Always tag with the immutable commit SHA as the primary identifier.

• Unbounded artifact retention. Storing every build artifact forever fills storage, inflates costs, and makes it harder to find the artifacts that matter. Set explicit retention policies: ephemeral CI artifacts expire in days, release artifacts go to a permanent registry with defined lifecycle rules.

• Ignoring artifact size. Uploading multi-gigabyte artifacts through the CI platform's built-in artifact system instead of external storage (S3, GCS) creates slow pipelines, flaky uploads, and storage limit violations. Use the right storage tier for the artifact size and access pattern.

Best Practices

• Use content-addressable cache keys (hash of lockfiles) so caches automatically invalidate when dependencies change.
• Set artifact retention policies; CI artifacts rarely need to persist beyond 7 days. Release artifacts go to a permanent registry.
• Tag container images with the immutable commit SHA as the primary identifier; use mutable tags (latest, v1) only as convenience aliases.
• Use multi-stage Docker builds to keep final images small and separate build-time from runtime dependencies.
• Leverage registry-based Docker layer caching (cache-from/cache-to) for faster image builds in CI.
• Sign and verify artifacts (cosign, GPG, npm provenance) to ensure supply chain integrity.
• Use if-no-files-found: error when uploading artifacts to catch build failures that silently produce no output.
• Separate caches by OS, language version, and lockfile hash to prevent cross-contamination.
• Use policy: pull for jobs that consume but should not update a cache (e.g., deploy jobs).
• Prefer downloading artifacts from a prior job over rebuilding, ensuring you deploy exactly what was tested.

Common Pitfalls

• Caching node_modules directly instead of the npm/yarn cache directory means platform-specific binaries (e.g., esbuild) may not match the runner OS.
• Mutable Docker tags (latest) in production mean you cannot reproduce or roll back to a known image version.
• Uploading massive artifacts (>1GB) in CI slows pipelines and may hit storage limits; use external storage (S3, GCS) for large files.
• Cache stampede: when cache is cold, many parallel jobs all rebuild and try to save the same cache simultaneously.
• Not versioning cache keys (no v1- prefix) means you cannot force a cache bust when the cache is corrupted.
• GitLab CI caches are per-runner by default; distributed runners lead to cache misses unless using distributed cache (S3 backend).
• Docker COPY . . before RUN npm install invalidates the dependency layer on every code change. Copy lockfiles first.
• Artifact download actions pull all artifacts if no name is specified, which can be very slow with many jobs.