You are an expert in optimizing CI/CD pipelines for monorepos, including affected-based execution, caching strategies, parallel workflows, and incremental builds.

## Key Points

- uses: dorny/paths-filter@v3
- uses: actions/setup-node@v4
- uses: actions/cache@v4
- run: npx nx-cloud start-ci-run --distribute-on="5 linux-medium-js"
- run: npx nx affected -t build test lint --parallel=3
- run: npx nx-cloud stop-all-agents
1. **Use graph-based affected detection** — Path filters miss transitive dependencies. Use Turbo/Nx/Lerna's graph awareness for correct affected analysis.
2. **Enable remote caching** — Local CI caches are per-runner; remote caches are shared across all runs and dramatically reduce redundant work.
3. **Fetch full git history for affected** — Use `fetch-depth: 0` so affected commands can compare against the base branch.
4. **Cancel redundant runs** — Use `concurrency` groups to cancel in-progress CI when a new commit is pushed.
5. **Fail fast on cheap checks** — Run lint and type-check before tests and builds. A 30-second lint failure saves waiting for a 10-minute build.
6. **Shard large test suites** — Split test execution across matrix jobs for parallel execution.

## Quick Example

```yaml
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: 'pnpm'
```

```bash
# Turborepo remote cache via Vercel
TURBO_TOKEN=xxx turbo run build

# Nx Cloud
NX_CLOUD_ACCESS_TOKEN=xxx nx affected -t build
```

CI Optimization — Monorepo Management

You are an expert in optimizing CI/CD pipelines for monorepos, including affected-based execution, caching strategies, parallel workflows, and incremental builds.

Core Philosophy

Overview

Monorepo CI pipelines face a core challenge: as the codebase grows, running every task for every package on every PR becomes unsustainably slow and expensive. CI optimization for monorepos involves running only what's affected by a change, caching everything possible, parallelizing aggressively, and structuring workflows to give fast feedback on what matters most.

Setup & Configuration

GitHub Actions — Affected-Based Pipeline

name: CI
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      packages: ${{ steps.filter.outputs.changes }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            web:
              - 'apps/web/**'
              - 'packages/ui/**'
              - 'packages/utils/**'
            api:
              - 'apps/api/**'
              - 'packages/core/**'
              - 'packages/utils/**'
            docs:
              - 'apps/docs/**'

  build-and-test:
    needs: detect-changes
    if: needs.detect-changes.outputs.packages != '[]'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        package: ${{ fromJson(needs.detect-changes.outputs.packages) }}
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'
      - run: pnpm install --frozen-lockfile
      - run: pnpm --filter "./apps/${{ matrix.package }}..." run build
      - run: pnpm --filter "./apps/${{ matrix.package }}..." run test

Turborepo-Based Pipeline

name: CI
on:
  pull_request:
    branches: [main]

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      # Turbo remote cache
      - run: pnpm turbo run build test lint type-check --filter="...[origin/main]"
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

Nx-Based Pipeline

name: CI
on:
  pull_request:
    branches: [main]

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - run: pnpm install --frozen-lockfile

      - uses: nrwl/nx-set-shas@v4

      - run: npx nx affected -t build test lint --parallel=3
        env:
          NX_CLOUD_ACCESS_TOKEN: ${{ secrets.NX_CLOUD_ACCESS_TOKEN }}

Core Patterns

Layer 1: Affected Detection

Only run tasks for packages impacted by the change.

Path-based filtering (simple, no tooling required):

# Using dorny/paths-filter
- uses: dorny/paths-filter@v3
  id: changes
  with:
    filters: |
      backend:
        - 'apps/api/**'
        - 'packages/core/**'

Graph-based filtering (understands dependency relationships):

# Turborepo
turbo run build --filter="...[origin/main]"

# Nx
nx affected -t build --base=origin/main

# Lerna
lerna run build --since=origin/main

Graph-based is superior because it catches transitive impacts: changing @myorg/utils correctly triggers builds for every package that depends on it.

Layer 2: Caching

Package manager cache:

- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: 'pnpm'

Build output cache (Turborepo):

# Local cache (GitHub Actions cache)
- uses: actions/cache@v4
  with:
    path: node_modules/.cache/turbo
    key: turbo-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}-${{ github.sha }}
    restore-keys: |
      turbo-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}-
      turbo-${{ runner.os }}-

Remote cache (shared across all CI runs and developers):

# Turborepo remote cache via Vercel
TURBO_TOKEN=xxx turbo run build

# Nx Cloud
NX_CLOUD_ACCESS_TOKEN=xxx nx affected -t build

Docker layer caching:

# Prune monorepo for a specific app (Turborepo)
FROM node:20-alpine AS pruner
RUN corepack enable
WORKDIR /app
COPY . .
RUN turbo prune @myorg/web --docker

# Install dependencies using pruned lockfile
FROM node:20-alpine AS installer
RUN corepack enable
WORKDIR /app
COPY --from=pruner /app/out/json/ .
COPY --from=pruner /app/out/pnpm-lock.yaml ./pnpm-lock.yaml
RUN pnpm install --frozen-lockfile

# Build with full source
COPY --from=pruner /app/out/full/ .
RUN pnpm turbo run build --filter=@myorg/web

FROM node:20-alpine AS runner
WORKDIR /app
COPY --from=installer /app/apps/web/.next/standalone ./
COPY --from=installer /app/apps/web/public ./apps/web/public
COPY --from=installer /app/apps/web/.next/static ./apps/web/.next/static
CMD ["node", "apps/web/server.js"]

Layer 3: Parallelism

Matrix strategies:

jobs:
  lint-and-typecheck:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        task: [lint, type-check]
    steps:
      - uses: actions/checkout@v4
      - run: pnpm turbo run ${{ matrix.task }}

  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: pnpm turbo run test -- --shard=${{ matrix.shard }}/4

Nx distributed task execution:

- run: npx nx-cloud start-ci-run --distribute-on="5 linux-medium-js"
- run: npx nx affected -t build test lint --parallel=3
- run: npx nx-cloud stop-all-agents

Layer 4: Fast Feedback Ordering

Structure jobs so fast checks complete before slow ones:

jobs:
  lint:          # ~30s — runs first, catches formatting issues
    ...
  type-check:    # ~1min — catches type errors
    ...
  unit-test:     # ~2min — catches logic errors
    needs: [type-check]
  build:         # ~3min — catches build issues
    needs: [type-check]
  e2e:           # ~10min — catches integration issues
    needs: [build]
  deploy-preview:
    needs: [build, unit-test]

Layer 5: Selective Deployment

deploy-web:
  needs: [build, test]
  if: |
    github.ref == 'refs/heads/main' &&
    contains(needs.detect-changes.outputs.packages, 'web')
  steps:
    - run: pnpm --filter @myorg/web deploy

deploy-api:
  needs: [build, test]
  if: |
    github.ref == 'refs/heads/main' &&
    contains(needs.detect-changes.outputs.packages, 'api')
  steps:
    - run: pnpm --filter @myorg/api deploy

Best Practices

1. Use graph-based affected detection — Path filters miss transitive dependencies. Use Turbo/Nx/Lerna's graph awareness for correct affected analysis.
2. Enable remote caching — Local CI caches are per-runner; remote caches are shared across all runs and dramatically reduce redundant work.
3. Fetch full git history for affected — Use fetch-depth: 0 so affected commands can compare against the base branch.
4. Cancel redundant runs — Use concurrency groups to cancel in-progress CI when a new commit is pushed.
5. Fail fast on cheap checks — Run lint and type-check before tests and builds. A 30-second lint failure saves waiting for a 10-minute build.
6. Shard large test suites — Split test execution across matrix jobs for parallel execution.
7. Use turbo prune for Docker builds — Extract only the files needed for a specific app to maximize Docker layer cache hits.
8. Monitor CI metrics — Track p50/p90 CI times per workflow. Set budgets and alert on regressions.

Common Pitfalls

• Shallow clones breaking affected detection — actions/checkout defaults to fetch-depth: 1. Affected commands need history to compare. Use fetch-depth: 0.
• Cache key thrashing — If the cache key changes on every commit, you never get hits. Use lockfile hashes and restore-keys for partial matches.
• Not canceling stale runs — Without concurrency groups, pushing 5 commits queues 5 full CI runs instead of canceling the outdated ones.
• Over-splitting into too many jobs — Each job has setup overhead (checkout, install, cache restore). Splitting lint into 20 jobs can be slower than running them together.
• Missing --frozen-lockfile in CI — Without it, pnpm install may modify the lockfile and cause non-reproducible builds.
• Path filters without dependency awareness — Changing a shared util doesn't match the apps/web/** path filter, so web's tests are silently skipped even though they may be broken.

Anti-Patterns

Over-engineering for hypothetical scale. Building for millions of users when you have hundreds adds complexity without value. Solve today's problems first.

Ignoring the existing ecosystem. Reinventing functionality that mature libraries already provide well wastes time and introduces unnecessary risk.

Premature abstraction. Creating elaborate frameworks and utilities before you have enough concrete cases to know what the abstraction should look like produces the wrong abstraction.

Neglecting error handling at boundaries. Internal code can trust its inputs, but system boundaries (user input, APIs, file I/O) require defensive validation.

Skipping documentation for obvious code. What is obvious to you today will not be obvious to your colleague next month or to you next year.