You are an expert in CircleCI for continuous integration and deployment.

## Key Points

- restore_cache:
- run: npm ci
- save_cache:
- Use orbs for common integrations (AWS, Docker, Slack) instead of writing custom steps.
- Leverage workspaces to pass build artifacts between jobs; use caches for dependencies that rarely change.
- Use contexts to group and manage secrets per environment, with security group restrictions.
- Version your cache keys (e.g., `v1-deps-...`) so you can bust caches by incrementing the prefix.
- Use `resource_class` to right-size executors; small for linting, large for compilation.
- Enable dynamic configuration with `setup` workflows for monorepo path filtering.
- Use `store_test_results` to get test insights and automatic test splitting with `circleci tests split`.
- Add `approval` jobs for manual gates before production deploys.
- Use OIDC tokens (`circleci/aws-cli` orb supports this) instead of long-lived credentials.

CircleCI — CI/CD

You are an expert in CircleCI for continuous integration and deployment.

Overview

CircleCI is a cloud-native CI/CD platform configured via .circleci/config.yml. It uses a concept of jobs, workflows, executors, and orbs (reusable config packages). CircleCI supports Docker, Linux, macOS, Windows, and Arm executors. Pipelines are triggered by VCS events (push, PR) or via API, and workflows orchestrate job execution order with fan-in/fan-out patterns.

Setup & Configuration

Configuration lives in .circleci/config.yml at the repository root.

Basic configuration:

version: 2.1

orbs:
  node: circleci/node@5.2.0

executors:
  default:
    docker:
      - image: cimg/node:20.11
    resource_class: medium
    working_directory: ~/project

jobs:
  build-and-test:
    executor: default
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run:
          name: Run tests
          command: npm test
      - store_test_results:
          path: test-results
      - store_artifacts:
          path: coverage

  deploy:
    executor: default
    steps:
      - checkout
      - run:
          name: Deploy to production
          command: ./deploy.sh

workflows:
  build-deploy:
    jobs:
      - build-and-test
      - deploy:
          requires:
            - build-and-test
          filters:
            branches:
              only: main

Core Patterns

Orbs for Reusable Configuration

Orbs encapsulate jobs, commands, and executors into reusable packages:

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@4.1.3
  docker: circleci/docker@2.6.0
  slack: circleci/slack@4.13.3

jobs:
  deploy:
    executor: aws-cli/default
    steps:
      - checkout
      - aws-cli/setup:
          role_arn: arn:aws:iam::123456789012:role/CircleCIRole
      - run: aws ecs update-service --cluster prod --service app --force-new-deployment
      - slack/notify:
          event: pass
          template: basic_success_1

Pipeline Parameters

Parameterize pipelines for conditional logic:

version: 2.1

parameters:
  run-integration-tests:
    type: boolean
    default: false
  deploy-environment:
    type: enum
    enum: ["staging", "production"]
    default: "staging"

jobs:
  integration-tests:
    docker:
      - image: cimg/node:20.11
      - image: cimg/postgres:16.2
    steps:
      - checkout
      - run: npm ci
      - run: npm run test:integration

workflows:
  main:
    jobs:
      - build-and-test
      - integration-tests:
          filters:
            branches:
              only: main
      - deploy:
          requires:
            - build-and-test
          context: << pipeline.parameters.deploy-environment >>

Workspaces for Sharing Data Between Jobs

jobs:
  build:
    executor: default
    steps:
      - checkout
      - run: npm ci && npm run build
      - persist_to_workspace:
          root: .
          paths:
            - dist
            - node_modules

  test:
    executor: default
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run: npm test

  deploy:
    executor: default
    steps:
      - attach_workspace:
          at: .
      - run: ./deploy.sh

Fan-Out / Fan-In Workflows

workflows:
  build-test-deploy:
    jobs:
      - build
      - unit-tests:
          requires: [build]
      - integration-tests:
          requires: [build]
      - security-scan:
          requires: [build]
      - deploy:
          requires:
            - unit-tests
            - integration-tests
            - security-scan
          filters:
            branches:
              only: main

Caching Dependencies

steps:
  - restore_cache:
      keys:
        - v1-deps-{{ checksum "package-lock.json" }}
        - v1-deps-
  - run: npm ci
  - save_cache:
      key: v1-deps-{{ checksum "package-lock.json" }}
      paths:
        - node_modules

Scheduled Pipelines

workflows:
  nightly:
    triggers:
      - schedule:
          cron: "0 2 * * *"
          filters:
            branches:
              only: main
    jobs:
      - full-test-suite
      - security-audit

Contexts for Secret Management

workflows:
  deploy:
    jobs:
      - deploy-staging:
          context:
            - aws-staging
            - slack-notifications
      - approve-production:
          type: approval
          requires: [deploy-staging]
      - deploy-production:
          context:
            - aws-production
            - slack-notifications
          requires: [approve-production]

Core Philosophy

CircleCI's design centers on composability: orbs, reusable executors, and workflow-level orchestration let you build pipelines from well-tested, shared components rather than copying and pasting YAML across repositories. This composability is not just a convenience feature — it is the key to maintaining CI/CD at scale. When a security patch needs to be applied to your Docker build process, updating a single orb version propagates the fix to every pipeline that uses it. The alternative — manually updating dozens of workflow files — is how CI configurations rot.

Workspaces and caching serve fundamentally different purposes and mixing them up is one of the most common sources of pipeline confusion. Caches are performance optimizations for dependencies that change infrequently; they are best-effort and may not be available. Workspaces are guaranteed data transfer between jobs within a single workflow run; they carry your build artifacts from the build job to the test job to the deploy job. Using caches where workspaces are needed introduces non-determinism. Using workspaces where caches would suffice wastes storage and slows pipelines.

The workflow layer is where CircleCI's real power lives. Fan-out/fan-in patterns let you run independent checks in parallel (unit tests, integration tests, security scans) and gate deployment on all of them passing. Approval jobs inject human judgment at precisely the right moment — after automated checks pass but before production deployment. This workflow-level orchestration turns a collection of scripts into a release process with safety guarantees.

Anti-Patterns

• Monolithic single-job pipelines. Putting build, test, lint, and deploy into one giant job means a linting failure blocks deployment even when the fix is trivial, and nothing runs in parallel. Break work into separate jobs orchestrated by workflows so independent steps run concurrently and failures are isolated.

• Floating orb versions. Using @volatile or unpinned orb versions means a breaking change in an orb can silently break your pipeline overnight. Pin orb versions to a specific release and update deliberately after reviewing changelogs.

• Caching for correctness. Relying on the cache to contain your build output instead of using persist_to_workspace means a cache miss causes the deploy job to deploy stale or missing artifacts. Caches are for speeding up dependency installation; workspaces are for passing data between jobs.

• Oversized resource classes everywhere. Using xlarge executors for every job because "faster is better" quadruples your CI bill. Right-size executors: small for linting, medium for unit tests, large only for compilation or resource-intensive work.

• Skipping test result reporting. Not using store_test_results means you lose CircleCI's test insights, timing data, and automatic test splitting capabilities. These features require JUnit XML or similar structured test output, and without them you are flying blind on test performance trends.

Best Practices

• Use orbs for common integrations (AWS, Docker, Slack) instead of writing custom steps.
• Leverage workspaces to pass build artifacts between jobs; use caches for dependencies that rarely change.
• Use contexts to group and manage secrets per environment, with security group restrictions.
• Version your cache keys (e.g., v1-deps-...) so you can bust caches by incrementing the prefix.
• Use resource_class to right-size executors; small for linting, large for compilation.
• Enable dynamic configuration with setup workflows for monorepo path filtering.
• Use store_test_results to get test insights and automatic test splitting with circleci tests split.
• Add approval jobs for manual gates before production deploys.
• Use OIDC tokens (circleci/aws-cli orb supports this) instead of long-lived credentials.
• Use when conditions on steps for conditional execution within jobs.

Common Pitfalls

• Caches are immutable once saved; a cache key that already exists will never be overwritten. Version your cache keys.
• Workspaces persist for 15 days and count against storage; use persist_to_workspace selectively.
• Docker layer caching (DLC) is a paid feature; without it, Docker builds in CI are uncached and slow.
• filters on workflows apply to branches AND tags differently; tag pushes require explicit tag filters.
• Orb versions should be pinned (@5.2.0) not floating (@volatile); use @5 for minor version flexibility.
• checkout fetches the default branch if the job is triggered by a tag without proper filter configuration.
• Resource class changes affect billing; xlarge costs 4x medium.
• Config validation errors are only caught when a pipeline is triggered; use circleci config validate locally.