Skip to main content

Anatomy of a CI/CD pipeline

Summary: a pipeline is the technical translation of CI/CD — a series of automated steps triggered by a Git event. This lesson breaks down each step (checkout, build, test, analysis, packaging, deployment), clarifies the difference between stage and job, explains the role of runners, artifacts and caches, and ends with a complete annotated example.

1. What a pipeline is​

The pipeline is code. It lives in a file versioned with your application: .github/workflows/ci.yml for GitHub Actions, .gitlab-ci.yml for GitLab, Jenkinsfile for Jenkins. This is what we call pipeline as code, and it is a major achievement of the 2015-2018 years.

2. The classic steps of a pipeline​

Golden rule of ordering: put the steps that fail fast and cost little first. A linter that takes 10 seconds must run before an 8-minute integration test suite. This is called the fail fast principle — fail early to make feedback faster.

3. Stage, job, step — the vocabulary to master​

These three words come up constantly and refer to different levels of organization.

TermDefinitionParallelism
PipelineThe complete execution, triggered by an event—
Stage (or phase)A logical group of jobs. The next stage waits for the success of the previous oneStages are sequential
JobAn execution unit that runs on a runner, with its own environmentJobs within the same stage are parallel
StepA command or an action within a jobSteps are sequential within their job

Important practical consequence: two jobs of the same stage do not share their file system. If the build job produces a binary that the deploy job needs, you must go through an artifact — which is the subject of the next section.

4. Runners — where the pipeline actually runs​

A runner (or agent, or executor) is the machine that executes a job.

Crucial security point: a self-hosted runner that executes pipelines from pull requests coming from outside is a gaping hole — a malicious contributor can run any code on your machine, with access to your internal network. That is why open source projects almost exclusively use hosted, ephemeral runners for external contributions.

5. Artifacts and caches — two notions not to confuse​

This is a very frequent confusion among beginners, with real consequences.

The test that settles it: "if I delete this, does my pipeline fail or does it just get slower?". Fails → it is an artifact. Slower → it is a cache.

Classic mistake to avoid: putting a build artifact in the cache. If the cache expires or is invalidated, the deployment fails in a mysterious and intermittent way — one of the most painful bugs to diagnose in CI.

6. The single-artifact principle​

This is the most important rule of continuous delivery, stated by Humble and Farley.

Corollary: configuration must be external to the artifact. The same container image runs in staging and in production, only the environment variables change. It is one of the principles of the Twelve-Factor App.

7. Complete example explained​

Here is a realistic GitHub Actions pipeline for a containerized Node.js application.

name: CI/CD

# Triggers: on push to main, and on any pull request
on:
  push:
    branches: [main]
  pull_request:

# Cancels previous runs on the same branch
# Avoids wasting runner minutes on stale code
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # ---------- STAGE 1: validation (jobs in parallel) ----------
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'          # automatic cache of ~/.npm
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    # Real database for the integration tests
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'
      - run: npm ci
      - run: npm test
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/postgres
      # Publishing an artifact: the coverage report
      - uses: actions/upload-artifact@v4
        with:
          name: coverage
          path: coverage/

  # ---------- STAGE 2: build (waits for stage 1 to succeed) ----------
  build:
    needs: [lint, test]          # explicit dependency
    if: github.ref == 'refs/heads/main'   # only on main
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write            # right to push to the registry
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          push: true
          # Tagged with the commit hash: traceable artifact
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha   # Docker layer cache
          cache-to: type=gha,mode=max

  # ---------- STAGE 3: deployment (protected environment) ----------
  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: production      # can require manual approval
    steps:
      - name: Deploy the image to the cluster
        run: |
          echo "Deploying ghcr.io/${{ github.repository }}:${{ github.sha }}"
          # kubectl set image deployment/my-app app=ghcr.io/...:${{ github.sha }}

The notable points of this example:

ElementWhy it matters
concurrency with cancel-in-progressCancels stale builds, saves runner minutes
lint and test without needsThey run in parallel, the pipeline is faster
services: postgresA real ephemeral database for the integration tests
needs: [lint, test]The build only happens if the validation succeeded
if: github.ref == 'refs/heads/main'No image is published for a mere pull request
tags: ...:${{ github.sha }}The artifact is traceable down to the exact commit
cache-from: type=ghaReuses Docker layers, build 3 to 10 times faster
environment: productionAllows requiring a human approval — this is the switch between continuous deployment and continuous delivery
Explicit permissionsPrinciple of least privilege for the pipeline token

Note this decisive detail: the line environment: production is exactly what makes this pipeline switch between continuous delivery (if the environment requires an approval) and continuous deployment (if it does not). A single line of configuration separates the two practices seen in lesson 2.

8. What makes a pipeline good or bad​

A pipeline's worst enemy: flaky tests. A test that fails one time out of ten for no reason teaches the team to re-run without thinking. From that moment on, the pipeline no longer protects anything: real failures are ignored just like the false ones. A flaky test must be fixed or deleted, never tolerated.

Remember in 30 seconds​

  • A pipeline is a series of automated steps triggered by a Git event, defined in a versioned file.
  • Typical steps: checkout, cache, dependencies, lint, build, unit tests, integration tests, security scan, packaging, publication, deployment, monitoring.
  • Fail fast: put what fails fast and costs little first.
  • Stage (sequential) > Job (parallel within a stage) > Step (sequential within a job).
  • A runner is the machine that executes a job. Hosted, self-hosted, or ephemeral in a container.
  • Artifact = what you produce and keep. Cache = what you reuse to go fast. Never confuse them.
  • One build: the same artifact travels through staging, pre-production and production.
  • A good pipeline is fast (under 10 min), deterministic, readable, informative and secure.

Next: Deployment strategies: rolling, blue-green, canary, feature flags →