ai-workflow-course/modules/18-continuous-delivery-and-deployment/README.md

# Module 18 — Continuous Delivery and Deployment

> **Merged isn't running.** This module closes the last gap in the pipeline — getting approved code
> from `main` to something actually serving traffic, automatically, with a way back when it's wrong.

---

## Prerequisites

- **Module 10 — Reviewing Code You Didn't Write.** The PR review gate. Auto-deploy is only safe
  because a human (or an agent under supervision) signed off on the diff first.
- **Module 14 — Continuous Integration.** You already have a pipeline that lints, builds, and tests
  on every push. CD is not a new system — it's **more stages on that same pipeline**, after the
  checks pass.
- **Module 15 — Security Scanning.** Dependency, secret, and static-analysis gates on the same
  pushes. These are part of what makes shipping without a human in the loop survivable.
- **Module 16 — Containers and Reproducible Environments.** The container image is *what you ship*.
  CD takes that image and runs it somewhere. This module assumes you can already build and tag an
  image of the `tasks-app`.
- **Module 17 — Secrets, Config, and Environments.** A running service needs configuration and
  secrets at runtime — *what it needs to run*. CD wires those into the deploy step instead of baking
  them into the image.

If you've done 14–17, you have all the parts. This module is the assembly.

---

## Learning objectives

By the end of this module you can:

1. State the precise difference between continuous **delivery** and continuous **deployment**, and
   decide which one a given project should use.
2. Extend your CI pipeline with build-and-publish stages that turn a merge into a versioned,
   deployable artifact.
3. Wire a deploy step that takes that artifact, injects runtime config/secrets, and brings up the
   new version — provider-neutrally.
4. Add a health check and an automatic **rollback** so a bad deploy reverts itself instead of
   staying down.
5. Reason about the deploy gate the way this audience already reasons about change windows: what's
   automated, what's manual, and where the stop button is.

---

## Key concepts

### The gap nobody automated yet

Walk the pipeline you've built so far. A change gets proposed (Module 9), implemented on a branch
(Module 6), reviewed as a PR (Module 10), checked by CI (Module 14), scanned for vulnerabilities
(Module 15). It merges. `main` is now correct, tested, and clean.

And then nothing happens. The code that's "done" is sitting in a Git history. The thing your users
touch is still running last week's version. Somebody — usually you, usually at 6pm — has to SSH in,
pull, build, restart, and pray. That manual last mile is where most outages are actually born:
inconsistent steps, a forgotten config flag, a half-restarted service, "wait, which version is in
prod right now?"

CI answered *"is this change good?"* CD answers the next question: ***"now get the good change
running, the same way every time."*** It's the same instinct that made CI worth it — replace an
error-prone manual ritual with an automated, repeatable one — pointed at the last step.

### Delivery vs. deployment: the distinction that matters

These two terms get used interchangeably and they are not the same thing. The difference is exactly
one decision: **who pushes the button to prod.**

- **Continuous Delivery** — every merge to `main` automatically produces a **deployable artifact**
  (a built, tagged, tested container image, sitting in a registry) and deploys it as far as a
  staging/pre-prod environment. Production deploy is **one click by a human**. The pipeline
  guarantees the artifact is *ready to ship at any moment*; a person decides *when*.

- **Continuous Deployment** — same pipeline, but there's **no button**. If it passes every gate, it
  goes all the way to production automatically. Merge is the last human action.

```
                 merge to main
                      │
        ┌─────────────┴──────────────┐
   CONTINUOUS DELIVERY          CONTINUOUS DEPLOYMENT
        │                            │
   build + test + scan          build + test + scan
        │                            │
   publish artifact             publish artifact
        │                            │
   deploy to staging            deploy to staging
        │                            │
   [human clicks "ship"] ──►    deploy to prod  (automatic)
        │                            │
   deploy to prod                  done
```

Both are "CD." When someone says "we do CD," ask which one — the operational risk is completely
different. Continuous deployment is not the more advanced/better option you graduate to; it's a
different risk posture that's appropriate for some systems and reckless for others. A blog,
internal dashboard, or stateless web service with good tests is a fine candidate. A billing engine,
a database migration, or anything with a regulatory change-control requirement usually is not — and
"a human clicks deploy" is a perfectly mature answer there, not a failure to automate.

The honest default for most teams adopting this: **start with continuous *delivery*.** Get the
artifact and the deploy step fully automated and trustworthy, keep the human on the prod button, and
remove that button only once you trust the gates more than you trust the click.

### The artifact is the unit of deploy

Here's the discipline that makes CD reliable, and it comes straight from Module 16: **you deploy a
built image, not a Git ref.** "Deploy `main`" is ambiguous — it means "go to the prod box, pull,
and rebuild," and that rebuild can pull a different base image or dependency version than CI tested.
"Deploy `tasks-app:9f3a2c1`" is not ambiguous. It's the exact bytes CI built and tested.

So the build-and-publish stage does this once, centrally:

1. Build the image from the merged code.
2. Tag it with something **immutable and traceable** — the Git commit SHA is the standard choice
   (`tasks-app:9f3a2c1`). Optionally also a moving tag like `:latest` or `:staging` for convenience,
   but the SHA tag is the one you trust.
3. Push it to a container registry — the durable, shared home for images, the same way a Git remote
   (Module 8) is the durable home for commits.

Every later deploy — to staging, to prod, a rollback — just says "run *this* tag." Build once, run
the identical artifact everywhere. That single property is what kills "works on my machine" at the
deploy layer.

### The deploy step, provider-neutrally

The shape of a deploy is the same everywhere, whatever the target — a cloud platform, a Kubernetes
cluster, a single VM, a PaaS:

1. **Pull** the specific image tag onto the target.
2. **Inject runtime config and secrets** (Module 17) — environment variables, mounted secret files,
   a secrets-manager lookup. Never baked into the image; supplied at run time so the *same* image
   runs in staging and prod with different config.
3. **Start the new version** alongside or in place of the old one.
4. **Health-check** it before sending real traffic.
5. **Cut over** if healthy; **roll back** if not.

This module is deliberately provider-agnostic on *where* — the same way Module 8 stayed neutral on
hosts. The mechanics differ (a `kubectl` apply, a platform CLI, a `docker run`, a `compose up`), but
the five steps don't. The lab does the simplest possible real version: a local container run. The
logic is identical at scale.

### Health checks and rollback: the part beginners skip

A deploy that can't tell whether it worked isn't a deploy, it's a gamble. The single most important
thing CD adds over "SSH in and restart" is that **the pipeline verifies the new version is alive
before trusting it, and reverses itself when it isn't.**

A health check is a cheap, honest signal that the new version is actually serving — typically an
endpoint like `/health` that returns `200` only when the app has started clean. The deploy step
hits it after starting the new version and **waits for green before cutting over.**

Rollback is the other half: if the health check fails, the deploy stops the broken new version and
brings the **previous known-good image tag** back up. Because you deploy immutable tags, rollback is
trivial — you still have `tasks-app:<previous-sha>`, so "go back" is just "run the old tag again."
No rebuild, no git revert race, no scramble. (Reverting the *source* is still Module 12's job for the
code; rollback here is about the *running artifact*.) The strategies have names you'll meet —
blue-green (run old and new side by side, flip a switch), canary (send 5% of traffic to new, watch,
ramp) — but they're all variations on "keep the old one ready until the new one proves itself."

> **Reframe for the ops reader:** you already know this instinct. It's the deployment equivalent of
> a maintenance window with a back-out plan — except the back-out plan is automated, tested on every
> single deploy, and takes seconds instead of a panicked hour. CD doesn't remove the discipline you
> already have; it encodes it so it runs every time instead of only when someone remembers.

---

## The AI angle

CI existed long before AI, and so did CD. What changed is the **rate**, and rate is everything for
the merged-to-prod gate.

AI writes and ships changes dramatically faster. More PRs open, more merge, and they merge sooner.
That's the upside — and it means the volume of code flowing toward production goes *up*, while the
human attention available to babysit each deploy stays flat. The gap between "merged" and "in prod"
stops being a quiet formality and becomes the place where the speed either pays off or hurts you.

Two consequences follow, and they pull in opposite directions:

- **Automating the deploy matters more.** If a human has to hand-deploy every AI-generated change,
  the manual last mile becomes the bottleneck that eats all the speed AI just gave you. CD is what
  lets the throughput actually reach users.
- **The gate matters more.** Faster shipping of code that *looks right* (the recurring AI failure
  mode from Modules 1 and 14) means a bad change reaches prod faster too — unless something catches
  it. This is the crucial point: **continuous deployment is only survivable because of the gates in
  front of it.** Review (Module 10), CI tests (Module 14), and security scanning (Module 15) are not
  bureaucracy you tolerate — they are the *entire reason* you're allowed to remove the human from the
  deploy button. Take auto-deploy without those gates and you've built a machine that ships AI
  mistakes to production at full speed.

So the AI-era posture is specific: **strengthen the early gates, then automate the late ones.** The
more you trust review + CI + scanning, the further right you can safely push automation — up to and
including no human on the prod button. The strength of the gates is the dial that decides whether
continuous *deployment* is responsible or reckless for a given repo. And when an agent itself is the
one merging (Unit 5), this stops being theoretical: the deploy gate is the last thing standing
between an autonomous contributor and your users.

---

## Hands-on lab

**Lab language:** shell, driving the container tooling from Module 16. You'll extend the `tasks-app`
into a tiny running service, then build a deploy script that ships it locally with a health check and
automatic rollback — the whole CD motion, simulated on your own machine.

This lab simulates deployment with a **local container run** so it works on any machine with no cloud
account. The five deploy steps are real; only the *target* is your laptop instead of a server.

**You'll need:**

- A container runtime from Module 16 — Docker or Podman. (Commands below use `docker`; if you run
  Podman, `alias docker=podman` or substitute.)
- The `tasks-app` from Modules 1–2, now a Git repo.
- `curl` (for the health check) and a bash-capable shell. On Windows, use WSL or Git Bash.
- Your AI assistant — by now, ideally editor-integrated (Module 4).

Starter files are in this module's `lab/` folder:

- `serve.py` — turns the `tasks-app` into a minimal HTTP service with a `/health` endpoint, using
  only the Python standard library (no dependencies). This is the long-running thing CD deploys.
- `Dockerfile` — the Module 16 container image, adjusted to run the service.
- `deploy.sh` — the deploy step: build, tag, run, health-check, cut over or roll back.
- `cd-starter.yml` — the CD pipeline stages, written as GitHub Actions and extending the Module 14
  CI file. GitLab/other-forge notes are in the comments.

### Part A — Make something worth deploying

A CLI that exits immediately is awkward to "deploy." Give the app a long-running face.

1. Copy `lab/serve.py` and `lab/Dockerfile` into your `tasks-app` folder next to `tasks.py` and
   `cli.py`. Read `serve.py` — it's ~40 lines wrapping the `TaskList` you already have in a stdlib
   HTTP server with two routes: `/health` and `/tasks`.

2. Run it locally first, no container, to see it work:

   ```bash
   python serve.py        # serves on http://localhost:8000
   ```

   In another terminal:

   ```bash
   curl localhost:8000/health     # {"status": "ok", "version": "dev"}
   curl localhost:8000/tasks      # your tasks as JSON
   ```

   Stop it with Ctrl-C. Commit this (`git add . && git commit -m "Add HTTP service + Dockerfile"`).

### Part B — Build and tag the artifact

3. Build the image and tag it with the current commit SHA — the immutable, traceable tag:

   ```bash
   SHA=$(git rev-parse --short HEAD)
   docker build -t tasks-app:$SHA -t tasks-app:latest .
   docker images tasks-app        # see both tags pointing at one image
   ```

   That `:$SHA` tag is the unit of deploy. Everything downstream refers to *this exact image*.

### Part C — Deploy it (with a net)

4. Read `lab/deploy.sh`. It does the five steps: stops any running `tasks-app` container, starts the
   new image with runtime config injected as env vars (Module 17 — note the `APP_VERSION` and the
   *absence* of any secret baked into the image), polls `/health` until green, and on failure rolls
   back to the previous tag it recorded. Make it executable and run it:

   ```bash
   chmod +x deploy.sh
   ./deploy.sh $SHA
   ```

   Watch it build, run, health-check, and report the deploy healthy. Hit it:

   ```bash
   curl localhost:8000/health     # now reports the SHA you deployed
   ```

   Run `./deploy.sh` again after another commit and notice it records the prior version as the
   rollback target. You now have continuous *delivery* in miniature: one command turns a commit into
   a running, version-tagged service.

### Part D — Break a deploy and watch it roll back

5. Now prove the net works. The service honors a `BREAK=1` env var that makes `/health` return `500`
   — a stand-in for "this build starts but is actually broken." Deploy a healthy version first so
   there's a known-good to fall back to, then force a bad one:

   ```bash
   ./deploy.sh $SHA               # healthy baseline
   BREAK=1 ./deploy.sh $SHA       # same image, but the new instance fails its health check
   ```

   The script starts the "new" version, the health check fails, and it **automatically stops the
   broken instance and brings the previous good one back up.** Confirm you're still serving:

   ```bash
   curl localhost:8000/health     # ok — the bad deploy reverted itself
   ```

   That automatic reversal — not the build, not the run — is the part that makes auto-deploy
   something you can sleep through.

### Part E — Wire it into the pipeline (read + reason)

6. Open `lab/cd-starter.yml` and compare it to the Module 14 `ci-starter.yml`. It's the **same
   pipeline with stages appended**: the lint/test/scan gates run first (unchanged), and only `on:
   push` to `main` (a merge) do the build-publish-deploy stages run. Trace the `needs:`/dependency
   chain that makes deploy run *only after* the checks pass.

7. Find the one line that is the delivery-vs-deployment switch — the deploy-to-prod step gated behind
   a manual approval (`environment:` with a required reviewer, commented in the file). Decide, for
   the `tasks-app`, which side you'd choose and why, and ask your AI assistant to make the case for
   the *other* choice. The goal isn't a "right" answer; it's being able to articulate the risk
   posture either way.

> **A note on running the full pipeline:** actually executing `cd-starter.yml` end to end needs a
> forge with a container registry and a deploy target wired up — that's environment-specific and
> partly Module 19's territory (the runners and compute underneath). Parts A–D give you the deploy
> *logic* runnable today on your own machine; the YAML shows how it slots into the automated
> pipeline you already started in Module 14.

---

## Where it breaks

Be honest about the edges — this is where teams get burned.

- **The deploy is only as safe as the gates in front of it.** Continuous deployment with weak tests
  and no review isn't "moving fast," it's an automated mistake-shipping machine. If you haven't done
  the Module 10/14/15 work, do *delivery* (human on the button), not *deployment*. Auto-deploy is a
  reward you earn by trusting your gates, not a default you turn on.
- **Health checks lie.** A `200` from `/health` means "the process started," not "the feature
  works." A shallow health check passes while the app returns garbage to users. Make the check
  meaningful (does it reach its database? can it serve a real request?) and lean on canary/gradual
  rollout for anything important — but know that no health check replaces real tests and real
  monitoring.
- **Rollback isn't free, and some things don't roll back.** Reverting the *running image* is cheap.
  Reverting a **database migration**, a sent email, a charged credit card, or a published message is
  not — those are forward-only. The cleaner the separation between code deploys and irreversible
  state changes, the more rollback actually saves you. Don't assume "we can always roll back" covers
  data.
- **This lab simulates the target.** A local `docker run` is the deploy logic, not the deploy
  reality. Real targets add networking, DNS cutover, load balancers, zero-downtime orchestration,
  and multiple instances. The five steps hold; the operational surface around them is larger. The
  *compute* that runs all of this — and why you might run your own — is Module 19.
- **"Build once" only holds if you actually do.** The instant someone rebuilds on the prod box "just
  to be sure," you've lost the guarantee that prod runs what CI tested. Deploy the artifact CI built.
  No rebuilds downstream.

---

## Check for understanding

**You're done when:**

- You can state the difference between continuous delivery and continuous deployment in one sentence
  — *who clicks the prod button* — and say which one `tasks-app` should use and why.
- `./deploy.sh` builds, tags by commit SHA, runs the container, and reports a healthy deploy you can
  `curl`.
- You have **watched a bad deploy roll itself back** to the previous good version, and the service
  stayed up.
- You can point at the line in `cd-starter.yml` that turns delivery into deployment, and explain what
  gates have to be trustworthy before you'd flip it.

When a deploy is one command, a bad one reverts itself, and you can argue the delivery-vs-deployment
call for a given repo, you've closed the merged-to-running gap. Module 19 goes underneath all of
this — the runners and compute actually executing your CI/CD, and why you'd own them.

---

## Verify-before-publish

This is expansion-zone material (Module 15+); some specifics drift. Re-check at build/publish time:

- [ ] **Action/runner versions** in `cd-starter.yml` (`actions/checkout`, `actions/setup-python`,
      any build/login/push actions) — pin to current major versions and confirm they still exist.
- [ ] **Registry login + push syntax** — the standard build-and-push action names and auth flow
      change; verify against current forge docs rather than the comments here.
- [ ] **Manual-approval mechanism** — the way a forge gates a job behind human approval
      (GitHub `environment` protection rules, GitLab `when: manual`, others) shifts in naming/UI.
      Confirm the delivery-vs-deployment switch still maps to the current feature.
- [ ] **Container runtime commands** — confirm `docker`/`podman` flags used in `deploy.sh`
      (`run`, `--health-*`, `inspect`) match current CLI behavior.
- [ ] **Cross-references** to Modules 16, 17, and 19 still match those modules' final content.