Reframe sweep M7-27 + capstone (AI drives git, lesson=theory, de-slop) (#93)
Sync course wiki / sync-wiki (push) Successful in 11s

Co-authored-by: claude <claude@jpaul.io>
Co-committed-by: claude <claude@jpaul.io>
This commit was merged in pull request #93.
This commit is contained in:
2026-06-22 21:58:36 -04:00
committed by Claude (agent)
parent a29823f4b3
commit 513d7e7ac8
38 changed files with 1735 additions and 1424 deletions
+114 -86
View File
@@ -14,7 +14,7 @@
them on.
- **Module 2 — Version Control as a Safety Net.** Scanners flag findings in a diff; you'll commit,
re-scan, and confirm a gate goes red then green. Secret scanning in particular cares about *history*,
not just the working tree that only makes sense once you think in commits.
not just the working tree; that only makes sense once you think in commits.
- **Module 1 — the `tasks-app`.** The running example. We'll let the AI bolt a "cloud sync" feature
onto it and watch it introduce all three failure modes at once.
@@ -74,7 +74,7 @@ things through automatically* — pointed at a different failure mode.
| **SAST** (Static Application Security Testing) | Insecure code *you wrote* — injection, weak crypto, unsafe deserialization | Static analyzers / linters with a security ruleset |
SCA and SAST split the world cleanly: **SCA scans the code you didn't write (your dependencies);
SAST scans the code you did.** Secret scanning cuts across both a leaked key is neither a
SAST scans the code you did.** Secret scanning cuts across both: a leaked key is neither a
dependency nor a logic bug, it's a string that should never have been committed.
### Gate 1 — SCA: scanning the code you didn't write
@@ -91,8 +91,8 @@ the dependency that **doesn't exist at all.**
#### Slopsquatting: the AI supply-chain attack
LLMs generate plausible text, and a package name is plausible text. Ask for code that talks to a
service and the model will confidently `import` or list a dependency that *sounds* exactly right
`requests-oauth`, `python-jsonlogger2`, `task-store-client` but was never published. This isn't
service and the model will `import` or list a dependency that *sounds* exactly right
(`requests-oauth`, `python-jsonlogger2`, `task-store-client`) but was never published. This isn't
rare; studies of AI-generated code find a meaningful fraction of suggested packages are
hallucinations, and crucially, **the model hallucinates the same plausible names repeatedly.**
@@ -102,12 +102,12 @@ rather than human typos) — is:
1. Watch what package names LLMs commonly invent.
2. Register those exact names on the public package index, with malware inside.
3. Wait. The next developer who pastes AI output and runs `pip install -r requirements.txt`
(or `npm install`) pulls your payload which now runs with that developer's privileges, in their
(or `npm install`) pulls your payload, which now runs with that developer's privileges, in their
dev environment or, worse, in CI.
The defense has two layers, and SCA is where they live:
- **The package doesn't exist (yet).** The install or the resolver fails outright "no matching
- **The package doesn't exist (yet).** The install or the resolver fails outright with "no matching
distribution." Annoying, but *safe*: a name that 404s can't hurt you. The danger is treating that
as a mere typo and "fixing" it by finding the closest real name without checking it.
- **The package exists but you didn't vet it.** This is the live wire. SCA flags newly-published,
@@ -121,8 +121,8 @@ same way you'd treat a stranger handing you a USB stick.
### Gate 2 — Secret scanning
AI loves to hardcode credentials. Ask for code that calls an authenticated API and a model will
cheerfully write `API_KEY = "sk-live-..."` straight into the source, because that makes the example
*work* and "make it work" is what it optimizes for. It has no instinct that the key is sensitive.
write `API_KEY = "sk-live-..."` straight into the source, because that makes the example
*work*, and "make it work" is what it optimizes for. It has no instinct that the key is sensitive.
Secret scanners catch this by scanning files (and crucially, **git history**) for two signals:
@@ -132,7 +132,7 @@ Secret scanners catch this by scanning files (and crucially, **git history**) fo
when they match no known pattern.
The non-obvious part for this audience: **a secret committed once is leaked forever.** Deleting it in
a later commit doesn't help it's still sitting in history, and anyone with the repo can
a later commit doesn't help; it's still sitting in history, and anyone with the repo can
`git log -p` their way to it. So secret scanning runs over *history*, not just the current files, and
a true hit means two jobs, not one: (1) get it out of the code, and (2) **rotate the credential**,
because you must assume it's compromised. Scrubbing history is harder than it looks and is a
@@ -157,7 +157,7 @@ SAST flags the *shape* of the bug regardless of whether any test happens to trig
SAST is also the noisiest of the three. Expect false positives, expect to tune the ruleset, and
expect to mark some findings "won't fix" with a reason. That's normal and it's why SAST is introduced
*after* the two higher-signal gates it's the most valuable to tune and the easiest to turn into
*after* the two higher-signal gates: it's the most valuable to tune and the easiest to turn into
ignored red noise if you don't.
### Where the gates run
@@ -167,7 +167,8 @@ You want these in more than one place, cheapest-and-earliest first:
- **Local / pre-commit** — fastest feedback, and the only place that stops a secret *before* it
enters history. A pre-commit hook running secret scanning is the single highest-value placement.
- **CI (the Module 14 pipeline)** — the enforcement gate. Local hooks can be skipped; the pipeline
can't be, if you require it to pass before merge. This is where "the build goes red" has teeth.
can't be, if you require it to pass before merge. This is where "the build goes red" actually
blocks a merge.
- **Host-native, on the remote** — most git hosts (Module 8) offer some of this for free:
dependency alerts that watch your manifest against advisory feeds and open issues/PRs when a new
CVE drops, and push protection that rejects a commit containing a recognized secret at the server.
@@ -181,8 +182,8 @@ CI, so there's one source of truth for "what counts as a finding."
## The AI angle
These three gates exist in any DevSecOps practice. What makes them *load-bearing* here is that
AI-assisted coding doesn't just fail to prevent these problems it actively manufactures all three,
These three gates exist in any DevSecOps practice. What makes them matter here is that
AI-assisted coding doesn't just fail to prevent these problems; it actively manufactures all three,
and does it in the exact form that slips past a human skim and a green build:
- **It invents dependencies.** Hallucinated package names are a failure mode unique to generated
@@ -190,8 +191,8 @@ and does it in the exact form that slips past a human skim and a green build:
human typing dependencies by hand produces this risk at the same rate.
- **It hardcodes secrets** because hardcoding makes the example run, and running is what the model is
rewarded for. The instinct that "this string is dangerous" is exactly the instinct it lacks.
- **It reproduces insecure idioms** with total confidence, because plausible-looking code is the
whole game, and insecure code is extremely plausible it's all over the training data.
- **It reproduces insecure idioms** by default, because plausible-looking code is the
whole game, and insecure code is extremely plausible: it's all over the training data.
And the volume multiplies all of it. You're merging more code, faster, with less of it read
line-by-line, precisely because the AI made generation cheap. The one defense that scales with that
@@ -212,73 +213,83 @@ and wire the catch into your pipeline.
**You'll need:**
- The `tasks-app` folder under version control from Module 2, and your CI pipeline from Module 14.
- The `tasks-app` repo at `~/ai-workflow-course/tasks-app` under version control from Module 2, and
your CI pipeline from Module 14.
- Python 3.10+ and `pip`.
- Two scanners installed into your environment:
- Two scanners installed into your environment. Direct your agent (Claude Code is the worked example;
sub your own) to install them: *"Install the pip-audit and detect-secrets scanners into this
project's environment; if pip refuses with an externally-managed-environment error, make a venv
first and install into that."* The command it runs is `pip install pip-audit detect-secrets`.
Verify both landed (`pip-audit --version`, `detect-secrets --version`) before you go on.
```bash
pip install pip-audit detect-secrets
```
> **If `pip install` is refused** with "externally-managed-environment" (PEP 668 — common on
> recent Debian/Ubuntu and Homebrew Python), install into a per-project virtual environment
> **If `pip install` is refused** with "externally-managed-environment" (PEP 668, common on recent
> Debian/Ubuntu and Homebrew Python), the scanners install into a per-project virtual environment
> instead: `python3 -m venv .venv && source .venv/bin/activate` (Windows: `.venv\Scripts\activate`),
> then re-run the install. (`pipx` or `pip install --break-system-packages` also work; a venv is the
> clean default.)
> clean default.) Point your agent at this note if it gets stuck.
These are concrete, currently-maintained examples of the **SCA** and **secret-scanning**
categories not the only choices (see *Where it breaks* and *Verify-before-publish*). The lab
categories, not the only choices (see *Where it breaks* and *Verify-before-publish*). The lab
teaches the moves; the moves transfer to any tool in the category.
- Your AI assistant (browser or editor-integrated — by now you have Module 4 tooling; either is fine).
- Your coding agent (Claude Code is the worked example; sub your own).
### Part A — Let the AI introduce the problems
Copy this module's starter files into your project — they're a realistic snapshot of what an AI hands
you when you ask the `tasks-app` to "sync tasks to a cloud service":
Direct your agent (Claude Code is the worked example; sub your own) to place this module's starter
files: *"Copy `~/ai-workflow-course/modules/15-security-scanning/lab/config.py` and
`~/ai-workflow-course/modules/15-security-scanning/lab/requirements.txt` into
`~/ai-workflow-course/tasks-app`."* They're a realistic snapshot of what an AI hands you when you ask
the `tasks-app` to "sync tasks to a cloud service":
- `lab/config.py` → a new module the AI "wrote," complete with a **hardcoded API key**.
- `lab/requirements.txt` → the dependencies the AI "suggested," containing a **vulnerable real
- `config.py` → a new module the AI "wrote," complete with a **hardcoded API key**.
- `requirements.txt` → the dependencies the AI "suggested," containing a **vulnerable real
package**, a **typosquatted** name, and a **hallucinated** name that doesn't exist.
Open both and read them. They look completely normal that's the point. Nothing here would fail a
lint or a test.
Now open both and read them yourself. They look completely normal, and that's the point: nothing here
would fail a lint or a test. Reading what the agent dropped in, instead of trusting that it landed,
is the move the whole module trains.
If you'd rather generate them yourself, ask your AI: *"Add a module to tasks-app that syncs tasks to
a cloud API, and give me a requirements.txt for it."* You'll very likely get a hardcoded key and at
least one questionable dependency for free. Use the provided files if you want the lab to be
If you'd rather generate them instead, tell your agent: *"Add a module to tasks-app that syncs tasks
to a cloud API, and give me a requirements.txt for it."* You'll very likely get a hardcoded key and
at least one questionable dependency for free. Use the provided files if you want the lab to be
reproducible.
### Part B — Gate 1: SCA, and meeting a hallucinated package
Try to resolve the AI's dependencies:
From the repo, try to resolve the AI's dependencies. Running the scanner is the lesson, so you run it
by hand:
```bash
cd ~/ai-workflow-course/tasks-app
pip-audit -r requirements.txt
```
It fails before it can audit anything the resolver can't find one or more packages. **That's
slopsquatting's first tripwire.** Read the error: it names the package it couldn't resolve. Ask
yourself the dangerous question and answer it correctly: *is this a typo I should "fix," or a name
that should not exist?* Do **not** silently swap in the nearest real name that's exactly the
reflex the attack relies on. Confirm against the real project's home page which dependency was
It fails before it can audit anything: the resolver can't find one or more packages. **That's
slopsquatting's first tripwire.** Read the error; it names the package it couldn't resolve. Now make
the call this module is really about, and make it *yourself* — this is the human-in-the-loop judgment
no tool and no agent should make for you: *is this a typo I should "fix," or a name that should not
exist?* Do **not** let the agent (or your own reflex) swap in the nearest real name; that reflex is
exactly what the attack relies on. Confirm against the real project's home page which dependency was
actually intended.
Now edit `requirements.txt`: comment out the typosquatted and hallucinated lines (the ones flagged as
unresolvable), leaving the real-but-vulnerable package. Re-run:
Once you've decided, hand the mechanical edit to your agent: *"In requirements.txt, comment out the
two unresolvable lines, `reqeusts==2.31.0` and `task-cloud-sync-client==1.4.2`, and leave the rest."*
Then re-run the scanner yourself:
```bash
pip-audit -r requirements.txt
```
This time it resolves and reports a known vulnerability with an advisory ID and a fixed version. Bump
the pin to the fixed version and run it once more until it's clean. You've now exercised both halves
of SCA: the package that *shouldn't exist*, and the package that exists but *shouldn't be at that
version*.
This time it resolves and reports a known vulnerability with an advisory ID and a fixed version. You
decide the advisory applies and the fix is safe, then direct your agent to apply it: *"Bump requests
to the fixed version the advisory names in requirements.txt."* Run `pip-audit` once more until it's
clean. You've now exercised both halves of SCA: the package that *shouldn't exist*, and the package
that exists but *shouldn't be at that version*.
### Part C — Gate 2: secret scanning
Scan for the hardcoded key:
Scan for the hardcoded key yourself:
```bash
detect-secrets scan config.py
@@ -287,10 +298,12 @@ detect-secrets scan config.py
The JSON output lists a detected secret with its file, line, and detector type. That's your tripwire
firing on the AI's hardcoded key.
Now do it right: remove the literal from `config.py` and read the key from the environment instead
(`os.environ`), then re-scan and confirm the finding is gone. And say the quiet part out loud — **if
that key had been real and ever pushed, removing it now is not enough; you'd have to rotate it,**
because it's in history. (Proper secret management is Module 17; this is just the catch.)
Now do it right. Direct your agent to apply the fix: *"In config.py, remove the hardcoded
SYNC_API_KEY literal and read it from os.environ instead."* (The file carries the fixed version at
the bottom, commented out, so you can confirm the agent matched it.) Re-scan yourself and confirm the
finding is gone. And say the quiet part out loud: **if that key had been real and ever pushed,
removing it now is not enough; you'd have to rotate it,** because it's in history. (Proper secret
management is Module 17; this is just the catch.)
> **Stretch — Gate 3 (SAST):** install a static analyzer for your language (for Python,
> `pip install bandit`, then `bandit -r .`) and watch it flag insecure *code you wrote* — here, the
@@ -307,26 +320,28 @@ because it's in history. (Proper secret management is Module 17; this is just th
A scan you have to remember to run is a scan you'll skip. Move it into the Module 14 pipeline so it
runs on every push and blocks the merge.
1. Copy `lab/security-scan.sh` into your project. It runs the SCA and secret-scan gates and **exits
non-zero on any finding** — which is what makes CI go red. Make it executable
(`chmod +x security-scan.sh`).
1. Have your agent place the gate script and make it runnable: *"Copy
`~/ai-workflow-course/modules/15-security-scanning/lab/security-scan.sh` into
`~/ai-workflow-course/tasks-app` and make it executable."* The script runs the SCA and secret-scan
gates and **exits non-zero on any finding**, which is what makes CI go red. Verify the copy landed
and is executable (`ls -l security-scan.sh` shows the `x` bit) before you trust it.
Before you run it, **stage the starter files** so the secret gate can see them:
Before you run it, the starter files have to be **staged** so the secret gate can see them. Direct
your agent to stage them, *"Stage config.py and requirements.txt,"* then confirm with `git status`
that both show as staged.
```bash
git add config.py requirements.txt
```
This is not a footnote. `detect-secrets scan` with no path argument scans the files Git
*tracks* — an *untracked* `config.py` is invisible to it, so the gate would report "no secrets"
That staging step is not a footnote. `detect-secrets scan` with no path argument scans the files
Git *tracks*; an *untracked* `config.py` is invisible to it, so the gate would report "no secrets"
on a file that's full of them (a silent false pass, the worst kind). Staging puts the file in
front of the scanner. It's the same reason the explicit `detect-secrets scan config.py` in
Part C worked, and the same reason "secrets live in history": the moment Git knows about a file,
so does the gate.
so does the gate. Verifying with `git status` that the files are actually staged is the point, so
don't skip it.
To watch the gate catch both planted problems at once, restore the original booby-trapped files
first (you fixed them in Parts B and C) — re-copy `config.py` and `requirements.txt` from this
module's starter, re-stage, then run:
To watch the gate catch both planted problems at once, you need the original booby-trapped files
back (you fixed them in Parts B and C). Direct your agent: *"Re-copy config.py and requirements.txt
from `~/ai-workflow-course/modules/15-security-scanning/lab/` into the repo, overwriting my fixes,
and stage them again."* Then run the gate yourself:
```bash
./security-scan.sh
@@ -334,18 +349,26 @@ runs on every push and blocks the merge.
It should **fail on both gates** — the SCA gate on the unresolvable/vulnerable dependencies and
the secret gate on the hardcoded key — and you should be able to point at which finding caused
each non-zero exit. Re-apply your Part B/C fixes (and re-stage), run it once more, and it should
pass.
each non-zero exit. Direct your agent to re-apply your Part B/C fixes and re-stage, run the gate
once more yourself, and it should pass.
2. Merge the security steps into your pipeline. `lab/ci-security.yml` shows the gate as a
self-contained, provider-neutral job check out, set up Python, install the scanners, run the
self-contained, provider-neutral job: check out, set up Python, install the scanners, run the
script. But the `check` job you built in Module 14 *already* checks out the code and sets up
Python, so you don't want a second job duplicating that work. You want its two **new** steps
**install the scanners** and **run the gate** added to the steps you already have. (Checkout and
Python are in the snippet only so it reads as a complete example; skip them when you merge.)
Python, so you don't want a second job duplicating that work. You want its two **new** steps,
**install the scanners** and **run the gate**, added to the steps you already have. (Checkout and
Python are in the snippet only so it reads as a complete example; the agent should skip them when
it merges.)
Here is exactly where they go. **Before** — the tail of your Module 14 `check` job (GitHub Actions
flavor, matching `ci-starter.yml`; on GitLab the same two steps drop into the job's `script:`):
This is a careful edit to an indentation-sensitive file, so direct your agent and then check its
work against the spec below: *"In my CI workflow, append two steps to the existing `check` job
after the Test step: one that installs the pip-audit and detect-secrets scanners, and one that
runs `./security-scan.sh` (chmod it first). Don't add a second job, and don't touch the checkout
or Python steps."*
Here is exactly what the result should look like. **Before** — the tail of your Module 14 `check`
job (GitHub Actions flavor, matching `ci-starter.yml`; on GitLab the same two steps drop into the
job's `script:`):
```yaml
jobs:
@@ -381,17 +404,22 @@ runs on every push and blocks the merge.
+ ./security-scan.sh
```
> **YAML is indentation-sensitive match the existing steps' indentation exactly.** Each new
> `- name:` lines up in the *same column* as the steps above it, and the keys under it (`run:`) sit
> one level deeper. A step pasted even one space off will silently attach to the wrong block or
> fail to parse, and the whole workflow breaks. If you'd rather keep the gate as its own job (some
> teams prefer the isolation), copy `ci-security.yml` in whole as a second job under `jobs:` in the
> same workflow file instead that is exactly why it carries its own checkout and Python steps.
> The *shape* install tools, run the gate, fail on findings — is identical everywhere.
> **YAML is indentation-sensitive, so verify the agent matched the existing steps' indentation
> exactly.** Each new `- name:` should line up in the *same column* as the steps above it, and the
> keys under it (`run:`) sit one level deeper. A step placed even one space off will silently
> attach to the wrong block or fail to parse, and the whole workflow breaks. If you'd rather keep
> the gate as its own job (some teams prefer the isolation), have the agent copy `ci-security.yml`
> in whole as a second job under `jobs:` in the same workflow file instead; that is exactly why it
> carries its own checkout and Python steps. The *shape* (install tools, run the gate, fail on
> findings) is identical everywhere.
3. Prove the gate has teeth: re-introduce the hardcoded key in `config.py`, commit, and push. Watch
the pipeline go **red** on the security step even though lint, build, and tests are still green.
Remove it, push again, watch it go green. That red-then-green is the whole module in one push.
3. Now prove the gate works on a live push, and notice the angle: the AI itself commits the mistake,
and the gate catches it. Direct your agent to plant and ship the regression: *"Re-add the
hardcoded SYNC_API_KEY to config.py, then commit and push it."* Watch the pipeline go **red** on
the security step even though lint, build, and tests are still green: your own agent's change,
blocked by your own gate. Then direct it to undo and push again, *"Remove the hardcoded key again
and push,"* and watch the pipeline go green. The agent does the git; you verify each result on the
pipeline.
---
@@ -408,7 +436,7 @@ The honest limits — these gates are necessary, not sufficient:
scrubbing it from history is a separate, harder, recovery-grade job. Prevention (Module 17) beats
detection here.
- **False positives are real and they erode trust.** SAST especially will flag things that aren't
exploitable in your context. If every push has noise, people start ignoring red the worst
exploitable in your context. If every push has noise, people start ignoring red, the worst
outcome. Budget time to tune rulesets and triage findings, or the gate becomes decoration.
- **SCA depends on a manifest it can read.** If dependencies aren't declared in a file the scanner
understands (a pinned requirements/lock file, a package manifest), it can't see them. Vendored code,
@@ -454,7 +482,7 @@ reproducible.
check the Module 14 and Module 18 CI/CD checklists carry.
- [ ] **Scanner names and install methods.** Confirm `pip-audit`, `detect-secrets`, and `bandit` are
still maintained and still install as shown. If any has stalled, swap in a current equivalent
from the *same category* and keep the prose category-first, not tool-first.
from the *same category* and keep the writing category-first, not tool-first.
- [ ] **Category roster.** Verify the named alternatives still exist and are reasonable to recommend:
SCA (Trivy, Grype, OWASP Dependency-Check, Snyk, Safety, language-native `npm audit` etc.);
secret scanning (gitleaks, trufflehog, git-secrets, detect-secrets); SAST (Semgrep, CodeQL,
+4 -4
View File
@@ -1,9 +1,9 @@
"""Cloud-sync config for tasks-app — a realistic snapshot of what an AI hands you.
Asked to "sync tasks to a cloud service," a model will cheerfully produce something like this: it
works, it reads naturally, it passes lint and tests... and it carries two planted flaws a live
credential baked straight into the source (caught by Gate 2, secret scanning) and a weak-crypto
"signature" using MD5 (caught by Gate 3, SAST). Two different gates, two different blind spots.
Asked to "sync tasks to a cloud service," a model will produce something like this: it works, it
reads naturally, it passes lint and tests... and it carries two planted flaws: a live credential
baked straight into the source (caught by Gate 2, secret scanning) and a weak-crypto "signature"
using MD5 (caught by Gate 3, SAST). Two different gates, two different blind spots.
DO NOT copy these patterns. The point of this file is to be caught by a scanner, not imitated.
The fix (read from the environment) is shown at the bottom, commented out, so you can see the