Leaderboard workflow: open a PR instead of pushing to protected main (#45)
The eval run worked (12 scored runs) but the final step failed: it pushed
evals/results.json directly to main, which the branch ruleset blocks
("Changes must be made through a pull request").
- eval-leaderboard.yml: replace the direct commit/push with
peter-evans/create-pull-request@v7 (branch eval-results), add
pull-requests: write. Merging that PR triggers the Pages deploy (which
watches evals/results.json) to publish real numbers.
- evals/README documents the PR flow + the required "Allow GitHub Actions to
create and approve pull requests" setting.
Claude-Session: https://claude.ai/code/session_016JWn5jRD5tcEFKrubjQ6Px
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,8 @@ each new wave of skills bumps the **major** version, extensions and fixes bump
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
- **Leaderboard workflow opens a PR** instead of pushing to `main` (which the branch
|
||||
ruleset blocks). After it runs, merge the auto-created results PR to publish real numbers.
|
||||
- **Faster, hang-proof evals.** The Anthropic client now has a per-request timeout (120s)
|
||||
and limited retries (429/5xx/timeout); the eval harness runs cases concurrently
|
||||
(default 4). The leaderboard workflow has a 20-minute job timeout. A 24-call run that
|
||||
|
||||
Reference in New Issue
Block a user