The artifact exists.
PR Quiz Gate is a third-party GitHub Action. It generates questions from pull-request evidence, accepts answers from the pull-request author, and reports an advisory or required status. Multiple-choice answers are checked directly. A model grades free-text answers. The current Marketplace listing is version 0.1.0 and is not certified by GitHub.[1]
We cloned commit feb9559978eb991794046b4afcfd52ff3aa729d5, installed its locked Python environment, and ran its test command. All 37 repository tests passed. They cover stale revisions, answer ownership, encrypted state, unresolved decisions, overrides, provider tool refusal, context limits, and installer behavior. The tests mock providers and GitHub APIs. We did not run the Action on a live pull request.
A comprehension check measures an answer to a question. It does not measure the whole patch.
The useful part is the reset.
The gate ties a quiz to the pull-request revision and author. A new head commit resets a passed quiz. Unresolved decisions can still block a perfect score. Those rules matter because an explanation can go stale the moment the patch changes.[1]
The tool also admits its limits. The provider cannot run code or use tools. Binary files and oversized changes can stop generation. Unchanged dependencies and review threads are outside the prompt. Grading can be wrong, and an author can ask another model to answer. Merge queues are unsupported. That is a clear boundary, not a footnote to hide.
Put the gate in the right lane.
GitHub rulesets can require status checks and combine several protections. When rules overlap, GitHub applies all of them and uses the most restrictive form of a duplicated rule.[2] That makes a comprehension status one lane in a release route. It does not need to impersonate unit tests, security analysis, or human approval.
Start in advisory mode. Use it on unfamiliar or agent-heavy changes where explanation debt is visible. Read the generated questions. Reject trivia. Check whether the questions point to decisions, failure behavior, and maintenance consequences. Promote the status to required only after the team has evidence that it catches useful gaps without turning review into theater.
Treat the Action as code with keys.
The Action receives repository content, pull-request comments, and a provider credential. GitHub warns that a compromised third-party Action can reach secrets and repository permissions. GitHub recommends auditing the source and pinning third-party Actions to a full commit SHA.[3] The project's own setup guide also tells users to pin a tested SHA.
The repository had one commit and no license file when we inspected it. The Marketplace page gives an install route and the README tells people how to use it, but no license text was present in the cloned tree. Review that gap before adopting or modifying the code. Keep workflow permissions narrow. Add a code owner for workflow changes. Record the exact Action revision in the rollout sheet.
Run the brake test before enforcement.
A useful trial needs bad cases, not only a clean demo. Change the pull-request head after a pass and confirm the status resets. Submit an answer from the wrong account. Leave a decision unresolved. Exceed the configured file limit. Rotate the provider token while using a stable state key. Confirm that each case fails or resets in the way your branch rule expects.
ROLL OUT IN THIS ORDER 1. Pin one reviewed commit SHA. 2. Run advisory mode on a narrow repository. 3. Save useful and useless questions. 4. Replay stale-head and wrong-author cases. 5. Keep tests and human approval as separate checks. 6. Name the person who can override, with a reason. 7. Promote only after the trial has a written result.