#rlsbl.commands.watch

#rlsbl.commands.watch

Watch command that polls GitHub Actions CI workflow runs for a given commit SHA and reports pass, fail, or in-progress status.

#_release_at

def _release_at(commit_sha)

The release commit_sha is, from the release record, or None.

rlsbl watch takes a bare commit and runs from wherever it is invoked, so the release record is resolved from the PROJECT the cwd is in -- the same resolution every other command's project root goes through -- and not by joining .rlsbl/releases onto the process cwd. That relative path answered "nothing was released here" for every invocation from a subdirectory, and for every releasable member, whose archives live under the releasable rather than under the package.

Outside an rlsbl project there is no release record to read and no label to give: None. A directory whose release record holds no archive answers None too -- nothing was released here -- while a release record that CANNOT answer (a tag disagreeing with a release commit, an ancestry git cannot decide) raises, because a label derived from a release record rlsbl could not read would be a guess presented as a fact.

#_open_url

def _open_url(url)

Open a URL in the default browser. Non-fatal if unavailable.

#_release_url

def _release_url(repo_slug)

Try to find the latest release tag and return its GitHub URL. Returns None on failure.

#_notify

def _notify(title, body, url=None)

Send a desktop notification. If url is provided, opens it only when the user clicks the notification action.

#_classify_failure

def _classify_failure(log_text)

Classify a CI failure log tail to decide whether a retry is worthwhile.

Returns one of: - "infra": the run died at the infrastructure layer, below the code under test (no runner acquired, GitHub's action download service failed, or nothing executed at all). Rerun its FAILED jobs once: the run established nothing, and on a resumed release it is the only run the candidate will ever have. - "deterministic": a signature indicating the failure recurs identically on retry (test failures, compile/build errors, config/validation errors, workflow syntax errors, missing-secret/auth denials). Never retry. - "transient": an infrastructure-flake signature (network timeouts, 5xx, rate limits, runner-lost/cancelled). Retry once. - "unknown": no signature matched. Treated by the caller as transient (retry once) -- this DEFAULT preserves the historical blind-retry behavior for failures we don't yet recognize, rather than suppressing a retry that might have succeeded.

Precedence is infra, then deterministic, then transient. Infra comes first because a run that never executed still emits a log tail full of job names, echoed commands and workflow text, which the deterministic signatures match by accident -- that is precisely how a provider-wide outage was read as a code failure and left a resumed release permanently unrunnable. Determin- istic then outranks transient so a log holding both a hard error and incidental network chatter follows the hard error, which is the real cause.

An EMPTY tail is infra, not unknown: a failed run whose jobs produced no log output at all -- or which has no failing job to read a log from -- died before execution, so nothing about the code was established. Runner never acquired, actions never resolved, or the run was cancelled while queued.

#_failure_region

def _failure_region(lines)

The part of one job's log worth classifying.

A failed job's log ends in its post-steps and cleanup, so a blind tail of a long log holds runner housekeeping and not the failure. Actions marks every failure with an ##[error] line, so those lines -- with a few lines of context each -- are the region. A job that emitted no marker at all (killed before it could) has no region to prefer, and its own tail is taken instead.

#_fetch_failure_log

def _fetch_failure_log(run_id, config=None)

Fetch the failing jobs' logs for a run, as one classifiable string.

Reads the run's jobs through the attempt-scoped endpoint (:func:rlsbl.ci_checks.fetch_run_jobs) -- the same single endpoint the release gate uses -- and then each failing job's own log (/actions/jobs/<id>/logs). Both are keyed by ids rlsbl already holds, so neither depends on the repo-level Actions collections that gh run view walks and that 404 on some repositories, taking the whole failure classification with them.

Every failing job is named in the returned text, so the operator reading a fifty-job router run sees WHICH jobs failed rather than only the workflow. At most _LOG_FETCH_MAX_JOBS of them are fetched -- enough to classify, bounded so a mass failure cannot stall the watch. Propagates any exception from the gh calls so the caller can emit a loud note.

#_run_state

def _run_state(run_id, config=None)

The run's own status and conclusion as GitHub reports them, or None.

None means the question could not be answered -- gh failed, the payload was unreadable, or the run object carried no status. It never means "the run is fine" or "the run failed": a caller that cannot read the state has no verdict to report.

Read through gh api, keyed by the run id the caller already holds, so it does not depend on the repo-level Actions collections that 404 on some repositories.

#_watch_to_conclusion

def _watch_to_conclusion(run_id, workflow_name, label, timeout, config=None)

Block until run_id concludes, and report only what was established.

Returns one of :data:WATCH_PASSED, :data:WATCH_FAILED, :data:WATCH_TIMEOUT (the caller's budget ran out with the run still going) or :data:WATCH_UNRESOLVED (the watch ended early and the run's state could not be read, so nothing was established at all).

:data:WATCH_FAILED is returned ONLY for a run GitHub reports as completed with a non-success conclusion. A non-zero gh run watch exit over a run that is still queued or in progress means gh dropped the watch, not that the run failed: the watch is resumed instead, within the same budget. A state that cannot be read at all resolves nothing and is reported as such -- guessing "failed" there would fail-forward a release on the strength of a broken API call, and would fire gh run rerun at a run that is still running (which GitHub refuses anyway).

The budget is measured on the monotonic clock, so a system clock adjustment mid-wait cannot shrink or extend it.

#_unresolved_result

def _unresolved_result(workflow_name, run_id, label, outcome, timeout)

The result record for a run that reached no conclusion.

Carries timed_out so :func:_timeout_verdict reports :data:CI_TIMEOUT rather than :data:CI_RED: the runs may still be in flight, and the remedy is to check them and resume, never to fix code that may be perfectly fine.

#_retry_workflow

def _retry_workflow(workflow_name, repo_slug, label, failed_run_id, failed_only=False)

Re-run a failed workflow run in place once and watch the new attempt.

Uses gh run rerun <failed_run_id> on the SAME run id. GitHub re-executes the run as a new attempt on the failed run's original commit, so (a) no duplicate check-run is created (a fresh gh workflow run dispatch would poison the publish gate) and (b) the retry runs on the failed commit rather than branch HEAD.

failed_only adds --failed, restarting only the jobs that failed. That is the right shape for an infrastructure-killed run: the jobs that DID acquire a runner and pass keep their result instead of being thrown back into a queue that just proved unreliable. A transient flake keeps the full rerun -- nothing there says which jobs the flake really touched.

Because the run id is unchanged, there is no dispatched-run-identification dance: we simply watch failed_run_id for its new attempt's conclusion.

Returns a result dict with name, passed, and run_id. Returns None if the rerun could not be triggered.

#_watch_single_run

def _watch_single_run(ci_run, label, repo_slug, retried_lock=None, retried_workflows=None, timeout=3600)

Watch a single CI run. Returns a dict with name, passed, and run_id.

When retried_lock and retried_workflows are provided, deduplicates retries so that only one retry is dispatched per workflow name even when multiple runs from the same workflow fail concurrently.

Retries are in-place reruns (gh run rerun) that reuse the failed run's own id, so no cross-thread run-id bookkeeping is needed: the late re-poll recognizes the reran run by its unchanged id.

#_watch_runs

def _watch_runs(runs, label, repo_slug, retried_lock=None, retried_workflows=None, timeout=3600)

Watch all runs in parallel. Returns list of result dicts.

retried_lock and retried_workflows may be passed in so that retry deduplication state is shared across multiple _watch_runs calls (initial watch + late re-poll watch). Fresh state is created when omitted.

Single-run pools deliberately go through the same thread-pool path so every run participates in the shared retry-dedup machinery.

#_repo_root

def _repo_root()

Best-effort git toplevel; falls back to cwd when not in a git repo.

#_has_publish_workflow_on_disk

def _has_publish_workflow_on_disk()

Check if any .github/workflows file looks like a publish workflow.

Resolved from the git repo root (not cwd) so monorepo package-dir invocations still find the repo-level workflow files.

#_is_publish_workflow

def _is_publish_workflow(name)

Return True if the workflow name matches a publish/deploy/release pattern.

#_print_workflow_audit

def _print_workflow_audit(results)

Print a summary of which workflows ran and flag missing publish workflows.

Returns True if a missing-publish warning was printed (for testability).

#_resolve_run_ids

def _resolve_run_ids(run_ids)

Resolve run IDs to run info dicts via gh run view.

#poll_runs

def poll_runs(commit_sha, max_attempts=30, interval=4)

Poll gh run list until at least one run appears.

Returns a list of run dicts (may be empty if nothing found after all attempts). Default timeout is ~120s (30 attempts * 4s interval).

#_discovery_budget

def _discovery_budget(timeout, discovery_grace)

Clamp the discovery grace so it never eats the whole CI budget.

Returns (effective_grace, clamped). At least one poll interval is always granted -- a budget too small to poll even once is the operator's declaration, not a reason to skip discovery entirely.

#_timeout_verdict

def _timeout_verdict(results)

Aggregate per-run results into a verdict.

A genuine failure outranks a timeout: if any run definitively failed, the answer is known (:data:CI_RED) and fix-forward is the right remedy, even if a sibling run was still going when the budget ran out.

#CIWaitError

Raised when the CI wait cannot reach a verdict at all.

Distinct from a red verdict: this means the repository declares push-triggered CI but the pushed candidate produced no runs, so there is nothing to gate on and proceeding would publish an unverified commit.

#_workflow_triggers_on_push

def _workflow_triggers_on_push(path)

Return True if a workflow file declares a push trigger.

Parsed rather than grepped so a push mentioned in a job step or a comment is not mistaken for a trigger. Unparseable files are treated as NOT push-triggered -- an unreadable workflow cannot be evidence that CI is expected.

#push_triggered_workflows

def push_triggered_workflows(repo_root=None)

Return the names of .github/workflows files that trigger on push.

An empty result means the repository has no push-triggered CI: the release flow then proceeds without a CI gate instead of blocking forever on runs that can never appear. The distinction is an observable fact about the repository, not a fallback.

#RunAllDispatchError

The router's run_all dispatch could not be made or correlated.

Fail-closed: the release refuses rather than entering a CI gate that would read a run nobody established, or wait out its whole budget on runs that were never created.

#router_workflow_path

def router_workflow_path(workspace_root)

The generated CI router's path under workspace_root, or None.

#dispatch_run_all

def dispatch_run_all(branch, commit_sha, *, config=None, log=None, attempts=None, interval=None)

Dispatch the CI router at branch with run_all=true, for commit_sha.

The router filters every project's job on the paths a PUSH touched, so an honestly narrow fix-forward leaves most members' jobs skipped -- a conclusion both the release gate and the publish gate refuse. Dispatching the router on the same commit with the filter short-circuited runs every member's real CI jobs; the dispatched run's conclusions supersede the skipped ones per name (:func:rlsbl.ci_checks.latest_check_runs), whichever suite GitHub stamped first. Nothing is waived: a job that fails in the dispatched run still blocks the release.

The dispatch names a REF, not a commit, so the run it creates is correlated back to commit_sha by head SHA before this returns. A run for any other commit -- something pushed to the branch between the release's own push and the dispatch -- proves nothing about the candidate and is a hard error.

Returns the correlated run dict. Raises :class:RunAllDispatchError.

#wait_for_ci_green

def wait_for_ci_green(commit_sha, *, timeout, check_filters, log=None, config=None, repo_root=None, label=None, discovery_grace=CI_DISCOVERY_GRACE_SECONDS)

Block until every CI run for commit_sha concludes.

Returns (verdict, results) where verdict is one of :data:CI_GREEN, :data:CI_RED, :data:CI_TIMEOUT, or :data:CI_NOT_CONFIGURED.

Five explicit outcomes, no silent waits:

• The repository declares no push-triggered workflow -> :data:CI_NOT_CONFIGURED

immediately (nothing can ever run; blocking would hang the release).

• Push-triggered workflows exist but no run appears for the commit within

the discovery grace -> :class:CIWaitError (hard error).

• Runs appear and conclude -> :data:CI_GREEN / :data:CI_RED (transient

failures retried once, deterministic ones not). Only a run GitHub reports as completed with a non-success conclusion is red -- see :func:_watch_to_conclusion.

• Runs appear but timeout expires with some still unresolved, or a watch

ends early over a run whose state cannot be read -> :data:CI_TIMEOUT. Distinct from red on purpose: nothing was proven about those runs, so the remedy is to check their status, not to fix code that may be perfectly fine.

• Every run concludes green, but the RELEASING PROJECT'S OWN check runs

were absent or did not conclude success (typically skipped by the monorepo CI router's paths filter) -> :class:rlsbl.ci_checks.ProjectCINotRunError (hard error). A green workflow run is not evidence that this project's CI ran: this is the exact predicate the publish gate applies later, checked here so the two gates cannot disagree and tag a version that can never publish.

check_filters is mandatory (a list of :class:rlsbl.ci_checks.CheckFilter, from :func:rlsbl.ci_checks.release_check_filters) precisely because it must never be forgotten: a caller that omitted it would silently re-open the divergence. Pass an empty list only when there is no project to verify.

timeout is the WHOLE budget: discovery is spent inside it and is capped at half of it, so the completion wait always keeps at least half.

#run_cmd

def run_cmd(registry, args, flags)

Watch all CI runs for a commit until they complete.

Usage: rlsbl watch [] rlsbl watch --run-id [--run-id ] Defaults to HEAD if no commit SHA is provided.