Repository navigation
design+impl: org-level GitHub Project as the conceptual-project layer — orchestrator-synced, dashboard-read #910
Description
design+impl: org-level GitHub Project as the conceptual-project layer — orchestrator-synced, dashboard-read
Drew's questions + direction (2026-10-25), building on the 2026-10-23/24 evaluation (recommended GitHub Projects: org-level, Board + Roadmap views over one item pool, label-gated auto-add, fleet-maintainable via gh/GraphQL, $0 incremental; write-up ~/workspace/your_files/github-projects-vs-linear-2026-10-23.md).
Why
Issues are the unit of work, but larger conceptual efforts (a program, a wave, a multi-repo set) have no first-class URL of their own today — they live in goal workspaces and epic-ish issues. An org-level Project gives each effort a stable place + URL, and the dashboard a canonical source for status that is not re-derived from scratch every hour.
Design
- One org Project now (developing-today), designed to split later without migration: an Area field carries the conceptual project (e.g.
id-train,pm-dashboard,lab-cluster). Splitting = filtered views per Area now; separate Projects per Area later is a view change, not a data move. The dashboard reads the same pool and can show all Areas together or filter to one. - Fields: Status (Todo / In Progress / In Review / Done), Area, Horizon (Today / Week / Month / Quarter — mirrors the dashboard horizons), Type (Task / Epic).
- Items are issues. Pre-issue ideas may sit as draft items briefly but should become issues.
- Epics = meta-issues: an issue with a task list of member issues + the
epiclabel, itself a Project item with Type=Epic. Milestones connect clusters inside one repo (a wave, a release); the Project connects across repos. Both mechanisms coexist: milestone = time-boxed cluster in a repo, epic = cross-cutting grouping, Project = the whole portfolio. - Sync is single-writer. Agents NEVER write the Project directly — they keep working in GitHub issues only (comments, branches, PRs that reference the issue, assignment). The orchestrator reconciles the Project on its hourly pass from observable signals: issue added (label-gated auto-add:
now/next/epic, or manual add) -> Todo; a branch or open PR referencing the issue, or an active session lined up on it -> In Progress; PR in review -> In Review; issue closed -> Done; blocked-by carried from issue text + the orchestrator's classification (same blockedBy data the dashboard already publishes). One writer = no sync races, no agent-side knowledge required. - Dashboard reads, never writes: the fleet-feed collector gains a projects section (GraphQL Projects v2 read), the aggregator renders Project views from it alongside the derived data; where the Project and derived state disagree, the disagreement is shown, not hidden (it is the reconciliation bug signal).
- Known caveat to verify at build time: org Projects and personal-repo items (dezren39/nix) — the 2026-10-24 research said an org Project can't hold personal-repo items; verify current behavior, and if still true, nix items stay dashboard-only (the feed already covers them) and that limitation is documented on the Project.
Staged implementation
- Drew's go (org-level creation is his call).
- Create the Project + fields + Board and Roadmap views; README on the Project explaining the conventions in one screen.
- Label-gated auto-add workflow + the epic/milestone conventions written into the four repos' contributor docs where the fleet reads them.
- Orchestrator: add the reconciliation step to the hourly pass (bounded: at most ~25 item updates per run, mirroring its classification cap).
- Collector + aggregator: projects section in the feed; dashboard project page (this is a slice-3-manifest page — coordinate with impl(dashboard): slice 3 — pluggable pages (DNS register first), comments everywhere, agent/session identity registry #906).
- Backfill: current epics/programs (ID train, PM dashboard slices, lab cluster set) become items with Areas set.
Acceptance
- Project exists with the fields/views above; a new labeled issue appears as a Todo item without human touching the Project.
- Opening a PR that references a tracked issue moves its item to In Progress on the next orchestrator pass; merging/closing moves it to Done.
- Dashboard shows the Project pool with an Area filter; an item's dashboard card links the Project item URL.
- A blocked item shows its blocker on both the Project and the dashboard, from one recorded source.
Acceptance criteria
- An org-level GitHub Project for developing-today exists with Status / Area / Horizon / Type fields and Board + Roadmap views over one item pool.
- Label-gated auto-add (
now/next/epic) is configured; epics are meta-issues carrying theepiclabel and Type=Epic. - The orchestrator's hourly pass is the single writer: it syncs item Status from issue/PR/session signals (Todo → In Progress → In Review → Done) and carries blocked-by from its existing classification.
- No agent writes the Project directly; the dashboard's fleet-feed collector reads the Project via GraphQL (Projects v2) and renders a projects section without writing back.
- A spot-check of several known issues shows Project Status matching their GitHub state after one orchestrator pass.
Activity
developing-today commentedon Oct 25, 2026
CODE-150
dezren39 commentedon Oct 25, 2026
Build report 2026-10-25 — stages 2, 3, 6(backfill): SPEC READY, creation BLOCKED on one auth step only Drew can do
Blocker (verified four ways): creating an org Project needs a token with the OAuth
projectscope. None of the fleet's existing GitHub credentials has it:ghCLI token (account dezren39): scopesadmin:org, gist, repo, user, workflow—gh project listanswers "missing required scopes [read:project]".github-cli.hosts-yml): scopesadmin:org, gist, repo, workflow— same gap.~/pm-stack/github-app-manifest.jsonon the laptop): repository permissions only (contents:read, issues:write, metadata:read, pull_requests:read) — no organization Projects permission, and adding one would itself need an org-owner UI approval.Unblock (one-time, ~1 minute, either option):
gh auth refresh -s project --hostname github.comand approve the consent prompt — the fleet's existingghthen does everything below with no further steps; orgithub.project-token.Everything else is prepared so creation is mechanical the moment either lands.
Prepared spec (stage 2)
developing-today, titleDeveloping Today, short description: "Org-level portfolio: every program, epic, and active task across the fleet's repos." Created viagh project create; remains private to the org.gh project field-create/ GraphQL):updateProjectV2SingleSelectField): Todo / In Progress / In Review / Done.id-train,pm-dashboard,lab-cluster,fleet-ops. Small on purpose: Area is the split dimension — a future split is a saved view per Area, not a migration. New Areas get added only when a program outgrows these four.ghsupport for creating project views, so the Board (grouped by Status) and Roadmap views are a one-time ~2-minute UI step at creation (or the default table view stands until then). Recorded here so nobody burns time re-discovering it.Nix caveat — VERIFIED (docs), supersedes the 2026-10-24 research
GitHub's current docs ("Adding items to your project") state a project can include issues and pull requests from any organization, and items are added by URL/search from any repository the adder can access — org projects are not restricted to org-owned repos. So the org Project can hold
dezren39/nixitems; the lab-cluster backfill below includes them. (One residual check at creation time: the firstaddProjectV2ItemByIdof a nix issue is the empirical proof — it is step 1 of the backfill run. Built-in auto-add remains per-repository and is not used; see stage 3.)Stage 3 — label gate DONE; auto-add realized as the orchestrator-side rule
now(d93f0b, "Org Project gate: active priority"),next(fbca04, "queued next"),epic(5319e7, "Meta-issue grouping multiple items") in developing-today/code, developing-today/opencode, developing-today/openchamber, dezren39/nix. No open issue carries them yet — the gate starts empty and correct.label:filters supported) and cannot be created via API/gh. Per the fallback in this issue's plan, auto-add is therefore an orchestrator-side rule, specified here for stage 4 to implement verbatim: on each pass, for each of the four repos, add every open issue labelednow,next, orepicthat is not already a project item, with Status=Todo and Type=Epic iff labeledepic; never remove items automatically. (Optional later: Drew enables the built-in UI workflows as a belt-and-suspenders duplicate — harmless because adds are idempotent.)Backfill manifest (stage 6) — 27 items, all verified open/closed 2026-10-25, ready to add
Counts per Area: id-train 4, pm-dashboard 6, lab-cluster 8, fleet-ops 9. Horizon is left unset at backfill except #906 (Week); the orchestrator's measurement fills horizons in.
Named next actions (unchanged ownership, not done here)
Project URL: none yet — this comment is the record until the auth unblock lands; the creation run appends the URL here and to the daily log.
dezren39 commentedon Oct 26, 2026
Intake review
Fleet monitor intake pass (2026-10-25 run). Acceptance criteria appended to the body this pass (original body preserved verbatim above it).
Related: #911 (the fleet work-picking jobs consulting the orchestrator's published state — enriched last monitor run; the Project described here is the conceptual layer above that same orchestrator), #902 (another standing-registry/dashboard build from the same 2026-10-25 direction batch).
Notes: This is a design+implementation item, not a bug: nothing to reproduce. The single-writer rule (orchestrator syncs; agents and the dashboard never write the Project) is the load-bearing design decision — it keeps the protocol identical to how agents already work issues today. Label-gated auto-add (
now/next/epic) means adoption is incremental: existing issues join the pool by label, no bulk migration needed.Risks: GitHub Projects v2 GraphQL field/option IDs are brittle across project edits — the sync should resolve fields by name each pass. Horizon/Status drift between the Project and the dashboard's own horizons is the main consistency risk; the design already mirrors the dashboard horizons deliberately.
Suggested approach: Stand up the Project + fields first (manually, once), then the collector read (dashboard projects section), then the orchestrator sync last — each step is independently verifiable, and the read path proves the field model before the writer depends on it. Effort classed l (multi-part build).
dezren39 commentedon Oct 26, 2026
Creation run 2026-10-25 — BUILT ✅
Project: Developing Today — https://github.com/orgs/developing-today/projects/4 (org
developing-today, #4, private, idPVT_kwDOAZNvF84Bouay)Built with the Bitwarden-filed fine-grained token (
github.project-token, Projects Read+write only, expires 2027-10-26). Verified end-to-end via API re-list after the build.Existing project #1 — inspected, left untouched (different design, not a partial build)
Project #1 "code" (public) uses a lifecycle Status scheme (CREATED / DESIGNED / READY / BLOCKED / ACTIVE / STALLED / DONE / CANCELLED), has no Area/Horizon/Type fields, an empty README, and a single draft item ("dns alias in tf provider"). That is an older, different tracker — not an earlier cut of this spec — so the designed project was created fresh as #4 rather than bending #1 into a different shape. #1 is unchanged.
What was built (per the spec in comment 6404216307)
createProjectV2Field: "Name cannot have a reserved value"). Same options, same meaning; README/stage-4 references to "Type=Epic" map to Kind=Epic.dezren39/nixadd (nix#508) succeeded with the Projects-only token, which is the empirical proof for the nix caveat: cross-account items work as the current docs state.Remainders (named, unchanged ownership)
ghsupport for project views; the laptop Helium CDP endpoint is live, but driving the Projects UI automation to build and configure both views is not a quick operation, so per the run's stop rule they remain a one-time ~2-minute manual UI step (Board grouped by Status; Roadmap by Horizon/dates). The default table view shows all 27 items with every field until then.Stages 2, 3, and 6 of this issue are now complete; the issue stays open only as the tracker for stages 4–5 and the views remainder.
Merge pull request #991 from developing-today/feat/dashboard-project-…
dezren39 commentedon Oct 26, 2026
Stage 4 — FIRST reconciliation pass run 2026-10-26 ✅
The reconciliation machinery is built and its first pass is complete. The Project is current against GitHub ground truth as of this run.
Helper:
project-reconcile.sh(bash) atworkspace/goals/keep-the-four-repos-healthy/hidden_files/project-reconcile.shin hatch-personal (committed).--dry-runprints proposals;--applyperforms at most 25 Status updates per run, closed→Done first. Design choices per this issue's spec: Status field + option IDs are resolved by name on every run (IDs are brittle across project edits); Project reads/writes use the Bitwardengithub.project-token(Projects-only) passed as GH_TOKEN per-command; all issue/PR/branch signals use the normal fleetghauth. Signals per item: issue state + open PRs referencing it (issue-timeline cross-references ∪ PR closing-issue references ∪ title/body#Nmentions ∪ PR head-branch naming) + branch-name matches, mapped closed → Done, open non-draft PR → In Review, draft PR or branch → In Progress, else Todo.First pass results — 27 items reviewed, 6 transitioned, 21 unchanged, 0 failed:
fix-issue-811-…).Acceptance spot-check (fresh reads both sides, all match):
Auto-add discovery (report-only this run; the hourly step applies it): gate labels
now/next/epicverified present in all four repos. Labeled open issues not yet Project items: code 0, opencode 0, openchamber 0, dezren39/nix 1 — nix#672 (carriesepic; would join as Status=Todo, Kind=Epic).Remainders (unchanged ownership): the hourly orchestration-pass body step that calls this helper is being added separately (dashboard-fix lane — this run deliberately touched no cron body); Board + Roadmap views are being created separately; Stage 5 (collector feed projects section + dashboard rendering) unchanged. A full pass takes ~2.5 min of
ghcalls, comfortable at hourly cadence. This issue stays open as the tracker for stages 4-hourly/5 and the views.developing-today commentedon Nov 5, 2026
Stage 4 — reconciliation wired onto the gh-post ladder; deferred Project #4 reconciliation computed in full, landing via the ladder (2026-11-05)
Ladder (live in
project-reconcile.sh+project-sync-apply.sh, protocol infiles/action-outbox-protocol-2026-10-28.md, kind documented in the action-outbox-processor cron):github.project-token) — live writes.project-sync, idempotent set-to-value ops with acomputed_attimestamp) for the processor to drain viaproject-sync-apply.sh, whose state guards make re-execution safe (current == to → NOOP; current != from → SUPERSEDED, never overwritten). Single-writer semantics preserved: the orchestration passes remain the only reconciliation writers; the outbox only executes what a pass computed.Ground truth established before applying anything (the previous lane died mid-run at ~09:44 CST): the interrupted apply run landed zero writes (it died in the proposal phase; no outbox entry existed). Independent readback of Project #4: 374 items (Done 115 / Todo 117 / In Review 13 / In Progress 11 / UNSET 118).
Root cause of the "deferred" backlog:
ITEM_LIMIT=200silently truncatedgh project item-listat exactly the project's old size. The project grew to 374 items and 174 of them — 118 with no Status at all — were invisible to every hourly pass, which kept reporting "0 changes" against the truncated view. Fixed: limit raised to 1000. Two more latent defects found and fixed the same day:> 50; App installation tokens arrive at 40 or 383 chars depending on mint, so the leg could silently never arm. Replaced with aghs_prefix + sanity-length check.fetch_issue_signalsnow chunks at 40 issues per query (any slice failure keeps the legacy fallback semantics).Deferred reconciliation, fully computed (two independent full computations agree): 161 status changes + 43 label-gated adds owed (40 code, 3 darwin-nix; the candidate count dropped from 45 to 43 once the full item list was visible — code#1233/#1234 were already items past the truncation horizon and would have been double-added by the old view).
Proof the ladder lands writes: a forced-limit run (
RECONCILE_FORCE_LIMIT=1, the analogue of gh-post's skip hooks) queued 25 writes → outbox entry OB-2026-11-05-003 → drained byproject-sync-apply.sh: 2 adds landed live in Project #4 before a GitHub-side error burst (GraphQL 502 / "something went wrong", 16:12:44Z) and then the user GraphQL bucket limit stopped the drain at the designed exit-4 with 23 writes still queued inhidden_files/project-sync-20261105-161214.jsonfor the processor.The one Drew action — App grant: probed three times independently: the App token mints and reads issues fine, but
organization.projectV2does not resolve under the App identity (FORBIDDEN/NOT_FOUND) — the developing.today App (5454422) holds no org Projects permission, so ladder leg 2 cannot arm. Exact fix: App settings → Organization permissions → Projects: Read & write, then approve the permission change on the developing-today org installation. The probes in both scripts pick the grant up automatically when it lands — no code change needed.Remaining landing: as of this comment the shared user GraphQL bucket (user ID 11225574) is exhausted at the executor level — even
{viewer{login}}returnsgraphql_rate_limitunder both the project token and the normal user token, while the RESTrate_limitendpoint misleadingly reports ~4,430 GraphQL points remaining (executor verdict is ground truth). Project writes are all GraphQL, so the 159 remaining status sets + 41 adds land via the queued outbox entry + the hourly capped passes as the bucket refills — that convergence is the design working as intended, not a stall. Filed from the Stage-4 lane; dashboard/projector readers will see statuses fill in pass by pass.dezren39 commentedon Nov 5, 2026
App Projects-permission request — blocked at sudo mode (2026-11-05 ~10:52 CST). Acting on Drew's directive ("you do it… I'll handle github mobile app approval"), a lane opened the developing.today App settings in the laptop Helium session: the session is live as @dezren39 and the GitHub Apps list loads fine, but the app settings page itself is gated behind sudo mode. The Confirm-access page offers ONLY passkey/security key, GitHub Mobile, authenticator app, or an emailed code — no password path exists. Per the standing rule that 2FA/passkey factors are Drew's alone, the lane stopped before the permission form; nothing was changed.
State left ready: the Helium tab is parked on the sudo Confirm-access page. Once Drew completes that one factor there (passkey, or "Use GitHub Mobile"), the Projects: Read & write permission change can be saved immediately after; his GitHub Mobile installation approval remains his step as planned. Follow-up once both land: re-probe
organization.projectV2under the App identity.dezren39 commentedon Nov 5, 2026
App Projects permission: SAVED at the App level — installation acceptance still pending (follow-up run, 2026-11-05 ~10:52–11:00 CST).
What was done: after Drew approved the GitHub Mobile request, the sudo elevation carried on the laptop Helium session. One correction to the earlier record: the app is ORG-owned, so its settings live at
github.com/organizations/developing-today/settings/apps/developing-today— the user-levelsettings/apps/developing-todayURL 404s (that 404 was the parked tab's state when this run started; the session itself was live as @dezren39). On the org app's Permissions page: Organization permissions → Projects → Read & write, saved; GitHub flashed "Your GitHub App has been updated."Verified twice, independently:
GET /appunder the App JWT reports requestedorganization_projects: "write".NOT yet armed — the remaining step is Drew's, and it is a different approval than the one he just gave (that one was the sudo elevation):
GET /app/installationsshows the developing-today org installation (174792676) with no organization_projects grant, and a GraphQL probe under a freshly minted App installation token still returns NOT_FOUND fororganization { projectV2(number: 4) }("Could not resolve to a ProjectV2 with the number 4"). The org installation must review/accept the App's new permissions (org GitHub Apps settings shows a review request; it also surfaces via GitHub Mobile/notifications). Once accepted, the ladder scripts' probes pick the grant up automatically — no further code or config change needed.dezren39 commentedon Nov 5, 2026
App permission batch — COMPLETE for both apps (follow-up to the comment above)
Drew's scope extension (2026-11-05 10:54–10:55 CST): approve the privilege on each org, make the same change on the fleet GitHub App, and batch every likely-needed permission into one save per app. All done and verified 2026-11-05 ~11:05–11:45 CST. Method: laptop Helium owner session (sudo via Drew's mobile approval), each save verified on a fresh page reload, each approval verified by re-reading the installation page, and each app probed with real App tokens afterward.
App 1 — developing.today (5454422, owned by the developing-today org)
Note: this app is ORG-owned — its settings live at
organizations/developing-today/settings/apps/developing-today(the user-level settings URL 404s; that was the earlier lane's dead end, not a sudo park).In-scope permission table (everything not listed was and remains No access; excluded classes — org Administration, Members write, repo Administration, Secrets/Security grants — were NOT added):
Section headers after: Repository 7 selected + 1 mandatory; Organization 1 selected (was 0).
Installation approvals (each review listed exactly the three new permissions; accepted in the owner session; verified on the page afterward — no pending banner, grants text includes organization projects):
Probe (App JWT + installation tokens via infra/fleet-paperwork): all four org installations report
organization_projects=write,checks=write,statuses=write; the developing-today installation token now resolvesprojectV2(number: 4)→PVT_kwDOAZNvF84Bouay("Developing Today"); the three sibling orgs'projectsV2queries execute (totalCount 0 — they have no projects). The Project #4 grant pipeline is fully armed on this app.App 2 — developing.today fleet (5612056, owned by the developing-today org)
Identification (three independent lines): its name; its own description — "Standby/backup of the developing.today app (App ID 5454422)"; and its private key held in BWS as
github-app.developing-today-fleet.private-key. Ruled out:developing-today-backup(5562183) is the github-backup system's app — BWSgithub-app.backup-app-idequals 5562183 (verified in-process) — and was left untouched.Before state was the same shape as App 1's original (Repository 5 selected, Organization 0 selected). Same batch applied in one save: Org Projects No access → Read & write; Repo Checks No access → Read & write; Repo Commit statuses No access → Read & write; Actions/Contents/Issues/Pull requests/Workflows Read & write (no-op); Metadata Read-only (no-op). After: Repository 7 selected, Organization 1 selected. Verified on fresh reload AND API-side (
GET /appreports requestedorganization_projects: write).Installation approvals (all four of its installations; same review-accept-verify flow):
Probe (JWT minted from the BWS-held key): all four installations report
organization_projects=write,checks=write,statuses=write; a fleet-app installation token for developing-today resolvesprojectV2(number: 4)→PVT_kwDOAZNvF84Bouay. The standby is now permission-identical to the primary for the whole automation set.Net state
Both automation apps carry: Org Projects RW; Repo Metadata RO, Issues RW, Pull requests RW, Contents RW, Actions RW, Checks RW, Commit statuses RW (Workflows RW pre-existing). Every installation has accepted. No further mobile approvals are outstanding for this change. Project writes under either App identity are unblocked permission-wise; the outbox drain noted in the Stage 4 record can proceed on the App leg as the user GraphQL bucket refills.