Skip to content

aggregator(charges): schedule the charges producer — candidates file only refreshes when an agent runs it by hand #2205

New issue
New issue
Closed
Closed
aggregator(charges): schedule the charges producer — candidates file only refreshes when an agent runs it by hand#2205
Assignees
dezren39
Labels
effort/sSmall: expected under 2 hours of workkind/infrastructureInfrastructure / host-portability workmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)

Description

@dezren39
dezren39
opened on Oct 29, 2026

Gap

#1057 is implemented and live-verified (2026-10-29, queue task 2990): scripts/produce-charge-candidates.ts (Plaid + Gmail → charges-candidates.json) feeds the fleet-feed collector, which publishes the /quota charges strip. But nothing runs the producer on a schedule:

  • The collector (fleet-feed-collector.timer, ~5 min, from the ~/git/pm-feed clone) consumes the candidates file fail-soft — a stale file silently means a stale charges strip.
  • The only producer runs so far were by hand from agent sessions (2026-10-29 06:45Z during PR feat(aggregator): wire charges auto-matcher into collector + Plaid/Gmail producer (#1057) #2149 development, and 09:11Z during task 2990 verification).
  • No systemd timer, cron job, or Muse scheduled job references produce-charge-candidates (checked 2026-10-29: systemd timers, ~/tools, and the Muse cron list).

Fix options

  1. A systemd timer on the sandbox mirroring fleet-feed-collector (the producer is non-interactive: it shells out to /opt/hatch/bin/plaid and /opt/hatch/bin/hatch_gws_cli, both already usable from the collector unit context), daily or a few times a day, before a collector cycle.
  2. Or fold the producer run into an existing daily Muse job that already touches the hub.

Acceptance

  • charges-candidates.json generatedAt is never more than ~24h old without an agent remembering to run the producer.
  • A stale-candidates condition is visible somewhere (collector log line at minimum) instead of silently publishing old charges.

Refs #1057, #1349 (pm-feed staleness class). Filed by queue worker Lane B from task 2990 verification.

Acceptance criteria

  • A scheduled mechanism (systemd timer, Muse scheduled job, or OpenChamber scheduled task — the choice recorded on this issue) runs scripts/produce-charge-candidates.ts on a cadence at least as frequent as the fleet-feed collector's consumption, without any agent session involved.
  • A fresh producer run's output is observed landing in charges-candidates.json on schedule (timestamp evidence), and the /quota charges strip reflects data no older than one cadence interval.
  • The producer's failure mode is visible somewhere (a failed run leaves a record a monitor can find), rather than the current silent fail-soft staleness.

Activity

  1. dezren39
    added
    queue/queuedHas a queued task in the Supabase agent queue
    on Oct 29, 2026

  2. developing-today commented on Oct 29, 2026

    @developing-today
    developing-todayBot
    on Oct 29, 2026 – with developing.today
    Contributor

    CODE-1087

  3. github-actions
    assigned
    dezren39
    on Oct 29, 2026
  4. dezren39
    added
    kind/infrastructureInfrastructure / host-portability work
    status/needs-triageNot yet triaged; needs a first look
    priority/normalOrdinary priority (the default)
    effort/sSmall: expected under 2 hours of work
    meta/enrichedIntake review done: acceptance criteria, links, and normalized labels added
    on Oct 29, 2026

  5. dezren39 commented on Oct 29, 2026

    @dezren39
    dezren39
    on Oct 29, 2026
    MemberAuthor

    Intake review

    • Fleet monitor intake (repo-fleet-monitor, 2026-10-29 05:12 CDT run): this issue was filed with no labels at all besides queue/queued. Normalized: kind/infrastructure, status/needs-triage, priority/normal, effort/s, meta/enriched; AC appended. A triage task (queue task 3387) was enqueued by this same monitor run under Duty 1 — the issue carried queue/queued with no live task behind it.
    • Still live? Yes — the body verifies no timer/cron/scheduled job references the producer (checked 2026-10-29), and nothing since changes that.
    • Related: feat(usage-hub): v3 charges auto-matching — Plaid + Gmail alerts to services #1057 (the implemented producer, queue task 2990 live-verified), PR feat(aggregator): wire charges auto-matcher into collector + Plaid/Gmail producer (#1057) #2149 (producer development).
    • Risks: the producer reads Plaid + Gmail — its schedule inherits those credentials' availability; a timer on the sandbox must use the existing secret references, never a copied value.
    • Suggested approach: option 1 from the body (a systemd timer mirroring fleet-feed-collector) is the smallest step; whichever mechanism is chosen, verify by watching one unattended run land.
  6. dezren39
    added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    status/needs-triageNot yet triaged; needs a first look
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 1, 2026

  7. dezren39 commented on Nov 1, 2026

    @dezren39
    dezren39
    on Nov 1, 2026
    MemberAuthor

    🤖 Agent note · picked up

    • Agent: openchamber-scheduled-lane (repo-worker-code)
    • Session / ID: ses_e5e69f2e656ffevtHBpRWvT91Y · queue task 3387
    • Task: Revalidate the scheduling gap and route the appropriate follow-up.

  8. dezren39 commented on Nov 1, 2026

    @dezren39
    dezren39
    on Nov 1, 2026
    MemberAuthor

    🤖 Agent note · triaged and handed off

    • Agent: openchamber-scheduled-lane (repo-worker-code) · task 3387
    • Verdict: still apparently live. At main 0f4731e, the NixOS scheduled service runs the collector, not the producer; repository search found no producer schedule. Collector logic does not detect readable stale candidates. #3368 moved the collector schedule to NixOS but did not schedule the producer.
    • Follow-up: queued fix task 5232. Verify the live laptop and external scheduler first; schedule no less frequently than the collector, use existing credential references, and verify an unattended refresh. Repository evidence alone does not establish live scheduler state or freshness.
  9. dezren39
    added
    queue/queuedHas a queued task in the Supabase agent queue
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    status/in-progressQueue worker lifecycle state
    and removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 1, 2026

  10. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    🤖 Agent note · picked up

  11. dezren39 commented on Nov 2, 2026

    @dezren39
    dezren39
    on Nov 2, 2026
    MemberAuthor

    🤖 Agent note · done

    Deployment choice (recorded per acceptance): systemd timer on the sandbox.

    • Verified live first: the fleet-feed collector runs on the LAPTOP as the NixOS user timer fleet-feed-collector.timer (every 5 min, last runs successful), reading charges-candidates.json from the laptop hatch-personal checkout. The producer cannot run there — the laptop has no /opt/hatch/bin/plaid or /opt/hatch/bin/hatch_gws_cli; those connector CLIs live only on the sandbox, which is also where the producer header says it runs. No existing scheduler referenced the producer on either host (checked systemd system+user timers, Muse cron list, crontab).
    • Installed on the sandbox: charge-candidates-producer.service + charge-candidates-producer.timer in /etc/systemd/system (mirrors the fleet-fetch system-unit precedent), running wrapper ~/tools/produce-charge-candidates-scheduled.sh every 30 min. The wrapper ff-pulls ~/git/devtoday-code (fail-soft), runs bun run scripts/produce-charge-candidates.ts from pm-stack/aggregator, and uses only the connectors' existing credential handling — no secrets copied or exposed.
    • Cadence reasoning: the headline acceptance is generatedAt never more than ~24h old; 30 min gives a 48x margin and refreshes far more often than the collector can surface a stale strip, without hammering Plaid/Gmail on the collector's 5-min cycle.
    • Unattended refresh verified: before this work the file was stale at generatedAt 2026-10-29T09:10:17Z on both hosts. The timer fired on its own at 2026-11-02T02:18:00Z (no agent session), service exited SUCCESS, file advanced to generatedAt 2026-11-02T02:18:01Z; next fire 02:48Z. The laptop copy synced within minutes and the collector's 02:17Z run consumed the refreshed file successfully. Producer health spot-check: an October-period run to a temp file produced 80 candidates (plaid 64, gmail 16), so the current month's 0 candidates is a genuinely empty Nov 1–2, not a silent failure.
    • Failure visibility: every run appends one JSON line (ok/failed + generatedAt + candidate count, or the error tail) to hidden_files/charges-producer-runs.jsonl in this goal's workspace, and a failed run exits non-zero into the systemd journal (journalctl -u charge-candidates-producer) — no longer silent fail-soft.
  12. dezren39
    closed this as completedon Nov 2, 2026
  13. dezren39
    removed
    status/in-progressQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    work/assignedQueue worker lifecycle state
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

effort/sSmall: expected under 2 hours of workkind/infrastructureInfrastructure / host-portability workmeta/enrichedIntake review done: acceptance criteria, links, and normalized labels addedpriority/normalOrdinary priority (the default)

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions