Skip to content

Define import log and source commit policy #12

Closed
Closed
Define import log and source commit policy#12
Labels
P0Must havedocsDocumentation or process docs workmigrationImport or migration work

Description

@alexeygrigorev
alexeygrigorev
opened on Jun 27, 2026

Define import log and source commit policy

Status: pending
Tags: docs, migration, P0
Blocks: #4, #7

Current Evidence

_docs/import-log.md already exists and records the initial 2026-06-27 import state for DTC Operations, DataTasks, and Podcast Assistant. It includes source paths, source commits/source state, imported locations, exclusions, and purpose notes.

The issue is not fully satisfied yet because the file does not define the reusable import/source-state policy requested in the intake, does not record validation commands, and does not make the expected fields explicit enough for dependent imports (#4 and #7) to follow without inventing their own convention.

Scope

Update _docs/import-log.md into the canonical policy and register for copied source systems in DataTalksClub/dataops.

The implementation should preserve the existing import evidence and add a clear policy that every import or re-import records enough information for reviewers to identify what was copied, from where, at what source revision/state, what was deliberately excluded, and how the import was validated.

For each copied source system, the log should record:

  • Source system name.
  • Source path and, when applicable, source repository URL or local source identity.
  • Source commit hash for Git-backed sources, or explicit non-Git/local source state when no commit exists.
  • Destination path(s) in this repo.
  • Copied path groups or major copied contents.
  • Deliberate exclusions, especially generated, dependency, cache, secret, local runtime, and private output paths.
  • Validation commands run after import, including whether each command passed, failed, or was skipped with a reason.
  • Follow-up issue references when an import is transitional or will be moved/normalized later.

Keep this issue documentation-only. Do not copy, move, delete, refactor, or re-import source code as part of #12.

Acceptance Criteria

  • _docs/import-log.md contains a named policy section that defines the required fields for future source imports and re-imports.
  • The policy states that source repos/directories such as ../dtc-operations, ../datatasks, and ../podcast-assistant are read-only unless a specific issue explicitly scopes changes there.
  • The policy explains how to record Git-backed sources with git -C <source> rev-parse HEAD and how to record non-Git local directories such as Podcast Assistant.
  • The existing DTC Operations entry records source path, source commit, destination path(s), copied path groups, exclusions, validation commands/statuses, and any known follow-up context.
  • The existing DataTasks entry records source path, source commit, destination path(s), copied path groups, exclusions, validation commands/statuses, and dependency/follow-up context for Import DataTasks into work-engine #4.
  • The existing Podcast Assistant entry records source path, non-Git source state, current destination path, copied path groups, exclusions, validation commands/statuses, and the follow-up context that Import Podcast Assistant into assistants/podcast #7 should move/canonicalize it under assistants/podcast/.
  • Validation command entries are explicit enough that a reviewer can distinguish commands that passed from commands that were not run or were blocked by local environment constraints.
  • The final document does not claim unverified commands passed.
  • No files outside _docs/import-log.md are changed for this issue.

Test Scenarios

  • From the repo root, verify the import log exists and contains the policy and all three current source systems:

    test -f _docs/import-log.md
rg -n "policy|DTC Operations|DataTasks|Podcast Assistant|Validation" _docs/import-log.md

  • For Git-backed local sources that are available, verify the recorded commit values can be reproduced:

    git -C ../dtc-operations rev-parse HEAD
git -C ../datatasks rev-parse HEAD

  • For the local non-Git Podcast Assistant source, verify the source-state statement is accurate:

    test -d ../podcast-assistant
test ! -d ../podcast-assistant/.git

  • Review _docs/import-log.md manually and confirm every source entry has source state, destination path(s), copied paths, exclusions, validation commands/statuses, and follow-up references where relevant.

Dependencies

Out of Scope

  • Importing or re-importing DataTasks.
  • Moving Podcast Assistant from podcast-assistant/ to assistants/podcast/; that remains in Import Podcast Assistant into assistants/podcast #7.
  • Renaming lambda-functions/, restructuring backend code, or changing runtime paths.
  • Adding automated tests beyond lightweight documentation/source-state verification commands.
  • Modifying any source repository or directory outside dataops.

Activity

  1. alexeygrigorev
    added
    needs groomingRaw intake that needs PM grooming
    docsDocumentation or process docs work
    migrationImport or migration work
    P0Must have
    and removed
    needs groomingRaw intake that needs PM grooming
    on Jun 27, 2026
  2. alexeygrigorev
    changed the title [-]Add import log and source commit policy[/-] [+]Define import log and source commit policy[/+] on Jun 27, 2026

  3. alexeygrigorev commented on Jun 27, 2026

    @alexeygrigorev
    alexeygrigorev
    on Jun 27, 2026
    MemberAuthor

    PM triage complete.

    Decision: not already satisfied by current repo evidence, so I groomed #12 instead of closing it.

    Evidence reviewed:

    • _docs/import-log.md exists and records initial DTC Operations, DataTasks, and Podcast Assistant import/source state.
    • The existing log includes source paths, Git commits or non-Git source state, imported locations, exclusions, and purpose notes.
    • It does not yet define a reusable import/source-state policy.
    • It does not record validation commands/statuses.
    • It does not make copied path groups and future re-import requirements explicit enough for dependent issues Import DataTasks into work-engine #4 and Import Podcast Assistant into assistants/podcast #7.
    • Import Podcast Assistant into assistants/podcast #7 also needs the policy to distinguish the current transitional podcast-assistant/ import from the target assistants/podcast/ location.

    I updated the issue into an implementable documentation-only task with scope, acceptance criteria, test scenarios, dependencies, and out-of-scope boundaries, then removed needs grooming. Remaining labels are docs, migration, and P0.

  4. alexeygrigorev commented on Jun 27, 2026

    @alexeygrigorev
    alexeygrigorev
    on Jun 27, 2026
    MemberAuthor

    TEST PASS

    Tester verification for #12 completed against the current repo/source state.

    Evidence:

    • test -f _docs/import-log.md: passed.
    • rg -n "policy|DTC Operations|DataTasks|Podcast Assistant|Validation" _docs/import-log.md: passed; matched the named policy section, all three source-system entries, and validation sections.
    • git -C ../dtc-operations rev-parse HEAD: passed, output e6f7b43f641389fa58a8e027485f5aac18e8ace5, matching the DTC Operations entry.
    • git -C ../datatasks rev-parse HEAD: passed, output 9adbc21da3a4db3990a58965950513c618abcda0, matching the DataTasks entry.
    • test -d ../podcast-assistant: passed.
    • test ! -d ../podcast-assistant/.git: passed, matching the non-Git Podcast Assistant source-state handling.
    • git diff --check: passed.
    • git diff --name-only -- . ':(exclude)_docs/import-log.md': no output.
    • git status --short: only _docs/import-log.md is modified.

    Manual/repo checks:

    • _docs/import-log.md contains a named import/source-state policy with required fields, read-only source boundary language, Git-backed source commit instructions, and non-Git local source-state instructions.
    • DTC Operations, DataTasks, and Podcast Assistant entries each include source state, destinations, copied path groups, exclusions, validation command/status entries, and follow-up context where applicable.
    • Destination/path groups listed for all three entries exist in the current dataops checkout.
    • Validation statuses distinguish commands that passed from full app suites that were not rerun for this documentation-only issue.

    No app suite was run because #12 is documentation-only and no app code changed.

  5. alexeygrigorev commented on Jun 27, 2026

    @alexeygrigorev
    alexeygrigorev
    on Jun 27, 2026
    MemberAuthor

    ACCEPTED

    PM acceptance after Tester pass. _docs/import-log.md now defines the reusable import/source-state policy, keeps the source repos/directories read-only unless explicitly scoped, and records the required source state, destinations, copied path groups, exclusions, validation statuses, and follow-up context for DTC Operations, DataTasks, and Podcast Assistant.

    This is acceptable as the source policy needed by #4 and #7. The DataTasks entry gives #4 a source-state convention for re-import/normalization work, and the Podcast Assistant entry clearly identifies the current transitional import plus the #7 move/canonicalization target under assistants/podcast/.

    PM spot-checks passed: issue criteria reviewed, Tester pass reviewed, source-state commands reproduce the recorded DTC Operations/DataTasks commits, Podcast Assistant non-Git state is accurate, git diff --check passes, and only _docs/import-log.md is changed.

  6. alexeygrigorev
    closed this as completedin ab0b290on Jun 27, 2026

  7. alexeygrigorev commented on Jun 27, 2026

    @alexeygrigorev
    alexeygrigorev
    on Jun 27, 2026
    MemberAuthor

    ON-CALL PASS (expected CI skip)

    Post-push verification for ab0b2909b222fa351b48b8e1c5d0a5e8f996662c (Define import source-state policy).

    Checks:

    • git ls-remote origin refs/heads/main points to ab0b2909b222fa351b48b8e1c5d0a5e8f996662c.
    • Commit changes only _docs/import-log.md.
    • GitHub check-runs API for this SHA returned total_count: 0; gh run list --commit ab0b2909b222fa351b48b8e1c5d0a5e8f996662c returned no runs.
    • .github/workflows/deploy-dataops-v1.yml push paths exclude _docs/** and include only workflow/app/content/work-engine/build inputs.
    • .github/workflows/validate-dataops-content.yml push paths exclude _docs/** and include only content/** plus the workflow file.

    Result: pass. No GitHub Actions run IDs are expected for this docs-only _docs/import-log.md commit because both push workflows were path-filtered out. No CI/deploy failure found, and no on-call fix was needed.

  8. alexeygrigorev
    added a commit that references this issue on Aug 14, 2026

    Define import source-state policy

    6d6dd7a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Must havedocsDocumentation or process docs workmigrationImport or migration work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions