Skip to content

CommandExecutor for external CLI tools as pipeline steps #81

Open

Description

@ddeboer

Summary

Add a CommandExecutor that runs an external CLI command as a pipeline step, reading quads from stdin or a file and producing quads (or other output) on stdout or to a file.

Context

Some pipeline steps are best handled by external tools rather than in-process Node.js code. For example, loda-pipeline's post-processing chain:

  1. Deduplication — Jena's sparql CLI running a CONSTRUCT query against an N-Triples file
  2. SHACL validation — Jena's shacl validate against EDM shapes
  3. Format conversion — Jena's riot --output=rdfxml for N-Triples → RDF/XML
  4. EDM XML packaging — a custom Java tool (Rdf2EdmCl) that splits RDF/XML into individual EDM XML files in a ZIP

These don't fit the SPARQL CONSTRUCT executor model — they're file-in, file-out CLI invocations. A CommandExecutor would make them composable as pipeline steps.

Approach

A CommandExecutor implements the same interface as SparqlConstructExecutor but delegates to an external process:

class CommandExecutor implements Executor {
  constructor(options: {
    command: string;       // e.g. 'riot'
    args: string[];        // e.g. ['--output=rdfxml']
    // How to pass input: stdin pipe, file path substitution, or no input
    input?: 'stdin' | 'file';
    // How to read output: stdout pipe or file path
    output?: 'stdout' | 'file';
  });
}

This is also the pattern proposed for SparqlAnythingExecutor in the design doc — spawning java -jar sparql-anything.jar per query, parsing N-Triples from stdout.

Uses @lde/task-runner for process lifecycle management (Docker or host execution).

Subsumes #18

SparqlAnythingExecutor is a specialised CommandExecutor — it spawns java -jar sparql-anything.jar and parses N-Triples from stdout. Rather than building a one-off executor for SPARQL Anything, the generic CommandExecutor covers it and any other CLI tool.

Relates to

Activity

  1. ddeboer commented on Jun 22, 2026

    @ddeboer
    MemberAuthor

    From hardening netwerk-digitaal-erfgoed/geonames-rdf (a hand-rolled SPARQL Anything → RDF converter), four lessons that map onto the layers here. Detail in #18; the split is the point:

    1. Failure contract → CommandExecutor (generic). A non-zero exit → hard step failure, never a silently-skipped item. Our incident was exactly this: a chunk crashed (non-zero exit) and SPARQL Anything deleted its own output, but the script ignored the exit code, so the concatenation dropped ~1M rows on a green build. Make the exit code the signal – not missing/empty output, which can be legitimate for some tools (a CONSTRUCT/filter that matches nothing). Enforce it once (via @lde/task-runner) and every CLI step here – Jena dedup, shacl validate, riot, Rdf2EdmCl, SPARQL Anything – inherits it. This argues for the generic executor over a one-off.

    2. Tool specifics → SparqlAnythingExecutor (specialised). CSV/TSV options (the literal-quote csv.quote-char workaround, null-string, delimiter, headers), fx:ondisk, N-Triples-from-stdout parsing.

    3. Chunking → pipeline fan-out, not executor internals. Large inputs split into items run through the executor; the per-item failure contract (1) is what makes the fan-out safe.

    4. Completeness gating → a downstream validation step, not an executor. A tight relative-regression count (drop vs. previous run) after convert, before publish – @lde/pipeline-void plus a stored previous count. A static floor was tried and dropped: it just re-detects the crash case (1) already catches.

    One wrinkle for the interface: it is framed as reading quads in/out, but SPARQL Anything’s input is non-RDF (a source file named inside the query via fx:location), which is the whole point of #18. The input model needs an opaque-file / no-quad-input mode, or this case does not fit cleanly.

  2. ddeboer commented on Jun 22, 2026

    @ddeboer
    MemberAuthor

    Refinement: split CommandExecutor (quads) from CommandConverter (file)

    Came out of designing the SPARQL Anything migration (#18). Applying LDE’s own rule — an Executor produces quads (execute() → AsyncIterable<Quad>) — exposes a conflation in this issue: the motivating tools here are not all executors.

    • Jena dedup, riot format conversion, and Rdf2EdmCl are file-in/file-out. Rdf2EdmCl produces a ZIP of EDM XML — it can’t yield quads at all. By the executor=produces-quads rule, none of these are executors.
    • Only a command whose output you parse back into quads (e.g. SPARQL Anything streaming N-Triples on stdout) can implement Executor.

    So one name (CommandExecutor) is covering two different shapes. Proposed split into three levels, shared by SPARQL Anything and the loda-pipeline chain:

    1. Process-runner core (generic, lowest): spawn via @lde/task-runner, fail-on-nonzero-exit contract, optional per-chunk orchestration. Independent of how output is consumed.
    2. CommandExecutor implements Executor = core + parse output → quads (streaming). Slots into a Stage.
    3. CommandConverter (file-in/file-out, not an Executor, not a node in Pipeline → Stage → Executor) = core + file output. This is where riot, Rdf2EdmCl, and file-dedup actually belong.

    SPARQL Anything ends up with both surfaces over the one core: a SparqlAnythingExecutor (streaming) and a SparqlAnythingConverter (file). We are building only the converter first (the geonames convert-only case is file-out with no in-Node post-processing — see #18); the streaming executor is deferred but the core is factored so it tucks on later without a refactor.

    One concrete seam this surfaces for the streaming side: TaskRunner.wait(task): Promise<string> buffers stdout into a string — fine for converters (they ignore stdout; the tool writes the file), but it cannot stream multi-GB output. A streaming CommandExecutor will need streamed stdout access, not wait→string.

    Net ask: rename/retitle so non-quad, file-in/file-out tools live under CommandConverter rather than CommandExecutor, with a shared process-runner core underneath both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions