You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CommandExecutor for external CLI tools as pipeline steps #81
Add a CommandExecutor that runs an external CLI command as a pipeline step, reading quads from stdin or a file and producing quads (or other output) on stdout or to a file.
Context
Some pipeline steps are best handled by external tools rather than in-process Node.js code. For example, loda-pipeline's post-processing chain:
Deduplication — Jena's sparql CLI running a CONSTRUCT query against an N-Triples file
SHACL validation — Jena's shacl validate against EDM shapes
Format conversion — Jena's riot --output=rdfxml for N-Triples → RDF/XML
EDM XML packaging — a custom Java tool (Rdf2EdmCl) that splits RDF/XML into individual EDM XML files in a ZIP
These don't fit the SPARQL CONSTRUCT executor model — they're file-in, file-out CLI invocations. A CommandExecutor would make them composable as pipeline steps.
Approach
A CommandExecutor implements the same interface as SparqlConstructExecutor but delegates to an external process:
classCommandExecutorimplementsExecutor{constructor(options: {command: string;// e.g. 'riot'args: string[];// e.g. ['--output=rdfxml']// How to pass input: stdin pipe, file path substitution, or no inputinput?: 'stdin'|'file';// How to read output: stdout pipe or file pathoutput?: 'stdout'|'file';});}
This is also the pattern proposed for SparqlAnythingExecutor in the design doc — spawning java -jar sparql-anything.jar per query, parsing N-Triples from stdout.
Uses @lde/task-runner for process lifecycle management (Docker or host execution).
SparqlAnythingExecutor is a specialised CommandExecutor — it spawns java -jar sparql-anything.jar and parses N-Triples from stdout. Rather than building a one-off executor for SPARQL Anything, the generic CommandExecutor covers it and any other CLI tool.
From hardening netwerk-digitaal-erfgoed/geonames-rdf (a hand-rolled SPARQL Anything → RDF converter), four lessons that map onto the layers here. Detail in #18; the split is the point:
Failure contract → CommandExecutor (generic). A non-zero exit → hard step failure, never a silently-skipped item. Our incident was exactly this: a chunk crashed (non-zero exit) and SPARQL Anything deleted its own output, but the script ignored the exit code, so the concatenation dropped ~1M rows on a green build. Make the exit code the signal – not missing/empty output, which can be legitimate for some tools (a CONSTRUCT/filter that matches nothing). Enforce it once (via @lde/task-runner) and every CLI step here – Jena dedup, shacl validate, riot, Rdf2EdmCl, SPARQL Anything – inherits it. This argues for the generic executor over a one-off.
Chunking → pipeline fan-out, not executor internals. Large inputs split into items run through the executor; the per-item failure contract (1) is what makes the fan-out safe.
Completeness gating → a downstream validation step, not an executor. A tight relative-regression count (drop vs. previous run) after convert, before publish – @lde/pipeline-void plus a stored previous count. A static floor was tried and dropped: it just re-detects the crash case (1) already catches.
One wrinkle for the interface: it is framed as reading quads in/out, but SPARQL Anything’s input is non-RDF (a source file named inside the query via fx:location), which is the whole point of #18. The input model needs an opaque-file / no-quad-input mode, or this case does not fit cleanly.
Refinement: split CommandExecutor (quads) from CommandConverter (file)
Came out of designing the SPARQL Anything migration (#18). Applying LDE’s own rule — an Executor produces quads (execute() → AsyncIterable<Quad>) — exposes a conflation in this issue: the motivating tools here are not all executors.
Jena dedup, riot format conversion, and Rdf2EdmCl are file-in/file-out. Rdf2EdmCl produces a ZIP of EDM XML — it can’t yield quads at all. By the executor=produces-quads rule, none of these are executors.
Only a command whose output you parse back into quads (e.g. SPARQL Anything streaming N-Triples on stdout) can implement Executor.
So one name (CommandExecutor) is covering two different shapes. Proposed split into three levels, shared by SPARQL Anything and the loda-pipeline chain:
Process-runner core (generic, lowest): spawn via @lde/task-runner, fail-on-nonzero-exit contract, optional per-chunk orchestration. Independent of how output is consumed.
CommandExecutor implements Executor = core + parse output → quads (streaming). Slots into a Stage.
CommandConverter (file-in/file-out, not an Executor, not a node in Pipeline → Stage → Executor) = core + file output. This is where riot, Rdf2EdmCl, and file-dedup actually belong.
SPARQL Anything ends up with both surfaces over the one core: a SparqlAnythingExecutor (streaming) and a SparqlAnythingConverter (file). We are building only the converter first (the geonames convert-only case is file-out with no in-Node post-processing — see #18); the streaming executor is deferred but the core is factored so it tucks on later without a refactor.
One concrete seam this surfaces for the streaming side: TaskRunner.wait(task): Promise<string> buffers stdout into a string — fine for converters (they ignore stdout; the tool writes the file), but it cannot stream multi-GB output. A streaming CommandExecutor will need streamed stdout access, not wait→string.
Net ask: rename/retitle so non-quad, file-in/file-out tools live under CommandConverter rather than CommandExecutor, with a shared process-runner core underneath both.
Activity
ddeboer commentedon Jun 22, 2026
From hardening
netwerk-digitaal-erfgoed/geonames-rdf(a hand-rolled SPARQL Anything → RDF converter), four lessons that map onto the layers here. Detail in #18; the split is the point:Failure contract →
CommandExecutor(generic). A non-zero exit → hard step failure, never a silently-skipped item. Our incident was exactly this: a chunk crashed (non-zero exit) and SPARQL Anything deleted its own output, but the script ignored the exit code, so the concatenation dropped ~1M rows on a green build. Make the exit code the signal – not missing/empty output, which can be legitimate for some tools (a CONSTRUCT/filter that matches nothing). Enforce it once (via@lde/task-runner) and every CLI step here – Jena dedup,shacl validate,riot,Rdf2EdmCl, SPARQL Anything – inherits it. This argues for the generic executor over a one-off.Tool specifics →
SparqlAnythingExecutor(specialised). CSV/TSV options (the literal-quotecsv.quote-charworkaround,null-string, delimiter, headers),fx:ondisk, N-Triples-from-stdout parsing.Chunking → pipeline fan-out, not executor internals. Large inputs split into items run through the executor; the per-item failure contract (1) is what makes the fan-out safe.
Completeness gating → a downstream validation step, not an executor. A tight relative-regression count (drop vs. previous run) after convert, before publish –
@lde/pipeline-voidplus a stored previous count. A static floor was tried and dropped: it just re-detects the crash case (1) already catches.One wrinkle for the interface: it is framed as reading quads in/out, but SPARQL Anything’s input is non-RDF (a source file named inside the query via
fx:location), which is the whole point of #18. Theinputmodel needs an opaque-file / no-quad-input mode, or this case does not fit cleanly.ddeboer commentedon Jun 22, 2026
Refinement: split
CommandExecutor(quads) fromCommandConverter(file)Came out of designing the SPARQL Anything migration (#18). Applying LDE’s own rule — an
Executorproduces quads (execute() → AsyncIterable<Quad>) — exposes a conflation in this issue: the motivating tools here are not all executors.riotformat conversion, andRdf2EdmClare file-in/file-out.Rdf2EdmClproduces a ZIP of EDM XML — it can’t yield quads at all. By the executor=produces-quads rule, none of these are executors.Executor.So one name (
CommandExecutor) is covering two different shapes. Proposed split into three levels, shared by SPARQL Anything and the loda-pipeline chain:@lde/task-runner, fail-on-nonzero-exit contract, optional per-chunk orchestration. Independent of how output is consumed.CommandExecutorimplementsExecutor= core + parse output → quads (streaming). Slots into aStage.CommandConverter(file-in/file-out, not anExecutor, not a node inPipeline → Stage → Executor) = core + file output. This is whereriot,Rdf2EdmCl, and file-dedup actually belong.SPARQL Anything ends up with both surfaces over the one core: a
SparqlAnythingExecutor(streaming) and aSparqlAnythingConverter(file). We are building only the converter first (the geonames convert-only case is file-out with no in-Node post-processing — see #18); the streaming executor is deferred but the core is factored so it tucks on later without a refactor.One concrete seam this surfaces for the streaming side:
TaskRunner.wait(task): Promise<string>buffers stdout into a string — fine for converters (they ignore stdout; the tool writes the file), but it cannot stream multi-GB output. A streamingCommandExecutorwill need streamed stdout access, notwait→string.Net ask: rename/retitle so non-quad, file-in/file-out tools live under
CommandConverterrather thanCommandExecutor, with a shared process-runner core underneath both.