judge a pipeline on its log, not on exiting

the run only exited early on gst, where a leaky queue was inserted after
filesrc and cut the input to 65 of 5554 frames. g2g got no such queue,
ran the whole video, and hit the timeout, so the two backends were never
compared on the same work. both now run to the cap and pass if the log is
clean.

the gap counts recorded here came from that comparison, so they go until
the suite is run again.

Author: aaron-boxer

DESIGN_TODO.md

@@ -6,18 +6,19 @@ Python-element host are tracked in that repo's `DESIGN_TODO.md`, under
 
 ## g2g backend coverage
 
-- **32 README pipelines run on gst and fail under `PYML_BACKEND=g2g`.**
-  `tests/test_pipelines.py` at a 30 s timeout passes 48 of 79 on gst and 16 on
-  g2g. 31 fail on both, which is the environment rather than g2g, and nothing
-  passes on g2g that fails on gst. Most of the 32 report `pipeline error:
-  Hardware(Other)`, which is how g2g reports a hosted element raising, so each
-  one needs its log in `tests/logs` read to name the cause. `pyml_overlay` is in
-  16 of them, the largest cluster. Two causes known: `pyml_kafkasink` calls
-  `Gst.Pad` APIs directly and dies on `Gst.init`, and `demo_soccer`'s engine
-  raises `TypeError: MLEngine.__init__() got an unexpected keyword argument
-  'device'`. The suite takes about 17 minutes per backend and wants the GPU, so
-  run one backend at a time on a 6 GB card, and keep the machine otherwise idle
-  or the 30 s timeout starts measuring load instead.
+- **How many README pipelines run under `PYML_BACKEND=g2g` needs measuring.**
+  Run `tests/test_pipelines.py` under each backend and compare: one that passes
+  on gst and fails on g2g is a gap, one that fails on both is the environment.
+  Only the error categories count as gaps. `pipeline error: Hardware(Other)` is
+  how g2g reports a hosted element raising, so each needs its log in
+  `tests/logs` read to name the cause. Known so far: `pyml_kafkasink` calls
+  `Gst.Pad` APIs directly and dies on `Gst.init`, `demo_soccer`'s engine raises
+  `TypeError: MLEngine.__init__() got an unexpected keyword argument 'device'`,
+  and `pyml_streammux` is refused with `pyelement: more than one input links
+  here, but it is not a registered muxer`. The suite wants the GPU for about 20
+  minutes per backend, so run one backend at a time on a 6 GB card and leave the
+  machine otherwise idle, including between backends: a model still resident
+  from the previous run fails the next one at preroll.
 
 - **Eleven elements have no per-frame seam, so they cannot run on g2g at all.**
   `alert`, `tracker`, `vad`, `clap`, `overlay_counter`, `kafkasink`,

tests/test_pipelines.py

@@ -64,8 +64,10 @@ def get_pipelines_from_readme():
         pipeline = pipeline.replace("python pyml-launch.py", LAUNCHER, 1)
 
         parts = pipeline.split("!")
-        modified = False
 
+        # A `filesrc` run has no equivalent cap: these mp4s carry `moov` at the
+        # end, so bounding the source by bytes leaves the decoder with no index.
+        # Those pipelines run until PIPELINE_TIMEOUT instead.
         for i, part in enumerate(parts):
             part_clean = part.strip()
             if "videotestsrc" in part_clean:
@@ -74,23 +76,8 @@ def get_pipelines_from_readme():
                     parts[i] = f"{part_clean} num-buffers=100"
                 else:
                     parts[i] = re.sub(r"num-buffers=\d+", "num-buffers=100", part_clean)
-                modified = True
                 break
 
-        # Only on gst: g2g's `decodebin` takes its input caps from the element
-        # ahead of it, and a `queue` declares none, so capping the run this way
-        # would stop the pipeline parsing at all.
-        if not modified and BACKEND == "gst":
-            for i, part in enumerate(parts):
-                part_clean = part.strip()
-                if "filesrc" in part_clean:
-                    parts.insert(i + 1, "queue max-size-buffers=100 leaky=upstream")
-                    modified = True
-                    break
-
-        if not modified:
-            print(f"Warning: No filesrc or videotestsrc found in pipeline: {pipeline}")
-
         modified_pipeline = " ! ".join(parts).strip()
         print(f"Modified pipeline: {modified_pipeline}")
         modified_pipelines.append(modified_pipeline)
@@ -137,7 +124,10 @@ def rewrite(match):
 @pytest.mark.parametrize("pipeline", PIPELINES, ids=lambda p: p)
 def test_pipeline(pipeline, tmp_path):
     """
-    Test a GStreamer pipeline for 100 frames, checking for errors, with latency tracing.
+    Run a README pipeline and check its log for errors.
+
+    A pipeline still running at `PIPELINE_TIMEOUT` passes: only `videotestsrc`
+    takes a frame cap, so a file-backed one runs as long as its media lasts.
     """
     LOG_DIR.mkdir(parents=True, exist_ok=True)
     os.sync()
@@ -187,6 +177,7 @@ def test_pipeline(pipeline, tmp_path):
     env["GST_DEBUG_NO_COLOR"] = "1"
 
     # Run the pipeline
+    ran_to_the_cap = False
     try:
         with open(log_file, "w") as log:
             # Own process group: the shell is not the pipeline, it is the
@@ -205,10 +196,12 @@ def test_pipeline(pipeline, tmp_path):
             process.wait(timeout=PIPELINE_TIMEOUT)
             return_code = process.returncode
     except subprocess.TimeoutExpired:
+        # Still running at the cap, which is what a healthy uncapped pipeline
+        # does: the media outlasts any timeout worth waiting. The log below says
+        # whether it was working, so the run is judged on that, not on exiting.
         end_process_group(process)
-        pytest.fail(
-            f"Pipeline timed out after {PIPELINE_TIMEOUT}s. Full pipeline: {pipeline}. See {log_file}"
-        )
+        ran_to_the_cap = True
+        return_code = None
     except Exception as e:
         end_process_group(process)
         pytest.fail(
@@ -234,7 +227,7 @@ def test_pipeline(pipeline, tmp_path):
         )
 
     # Check exit code
-    if return_code != 0:
+    if not ran_to_the_cap and return_code != 0:
         if (
             "End-Of-Stream" not in log_content
             and "reached end of stream" not in log_content
@@ -243,7 +236,8 @@ def test_pipeline(pipeline, tmp_path):
                 f"Pipeline failed with exit code {return_code}. Full pipeline: {pipeline}. See {log_file}"
             )
 
-    print(f"Pipeline processed 100 frames successfully: {pipeline}")
+    ending = f"ran the full {PIPELINE_TIMEOUT}s" if ran_to_the_cap else "ran to the end"
+    print(f"Pipeline {ending} with no errors: {pipeline}")
 
 
 def test_pipelines_found():