[WIP] Leverage UI5 CLI V5 and UI5 Test Runner V6 by flovogt · Pull Request #582 · UI5/sample-app

flovogt · 2026-11-03T13:52:23Z

Upgrade to UI5 CLI v5
Upgrade UI5 YAML specification version to v5
Upgrade to UI5 Test Runner v6

flovogt · 2026-11-03T13:57:34Z

ArnaudBuchholz · 2026-11-05T20:56:33Z

@flovogt Good news / bad news time.

After creating several patches (now 6.0.7), I was able to make ui5-test-runner run your config with thepackage.json listed below.

Several remarks :

I use--start "npm start" to start the server
I use --start-wait-url http://localhost:8080 to wait for the server to be started
-ccb 100 was not working (because % was required... I changed that) but now it is OK

package.json

{
	"name": "@ui5/sample-app",
	"version": "0.5.0",
	"description": "Sample of an OpenUI5 app",
	"private": true,
	"engines": {
		"node": "^20.11.0 || >=22.0.0",
		"npm": ">= 8"
	},
	"scripts": {
		"start": "ui5 serve --port 8080",
		"lint": "eslint webapp && ui5lint",
		"test-runner": "ui5-test-runner --url http://localhost:8080/test/testsuite.qunit.html --coverage-source-dir webapp --start \"npm start\" --start-wait-url http://localhost:8080 --coverage -ccb 100 -ccf 100 -ccl 100 -ccs 100",
		"test": "npm run lint && npm run test-runner",
		"build": "ui5 build -a --clean-dest",
		"build-self-contained": "ui5 build self-contained -a --clean-dest",
		"serve-dist": "ws --compress -d dist"
	},
	"devDependencies": {
		"@eslint/js": "^10.0.1",
		"@ui5/cli": "^5.0.0-alpha.14",
		"@ui5/linter": "^1.23.7",
		"@ui5/middleware-code-coverage": "^3.0.0-alpha.1",
		"eslint": "^10.14.0",
		"globals": "^17.14.0",
		"local-web-server": "^5.4.2",
		"rimraf": "^6.1.3",
		"ui5-test-runner": "^6.0.7"
	}
}

Output :

   _,     ,_           _ ____        _            _
 .'/   _,  \'.   _   _(_) ___|      | |_ ___  ___| |_      _ __ _   _ _ __  _ __   ___ _ __
|  \__< )__/  | | | | | |___ \ _____| __/ _ \/ __| __|____| '__| | | | '_ \| '_ \ / _ \ '__|
\             / | |_| | |___) |_____| ||  __/\__ \ ||_____| |  | |_| | | | | | | |  __/ |
 '-..(___)..-'   \__,_|_|____/       \__\___||___/\__|    |_|   \__,_|_| |_|_| |_|\___|_|
   0.115 ui5-test-runner@6.0.7 / Node.js v24.22.2 / 2026-11-05T20:50:59.664Z (300)
   0.115 arm64 / 12x Apple M3 Pro
   0.131 executing start command...
   0.227 waiting for start URL to be reachable {"url":"http://localhost:8080"}
   0.313
   0.313 > @ui5/sample-app@0.5.0 start
   0.313 > ui5 serve --port 8080
   0.313
(X)00:02 info Root project: application project @ui5/sample-app (0.5.0)
(X)00:02 info graph:helpers:ui5Framework Using OpenUI5 version: 1.153.0
(X)00:04 [info] Building...
(X)00:04 info ProjectBuilder   Requested dependencies:
(X)00:04     + sap.ui.core
(X)00:04 info ProjectBuilder Preparing build for projects: sap.ui.core
   00:07 Server started
   00:07 URL: http://localhost:8080
(X)00:07 info Project 1 of 1: ✔ Skipping build of library project sap.ui.core
(X)00:07 info ProjectBuilder Build succeeded in 15 ms
(X)00:07 info ProjectBuilder Executing cleanup tasks...
(X)00:07 [info] Build finished in 2.95 s
(X)00:07 [info] Validating caches for: @ui5/sample-app, sap.f, sap.m, sap.ui.layout, sap.ui.unified, themelib_sap_horizon
(X)00:08 [info] Building...
(X)00:08 info ProjectBuilder Preparing build for projects: @ui5/sample-app
(X)00:08 info Project 1 of 1: ✔ Skipping build of application project @ui5/sample-app
(X)00:08 info ProjectBuilder Build succeeded in 797 μs
(X)00:08 info ProjectBuilder Executing cleanup tasks...
(X)00:08 [info] Build finished in 1.06 ms
(X)00:08 [info] Validating caches for: sap.f, sap.m, sap.ui.layout, sap.ui.unified, themelib_sap_horizon
(X)00:08 [info] Server ready
(X)00:08 [info] All projects up-to-date
(X)00:08 [info] Building...
(X)00:08 [info] Build finished in 11 μs
(X)00:08 [info] Server ready
   00:08 coverageSourceDir is set, skipping local instrumentation
/!\00:08 Module nyc not found locally
   00:08 Installed version of nyc is 18.0.0
   00:09 server listening on: http://10.124.77.12:57435/
/!\00:09 Module puppeteer not found locally
   00:09 Installed version of puppeteer is 25.16.0
   00:10 UI5 version 1.153.0 (from http://localhost:8080)
/!\00:28 no coverage data found for page {"url":"http://localhost:8080/test/Test.qunit.html?testsuite=test-resources/sap/ui/demo/todo/testsuite.qunit&test=integration/opaTests"}
   00:28 generating coverage report...
   00:28 --------------------------------------|---------|----------|---------|---------|--------------------
   00:28 File                                  | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
   00:28 --------------------------------------|---------|----------|---------|---------|--------------------
   00:28 All files                             |    6.78 |    36.17 |    6.94 |    6.91 |
   00:28  webapp                               |       0 |      100 |       0 |       0 |
   00:28   Component.js                        |       0 |      100 |       0 |       0 | 1-3
   00:28  webapp/controller                    |   26.78 |    62.96 |   31.25 |    28.3 |
   00:28   App.controller.js                   |   26.78 |    62.96 |   31.25 |    28.3 | 14-17,35-53,85-148
   00:28  webapp/test                          |       0 |      100 |       0 |       0 |
   00:28   testsuite.qunit.js                  |       0 |      100 |       0 |       0 | 1-3
   00:28  webapp/test/integration              |       0 |        0 |       0 |       0 |
   00:28   FilterJourney.js                    |       0 |      100 |       0 |       0 | 3-59
   00:28   SearchJourney.js                    |       0 |        0 |       0 |       0 | 3-108
   00:28   TodoListJourney.js                  |       0 |      100 |       0 |       0 | 3-77
   00:28   opaTests.qunit.js                   |       0 |      100 |       0 |       0 | 1-10
   00:28  webapp/test/integration/arrangements |       0 |      100 |       0 |       0 |
   00:28   Startup.js                          |       0 |      100 |       0 |       0 | 1-9
   00:28  webapp/test/integration/pages        |       0 |        0 |       0 |       0 |
   00:28   App.js                              |       0 |        0 |       0 |       0 | 1-189
   00:28  webapp/test/unit                     |       0 |      100 |     100 |       0 |
   00:28   unitTests.qunit.js                  |       0 |      100 |     100 |       0 | 1
   00:28  webapp/test/unit/controller          |       0 |      100 |       0 |       0 |
   00:28   App.controller.js                   |       0 |      100 |       0 |       0 | 3-57
   00:28  webapp/util                          |       0 |      100 |       0 |       0 |
   00:28   Helper.js                           |       0 |      100 |       0 |       0 | 1-6
   00:28 --------------------------------------|---------|----------|---------|---------|--------------------
(X)00:28 ERROR: Coverage for lines (6.91%) does not meet global threshold (100%)
(X)00:28 ERROR: Coverage for functions (6.94%) does not meet global threshold (100%)
(X)00:28 ERROR: Coverage for branches (36.17%) does not meet global threshold (100%)
(X)00:28 ERROR: Coverage for statements (6.78%) does not meet global threshold (100%)
(X)00:28 Coverage thresholds not met (branches=100%, functions=100%, lines=100%, statements=100%)
   00:28 coverage report complete
   00:28 Test report:
   00:28 PASS   3/3  http://localhost:8080/test/Test.qunit.html?testsuite=test-resources/sap/ui/demo/todo/testsuite.qunit&test=unit/unitTests
   00:28 FAIL   0/14  http://localhost:8080/test/Test.qunit.html?testsuite=test-resources/sap/ui/demo/todo/testsuite.qunit&test=integration/opaTests
   00:28 FAIL   0/1  coverage
(X)00:28 [http://localhost:8080/test/Test.qunit.html?testsuite=test-resources/sap/ui/demo/todo/testsuite.qunit&test=integration/opaTests]
Filter ▶ should show correct items when filtering for 'Active' items
  Opa timeout after 15 seconds
This is what Opa logged:
Found 0 blocking out of 168 tracked timeouts -  sap.ui.test.autowaiter._timeoutWaiter#hasPending
AutoWaiter syncpoint -  sap.ui.test.autowaiter._autoWaiter
Found control with the global ID '__component0---app--toolbar-overflowButton' -  sap.ui.test.Opa5
Control 'Element sap.m.ToggleButton#__component0---app--toolbar-overflowButton'' is not rendered -  sap.ui.test.matchers.Visible
0 out of 1 controls met the matchers pipeline requirements -  sap.ui.test.pipelines.MatcherPipeline
Matchers found no controls so check function will be skipped -  sap.ui.test.Opa5
Callstack:
    at c.success (http://localhost:8080/test/integration/pages/App.js?instrument=true:1:2054)
    at u.success (http://localhost:8080/resources/sap/ui/test/Opa5.js:6:5822)
    at c.<anonymous> (http://localhost:8080/resources/sap/ui/test/Opa.js:6:3997)
    at n (http://localhost:8080/resources/sap/ui/test/Opa.js:6:627)
    at e (http://localhost:8080/resources/sap/ui/test/autowaiter/_timeoutWaiter.js:6:2649)
Callstack:
    at c.success (http://localhost:8080/test/integration/pages/App.js?instrument=true:1:1905)
    at u.success (http://localhost:8080/resources/sap/ui/test/Opa5.js:6:5822)
    at c.<anonymous> (http://localhost:8080/resources/sap/ui/test/Opa.js:6:3997)
    at n (http://localhost:8080/resources/sap/ui/test/Opa.js:6:627)
    at e (http://localhost:8080/resources/sap/ui/test/autowaiter/_timeoutWaiter.js:6:2649)
Callstack:
    at c._waitForToolbar (http://localhost:8080/test/integration/pages/App.js?instrument=true:1:1864)
    at c.iFilterForItems (http://localhost:8080/test/integration/pages/App.js?instrument=true:1:1656)
    at Object.<anonymous> (http://localhost:8080/test/integration/FilterJourney.js?instrument=true:1:199)
    at Object.<anonymous> (http://localhost:8080/resources/sap/ui/test/opaQunit.js:6:1998)
    at n (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1301:13504)
    at G.e [as run] (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1301:13327)
    at http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1301:16109
      at e (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1301:21830)
    at http://localhost:8080/resources/sap/ui/test/opaQunit.js:6:1015
    at Object.<anonymous> (http://localhost:8080/resources/sap/ui/test/opaQunit.js:6:2148)
    at u (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1275:25639)
    at Object.fireWith [as rejectWith] (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1275:26474)
    at t.reject (http://localhost:8080/resources/sap/ui/test/starter/runTest.js:1229:10108)
/!\00:28 [http://localhost:8080/test/Test.qunit.html?testsuite=test-resources/sap/ui/demo/todo/testsuite.qunit&test=integration/opaTests]
13 failures; showing first OPA error only
(X)00:28 [coverage]
Global code coverage check
  Coverage thresholds not met (branches=100%, functions=100%, lines=100%, statements=100%)
   00:28 Tests completed: passed=3 failed=14 tests=18 (00:28)

ArnaudBuchholz · 2026-11-05T20:58:53Z

Side note : the OPA page does not contain coverage data, seems to be related to the fact that the app is bundled (but I am not sure)

flovogt · 2026-11-06T10:38:06Z

@ArnaudBuchholz Thank you! I got it working but I see a mismatch in how the coverage configuration in the testsuite is treated by UI5 Test Runner.

In webapp/test/testsuite.qunit.js the coverage reporting has disabled reporting for the test files.

	coverage: {
		only: "sap/ui/demo/todo/",
		never: "test-resources/sap/ui/demo/todo/"
	},
	loader: {
		paths: {
			"sap/ui/demo/todo": "../"
		}
	}

This config seems to be ignored by UI5 Test Runner:

   00:24 --------------------------------------|---------|----------|---------|---------|-------------------
   00:24 File                                  | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
   00:24 --------------------------------------|---------|----------|---------|---------|-------------------
   00:24 All files                             |   99.09 |    91.48 |     100 |   99.07 |                   
   00:24  webapp                               |     100 |      100 |     100 |     100 |                   
   00:24   Component.js                        |     100 |      100 |     100 |     100 |                   
   00:24  webapp/controller                    |     100 |      100 |     100 |     100 |                   
   00:24   App.controller.js                   |     100 |      100 |     100 |     100 |                   
   00:24  webapp/test                          |     100 |      100 |     100 |     100 |                   
   00:24   testsuite.qunit.js                  |     100 |      100 |     100 |     100 |                   
   00:24  webapp/test/integration              |    98.7 |       50 |     100 |    98.7 |                   
   00:24   FilterJourney.js                    |     100 |      100 |     100 |     100 |                   
   00:24   SearchJourney.js                    |   97.05 |       50 |     100 |   97.05 | 14                
   00:24   TodoListJourney.js                  |     100 |      100 |     100 |     100 |                   
   00:24   opaTests.qunit.js                   |     100 |      100 |     100 |     100 |                   
   00:24  webapp/test/integration/arrangements |     100 |      100 |     100 |     100 |                   
   00:24   Startup.js                          |     100 |      100 |     100 |     100 |                   
   00:24  webapp/test/integration/pages        |      98 |    83.33 |     100 |   97.95 |                   
   00:24   App.js                              |      98 |    83.33 |     100 |   97.95 | 109               
   00:24  webapp/test/unit                     |     100 |      100 |     100 |     100 |                   
   00:24   unitTests.qunit.js                  |     100 |      100 |     100 |     100 |                   
   00:24  webapp/test/unit/controller          |     100 |      100 |     100 |     100 |                   
   00:24   App.controller.js                   |     100 |      100 |     100 |     100 |                   
   00:24  webapp/util                          |     100 |      100 |     100 |     100 |                   
   00:24   Helper.js                           |     100 |      100 |     100 |     100 |                   
   00:24 --------------------------------------|---------|----------|---------|---------|-------------------

ArnaudBuchholz · 2026-11-06T12:20:15Z

@flovogt ui5-test-runner has never used coverage information from the test suite configuration. As of today, it is entirely based on .nycrc.json. Yet, I am assuming that this is properly documented and I can probably find a way to "merge" those settings with .nycrc.json (if any).

I will look into that

flovogt · 2026-11-09T08:12:47Z

Side note : the OPA page does not contain coverage data, seems to be related to the fact that the app is bundled (but I am not sure)

Created an issue for this UI5/cli-extensions#627.

flovogt force-pushed the ui5-cli-v5-ui5-test-runner-v6 branch from 61fe608 to 8f78555 Compare November 3, 2026 13:56

ArnaudBuchholz mentioned this pull request Nov 3, 2026

Use sample app in e2e test cases ArnaudBuchholz/ui5-test-runner#203

Open

flovogt force-pushed the ui5-cli-v5-ui5-test-runner-v6 branch 4 times, most recently from 0a432fc to d6213e2 Compare November 9, 2026 08:01

flovogt mentioned this pull request Nov 9, 2026

Code Coverage for OPA5 tests is no reported properly UI5/cli-extensions#627

Closed

flovogt force-pushed the ui5-cli-v5-ui5-test-runner-v6 branch 2 times, most recently from a24cb94 to 9dfa220 Compare November 9, 2026 09:03

[WIP] Leverage UI5 CLI V5 and UI5 Test Runner V6

a76189c

flovogt force-pushed the ui5-cli-v5-ui5-test-runner-v6 branch from 9dfa220 to a76189c Compare November 9, 2026 09:16

Navigation Menu

Repository navigation

[WIP] Leverage UI5 CLI V5 and UI5 Test Runner V6 - #582

[WIP] Leverage UI5 CLI V5 and UI5 Test Runner V6#582
flovogt wants to merge 1 commit into
mainfrom
ui5-cli-v5-ui5-test-runner-v6

flovogt commented Nov 3, 2026 •

edited

Loading

Uh oh!

flovogt commented Nov 3, 2026

Uh oh!

ArnaudBuchholz commented Nov 5, 2026

Uh oh!

ArnaudBuchholz commented Nov 5, 2026

Uh oh!

flovogt commented Nov 6, 2026

Uh oh!

ArnaudBuchholz commented Nov 6, 2026

Uh oh!

flovogt commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

flovogt commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flovogt commented Nov 3, 2026

Uh oh!

ArnaudBuchholz commented Nov 5, 2026

Uh oh!

ArnaudBuchholz commented Nov 5, 2026

Uh oh!

flovogt commented Nov 6, 2026

Uh oh!

ArnaudBuchholz commented Nov 6, 2026

Uh oh!

flovogt commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

flovogt commented Nov 3, 2026 •

edited

Loading