Skip to content

MultilineStrings miscounts lines in UTF-8 text, and VariableAnalysis flags named by-reference arguments - #398

Merged
mikebronner merged 6 commits into
mainfrom
fix/397-multilinestrings-utf8-and-named-byref-args
Oct 20, 2026
Merged

mikebronner merged 6 commits into
mainfrom
fix/397-multilinestrings-utf8-and-named-byref-args

Conversation

@mikebronner

@mikebronner mikebronner commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements #397. Two sniffs reported errors on correct code: MultilineStrings counted bytes in UTF-8 text, and the bundled VariableAnalysis sniff did not understand named by-reference arguments or nested arrow functions.

Changes

Part 1: line breaks in UTF-8 text

  • New CleanCode/Support/LineBreaks. It gives the line-break pattern for a text. For valid UTF-8 the pattern is LF, CRLF, CR, vertical tab, form feed and NEL, and NEL counts only as its own encoding C2 85. A 0x85 byte inside υ, Ņ or U+2005 is no longer a break. For text that is not valid UTF-8 the pattern stays the old byte-mode \R.
  • MultilineStrings counts with it. If the count itself fails, the sniff reports nothing. It never takes a failed match as zero breaks.
  • StructuredText builds MARKDOWN_LINE, INI_SECTION and SETTING_LINE with the same pattern, at all four call sites (isSerialised included).
  • Decision (AC2): U+0085 counts as a line break. U+2028 and U+2029 do not, because an editor does not show them as a new line. Both cases behave as before. line-break-kinds.php pins all three.
  • Why not the u flag: PHP turns on Unicode character properties with /u, so \w and \s would start to match Greek letters and Unicode spaces. That changes what StructuredText treats as a key or a space, which is outside this issue. The patterns stay in byte mode, and only the break token changes.

Part 2: VariableAnalysis.CodeAnalysis.VariableAnalysis

  • New subclass CleanCode/Overrides/VariableAnalysis/Sniffs/CodeAnalysis/VariableAnalysisSniff.php. Its path repeats the upstream shape, so PHP_CodeSniffer gives it the upstream code. The reported code stays VariableAnalysis.CodeAnalysis.VariableAnalysis.UndefinedVariable, and existing excludes and ignore comments keep working.
  • A named argument defines its variable when the call is a global function call (FunctionCalls::isGlobalFunctionCall), PHP knows the function, its reflected parameter with that name is by-reference, and the variable is the whole argument. Positional arguments still go to the upstream code unchanged. An unknown parameter name, a method call, or an unknown function is still a read.
  • A read inside nested arrow functions now finds the outer arrow function that defines the variable.
  • CleanCode/ruleset.xml refers to the subclass by path, in place of the upstream code. The <type>, <exclude>s and <properties> are unchanged. composer.json is unchanged: no newer upstream release is needed.

Upstream status (AC10): sirbrillig/phpcs-variable-analysis v3.0.0 is the newest tag, and trunk is identical to it in both affected files. Upstream keys a named argument by its name, so its positional by-reference check never matches it, and it has no option for named arguments. For nested arrow functions, upstream #329 fixed a different shape (a read after a nested arrow function). Reading an outer arrow function's parameter inside an inner one is still reported on trunk, and there is no option for it. So this PR takes the subclass approach. Reporting both upstream is a separate step.

Docs: the multiline-strings reference states what a line break is. The UndefinedVariable reference and the README describe the subclass. CONTRIBUTING documents CleanCode/Overrides/.

Acceptance Criteria

  • AC1. utf8-at-limit.php: three strings at exactly three lines, with several υ, Ņ and U+2005 per line. No error.
  • AC2. utf8-over-limit.php: the same strings at four lines. One error each, and the message states 4. line-break-kinds.php: LF, CRLF and CR each count once. U+0085 counts, and U+2028 and U+2029 do not.
  • AC3. invalid-utf8.php: the over-limit string is reported as 5 lines, which is the byte-mode count including a lone 0x85. The at-limit string is not reported. A forced count failure reports nothing and raises no diagnostic. Note: StringLiteral decodes only \n, \r\n, \r and \t, so an escape such as "\xCF" reaches the sniff as four ASCII characters and cannot exercise this path. The fixtures carry raw Latin-1 bytes instead. The PHPCS tokenizer raises an iconv_strlen() notice while it measures columns in such a file, before any sniff runs and on main too. The phpcs command line does not show it. The tests assert no diagnostic other than that one.
  • AC4. RequireHeredocForStructuredText/utf8-line-breaks.php: one case per pattern, with υ directly before markdown, an INI section and setting lines. None is reported. The three controls, with a real newline in that place, are reported. invalid-utf8.php is classified byte by byte, as before.
  • AC5. No existing fixture or expected-error list is edited.
  • AC6. named-by-reference.php: preg_match, preg_match_all, str_replace, similar_text and parse_str, with the by-reference argument first, in the middle and last, one mixed call, and one fully qualified call. No error.
  • AC7. nested-arrow-functions.php: two levels, three levels, an arrow function in a closure with use, a method local, and a read inside a string. No error.
  • AC8. failing.php keeps the same lines and columns. still-undefined.php reports a by-value named argument, a read before the call, another undefined variable in the same call, an unknown parameter name, a by-value parameter name on another function, a method call, an unknown function, a non-variable argument, and a nested arrow function reading nothing defined. The unknown name raises no PHP error.
  • AC9. The fix ships in this package alone. The code is unchanged, the other VariableAnalysis codes are untouched, and the UnusedLocalVariable tests pass unedited.
  • AC10. See "Upstream status" above.
  • AC11. Pre-change commit: 02049c2 (main at 4dfd55a plus the empty start commit). The new tests, run against that source (test names as they were at that time):
    ⨯ it stays silent on UTF-8 text at the limit whose characters contain…
    ⨯ it reports UTF-8 text one line over the limit once, with its count
    ⨯ it never reads a byte inside a UTF-8 character as a line break
    ⨯ it treats a by-reference argument passed by name as a definition with ('named-by-reference.php')
    ⨯ it treats a by-reference argument passed by name as a definition with ('nested-arrow-functions.php')
    ⨯ it still reports every variable that nothing defines
    Tests:    8 failed, 52 passed (89 assertions)
    
    Before the change, utf8-at-limit.php reported 3 errors and utf8-line-breaks.php reported 6. named-by-reference.php reported 16 errors, and nested-arrow-functions.php reported 6. Every genuine report in still-undefined.php appeared before as well. That test failed only on the extra false positives this PR removes. AC2's Greek strings were reported before as well, but with counts of 14, 16 and 11 lines. The LF, CR, CRLF, NEL and invalid-UTF-8 cases give the same result before and after.
  • AC12. README.md, both named references and CONTRIBUTING are updated. No other statement in the repo outside vendor/ was false.
  • AC13. The full suite passes. composer lint and composer lint:self report no errors. No new phpcs:ignore. No mbstring function is used.

Test Plan

  • All existing tests pass (2991 in the full suite, unchanged files untouched)
  • tests/Support/StructuredTextTest.php forces preg_match_all to fail inside StructuredText. isMarkdown, isConfig and isSerialised then return false with no PHP diagnostic. Changing the fallback to return $count; turns all three cases red with a TypeError.
  • New tests cover the changes. Nine mutations of the new branches each turn the suite red: both LineBreaks patterns, the outer arrow function lookup, the global-call check, the whole-argument check, the by-reference check, the function_exists check, the named-argument lookup, and the failed-count guard.
  • vendor/bin/phpcs --standard=CleanCode run by hand on each new fixture gives the same results.

Fixes #397

@mikebronner
mikebronner marked this pull request as ready for review October 20, 2026 00:12

@mr-sherlock-holmes mr-sherlock-holmes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔄 Changes Requested

Two small blockers, both in code this PR added. Everything else is in good shape, and all 13 acceptance criteria are met.

Issues Found

  1. The new failure branch in StructuredText::lineMatches has no test. CleanCode/Support/StructuredText.php:100 returns 0 when preg_match_all fails. Nothing can force that failure. tests/PregOverrides.php gained a preg_match_all override only in the Sniffs\Strings namespace. StructuredText lives in MikeBronner\CleanCode\Support, so PregFailure cannot reach it. If someone changes line 100 to return $count;, the suite stays green. A real PCRE failure would then throw a TypeError under strict_types and crash phpcs.
    Missing test: add a MikeBronner\CleanCode\Support preg_match_all override to tests/PregOverrides.php, gated by pregArmed. Then write a unit test in tests/Support/ that arms PregFailure for preg_match_all and calls isMarkdown, isConfig and isSerialised on text that is structured. It asserts that each returns false and that no PHP diagnostic is raised.
    This repo bounced the same gap before: unexercised defensive branches in MemberOrderingSniff (PR #291).

  2. The new fixtures carry comments that no linter requires. The rule is no comments in code unless the project's linter or config requires them. This holds even where existing files are full of comments. phpcs.lint.xml excludes */fixtures/*, and no sniff under test reads comment tokens, so none of these is needed. That covers about 20 // lines across all nine new fixtures. Examples: MultilineStringsSniff/utf8-at-limit.php ("Greek upsilon, U+03C5, encodes as CF 85."), VariableAnalysisSniff/named-by-reference.php, and MultilineStringsSniff/invalid-utf8.php. The last one also quotes "\xCF", which reads as if the fixture used escapes.
    Fix: delete them, and update the line and column expectations that move. Where a case needs explaining (the invisible U+2005 spaces, the raw Latin-1 bytes), carry it in the test name or the dataset key.

What's Good

  • LineBreaks is correct. In valid UTF-8, 0xC2 is only ever a lead byte, so \xC2\x85 matches a real U+0085 and never the 0x85 inside υ, Ņ or U+2005. CRLF counts once because it is the first alternative. Keeping byte mode instead of /u is a good call, because /u would have changed what \w and \s match in StructuredText.
  • The sniff no longer takes a failed count as zero breaks, and that path is tested through PregFailure.
  • Fixtures assert exact lines, columns and messages, and they include the real-newline controls.
  • The VariableAnalysis subclass keeps the upstream sniff code, so existing excludes and ignore comments keep working. It also fails safe on every unresolved name.
  • AC3 is met by a different route than its wording. StringLiteral decodes only \r\n, \n, \r and \t (CleanCode/Support/StringLiteral.php:85), so "\xCF" never reaches the invalid-UTF-8 path. Raw bytes are the only way to exercise it. The iconv_strlen notice comes from the PHPCS tokenizer, it happens on main too, and no CleanCode code calls iconv. So filtering it out of the assertion hides nothing from the code under test.
  • CI is green on PHP 8.3, 8.4 and 8.5.

📋 Non-blocking follow-ups

  • namesReferenceParameter does not check ReflectionFunction::isInternal() (CleanCode/Overrides/VariableAnalysis/Sniffs/CodeAnalysis/VariableAnalysisSniff.php:84-95). ArrayAccessorsSniff.php:423-426 already does this. Adding it would stop a function loaded in the phpcs process from standing in for a project function. Disposition: Noted, not tracked.
  • The test "reports both cases through the upstream sniff alone" (tests/Ruleset/UndefinedVariableTest.php:181) only checks that some violation appears. Pinning UNDEFINED_VARIABLE at exact lines would make it stronger. The fixtures leave room for nothing else, so this is hardening, not a hole. Disposition: Noted, not tracked.

(Watson: fix the two blockers above. The follow-ups are optional. Do them if they are cheap while you are in there, or skip them.)

Please address the above and re-request review.

A preg_match_all override in the Support namespace lets PregFailure reach StructuredText. The new test checks that isMarkdown, isConfig and isSerialised return false, with no PHP diagnostic, when the match fails.
No linter requires them. The line expectations move with them, and the test names now say where a fixture holds invisible U+2005 spaces or raw Latin-1 bytes.
@dr-john-h-watson

Copy link
Copy Markdown

Both blockers are fixed, including every unit-belonging finding. I skipped the unrelated cosmetics.

  • tests/PregOverrides.php has a preg_match_all override in the Support namespace. The new tests/Support/StructuredTextTest.php forces the match to fail and checks that isMarkdown, isConfig and isSerialised return false with no PHP diagnostic. Changing the fallback to return $count; turns all three cases red with a TypeError.
  • All comment lines are gone from the nine new fixtures. The line expectations moved with them. The test names now say where a fixture holds invisible U+2005 spaces or raw Latin-1 bytes.

The two non-blocking follow-ups (isInternal() and pinning exact lines in the upstream-only test) are not in this PR.

@mr-sherlock-holmes mr-sherlock-holmes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔄 Changes Requested

Both blockers from the last round are fixed. One small blocker is left, in a doc paragraph this PR added. All 13 acceptance criteria are still met.

Issues Found

  1. The new paragraph in the multiline-strings reference breaks the file's wrap. resources/boost/skills/clean-code/references/code-style-multiline-strings-heredoc.md:34 is 110 characters ("…is never mistaken for NEL. Text that is not valid UTF-8 is read byte"). Lines 36 and 38 then wrap short, at 32 and 27 characters. The rest of this file, and nearly every file in that folder, wraps at about 80. The paragraph reads as if two lines were joined by an edit.
    Fix: re-wrap lines 30-38 at about 80 characters. No wording change is needed.

What's Good

  • Round-1 blocker 1 is fixed. The MikeBronner\CleanCode\Support preg_match_all override in tests/PregOverrides.php is loaded by tests/bootstrap.php before any test runs, so it reaches StructuredText::lineMatches. tests/Support/StructuredTextTest.php forces the failure and checks that isMarkdown, isConfig and isSerialised return false with no diagnostic. The positive cases in the same file show the texts really are structured, so the test can fail.
  • Round-1 blocker 2 is fixed. The new fixtures hold no comments, and no new comment appears anywhere in the diff. Every line and column expectation was checked by hand against the trimmed fixtures, and they all line up. The test names now carry the invisible U+2005 spaces and the raw Latin-1 bytes.
  • CI is green on PHP 8.3, 8.4 and 8.5.
  • The earlier notes on LineBreaks, the subclass approach and the AC3 raw-byte route still hold.

Checked and not blocking: the switch from preg_match to preg_match_all in StructuredText does not add new regex cost. main already had the same (?:^|\R)\s* shape, and on text that does not match, both functions try every offset. The AC4 sniff-level fixtures are the right tier, because they also prove that StringLiteral decodes the \n escapes. tests/Support/LineBreaksTest.php already covers the 0x85 rule at unit level.

📋 Non-blocking follow-ups

  • tests/Ruleset/UndefinedVariableTest.php:179-181 checks only that the upstream run reports something. That run loads only the upstream sniff, and the fixtures are clean apart from the bug, so nothing else can pass it. Pinning UNDEFINED_VARIABLE at exact lines would still make it more exact. Disposition: Noted, not tracked.
  • namesReferenceParameter does not check ReflectionFunction::isInternal(), from round 1. Disposition: Noted, not tracked.

(Watson: fix the one blocker above. The follow-ups are optional. Do them if they are cheap while you are in there, or skip them.)

Please address the above and re-request review.

@dr-john-h-watson

Copy link
Copy Markdown

I fixed the blocker and every finding that belongs to this unit. I skipped the unrelated cosmetics.

  • code-style-multiline-strings-heredoc.md lines 30-38 now wrap at about 80 characters. The wording did not change.
  • A sweep of every Markdown line this PR adds found one more instance. In CONTRIBUTING.md, line 217 of the Overrides paragraph was 84 characters long. That paragraph now wraps at 80, and its wording did not change either. The long README.md bullet is a one-line list item, the same as the bullets around it.

The two non-blocking follow-ups (isInternal(), and pinning exact lines in the upstream-only test) are still out of this PR. CI is green on PHP 8.3, 8.4 and 8.5.

@mr-sherlock-holmes mr-sherlock-holmes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approved

The round-2 blocker is fixed, and a full re-review of the current head found nothing new. All 13 acceptance criteria are met.

Review Summary

  • Round-2 blocker is fixed. The new paragraph in code-style-multiline-strings-heredoc.md now wraps at 80 like the rest of the file. The CONTRIBUTING paragraph was re-wrapped the same way. Neither commit changes any wording.
  • Each acceptance criterion is met. Two were met by a different route than their wording, and both were noted in round 1. AC3 uses raw bytes because StringLiteral does not decode \xCF. Both of its cases sit in one fixture file, and the test pins each of them.
  • One possible blocker was checked and dropped. StructuredText::lineMatches returns 0 when preg_match_all fails (CleanCode/Support/StructuredText.php:100). That is not a line-break count. It is a "does any line match" check, and a failure means "not structured", which is how main already behaved. AC4 asks for exactly that. AC3's "never zero breaks" rule covers the sniff's own count, and MultilineStringsSniff returns early on false there. tests/Support/StructuredTextTest.php pins the failure path.
  • The VariableAnalysis override runs no analysed code. function_exists does not autoload, and ReflectionFunction executes nothing.
  • No comments or docblocks were added. No phpcs:ignore was added. No mbstring function is used.
  • CI is green on PHP 8.3, 8.4 and 8.5.
  • Everything belonging to the coherent unit is clean.

📋 Non-blocking follow-ups

  • tests/Ruleset/UndefinedVariableTest.php:179-182 checks only that the upstream-only run reports something. Pinning UNDEFINED_VARIABLE at exact lines would make it more exact. Disposition: Noted, not tracked.
  • namesReferenceParameter does not check ReflectionFunction::isInternal(). This carries over from round 1. Disposition: Noted, not tracked.

Ready for @mikebronner to merge.

@mikebronner
mikebronner merged commit 17c0e3c into main Oct 20, 2026
3 checks passed
@mikebronner
mikebronner deleted the fix/397-multilinestrings-utf8-and-named-byref-args branch October 20, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MultilineStrings miscounts lines in UTF-8 text, and VariableAnalysis flags named by-reference arguments

1 participant