Description
FileSplitterTask does not split the source file correctly:
- the first line of the source file is lost,
- every line break is doubled (each line is followed by an extra empty line),
- an extra empty file is emitted at the end of the iteration.
The produced files contain neither max_lines lines of the input nor the original content.
Reproduction
var/data/file_splitter.txt contains a\nb\nc\nd\ne\n:
clever_age_process:
configurations:
demo.file_splitter_content:
tasks:
split:
service: '@CleverAge\ProcessBundle\Task\File\FileSplitterTask'
options:
file_path: '%kernel.project_dir%/var/data/file_splitter.txt'
max_lines: 2
outputs: [ read_chunk ]
read_chunk:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
callback#1:
callback: 'file_get_contents'
callback#2:
callback: 'json_encode'
outputs: [ dump ]
dump:
service: '@CleverAge\ProcessBundle\Task\Debug\DebugTask'
$ printf 'a\nb\nc\nd\ne\n' > var/data/file_splitter.txt
$ bin/console cleverage:process:execute demo.file_splitter_content
Starting process 'demo.file_splitter_content'...
""b\n\nc\n\n""
""d\n\ne\n\n""
""""
Process 'demo.file_splitter_content' executed successfully
Expected: "a\nb\n", "c\nd\n", "e\n". Tested on main (c1a2804), PHP 8.5, Symfony 7.4.
Cause
Proposed fix
- Open the source without SplFileObject flags and without counting and rewinding, and read one line ahead to know whether another chunk remains.
- Write exactly
max_lines lines per file (the last one fewer), stripping only the trailing \r\n/\n of each line before writeLine(): no line is lost, content is preserved, line breaks are not doubled.
next() stops as soon as no line is left, so no empty file is emitted; an empty source file is skipped and produces no output.
- Update the reference documentation, add unit tests.
Requirements
Breaking changes
None: this fixes the content of the produced files (the task was introduced in v5.0, #174). Line breaks in produced files are written as PHP_EOL, and an empty source file now produces no output instead of an empty file.
Description
FileSplitterTaskdoes not split the source file correctly:The produced files contain neither
max_lineslines of the input nor the original content.Reproduction
var/data/file_splitter.txtcontainsa\nb\nc\nd\ne\n:Expected:
"a\nb\n","c\nd\n","e\n". Tested onmain(c1a2804), PHP 8.5, Symfony 7.4.Cause
FileSplitterTask::execute()opens the file withREAD_AHEADand callsSplFile::getLineCount(), whichrewind()s the file. WithREAD_AHEAD,rewind()already reads the first line, so the nextfgets()inSplFile::readLine()returns line 2: line 1 is lost.FileSplitterTask::splitFile()writes lines read withfgets(), which keeps the line break (DROP_NEW_LINEdoes not apply tofgets()), andSplFile::writeLine()appendsPHP_EOL: line breaks are doubled.FileSplitterTask::next()relies oneof(), which is only true after a read attempt past the last line, so one more (empty) file is produced.Proposed fix
max_lineslines per file (the last one fewer), stripping only the trailing\r\n/\nof each line beforewriteLine(): no line is lost, content is preserved, line breaks are not doubled.next()stops as soon as no line is left, so no empty file is emitted; an empty source file is skipped and produces no output.Requirements
Breaking changes
None: this fixes the content of the produced files (the task was introduced in v5.0, #174). Line breaks in produced files are written as
PHP_EOL, and an empty source file now produces no output instead of an empty file.