Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/batch_release_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,9 @@ jobs:
cut until the version bump and changelog on
`${{ needs.create_batch_release_branch.outputs.blocking_branch }}` are on
`main`. That release is already published, so these changes have to
land. If this PR can't be landed as-is, apply the same version bump and
changelog to `main` by hand.
land. If this PR can't be landed as-is, apply the same version bump,
changelog, and `{{NEXT_VERSION}}` placeholder replacements to `main` by
hand.
secrets:
BOT_TOKEN: ${{ secrets.FLUTTERGITHUBBOT_TOKEN }}

Expand Down
12 changes: 8 additions & 4 deletions script/tool/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
## NEXT
## 0.14.4+4

* Exempts `test_apps/` from requiring version and changelog changes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, when does the tool changelog actually bump?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm based on the history on Github and pub, it looks like it gets bumped after being published, and I'm guessing publishing is supposed to happen manually.

However, I noticed we are on [0.14.4+1](https://pub.dev/packages/flutter_plugin_tools/versions/0.14.4+1) on pub, 0.14.4+3 in the pubspec, and then had both 0.14.4+4 and NEXT entries in the changelog. I removed NEXT so now we only have 0.14.4+4 in the changelog, but probably we should establish a clearer process here around version bumps / publishing.

* Fixes `dart-test --package-tags` not matching any packages when the
repository is checked out into a directory named `packages`, as it is in CI.

## 0.14.4+4

* Updates `branches-for-batch-release` to replace `{{NEXT_VERSION}}`
placeholders in Dart files in `lib/` with the new release version.
* Updates `validate` to fail if `{{NEXT_VERSION}}` placeholders are present in
packages that don't use batch release, in packages without pending changelog
entries, or in files other than Dart files in `lib/`.
* Updates `publish` to refuse to publish packages containing `{{NEXT_VERSION}}`
placeholders.
* Uses `flutter pub` instead of `dart pub` for non-Flutter packages that depend
on `jni`.

Expand Down
72 changes: 69 additions & 3 deletions script/tool/lib/src/branches_for_batch_release_command.dart
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,16 @@ import 'package:pub_semver/pub_semver.dart';
import 'package:yaml_edit/yaml_edit.dart';

import 'common/core.dart';
import 'common/file_utils.dart';
import 'common/output_utils.dart';
Comment thread
elliette marked this conversation as resolved.
Resolved
import 'common/package_command.dart';
import 'common/pub_utils.dart';
import 'common/repository_package.dart';
import 'common/version_placeholder.dart';

const int _kExitPackageMalformed = 3;
const int _kGitFailedToPush = 4;
const int _kFormatFailed = 5;

/// A command to create remote branches with release changes for a single package.
class BranchesForBatchReleaseCommand extends PackageCommand {
Expand Down Expand Up @@ -211,10 +215,70 @@ class BranchesForBatchReleaseCommand extends PackageCommand {

_updatePubspec(package, releaseInfo.newVersion!);
_updateChangelog(package, releaseInfo);
await _stageAndCommitChanges(git, package, pendingChangelogFiles);
final List<File> filesWithPlaceholders = await _replaceVersionPlaceholders(
package,
releaseInfo.newVersion!,
);
await _stageAndCommitChanges(
git,
package,
pendingChangelogFiles,
additionalFiles: filesWithPlaceholders,
);
await _pushBranch(git, remoteName, branchName);
}

/// Replaces every occurrence of [nextVersionPlaceholder] in the Dart files in
/// `lib/` of [package] with [newVersion], returning the list of modified
/// files.
///
/// This allows contributors to reference the not-yet-known release version
/// (e.g., in deprecation messages) before the batch release determines it.
Future<List<File>> _replaceVersionPlaceholders(
Comment thread
elliette marked this conversation as resolved.
Resolved
RepositoryPackage package,
Version newVersion,
) async {
final modifiedFiles = <File>[];
for (final (:File file, :String content) in await findVersionPlaceholderFiles(
package.directory,
getFilesForPackage(package),
scope: VersionPlaceholderScope.replaceable,
)) {
file.writeAsStringSync(content.replaceAll(nextVersionPlaceholder, newVersion.toString()));
modifiedFiles.add(file);
}
if (modifiedFiles.isNotEmpty) {
print(' Replaced $nextVersionPlaceholder with $newVersion in:');
for (final file in modifiedFiles) {
print(' ${relativePosixPath(file, from: package.directory, platformContext: path)}');
}
Comment thread
elliette marked this conversation as resolved.
Resolved
await _formatDartFiles(package, modifiedFiles);
}
return modifiedFiles;
}

/// Runs `dart format` on [files] in [package].
///
/// Replacing [nextVersionPlaceholder] with a (shorter) version can allow
/// lines that were previously split to fit on one line, which would fail the
/// format check on the release PR.
Future<void> _formatDartFiles(RepositoryPackage package, List<File> files) async {
// The formatter's behavior depends on the package's language version, so
// make sure it has been resolved.
if (!await runPubGet(package, processRunner, platform)) {
printError('Unable to fetch dependencies for ${package.displayName}.');
throw ToolExit(_kFormatFailed);
}
final io.ProcessResult formatResult = await processRunner.run('dart', <String>[
'format',
...files.map((File f) => f.path),
], workingDir: package.directory);
if (formatResult.exitCode != 0) {
printError('Failed to format files: ${formatResult.stderr}');
throw ToolExit(_kFormatFailed);
}
}

void _updatePubspec(RepositoryPackage package, Version newVersion) {
final editablePubspec = YamlEditor(package.pubspecFile.readAsStringSync());
editablePubspec.update(<String>['version'], newVersion.toString());
Expand Down Expand Up @@ -250,8 +314,9 @@ class BranchesForBatchReleaseCommand extends PackageCommand {
Future<void> _stageAndCommitChanges(
GitDir git,
RepositoryPackage package,
List<File> pendingChangelogFiles,
) async {
List<File> pendingChangelogFiles, {
List<File> additionalFiles = const <File>[],
}) async {
final List<String> paths = pendingChangelogFiles.map((File f) => f.path).toList();
final io.ProcessResult rmResult = await git.runCommand(<String>['rm', ...paths]);
if (rmResult.exitCode != 0) {
Expand All @@ -263,6 +328,7 @@ class BranchesForBatchReleaseCommand extends PackageCommand {
'add',
package.pubspecFile.path,
package.changelogFile.path,
...additionalFiles.map((File f) => f.path),
]);
if (addResult.exitCode != 0) {
printError('Failed to git add: ${addResult.stderr}');
Expand Down
78 changes: 78 additions & 0 deletions script/tool/lib/src/common/version_placeholder.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
// Copyright 2013 The Flutter Authors
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:convert';

import 'package:file/file.dart';
import 'package:path/path.dart' as path;

/// Placeholder text that is replaced with the new release version in a batch
/// release package's Dart files in `lib/` when creating a batch release.
///
/// For example, a deprecation message can use
/// `'This feature was deprecated in v{{NEXT_VERSION}}.'`.
const String nextVersionPlaceholder = '{{NEXT_VERSION}}';

/// A text file containing [nextVersionPlaceholder].
typedef VersionPlaceholderFile = ({File file, String content});

/// The set of files to search for [nextVersionPlaceholder].
enum VersionPlaceholderScope {
/// Files in which the batch release tooling replaces placeholders: Dart files
/// in `lib/`.
replaceable,

/// All text files in the package, other than those in build output or tool
/// state directories. Used to find placeholders that won't be replaced.
all;

/// Whether the file at [relativePath], relative to the package root, is in
/// this scope.
bool includes(String relativePath, path.Context pathContext) {
final List<String> components = pathContext.split(relativePath);
switch (this) {
case VersionPlaceholderScope.replaceable:
return components.first == 'lib' && pathContext.extension(relativePath) == '.dart';
case VersionPlaceholderScope.all:
final List<String> directories = components.sublist(0, components.length - 1);
return !directories.any((String name) => name.startsWith('.') || name == 'build');
}
}
}

/// Returns the files in [files] that are in [scope] and contain
/// [nextVersionPlaceholder], along with their contents, in path order.
///
/// [files] should be the files of the package rooted at [packageDirectory],
/// as returned by `PackageCommand.getFilesForPackage`.
Future<List<VersionPlaceholderFile>> findVersionPlaceholderFiles(
Directory packageDirectory,
Stream<File> files, {
required VersionPlaceholderScope scope,
}) async {
final path.Context pathContext = packageDirectory.fileSystem.path;
final List<File> candidates =
await files
.where(
(File file) => scope.includes(
pathContext.relative(file.path, from: packageDirectory.path),
pathContext,
),
)
.toList()
..sort((File a, File b) => a.path.compareTo(b.path));
return <VersionPlaceholderFile>[
for (final File file in candidates)
if (_readUtf8(file) case final String content when content.contains(nextVersionPlaceholder))
(file: file, content: content),
];
}

String? _readUtf8(File file) {
try {
return utf8.decode(file.readAsBytesSync());
} on FormatException {
return null;
}
}
Comment thread
elliette marked this conversation as resolved.
Resolved
24 changes: 24 additions & 0 deletions script/tool/lib/src/publish_command.dart
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import 'common/package_looping_command.dart';
import 'common/pub_utils.dart';
import 'common/pub_version_finder.dart';
import 'common/repository_package.dart';
import 'common/version_placeholder.dart';

@immutable
class _RemoteInfo {
Expand Down Expand Up @@ -256,6 +257,10 @@ class PublishCommand extends PackageLoopingCommand {
return PackageResult.fail(<String>['uncommitted changes']);
}

if (!await _checkForVersionPlaceholders(package)) {
return PackageResult.fail(<String>['unreplaced $nextVersionPlaceholder placeholder']);
}

final bool tagOnly = getBoolArg(_tagForAutoPublishFlag);
if (!tagOnly) {
if (!await _publish(package)) {
Expand Down Expand Up @@ -411,6 +416,25 @@ Safe to ignore if the package is deleted in this commit.
return statusOutput.isEmpty;
}

/// Returns false, printing the offending files, if [package] contains any
/// [nextVersionPlaceholder]s.
Future<bool> _checkForVersionPlaceholders(RepositoryPackage package) async {
final List<VersionPlaceholderFile> placeholderFiles = await findVersionPlaceholderFiles(
package.directory,
getFilesForPackage(package),
scope: VersionPlaceholderScope.all,
);
if (placeholderFiles.isEmpty) {
return true;
}
printError(
'Refusing to publish a package containing $nextVersionPlaceholder '
'placeholders. Replace them with the release version:\n'
'${placeholderFiles.map((VersionPlaceholderFile f) => ' ${getRelativePosixPath(f.file, from: package.directory)}').join('\n')}',
);
return false;
}

Future<String?> _verifyRemote(String remote) async {
final io.ProcessResult getRemoteUrlResult = await (await gitDir).runCommand(<String>[
'remote',
Expand Down
1 change: 1 addition & 0 deletions script/tool/lib/src/validate_command.dart
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,7 @@ class ValidateCommand extends PackageLoopingCommand {

return validator.validateChangelogAndVersion(
package,
packageFiles: getFilesForPackage(package),
checkForMissingChanges: getBoolArg(_checkForMissingChanges),
ignorePlatformInterfaceBreaks: getBoolArg(_ignorePlatformInterfaceBreaks),
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import '../common/git_version_finder.dart';
import '../common/output_utils.dart';
import '../common/package_state_utils.dart';
import '../common/repository_package.dart';
import '../common/version_placeholder.dart';

/// Categories of version change types.
enum NextVersionType {
Expand Down Expand Up @@ -112,9 +113,13 @@ class VersionAndChangelogValidator {
/// and match the policy for the files changed, returning a list of resulting
/// error strings.
///
/// [packageFiles] should be all of the files in [package], and is used to
/// check for version placeholders.
///
/// If no errors are found, an empty list is returned.
Future<List<String>> validateChangelogAndVersion(
RepositoryPackage package, {
required Stream<File> packageFiles,
required bool checkForMissingChanges,
required bool ignorePlatformInterfaceBreaks,
}) async {
Expand Down Expand Up @@ -159,6 +164,15 @@ class VersionAndChangelogValidator {
}
}

final String? placeholderError = await _checkForVersionPlaceholders(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like _checkForVersionPlaceholders always allows {{NEXT_VERSION}} in lib/ whenever usesBatchRelease is true, even when the PR is the batch release PR itself (or when there are no version-bumping entries in pending_changelogs).

I see the comment in the test mentioning that the sync-to-main PR also uses the override: batch- label and might merge in new placeholders from main. However, in that sync-to-main case, any new placeholder on main would come with a new pending changelog entry in pending_changelogs/ (whereas on the batch release PR itself, pending_changelogs/ has just been emptied and the version in pubspec.yaml is being bumped for release).

If we allow {{NEXT_VERSION}} in lib/ when there are no version-bumping pending_changelogs entries (for example, on the batch release PR itself, or on a PR that only has a version: skip changelog), validate will pass in pre-submit and then publish will fail in post-submit when release_from_branches runs:

Details
  test('validate passes on batch release PR with unreplaced {{NEXT_VERSION}} in '
      'lib/ while publish fails on the same package state', () async {
    const thisPackageName = 'a_package';
    final RepositoryPackage package = createFakePackage(
      thisPackageName,
      packagesDir,
      version: '1.1.0',
    );
    package.ciConfigFile.writeAsStringSync('release:\n  batch: true\n');
    package.pendingChangelogsDirectory.createSync();
    package.changelogFile.writeAsStringSync('''
## 1.1.0

* Deprecate oldMethod.
''');
    package.libDirectory.childFile('a_package.dart').writeAsStringSync('''
@Deprecated('Deprecated in v{{NEXT_VERSION}}.')
void oldMethod() {}
''');

    gitProcessRunner.mockProcessesForExecutable['git-diff'] = <FakeProcessInfo>[
      FakeProcessInfo(
        MockProcess(
          stdout: '''
packages/a_package/pubspec.yaml
packages/a_package/CHANGELOG.md
packages/a_package/lib/a_package.dart
''',
        ),
      ),
    ];
    gitProcessRunner.mockProcessesForExecutable['git-show'] = <FakeProcessInfo>[
      FakeProcessInfo(MockProcess(stdout: 'version: 1.0.0')),
    ];

    final validateRunner = CommandRunner<void>('validate_test', 'test')
      ..addCommand(
        ValidateCommand(
          packagesDir,
          processRunner: processRunner,
          platform: mockPlatform,
          gitDir: gitDir,
          targetedValidators: <Validator>{Validator.version},
        ),
      );

    final List<String> validateOutput = await runCapturingPrint(validateRunner, <String>[
      'validate',
      '--base-sha=main',
      '--packages=$thisPackageName',
      '--pr-labels=override: batch-$thisPackageName',
    ]);
    expect(
      validateOutput,
      containsAllInOrder(<Matcher>[
        contains('Running for a_package'),
        contains('No issues found!'),
      ]),
    );

    final publishRunner = CommandRunner<void>('publish_test', 'test')
      ..addCommand(PublishCommand(packagesDir, processRunner: processRunner, gitDir: gitDir));

    Error? publishError;
    final List<String> publishOutput = await runCapturingPrint(
      publishRunner,
      <String>['publish', '--packages=$thisPackageName'],
      errorHandler: (Error e) {
        publishError = e;
      },
    );
    expect(publishError, isA<ToolExit>());
    expect(
      publishOutput,
      containsAllInOrder(<Matcher>[
        contains(
          'Refusing to publish a package containing {{NEXT_VERSION}} placeholders. Replace them with the release version:\n'
          '  lib/a_package.dart',
        ),
        contains('unreplaced {{NEXT_VERSION}} placeholder'),
      ]),
    );
  });

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good idea! We are now checking whether or not there is a pending changelog entry. If there is (e.g. on a feature PR opened against main, or the sync-to-main case) then version placeholders are allowed. If there isn't (e.g. on batch release PR) then version placeholders aren't allowed.

Currently it's only checking for a pending changelog entry, not whether or not the entry specifies skip or not. My thinking here was there might be a valid case to use a next version placeholder without needing a version bump, since version placeholders don't need to be restricted to only deprecation messages. But happy to add a non-skip check as well if that's preferred.

package,
packageFiles,
usesBatchRelease: usesBatchRelease,
);
if (placeholderError != null) {
errors.add(placeholderError);
}

final _CurrentVersionState versionState = await _getVersionState(
package,
pubspec: pubspec,
Expand Down Expand Up @@ -649,6 +663,82 @@ ${_indentation}The first version listed in CHANGELOG.md is $fromChangeLog.
return versionChanged;
}

/// Returns an error string if any of [packageFiles] contain a
/// [nextVersionPlaceholder] that won't be replaced automatically, printing
/// the offending files.
///
/// Placeholders are only replaced by the batch release tooling, and only in
/// [VersionPlaceholderScope.replaceable] files, so they are not allowed in
/// packages that don't use batch release, in PRs without a pending changelog
/// entry, or in any other files.
Future<String?> _checkForVersionPlaceholders(
RepositoryPackage package,
Stream<File> packageFiles, {
required bool usesBatchRelease,
}) async {
final bool hasPendingChangelogs = usesBatchRelease && _hasPendingChangelogs(package);
final path.Context fsPath = package.directory.fileSystem.path;
final placeholderPaths = <String>[
for (final VersionPlaceholderFile f in await findVersionPlaceholderFiles(
package.directory,
packageFiles,
scope: VersionPlaceholderScope.all,
))
if (!hasPendingChangelogs ||
!VersionPlaceholderScope.replaceable.includes(
fsPath.relative(f.file.path, from: package.directory.path),
fsPath,
))
relativePosixPath(f.file, from: package.directory, platformContext: _path),
];
if (placeholderPaths.isEmpty) {
return null;
}
final String reason = _versionPlaceholderErrorMessage(
usesBatchRelease: usesBatchRelease,
hasPendingChangelogs: hasPendingChangelogs,
);
printError(
'$_indentation$reason\n'
'${_indentation}Found $nextVersionPlaceholder in:\n'
'${placeholderPaths.map((String p) => '$_indentation $p').join('\n')}',
);
return 'Unreplaced $nextVersionPlaceholder placeholder.';
}

/// Returns the explanation for why unreplaced [nextVersionPlaceholder]s are
/// not allowed.
String _versionPlaceholderErrorMessage({
required bool usesBatchRelease,
required bool hasPendingChangelogs,
}) {
if (!usesBatchRelease) {
return 'This package does not use batch release, so $nextVersionPlaceholder '
'placeholders will not be replaced automatically. Use the actual '
'version instead.';
} else if (!hasPendingChangelogs) {
return 'Using $nextVersionPlaceholder is only allowed in PRs with a '
'pending changelog entry. To resolve, create a pending changelog entry '
'or remove $nextVersionPlaceholder instead.';
} else {
return '$nextVersionPlaceholder placeholders are only supported '
'in Dart files in lib/.';
}
}

/// Returns true if [package] has at least one valid pending changelog entry.
///
/// Entries with `version: skip` count, since their placeholders will be
/// replaced by the next release that does change the version.
bool _hasPendingChangelogs(RepositoryPackage package) {
try {
return package.getPendingChangelogs().isNotEmpty;
} on FormatException {
// Invalid pending changelogs are reported separately.
return false;
}
}

Future<String> _getRelativePackagePath(RepositoryPackage package) async {
return relativePosixPath(package.directory, from: _repoRoot, platformContext: _path);
}
Expand Down
Loading
Loading