Repository navigation
Make --strict imply --strict-config / xfail_strict? #7503
Description
With pytest 4.5.0 (May 2019), we introduced --strict-markers as a replacement for --strict, and said:
The existing
--strictoption has the same behavior currently, but can be augmented in the future for additional checks.
With #7286, now we also have --strict-config. Yet, --strict is still an alias for --strict-markers.
Given that pytest 6 is allowed to do breaking changes, wouldn't it make sense for --strict to imply both --strict-markers and --strict-config now?
(Though, on a second thought: Do we actually need --strict-markers in pytest 6 still, given the warning system? Probably still useful as a convenient way to turn those warnings into errors?)
This would also make pytest more in line with other tools with a --strict flag - mypy and tsc (the typescript compiler) come to mind, where --strict implies various different strict flags.
cc @gnikonorov (who added --strict-config), @webknjaz and @DahlitzFlorian (some existing discussion in #7233), and @nicoddemus (#5023) - what do you think?
Activity
webknjaz commentedon Jul 15, 2020
I think it's a good idea to have some
--strict-all-the-thingsthat would imply enabling all strictness-related settings. What I'm not sure about is how it should be called. I'm okay with just--strictbut Bruno mentioned earlier that it used to mean just "strict-markers" for a while so such semantics change may be confusing. OTOH personally I can live with that and I'm not really concerned about this name. I think it should be decided by the folks who feel differently.webknjaz commentedon Jul 15, 2020
The only thing I want to add is that I'd probably expect it to also be a config value rather than just a CLI option.
Zac-HD commentedon Jul 15, 2020
I like the idea of
--strictorstrict = Trueimplying all the strictness options.Whether we still need it with the warnings system or not, I think we should keep it until pytest 7 rather than scrambling to make such a large change now.
DahlitzFlorian commentedon Jul 15, 2020
I really like the idea of using
--strictto activate all--strict* options. However, I would suggest adding a note to the documentation that--strictwill have that behavior in version x.x and people should be aware of it. Of course, we already added a note that it might happen, but an explicit note that it is going to happen in version x.x is highly preferred.IMHO we can discuss whether we need to wait until version 7 or can include it in some minor version of pytest 6.
symonk commentedon Jul 15, 2020
Should we consider giving the user the option?
--strict=all | --strict=marker | --strict=configIm just wondering if we add future 'strictness' but have then coupled the user into all or none situations.
The-Compiler commentedon Jul 15, 2020
@symonk We already have
--strict-markersand--strict-config, I don't think we should have a second way to do the same thing.DahlitzFlorian commentedon Jul 15, 2020
Totally agree with @The-Compiler. In fact,
--strictshould be a shortcut for using all--strict* options.gnikonorov commentedon Jul 15, 2020
No objections from me at all.
--strictshould imply all strict flags IMO, as @DahlitzFlorian mentions above.The-Compiler commentedon Jul 22, 2020
I opened a PR: #7529
graingert commentedon Jul 22, 2020
Can we flip the default here so markers default to strict in v6?
What about xfail_strict
The-Compiler commentedon Jul 22, 2020
We already have a warning for that since 4.5.0, so it should be quite clear what's going on even without
--strict-markers. -1 on making that the default now, that seems a bit late.That one is much more controversial and I'm not sure if starting the discussion again will produce a different outcome. See #1299 and #814.
graingert commentedon Jul 22, 2020
From @nicoddemus seems like this is exactly that future release
graingert commentedon Jul 22, 2020
I think if we add strict-markers and strict-config to
--strictwe should also have it coverxfail_strict14 remaining items
nicoddemus commentedon Jul 23, 2020
Good point, but I think we should rather close #7233 and keep this one open, as I believe this discussion is more through and brought more ideas on how to move things forward. 👍
[-]pytest 6: Make --strict imply --strict-config?[/-][+]Make --strict imply --strict-config / xfail_strict?[/+]graingert commentedon Jul 5, 2021
what about passing a pypi major version to strict:
--strict=7any new strictness settings will be graded with the major version they are added in, then--strictwill enable any flags that are greater than that version.Then people can pass
--strict=infto opt into all strictness settingsgraingert commentedon Jul 5, 2021
then you can keep
--strictwith no=deprecated foreverThe-Compiler commentedon Jul 5, 2021
I'm not sure what problem that would solve. Major releases come with some breakage, so projects likely need to adapt a little anyways - I don't see an issue with making
--strictimply ~all strict settings and adding to it, especially in major releases. This just seems like very unusual UX and a solution without a problem IMHO 😉graingert commentedon Jul 5, 2021
Usually new strict options get added in minor releases because they're not a breaking change. If we made
--stricta recommendation then release managers would have to delay every new flag to a new major releaseRonnyPfannschmidt commentedon Jul 6, 2021
Having new strict flags trigger warnings until the next major release seems like a good idea anyway
Having strict simply error on all from the last major release and warn on the new ones seems like a nice plan to me
The-Compiler commentedon Jul 6, 2021
I don't follow. The whole point of the strict flags (at least the ones we have so far) is to turn warnings into errors?
graingert commentedon Jul 6, 2021
You're saying Pytest 7 should just enable strict-config and strict-markers by default?
The-Compiler commentedon Jul 6, 2021
I'm not sure where you see me saying that. I'm saying
--strictshould imply both--strict-configand--strict-markers, with us being free to add more strictness settings over time if needed.graingert commentedon Jul 6, 2021
My usecase/workflow is that I want to opt into all breaking warnings and exceptions and strictness settings and then dependabot will make a new PR where I manually disable any errors I can't fix immediately
The-Compiler commentedon Jul 6, 2021
For the unlikely situation that
--strictwith a new pytest version implies--strict-frobnicationand you can't fix that right away, you could always replace--strictby--strict-config --strict-markerstemporarily. It's not like we're going to have hundreds of new strict options suddenly. I feel like we're thoroughly overengineering something simple at this point.