Repository navigation
Giving file_or_dir twice collects tests twice #1187
Description
When passing the same directory twice to py.test, all tests are run twice. The same is true if passing a sub-directory and a parent-directory:
$ python3 -m py.test --collect-only tests/ | grep collected
collecting ... collected 338 items
$ python3 -m py.test --collect-only tests/ tests/ | grep collected
collecting ... collected 676 items
$ python3 -m py.test --collect-only tests/unit/ tests/ | grep collected
collecting ... collected 418 items
platform linux -- Python 3.4.3, pytest-2.8.3, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python3
Rationale: I want to run the unit-test (which are in a certain directory) prior to he other ones. Thus I put the tests/unit directory in front, but this gave me duplicate tests.
Reacted by Daniel Hahler and Kyle King
Activity
nicoddemus commentedon Nov 23, 2015
Thanks for the report @htgoebel!
I had used this behavior of collecting tests more than once from the command-line in order to run the same test multiple times to track down some flaky behavior, but I think my use-case is a little contrived and yours makes more sense IMO.
Any more thoughts, @hpk42 @RonnyPfannschmidt @The-Compiler?
The-Compiler commentedon Nov 23, 2015
I usually do something like
while (($? == 0)); do py.test -k 'test_foo'; donein that case... I guess I should write apytest-repeatplugin 😆I don't really have an opinion on this, I can find arguments for and against both behaviours, and I'd be fine with both.
hpk42 commentedon Nov 24, 2015
don't have a strong opinion either but if in doubt i'd rather not change the default and better introduce a
--drop-duplicateoption or so. Maybe it's even better to write apytest-drop-duplicatesplugin which, when installed, will drop duplicates? It's easy to implement through thepytest_collect_modifyitemshook, basically a plugin with 20 lines of code or so.RonnyPfannschmidt commentedon Nov 24, 2015
I'm for the plugin solution
htgoebel commentedon Nov 24, 2015
Plugin sounds complicated. So I'm for a plugin
keep-duplicates.RonnyPfannschmidt commentedon Nov 24, 2015
a propperly separate
keep-duplicatesplugin would be impossible to do cleanly since the duplicates would be gone by the time it gets to the pluginnicoddemus commentedon Nov 24, 2015
Making a plugin is actually really easy using the cookiecutter-pytest-plugin, in fact just made one myself to quickly see it in action.
@htgoebel, could you try it out? If all is well I will publish it to PyPI, it still needs more tests and README. To install it use:
No other configuration is necessary.
RonnyPfannschmidt commentedon Nov 24, 2015
i'm closing this as working as intended given that @nicoddemus already finished a plugin
nicoddemus commentedon Nov 24, 2015
Btw kudos to @hackebrot, it took me 10 minutes to have that plugin working, well done! cookiecutter-pytest-plugin makes it really simple to create new plugins, covering a lot of the boiler plate of setting up a new repository, setup.py, C.I., etc etc. Highly recommended. 👍
htgoebel commentedon Nov 24, 2015
@nicoddemus The plugin does not show any effect here. I still get duplicates.
nicoddemus commentedon Nov 24, 2015
Can you show me the output of a sample session?
htgoebel commentedon Nov 24, 2015
It is the same as in my very fist comment here. I used
--traceconfigto verified the plugin is loaded:hackebrot commentedon Nov 24, 2015
Thank you @nicoddemus! 🙇 Happy to hear that 😸
nicoddemus commentedon Nov 24, 2015
@htgoebel hmm that's strange. I will investigate this later (most probably tomorrow). I moved over this discussion over to nicoddemus/pytest-drop-dup-tests#2
foxx commentedon Jan 5, 2016
Just came up against this problem, I'm confused why this functionality has to be placed into a separate plugin, why can't it be submitted as a
pytestpatch? Surely collecting tests twice, when given a directory to look for tests, would be considered a bug?nicoddemus commentedon Jan 6, 2016
Mostly because it is a behavior change and no one seemed to have a strong opinion for it... the change is trivial, as the plugin demonstrates. If more people are in favor of changing the behavior, I would gladly merge it into the core. 😄
foxx commentedon Jan 6, 2016
Ah okay. I'm struggling to think of a scenario where allowing duplicates would be considered an advantage. Can someone give me an example of where this behaviour might be desired? If not, I'd vote for re-opening this for merge into the core.
PS) It turns out my duplication issue was caused by different bug, which I've raised in another issue.
The-Compiler commentedon Jan 6, 2016
@foxx Check the history of this discussion - e.g. to run the same test multiple times to track down some flaky behaviour.
foxx commentedon Jan 6, 2016
I might have misinterpreted the use case, but I'd argue that if a test suite is producing different results on each run, with no actual change to any of the code, then the tests are broken.
That being said, most of the GNU core utils do not have any sort of de-duplication, for example if you run
cat /etc/issue /etc/issuethe file will be outputted twice. The same goes forfind tests/ tests/, andls /etc/passwd /etc/passwd.In fact it could be argued that changing the behaviour to be different than the standard of most other utilities, would be the wrong thing to do.
In my situation, the problem was caused by a bug with
setup.cfghandling (see #1311), and de-duplication would actually have hidden the problem, rather than fix it.On that basis, I'm changing my vote to -1
The-Compiler commentedon Jan 7, 2016
@nicoddemus FYI, for your usecase: I just wanted to write a
pytest-repeatand then noticed it already exists 😆nicoddemus commentedon Jan 7, 2016
😆 thanks
LewisGaul commentedon May 23, 2020
FWIW I'd like to see this implemented in core pytest, so just giving my +1 here.
See nicoddemus/pytest-drop-dup-tests#5 for my use-case and issue with the plugin approach (although thanks to niccodemus for making the plugin usable for my case!).