Repository navigation
Add test guidelines #1129
Description
Activity
@jku kindly provided some insights about test metadata in #870 (comment). They might be a good fit for a test guideline.
I personally find CodeCov to be next to useless
I will add to this issue to remove the requirement (which is seen in
test_updater.py) to use this format for test names:
"test_" + <id of the test in the file> + "_" + <function_to_be_tested_".Probably, they have decided to use that pattern before to have an alphabetical order of the tests in the order they are in
updater.py, but we have multiple tests with the nametest_2_*,test_1_*andtest_X_*as a whole.It's really annoying when you have to add a single test which is in the middle of the test file sequence how you have to change all test names below yours. Even more annoying is you are unlucky to place your test at the begging of the test sequence...
Reacted by Lukas Pühringer and Jussi Kukkonenthe
test_1_...style is typically used to achieve a specific order for the tests to run in (you might want that because the tests are borken so they require state from previous tests, but more often just because you want to bail out early if some fast preliminary tests fail and not spend time in slower more complex tests). This ordering actually fails in tuf because unittest of course sorts alphabetically: "test_10_" is run before "test_1_".I'd say there's probably no reason to use that numbering for new tests -- and there's no problem in multiple tests having the same number: the point is usually just to define a rough order (so all "test_1_..." get executed before all "test_2_...").
I'd say there's probably no reason to use that numbering for new tests -- and there's no problem in multiple tests having the same number: the point is usually just to define a rough order (so all "test_1_..." get executed before all "test_2_...").
Still, I think this is unnecessary and confusing and it's better if it's removed.
We don't need to execute our tests in a particular order.If the others agree with this, I can push a pr to rename the tests?
- added a commit that references this issue
on Dec 1, 2020 See lukpueh/code-style-guidelines@57ed297 and secure-systems-lab/code-style-guidelines#21 (comment) pp. for basic recommendations that used to be in the code style guide.
- added a commit that references this issue
on Dec 1, 2020 The code base is a lot more stable, compared to the time of creating this issue. This means:
- huge amounts of new test code are not expected
- existing test code can be used as guideline
The contribution docs say, new software features or changes must be unit tested, but the current test suite is a mix of unit-, integration-, system-, regression-, etc. tests.
It would be helpful to better structure different types of tests in the test suite, and to add some guidelines for testing.
Personally, I have chosen a pragmatic non-SWE approach, a la
"Tailor your tests towards checking the desired functionality of your code and not the coverage report!