Split the test suite for spec versions #20

Activity

1. asmeurer

   changed the title [-]Split test suite for spec versions[/-] [+]Split the test suite for spec versions[/+] on Sep 20, 2021
2. honno commented on Oct 5, 2021

   

   honno

   on Oct 5, 2021

   Member

   More actions

   Something useful I learnt recently—we can specify markers for all test methods in a module via assigning the pytestmark variable once per-file.

   pytestmark = [...]
3. honno commented on Oct 22, 2021

   

   honno

   on Oct 22, 2021

   Last edited by honno

   Member

   More actions

   Okay, a proposal. By default, test methods are assumed for all versions. If test methods do not cover every version, we pin the version via a mark.

   # No marker
   test_method_in_all_versions(...):
       ...
   
   @pytest.mark.xp_version("2021.12")
   test_method_only_in_one_version(...):
       ...
   
   @pytest.mark.xp_version(">2021.12")
   test_method_from_version(...):
       ...
   
   @pytest.mark.xp_version("<=2022.09")
   test_method_to_version(...):
       ...
   
   @pytest.mark.xp_version(">2022.03,<=2022.09")
   test_method_in_version_range(...):
       ...

   In the pytest collection hook, we can get the xp version from a custom option, fallbacking to something reasonable (e.g. only testing with the unmarked methods? assuming the most recent version?). Then we can just dynamically add the skip mark pytest.mark.skip to any test case that doesn't fit the (parsed) acceptable version range.
4. asmeurer commented on Oct 22, 2021

   

   asmeurer

   on Oct 22, 2021

   MemberAuthor

   More actions

   That sounds reasonable, but we're also going to want to have a way to do skipping inside of the test function, like

   if version == '2021.1':
       assert ...
   elif version == '2022.1':
       assert ...
   

   since I expect most changes will be minor things. Some changes will require separate functions, especially if they involve changing a function's signature.

   Relatedly, we should probably just split the stubs like function_stubs/2021.1/ and so on (I think it's better to just duplicate the stubs rather than trying to reuse code as that makes it easier for library implementers to copy-paste them).

   👍React with 👍1Reacted by Matthew Barber
5. honno

   added

   medium priorityMedium priority issueMedium priority issue

   on May 10, 2022
6. honno

   self-assigned this

   on Sep 30, 2022
7. honno commented on Sep 30, 2022

   

   honno

   on Sep 30, 2022

   Member

   More actions

   I'll formalise my thoughts later, just to say we added versioning support to Hypothesis in HypothesisWorks/hypothesis#3456, which I'll probably take inspiration from when working on this issue.
8. asmeurer

   closed this as completedin #165on Mar 13, 2023