From 83245383189c4f12d48325104f324996e1580a8d Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Mon, 6 Sep 2021 18:43:50 +0530 Subject: [PATCH 1/2] Fix doc lint errors Signed-off-by: Ayan Sinha Mahapatra --- .../source/contribute/gsoc21_final_report.rst | 120 +++++++++++------- docs/source/contribute/index.rst | 1 + 2 files changed, 75 insertions(+), 46 deletions(-) diff --git a/docs/source/contribute/gsoc21_final_report.rst b/docs/source/contribute/gsoc21_final_report.rst index 849f26d29c7..ecc2b543a82 100644 --- a/docs/source/contribute/gsoc21_final_report.rst +++ b/docs/source/contribute/gsoc21_final_report.rst @@ -1,13 +1,15 @@ Google Summer of Code 2021 Final report ========================================= + Organisation - `AboutCode `_ --------------------------------------------------------- -Akanksha Garg +Akanksha Garg `GITHUB `_ + Project: Detect Unknown Licenses and Indirect License References in Scancode ----------------------------------------------------------------------------- @@ -17,58 +19,84 @@ Project: Detect Unknown Licenses and Indirect License References in Scancode `Proposal `_ + Description ------------ -- The main motive of this project was to improve license detection of unknown licenses and follow references to indirect license references in Scancode-TK - - **Improvement in the License Data Model Definition** -- Unknown Licenses are the ones which are matched to a license rule tagged with 'unknown' license key . Since these are some of the 'special' licenses , reporting them with special attributes will - provide more clarification. Now unknown licenses are tagged with a new flag **"is_unknown"** to identify them beyond just the naming convention of having "unknown" as part of their name. - Rules that match at least one unknown license have a flag **"has_unknown"** set - in the returned match results. - - `nexB/scancode-toolkit#2548 `_ - - **Reporting known and Unknown licenses separately** -- We considered having a separate section for of scan results to report 'unknown licenses' separately and not mixed with main license detection results. But after implementing a separate section for - unknown ones ,it doesn't seem to be good idea to have currently. - - `nexB/scancode-toolkit#2578 `_ - - **Follow License References to another file** -- Some license references such as "see license in file LICENSE.txt" e.g. mentions to look for license details in another file are reported as unknown license references and we could instead follow - the referenced file to find what was detected there. The approach was to use already contained attribute ```refrenced_filenames``` in license RULE data files. Since this was a ```process_codebase``` - step in scan plugin , it was needed that our API function should return ```refrenced_filenames``` to keep track of these files corresponding to licenses detected. This was tracked in - - - `nexB/scancode-toolkit#2632 `_ - -- The ```process_codebase``` step is tracked in - - - `nexB/scancode-toolkit#2616 `_ - - **Improve license detection of Unknown Licenses** -- The approach was to use index of n-grams for detecting unknowns besides having our actual detection of "unknown" license rules. Firstly matches were filtered after running our normal procedure - of license detection and the remaining spans are run through a automaton index containing n-grams from all regular license texts and rules. This is tracked in- - - `nexB/scancode-toolkit#2592 `_ - - **Addition of some new Licenses** -- There were some licenses that were not present in Scancode-toolkit as for now. They have been added now. - - `nexB/scancode-toolkit#2625 `_ - - + +The main motive of this project was to improve license detection of unknown licenses +and follow references to indirect license references in Scancode-TK + +**Improvement in the License Data Model Definition** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Unknown Licenses are the ones which are matched to a license rule tagged with 'unknown' license +key. Since these are some of the 'special' licenses , reporting them with special attributes +will provide more clarification. Now unknown licenses are tagged with a new flag **"is_unknown"** +to identify them beyond just the naming convention of having "unknown" as part of their name. + +Rules that match at least one unknown license have a flag **"has_unknown"** set +in the returned match results. + +`nexB/scancode-toolkit#2548 `_ + +**Reporting known and Unknown licenses separately** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We considered having a separate section for of scan results to report 'unknown licenses' +separately and not mixed with main license detection results. But after implementing +a separate section for unknown ones ,it doesn't seem to be good idea to have currently. + +`nexB/scancode-toolkit#2578 `_ + +**Follow License References to another file** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Some license references such as "see license in file LICENSE.txt" e.g. mentions to look +for license details in another file are reported as unknown license references and +we could instead follow the referenced file to find what was detected there. The approach +was to use already contained attribute ``refrenced_filenames`` in license RULE data files. +Since this was a ``process_codebase`` step in scan plugin , it was needed that our API function +should return ``refrenced_filenames`` to keep track of these files corresponding to licenses +detected. This was tracked in - + +`nexB/scancode-toolkit#2632 `_ + +The ```process_codebase``` step is tracked in - + +`nexB/scancode-toolkit#2616 `_ + +**Improve license detection of Unknown Licenses** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The approach was to use index of n-grams for detecting unknowns besides having our actual +detection of "unknown" license rules. Firstly matches were filtered after running our normal +procedure of license detection and the remaining spans are run through a automaton index +containing n-grams from all regular license texts and rules. This is tracked in - + +`nexB/scancode-toolkit#2592 `_ + +**Addition of some new Licenses** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There were some licenses that were not present in Scancode-toolkit as for now. +They have been added now. + +`nexB/scancode-toolkit#2625 `_ + + Pre-GSoC ----------- - +-------- + **Contributions** - + - `nexB/scancode-toolkit#2423 `_ - `nexB/scancode-toolkit#2473 `_ - `nexB/scancode-toolkit#2464 `_ - `nexB/scancode-toolkit#2381 `_ - -I’ve had a wonderful summer during these 10 weeks journey and have learned plenty of things. I am thankful to Google and Aboutcode for giving me this opportunity to work with such an amazing -community. I am fortunate to have mentors `Philippe Ombredanne `_ and `Ayan Sinha Mahapatra `_ who helped me a lot throughout + +I’ve had a wonderful summer during these 10 weeks journey and have learned plenty of things. +I am thankful to Google and Aboutcode for giving me this opportunity to work with such an amazing +community. I am fortunate to have mentors `Philippe Ombredanne `_ +and `Ayan Sinha Mahapatra `_ who helped me a lot throughout my GSoC project and provided constant support. diff --git a/docs/source/contribute/index.rst b/docs/source/contribute/index.rst index be58f651585..69d83fe7ecc 100644 --- a/docs/source/contribute/index.rst +++ b/docs/source/contribute/index.rst @@ -10,4 +10,5 @@ roadmap gsoc17_final_report gsoc19_final_report + gsoc21_final_report long_running_issues From 534cd4eeb5fb4e4b2a5296e25cf5906750d2ada4 Mon Sep 17 00:00:00 2001 From: Ayan Sinha Mahapatra Date: Mon, 6 Sep 2021 22:34:05 +0530 Subject: [PATCH 2/2] Add CI for docs and ABOUT files Adds GitHub actions CI for docs and about files. Signed-off-by: Ayan Sinha Mahapatra --- .github/workflows/ci-about-files.yml | 39 ++++++++++++++++++++++++++++ .github/workflows/ci-docs.yml | 37 ++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100644 .github/workflows/ci-about-files.yml create mode 100644 .github/workflows/ci-docs.yml diff --git a/.github/workflows/ci-about-files.yml b/.github/workflows/ci-about-files.yml new file mode 100644 index 00000000000..27545625517 --- /dev/null +++ b/.github/workflows/ci-about-files.yml @@ -0,0 +1,39 @@ +name: CI About Files + +on: [push, pull_request] + +jobs: + build: + runs-on: ubuntu-20.04 + + strategy: + max-parallel: 4 + matrix: + python-version: [3.7] + + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python-version }} + + - name: Configure VirtualEnv + run: ./configure --dev + + - name: Activate VirtualEnv + run: source bin/activate + + #- name: Check About Files in thirdparty + # run: ./bin/about check thirdparty/ + + - name: Check About Files in thirdparty + run: ./bin/about check src/ + + - name: Check About Files in thirdparty + run: ./bin/about check etc/ + + - name: Check About Files in thirdparty + run: ./bin/about check scancode-toolkit.ABOUT diff --git a/.github/workflows/ci-docs.yml b/.github/workflows/ci-docs.yml new file mode 100644 index 00000000000..a0ea74d3458 --- /dev/null +++ b/.github/workflows/ci-docs.yml @@ -0,0 +1,37 @@ +name: CI Documentation + +on: [push, pull_request] + +jobs: + build: + runs-on: ubuntu-20.04 + + strategy: + max-parallel: 4 + matrix: + python-version: [3.7] + + steps: + - name: Checkout code + uses: actions/checkout@v2 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python-version }} + + - name: Give permission to run scripts + run: chmod +x ./docs/scripts/doc8_style_check.sh + + - name: Install Dependencies + working-directory: ./docs + run: pip install -r requirements.txt + + - name: Check Sphinx Documentation build minimally + working-directory: ./docs + run: sphinx-build -E source build + + - name: Check for documentation style errors + working-directory: ./docs + run: ./scripts/doc8_style_check.sh +