Update documentation for v32

.github/PULL_REQUEST_TEMPLATE.md

@@ -17,7 +17,8 @@ Make sure these boxes are checked before your pull request (PR) is ready to be r
 * [ ] Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
   Run [tests](https://scancode-toolkit.readthedocs.io/en/latest/contribute/contrib_dev.html#running-tests) locally to check for errors. 
 * [ ] Commits are in uniquely-named feature branch and has no merge conflicts 📁
-
+* [ ] Looked for possible updates in documentation and added updates if applicable
+* [ ] Updated CHANGELOG.rst
 <!--
 We're happy to help you get this ready -- don't be afraid to ask for help, and **don't be discouraged**
 if your tests fail at first!

docs/Makefile

@@ -18,3 +18,4 @@ help:
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	./scripts/regen_package_docs.sh

docs/scripts/regen_package_docs.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# halt script on error
+set -e
+
+# Path to the available packages doc
+DOC_PATH="source/reference/available_package_parsers.rst"
+
+# Regen docs for available package parsers
+regen-package-docs --path "$DOC_PATH"
+
+# Delete whitespace from last line to not fail doc8 tests
+sed -i '$ d' "$DOC_PATH"

docs/source/contribute/contrib_dev.rst

@@ -198,3 +198,9 @@ Using ScanCode as a Python library
 ScanCode can be used alright as a Python library and is available as as a
 Python wheel in Pypi and installed with ``pip install scancode-toolkit`` or
 ``pip install scancode-toolkit-mini``.
+
+.. _note:
+
+   Since we do not pin dependencies to avoid dependency resolution conflicts
+   for downstream users, there are possibilities of issues arising from
+   dependencies silently changing API/functions which scancode uses.

docs/source/contribute/contrib_doc.rst

@@ -312,3 +312,15 @@ bugs. But this is definitely better than converting everything by yourself.
 
 This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for
 Sphinx/ReadTheDocs hosting.
+
+Automatic Docs Generation
+-------------------------
+
+It's possible to generate docs automatically from data by using a combination of:
+
+- `shell scripts: example <https://github.com/nexB/scancode-toolkit/blob/develop/docs/scripts/regen_package_docs.sh>`_
+- `python scripts: example <https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/regen_package_docs.py>`_
+- `jinja templates: example <https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/templates/available_package_parsers.rst>`_
+
+And we do this currently to keep a documentation page for all the supported package formats.
+See :ref:`supported_packages` for details.

docs/source/contribute/cut_new_release.rst

@@ -1,59 +1,68 @@
+.. _release_process:
+
 How to cut a new release:
 =========================
 
 Update version
 --------------
 
-- Run bumpversion with major, minor or patch to bump the version in
-  ``setup.cfg`` and ``src/scancode_config.py``. Note that this is CalVer.
+- Bump version to update major, minor or patch version in ``setup.cfg``
+  ``setup-mini.cfg`` and ``src/scancode_config.py``. Note that this is SemVer,
+  though we used CalVer previously, we have switched back to SemVer.
+
+- If scancode output data format is changed, increment manually the major,
+  minor or patch version to bump the version in ``src/scancode_config.py``.
+  Note that this is SemVer.
 
-- If scancode output data format is changed, increment manually the major, minor or patch
-  to bump the version in ``src/scancode_config.py``. Note that this is SemVer.
+See our :ref:``versioning`` for more details.
 
 Tag and publish
 ---------------
 
-- Update the CHANGELOG.rst
+- Changes for a release should also be pushed to a branch and a pull
+  Request should be created for it, for review.
+
+- Update the CHANGELOG.rst with detailed documentation of updates
+  and API/CLI option changes, or any significant changes.
 
-- Commit these changes and push changes to develop:
+- Commit these changes and push changes to develop (here we use an
+  example tag ``v1.6.1``):
 
-    - ``git commit -m "commit message"``
-    - ``git push --set-upstream origin develop``
+    - ``git commit -s``
+    - ``git push --set-upstream origin release-prep-v1.6.1``
 
-- Merge develop branch in master and tag the release.
+- Merge this ``release-prep-v1.6.1`` branch in master after review approval
+  and tag the release:
 
-    - ``git checkout master``
-    - ``git merge develop``
     - ``git tag -a v1.6.1 -m "Release v1.6.1"``
-    - ``git push --set-upstream origin master``
+    - ``git push --set-upstream origin release-prep-v1.6.1``
     - ``git push --set-upstream origin v1.6.1``
 
-- Draft a new release in GitHub, using the previous release blurb as a base. Highlight new and
-  noteworthy changes from the CHANGELOG.rst.
-
-Create Release on GitHub
-------------------------
-
-- Run ``etc/release/scancode_release.sh`` locally.
+Automated Release Process
+-------------------------
 
-- Upload the release archives created in the ``dist/`` directory to the GitHub release page.
+- We have a `automated release script <https://github.com/nexB/scancode-toolkit/actions/workflows/scancode-release.yml>`_
+  triggered by a pushed tag, where jobs run to:
 
-- Save the release as a draft. Use the previous release notes to create notes in the same style.
-  Ensure that the link to third-party source code is present.
+  - Build pypi wheels and sdist archives
+  - Build app release archives for linux/mac/windows
+  - This happens for all supported python versions
+  - Test these wheels and app archives in linux/mac/windows for all supported
+    versions of python
+  - Create a GitHub release (draft by default) with all wheels, sdists and app arcives
+    (for all os/python combinations)
+  - Upload sdists and wheels (all python versions) and publish a release
+    (This won't be a stable release for beta/release-candidate tags)
 
-- test the downloads.
+- Populate the draft GitHub release by clicking the ``Generate Release Notes`` button
+  and this pre-populates the release notes with PRs and contributors.
 
-- publish the release on GitHub
+- Add more details to the release notes talking about the key features and changes in the
+  release.
 
-Upload wheels to pypi and Test
-------------------------------
+- Publish the release on GitHub
+  (Note the ``Set as a pre-release`` vs ``Set as the latest release`` checkboxes)
 
-- then build and publish the released wheel on Pypi. For this you need your own Pypi credentials
-  (and get authorized to publish Pypi release: ask @pombredanne) and you need to have the ``twine``
-  package installed and configured.
+- Announce in public channels and chats about the release
 
-  - Build a ``.whl`` and source distribution with ``python setup.py release``
-  - Run twine with ``twine upload dist/``
-  - Once uploaded check the published release at https://pypi.python.org/pypi/scancode-toolkit/
-  - Then create a new fresh local virtualenv and test the wheel installation with:
-    ``pip install scancode-toolkit[full]``
+- Do test the release archives yourself.

docs/source/contribute/index.rst

@@ -11,4 +11,3 @@
    gsoc17_final_report
    gsoc19_final_report
    gsoc21_final_report
-   long_running_issues

docs/source/contribute/roadmap.rst

@@ -5,6 +5,14 @@ Roadmap
 
 This is a high level list of what we are working on and what is completed.
 
+.. ToDo: Move this to Discussions or wikis as this is not used or updated
+   regularly.
+
+.. _note:
+
+  This is not updated regularly, see the `milstones <https://github.com/nexB/scancode-toolkit/milestones>`_
+  instead for updated shorter and longer term roadmaps.
+
 Legend
 ------
 
@@ -26,7 +34,7 @@ Package manifest and dependency parsers
 - |white_large_square| RPM dependencies #649
 - |white_large_square| Windows Nuget dependencies #648
 - |white_check_mark| Bower packages #654
-- |clock1030| Python dependencies #653
+- |white_check_mark|  Python dependencies #653
 - |white_check_mark| CRAN
 - |white_check_mark| Plain packages
 - |white_large_square| other Java-related meta files (SBT, Ivy, Gradle, etc.)
@@ -39,7 +47,7 @@ License Detection
 
 - |white_check_mark| support and detect license expressions (code in https://github.com/nexB/license-expression)
 - |clock1030| support and detect composite licenses
-- |white_large_square| support custom licenses
+- |white_check_mark| support custom licenses
 - |white_large_square| move licenses data set to external separate repository
 - |white_check_mark| Improved unknown license detection
 - |white_check_mark| sync with external sources (DejaCode, SPDX, etc.)
@@ -61,8 +69,8 @@ Core features
 - |white_check_mark| scan plugins (e.g. plugins that run a scan to collect data)
 - |white_check_mark| support Python 3 #295
 - |clock1030| transparent archive extraction (as opposed to on-demand with extractcode)
-- |clock1030| scancode.yml configuration file for exclusions, defaults, scan failure conditions,
-  etc.
+- |white_large_square| scancode.yml configuration file for exclusions, defaults, scan
+  failure conditions, etc.
 - |white_large_square| support scan pipelines and rules to organize more complex scans
 - |white_check_mark| scan baselining, delta scan and failure conditions (such as license change,
   etc) ( spawned as its the `DeltaCode <https://github.com/nexB/deltacode/>`_ project)