Add dataset and composite required phrase commands

AUTHORS.rst

@@ -1,3 +1,4 @@
 The following organizations or individuals have contributed to this repo:
 
--
+- AboutCode.org contributors
+- Kaushik Kumar

CHANGELOG.rst

@@ -5,4 +5,7 @@ Changelog
 v0.0.0
 ------
 
-*xxxx-xx-xx* -- Initial release.
+*Unreleased*
+
+- Add required phrase dataset extraction.
+- Add composite rule required phrase updates.

MANIFEST.in

@@ -1,5 +1,6 @@
 graft src
 graft docs
+prune docs/_build
 graft etc
 
 include *.LICENSE

Makefile

@@ -4,7 +4,7 @@
 # ScanCode is a trademark of nexB Inc.
 # SPDX-License-Identifier: Apache-2.0
 # See http://www.apache.org/licenses/LICENSE-2.0 for the license text.
-# See https://github.com/aboutcode-org/skeleton for support or download.
+# See https://github.com/aboutcode-org/scancode-required-phrases for support or download.
 # See https://aboutcode.org for more information about nexB OSS projects.
 #
 

README.rst

@@ -1,107 +1,51 @@
-A Simple Python Project Skeleton
-================================
+ScanCode Required Phrases
+=========================
 
-This repo attempts to standardize the structure of the Python-based project's
-repositories using modern Python packaging and configuration techniques.
-Using this `blog post`_ as inspiration, this repository serves as the base for
-all new Python projects and is mergeable in existing repositories as well.
+``scancode-required-phrases`` provides commands for working with required
+phrases in ScanCode license rules. ScanCode Toolkit provides the rule models,
+tokenization, matching, and validation used by these commands.
 
-.. _blog post: https://blog.jaraco.com/a-project-skeleton-for-python-projects/
+Installation
+============
 
+Install this package in a Python 3.10 or newer environment:
 
-Usage
-=====
+.. code-block:: console
 
-A brand new project
--------------------
+    python -m pip install .
 
-.. code-block:: bash
+Build a dataset
+===============
 
-    git init my-new-repo
-    cd my-new-repo
-    git pull git@github.com:nexB/skeleton
+.. code-block:: console
 
-    # Create the new repo on GitHub, then update your remote
-    git remote set-url origin git@github.com:nexB/your-new-repo.git
+    build-required-phrases-dataset --rules-dir path/to/rules --output-dir dataset-output
 
-From here, you can make the appropriate changes to the files for your specific project.
+The command writes ``train.jsonl``, ``val.jsonl``, and ``test.jsonl``. If
+``--rules-dir`` is omitted, the installed ScanCode rules directory is used.
 
-Update an existing project
----------------------------
+Update composite rules
+======================
 
-.. code-block:: bash
+.. code-block:: console
 
-    cd my-existing-project
-    git remote add skeleton git@github.com:nexB/skeleton
-    git fetch skeleton
-    git merge skeleton/main --allow-unrelated-histories
+    add-composite-required-phrases --dry-run --verbose
 
-This is also the workflow to use when updating the skeleton files in any given repository.
+The command uses existing required phrases from single-key rules to update
+composite rules. A rule is updated only when every relevant license key has a
+non-overlapping match. It operates on the installed ScanCode rules directory.
 
-More usage instructions can be found in ``docs/skeleton-usage.rst``.
+Development
+===========
 
+Create a development environment and run the tests:
 
-Release Notes
-=============
+.. code-block:: console
 
-- 2025-03-31:
+    configure --dev
+    venv\Scripts\pytest
 
-    - Use ruff as the main code formatting tool, add ruff rules to pyproject.toml
+On POSIX systems, run ``./configure --dev`` and ``venv/bin/pytest``.
 
-- 2025-03-29:
-
-    - Add support for beta macOS-15
-    - Add support for beta windows-2025
-
-- 2025-02-14:
-
-    - Drop support for Python 3.8, add support in CI for Python 3.13, use Python 3.12 as default
-      version.
-
-- 2025-01-17:
-
-    - Drop support for macOS-12, add support for macOS-14
-    - Add support in CI for ubuntu-24.04
-    - Add support in CI for Python 3.12
-
-- 2024-08-20:
-
-    - Update references of ownership from nexB to aboutcode-org
-
-- 2024-07-01:
-
-    - Drop support for Python 3.8
-    - Drop support for macOS-11, add support for macOS-14
-
-- 2024-02-19:
-
-    - Replace support in CI of default ubuntu-20.04 by ubuntu-22.04
-
-- 2023-10-18:
-
-    - Add dark mode support in documentation
-
-- 2023-07-18:
-
-    - Add macOS-13 job in azure-pipelines.yml
-
-- 2022-03-04:
-
-    - Synchronize configure and configure.bat scripts for sanity
-    - Update CI operating system support with latest Azure OS images
-    - Streamline utility scripts in etc/scripts/ to create, fetch and manage third-party
-      dependencies. There are now fewer scripts. See etc/scripts/README.rst for details
-
-- 2021-09-03:
-    - ``configure`` now requires pinned dependencies via the use of ``requirements.txt``
-      and ``requirements-dev.txt``
-    - ``configure`` can now accept multiple options at once
-    - Add utility scripts from scancode-toolkit/etc/release/ for use in generating project files
-    - Rename virtual environment directory from ``tmp`` to ``venv``
-    - Update README.rst with instructions for generating ``requirements.txt``
-      and ``requirements-dev.txt``, as well as collecting dependencies as wheels and generating
-      ABOUT files for them.
-
-- 2021-05-11:
-    - Adopt new configure scripts from ScanCode TK that allows correct configuration of which
-      Python version is used.
+See ``docs/source/dataset.rst`` and ``docs/source/composite_rules.rst`` for
+details.

docs/source/composite_rules.rst

@@ -0,0 +1,34 @@
+Update composite rules
+======================
+
+The ``add-composite-required-phrases`` command adds existing required phrases
+to ScanCode rules with multi-key license expressions.
+
+Run the command
+---------------
+
+Preview changes without saving rules:
+
+.. code-block:: console
+
+    add-composite-required-phrases --dry-run --verbose
+
+Use ``--license-expression`` to process one expression. Use
+``--write-phrase-source`` to record the source required phrase rules in updated
+rules.
+
+Matching
+--------
+
+Candidates come from existing ``is_required_phrase`` rules with one
+non-generic license key. A composite rule is updated only when every relevant
+key has a non-overlapping phrase match. Existing required phrase markers are
+kept, and filenames and URLs are not changed.
+
+Writing and validation
+----------------------
+
+The command operates on the installed ScanCode rules directory. Without
+``--dry-run``, each changed rule is written once after all required phrases are
+added. Use ``--validate`` to validate the rules and licenses and ``--reindex``
+to rebuild the cached license index after updating.

docs/source/conf.py

@@ -17,7 +17,7 @@
 
 # -- Project information -----------------------------------------------------
 
-project = "nexb-skeleton"
+project = "ScanCode Required Phrases"
 copyright = "nexB Inc., AboutCode and others."
 author = "AboutCode.org authors and contributors"
 
@@ -79,9 +79,9 @@
 
 html_context = {
     "display_github": True,
-    "github_user": "nexB",
-    "github_repo": "nexb-skeleton",
-    "github_version": "develop",  # branch
+    "github_user": "aboutcode-org",
+    "github_repo": "scancode-required-phrases",
+    "github_version": "main",  # branch
     "conf_py_path": "/docs/source/",  # path in the checkout to the docs root
 }
 

docs/source/contribute/contrib_doc.rst

@@ -10,9 +10,9 @@ Setup Local Build
 
 To get started, check out and configure the repository for development::
 
-    git clone https://github.com/aboutcode-org/<your-repo>.git
+    git clone https://github.com/aboutcode-org/scancode-required-phrases.git
 
-    cd your-repo
+    cd scancode-required-phrases
     ./configure --dev
 
 (Or use "make dev")
@@ -55,7 +55,7 @@ documentation standards are enforced. We checks for these aspects of the documen
 You myst run these scripts locally before creating a pull request::
 
     make doc8
-    make check-docs
+    make docs-check
 
 
 .. _doc_style_docs8:

docs/source/dataset.rst

@@ -0,0 +1,43 @@
+Build a required phrase dataset
+===============================
+
+The ``build-required-phrases-dataset`` command reads ScanCode ``.RULE`` files
+and creates a BIOES-labelled JSONL dataset from required phrases marked with
+``{{ }}``.
+
+Run the command
+---------------
+
+.. code-block:: console
+
+    build-required-phrases-dataset \
+        --rules-dir path/to/rules \
+        --output-dir dataset-output
+
+``--rules-dir`` defaults to the installed ScanCode rules directory.
+``--output-dir`` defaults to ``dataset-output``.
+
+Output
+------
+
+The output directory contains:
+
+* ``train.jsonl``
+* ``val.jsonl``
+* ``test.jsonl``
+
+Each record contains the rule identifier, license expression, rule type,
+unmarked text, tokens, and BIOES labels.
+
+Splitting
+---------
+
+Rules with common license expressions are assigned by a deterministic hash of
+their identifiers. Rules with rarer expressions stay together in one split.
+The target proportions are 80 percent training, 10 percent validation, and
+10 percent test.
+
+Only eligible rules containing a marked required phrase are included.
+Dedicated ``is_required_phrase`` rules, deprecated rules, false positives,
+license clues, license introductions, and rules without a license expression
+are excluded.

docs/source/index.rst

@@ -1,11 +1,15 @@
-Welcome to nexb-skeleton's documentation!
-=========================================
+ScanCode Required Phrases
+=========================
+
+Commands for building required phrase datasets and updating composite ScanCode
+license rules.
 
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
-   skeleton-usage
+   dataset
+   composite_rules
    contribute/contrib_doc
 
 Indices and tables