diff --git a/docs/source/aboutcode-data/abcd.rst b/docs/source/aboutcode-data/abcd.rst index 8d45de47..f187303e 100644 --- a/docs/source/aboutcode-data/abcd.rst +++ b/docs/source/aboutcode-data/abcd.rst @@ -1,6 +1,6 @@ .. _aboutcode_data: -AboutCode Data : ABCD +AboutCode Data : ABCD ===================== Summary @@ -57,7 +57,7 @@ technology-specific. Recently there have been efforts to collect and expose more data such as: -- libraries.io (a catalog of packages, AGPL-licensed) +- libraries.io (a catalog of packages, AGPL-licensed) and dependencyci.com its companion commercial service, - versioneye.com (a catalog of package versions updates, now MIT-licensed), @@ -76,9 +76,9 @@ consistently exchange data about code in a user-centric and technology-neutral, normalized way. Why does this matter? Software and code are everywhere. FLOSS code is -exploding with millions of components and packages.  The data about this -code is out there somewhere but getting it is harder than needed. This a -problem of data normalization, aggregation and exchange.   +exploding with millions of components and packages. The data about this +code is out there somewhere but getting it is often too difficult. This +is a problem of data normalization, aggregation and exchange. Whether you consume or produce software, accessing and creating normalized data about your code and the code you use should be made @@ -291,12 +291,12 @@ Name conventions {"http://someurl.com": "this is the home URL"} - Use rather this form to specify a name for the URL attribute:: - + {"url": "http://someurl.com", "note": "this is the home URL"} - But this would be correct when using a list of plain values where "urls" is plural:: - + {"urls": ["http://someurl.com", "http://someurl2.com"]} - An attribute names without a value is not needed. Only names with @@ -319,7 +319,7 @@ Name conventions - urn: uniform resource name - vcs: version control system - uuid: universally unique identifier, used for uuid4 string - `https://tools.ietf.org/html/rfc4122.html `__  + `https://tools.ietf.org/html/rfc4122.html `__ - id: identifier - info: information - os: operating system @@ -351,7 +351,7 @@ Well known attribute names include: other objects. For instance a Party may be the "owner" or "author" of a Component or Package. - uuid: a uuid4 string - `https://tools.ietf.org/html/rfc4122.html `__  + `https://tools.ietf.org/html/rfc4122.html `_ - algorithms for checksums: to store checksums we use a name/value pairs where the name is an algorithm such as sha1 and the value is a checksum in hexadecimal such as "sha1": "asasa231212" . The value is @@ -441,10 +441,10 @@ Hence each object may need several attributes to be fully identifiable. For example, public package managers ensure that a name is unique within the confines of a source. "logging" is the unique name of a single -Sourceforge project at -`https://sourceforge.net/projects/logging/ `__ . +Sourceforge project at +`https://sourceforge.net/projects/logging/ `_. "logging" is the unique name of an Apache project at the Apache -Foundation `http://logging.apache.org/ `__ . +Foundation `http://logging.apache.org/ `_. Yet, these two names point to completely different software. In most cases, providing information about the "source" where an identifier is @@ -535,7 +535,7 @@ lists. When embedding, you include not only the reference but also the related object details in another object data. This could include all data about an object or a subset as needed. -For example, this components list embeds a list of two packages. +For example, this components list embeds a list of two packages. Note: "components" is always a list, *even when it has a single component*:: @@ -544,19 +544,19 @@ Note: "components" is always a list, *even when it has a single component*::     "name": "Apache httpd",     "version": "2.3",     "packages": [ -         {"name": "httpd", - "version": "2.3.4", +         {"name": "httpd", + "version": "2.3.4", "download_url": "http://apache.org/dist/httpd/httpd-2.3.4.zip", -         "sha1": "acbf23256361abcdf", - "size": 3267, +         "sha1": "acbf23256361abcdf", + "size": 3267, "filename": "httpd-2.3.4.zip" - },         - -         {"name": "httpd", + }, + +         {"name": "httpd", "version": "2.3.5", "download_url": "http://apache.org/dist/httpd/httpd-2.3.5.tar.gz", -          "sha1": "ac8823256361adfcdf", - "size": 33267, +          "sha1": "ac8823256361adfcdf", + "size": 33267, "filename": "httpd-2.3.5.tar.gz" }     ] @@ -579,14 +579,14 @@ organized differently::         {"sha1": "ac8823256361adfcdf"}     ] }], - + "packages": [ -     {"name": "httpd", "version": "2.3.4", +     {"name": "httpd", "version": "2.3.4", "download_url": "http://apache.org/dist/httpd/httpd-2.3.4.zip",      "sha1": "acbf23256361abcdf", "size": 23267, "filename": "httpd-2.3.4.zip"}, - -     {"name": "httpd", "version": "2.3.5", + +     {"name": "httpd", "version": "2.3.5", "download_url": "http://apache.org/dist/httpd/httpd-2.3.5.tar.gz",      "sha1": "ac8823256361adfcdf", "size": 33267, "filename": "httpd-2.3.5.tar.gz"} ]} @@ -597,15 +597,15 @@ component detail data is not provided. The details may be available elsewhere in a tool that tracks components:: "packages": [ -     {"name": "httpd", "version": "2.3.4", +     {"name": "httpd", "version": "2.3.4", "download_url": "http://apache.org/dist/httpd/httpd-2.3.4.zip",      "sha1": "acbf23256361abcdf", "size": 23267, "filename": "httpd-2.3.4.zip",      "components": [ {"source": "http://apache.org", "name": "Apache httpd", "version": "2.3"} ]     }, - -     {"name": "httpd", "version": "2.3.5", + +     {"name": "httpd", "version": "2.3.5", "download_url":"http://apache.org/dist/httpd/httpd-2.3.5.tar.gz",      "sha1": "ac8823256361adfcdf", "size": 33267, "filename": "httpd-2.3.5.tar.gz",      "components": [ @@ -833,8 +833,8 @@ writing primarily by humans. - Multi-document YAML documents should be avoided (when using the --- separators). -- Beware of parser shenanigans: Most YAML parsers recognize and convert - automatically certain data types such as numbers, booleans or dates. +- Beware of parser shenanigans: Most YAML parsers recognize and convert + automatically certain data types such as numbers, booleans or dates. You should be aware of this because the ABC Data strings may contain date stamps. You may want to configure a YAML parser to deactivate some of these automated format conversions to avoid unwanted conversions. @@ -973,17 +973,17 @@ in YAML::         - path: bitarray-0.8.1-cp27-cp27mu-manylinux1_x86_64.whl         - path: bitarray-0.8.1-cp27-none-macosx_10_6_intel.whl         - path: bitarray-0.8.1.tar.gz - +     parties:       - role: owner         name: Ilan Schnell - +     packages:       - download_url: http://pypi.python.org/packages/source/b/bitarray/bitarray-0.8.1.tar.gz         sha1: 468456384529abcdef342 - +     license_expression: psf - +     licenses:       - source: scancode.com         key: psf @@ -1003,7 +1003,7 @@ And here would be similar data in JSON::                             "sha1": "468456384529abcdef342"                 }],                "parties": [{"name": "Ilan Schnell", "role": "owner"}], - +                "files": [{"path": "some/directory/", "type": "dir"},                          {"path": "bitarray-0.8.1-cp27-cp27m-macosx_10_9_intel.whl"},                          {"path": "bitarray-0.8.1-cp27-cp27m-manylinux1_i686.whl"}, @@ -1015,7 +1015,7 @@ And here would be similar data in JSON::                          {"path": "bitarray-0.8.1-cp27-none-macosx_10_6_intel.whl"},                          {"path": "bitarray-0.8.1.tar.gz"}],                }], - +  aboutcode_version: "4.0"} @@ -1116,7 +1116,7 @@ format::       }],       "licenses": [         { "detected": true, -           "key": "lgpl-2.1-plus",   +           "key": "lgpl-2.1-plus",           "category": "Copyleft Limited",           "homepage_url": "http://www.gnu.org/licenses/old-licenses/lgpl-2.1-standalone.html",           "start_line": 7, @@ -1178,7 +1178,7 @@ AboutCode Manager As a primary GUI for data review and integration, AboutCode Manager will need to be fluent in ABC Data to read/write ABC Data locally and -remotely through API from several sources.  +remotely through API from several sources. The short term changes would include: diff --git a/docs/source/aboutcode-docs/contributor_project_ideas.rst b/docs/source/aboutcode-docs/contributor_project_ideas.rst index c4d26d72..10dedcd2 100644 --- a/docs/source/aboutcode-docs/contributor_project_ideas.rst +++ b/docs/source/aboutcode-docs/contributor_project_ideas.rst @@ -43,7 +43,7 @@ Table of Contents * :ref:`aboutcode_data_server` * :ref:`vulnerablecode_package_security_vulnerability_data_feed_and_scanner` - * :ref:`integrate_the_license_expression_library_in_scancode_python_and_aboutcode_manager_javascript` + * :ref:`integrate_the_license_expression_library_in_scancode_python_and_aboutcode_manager_jscript` * :ref:`high_volume_matching_automatons_and_data_structures` * :ref:`scancode_scan_deduction` * :ref:`license_and_copyright_detection_benchmark` @@ -61,20 +61,25 @@ Table of Contents AboutCode projects are... ------------------------- -- **ScanCode Toolkit** a popular command line tool to scan code for licenses, copyrights and packages, used by many organizations and FOSS projects, small and large. +- **ScanCode Toolkit** a popular command line tool to scan code for licenses, copyrights and + packages, used by many organizations and FOSS projects, small and large. -- **AboutCode Manager** a JavaScript, Electron-based desktop application to review scan results and document your conclusions +- **AboutCode Manager** a JavaScript, Electron-based desktop application to review scan results + and document your conclusions - **AboutCode Toolkit** a set of command line tools to document and inventory known packages and licenses and generate attribution docs -- TraceCode Toolkit: a set of command line tools to find which source code is used to create a compiled binary +- TraceCode Toolkit: a set of command line tools to find which source code is used to create a + compiled binary -- DeltaCode Toolkit: a new command line tool to compare codebases based on scan and determine if and where there are material differences that affect licensing +- DeltaCode Toolkit: a new command line tool to compare codebases based on scan and determine if + and where there are material differences that affect licensing - VulnerableCode Server: a new server-side application to track package vulnerabilities -- AboutCode Server: a new server-side application to run and organize scans and ABC data (formerly ScanCode server) +- AboutCode Server: a new server-side application to run and organize scans and ABC data + (formerly ScanCode server) - ConAn: a command line tool to analyze the code in Docker and container images @@ -86,9 +91,11 @@ AboutCode projects are... We also work closely with other orgs and projects: -- purl aka. Package URLs https://github.com/package-url which is an emerging standard to reference software packages of all types. +- purl aka. Package URLs https://github.com/package-url which is an emerging standard to reference + software packages of all types. -- SPDX.org aka. Software Package Data Exchange, a spec to document the origin and licensing of packages +- SPDX.org aka. Software Package Data Exchange, a spec to document the origin and licensing of + packages .. _contact: @@ -139,7 +146,7 @@ following information, plus anything you think is relevant: innovative and what it will contribute -- hint: explain your data structures and the main processing flows in details. +- hint: explain your data structures and the main processing flows in details. - Description of previous work, existing solutions (links to prototypes, bibliography are more than welcome) @@ -163,7 +170,7 @@ introduce yourself and start the discussion! You need to understand something about open source licensing or package managers or code and binaries static analysis or low level data structures. The best way to demonstrate your capability would be to submit a small patch ahead of the -project selection for an existing issue or a new issue. +project selection for an existing issue or a new issue. We will **always** consider and prefer a project submissions where you have submitted a patch over any otherr submission without a patch. @@ -198,7 +205,7 @@ The features of this updated server would be: - Integrate the storage and retrieval of scans and ABC data with the AboutCode Manager app through the JSON API. -- Add a Github integration to scan/run an ABC tool on commit with webhooks. +- Add a Github integration to scan/run an ABC tool on commit with webhooks. * Bonus feature is to scan based on a received tweet of similar IRC or IM integration. @@ -239,26 +246,20 @@ relate to packages as they are detected by ScanCode or else. The features and TODO for this updated server would be: -* - Aggregate more and new packages vulnerabilities feeds, +* Aggregate more and new packages vulnerabilities feeds, -* - Automating correlation: add smart relationship detection to infer new +* Automating correlation: add smart relationship detection to infer new relatiosnhips between available packages and vulnerabilities from mining the graph of existing relations. -* - Create a ScanCode plugin to report vulnerabilities with detected packages +* Create a ScanCode plugin to report vulnerabilities with detected packages using this data. -* - Integrate API lookup on the server withe the AboutCode Manager UI +* Integrate API lookup on the server withe the AboutCode Manager UI -* - Create a UI and model for community curation of vulnerability to package +* Create a UI and model for community curation of vulnerability to package mappings, correlations and enhancements. - - **Tech** * Python 2, Django, PostgreSQL, DRF, JavaScript, Electron @@ -287,10 +288,10 @@ The features and TODO for this updated server would be: * @JonoYang https://github.com/JonoYang * @pombredanne https://github.com/pombredanne -.. _integrate_the_license_expression_library_in_scancode_python_and_aboutcode_manager_javascript: +.. _integrate_the_license_expression_library_in_scancode_python_and_aboutcode_manager_jscript: -**Integrate the license expression library in ScanCode (Python) and AboutCode Manager (JavaScript)** -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +**Integrate the license expression library in ScanCode (Python) and AboutCode Manager (JScript)** +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In GSoC 2017, this Python library was ported to JavaScript using Transcrypt. @@ -298,10 +299,10 @@ The goal of this project is to add support for license expressions in multiple projects and evolve the license expression library as needed: -- in Python: +- in Python: * the SPDX Python library - * the ScanCode toolkit. This also include the proper detection of license + * the ScanCode toolkit. This also include the proper detection of license expressions in SPDX-License-Identifier tags. * the AboutCode toolkit @@ -309,7 +310,7 @@ projects and evolve the license expression library as needed: * the AboutCode Manager -- in both languages in the core license expression proper, add support for a +- in both languages in the core license expression proper, add support for a built-in mode for strict SPDX expressions @@ -319,14 +320,14 @@ projects and evolve the license expression library as needed: - **URLS** - * https://github.com/nexB/license-expression + * https://github.com/nexB/license-expression * https://github.com/bastikr/boolean.py * https://github.com/nexB/aboutcode-manager * https://github.com/nexB/aboutcode-toolkit * https://github.com/nexB/scancode-toolkit * https://github.com/spdx/tools-python -- **Mentors** +- **Mentors** * @JonoYang https://github.com/JonoYang * @majurg https://github.com/majurg @@ -415,7 +416,7 @@ whole directory tree. This should be implemented as a set of ScanCode plugins * https://github.com/nexB/scancode-toolkit/issues/426 * https://github.com/nexB/scancode-toolkit/issues/377 -- **Mentors** +- **Mentors** * @pombredanne https://github.com/pombredanne * @JonoYang https://github.com/JonoYang @@ -458,9 +459,9 @@ adding more as needed. * Python -- **Mentors** +- **Mentors** - * @mjherzog https://github.com/mjherzog + * @mjherzog https://github.com/mjherzog * @pombredanne https://github.com/pombredanne .. _improved_copyright_parsing_and_speed_in_scancode: @@ -475,7 +476,7 @@ The goal of this project is to refactor Copyright detection for speed and simplicity possibly implementaing a new parser (PEG?, etc) or reimplementing core elements in Rust with a Python binding. -This would include also keeping track of line numbers and offsets where copyrights are found. +This would include also keeping track of line numbers and offsets where copyrights are found. This would likely require either replacing or enhancing NLTK which is used as a natural language parser. @@ -489,7 +490,7 @@ natural language parser. * https://github.com/nexB/scancode-toolkit/tree/develop/src/cluecode -- **Mentor** +- **Mentor** * @JonoYang https://github.com/JonoYang @@ -510,9 +511,9 @@ architecture. - **URLS** - * https://github.com/nexB/scancode-toolkit/issues/14 + * https://github.com/nexB/scancode-toolkit/issues/14 -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -533,7 +534,7 @@ ScanCode to support both Python 2 and Python 3. * https://github.com/nexB/scancode-toolkit/issues/295 -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -557,10 +558,10 @@ of ScanCode. - **URLS** - * https://github.com/pombredanne/conan + * https://github.com/pombredanne/conan * https://github.com/nexB/scancode-toolkit -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -569,7 +570,7 @@ of ScanCode. **Static analysis of binaries for build tracing in TraceCode** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -TraceCode does system call tracing only today. +TraceCode does system call tracing only today. - The primary goal of this project is to do the same using symbol, debug symbol @@ -591,9 +592,10 @@ TraceCode does system call tracing only today. - **URLS** * https://github.com/nexB/tracecode-toolkit for the existing non-static tool - * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols parsers/extractors + * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols + parsers/extractors -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -609,7 +611,8 @@ The first step is to debundle pre-built binaries that exist in ScanCode such that they come either from system-packages or pre-built Python wheels. This covers libarchive, libmagic and a few other native libraries. -The next step is to ensure that all the dependencies from ScanCode are also available as distro packages. +The next step is to ensure that all the dependencies from ScanCode are also available as distro +packages. The last step is to create proper distro packages for RPM, Debian, Nix and GUIX, Alpine, Arch and Gentoo and also an AppImage.org package as well as a proper @@ -629,7 +632,7 @@ This requires a good understanding of packaging and Python. * https://github.com/nexB/scancode-toolkit/issues/487 * https://github.com/nexB/scancode-toolkit/issues/469 -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne diff --git a/docs/source/aboutcode-docs/gsoc_2017.rst b/docs/source/aboutcode-docs/gsoc_2017.rst index 772cfba0..5e708385 100644 --- a/docs/source/aboutcode-docs/gsoc_2017.rst +++ b/docs/source/aboutcode-docs/gsoc_2017.rst @@ -15,23 +15,31 @@ This year AboutCode is a mentoring Organization for the Google Summer of Code 20 * where does it come from? * what is its license? copyright? -* is it secure, maintained, well coded? +* is it secure, maintained, well coded? -All these are questions that are important to find answers to when there are million of free and open source software components available on the web. +All these are questions that are important to find answers to when there are million of free and +open source software components available on the web. -Where software comes from and what is its license should be a problem of the past, such that everyone can safely consume more free and open source software. Come and join us to make it so! +Where software comes from and what is its license should be a problem of the past, such that +everyone can safely consume more free and open source software. Come and join us to make it so! -Our tools are used to help detect and report the origin and license of source code, packages and binaries, as well as discover software and package dependencies, track vulnerabilities, bugs and other important software component attributes. +Our tools are used to help detect and report the origin and license of source code, packages and +binaries, as well as discover software and package dependencies, track vulnerabilities, bugs and +other important software component attributes. Contact ------- -Subscribe to the mailing list at https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss and introduce yourself and start the discussion! The mailing list is usually the better option to avoid timezone gaps. +Subscribe to the mailing list at https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss +and introduce yourself and start the discussion! The mailing list is usually the better option +to avoid timezone gaps. -The list archive have also plenty of interesting information. Someone may have asked your question before. -Search and browse the archives at https://sourceforge.net/p/aboutcode/mailman/aboutcode-discuss/ ! +The list archive have also plenty of interesting information. Someone may have asked your question +before. Search and browse the archives at +https://sourceforge.net/p/aboutcode/mailman/aboutcode-discuss/ ! -For short chats, you can also join the #aboutcode IRC channel on Freenode or the Gitter channel at https://gitter.im/aboutcode-org/discuss +For short chats, you can also join the #aboutcode IRC channel on Freenode or the Gitter channel at +https://gitter.im/aboutcode-org/discuss For personal issues, you can contact the org admin directly: @pombredanne and pombredanne@gmail.com @@ -40,32 +48,46 @@ Please ask questions the smart way: http://www.catb.org/~esr/faqs/smart-question Technology ---------- -Discovering the origin of code is a vast topic. We primarily use Python for this and some C/C++ and JavaScript, but we are open to using any other language within reason. +Discovering the origin of code is a vast topic. We primarily use Python for this and some C/C++ and +JavaScript, but we are open to using any other language within reason. -Our domain includes text analysis and processing (for instance for copyrights and licenses), parsing (for package manifest formats), binary analysis (to detect the origin and license of binaries, which source code they come from, etc) as well as web based tools and APIs (to expose the tools and libraries as web services). +Our domain includes text analysis and processing (for instance for copyrights and licenses), +parsing (for package manifest formats), binary analysis (to detect the origin and license of +binaries, which source code they come from, etc) as well as web based tools and APIs (to expose the +tools and libraries as web services). About your project application ------------------------------ -We expect your application to be in the range of 1000 words. Anything less than that will probably not contain enough information for us to determine whether you are the right person for the job. Your proposal should contain at least the following information, plus anything you think is relevant: +We expect your application to be in the range of 1000 words. Anything less than that will probably +not contain enough information for us to determine whether you are the right person for the job. +Your proposal should contain at least the following information, plus anything you think is +relevant: * Your name * Title of your proposal * Abstract of your proposal -* Detailed description of your idea including explanation on why is it innovative and what it will contribute +* Detailed description of your idea including explanation on why is it innovative and what it will + contribute - * hint: explain your data structures and the main processing flows in details. + * hint: explain your data structures and the main processing flows in details. -* Description of previous work, existing solutions (links to prototypes, bibliography are more than welcome) +* Description of previous work, existing solutions (links to prototypes, bibliography are more + than welcome) * Mention the details of your academic studies, any previous work, internships * Relevant skills that will help you to achieve the goal (programming languages, frameworks)? * Any previous open-source projects (or even previous GSoC) you have contributed to and links. -* Do you plan to have any other commitments during GSoC that may affect your work? Any vacations/holidays? Will you be available full time to work on your project? (Hint: do not bother applying if this is not a serious full time commitment) +* Do you plan to have any other commitments during GSoC that may affect your work? Any + vacations/holidays? Will you be available full time to work on your project? (Hint: do not bother + applying if this is not a serious full time commitment) -Subscribe to the mailing list at https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss or join the #aboutcode IRC channel on Freenode and introduce yourself and start the discussion! +Subscribe to the mailing list at https://lists.sourceforge.net/lists/listinfo/aboutcode-discuss or +join the #aboutcode IRC channel on Freenode and introduce yourself and start the discussion! -You need to understand something about open source licensing or package managers or code and binaries static analysis. The best way to demonstrate your capability would be to submit a small patch ahead of the project selection for an existing issue or a new issue. +You need to understand something about open source licensing or package managers or code and +binaries static analysis. The best way to demonstrate your capability would be to submit a small +patch ahead of the project selection for an existing issue or a new issue. Project ideas ------------- @@ -73,7 +95,10 @@ Project ideas **ScanCode live scan server** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This project is to use ScanCode as a library in a web and REST API application that allows you to scan code on demand by entering a URL and then store the scan results. It could also be made available as a Travis or Github integration to scan on commit with webhooks. Bonus feature is to scan based on a received tweet of similar IRC or IM integration. +This project is to use ScanCode as a library in a web and REST API application that allows you to +scan code on demand by entering a URL and then store the scan results. It could also be made +available as a Travis or Github integration to scan on commit with webhooks. Bonus feature is to +scan based on a received tweet of similar IRC or IM integration. - **URLS** : @@ -88,23 +113,28 @@ This project is to use ScanCode as a library in a web and REST API application t **Package security vulnerability data feed (and scanner)** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The end goal for this project is to build on existing projects to match packages identified by ScanCode to existing vulnerability alerts. This is not trivial as there are several gaps in the CVE data and how they relate to packages as they are detected by ScanCode or else. This is a green field project. +The end goal for this project is to build on existing projects to match packages identified by +ScanCode to existing vulnerability alerts. This is not trivial as there are several gaps in the CVE +data and how they relate to packages as they are detected by ScanCode or else. This is a green +field project. The key points to tackle are: -#. - create the tools to build a free and open source structured and curate security feed +#. create the tools to build a free and open source structured and curate security feed - * the aggregation of packages vulnerabilities feeds in a common and structured model (CVE, distro trackers, etc), + * the aggregation of packages vulnerabilities feeds in a common and structured model (CVE, + distro trackers, etc), * the aggregation of additional security data (CWE, CPE, and more) in that model, - * the correlation of the different data items, creating accurate relationships and matching of actual package identifiers to vulnerabilities, - * an architecture for community curation of vulnerabilities, correlation and enhancement of the data. + * the correlation of the different data items, creating accurate relationships and matching of + actual package identifiers to vulnerabilities, + * an architecture for community curation of vulnerabilities, correlation and enhancement of the + data. -#. - as a side bonus, build the tools in ScanCode to match detected packages to this feed. - Note there is no FOSS tool and DB that does all of this today (only proprietary solutions such as vfeed or vulndb). +#. as a side bonus, build the tools in ScanCode to match detected packages to this feed. + Note there is no FOSS tool and DB that does all of this today (only proprietary solutions such + as vfeed or vulndb). - **Some Related URLS for other projects in the same realm** : @@ -126,20 +156,22 @@ The key points to tackle are: * @JonoYang https://github.com/JonoYang * @pombredanne https://github.com/pombredanne -**Port the Python license expression library to JavaScript and prepare and publish an NPM package** : -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +**Port the Python license expression library to JScript and prepare and publish an NPM package**: +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Use automated code translation (for JS) for the port. Add license expression support to AboutCodeMgr with this library. As a bonus, create a web server app and API service to parse and normalize ScanCode and SPDX license expressions either in Python or JavaScript. +Use automated code translation (for JS) for the port. Add license expression support to +AboutCodeMgr with this library. As a bonus, create a web server app and API service to parse and +normalize ScanCode and SPDX license expressions either in Python or JavaScript. - **URLS** : - * https://github.com/nexB/license-expression + * https://github.com/nexB/license-expression * https://github.com/bastikr/boolean.py * https://github.com/nexB/aboutcode-manager * https://github.com/jashkenas/coffeescript/wiki/list-of-languages-that-compile-to-js -- **Mentors** : +- **Mentors** : * @JonoYang https://github.com/JonoYang * @majurg https://github.com/majurg @@ -168,7 +200,8 @@ Bitbucket commit hash searches/API or the SoftwareHeritage.org API. **ScanCode scan deduction** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The goal of this project is to take existing scan and match results and infer summaries and deduction at a higher level, such as the licensing of a whole directory tree. +The goal of this project is to take existing scan and match results and infer summaries and +deduction at a higher level, such as the licensing of a whole directory tree. - **URLS** : @@ -176,7 +209,7 @@ The goal of this project is to take existing scan and match results and infer su * https://github.com/nexB/scancode-toolkit/issues/426 * https://github.com/nexB/scancode-toolkit/issues/377 -- **Mentors** : +- **Mentors** : * @pombredanne https://github.com/pombredanne * @JonoYang https://github.com/JonoYang @@ -184,19 +217,26 @@ The goal of this project is to take existing scan and match results and infer su **DeltaCode** : ^^^^^^^^^^^^^^^^^^ -A new tool to help determine at a high level if the licensing for two codebases or versions of code has changed, and if so how. This is NOT a generic diff tool that identifies all codebase differences, rather it focuses on changes in licensing based on differences between ScanCode files. +A new tool to help determine at a high level if the licensing for two codebases or versions of +code has changed, and if so how. This is NOT a generic diff tool that identifies all codebase +differences, rather it focuses on changes in licensing based on differences between ScanCode files. -- **Mentor** : +- **Mentor** : * @majurg https://github.com/majurg **License and copyright detection benchmark** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Compare ScanCode runtimes with Fossology, licensee, LicenseFinder, license-check, ninka, slic, LiD and others. This project is to create a comprehensive test suite and a benchmark for several FOSS open source license and copyright detection engines, establish mappings between the different conventions they use for license identification and evaluate and publish the results of detection accuracy and precision. +Compare ScanCode runtimes with Fossology, licensee, LicenseFinder, license-check, ninka, slic, LiD +and others. This project is to create a comprehensive test suite and a benchmark for several FOSS +open source license and copyright detection engines, establish mappings between the different +conventions they use for license identification and evaluate and publish the results of detection +accuracy and precision. -Note that this not about the speed of scanning: the performance and time taken is accessory and a nice to have result only. What matters is the accuracy of the license detection: +Note that this not about the speed of scanning: the performance and time taken is accessory and a +nice to have result only. What matters is the accuracy of the license detection: #. is the right license detected and how correct is this detection? @@ -204,27 +244,34 @@ Note that this not about the speed of scanning: the performance and time taken i So what is needed is a (large) test set of files. -Then establishing a ground truth for reference e.g. detecting then reviewing manually possibly with scancode to set up the baseline that will be used to compare all the scanners. +Then establishing a ground truth for reference e.g. detecting then reviewing manually possibly with +scancode to set up the baseline that will be used to compare all the scanners. -Then run the other tools and scancode to see how well they perform and of course establish a mapping of license identifiers: each tool may use different license ids so we need to map these to the ids used in the test baseline (e.g. the scancode license keys): all this has to be built, possibly reusing some or all of the scancode tests and lacing in all the tests from the other tools and adding more ass needed. +Then run the other tools and scancode to see how well they perform and of course establish a +mapping of license identifiers: each tool may use different license ids so we need to map these +to the ids used in the test baseline (e.g. the scancode license keys): all this has to be built, +possibly reusing some or all of the scancode tests and lacing in all the tests from the other tools +and adding more ass needed. -- **Mentors** : +- **Mentors** : - * @mjherzog https://github.com/mjherzog + * @mjherzog https://github.com/mjherzog * @pombredanne https://github.com/pombredanne **Improved copyright parsing in ScanCode** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -by keeping track of line numbers and offsets where copyrights are found. This would likely require either replacing or enhancing NLTK which is used as a natural language parser to add support for tracking where a copyright has been detected in a scanned text. +by keeping track of line numbers and offsets where copyrights are found. This would likely require +either replacing or enhancing NLTK which is used as a natural language parser to add support for +tracking where a copyright has been detected in a scanned text. - **URLS** : * https://github.com/nexB/scancode-toolkit/tree/develop/src/cluecode -- **Mentor** : +- **Mentor** : * @JonoYang https://github.com/JonoYang @@ -236,35 +283,37 @@ by keeping track of line numbers and offsets where copyrights are found. This wo * https://github.com/nexB/attributecode/issues/277 -- **Mentor** : +- **Mentor** : * @chinyeungli https://github.com/chinyeungli **Transparent archive extraction in ScanCode** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -ScanCode archive extraction is currently done with a separate command line invocation. The goal of this project is to integrate archive extraction transparently into the ScanCode scan loop. +ScanCode archive extraction is currently done with a separate command line invocation. The goal +of this project is to integrate archive extraction transparently into the ScanCode scan loop. - **URLS** : - * https://github.com/nexB/scancode-toolkit/issues/14 + * https://github.com/nexB/scancode-toolkit/issues/14 -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne **Automated docker and VM images static package analysis** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -to determine which packages are installed in Docker layers for RPMs, Debian or Alpine Linux. This is for the conan Docker image analysis tool. +to determine which packages are installed in Docker layers for RPMs, Debian or Alpine Linux. This +is for the conan Docker image analysis tool. - **URLS** : - * https://github.com/pombredanne/conan + * https://github.com/pombredanne/conan -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne @@ -279,43 +328,48 @@ Create ScanCode plugins for outputs to multiple formats (CSV, JSON, SPDX, Debian * https://github.com/nexB/scancode-toolkit/issues/552 * https://github.com/nexB/scancode-toolkit/issues/381 -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne **Static analysis of binaries for build tracing in TraceCode** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -TraceCode does system call tracing. The goal of this project is to do the same using symbol, debug symbol or string matching to accomplish something similar, +TraceCode does system call tracing. The goal of this project is to do the same using symbol, debug +symbol or string matching to accomplish something similar, - **URLS** : * https://github.com/nexB/tracecode-build for the existing non-static tool - * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols parsers/extractors + * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols + parsers/extractors -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne **Better support tracing the lifecycle of file descriptors in TraceCode build** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -TraceCode does system call tracing. The goal of this project is to improve the way we track open/close file descriptors in the trace to reconstruct the life of a file. +TraceCode does system call tracing. The goal of this project is to improve the way we track +open/close file descriptors in the trace to reconstruct the life of a file. - **URLS** : - * https://github.com/nexB/tracecode-build + * https://github.com/nexB/tracecode-build -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne **Create Debian and RPM packages for ScanCode, AttributeCode and TraceCode**. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Consider also including an AppImage.org package. If you think this may not fill in a full three months project, consider also adding some extras such as submitting the packages to Debian and Fedora. +Consider also including an AppImage.org package. If you think this may not fill in a full three +months project, consider also adding some extras such as submitting the packages to Debian and +Fedora. - **URLS** : @@ -323,14 +377,15 @@ Consider also including an AppImage.org package. If you think this may not fill * https://github.com/nexB/scancode-toolkit/issues/487 * https://github.com/nexB/scancode-toolkit/issues/469 -- **Mentor** : +- **Mentor** : * @pombredanne https://github.com/pombredanne **AboutCode Manager test suite and Ci** : ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Create an extensive test suite for the Electron app and setup the CI to run unit, integration and smoke tests on Ci for Windows, Linux and Mac. +Create an extensive test suite for the Electron app and setup the CI to run unit, integration and +smoke tests on Ci for Windows, Linux and Mac. - **URLS** : diff --git a/docs/source/aboutcode-docs/gsoc_2019.rst b/docs/source/aboutcode-docs/gsoc_2019.rst index 9ceb2b98..27bd3a29 100644 --- a/docs/source/aboutcode-docs/gsoc_2019.rst +++ b/docs/source/aboutcode-docs/gsoc_2019.rst @@ -3,7 +3,8 @@ Google Summer of Code 2019 ========================== -AboutCode is participating in the Google Summer of Code in 2019 as a mentoring org. This page contain all the information for students and anyone else interested in helping. +AboutCode is participating in the Google Summer of Code in 2019 as a mentoring org. This page +contain all the information for students and anyone else interested in helping. AboutCode is a family of FOSS projects to uncover data ... about software code: @@ -54,21 +55,21 @@ Table of Contents AboutCode projects are... ------------------------- -- `ScanCode Toolkit `_ is a popular +- `ScanCode Toolkit `_ is a popular command line tool to scan code for licenses, copyrights and packages, used by many organizations and FOSS projects, small and large. -- `Scancode Workbench `_ +- `Scancode Workbench `_ (formerly AboutCode Manager) is a JavaScript, Electron-based desktop - application to review scan results and document your origin and license + application to review scan results and document your origin and license conclusions. -- `AboutCode Toolkit `_ is a +- `AboutCode Toolkit `_ is a command line tool to document and inventory known packages and licenses and generate attribution docs, typically using the results of analyzed and reviewed scans. -- `TraceCode Toolkit `_ is +- `TraceCode Toolkit `_ is a command line tool to find which source code file is used to create a compiled binary and trace and graph builds. @@ -80,24 +81,24 @@ AboutCode projects are... the code in Docker and container images - `VulnerableCode `_ : an emerging - server-side application to collect and track known package vulnerabilities. - + server-side application to collect and track known package vulnerabilities. + - `license-expression `_ : a library to parse, analyze, simplify and render boolean license expression (such as SPDX) We also work closely, contribute and co-started several other orgs and projects: -- +- `Package URL `_ which is an emerging standard to reference software packages of all types with simple, readable and concise URLs. -- +- `SPDX `_ aka. Software Package Data Exchange, a spec to document the origin and licensing of packages. -- +- `ClearlyDefined `_ to review and help FOSS projects improve their licensing and documentation clarity. @@ -138,12 +139,14 @@ Skills Incoming students will need the following skills: -- Intermediate to strong Python programming. For some projects, strong C/C++ and/or Rust is needed too. +- Intermediate to strong Python programming. For some projects, strong C/C++ and/or Rust is needed + too. - Familiarity with git as a version control system - Ability to set up your own development environment - An interest in FOSS licensing and software code and origin analysis -We are happy to help you get up to speed, but the more you are able to demonstrate ability and skills in advance, the more likely we are to choose your application! +We are happy to help you get up to speed, but the more you are able to demonstrate ability and +skills in advance, the more likely we are to choose your application! .. _about_your_project_application_gsoc19: @@ -165,7 +168,7 @@ following information, plus anything you think is relevant: * hint: explain your data structures and you planned main processing flows in - details. + details. - Description of previous work, existing solutions (links to prototypes, bibliography are more than welcome) @@ -212,8 +215,8 @@ ScanCode ideas ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Copyright detection is reasonably good by the slowest scanner in ScanCode. -It is based on NLTK part of speech (PoS) tagging and a copyright grammar. -The exact start and end lines where a copyright is found are approximate. +It is based on NLTK part of speech (PoS) tagging and a copyright grammar. +The exact start and end lines where a copyright is found are approximate. The goal of this project is to refactor Copyright detection for speed and simplicity possibly implementing a new parser (PEG?, etc.) or re-implementing @@ -221,10 +224,10 @@ core elements in Rust with a Python binding for speed or using a fork of NLTK or any other tool to be faster and more accurate. This would include also keeping track of line numbers and offsets where -copyrights are found. +copyrights are found. Also we detect copyrights that are part of a standard license text (e.g. FSF -copyright in a GPL text) and we should be able to filter these out. +copyright in a GPL text) and we should be able to filter these out. - **Level** @@ -264,7 +267,7 @@ ScanCode to support both Python 2 and Python 3. * https://github.com/nexB/scancode-toolkit/issues/295 -- **Mentors** +- **Mentors** * @majurg https://github.com/majurg @@ -275,7 +278,7 @@ ScanCode to support both Python 2 and Python 3. ScanCode programming language detection is not as accurate as it could be and this is important to get this right to drive further automation. We also need to -automatically classify each file in facets when possible. +automatically classify each file in facets when possible. The goal of this project is to improve the quality of programming language detection (which is using only Pygments today and could use another tool, e.g. @@ -298,7 +301,7 @@ which could use some machine learning and classifier. * https://github.com/nexB/scancode-toolkit/issues/1012 * https://github.com/nexB/scancode-toolkit/issues/1036 -- **Mentors** +- **Mentors** * @pombredanne https://github.com/pombredanne @@ -314,7 +317,7 @@ notices). Other improvements would be welcomed to ensure the proper detected license text is collected in an improved way. Dealing with large files sometimes trigger a timeout and handling these cases would be needed too (by breaking files in chunks). The detection speed could also be improved possibly by porting -some critical code sections to C or Rust and that would need extensive profiling. +some critical code sections to C or Rust and that would need extensive profiling. - **Level** @@ -325,9 +328,9 @@ some critical code sections to C or Rust and that would need extensive profiling * Python, C/C++, Rust, Go -- **Mentors** +- **Mentors** - * @mjherzog https://github.com/mjherzog + * @mjherzog https://github.com/mjherzog * @pombredanne https://github.com/pombredanne .. _improve_scancode_scan_summarization_and_deduction: @@ -340,7 +343,7 @@ summaries and perform some deduction of license and origin at a higher level, such as the licensing or origin of a whole directory tree. The ultimate goal is to automate the conclusion of a license and origin based on scans. This could include using statistics and machine learning techniques such as -classifiers where relevant and efficient. +classifiers where relevant and efficient. This should be implemented as a set of ScanCode plugins and further the summarycode module plugins. @@ -359,7 +362,7 @@ summarycode module plugins. * https://github.com/nexB/scancode-toolkit/issues/426 * https://github.com/nexB/scancode-toolkit/issues/377 -- **Mentors** +- **Mentors** * @pombredanne https://github.com/pombredanne * @JonoYang https://github.com/JonoYang @@ -402,7 +405,7 @@ This requires a good understanding of packaging and Python. * https://github.com/nexB/scancode-toolkit/issues/487 * https://github.com/nexB/scancode-toolkit/issues/469 -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -415,7 +418,7 @@ DeltaCode projects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ DeltaCode is a tool to compare and report scan differences. -When comparing files, it only uses exact comparison. +When comparing files, it only uses exact comparison. The goal of this project is to improve the usefulness of the delta by also finding files that are mostly the same (e.g. quasi or nrea duplicates) vs. files that are completely different. Then the DeltaCode comparison core should be @@ -436,7 +439,7 @@ meterial in the code files (e.g. such that small changes may be ignored) * https://github.com/nexB/deltacode/ -- **Mentors** +- **Mentors** * @majurg https://github.com/majurg * @johnmhoran https://github.com/johnmhoran @@ -454,7 +457,7 @@ is to create a tool that provides the same results as the strace-based tracing but would be using using ELF symbols, DWARF debug symbols, signatures or string matching to determine when and how a source code file is built in a binary using only a static analysis. The primary target should be Linux executables, -though the code should be designed to be extensible to Windows PE and macOS Dylib and exes. +though the code should be designed to be extensible to Windows PE and macOS Dylib and exes. - **Level** @@ -468,9 +471,10 @@ though the code should be designed to be extensible to Windows PE and macOS Dyli - **URLS** * https://github.com/nexB/tracecode-toolkit for the existing non-static tool - * https://github.com/nexB/scancode-toolkit-contrib for some work in progress on binaries/symbols parsers/extractors + * https://github.com/nexB/scancode-toolkit-contrib for some work in progress on binaries/symbols + parsers/extractors -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -501,9 +505,10 @@ line toolkit (like for ScanCode). - **URLS** * https://github.com/nexB/tracecode-toolkit for the existing non-static tool - * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols parsers/extractors + * https://github.com/nexB/scancode-toolkit-contrib for the work in progress on binaries/symbols + parsers/extractors -- **Mentor** +- **Mentor** * @pombredanne https://github.com/pombredanne @@ -534,9 +539,9 @@ the integration with ScanCode. - **URLS** - * https://github.com/nexB/conan + * https://github.com/nexB/conan -- **Mentor** +- **Mentor** * @JonoYang https://github.com/JonoYang @@ -585,23 +590,18 @@ relate to packages as they are detected by ScanCode or else. The features and TODO for this updated server would be: -- - Aggregate more and new packages vulnerabilities feeds, +- Aggregate more and new packages vulnerabilities feeds, -- - Automating correlation: add smart relationship detection to infer new +- Automating correlation: add smart relationship detection to infer new relatiosnhips between available packages and vulnerabilities from mining the graph of existing relations. -- - Create a ScanCode plugin to report vulnerabilities with detected packages +- Create a ScanCode plugin to report vulnerabilities with detected packages using this data. -- - Integrate API lookup on the server withe the AboutCode Manager UI +- Integrate API lookup on the server withe the AboutCode Manager UI -- - Create a UI and model for community curation of vulnerability to package +- Create a UI and model for community curation of vulnerability to package mappings, correlations and enhancements. @@ -654,23 +654,20 @@ bindings. There are several areas to research and prototype such as: -- - A data structure to match efficiently a batch of fix-width checksums (e.g. +- A data structure to match efficiently a batch of fix-width checksums (e.g. SHA1) against a large index of such checksums, where each checksum points to one or more files or packages. A possible direction is to use finite state transducers, specialized B-tree indexes, blomm-like filters. Since when a codebase is being matched there can be millions of lookups to do, the batch matching is preferred. -- - A data structure to match efficiently a batch of fix-width byte strings (e.g. +- A data structure to match efficiently a batch of fix-width byte strings (e.g. LSH) against a large index of such LSH within a fixed hamming distance, where each points to one or more files or packages. A possible direction is to use finite state transducers (possibly weighted), specialized B-tree indexes or multiple hash-like on-disk tables. -- - A memory-mapped Aho-Corasick automaton to build large batch tree matchers. +- A memory-mapped Aho-Corasick automaton to build large batch tree matchers. Available Aho-Corasick automatons may not have a Python binding or may not allow memory-mapping (like pyahocorasick we use in ScanCode). The volume of files we want to handle requires to reuse, extend or create specialized tree/paths @@ -678,8 +675,7 @@ There are several areas to research and prototype such as: than the available memory. A possible direction is to use finite state transducers (possibly weighted). -- - Feature hashing research: we deal with many "features" and hashing to limit +- Feature hashing research: we deal with many "features" and hashing to limit the number and size of the each features seems to be a valuable thing. The goal is to research the validaty of feature hashing with short hashes (15, 16 and 32 bits) and evaluate if this leads to acceptable false-positive and loss of @@ -712,4 +708,5 @@ This is a green field project. Mentoring --------- -We welcome new mentors to help with the program and require some good unerstanding of the project codebase and domain to join as a mentor. Contact the team on Gitter. +We welcome new mentors to help with the program and require some good unerstanding of the project +codebase and domain to join as a mentor. Contact the team on Gitter. diff --git a/docs/source/aboutcode-docs/gsod_2019.rst b/docs/source/aboutcode-docs/gsod_2019.rst index 79be2807..01757e57 100644 --- a/docs/source/aboutcode-docs/gsod_2019.rst +++ b/docs/source/aboutcode-docs/gsod_2019.rst @@ -3,7 +3,9 @@ Google Season of Docs 2019 ========================== -AboutCode has been accepted as a participant in the Google Season of Documents in 2019 as a mentoring org, and is looking for people with technical writing skills. This page contains information for technical writers and anyone else interested in helping. +AboutCode has been accepted as a participant in the Google Season of Documents in 2019 as a +mentoring org, and is looking for people with technical writing skills. This page contains +information for technical writers and anyone else interested in helping. AboutCode is a family of FOSS projects to uncover data about software code: @@ -12,11 +14,16 @@ AboutCode is a family of FOSS projects to uncover data about software code: * What is its license? copyright? * Is the code secure, maintained, well coded? -All these questions are important, and are relevant to millions of free and open source software components available on the web for reuse. The answers are critical to ensure that everyone can safely consume free and open source software. +All these questions are important, and are relevant to millions of free and open source software +components available on the web for reuse. The answers are critical to ensure that everyone can +safely consume free and open source software. *Join us to make it so!* -Our tools are used to help detect and report the origin and license of source code, packages and binaries as well as to discover software and package dependencies, and in the future track security vulnerabilities, bugs and other important software package attributes. This is a suite of command line tools, web-based and API servers and desktop applications. +Our tools are used to help detect and report the origin and license of source code, packages and +binaries as well as to discover software and package dependencies, and in the future track security +vulnerabilities, bugs and other important software package attributes. This is a suite of command +line tools, web-based and API servers and desktop applications. Table of Contents ----------------- @@ -32,10 +39,10 @@ Table of Contents * :ref:`scan_a_codebase_and_analyze_the_results` -- :ref:`how_to_ideas_gsod` +- :ref:`how_to_ideas_gsod` * :ref:`how_to_get_the_license_clarity_score_of_a_package` - * :ref:`how_to_discover_licensing_issues_in_a_software_project` + * :ref:`how_to_discover_licensing_issues_in_a_software_project` - :ref:`referance_ideas_gsod` @@ -53,22 +60,31 @@ Table of Contents List of AboutCode projects -------------------------- -Note that the AboutCode focus for GSOD 2019 is on **ScanCode Toolkit** and **ScanCode Workbench**\ , although proposals to improve the documents of other AboutCode projects are welcome. +Note that the AboutCode focus for GSOD 2019 is on **ScanCode Toolkit** and **ScanCode Workbench**\, +although proposals to improve the documents of other AboutCode projects are welcome. -- `ScanCode Toolkit `_ is a popular command line tool to scan code for licenses, copyrights and packages, used by many organizations and FOSS projects, small and large. +- `ScanCode Toolkit `_ is a popular command line tool to + scan code for licenses, copyrights and packages, used by many organizations and FOSS projects, + small and large. -- `Scancode Workbench `_ (formerly AboutCode Manager) is a JavaScript, Electron-based desktop application to review scan results and document your origin and license conclusions. +- `Scancode Workbench `_ (formerly AboutCode Manager) + is a JavaScript, Electron-based desktop application to review scan results and document your + origin and license conclusions. -- Other AboutCode projects are described at https://www.aboutcode.org and https://github.com/nexB/aboutcode +- Other AboutCode projects are described at https://www.aboutcode.org and + https://github.com/nexB/aboutcode We also work closely with, contribute to and have co-started several other orgs and projects: -- `Package URL `_ is an emerging standard to reference software packages of all types with simple, readable and concise URLs. +- `Package URL `_ is an emerging standard to reference software + packages of all types with simple, readable and concise URLs. -- `SPDX `_ is the Software Package Data Exchange, a specification to document the origin and licensing of software packages. +- `SPDX `_ is the Software Package Data Exchange, a specification to document + the origin and licensing of software packages. -- `ClearlyDefined `_ is a project to review FOSS software and help FOSS projects to improve their licensing and documentation clarity. +- `ClearlyDefined `_ is a project to review FOSS software and help FOSS projects + to improve their licensing and documentation clarity. .. _contact_gsod: @@ -92,9 +108,14 @@ Technology We primarily use Python (and some C/C++) for code analysis. We use Electron/JavaScript for GUI. -Our domain includes text analysis and processing (for instance for copyright and license detection), parsing (for package manifest formats), binary analysis (to detect the origin and license of binaries, which source code they come from, etc.) as well as web based tools and APIs (to expose the tools and libraries as web services) and low-level data structures for efficient matching (such as Aho-Corasick and other automata). +Our domain includes text analysis and processing (for instance for copyright and license +detection), parsing (for package manifest formats), binary analysis (to detect the origin and +license of binaries, which source code they come from, etc.) as well as web based tools and APIs +(to expose the tools and libraries as web services) and low-level data structures for efficient +matching (such as Aho-Corasick and other automata). -Our documentation is provided in text files that support the help function of our command line tools. We also have begun to provide documentation in the Wiki section of some AboutCode projects. +Our documentation is provided in text files that support the help function of our command line +tools. We also have begun to provide documentation in the Wiki section of some AboutCode projects. .. _writing_skills_gsod: @@ -106,18 +127,21 @@ Incoming technical writers will need the following skills: * Ability to install and configure open source code from GitHub. * Ability to understand and run programs from the command line in a terminal window. -* Familiarity with the four document functions described at https://www.divio.com/blog/documentation/ +* Familiarity with the four document functions described at + https://www.divio.com/blog/documentation/ * Ability to create and edit wiki pages with multiple markdown languages. * An interest in FOSS licensing and software code and origin analysis. -We are happy to help you get up to speed, and the more you are able to demonstrate ability and skills in advance, the more likely we are to choose your application! +We are happy to help you get up to speed, and the more you are able to demonstrate ability and +skills in advance, the more likely we are to choose your application! .. _application_gsod: About your project application ------------------------------ -Your application should be in the range of 1000 words, and should contain the following information, plus anything else that you think is relevant: +Your application should be in the range of 1000 words, and should contain the following +information, plus anything else that you think is relevant: - Your name and contact details @@ -126,7 +150,9 @@ Your application should be in the range of 1000 words, and should contain the fo - Abstract of your proposal -- Description of your idea including an explanation of what it will contribute to the project, such as the software development life cycle requirements that you expect to help with the documentation improvements. +- Description of your idea including an explanation of what it will contribute to the project, + such as the software development life cycle requirements that you expect to help with the + documentation improvements. - Description of previous work, existing solutions, open-source projects, preferably with links. @@ -134,25 +160,32 @@ Your application should be in the range of 1000 words, and should contain the fo - Descriptions of your relevant skills. -- Do you plan to have any other commitments during GSOD that may affect your work? Any vacations/holidays? Will you be available full time to work on your project? Please apply only if this is a serious full time commitment during the GSOD time frame. +- Do you plan to have any other commitments during GSOD that may affect your work? Any + vacations/holidays? Will you be available full time to work on your project? Please apply + only if this is a serious full time commitment during the GSOD time frame. Join the chat online or by IRC at https://gitter.im/aboutcode-org/discuss Introduce yourself and start the discussion! -An excellent, competitive way to demonstrate your capability would be to submit a documentation improvement to an AboutCode project, especially to ScanCode Toolkit or ScanCode Workbench. +An excellent, competitive way to demonstrate your capability would be to submit a documentation +improvement to an AboutCode project, especially to ScanCode Toolkit or ScanCode Workbench. -You can pick any project idea from the list below. You can also submit `Your Documentation Project ideas <#your-documentation-project-ideas>`_. +You can pick any project idea from the list below. You can also submit +`Your Documentation Project ideas <#your-documentation-project-ideas>`_. -.. _doc_project_ideas_gsod: +.. _doc_project_ideas_gsod: Our Documentation Project ideas ------------------------------- -Here is a list of candidate project ideas for your consideration, organized by documentation function: **Tutorial** , **How-To** , **Reference** , **Discussion**. +Here is a list of candidate project ideas for your consideration, organized by documentation +function: **Tutorial** , **How-To** , **Reference** , **Discussion**. -Note that the AboutCode focus for GSOD 2019 is on ScanCode Toolkit and ScanCode Workbench, although proposals to improve the documents of other AboutCode projects are welcome. +Note that the AboutCode focus for GSOD 2019 is on ScanCode Toolkit and ScanCode Workbench, +although proposals to improve the documents of other AboutCode projects are welcome. -`Your Documentation Project ideas <#your-documentation-project-ideas>`_ are welcome too! Please chat about them to increase your chances of success! +`Your Documentation Project ideas <#your-documentation-project-ideas>`_ are welcome too! Please +chat about them to increase your chances of success! .. _tutorial_ideas_gsod: @@ -164,13 +197,12 @@ Tutorial ideas **Scan a Codebase and Analyze the Results** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Provide specific instructions to guide a new user to: +Provide specific instructions to guide a new user to: * Scan a somewhat complex sample codebase using scancode-toolkit. -* Import the results into ScanCode Workbench. -* - Analyze the scan results. +* Import the results into ScanCode Workbench. +* Analyze the scan results. * **Level** @@ -200,9 +232,9 @@ How-To ideas .. _how_to_get_the_license_clarity_score_of_a_package: **How To Get the License Clarity Score of a Package** -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Explain the recommended scancode-toolkit options to get a license clarity score. +Explain the recommended scancode-toolkit options to get a license clarity score. * **Level** @@ -231,9 +263,7 @@ Explain the recommended scancode-toolkit options to get a license clarity score. * Explain the recommended scancode-toolkit options to discover licenses. -* - Explain how to take advantage of license policy support. - +* Explain how to take advantage of license policy support. * **Level** @@ -264,7 +294,7 @@ Reference ideas **ScanCode Output Formats** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Explain the various ScanCode output formats and their business purposes. +Explain the various ScanCode output formats and their business purposes. * **Level** @@ -296,13 +326,13 @@ Discussion ideas **Integrating ScanCode into a Software Development Lifecycle** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Discuss options and techniques to integrate ScanCode into a software development lifecycle workflow: +Discuss options and techniques to integrate ScanCode into a software development lifecycle +workflow: * During software creation and maintenance. * During software check-out/check-in. -* - During sofware build and test. +* During sofware build and test. * **Level** @@ -329,13 +359,18 @@ Discuss options and techniques to integrate ScanCode into a software development Your Documentation Project ideas -------------------------------- -Download and install ScanCode Toolkit and ScanCode Workbench and try them out. For example, you may try scanning an open source software package in a technology with which you are familiar. What are the documentation weak points? +Download and install ScanCode Toolkit and ScanCode Workbench and try them out. For example, you +may try scanning an open source software package in a technology with which you are familiar. +What are the documentation weak points? * Is it difficult to get started? A **Tutorial** document opportunity. * Is it difficult to accomplish a specific objective? A **How-To** document opportunity. -* Are the capabilities of the tool too mysterious? Do you want to know more about what you can do with it? A **Reference** document opportunity. -* Do you feel that you need to understand its concepts better in order to use it and trust it? Do you want to know more about how the code scanning actually works? A **Discussion** document opportunity. +* Are the capabilities of the tool too mysterious? Do you want to know more about what you can do + with it? A **Reference** document opportunity. +* Do you feel that you need to understand its concepts better in order to use it and trust it? Do + you want to know more about how the code scanning actually works? A **Discussion** document + opportunity. Feel free to propose and describe your own documentation ideas. @@ -344,4 +379,5 @@ Feel free to propose and describe your own documentation ideas. Mentoring --------- -We welcome new mentors to help with the program. We require some understanding of the project domain to join as a mentor. Contact the team on Gitter at https://gitter.im/aboutcode-org/discuss +We welcome new mentors to help with the program. We require some understanding of the project +domain to join as a mentor. Contact the team on Gitter at https://gitter.im/aboutcode-org/discuss diff --git a/docs/source/aboutcode-docs/home.rst b/docs/source/aboutcode-docs/home.rst index 17284b9a..69549902 100644 --- a/docs/source/aboutcode-docs/home.rst +++ b/docs/source/aboutcode-docs/home.rst @@ -2,7 +2,7 @@ AboutCode Docs ============== Welcome to the AboutCode wiki! - + If you are interested in the Google Summer of Code 2019, check out this page. :ref:`GSoC2019` If you are interested in the Google Season of Documents 2019, go through this page. :ref:`GSoD2019` @@ -15,4 +15,4 @@ If you are interested in the Google Season of Documents 2019, go through this pa gsoc_2017 gsoc_2018 gsoc_2019 - writing_good_commit_messages \ No newline at end of file + writing_good_commit_messages diff --git a/docs/source/aboutcode-docs/writing_good_commit_messages.rst b/docs/source/aboutcode-docs/writing_good_commit_messages.rst index 34d51986..1a032f25 100644 --- a/docs/source/aboutcode-docs/writing_good_commit_messages.rst +++ b/docs/source/aboutcode-docs/writing_good_commit_messages.rst @@ -32,27 +32,24 @@ Body: Other comments: -We like to suffix the subject line with an issue number. If this was a trivial change it may not have one though. If it had one a you would use ``#156`` as a suffix to the first line. +We like to suffix the subject line with an issue number. If this was a trivial change it may not +have one though. If it had one a you would use ``#156`` as a suffix to the first line. -We like to tell why the commit is there and use an imperative style, like if you were giving an order to the codebase with your commit: +We like to tell why the commit is there and use an imperative style, like if you were giving an +order to the codebase with your commit: e.g rather than : ``Minor fix for unnecessary operations.`` may be ``Remove unnecessary operations #123`` -or +or:: -:: + Remove unnecessary operations #123 - Remove unnecessary operations #123 + * If the ts timestamp does not exist, do not compare with old one. - * If the ts timestamp does not exist, do not compare with old one. +You need to add a signoff to your commit. So the final message would have looked like this:: + Remove unnecessary operations #123 -You need to add a signoff to your commit. So the final message would have looked like this: + * If the ts timestamp does not exist, do not compare with old one. -:: - - Remove unnecessary operations #123 - - * If the ts timestamp does not exist, do not compare with old one. - - Signed-off-by: Philippe Ombredanne + Signed-off-by: Philippe Ombredanne diff --git a/docs/source/aboutcode-toolkit/home.rst b/docs/source/aboutcode-toolkit/home.rst index 51292e19..2551512c 100644 --- a/docs/source/aboutcode-toolkit/home.rst +++ b/docs/source/aboutcode-toolkit/home.rst @@ -101,7 +101,7 @@ The AboutCode Toolkit is available through GitHub. For the latest version visit: HACKING ------- We accept pull requests provided under the same license as this tool. -You agree to the http://developercertificate.org/ +You agree to the http://developercertificate.org/ LICENSE diff --git a/docs/source/aboutcode-toolkit/index.rst b/docs/source/aboutcode-toolkit/index.rst index 20a7f03c..0778a96c 100644 --- a/docs/source/aboutcode-toolkit/index.rst +++ b/docs/source/aboutcode-toolkit/index.rst @@ -6,4 +6,4 @@ home reference - specification \ No newline at end of file + specification diff --git a/docs/source/aboutcode-toolkit/reference.rst b/docs/source/aboutcode-toolkit/reference.rst index e8777f3e..e048b92f 100644 --- a/docs/source/aboutcode-toolkit/reference.rst +++ b/docs/source/aboutcode-toolkit/reference.rst @@ -110,7 +110,7 @@ Options Users can use the following in the template to get the vartext: {{ vartext_dict['title'] }} - {{ vartext_dict['header'] }} + {{ vartext_dict['header'] }} --verbose @@ -121,9 +121,12 @@ Options The following data are passed to jinja2 and, therefore, can be used for a custom template: * about object: the about objects * common_licenses: a common license keys list in licenses.py - * license_key_and_context: a dictionary list with license_key as a key and license text as the value - * license_file_name_and_key: a dictionary list with license file name as a key and license key as the value - * license_key_to_license_name: a dictionary list with license key as a key and license file name as the value + * license_key_and_context: a dictionary list with license_key as a key and license text as the + value + * license_file_name_and_key: a dictionary list with license file name as a key and license key + as the value + * license_key_to_license_name: a dictionary list with license key as a key and license file name + as the value check @@ -225,7 +228,7 @@ Options --license-notice-text-location - Copy the license files and notice files to the generated location based on the + Copy the license files and notice files to the generated location based on the 'license_file' and 'notice_file' value in the input from the directory For instance, @@ -288,7 +291,7 @@ Options :: -filter TEXT - + Filter for the output inventory. $ about inventory --filter "license_expression=gpl-2.0" LOCATION OUTPUT @@ -296,7 +299,7 @@ Options The above command will only inventory the ABOUT files which have the "license_expression: gpl-2.0" -f, --format [json|csv] - + Set OUTPUT file format. [default: csv] $ about inventory -f json LOCATION OUTPUT @@ -317,7 +320,7 @@ Options In the custom mapping file, the left side is the custom key name where the right side is the ABOUT field name. For instance, Component: name - + The "Component" is a custom field name for the output The "name" is one of the defaul ABOUT field name that user want to convert diff --git a/docs/source/aboutcode-toolkit/specification.rst b/docs/source/aboutcode-toolkit/specification.rst index df4ead8f..d58b473c 100644 --- a/docs/source/aboutcode-toolkit/specification.rst +++ b/docs/source/aboutcode-toolkit/specification.rst @@ -228,7 +228,7 @@ Referencing the file or directory documented by an ABOUT file An ABOUT file documents one file or directory. The mandatory "about_resource" field reference the documented file or directory. The value of the -"about_resource" field is the name or path of the referenced file or directory. +"about_resource" field is the name or path of the referenced file or directory. A tool processing an ABOUT file must report an error if this field is missing. @@ -402,7 +402,7 @@ These fields support checksums (such as SHA1 and MD5)commonly provided with downloaded archives to verify their integrity. A tool can optionally use these to verify the integrity of a file documented by an ABOUT file. -- checksum_md5: MD5 for the file documented by this ABOUT file in the +- checksum_md5: MD5 for the file documented by this ABOUT file in the "about_resource" field. - checksum_sha1: SHA1 for the file documented by this ABOUT file in the diff --git a/docs/source/deltacode/comprehensive_installation.rst b/docs/source/deltacode/comprehensive_installation.rst index bd2b0af3..e387a869 100644 --- a/docs/source/deltacode/comprehensive_installation.rst +++ b/docs/source/deltacode/comprehensive_installation.rst @@ -1,45 +1,52 @@ Comprehensive Installation ========================== -DeltaCode requires Python 2.7.x and is tested on Linux, Mac, and Windows. Make sure Python 2.7 is installed first. +DeltaCode requires Python 2.7.x and is tested on Linux, Mac, and Windows. Make sure Python 2.7 is +installed first. System Requirements ------------------- -- Hardware : DeltaCode will run best with a modern X86 processor and at least 1GB of RAM and 250MB of disk. +- Hardware : DeltaCode will run best with a modern X86 processor and at least 1GB of RAM and 250MB + of disk. - Supported operating systems : DeltaCode should run on these OSes: - #. Linux: on most recent 64-bit Linux distributions (32-bit distros are only partially supported), - #. Mac: on recent Mac OSX (10.6.8 and up), - #. Windows: on Windows 7 and up (32- or 64-bit) using a 32-bit Python. + #. Linux: on most recent 64-bit Linux distributions (32-bit distros are only partially + supported), + #. Mac: on recent Mac OSX (10.6.8 and up), + #. Windows: on Windows 7 and up (32- or 64-bit) using a 32-bit Python. Prerequisites ------------- DeltaCode needs a Python 2.7 interpreter. -- **On Linux**: - - Use your package manager to install ``python2.7``. If Python 2.7 is not available from your package manager, you must compile it from sources. - - For instance, visit https://github.com/dejacode/about-code-tool/wiki/BuildingPython27OnCentos6 for instructions to compile Python from sources on Centos. +- **On Linux**: -- **On Windows**: - - Use the Python 2.7 32-bit (e.g. the Windows x86 MSI installer) for X86 regardless of whether you run Windows on 32-bit or 64-bit. **DO NOT USE Python X86_64 installer** even if you run 64 bit Windows. + Use your package manager to install ``python2.7``. If Python 2.7 is not available from your + package manager, you must compile it from sources. - Download Python from this url: - https://www.python.org/ftp/python/2.7.14/python-2.7.14.msi + For instance, visit https://github.com/dejacode/about-code-tool/wiki/BuildingPython27OnCentos6 + for instructions to compile Python from sources on Centos. - Install Python on the c: drive and use all default installer options. - See the Windows installation section for more installation details. +- **On Windows**: -- **On Mac**: + Use the Python 2.7 32-bit (e.g. the Windows x86 MSI installer) for X86 regardless of whether + you run Windows on 32-bit or 64-bit. **DO NOT USE Python X86_64 installer** even if you run + 64 bit Windows. - Download and install Python from this url: + Download Python from this url: + https://www.python.org/ftp/python/2.7.14/python-2.7.14.msi - https://www.python.org/ftp/python/2.7.14/python-2.7.14-macosx10.6.pkg + Install Python on the c: drive and use all default installer options. + See the Windows installation section for more installation details. + +- **On Mac**: + + Download and install Python from this url: + + https://www.python.org/ftp/python/2.7.14/python-2.7.14-macosx10.6.pkg Installation on Linux and Mac ----------------------------- @@ -49,7 +56,7 @@ https://github.com/nexB/deltacode/releases/latest Open a terminal in the extracted directory and run:: - ./deltacode --help + ./deltacode --help This will configure DeltaCode and display the command line help. @@ -79,7 +86,7 @@ https://github.com/nexB/deltacode/releases/latest :: - cmd + cmd - Select 'cmd.exe' listed in the search results. @@ -89,19 +96,21 @@ https://github.com/nexB/deltacode/releases/latest :: - cd + cd -- then right-click in this window and select Paste. This will paste the path where you extracted DeltaCode. +- then right-click in this window and select Paste. This will paste the path where you extracted + DeltaCode. - Press Enter. -- This will change the current location of your command prompt to the root directory where DeltaCode is installed. +- This will change the current location of your command prompt to the root directory where + DeltaCode is installed. - Then type :: - deltacode --help + deltacode --help - Press enter. This will configure your DeltaCode installation. diff --git a/docs/source/deltacode/deltacode_output.rst b/docs/source/deltacode/deltacode_output.rst index 635c1aa1..ea28c45b 100644 --- a/docs/source/deltacode/deltacode_output.rst +++ b/docs/source/deltacode/deltacode_output.rst @@ -6,8 +6,8 @@ Deltacode Output: Format, Fields and Structure Usage: deltacode [OPTIONS] Identify the changes that need to be made to the 'old' scan file (-o or --old) - in order to generate the 'new' scan file (-n or --new). Write the results to - a .json file (-j or --json-file) at a user-designated location. If no file + in order to generate the 'new' scan file (-n or --new). Write the results to + a .json file (-j or --json-file) at a user-designated location. If no file option is selected, print the JSON results to the console. Options: @@ -16,21 +16,26 @@ Deltacode Output: Format, Fields and Structure -n, --new PATH Identify the path to the "new" scan file [required] -o, --old PATH Identify the path to the "old" scan file [required] -j, --json-file FILENAME Identify the path to the .json output file - -a, --all-delta-types Include unmodified files as well as all changed - files in the .json output. If not selected, only + -a, --all-delta-types Include unmodified files as well as all changed + files in the .json output. If not selected, only changed files are included. Output Formats -------------- -DeltaCode provides two output formats for the results of a DeltaCode codebase comparison: ``JSON`` and ``CSV``. +DeltaCode provides two output formats for the results of a DeltaCode codebase comparison: ``JSON`` +and ``CSV``. -The default output format is ``JSON``. If the command-line input does not include an output flag (``-j`` or ``--json-file``) and the path to the output file, the results of the DeltaCode comparison will be displayed in the console in ``JSON`` format. Alternatively, the results will be saved to a ``.json`` file if the user includes the ``-j`` or ``--json-file`` flag and the output file's path, -e.g.:: +The default output format is ``JSON``. If the command-line input does not include an output flag +(``-j`` or ``--json-file``) and the path to the output file, the results of the DeltaCode +comparison will be displayed in the console in ``JSON`` format. Alternatively, the results will be +saved to a ``.json`` file if the user includes the ``-j`` or ``--json-file`` flag and the output +file's path, e.g.:: deltacode -n [path to the 'new' codebase] -o [path to the 'old' codebase] -j [path to the JSON output file] -Once a user has generated a DeltaCode JSON output file, he or she can convert that ``JSON`` output to ``CSV`` format by running a command with this structure::: +Once a user has generated a DeltaCode JSON output file, he or she can convert that ``JSON`` output +to ``CSV`` format by running a command with this structure::: python etc/scripts/json2csv.py [path to the JSON input file] [path to the CSV output file] @@ -46,25 +51,39 @@ JSON DeltaCode's ``JSON`` output comprises the following six fields/keys and values at the top level: -- ``deltacode_notice`` -- A string of the terms under which the DeltaCode output is provided. +#. ``deltacode_notice`` -- A string of the terms under which the DeltaCode output is provided. -- ``deltacode_options`` -- A JSON object containing three key/value pairs: +#. ``deltacode_options`` -- A JSON object containing three key/value pairs: - * ``--new`` -- A string identifying the path to the ``JSON`` file containing the ScanCode output of the codebase the user wants DeltaCode to treat as the 'new' codebase. - * ``--old`` -- A string identifying the path to the JSON file containing the ScanCode output of the codebase the user wants DeltaCode to treat as the 'old' codebase. - * ``--all-delta-types`` -- A ``true`` or ``false`` value. - - This value will be true if the command-line input includes the ``-a`` or ``--all-delta-types`` flag, in which case the deltas field described below will include details for unmodified files as well as all changed files. - - If the user does not include the ``-a`` or ``--all-delta-types`` flag, the value will be false and unmodified files will be omitted from the DeltaCode output. + * ``--new`` -- A string identifying the path to the ``JSON`` file containing the ScanCode + output of the codebase the user wants DeltaCode to treat as the 'new' codebase. + * ``--old`` -- A string identifying the path to the JSON file containing the ScanCode output of + the codebase the user wants DeltaCode to treat as the 'old' codebase. + * ``--all-delta-types`` -- A ``true`` or ``false`` value. + - This value will be true if the command-line input includes the ``-a`` or + ``--all-delta-types`` flag, in which case the deltas field described below will include + details for unmodified files as well as all changed files. + - If the user does not include the ``-a`` or ``--all-delta-types`` flag, the value will be + false and unmodified files will be omitted from the DeltaCode output. -- ``deltacode_version`` -- A string representing the version of DeltaCode on which the codebase comparison was run. +#. ``deltacode_version`` -- A string representing the version of DeltaCode on which the codebase + comparison was run. -- ``deltacode_errors`` -- A list of one or more strings identifying errors (if any) that occurred during the codebase-comparison process. +#. ``deltacode_errors`` -- A list of one or more strings identifying errors (if any) that occurred + during the codebase-comparison process. -- ``deltas_count`` -- An integer representing the number of 'Delta' objects -- the file-level comparisons of the two codebases (discussed in the next section) -- contained in the DeltaCode output's ``deltas`` key/value pair. +#. ``deltas_count`` -- An integer representing the number of 'Delta' objects -- the file-level + comparisons of the two codebases (discussed in the next section) -- contained in the DeltaCode + output's ``deltas`` key/value pair. - * If the user's command-line input does not include the ``-a`` or ``--all-delta-types`` flag (see the discussion above of the ``--all-delta-types`` field/key), the DeltaCode output will omit details for unmodified files and consequently the deltas_count field will not include unmodified files. + * If the user's command-line input does not include the ``-a`` or ``--all-delta-types`` flag + (see the discussion above of the ``--all-delta-types`` field/key), the DeltaCode output will + omit details for unmodified files and consequently the deltas_count field will not include + unmodified files. -- ``deltas`` -- A list of 'Delta' objects, each of which represents a file-level comparison (i.e., the "delta") of the 'new' and 'old' codebases. The Delta object is discussed in further detail in the next section. +#. ``deltas`` -- A list of 'Delta' objects, each of which represents a file-level comparison (i.e., + the "delta") of the 'new' and 'old' codebases. The Delta object is discussed in further detail + in the next section. This is the top-level ``JSON`` structure of the key/value pairs described above:: @@ -85,7 +104,8 @@ This is the top-level ``JSON`` structure of the key/value pairs described above: Each Delta object consists of four key/value pairs: -- ``factors``: A list of one or more strings representing the factors that characterize the file-level comparison and are used to calculate the resulting score, e.g. :: +- ``factors``: A list of one or more strings representing the factors that characterize the + file-level comparison and are used to calculate the resulting score, e.g. :: "factors": [ "added", @@ -93,11 +113,20 @@ Each Delta object consists of four key/value pairs: "copyright info added" ], -The possible values for the factors field are discussed in some detail in DeltaCode Scoring :ref:`deltacode_scoring`. - -- ``score``: An integer representing the magnitude/importance of the file-level change -- the higher the ``score``, the greater the change. For further details about the DeltaCode scoring system, see DeltaCode Scoring :ref:`deltacode_scoring`. -- ``new``: A 'File' object containing key/value pairs of certain ScanCode-based file attributes (``path``, ``licenses``, ``copyrights`` etc.) for the file in the codebase designated by the user as ``new``. If the Delta object represents the removal of a file (the ``factors`` value would be ``removed``), the value of ``new`` will be ``null``. -- ``old``: A 'File' object containing key/value pairs of certain ScanCode-based file attributes for the file in the codebase designated by the user as ``old``. If the Delta object represents the addition of a file (the ``factors`` value would be ``added``), the value of ``old`` will be ``null``. +The possible values for the factors field are discussed in some detail in DeltaCode Scoring +:ref:`deltacode_scoring`. + +- ``score``: An integer representing the magnitude/importance of the file-level change -- the + higher the ``score``, the greater the change. For further details about the DeltaCode scoring + system, see DeltaCode Scoring :ref:`deltacode_scoring`. +- ``new``: A 'File' object containing key/value pairs of certain ScanCode-based file attributes + (``path``, ``licenses``, ``copyrights`` etc.) for the file in the codebase designated by the + user as ``new``. If the Delta object represents the removal of a file (the ``factors`` value + would be ``removed``), the value of ``new`` will be ``null``. +- ``old``: A 'File' object containing key/value pairs of certain ScanCode-based file attributes + for the file in the codebase designated by the user as ``old``. If the Delta object represents + the addition of a file (the ``factors`` value would be ``added``), the value of ``old`` will be + ``null``. The JSON structure of a Delta object looks like this::: @@ -145,18 +174,26 @@ As you saw in the preceding section, the File object has the following JSON stru A File object consists of eight key/value pairs: - ``path``: -- A string identifying the path to the file in question. - In processing the 'new' and 'old' codebases to be compared, DeltaCode may modify the codebases' respective file paths in order to properly align them for comparison purposes. As a result, a File object's ``path`` value may differ to some extent from its ``original_path`` value (see below). + In processing the 'new' and 'old' codebases to be compared, DeltaCode may modify the codebases' + respective file paths in order to properly align them for comparison purposes. As a result, a + File object's ``path`` value may differ to some extent from its ``original_path`` value + (see below). - ``type``: -- A string indicating whether the object is a ``file`` or a ``directory``. - ``name``: -- A string reflecting the name of the file. - ``size``: -- An integer reflecting the size of the file in KB. - ``sha1``: -- A string reflecting the file's sha1 value. -- ``original_path``: -- A string identifying the file's path as it exists in the codebase, prior to any processing by DeltaCode to modify the path for purposes of comparing the two codebases. -- ``licenses``: -- A list of License objects reflecting all licenses identified by ScanCode as associated with the file. This list can be empty. -- ``copyrights``: -- A list of Copyright objects reflecting all copyrights identified by ScanCode as associated with the file. This list can be empty. +- ``original_path``: -- A string identifying the file's path as it exists in the codebase, prior to + any processing by DeltaCode to modify the path for purposes of comparing the two codebases. +- ``licenses``: -- A list of License objects reflecting all licenses identified by ScanCode as + associated with the file. This list can be empty. +- ``copyrights``: -- A list of Copyright objects reflecting all copyrights identified by ScanCode + as associated with the file. This list can be empty. **Example of Detailed JSON output** -Here is an example of the detailed DeltaCode output in ``JSON`` format displaying one Delta object in the ``deltas`` key/value pair -- in this case, an excerpt from the ``JSON`` output of a DeltaCode comparison of ``zlib-1.2.11`` and ``zlib-1.2.9``::: +Here is an example of the detailed DeltaCode output in ``JSON`` format displaying one Delta object +in the ``deltas`` key/value pair -- in this case, an excerpt from the ``JSON`` output of a +DeltaCode comparison of ``zlib-1.2.11`` and ``zlib-1.2.9``::: { "deltacode_notice": "Generated with DeltaCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nDeltaCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nDeltaCode is a free software codebase-comparison tool from nexB Inc. and others.\nVisit https://github.com/nexB/deltacode/ for support and download.", @@ -236,12 +273,28 @@ Here is an example of the detailed DeltaCode output in ``JSON`` format displayin CSV ^^^ -Compared with DeltaCode's JSON output, the CSV output is relatively simple, comprising the following seven fields as column headers, with each row representing one Delta object: +Compared with DeltaCode's JSON output, the CSV output is relatively simple, comprising the +following seven fields as column headers, with each row representing one Delta object: - ``Score`` -- An integer representing the magnitude/importance of the file-level change. -- ``Factors`` -- One or more strings -- with no comma or other separators -- representing the factors that characterize the file-level comparison and are used to calculate the resulting score. -- ``Path`` -- A string identifying the file's path in the 'new' codebase unless the Delta object reflects a ``removed`` file, in which case the string identifies the file's path in the 'old' codebase. As noted above, this path may vary to some extent from the file's actual path in its codebase as a result of DeltaCode processing for codebase comparison purposes. -- ``Name`` -- A string reflecting the file's name in the 'new' codebase unless the Delta object reflects a ``removed`` file, in which case the string reflects the file's name in the 'old' codebase. -- ``Type`` -- A string reflecting the file's type ('file' or 'directory') in the 'new' codebase unless the Delta object reflects a ``removed`` file, in which case the string reflects the file's type in the 'old' codebase. -- ``Size`` -- An integer reflecting the file's size in KB in the 'new' codebase unless the Delta object reflects a ``removed`` file, in which case the string reflects the file's size in the 'old' codebase. -- ``Old Path`` -- A string reflecting the file's path in the 'old' codebase if the Delta object reflects a ``moved`` file. If the Delta object does not involve a ``moved`` file, this field is empty. As with the ``Path`` field/column header above, this path may differ to some extent from the file's actual path in its codebase due to DeltaCode processing for codebase comparison purposes. +- ``Factors`` -- One or more strings -- with no comma or other separators -- representing the + factors that characterize the file-level comparison and are used to calculate the resulting + score. +- ``Path`` -- A string identifying the file's path in the 'new' codebase unless the Delta object + reflects a ``removed`` file, in which case the string identifies the file's path in the 'old' + codebase. As noted above, this path may vary to some extent from the file's actual path in its + codebase as a result of DeltaCode processing for codebase comparison purposes. +- ``Name`` -- A string reflecting the file's name in the 'new' codebase unless the Delta object + reflects a ``removed`` file, in which case the string reflects the file's name in the 'old' + codebase. +- ``Type`` -- A string reflecting the file's type ('file' or 'directory') in the 'new' codebase + unless the Delta object reflects a ``removed`` file, in which case the string reflects the file's + type in the 'old' codebase. +- ``Size`` -- An integer reflecting the file's size in KB in the 'new' codebase unless the Delta + object reflects a ``removed`` file, in which case the string reflects the file's size in the + 'old' codebase. +- ``Old Path`` -- A string reflecting the file's path in the 'old' codebase if the Delta object + reflects a ``moved`` file. If the Delta object does not involve a ``moved`` file, this field is + empty. As with the ``Path`` field/column header above, this path may differ to some extent from + the file's actual path in its codebase due to DeltaCode processing for codebase comparison + purposes. diff --git a/docs/source/deltacode/deltacode_scoring.rst b/docs/source/deltacode/deltacode_scoring.rst index c267cad3..def2aeb0 100644 --- a/docs/source/deltacode/deltacode_scoring.rst +++ b/docs/source/deltacode/deltacode_scoring.rst @@ -9,35 +9,53 @@ Delta Objects A File-Level Comparison of Two Codebases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -A Delta object represents the file-level comparison (i.e., the "delta") of two codebases, typically two versions of the same codebase, using ScanCode-generated ``JSON`` output files as input for the comparison process. +A Delta object represents the file-level comparison (i.e., the "delta") of two codebases, typically +two versions of the same codebase, using ScanCode-generated ``JSON`` output files as input for the +comparison process. -Based on how the user constructs the command-line input, DeltaCode's naming convention treats one codebase as the "new" codebase and the other as the "old" codebase::: +Based on how the user constructs the command-line input, DeltaCode's naming convention treats one +codebase as the "new" codebase and the other as the "old" codebase::: - deltacode -n [path to the 'new' codebase] -o [path to the 'old' codebase] [...] + deltacode -n [path to the 'new' codebase] -o [path to the 'old' codebase] [...] .. _basic_scoring: Basic Scoring ^^^^^^^^^^^^^ -A DeltaCode codebase comparison produces a collection of file-level Delta objects. Depending on the nature of the file-level change between the two codebases, each Delta object is characterized as belonging to one of the categories listed below. Each category has an associated score intended to convey its potential importance -- from a license/copyright compliance perspective -- to a user's analysis of the changes between the ``new`` and ``old`` codebases. +A DeltaCode codebase comparison produces a collection of file-level Delta objects. Depending on +the nature of the file-level change between the two codebases, each Delta object is characterized +as belonging to one of the categories listed below. Each category has an associated score intended +to convey its potential importance -- from a license/copyright compliance perspective -- to a +user's analysis of the changes between the ``new`` and ``old`` codebases. In descending order of importance, the categories are: #. ``added``: A file has been added to the ``new`` codebase. -#. ``modified``: The file is contained in both the ``new`` and ``old`` codebase and has been modified (as reflected, among other things, by a change in the file's ``sha1`` attribute). -#. ``moved``: The file is contained in both the ``new`` and ``old`` codebase and has been moved but not modified. +#. ``modified``: The file is contained in both the ``new`` and ``old`` codebase and has been + modified (as reflected, among other things, by a change in the file's ``sha1`` attribute). +#. ``moved``: The file is contained in both the ``new`` and ``old`` codebase and has been moved but + not modified. #. ``removed``: A file has been removed from the ``old`` codebase. -#. ``unmodified``: The file is contained in both the ``new`` and ``old`` codebase and has not been modified or moved. +#. ``unmodified``: The file is contained in both the ``new`` and ``old`` codebase and has not been + modified or moved. -The score of a Delta object characterized as ``added`` or ``modified`` may be increased based on the detection of license- and/or copyright-related changes. See :ref:`license_additions_and_changes` and :ref:`copyright_holder_additions_and_changes` below. +.. note:: + + Files are determined to be Moved by looping thru the `added` and `removed` Delta objects and + checking their sha1 values. + +The score of a Delta object characterized as ``added`` or ``modified`` may be increased based on +the detection of license- and/or copyright-related changes. See +:ref:`license_additions_and_changes` and :ref:`copyright_holder_additions_and_changes` below. Delta Object Fields and Values ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each Delta object includes the following fields and values: -- ``factors``: One or more strings representing the factors that characterize the file-level comparison and resulting score, e.g., in JSON format::: +- ``factors``: One or more strings representing the factors that characterize the file-level + comparison and resulting score, e.g., in JSON format::: "factors": [ "added", @@ -45,37 +63,52 @@ Each Delta object includes the following fields and values: "copyright info added" ], -- ``score``: A number representing the magnitude/importance of the file-level change -- the higher the score, the greater the change. -- ``new``: The ScanCode-based file attributes (``path``, ``licenses``, ``copyrights`` etc.) for the file in the codebase designated by the user as ``new``. -- ``old``: The ScanCode-based file attributes for the file in the codebase designated by the user as ``old``. +- ``score``: A number representing the magnitude/importance of the file-level change -- the + higher the score, the greater the change. +- ``new``: The ScanCode-based file attributes (``path``, ``licenses``, ``copyrights`` etc.) + for the file in the codebase designated by the user as ``new``. +- ``old``: The ScanCode-based file attributes for the file in the codebase designated by the user + as ``old``. -Note that an ``added`` Delta object will have a ``new`` file but no ``old`` file, while a ``removed`` Delta object will have an ``old`` file but not a ``new`` file. In each case, the ``new`` and ``old`` keys will be present but the value for the missing file will be ``null``. +Note that an ``added`` Delta object will have a ``new`` file but no ``old`` file, while a +``removed`` Delta object will have an ``old`` file but not a ``new`` file. In each case, the +``new`` and ``old`` keys will be present but the value for the missing file will be ``null``. .. _license_additions_and_changes: License Additions and Changes ----------------------------- -Certain file-level changes involving the license-related information in a Delta object will increase the object's score. +Certain file-level changes involving the license-related information in a Delta object will +increase the object's score. - An ``added`` Delta object's score will be increased: - * If the ``new`` file contains one or more licenses (``factors`` will include ``license info added``). - * If the the ``new`` file contains any of the following Commercial/Copyleft license categories (``factors`` will include, e.g., ``copyleft`` added): + * If the ``new`` file contains one or more licenses (``factors`` will include + ``license info added``). + * If the the ``new`` file contains any of the following Commercial/Copyleft license + categories (``factors`` will include, e.g., ``copyleft`` added): - - 'Commercial' - - 'Copyleft' - - 'Copyleft Limited' - - 'Free Restricted' - - 'Patent License' - - 'Proprietary Free' + - 'Commercial' + - 'Copyleft' + - 'Copyleft Limited' + - 'Free Restricted' + - 'Patent License' + - 'Proprietary Free' - A ``modified`` Delta object's score will be increased: - * If the ``old`` file has at least one license and the ``new`` file has no licenses (``factors`` will include ``license info removed``). - * If the ``old`` file has no licenses and the ``new`` file has at least one license (``factors`` will include ``license info added``). - * If both the ``old`` file and ``new`` file have at least one license and the license keys are not identical (e.g., the ``old`` file includes an ``mit`` license and an ``apache-2.0`` license and the ``new`` file includes only an ``mit`` license) (``factors`` will include ``license`` change). - * If any of the Commercial/Copyleft license categories listed above are found in the ``new`` file but not in the ``old`` file (``factors`` will include, e.g., ``proprietary free added``). + * If the ``old`` file has at least one license and the ``new`` file has no licenses + (``factors`` will include ``license info removed``). + * If the ``old`` file has no licenses and the ``new`` file has at least one license + (``factors`` will include ``license info added``). + * If both the ``old`` file and ``new`` file have at least one license and the license + keys are not identical (e.g., the ``old`` file includes an ``mit`` license and an + ``apache-2.0`` license and the ``new`` file includes only an ``mit`` license) + (``factors`` will include ``license`` change). + * If any of the Commercial/Copyleft license categories listed above are found in the + ``new`` file but not in the ``old`` file (``factors`` will include, e.g., + ``proprietary free added``). .. _copyright_holder_additions_and_changes: @@ -83,20 +116,29 @@ Certain file-level changes involving the license-related information in a Delta Copyright Holder Additions and Changes -------------------------------------- -- An ``added`` Delta object's score will be increased if the ``new`` file contains one or more copyright ``holders`` (``factors`` will include ``copyright info added``). +- An ``added`` Delta object's score will be increased if the ``new`` file contains one or more + copyright ``holders`` (``factors`` will include ``copyright info added``). - A ``modified`` Delta object's score will be increased: - * If the ``old`` file has at least one copyright ``holder`` and the ``new`` file has no copyright holders (``factors`` will include ``copyright info removed``). - * If the ``old`` file has no copyright ``holders`` and the ``new`` file has at least one (``actors`` will include ``copyright info added``). - * If both the ``old`` file and ``new`` file have at least one copyright ``holder`` and the ``holders`` are not identical (``factors`` will include ``copyright`` change). + * If the ``old`` file has at least one copyright ``holder`` and the ``new`` file has no + copyright holders (``factors`` will include ``copyright info removed``). + * If the ``old`` file has no copyright ``holders`` and the ``new`` file has at least one + (``actors`` will include ``copyright info added``). + * If both the ``old`` file and ``new`` file have at least one copyright ``holder`` and + the ``holders`` are not identical (``factors`` will include ``copyright`` change). Moved, Removed and Unmodified ----------------------------- -As noted above in Basic Scoring :ref:`basic_scoring`, from a license/copyright compliance perspective, the three least significant Delta categories are ``moved``, ``removed`` and ``unmodified``. - -In the current version of DeltaCode, each of these three categories is assigned a score of 0, with no options to increase that score depending on the content of the Delta object. +As noted above in Basic Scoring :ref:`basic_scoring`, from a license/copyright compliance +perspective, the three least significant Delta categories are ``moved``, ``removed`` and +``unmodified``. -However, it is possible that both ``moved`` and ``removed`` will be assigned some non-zero score in a future version. In particular, ``removed`` could be significant from a compliance viewpoint where, for example, the removal of a file results in the removal of a Commercial/Copyleft license obligation. +In the current version of DeltaCode, each of these three categories is assigned a score of 0, with +no options to increase that score depending on the content of the Delta object. +However, it is possible that both ``moved`` and ``removed`` will be assigned some non-zero score in +a future version. In particular, ``removed`` could be significant from a compliance viewpoint +where, for example, the removal of a file results in the removal of a Commercial/Copyleft license +obligation. diff --git a/docs/source/deltacode/development.rst b/docs/source/deltacode/development.rst index 50e65fd5..3dd740af 100644 --- a/docs/source/deltacode/development.rst +++ b/docs/source/deltacode/development.rst @@ -9,32 +9,49 @@ Code layout and conventions Source code is in ``src/``. Tests are in ``tests/``. -Each test script is named ``test_XXXX`` and while we love to use ``py.test`` as a test runner, most tests have no dependencies on ``py.test``\ , only on the ``unittest`` module (with the exception of some command line tests that depend on pytest monkeypatching capabilities. +Each test script is named ``test_XXXX`` and while we love to use ``py.test`` as a test runner, +most tests have no dependencies on ``py.test``\ , only on the ``unittest`` module (with the +exception of some command line tests that depend on pytest monkeypatching capabilities. When source or tests need data files, we store these in a ``data`` subdirectory. -We use PEP8 conventions with a relaxed line length that can be up to 90'ish characters long when needed -to keep the code clear and readable. +We use PEP8 conventions with a relaxed line length that can be up to 90'ish characters long when +needed to keep the code clear and readable. -We store pre-built **bundled** native binaries in ``bin/`` sub-directories of each ``src/`` packages. These binaries are organized by OS and architecture. This ensure that DeltaCode works out of the box either using a checkout or a download, without needing a compiler and toolchain to be installed. +We store pre-built **bundled** native binaries in ``bin/`` sub-directories of each ``src/`` +packages. These binaries are organized by OS and architecture. This ensure that DeltaCode works +out of the box either using a checkout or a download, without needing a compiler and toolchain to +be installed. -We store **bundled** thirdparty components and libraries in the ``thirdparty`` directory. Python libraries are stored as wheels, eventually pre-built if the corresponding wheel is not available in the Pypi repository. -Some of these components may be advanced builds with bug fixes or advanced patches. +We store **bundled** thirdparty components and libraries in the ``thirdparty`` directory. Python +libraries are stored as wheels, eventually pre-built if the corresponding wheel is not available +in the Pypi repository. -We write tests, a lot of tests, thousands of tests. Several tests are data-driven and use data files as test input and sometimes data files as test expectation (in this case using either JSON or YAML files). The tests should pass on Linux 64 bits, Windows 32 and 64 bits and on MacOSX 10.6.8 and up. We maintain two CI loops with Travis (Linux) at https://travis-ci.org/nexB/deltacode and Appveyor (Windows) at https://ci.appveyor.com/project/nexB/deltacode +Some of these components may be advanced builds with bug fixes or advanced patches. + +We write tests, a lot of tests, thousands of tests. Several tests are data-driven and use data +files as test input and sometimes data files as test expectation (in this case using either JSON or +YAML files). The tests should pass on Linux 64 bits, Windows 32 and 64 bits and on MacOSX 10.6.8 +and up. We maintain two CI loops with Travis (Linux) at https://travis-ci.org/nexB/deltacode and +Appveyor (Windows) at https://ci.appveyor.com/project/nexB/deltacode When finding bugs or adding new features, we add tests. See existing test code for examples. Running tests ^^^^^^^^^^^^^ -DeltaCode comes with over 130 unit tests to ensure detection accuracy and stability across Linux, Windows and macOS OSes: we kinda love tests, do we? +DeltaCode comes with over 130 unit tests to ensure detection accuracy and stability across Linux, +Windows and macOS OSes: we kinda love tests, do we? -We use pytest to run the tests: call the ``py.test`` script to run the whole test suite. This is installed by ``pytest`` which is bundled with a DeltaCode checkout and installed when you run ``./configure``\ ). +We use pytest to run the tests: call the ``py.test`` script to run the whole test suite. This is +installed by ``pytest`` which is bundled with a DeltaCode checkout and installed when you run +``./configure``\ ). -If you are running from a fresh git clone and you run ``./configure`` and then ``source bin/activate`` the ``py.test`` command will be available in your path. +If you are running from a fresh git clone and you run ``./configure`` and then +``source bin/activate`` the ``py.test`` command will be available in your path. -Alternatively if you have already configured but are not in an activated "virtualenv" the ``py.test`` command is available under ``/bin/py.test`` +Alternatively if you have already configured but are not in an activated "virtualenv" the +``py.test`` command is available under ``/bin/py.test`` (Note: paths here are for POSIX, but mostly the same applies to Windows) @@ -42,24 +59,39 @@ If you have a multiprocessor machine you might want to run the tests in parallel For instance: ``py.test -n4`` runs the tests on 4 CPUs. We typically run the tests in verbose mode with ``py.test -vvs -n4`` -See also https://docs.pytest.org for details or use the ``py.test -h`` command to show the many other options available. +See also https://docs.pytest.org for details or use the ``py.test -h`` command to show the many +other options available. -One useful option is to run a select subset of the test functions matching a pattern with the ``-k`` option for instance: ``py.test -vvs -k tcpdump`` would only run test functions that contain the string "tcpdump" in their name or their class name or module name . +One useful option is to run a select subset of the test functions matching a pattern with the +``-k`` option for instance: ``py.test -vvs -k tcpdump`` would only run test functions that contain +the string "tcpdump" in their name or their class name or module name . -Another useful option after a test run with some failures is to re-run only the failed tests with the ``--lf`` option for instance: ``py.test -vvs --lf`` would only run only test functions that failed in the previous run. +Another useful option after a test run with some failures is to re-run only the failed tests with +the ``--lf`` option for instance: ``py.test -vvs --lf`` would only run only test functions that +failed in the previous run. pip requirements and the configure script ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -DeltaCode use the ``configure`` and ``configure.bat`` (and ``etc/configure.py`` behind the scenes) scripts to install a `virtualenv `_\ , install required packaged dependencies `as pip `_ requirements and more configure tasks such that DeltaCode can be installed in a self-contained way with **no network connectivity** required. +DeltaCode use the ``configure`` and ``configure.bat`` (and ``etc/configure.py`` behind the scenes) +scripts to install a `virtualenv `_\ , install required +packaged dependencies `as pip `_ requirements and more configure tasks +such that DeltaCode can be installed in a self-contained way with **no network connectivity** +required. DeltaCode requirements and third-party Python libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In a somewhat unconventional way, all the required libraries are bundled aka. copied in the repo itself in the thirdparty/ directory. If DeltaCode were only a library it would not make sense. But its is first an application and having a well defined frozen set of dependent packages is important for an app. -The benefit of this approach (combined with the ``configure`` script) means that a mere checkout of the repository contains **everything** needed to run DeltaCode except for a Python interpreter. +In a somewhat unconventional way, all the required libraries are bundled aka. copied in the repo +itself in the thirdparty/ directory. If DeltaCode were only a library it would not make sense. But +its is first an application and having a well defined frozen set of dependent packages is important +for an app. + +The benefit of this approach (combined with the ``configure`` script) means that a mere checkout of +the repository contains **everything** needed to run DeltaCode except for a Python interpreter. Using DeltaCode as a Python library ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -(Coming Soon) DeltaCode can be used alright as a Python library and is available as as a Python wheel in Pypi and installed with ``pip install deltacode``. +(Coming Soon) DeltaCode can be used alright as a Python library and is available as as a Python +wheel in Pypi and installed with ``pip install deltacode``. diff --git a/docs/source/deltacode/gsoc19_final_report.rst b/docs/source/deltacode/gsoc19_final_report.rst new file mode 100644 index 00000000..b2aab0c9 --- /dev/null +++ b/docs/source/deltacode/gsoc19_final_report.rst @@ -0,0 +1,69 @@ +Google Summer of Code 2019 - Final report +========================================= + +**Project: Approximately similar file detection in DeltaCode** +-------------------------------------------------------------- + +**Arnav Mandal ** + +Project Overview +---------------- + +DeltaCode is a tool to compare and report scan differences. It takes JSON files as an input which +is the output of ScanCode-toolkit as well. When comparing files, it only uses the exact comparison. +By exact comparison, I mean it compares the hash value of the files. The output of DeltaCode is a +JSON/CSV file which includes the details of the scan such as delta score, delta count, etc. The +goal of this project is to improve the usefulness of the delta by also finding files that are +mostly the same (e.g. quasi or near duplicates) vs. files that are completely different. After this +project, DeltaCode would be able to detect similar files in a directory approximately. + +Requirements of the project +--------------------------- + +- Provided two files using ScanCode-toolkit, the new near-duplicate detection should return the + distance between the two files. +- The code should be seamlessly integrated with ScanCode-toolkit. It should be highly configurable + by the maintainers. +- The strictness of near-duplicates should be noted and adjusted by a threshold variable. + +The Project +----------- + +- Addition of new fingerprint plugin in the ScanCode Toolkit. +- Implementation and integration of the fingerprint generation algorithm in the ScanCode Toolkit + codebase. +- Implementation of distance finding algorithm between the files and process them further in the + DeltaCode codebase. +- Integration of fingerprint field in the JSON file to compare the deltas and provide them with + appropriate scores. +- Make changes to old unit tests and addition of new unit tests in ScanCode Toolkit as well as + DeltaCode. + +I have completed all the tasks that were in the scope of this GSoC project. + +Pull Requests +------------- + +- https://github.com/nexB/scancode-toolkit/pull/1576 [Closed] (something went wrong while rebasing) +- https://github.com/nexB/scancode-toolkit/pull/1651 [Merged] +- https://github.com/nexB/deltacode/pull/128 [Merged] + +Links +----- + +.. + [Org Link] https://summerofcode.withgoogle.com/organizations/6118953540124672/ + [Project Link] https://summerofcode.withgoogle.com/projects/#6422961651712000 + +- `Project Details `_ +- `Proposal `_ +- `ScanCode Toolkit `_ +- `DeltaCode `_ + +------------ + +I’ve had a wonderful time during these three months and have learned plenty of things. I would +really like to thank `@pombredanne `_, +`@majurg `_, and `@JonoYang `_ for their +constant support throughout the journey. From good job claps to nit-picky constructive +code-reviews, I enjoyed every bit of this GSoC project. diff --git a/docs/source/deltacode/index.rst b/docs/source/deltacode/index.rst index 5a198ae1..869a33d6 100644 --- a/docs/source/deltacode/index.rst +++ b/docs/source/deltacode/index.rst @@ -1,7 +1,7 @@ **Deltacode Documentation** -================================== +=========================== -Welcome to Deltacode Documentation. +Welcome to Deltacode Documentation. .. toctree:: :maxdepth: 2 @@ -11,4 +11,5 @@ Welcome to Deltacode Documentation. deltacode_scoring development json_to_csv - release_process \ No newline at end of file + release_process + gsoc19_final_report diff --git a/docs/source/deltacode/json_to_csv.rst b/docs/source/deltacode/json_to_csv.rst index 158148d6..2b051cd3 100644 --- a/docs/source/deltacode/json_to_csv.rst +++ b/docs/source/deltacode/json_to_csv.rst @@ -3,14 +3,14 @@ JSON to CSV Conversion ====================== -The default output format for a DeltaCode codebase comparison is JSON. If the ``-j`` or ``--json-file`` option is included in the ``deltacode`` command, the output will be written to a ``.json`` file at the user-designated location. For example: - -:: +The default output format for a DeltaCode codebase comparison is JSON. If the ``-j`` or +``--json-file`` option is included in the ``deltacode`` command, the output will be written to a +``.json`` file at the user-designated location. For example:: deltacode -n [path to the 'new' codebase] -o [path to the 'old' codebase] -j [path to the JSON output file] -We have also created an easy-to-use script for users who want to convert their JSON output to CSV format. Located at ``etc/scripts/json2csv.py``\ , the conversion can be run with this command template: - -:: +We have also created an easy-to-use script for users who want to convert their JSON output to CSV +format. Located at ``etc/scripts/json2csv.py``\ , the conversion can be run with this command +template:: python etc/scripts/json2csv.py [path to the JSON input file] [path to the CSV output file] diff --git a/docs/source/deltacode/release_process.rst b/docs/source/deltacode/release_process.rst index cc796129..27674ea2 100644 --- a/docs/source/deltacode/release_process.rst +++ b/docs/source/deltacode/release_process.rst @@ -30,21 +30,26 @@ merge develop branch in master and tag the release. git push --set-upstream origin master git push --set-upstream origin v1.6.1 -Draft a new release in GitHub, using the previous release blurb as a base. Highlight new and noteworthy changes from the CHANGELOG.rst. +Draft a new release in GitHub, using the previous release blurb as a base. Highlight new and +noteworthy changes from the CHANGELOG.rst. Run etc/release/release.sh locally. Upload the release archives created in the dist/ directory to the GitHub release page. -Save the release as a draft. Use the previous release notes to create notes in the same style. Ensure that the link to thirdparty source code is present. +Save the release as a draft. Use the previous release notes to create notes in the same style. +Ensure that the link to thirdparty source code is present. Test the downloads. Publish the release on GitHub -Then build and publish the released wheel on Pypi. For this you need your own Pypi credentials (and get authorized to publish Pypi release: ask @pombredanne) and you need to have the twine package installed and configured. +Then build and publish the released wheel on Pypi. For this you need your own Pypi credentials +(and get authorized to publish Pypi release: ask @pombredanne) and you need to have the twine +package installed and configured. Build a .whl with ``python setup.py bdist_wheel`` Run twine with ``twine upload dist/`` Once uploaded check the published release at https://pypi.python.org/pypi/deltacode/ -Then create a new fresh local virtualenv and test the wheel installation with: ``pip install deltacode`` +Then create a new fresh local virtualenv and test the wheel installation with: +``pip install deltacode`` diff --git a/docs/source/doc_maintenance.rst b/docs/source/doc_maintenance.rst index a2a8a70d..49d9dc89 100644 --- a/docs/source/doc_maintenance.rst +++ b/docs/source/doc_maintenance.rst @@ -43,11 +43,13 @@ Now you can build the HTML documents locally:: cd docs make html -Assuming that your Sphinx installation was successful, Sphinx should build a local instance of the documentation .html files:: +Assuming that your Sphinx installation was successful, Sphinx should build a local instance of the +documentation .html files:: open build/html/index.html -In case this command did not work, for example on Ubuntu 18.04 you may get a message like “Couldn’t get a file descriptor referring to the console”, try: :: +In case this command did not work, for example on Ubuntu 18.04 you may get a message like “Couldn’t +get a file descriptor referring to the console”, try: :: see build/html/index.html @@ -56,14 +58,17 @@ You now have a local build of the AboutCode documents. Improve AboutCode Documents --------------------------- -Before you begin creating and modifying AboutCode documents, be sure that you understand the basics of reStructuredText as explained at http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +Before you begin creating and modifying AboutCode documents, be sure that you understand the basics +of reStructuredText as explained at +http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html Ensure that you have the latest AboutCode files:: git pull git status -Use your favorite text editor to create and modify .rst files to make your documentation improvements. +Use your favorite text editor to create and modify .rst files to make your documentation +improvements. Review your work:: @@ -71,7 +76,8 @@ Review your work:: make html open build/html/index.html -AboutCode uses Travis-CI to test build status and check links, so run this script at your local system before creating a Pull Request. +AboutCode uses Travis-CI to test build status and check links, so run this script at your local +system before creating a Pull Request. :: @@ -81,7 +87,8 @@ AboutCode uses Travis-CI to test build status and check links, so run this scrip Share AboutCode Document Improvements ------------------------------------- -Follow standard git procedures to upload your new and modified files. The following commands are examples:: +Follow standard git procedures to upload your new and modified files. The following commands are +examples:: git status git add source/index.rst @@ -92,4 +99,5 @@ Follow standard git procedures to upload your new and modified files. The follow git push git status -The AboutCode webhook with ReadTheDocs should rebuild the documentation. You can review your results online. +The AboutCode webhook with ReadTheDocs should rebuild the documentation. You can review your +results online. diff --git a/docs/source/index.rst b/docs/source/index.rst index a32e25dd..390791ca 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -19,7 +19,7 @@ Documentation Guide .. toctree:: :maxdepth: 2 - + help license doc_maintenance @@ -30,7 +30,7 @@ Tutorial Documents .. toctree:: :maxdepth: 2 - + scancode-toolkit/tutorials/index @@ -39,7 +39,7 @@ How-To Documents .. toctree:: :maxdepth: 2 - + scancode-toolkit/how-to-guides/index @@ -48,7 +48,7 @@ Reference Documents .. toctree:: :maxdepth: 2 - + scancode-toolkit/cli-reference/index @@ -57,7 +57,7 @@ Discussion Documents .. toctree:: :maxdepth: 2 - + scancode-toolkit/explanations/index Indices and Tables diff --git a/docs/source/scancode-toolkit/cli-reference/output-format.rst b/docs/source/scancode-toolkit/cli-reference/output-format.rst index ca634622..f36b9a81 100644 --- a/docs/source/scancode-toolkit/cli-reference/output-format.rst +++ b/docs/source/scancode-toolkit/cli-reference/output-format.rst @@ -12,6 +12,12 @@ following options. ---- +.. _output_to_stdout: + +.. include:: /scancode-toolkit/rst_snippets/stdout.rst + +---- + ``--json FILE`` --------------- diff --git a/docs/source/scancode-toolkit/cli-reference/synopsis.rst b/docs/source/scancode-toolkit/cli-reference/synopsis.rst index 4074b85a..a7dc33ad 100644 --- a/docs/source/scancode-toolkit/cli-reference/synopsis.rst +++ b/docs/source/scancode-toolkit/cli-reference/synopsis.rst @@ -70,6 +70,8 @@ is JSON, the entire file being in one line, without whitespace characters. The following example scans will show you how to run a scan with each of the result formats. For the scans, we will use the ``samples`` directory provided with the ScanCode Toolkit. +.. include:: /scancode-toolkit/rst_snippets/tip_snippets/stdout.rst + JSON file output ^^^^^^^^^^^^^^^^ diff --git a/docs/source/scancode-toolkit/contribute/contrib_doc.rst b/docs/source/scancode-toolkit/contribute/contrib_doc.rst index 0c8ee91d..0c705c3a 100644 --- a/docs/source/scancode-toolkit/contribute/contrib_doc.rst +++ b/docs/source/scancode-toolkit/contribute/contrib_doc.rst @@ -26,7 +26,7 @@ In the project root, run the following command:: .. note:: - Only the scancode-toolkit documentation style standards are enforced presently. + Only the scancode-toolkit documentation style standards are enforced presently. A sample output is:: @@ -50,8 +50,8 @@ A sample output is:: Now fix the errors and run again till there isn't any style error in the documentation. -What is Checked -^^^^^^^^^^^^^^^ +What is Checked? +^^^^^^^^^^^^^^^^ PyCQA is an Organization for code quality tools (and plugins) for the Python programming language. Doc8 is a sub-project of the same Organization. Refer this `README `_ for more details. @@ -143,3 +143,18 @@ Extra Style Checks temporary ``.bak`` files generated. Refer the `manual `_ for more information on how to use. +7. Notes and Warning Snippets + + Every ``Note`` and ``Warning`` sections are to be kept in ``rst_snippets/note_snippets/`` and + ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are + frequently used in multiple files. + +Converting from Markdown +------------------------ + +If you want to convert a ``.md`` file to a ``.rst`` file, this `tool `_ +does it pretty well. You'd still have to clean up and check for errors as this contains a lot of +bugs. But this is definitely better than converting everything by yourself. + +This will be helpful in converting GitHub wiki's (Markdown Files) to reStructuredtext files for +Sphinx/ReadTheDocs hosting. diff --git a/docs/source/scancode-toolkit/contribute/gsoc_17_final_report.rst b/docs/source/scancode-toolkit/contribute/gsoc17_final_report.rst similarity index 100% rename from docs/source/scancode-toolkit/contribute/gsoc_17_final_report.rst rename to docs/source/scancode-toolkit/contribute/gsoc17_final_report.rst diff --git a/docs/source/scancode-toolkit/contribute/gsoc19_final_report.rst b/docs/source/scancode-toolkit/contribute/gsoc19_final_report.rst new file mode 100644 index 00000000..c5147358 --- /dev/null +++ b/docs/source/scancode-toolkit/contribute/gsoc19_final_report.rst @@ -0,0 +1,102 @@ +Google Summer of Code 2019 - Final report +========================================= + +Project: **scancode-toolkit** to Python 3 +----------------------------------------- + +**Owner:** `Abhishek Kumar `_ + +**Mentor:** `Philippe Ombredanne `_ + +Overview +-------- + +**Problem:** Since Python 2.7 will retire in few months and will not be maintained any longer. + +**Solution:** `Scancode `__ needs to be ported to +python 3 and all test suites must pass on both version of Python. The main difference that +makes Python 3 better than Python 2.x is that the support for unicode is greatly improved in +Python 3. This will also be useful for scancode as scancode has users in more than 100 languages +and it's easy to translate strings from unicode to other languages. + +**Objective**: To make scancode-toolkit installable on on Python 3.6 and higher, as presently it +installs with Python 2.7 only. + +Implementation +-------------- + +- It was started in development mode(editable mode) and then it was moved to work in virtual + environments. +- I have worked module by module according to the order of hierarchy of modules. For example :All + module is dependent on commoncode, so it must be ported first. In this way we have created the + Porting order: + + 1. commoncode + 2. plugincode + 3. typecode + 4. extractcode + 5. textcode + 6. scancode basics (some tests are integration tests and will have to wait to be ported) + 7. formattedcode, starting with JSON (some tests are integration tests and will have to wait + to be ported) + 8. cluecode + 9. licensedcode + 10. packagedcode (depends on licensecode) + 11. summarycode + 12. fixup the remaining bits and tests + +After porting each module, I have marked these modules as ported ``scanpy3`` with help of +**conffest** plugin (created by `@pombredanne `_). **Conffest** +plugin is heart of this project. Without this, it was very difficult to do. Dependencies was fixed +at the time of porting the module where it was used. + +Challenging part of Project +--------------------------- + +It is very difficult to deal with paths on different operating systems.The issue is around +macOS/Windows/Linux. The first two OS handle unicode paths comfortably on Python 2 and 3 but not +completely on macOS Mojave because its filesystem encoding is APFS. Linux paths are bytes and +os.listdir is broken on Python 2. As a result you can only sanely handle Linux paths as bytes +on Python 2. But on Python 3 path seems to be corrected as ``unicode`` on Linux. + +For more details visit here : + +- https://vstinner.github.io/painful-history-python-filesystem-encoding.html +- `jaraco/path.py#130 `__ + +We came with various Solution: + +- To use pathlib which generally handle paths correctly across platforms. And for backports we use + pathlib 2. But this solution also fails because pathlib 2 does not work as expected wrt unicode + vs bytes. And os.listdir also doesn't work properly. + +- To use `path.py `__ which handles the paths across all the + platforms even on macOS Mojave . + +- Use ``bytes`` on linux and python 3 and ``unicode`` everywhere. + +We choose the third solution because it is most fundamental and simple and easy to use. + +Project was tracked in this ticket `nexB/scancode-toolkit#295 `__ + +**Project link :** `Port Scancode to Python 3 `__ + +.. + [Org Link] https://summerofcode.withgoogle.com/organizations/6118953540124672/ + [Project Link] https://summerofcode.withgoogle.com/projects/#5969926387400704 + +**My contribution :** `List of Commits `__ + +**Note :** Please give your feedback `here `_ + +Outcome +------- + +Now we have liftoff on Python 3 . We are able to run basic scans without errors on develop branch. +You check it by running ``scancode -clipeu samples/ --json-pp - -n4`` . + +At last I would like to thanks my Mentor **@pombredanne** aka +`Philippe Ombredanne `__ . He has helped lot in completing this +project. He is very supportive and responsive. I have learned a lot from him. By his encouragement +and motivation, I am very improving day by day, building and developing my skills. I have completed +all the tasks that were in the scope of this GSoC project. diff --git a/docs/source/scancode-toolkit/contribute/index.rst b/docs/source/scancode-toolkit/contribute/index.rst index 8e289539..d35ce91e 100644 --- a/docs/source/scancode-toolkit/contribute/index.rst +++ b/docs/source/scancode-toolkit/contribute/index.rst @@ -8,4 +8,5 @@ cut_new_release contrib_doc roadmap - gsoc_17_final_report + gsoc17_final_report + gsoc19_final_report diff --git a/docs/source/scancode-toolkit/contribute/roadmap.rst b/docs/source/scancode-toolkit/contribute/roadmap.rst index 8a981bc2..2ba972a4 100644 --- a/docs/source/scancode-toolkit/contribute/roadmap.rst +++ b/docs/source/scancode-toolkit/contribute/roadmap.rst @@ -23,9 +23,9 @@ Packages manifests and dependencies parsers - |clock1030| Windows PE #652 - |white_large_square| RPMs dependencies #649 - |white_large_square| Windows Nuget dependencies #648 -- |white_large_square| Bower packages #654 -- |white_large_square| Python dependencies #653 -- |white_large_square| CRAN +- |white_check_mark| Bower packages #654 +- |clock1030| Python dependencies #653 +- |white_check_mark| CRAN - |white_check_mark| Plain packages - |white_large_square| other Java-related meta files (SBT, Ivy, Gradle, etc.) - |white_large_square| Debian debs @@ -57,22 +57,22 @@ Core features - |white_check_mark| pre scan filtering (ignore binaries, etc) - |white_check_mark| pre/post/ouput plugins! (worked as part of the GSoC by @yadsharaf ) - |white_check_mark| scan plugins (e.g. plugins that run a scan to collect data) -- |clock1030| support Python 3 #295 +- |white_check_mark| support Python 3 #295 - |clock1030| transparent archive extraction (as opposed to on-demand with extractcode) - |clock1030| scancode.yml configuration file for exclusions, defaults, scan failure conditions, etc. - |white_large_square| support scan pipelines and rules to organize more complex scans -- |white_large_square| scan baselining, delta scan and failure conditions (such as license change, - etc) (will be spawned as its own DeltaCode project) +- |white_check_mark| scan baselining, delta scan and failure conditions (such as license change, + etc) ( spawned as its the `DeltaCode `_ project) - |white_large_square| dedupe and similarities to avoid re-scanning. For now only identical files are scanned only once. -- |white_large_square| Improved logging, tracing and error diagnostics -- |clock1030| native support for ABC Data (See :ref:`aboutcode_data` ) +- |clock1030| Improved logging, tracing and error diagnostics +- |white_check_mark| native support for ABC Data (See :ref:`aboutcode_data` ) Classification, summarization and deduction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- |white_check_mark| File classification #426 +- |clock1030| File classification #426 - |white_check_mark| summarize and aggregate data #377 at the top level Source code support (some will be spawned as their own tool) @@ -100,7 +100,7 @@ Packaging - |white_large_square| simpler installation, automated installer - |white_check_mark| distro-friendly packaging -- |white_large_square| unbundle and package as multiple libaries (commoncode, extractcode, etc) +- |clock1030| unbundle and package as multiple libaries (commoncode, extractcode, etc) Documentation ^^^^^^^^^^^^^ @@ -120,12 +120,13 @@ CI integration Other work in progress ---------------------- -- ScanCode server: Spawned as its own project: https://github.com/nexB/scancode-server. Will - include Integration / webhooks for Github, Bitbucket. -- VulnerableCode: NVD and CVE lookups: Spawned as its own project: https://github.com/nexB/vulnerablecode -- ScanCode Workbench: desktop app for scan review: Spawned as its own project: +- |clock1030| ScanCode server: Spawned as its own project: https://github.com/nexB/scancode-server. + Will include Integration / webhooks for Github, Bitbucket. +- |clock1030| VulnerableCode: NVD and CVE lookups: Spawned as its own project: + https://github.com/nexB/vulnerablecode +- |white_check_mark| ScanCode Workbench: desktop app for scan review: Spawned as its own project: https://github.com/nexB/scancode-workbench -- DependentCode: dynamic dependencies resolutions: Spawned as its own project: +- |white_large_square| DependentCode: dynamic dependencies resolutions: Spawned as its own project: https://github.com/nexB/dependentcode Package mining and matching diff --git a/docs/source/scancode-toolkit/index.rst b/docs/source/scancode-toolkit/index.rst index f5d00939..ef228757 100644 --- a/docs/source/scancode-toolkit/index.rst +++ b/docs/source/scancode-toolkit/index.rst @@ -6,6 +6,7 @@ getting-started/index cli-reference/index + plugins/index tutorials/index how-to-guides/index explanations/index diff --git a/docs/source/scancode-toolkit/misc/index.rst b/docs/source/scancode-toolkit/misc/index.rst index 41d9ea08..884e0416 100644 --- a/docs/source/scancode-toolkit/misc/index.rst +++ b/docs/source/scancode-toolkit/misc/index.rst @@ -7,5 +7,3 @@ faq support perf_report - plugin_arch - licence_policy_plugin diff --git a/docs/source/scancode-toolkit/plugins/index.rst b/docs/source/scancode-toolkit/plugins/index.rst new file mode 100644 index 00000000..3091df17 --- /dev/null +++ b/docs/source/scancode-toolkit/plugins/index.rst @@ -0,0 +1,9 @@ +**Plugins** +================= + +.. toctree:: + :maxdepth: 2 + + plugin_arch + licence_policy_plugin + plugin_tutorials diff --git a/docs/source/scancode-toolkit/misc/licence_policy_plugin.rst b/docs/source/scancode-toolkit/plugins/licence_policy_plugin.rst similarity index 94% rename from docs/source/scancode-toolkit/misc/licence_policy_plugin.rst rename to docs/source/scancode-toolkit/plugins/licence_policy_plugin.rst index 97e4dfcf..4f278e0e 100644 --- a/docs/source/scancode-toolkit/misc/licence_policy_plugin.rst +++ b/docs/source/scancode-toolkit/plugins/licence_policy_plugin.rst @@ -10,7 +10,7 @@ attribute: ``license_policy``. Policy File Specification ------------------------- -The Policy file is a YAML (``.yml``) document with the following struture:: +The Policy file is a YAML (``.yml``) document with the following structure:: license_policies: - license_key: mit @@ -26,7 +26,7 @@ The Policy file is a YAML (``.yml``) document with the following struture:: color_code: '#FFcc33' icon: icon-warning-sign -The only required key is ``license_key``, which represents the scancode license key to match +The only required key is ``license_key``, which represents the ScanCode license key to match against the detected licenses in the scan results. In the above example, a descriptive label is added along with a color code and CSS ``id`` name diff --git a/docs/source/scancode-toolkit/misc/plugin_arch.rst b/docs/source/scancode-toolkit/plugins/plugin_arch.rst similarity index 98% rename from docs/source/scancode-toolkit/misc/plugin_arch.rst rename to docs/source/scancode-toolkit/plugins/plugin_arch.rst index a758b6a7..7c998426 100644 --- a/docs/source/scancode-toolkit/misc/plugin_arch.rst +++ b/docs/source/scancode-toolkit/plugins/plugin_arch.rst @@ -3,6 +3,9 @@ Plugin Architecture =================== +Notes: this is the initial design for ScanCode plugins. The actual architecture evolved and is +different. + Abstract: --------- diff --git a/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst b/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst new file mode 100644 index 00000000..d6b31e67 --- /dev/null +++ b/docs/source/scancode-toolkit/plugins/plugin_tutorials.rst @@ -0,0 +1,4 @@ +Plugin Tutorials +================ + +- :ref:`how_to_add_post_scan_plugin` diff --git a/docs/source/scancode-toolkit/rst_snippets/stdout.rst b/docs/source/scancode-toolkit/rst_snippets/stdout.rst new file mode 100644 index 00000000..2919ce28 --- /dev/null +++ b/docs/source/scancode-toolkit/rst_snippets/stdout.rst @@ -0,0 +1,9 @@ +Print to ``stdout`` (Terminal) +------------------------------ + +If you want to format the output in JSON and print it at stdout, you can replace the JSON filename +with a "-", like ``--json-pp -`` instead of ``--json-pp output.json``. + +The following command will output the scan results in JSON format to ``stdout`` (In the Terminal):: + + ./scancode -clpieu --json-pp - samples/ diff --git a/docs/source/scancode-toolkit/rst_snippets/tip_snippets/stdout.rst b/docs/source/scancode-toolkit/rst_snippets/tip_snippets/stdout.rst new file mode 100644 index 00000000..a600e5eb --- /dev/null +++ b/docs/source/scancode-toolkit/rst_snippets/tip_snippets/stdout.rst @@ -0,0 +1,4 @@ +.. Tip:: + + You can also output to ``stdout`` instead of a file. For more information refer + :ref:`output_to_stdout`. diff --git a/docs/source/scancode-toolkit/tutorials/how_to_add_a_post_scan_plugin.rst b/docs/source/scancode-toolkit/tutorials/how_to_add_a_post_scan_plugin.rst new file mode 100644 index 00000000..ae20ca60 --- /dev/null +++ b/docs/source/scancode-toolkit/tutorials/how_to_add_a_post_scan_plugin.rst @@ -0,0 +1,290 @@ +.. _how_to_add_post_scan_plugin: + +Add A Post-Scan Plugin +====================== + +Built-In vs. Optional Installation +---------------------------------- + +Built-In +^^^^^^^^ + +Some post-scan plugins are installed when ScanCode itself is installed, e.g., the +:ref:`license_policy_plugin`, whose code is located here:: + + https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/plugin_license_policy.py + +These plugins do not require any additional installation steps and can be used as soon as ScanCode +is up and running. + +Optional +^^^^^^^^ + +ScanCode is also designed to use post-scan plugins that must be installed separately from the +installation of ScanCode. The code for this sort of plugin is located here:: + + https://github.com/nexB/scancode-toolkit/tree/develop/plugins/ + +This wiki page will focus on optional post-scan plugins. + +Example Post-Scan Plugin: Hello ScanCode +---------------------------------------- + +To illustrate the creation of a simple post-scan plugin, we'll create a hypothetical plugin named +``Hello ScanCode``, which will print ``Hello ScanCode!`` in your terminal after you've run a scan. +Your command will look like something like this:: + + scancode -i -n 2 --hello --json + +We'll start by creating three folders: + +1. Top-level folder -- ``/scancode-hello/`` +2. 2nd-level folder -- ``/src/`` +3. 3rd-level folder -- ``/hello_scancode/`` + +1. Top-level folder -- ``/scancode-hello/`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- In the ``/scancode-toolkit/plugins/`` directory, add a folder with a relevant name, e.g., + ``scancode-hello``. This folder will hold all of your plugin code. + +- Inside the ``/scancode-hello/`` folder you'll need to add a folder named ``src`` and 7 files. + +1. ``/src/`` -- This folder will contain your primary Python code and is discussed in more detail + in the following section. + +The 7 Files are: + +1. ``.gitignore`` -- See, e.g., + `/plugins/scancode-ignore-binaries/.gitignore `_ + +:: + + /build/ + /dist/ + +2. ``apache-2.0.LICENSE`` -- See, e.g., + `/plugins/scancode-ignore-binaries/apache-2.0.LICENSE `_ + +3. ``MANIFEST.in`` + +:: + + graft src + + include setup.py + include setup.cfg + include .gitignore + include README.md + include MANIFEST.in + include NOTICE + include apache-2.0.LICENSE + + global-exclude *.py[co] __pycache__ *.*~ + +4. ``NOTICE`` -- See, e.g., + `/plugins/scancode-ignore-binaries/NOTICE `__ + +5. ``README.md`` + +6. ``setup.cfg`` + +:: + + [metadata] + license_file = NOTICE + + [bdist_wheel] + universal = 1 + + [aliases] + release = clean --all bdist_wheel + +7. ``setup.py`` -- This is an example of what our ``setup.py`` file would look like: + +:: + + #!/usr/bin/env python + # -*- encoding: utf-8 -*- + + from __future__ import absolute_import + from __future__ import print_function + + from glob import glob + from os.path import basename + from os.path import join + from os.path import splitext + + from setuptools import find_packages + from setuptools import setup + + + desc = '''A ScanCode post-scan plugin to to illustrate the creation of a simple post-scan plugin.''' + + setup( + name='scancode-hello', + version='1.0.0', + license='Apache-2.0 with ScanCode acknowledgment', + description=desc, + long_description=desc, + author='nexB', + author_email='info@aboutcode.org', + url='https://github.com/nexB/scancode-toolkit/plugins/scancode-categories', + packages=find_packages('src'), + package_dir={'': 'src'}, + py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')], + include_package_data=True, + zip_safe=False, + classifiers=[ + # complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers + 'Development Status :: 4 - Beta', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2.7', + 'Topic :: Utilities', + ], + keywords=[ + 'scancode', 'plugin', 'post-scan' + ], + install_requires=[ + 'scancode-toolkit', + ], + entry_points={ + 'scancode_post_scan': [ + 'hello = hello_scancode.hello_scancode:SayHello', + ], + } + ) + +2. 2nd-level folder -- ``/src/`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Add an ``__init__.py`` file inside the ``src`` folder. This file can be empty, and is used to + indicate that the folder should be treated as a Python package directory. + +#. Add a folder that will contain our primary code -- we'll name the folder ``hello_scancode``. + If you look at the example of the ``setup.py`` file above, you'll see this line in the + ``entry_points`` section: + +:: + + 'hello = hello_scancode.hello_scancode:SayHello', + +- ``hello`` refers to the name of the command flag. +- The first ``hello_scancode`` is the name of the folder we just created. +- The second ``hello_scancode`` is the name of the ``.py`` file containing our code (discussed in + the next section). +- ``SayHello`` is the name of the ``PostScanPlugin`` class we create in that file (see sample + code below). + +3. 3rd-level folder -- ``/hello_scancode/`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Add an ``__init__.py`` file inside the ``hello_scancode`` folder. As noted above, this file can + be empty. + +#. Add a ``hello_scancode.py`` file. + +Notice at the top of the file +""""""""""""""""""""""""""""" + +:: + + # + # Copyright (c) 2019 nexB Inc. and others. All rights reserved. + # http://nexb.com and https://github.com/nexB/scancode-toolkit/ + # The ScanCode software is licensed under the Apache License version 2.0. + # Data generated with ScanCode require an acknowledgment. + # ScanCode is a trademark of nexB Inc. + # + # You may not use this software except in compliance with the License. + # You may obtain a copy of the License at: http://apache.org/licenses/LICENSE-2.0 + # Unless required by applicable law or agreed to in writing, software distributed + # under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR + # CONDITIONS OF ANY KIND, either express or implied. See the License for the + # specific language governing permissions and limitations under the License. + # + # When you publish or redistribute any data created with ScanCode or any ScanCode + # derivative work, you must accompany this data with the following acknowledgment: + # + # Generated with ScanCode and provided on an "AS IS" BASIS, WITHOUT WARRANTIES + # OR CONDITIONS OF ANY KIND, either express or implied. No content created from + # ScanCode should be considered or used as legal advice. Consult an Attorney + # for any legal advice. + # ScanCode is a free software code scanning tool from nexB Inc. and others. + # Visit https://github.com/nexB/scancode-toolkit/ for support and download. + +Imports +""""""" + +:: + + from __future__ import absolute_import + from __future__ import division + from __future__ import print_function + from __future__ import unicode_literals + + from plugincode.post_scan import PostScanPlugin + from plugincode.post_scan import post_scan_impl + from scancode import CommandLineOption + from scancode import POST_SCAN_GROUP + +Create a ``PostScanPlugin`` class +""""""""""""""""""""""""""""""""" + +The ``PostScanPlugin`` class (see L40-L45 +`code `__) +inherits from the ``CodebasePlugin`` class (see L139-L150 +`code `_ ), +which inherits from the ``BasePlugin`` class (see L38-L136 +`code `__ ). + +:: + + @post_scan_impl + class SayHello(PostScanPlugin): + """ + Illustrate a simple "Hello World" post-scan plugin. + """ + + options = [ + CommandLineOption(('--hello',), + is_flag=True, default=False, + help='Generate a simple "Hello ScanCode" greeting in the terminal.', + help_group=POST_SCAN_GROUP) + ] + + def is_enabled(self, hello, **kwargs): + return hello + + def process_codebase(self, codebase, hello, **kwargs): + """ + Say hello. + """ + if not self.is_enabled(hello): + return + + print('\nHello ScanCode!!\n') + +Load the plugin +--------------- + +- To load and use the plugin in the normal course, navigate to the plugin's root folder (in this + example: ``/plugins/scancode-hello/``) and run ``pip install .`` (don't forget the final ``.``). + +- If you're developing and want to test your work, save your edits and run ``pip install -e .`` + from the same folder. + +More-complex examples +--------------------- + +This Hello ScanCode example is quite simple. For examples of more-complex structures and +functionalities you can take a look at the other post-scan plugins for guidance and ideas. + +One good example is the License Policy post-scan plugin. This plugin is installed when ScanCode +is installed and consequently is not located in the ``/plugins/`` directory used for +manually-installed post-scan plugins. The code for the License Policy plugin can be found at +`/scancode-toolkit/src/licensedcode/plugin_license_policy.py `_ +and illustrates how a plugin can be used to analyze the results of a ScanCode scan using external +data files and add the results of that analysis as a new field in the ScanCode JSON output file. diff --git a/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst b/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst index 8f34d456..76d6a3bb 100644 --- a/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst +++ b/docs/source/scancode-toolkit/tutorials/how_to_format_scan_output.rst @@ -19,15 +19,7 @@ The following commands will output scan results in a formatted json file: To compare the JSON output in different formats refer :ref:`comparing_json`. -Print to ``stdout`` (Terminal) ------------------------------- - -If you want to format the output in JSON and print it at stdout, you can replace the JSON filename -with a "-", like ``--json-pp -`` instead of ``--json-pp output.json``. - -The following command will output the scan results in JSON format to ``stdout`` (In the Terminal):: - - ./scancode -clpieu --jsonlines - samples/ +.. include:: /scancode-toolkit/rst_snippets/stdout.rst HTML ---- diff --git a/docs/source/scancode-toolkit/tutorials/index.rst b/docs/source/scancode-toolkit/tutorials/index.rst index dbb8adfc..4751d213 100644 --- a/docs/source/scancode-toolkit/tutorials/index.rst +++ b/docs/source/scancode-toolkit/tutorials/index.rst @@ -11,3 +11,4 @@ how_to_extract_archives how_to_format_scan_output how_to_set_what_will_be_detected_in_a_scan + how_to_add_a_post_scan_plugin diff --git a/docs/source/scancode-workbench/basics/building.rst b/docs/source/scancode-workbench/basics/building.rst index d7eed62f..95dba9a4 100644 --- a/docs/source/scancode-workbench/basics/building.rst +++ b/docs/source/scancode-workbench/basics/building.rst @@ -16,7 +16,8 @@ MacOS - Python 2.7 - `Node.js `_ >=6.x or later but <=8.9.4 - npm 3.10.x or later but <= 5.2.0 (run ``npm install npm@5.2.0 -g``) -- Command Line Tools for `Xcode `_ (run ``xcode-select --install to install``) +- Command Line Tools for `Xcode `_ + (run ``xcode-select --install to install``) Windows ------- @@ -25,14 +26,17 @@ Windows - npm 3.10.x or later but <= 5.2.0 (``run npm install npm@5.2.0 -g``) - Python v2.7.x - * Make sure your Python path is set. To verify, open a command prompt and type ``python --version``. Then, the version of python will be displayed. - + * Make sure your Python path is set. To verify, open a command prompt and type + ``python --version``. Then, the version of python will be displayed. + - Visual C++ Build Environment: * Either: - - Option 1: Install `Visual C++ Build Tools 2015 `_ (or modify an existing installation) and select Common Tools for Visual C++ during setup. This also works with the free Community and Express for Desktop editions. - - Option 2: `Visual Studio 2015 `_ (Community Edition or better) + - Option 1: Install `Visual C++ Build Tools 2015 `_ + (or modify an existing installation) and select Common Tools for Visual C++ during setup. + This also works with the free Community and Express for Desktop editions. + - Option 2: `Visual Studio 2015 `_ (Community Edition or better) - * Note: Windows 7 requires `.NET Framework 4.5.1 `_ + * Note: Windows 7 requires `.NET Framework 4.5.1 `_ * Launch cmd, ``npm config set msvs_version 2015`` diff --git a/docs/source/scancode-workbench/basics/check_for_errors_in_the_developer_tools.rst b/docs/source/scancode-workbench/basics/check_for_errors_in_the_developer_tools.rst index 2e2e6542..774b870a 100644 --- a/docs/source/scancode-workbench/basics/check_for_errors_in_the_developer_tools.rst +++ b/docs/source/scancode-workbench/basics/check_for_errors_in_the_developer_tools.rst @@ -1,11 +1,14 @@ Check for Errors in the Developer Tools ======================================= -When an unexpected error occurs in ScanCode Workbench, you will normally see a dialog message which provides details about the error and allows you to create an issue. +When an unexpected error occurs in ScanCode Workbench, you will normally see a dialog message which +provides details about the error and allows you to create an issue. .. image:: data/dialog-error.png - :scale: 50 % + :scale: 50 % -If you can reproduce the error, use this approach to get the stack trace and report the issue. Open the Developer Tools with ``Ctrl+Shift+I`` or ``Alt+Cmd+I``. From there, click the Console tab. Include the error that is logged in the issue in a code block or a file attachment. +If you can reproduce the error, use this approach to get the stack trace and report the issue. Open +the Developer Tools with ``Ctrl+Shift+I`` or ``Alt+Cmd+I``. From there, click the Console tab. +Include the error that is logged in the issue in a code block or a file attachment. .. image:: data/developer-tools-error.png diff --git a/docs/source/scancode-workbench/basics/index.rst b/docs/source/scancode-workbench/basics/index.rst index a9132b8a..0b2fdd99 100644 --- a/docs/source/scancode-workbench/basics/index.rst +++ b/docs/source/scancode-workbench/basics/index.rst @@ -7,4 +7,4 @@ Basics scancode_workbench_views building platform_support - check_for_errors_in_the_developer_tools \ No newline at end of file + check_for_errors_in_the_developer_tools diff --git a/docs/source/scancode-workbench/basics/platform_support.rst b/docs/source/scancode-workbench/basics/platform_support.rst index 25edddab..5c008e8f 100644 --- a/docs/source/scancode-workbench/basics/platform_support.rst +++ b/docs/source/scancode-workbench/basics/platform_support.rst @@ -1,48 +1,53 @@ ScanCode Workbench Platform Support =================================== -Our approach for platform support is to focus on one primary release for each of Linux, MacOS and Windows. The Priority definitions are: +Our approach for platform support is to focus on one primary release for each of Linux, MacOS and +Windows. The Priority definitions are: #. Primary - These are the primary platforms for build/test/release on an ongoing basis. -#. Secondary - These are platforms where the primary ScanCode Workbench release for the corresponding OS Group should be forward-compatible, e.g., Windows 7 build should work on Windows 10. Issues reported and traced to a Secondary platform may not be fixed. -#. Tertiary - These are any other platforms not listed as Primary or Secondary. In these cases, we will help users help themselves, but we are likely not to fix Issues that only surface on a Tertiary platform. +#. Secondary - These are platforms where the primary ScanCode Workbench release for the + corresponding OS Group should be forward-compatible, e.g., Windows 7 build should work on + Windows 10. Issues reported and traced to a Secondary platform may not be fixed. +#. Tertiary - These are any other platforms not listed as Primary or Secondary. In these cases, we + will help users help themselves, but we are likely not to fix Issues that only surface on a + Tertiary platform. -+-------------+------------------+------------+------------+--------------------------------------------+ -| OS Group | Desktop OS | Arch | Priority | Notes | -| | Version | | | | -+=============+==================+============+============+============================================+ -| Windows | Windows 7 SP1 | x64 | 1 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Windows | Windows 10 SP? | x64 | 2 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| MacOS | 10.9 Mavericks | x64 | 1 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| MacOS | 10.10 Yosemite | x64 | 2 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| MacOS | 10.11 El Capitan | x64 | 2 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| MacOS | 10.12 Sierra | x64 | 2 | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux Deb | Ubuntu 12.04 | x64 | 1 | From Electron Docs: The prebuilt ia32 | -| | | | | (i686) and x64 (amd64) binaries of | -| | | | | Electron are built on Ubuntu 12.04. | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux Deb | Ubuntu 14.xx | x64 | 2 | Verified to be able to run the prebuilt | -| | | | | binaries of Electron. | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux Deb | Ubuntu 16.xx | x64 | 2 | Verified to be able to run the prebuilt | -| | | | | binaries of Electron. | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux | Fedora 21 | x64 | 2 | Verified to be able to run the prebuilt | -| | | | | binaries of Electron. | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux | Debian 8 | x64 | 2 | Verified to be able to run the prebuilt | -| | | | | binaries of Electron. | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux RH | CentOS 7.xx | x64 | ? | | -+-------------+------------------+------------+------------+--------------------------------------------+ -| Linux RH | RHEL 7.xx | x64 | ? | | -+-------------+------------------+------------+------------+--------------------------------------------+ ++-------------+------------------+---------+------------+-----------------------------------------+ +| OS Group | Desktop OS | Arch | Priority | Notes | +| | Version | | | | ++=============+==================+=========+============+=========================================+ +| Windows | Windows 7 SP1 | x64 | 1 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Windows | Windows 10 SP? | x64 | 2 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| MacOS | 10.9 Mavericks | x64 | 1 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| MacOS | 10.10 Yosemite | x64 | 2 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| MacOS | 10.11 El Capitan | x64 | 2 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| MacOS | 10.12 Sierra | x64 | 2 | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux Deb | Ubuntu 12.04 | x64 | 1 | From Electron Docs: The prebuilt ia32 | +| | | | | (i686) and x64 (amd64) binaries of | +| | | | | Electron are built on Ubuntu 12.04. | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux Deb | Ubuntu 14.xx | x64 | 2 | Verified to be able to run the prebuilt | +| | | | | binaries of Electron. | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux Deb | Ubuntu 16.xx | x64 | 2 | Verified to be able to run the prebuilt | +| | | | | binaries of Electron. | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux | Fedora 21 | x64 | 2 | Verified to be able to run the prebuilt | +| | | | | binaries of Electron. | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux | Debian 8 | x64 | 2 | Verified to be able to run the prebuilt | +| | | | | binaries of Electron. | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux RH | CentOS 7.xx | x64 | ? | | ++-------------+------------------+---------+------------+-----------------------------------------+ +| Linux RH | RHEL 7.xx | x64 | ? | | ++-------------+------------------+---------+------------+-----------------------------------------+ Electron Supported Platforms ---------------------------- @@ -59,15 +64,21 @@ Only 64-bit binaries are provided for MacOS, and the minimum MacOS version suppo Windows ^^^^^^^ -Windows 7 and later are supported, while older operating systems are not supported (and do not work). Both ia32 (x86) and x64 (amd64) binaries are provided for Windows. Please note: the ARM version of Windows is not supported for now. +Windows 7 and later are supported, while older operating systems are not supported (and do not +work). Both ia32 (x86) and x64 (amd64) binaries are provided for Windows. Please note: the ARM +version of Windows is not supported for now. Linux ^^^^^ -The prebuilt ia32 (i686) and x64 (amd64) binaries of Electron are built on Ubuntu 12.04, and the ARM binary is built against ARM v7 with hard-float ABI and NEON for Debian Wheezy. +The prebuilt ia32 (i686) and x64 (amd64) binaries of Electron are built on Ubuntu 12.04, and the +ARM binary is built against ARM v7 with hard-float ABI and NEON for Debian Wheezy. -Whether the prebuilt binary can run on a distribution depends on whether the distribution includes the libraries that Electron is linked to on the building platform, so only Ubuntu 12.04 is guaranteed to work, but the following platforms are also verified to be able to run the prebuilt binaries of Electron: +Whether the prebuilt binary can run on a distribution depends on whether the distribution includes +the libraries that Electron is linked to on the building platform, so only Ubuntu 12.04 is +guaranteed to work, but the following platforms are also verified to be able to run the prebuilt +binaries of Electron: - Ubuntu 12.04 and later - Fedora 21 -- Debian 8 \ No newline at end of file +- Debian 8 diff --git a/docs/source/scancode-workbench/basics/scancode_workbench_views.rst b/docs/source/scancode-workbench/basics/scancode_workbench_views.rst index 81346713..f1b70311 100644 --- a/docs/source/scancode-workbench/basics/scancode_workbench_views.rst +++ b/docs/source/scancode-workbench/basics/scancode_workbench_views.rst @@ -6,20 +6,35 @@ Scancode Workbench Views Directory Tree -------------- -An interactive directory tree is always present on the left side of the application. The tree is expandable and collapsible. This allows the user to navigate the codebase structure. If a directory is selected, only that directory and its sub-files and folders will be shown in the view. Similarly, if a single file is selected, only information for that selected file will be shown. +An interactive directory tree is always present on the left side of the application. The tree is +expandable and collapsible. This allows the user to navigate the codebase structure. If a directory +is selected, only that directory and its sub-files and folders will be shown in the view. +Similarly, if a single file is selected, only information for that selected file will be shown. .. image:: data/navigate-code-tree.gif Table View ---------- -In the table view, the available clues detected by `ScanCode `_ are shown in a tabular format. A user can see provenance clues such as license and copyright information detected by ScanCode. A user can also see the file information (e.g. file type, file size, etc) and package information (package type, primary language of package) that was detected. The columns can be sorted as well as shown or hidden based on what the user’s preferences. Searching for specific clues (license names, copyrights, etc.) is also available in this view. +In the table view, the available clues detected by `ScanCode `_ +are shown in a tabular format. A user can see provenance clues such as license and copyright +information detected by ScanCode. A user can also see the file information (e.g. file type, +file size, etc) and package information (package type, primary language of package) that was +detected. The columns can be sorted as well as shown or hidden based on what the user’s +preferences. Searching for specific clues (license names, copyrights, etc.) is also available in +this view. .. image:: data/table-view.gif Chart Summary View ------------------ -With the chart summary view, a user can select a node in the directory tree (i.e., a directory, folder or file) and display a horizontal bar chart listing the values identified in the scanned codebase -- that is, the clues detected by ScanCode Toolkit -- for a number of different attributes. The attributes are a subset of the columns displayed in the table view, and can be selected by clicking the dropdown at the top of the view. The chart displays the full range of values for the selected directory tree node and attribute and the number of times each value occurs in the scanned codebase. +With the chart summary view, a user can select a node in the directory tree (i.e., a directory, +folder or file) and display a horizontal bar chart listing the values identified in the scanned +codebase -- that is, the clues detected by ScanCode Toolkit -- for a number of different +attributes. The attributes are a subset of the columns displayed in the table view, and can be +selected by clicking the dropdown at the top of the view. The chart displays the full range of +values for the selected directory tree node and attribute and the number of times each value occurs +in the scanned codebase. -.. image:: data/scancode-workbench-chart-summary.gif \ No newline at end of file +.. image:: data/scancode-workbench-chart-summary.gif diff --git a/docs/source/scancode-workbench/index.rst b/docs/source/scancode-workbench/index.rst index 8fd00801..8463b009 100644 --- a/docs/source/scancode-workbench/index.rst +++ b/docs/source/scancode-workbench/index.rst @@ -1,12 +1,17 @@ **Scancode-Workbench Documentation** ==================================== -ScanCode Workbench allows you take the scan results from the ScanCode Toolkit and create a software inventory annotated with your summaries or conclusions (we call these Conclusions) at any levels of the codebase you choose. +ScanCode Workbench allows you take the scan results from the ScanCode Toolkit and create a software +inventory annotated with your summaries or conclusions (we call these Conclusions) at any levels of +the codebase you choose. -The attributes you add (e.g., Name, Version, Owner, License Expression, Copyright) to your Conclusion about a single package or file -- or a higher-level group of packages and/or files -- can then be exported to a JSON or SQLite file. In addition, Conclusions created in ScanCode Workbench can be exported to `DejaCode `_. +The attributes you add (e.g., Name, Version, Owner, License Expression, Copyright) to your +Conclusion about a single package or file -- or a higher-level group of packages and/or files -- +can then be exported to a JSON or SQLite file. In addition, Conclusions created in ScanCode +Workbench can be exported to `DejaCode `_. .. toctree:: :maxdepth: 3 basics/index - tutorials/index \ No newline at end of file + tutorials/index diff --git a/docs/source/scancode-workbench/tutorials/creating_conclusions.rst b/docs/source/scancode-workbench/tutorials/creating_conclusions.rst index b4654c8e..61d86373 100644 --- a/docs/source/scancode-workbench/tutorials/creating_conclusions.rst +++ b/docs/source/scancode-workbench/tutorials/creating_conclusions.rst @@ -3,22 +3,26 @@ Creating Conclusions ==================== -A Conclusion in ScanCode Workbench refers to the documentation of your analysis and conclusions about the name, version, owner, copyright, license expression and other attributes for a single software package or file or -- if you conclude these attributes are shared by a group of packages and/or files -- for that group of packages/files. +A Conclusion in ScanCode Workbench refers to the documentation of your analysis and conclusions +about the name, version, owner, copyright, license expression and other attributes for a single +software package or file or -- if you conclude these attributes are shared by a group of packages +and/or files -- for that group of packages/files. -You can record your Conclusions throughout the codebase you're analyzing, at any level of the codebase (i.e., nodes in the directory tree representing the codebase) you think will best reflect the results of your analysis. +You can record your Conclusions throughout the codebase you're analyzing, at any level of the +codebase (i.e., nodes in the directory tree representing the codebase) you think will best reflect +the results of your analysis. To create a Conclusion, begin by navigating to the ``ScanDataTable`` view. +.. figure:: data/scandatatable-01.png -.. image:: data/scandatatable-01.png - - - -In the directory tree on the left, choose the directory, package or file you want to annotate, right-click that node, and select ``Edit Conclusion`` in the menu that appears. This will display a form with the path to that node displayed at the top of the form, and a series of attribute names (e.g., ``Status``\ , ``Name``\ , ``Version``\ , ``License Expression``\ , ``Owner``\ , ``Copyright``\ ) and associated textboxes, dropdowns or radio buttons to record your input. - - -.. image:: data/conclusion-form-01.png +In the directory tree on the left, choose the directory, package or file you want to annotate, +right-click that node, and select ``Edit Conclusion`` in the menu that appears. This will display a +form with the path to that node displayed at the top of the form, and a series of attribute names +(e.g., ``Status``\ , ``Name``\ , ``Version``\ , ``License Expression``\ , ``Owner``\ , +``Copyright``\ ) and associated textboxes, dropdowns or radio buttons to record your input. +.. figure:: data/conclusion-form-01.png Conclusion Definitions ---------------------- @@ -32,15 +36,31 @@ The following fields are available when creating a Conclusion: * - Conclusion Field - Description * - Status - - Used to document status of Conclusion creation. The dropdown choices: **Analyzed** - A Conclusion has been created, **Needs Attention** - A Conclusion is flagged for further review, **Original Code** - A Conclusion references code from your organization, **Not Reporting** - A Conclusion that will not be reported and can be ignored. + - Used to document status of Conclusion creation. The dropdown choices: **Analyzed** - A + Conclusion has been created, **Needs Attention** - A Conclusion is flagged for further + review, **Original Code** - A Conclusion references code from your organization, + **Not Reporting** - A Conclusion that will not be reported and can be ignored. * - Name - - The primary name for your Conclusion (usually a file, directory or library name). A Conclusion can represent any software-related object. Any Conclusion can contain one or more sub-Conclusions. The combined Conclusion Name and Version must be unique. + - The primary name for your Conclusion (usually a file, directory or library name). A + Conclusion can represent any software-related object. Any Conclusion can contain one or + more sub-Conclusions. The combined Conclusion Name and Version must be unique. * - Version - The Conclusion version number. * - License Expression - - This is the overall license (an individual license or combination of several licenses) for the Conclusion. The Conclusion form will populate a dropdown with any License Expression data detected by ScanCode when the imported scan was run, using the ScanCode License Expression syntax (e.g., ``gpl-2.0`` represents the GPL 2.0 license). The user can also manually add one or more License Expressions by typing in the textbox and then hitting the ``Enter`` key. The License Expression is intended to capture the facts of a license (primarily the text, as provided by its owner), as well as an organization's interpretation and policy regarding that license. + - This is the overall license (an individual license or combination of several licenses) for + the Conclusion. The Conclusion form will populate a dropdown with any License Expression + data detected by ScanCode when the imported scan was run, using the ScanCode License + Expression syntax (e.g., ``gpl-2.0`` represents the GPL 2.0 license). The user can also + manually add one or more License Expressions by typing in the textbox and then hitting the + ``Enter`` key. The License Expression is intended to capture the facts of a license + (primarily the text, as provided by its owner), as well as an organization's interpretation + and policy regarding that license. * - Owner - - An Owner identifies the original creator (copyright holder) of the code covered by the Conclusion. If this Conclusion code is in its original, unmodified state, the Conclusion owner is associated with the original author/publisher. If this Conclusion code has been copied and modified, the Conclusion owner should be the owner that has copied and modified it. + - An Owner identifies the original creator (copyright holder) of the code covered by the + Conclusion. If this Conclusion code is in its original, unmodified state, the Conclusion + owner is associated with the original author/publisher. If this Conclusion code has been + copied and modified, the Conclusion owner should be the owner that has copied and modified + it. * - Copyright - The Copyright notice that applies to the Conclusion code under the License Expression. * - Modified @@ -52,7 +72,8 @@ The following fields are available when creating a Conclusion: * - Notes - Any notes or comments the user wants to record. * - Feature - - The name of a product feature or codebase module that applies to the code covered by the Conclusion. + - The name of a product feature or codebase module that applies to the code covered by the + Conclusion. * - Purpose - The type of code covered by the Conclusion, e.g., Core, Test, Build, Documentation. * - Programming Language @@ -65,4 +86,3 @@ The following fields are available when creating a Conclusion: - The URL for the primary license represented by the License Expression. * - Notice URL - The URL for the license notice that applies to the code covered by the Conclusion. - diff --git a/docs/source/scancode-workbench/tutorials/import_json_and_export_json.rst b/docs/source/scancode-workbench/tutorials/import_json_and_export_json.rst index 2af85665..4d27e1e3 100644 --- a/docs/source/scancode-workbench/tutorials/import_json_and_export_json.rst +++ b/docs/source/scancode-workbench/tutorials/import_json_and_export_json.rst @@ -7,22 +7,30 @@ Import a ScanCode JSON File .. image:: data/import-json-file.gif -* - We have provided a set of sample scans that you can quickly review in ScanCode Workbench in order to get a sense of its functionality and the types of information captured by a scan. The samples are located at `https://github.com/nexB/scancode-workbench/tree/develop/samples `_. - -* - To import a ScanCode JSON file: - - * Open the ``File`` menu and select ``Import JSON File`` (keyboard shortcut: ``Ctrl+I`` or ``⌘+I`` ). - * In the dialog window that opens, navigate to the JSON file you want to import, select the file and click ``Open``. - * You will then be prompted to choose a filename and location to save the JSON file as a SQLite database file. Add a filename, select the folder in which you want to save the SQLite database file, and click ``Save``. - * ScanCode Workbench will then create a SQLite database file from your JSON file, indicated by the status message "Creating Database ..." - * Once the process has finished, the status message will be replaced by an expandable code tree and, to the right of the tree, a table displaying provenance information generated by ScanCode. +* We have provided a set of sample scans that you can quickly review in ScanCode Workbench in order + to get a sense of its functionality and the types of information captured by a scan. The samples + are located at `https://github.com/nexB/scancode-workbench/tree/develop/samples `_. + +* To import a ScanCode JSON file: + + * Open the ``File`` menu and select ``Import JSON File`` (keyboard shortcut: ``Ctrl+I`` or + ``⌘+I`` ). + * In the dialog window that opens, navigate to the JSON file you want to import, select the + file and click ``Open``. + * You will then be prompted to choose a filename and location to save the JSON file as a SQLite + database file. Add a filename, select the folder in which you want to save the SQLite database + file, and click ``Save``. + * ScanCode Workbench will then create a SQLite database file from your JSON file, indicated by + the status message "Creating Database ..." + * Once the process has finished, the status message will be replaced by an expandable code tree + and, to the right of the tree, a table displaying provenance information generated by ScanCode. Export a JSON file ------------------ * To export a JSON file: - * Select the ``File`` menu and then select ``Export JSON File`` (keyboard shortcut: ``Ctrl+E`` or ``⌘+E`` ). - * In the dialog window that opens, add a name for the file, navigate to the directory in which you want to save the file and click ``Save``. + * Select the ``File`` menu and then select ``Export JSON File`` (keyboard shortcut: ``Ctrl+E`` or + ``⌘+E`` ). + * In the dialog window that opens, add a name for the file, navigate to the directory in which + you want to save the file and click ``Save``. diff --git a/docs/source/scancode-workbench/tutorials/license_policy_support.rst b/docs/source/scancode-workbench/tutorials/license_policy_support.rst index fd494f6f..17a5db85 100644 --- a/docs/source/scancode-workbench/tutorials/license_policy_support.rst +++ b/docs/source/scancode-workbench/tutorials/license_policy_support.rst @@ -1,12 +1,16 @@ License Policy support in ScanCode Workbench ============================================ -ScanCode Workbench now has basic support for tracking and viewing license policies that have been applied to a ``scancode-toolkit`` scan. In order for things to work, your initial ``scancode`` scan must be run with the ``--license-policy`` option. You can read more about that here: :ref:`license_policy_plugin`. +ScanCode Workbench now has basic support for tracking and viewing license policies that have been +applied to a ``scancode-toolkit`` scan. In order for things to work, your initial ``scancode`` scan +must be run with the ``--license-policy`` option. You can read more about that here: +:ref:`license_policy_plugin`. The basics ---------- -While the :ref:`license_policy_plugin` can be customized with any number of custom fields and values, ScanCode Workbench currently only supports a pre-defined set of policy labels. +While the :ref:`license_policy_plugin` can be customized with any number of custom fields and +values, ScanCode Workbench currently only supports a pre-defined set of policy labels. .. list-table:: :header-rows: 1 @@ -22,31 +26,34 @@ While the :ref:`license_policy_plugin` can be customized with any number of cus * - scancode_license_key - Restricted License +This means in order to take advantage of ScanCode Workbench's policy features, your +``license-policy.yml`` needs to have ``license_key`` and ``label`` fields at the very least. -This means in order to take advantage of ScanCode Workbench's policy features, your ``license-policy.yml`` needs to have ``license_key`` and ``label`` fields at the very least. +Additionally, in order to take advantage of policy visualizations, ``label`` values must be +one of the 4 above values: Approved License, Prohibited License, Recommended License or Restricted +License. Later versions of ScanCode Workbench will eventually evolve to support more dynamic policy +values. -Additionally, in order to take advantage of policy visualizations, ``label`` values must be one of the 4 above values: Approved License, Prohibited License, Recommended License or Restricted License. Later versions of ScanCode Workbench will eventually evolve to support more dynamic policy values. - -Here is a simple example of a valid ``license-policy.yml`` file: - -:: +Here is a simple example of a valid ``license-policy.yml`` file:: license_policies: - license_key: apache-2.0 label: Approved License - - license_key: apache-1.1 + - license_key: apache-1.1 label: Prohibited License - license_key: lgpl-2.1-plus label: Recommended License - license_key: cpl-1.0 label: Restricted License -After running a scan with that particular ``license-policy.yml`` file, viewing the scan in ScanCode Workbench will look like the following: - - -.. image:: data/policy-view.png +After running a scan with that particular ``license-policy.yml`` file, viewing the scan in ScanCode +Workbench will look like the following: +.. figure:: data/policy-view.png -As you can see, files which have detected licenses that fall under a particular policy will be shown in the JSTree view with specific icons. This way, you are able to quickly see what files fall under a specific policy. +As you can see, files which have detected licenses that fall under a particular policy will be +shown in the JSTree view with specific icons. This way, you are able to quickly see what files +fall under a specific policy. -Additionally, policy details can be found in the scan data view in their own column: License Policy. This column has been added to both the "Origin" column set and "License info" column set. +Additionally, policy details can be found in the scan data view in their own column: License +Policy. This column has been added to both the "Origin" column set and "License info" column set. diff --git a/docs/source/scancode-workbench/tutorials/navigating_the_chart_summary_view.rst b/docs/source/scancode-workbench/tutorials/navigating_the_chart_summary_view.rst index 2781c244..f89c67a3 100644 --- a/docs/source/scancode-workbench/tutorials/navigating_the_chart_summary_view.rst +++ b/docs/source/scancode-workbench/tutorials/navigating_the_chart_summary_view.rst @@ -4,40 +4,35 @@ Navigating the Chart Summary View Display the view ---------------- -Once you have a SQLite file loaded into ScanCode Workbench, displaying the Chart +Once you have a SQLite file loaded into ScanCode Workbench, displaying the Chart Summary View is easy: - #. Select a file or directory in the Tree View on the left. -#. Click the chart icon in the sidebar or open the ``View`` menu and select - ``Chart Summary View`` (keyboard shortcut: ``Ctrl+Shift+D`` or +#. Click the chart icon in the sidebar or open the ``View`` menu and select + ``Chart Summary View`` (keyboard shortcut: ``Ctrl+Shift+D`` or ``⌘+Shift+D`` ). Select an attribute ------------------- +.. figure:: data/scancode-workbench-chart-summary.gif -.. image:: data/scancode-workbench-chart-summary.gif - - -Use the dropdown at the top of the view to select the attribute you want to -examine (e.g., ``Copyright Statements``\ , ``License Key``\ ). These attribute values +Use the dropdown at the top of the view to select the attribute you want to +examine (e.g., ``Copyright Statements``\ , ``License Key``\ ). These attribute values are detected from ScanCode, and can also be viewed in the Table View. -When you select an attribute, the Chart Summary View will automatically refresh -to display a horizontal bar chart showing -- in descending order of frequency -- -each value identified in the scanned codebase for the selected attribute and the -number of times it occurs in the codebase. You can also see the value for a -particular entry in the bar chart in a tooltip that appears when you move your -cursor over the text on the left or the bar on the right. +When you select an attribute, the Chart Summary View will automatically refresh +to display a horizontal bar chart showing -- in descending order of frequency -- +each value identified in the scanned codebase for the selected attribute and the +number of times it occurs in the codebase. You can also see the value for a +particular entry in the bar chart in a tooltip that appears when you move your +cursor over the text on the left or the bar on the right. Filter Chart Summary -------------------- +.. figure:: data/scancode-workbench-chart-summary-filter.gif -.. image:: data/scancode-workbench-chart-summary-filter.gif - - -You can further filter the summary results by choosing a specific directory or -file in the Tree View. The chart will then only show results for that selected +You can further filter the summary results by choosing a specific directory or +file in the Tree View. The chart will then only show results for that selected directory or file. diff --git a/docs/source/scancode-workbench/tutorials/open_and_save_a_sqlite_file.rst b/docs/source/scancode-workbench/tutorials/open_and_save_a_sqlite_file.rst index e0c67785..ed1c2a1d 100644 --- a/docs/source/scancode-workbench/tutorials/open_and_save_a_sqlite_file.rst +++ b/docs/source/scancode-workbench/tutorials/open_and_save_a_sqlite_file.rst @@ -6,13 +6,17 @@ Open a SQLite File * To open a SQLite File: - * Select the ``File`` menu and then select ``Open SQLite File`` (keyboard shortcut: ``Ctrl+O`` or ``⌘+O`` ). - * In the dialog window that opens, navigate to the SQLite file you want to open, select the file and click ``Open``. + * Select the ``File`` menu and then select ``Open SQLite File`` (keyboard shortcut: ``Ctrl+O`` + or ``⌘+O`` ). + * In the dialog window that opens, navigate to the SQLite file you want to open, select the file + and click ``Open``. Save as a New SQLite File ------------------------- * To save as a new SQLite file: - * Select the ``File`` menu and then select ``Save As New SQLite File`` (keyboard shortcut: ``Ctrl+S`` or ``⌘+S`` ). - * In the dialog window that opens, add a name for the file, navigate to the directory in which you want to save the file and click ``Save``. + * Select the ``File`` menu and then select ``Save As New SQLite File`` (keyboard shortcut: + ``Ctrl+S`` or ``⌘+S`` ). + * In the dialog window that opens, add a name for the file, navigate to the directory in which + you want to save the file and click ``Save``.