diff --git a/SPECIFICATION.rst b/SPECIFICATION.rst index bc401883..df83e28a 100644 --- a/SPECIFICATION.rst +++ b/SPECIFICATION.rst @@ -1,28 +1,29 @@ -ABOUT File Specification v3.1 +ABOUT File Specification v4.0 Purpose ~~~~~~~ -An ABOUT file provides a simple way to document the provenance (origin and +An ABOUT file provides a simple way to document the provenance (e.g. origin and license) and other important or interesting information about a software -component. An ABOUT file is a small YAML formatted text file stored in the -codebase side-by-side with the software component file or archive that it -documents. No modification of the documented software is needed. +package. An ABOUT file is a small text file stored in the codebase side-by-side +with the software package file or archive that it documents. No modification of +the documented software is needed. -The ABOUT format is plain text with field name/value pairs separated by a colon. -It is easy to read and create by hand and is designed first for humans, rather -than machines. The format is well-defined and structured just enough to make it -easy to process with software as well. It contains enough information to fulfill -key license requirements such as creating credits or attribution notices, -collecting redistributable source code, or providing information about new -versions of a software component. +The ABOUT file format is plain text using the YAML format e.g. "name: value" +pairs separated by a colon. It is easy to read and create by hand and is +designed first for processing by humans, rather than by machines. The format is +well-defined and structured just enough to make it easy to process with software +as well. An ABOUT file contains enough information to fulfill key license +requirements such as creating credits or attribution notices, collecting +redistributable source code, or providing information about new versions of a +software package. Getting Started ~~~~~~~~~~~~~~~ -A simple and valid ABOUT file named httpd.ABOUT may look like this:: +A simple and valid ABOUT file named httpd.ABOUT looks like this:: about_resource: httpd-2.4.3.tar.gz name: Apache HTTP Server @@ -31,8 +32,8 @@ A simple and valid ABOUT file named httpd.ABOUT may look like this:: download_url: http://archive.apache.org/dist/httpd/httpd-2.4.3.tar.gz license_expression: apache-2.0 licenses: - - key: apache-2.0 - - file: apache-2.0.LICENSE + - key: apache-2.0 + - file: apache-2.0.LICENSE notice_file: httpd.NOTICE copyright: Copyright (c) 2012 The Apache Software Foundation. @@ -41,34 +42,29 @@ The meaning of this ABOUT file is: - The file "httpd-2.4.3.tar.gz" is stored in the same directory and side-by-side with the ABOUT file "httpd.ABOUT" that documents it. -- The name of this component is "Apache HTTP Server" with version "2.4.3". +- The name of this package is "Apache HTTP Server" with version "2.4.3". -- The home URL for this component is http://httpd.apache.org +- The home URL for this package is http://httpd.apache.org - The file "httpd-2.4.3.tar.gz" was originally downloaded from http://archive.apache.org/dist/httpd/httpd-2.4.3.tar.gz -- In the same directory, "apache-2.0.LICENSE" and "httpd.NOTICE" are files that - contain respectively the license text and the notice text for this component. +- This package is licensed under the "apache-2.0" license. -- This component is licensed under "apache-2.0" +- In the same directory, "apache-2.0.LICENSE" and "httpd.NOTICE" are files that + contain respectively the license text and the notice text for this package. Specification ~~~~~~~~~~~~~ -An ABOUT file is an ASCII YAML formatted text file. -Note that while Unicode characters are not supported in -an ABOUT file proper, external files can contain UTF-8 Unicode. - - ABOUT file name ~~~~~~~~~~~~~~~ -An ABOUT file name can use a limited set of characters and is suffixed with a -".ABOUT" extension using any combination of uppercase and lowercase characters. +An ABOUT file name is suffixed with a ".ABOUT" extension (This extension can use +any combination of uppercase and lowercase characters) -A file name can contain only these US-ASCII characters: +An ABOUT file name can contain only these US-ASCII characters: - digits from 0 to 9 - uppercase and lowercase letters from A to Z @@ -81,73 +77,63 @@ A file name can contain only these US-ASCII characters: lowercase file name and an uppercase ABOUT extension. -Lines of text -~~~~~~~~~~~~~ +YAML format, UTF-encoded text +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An ABOUT file contains text using the YAML format. The textmust be UTF-8-encoded. +The YAML style to use is always the block style and never the JSON-like flow +style. When creating ABOUT files, tools must emit YAML in block style. +An ABOUT file can contain only a single YAML document. This document must be a +YAML mapping of field/values. Lines that start with "#" pound sign must be +ignored and treated as comments. + + +Line ending +~~~~~~~~~~~ -An ABOUT file contains lines of US-ASCII text. Lines contain field names/values -pairs. The standard line ending is the LF character. The line ending characters +The standard line ending is the LF character. The line ending characters can be any LF, CR or CR/LF and tools must normalize line endings to LF when -processing an ABOUT file. Empty lines and lines containing only white spaces -that are not part of a field value continuation are ignored. Empty lines are -commonly used to improve the readability of an ABOUT file. +parsing an ABOUT file. When creating ABOUT files, tools must emit LF line +endings. -Field name -~~~~~~~~~~ +Field names +~~~~~~~~~~~ -A field name can contain only these US-ASCII characters: +A field name can contain only these US-ASCII characters and no space. It must +start with a letter: +- lowercase letters from A to Z - digits from 0 to 9 -- uppercase and lowercase letters from A to Z - the "_" underscore sign. -- Field names are not case sensitive. For example, "HOMEPAGE_URL" and "HomePage_url" - represent the same field name. -- A field name must start at the beginning of a new line. It can be followed by - one or more spaces that must be ignored. These spaces are commonly used to - improve the readability of an ABOUT file. +Field values +~~~~~~~~~~~~ +Leading and trailing white spaces in values must be ignored. -Field value -~~~~~~~~~~~ - -The field value is separated from the field name by a ":" colon. The ":" colon -can be followed by one or more spaces that must be ignored. This also applies to -trailing white spaces: they must be ignored. +A field value is either: -The field value is composed of one or more lines of plain US-ASCII printable text. +- a string of one or lines of text. +- a list where each item is prefixed with a "-" dash +- a mapping of field name: value where the field name and value are separated + by ": " a colon and a space. -When a field value contains more than one line of text, a 'literal block' -(using |), or a 'folded block' (using '>') is need. +When a field string value contains more than one line of text, continuing lines +must start with one or more spaces. For instance:: - description: > - This is a long description for a software component that spans - multiple lines with arbitrary line breaks. - -or:: - - description: | - This is a long description for a software component that spans + description: This is a long description for a software package that spans multiple lines with arbitrary line breaks. Fields are mandatory or optional ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -As defined in this specification, a field can be mandatory or optional. Tools -must report an error for missing mandatory fields. - - -Extension and ignored fields -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An ignored field is a field with a name that is not defined in this -specification. Custom extension fields are also supported and must be processed -by tools as ignored fields unless a certain tool can process a certain extension -field. +A field can be mandatory or optional. Tools must report an error for missing +mandatory fields. There is only one mandatory field for now: "about_resource" Fields validation @@ -155,253 +141,186 @@ Fields validation When processing an ABOUT file, tools must report a warning or error if a field is invalid. A field can be invalid for several reasons, such as invalid field -name syntax or invalid content. Tools should report additional validation error +name or an invalid content. Tools should report additional validation error details. The validation process should check that each field name is syntactically correct and that fields contain correct values according to its concise, common sense definition in this specification. For certain fields, additional and specific validations are relevant such as checksum verification, -URL validation, path resolution and verification, and so forth. Tools should -report a warning for ignored fields. +URL validation, path resolution and verification, etc. can be done optionally. +Tools should report an info for custom fields. +Multiple occurrences of a field name is an error. -Fields order and multiple occurrences -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The field order does not matter. Tools should emit ABOUT files using a well +defined order that promotes readability and makes text diffing easier. -The field order does not matter. Multiple occurrences of a field name is not -supported. -The tool processing an ABOUT file or CSV/JSON input will issue an error when a -field name occurs more than once in the input file (as for any other ignored field). +Custom fields +~~~~~~~~~~~~~ + +A custom field is a field with a name that is not defined in this specification. +These fields must be processed by tools as any other fields but are not subject +to content validation. Field referencing a file ~~~~~~~~~~~~~~~~~~~~~~~~ -The actual value of some fields may be contained in another file. This is useful -for long texts or to reference a common text in multiple ABOUT files such as a -common license text. In this case the field name is suffixed with "_file" and -the field value must be a path pointing to the file that contains the actual -value of the field. This path must be a POSIX path relative to the path of the -ABOUT file. The file content must be UTF-8-encoded text. This is in contrast -with field values contained directly in an ABOUT file that must be US-ASCII- -encoded text and allows to support non-ASCII text content. +Certain fields reference a file path such "about_resource", or fields pointing +to a notice file or a license text file. In these case, the path must be a +POSIX path (using a slash "/" as path segments separator) and be relative to the +path of the ABOUT file. -For example, the full license text for a component is often stored in a separate -file named COPYING:: +For notice and license text files, the content must be UTF-8-encoded text. As a +(non-mandatory) convention, the notice files use a .NOTICE file extension and +the license file use a .LICENSE file extension. + +For example, here the license text is stored in a separate file named +gpl-2.0.LICENSE:: licenses: - - file: linux.COPYING + - key: gpl-2.0 + - file: gpl-2.0.LICENSE -In this example, the README file is stored in a doc directory, one directory -above the ABOUT file directory, using a relative POSIX path:: +In this example, the NOTICE file is stored in a "docs" sub-directory. +Note the usage of the POSIX path syntax:: + + notice_file: docs/NOTICE - licenses: - - file: ../docs/ruby.README Field referencing a URL ~~~~~~~~~~~~~~~~~~~~~~~ -The value of a field may reference URLs such as a homepage or a download. In -this case the field name is suffixed with "_url" and the field value must be a -valid absolute URL starting with ftp://, http:// or https://. URLs are -informational and the content they may reference is ignored. For example, a -download URL is referenced this way:: - - download_url: http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.20.tar.bz2 +Some fields contain a URL such as a homepage URL or a download URL. These are +purely informational. URL field names are suffixed with "_url" and the field +value must be a valid absolute URL. Flag fields ~~~~~~~~~~~ -Flag fields have a "true" or "false" value. True, T, Yes or Y , x must be -interpreted as "true" in any case combination. False, F, No or N must be -interpreted as "false" in any case combination. +Some fields are flags with either a true or false value. + +- "True", "T", "Yes", "Y" or "x" in any case combination must be interpreted as + a "true" value. +- "False", "F", "No", "N" in any case combination or the absence of a value must + be interpreted as "false". + Referencing the file or directory documented by an ABOUT file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An ABOUT file documents one file or directory. The mandatory "about_resource" field reference the documented file or directory. The value of the -"about_resource" field is the name or path of the referenced file or directory. +"about_resource" field is the name or path of the referenced file or directory +relative to the ABOUT file location. A tool processing an ABOUT file must report an error if this field is missing. -By convention, an ABOUT file is often stored in the same directory side-by-side -to the file or directory that it documents, but this is not mandatory. +By convention, an ABOUT file is stored side-by-side to the file or directory +that it documents. This is not mandatory but a very common convention. For example, a file named django.ABOUT contains the following field to document the django-1.2.3.tar.gz archive stored in the same directory:: about_resource: django-1.2.3.tar.gz -In this example, the ABOUT file documents a whole sub-directory:: +In this example, an ABOUT file documents a whole linux-kernel-2.6.23 directory:: about_resource: linux-kernel-2.6.23 -In this example, the ABOUT file documents the current directory, using a "." -period to reference it:: +In this example, the ABOUT file documents all the files in the directory where +it is stored, using "." (period) as its "about_resource" value:: about_resource: . -Other Mandatory fields -~~~~~~~~~~~~~~~~~~~~~~ - -When a tool processes an ABOUT file, it must issue an error if these mandatory -field are missing. - -- about_resource: The resource this file referencing to. -- name: Component name. - Optional Information fields ~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- version: Component or package version. A component or package usually has a version, such as a - revision number or hash from a version control system (for a snapshot checked - out from VCS such as Subversion or Git). If not available, the version should - be the date the component was provisioned, in an ISO date format such as - 'YYYY-MM-DD'. - - spec_version: The version of the ABOUT file format specification used for this file. This is provided as a hint to readers and tools in order to support future versions of this specification. -- description: Component description, as a short text. - -- download_url: A direct URL to download the original file or archive documented - by this ABOUT file. - -- homepage_url: URL to the homepage for this component. - -- changelog_file: Changelog file for the component. +- name: Package name. -- notes: Notes and comments about the component. +- version: Package version. A package usually has a version, such as a "1.2.6" + or a revision number or hash from a version control system. + If not available, the version could be the date the packages was created or + fetched in an ISO date format such as'YYYY-MM-DD'. +- description: Package description text. -Optional Owner and Author fields -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- owner: The name of the primary organization or person(s) that owns or provides - the component. - -- owner_url: URL to the homepage for the owner. - -- contact: Contact information (such as an email address or physical address) - for the component owner. - -- author: Name of the organization(s) or person(s) that authored the component. - -- author_file: Author file for the component. - - -Optional Licensing fields -~~~~~~~~~~~~~~~~~~~~~~~~~ - -- copyright: Copyright statement for the component. - -- notice_file: Legal notice or credits for the component. - -- notice_url: URL to a legal notice for the component. +- download_url: A direct URL to download the package file or archive documented + by this ABOUT file. -- license_file: License file that applies to the component. For example, the - name of a license file such as LICENSE or COPYING file extracted from a - downloaded archive. +- homepage_url: URL to the homepage for this package. -- license_url: URL to the license text for the component. +- changelog_file: Changelog file for the package. -- license_expression: The license expression that apply to the component. You - can separate each identifier using " or " and " and " to document the - relationship between multiple license identifiers, such as a choice among - multiple licenses. +- notes: Notes and comments about the package. -- license_name: The license short name for the license. +- vcs_url: a VCS URL as defined in the SPDX specification. For example:: -- license_key: The license key(s) for the component. + vcs_url: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git@b59958d90b3e75a3b66cd31 +- md5: MD5 for the file in the "download_url" field. -Optional Boolean flag fields -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- sha1: SHA1 for the file in the "download_url" field. -- redistribute: Set this flag to yes if the component license requires source - code redistribution. Defaults to no when absent. +- sha256: SHA256 for the file in the "download_url" field. -- attribute: Set this flag to yes if the component license requires publishing - an attribution or credit notice. Defaults to no when absent. +- sha512: SHA512 for the file in the "download_url" field. -- track_changes: Set this flag to yes if the component license requires tracking - changes made to a the component. Defaults to no when absent. +All the checksums above are hex-encoded strings and computed as in the GNU tools. +For example:: -- modified: Set this flag to yes if the component has been modified. Defaults to - no when absent. + md5: f30b9c173b1f19cf42ffa44f78e4b96c -- internal_use_only: Set this flag to yes if the component is used internal only. - Defaults to no when absent. -Optional Extension fields +Optional Licensing fields ~~~~~~~~~~~~~~~~~~~~~~~~~ -You can create extension fields by prefixing them with a short prefix to -distinguish these from the standard fields. You should provide documentation for -these extensions and create or extend existing tools to support these -extensions. Other tools must ignore these extensions. - - -Optional Extension fields to reference files stored in a version control system (VCS) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -These fields provide a simple way to reference files stored in a version control -system. There are many VCS tools such as CVS, Subversion, Git, ClearCase and GNU -Arch. Accurate addressing of a file or directory revision in each tool in a -uniform way may not be possible. Some tools may require access control via -user/password or certificate and this information should not be stored in an -ABOUT file. This extension defines the 'vcs' field extension prefix and a few -common fields to handle the diversity of ways that VCS tools reference files and -directories under version control: +- copyright: Copyright statement for the package. -- vcs_tool: VCS tool such as git, svn, cvs, etc. +- notice_file: Legal notice or credits file for the package. +- notice_url: URL to the notice for this package. -- vcs_repository: Typically a URL or some other identifier used by a VCS tool to - point to a repository such as an SVN or Git repository URL. +- license_expression: The license expression that apply to the package. The + syntax is the SPDX license expression synatx but the license keys should be + ScanCode or DEjaCode license keys. -- vcs_path: Path used by a particular VCS tool to point to a file, directory or - module inside a repository. +- licenses: a list of name/value pairs for each license key in the + license_expression field. + - key: A license key + - name: Short name for this license key. + - url: URL to the license text for this license key. + - file: Path to a file that contains the full text of this license. -- vcs_tag: tag name or path used by a particular VCS tool. +- redistribute: flag set to "yes" if the license requires source code redistribution. -- vcs_branch: branch name or path used by a particular VCS tool. +- attribute: flag set to "yes" if the license requires publishing an attribution + or credit notice. -- vcs_revision: revision identifier such as a revision hash or version number. +- track_changes: flag set to "yes" if the license requires tracking changes + made to a the package. +- modified: flag set to yes if the package has been modified. -Some examples for using the vcs_* extension fields include:: +- internal_use_only: flag set to yes if the package is for internal use only. - vcs_tool: svn - vcs_repository: http://svn.code.sf.net/p/inkscape/code/inkscape_project/ - vcs_path: trunk/inkscape_planet/ - vcs_revision: 22886 +- changelog_file: Path to a file that contains the log of changes made to this package. -or:: - vcs_tool: git - vcs_repository: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git - vcs_path: tools/lib/traceevent - vcs_revision: b59958d90b3e75a3b66cd311661535f94f5be4d1 - - -Optional Extension fields for checksums -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -These fields support checksums (such as SHA1 and MD5)commonly provided with -downloaded archives to verify their integrity. A tool can optionally use these -to verify the integrity of a file documented by an ABOUT file. +Optional Owner and Author fields +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- checksum_md5: MD5 for the file documented by this ABOUT file in the - "about_resource" field. +- owner: The name of the primary organization or person(s) that owns or provides + the package. -- checksum_sha1: SHA1 for the file documented by this ABOUT file in the - "about_resource" field. +- owner_url: URL to the homepage for the owner. -Some examples:: +- contact: Contact information (such as an email address or physical address) + for the package owner. - checksum_md5: f30b9c173b1f19cf42ffa44f78e4b96c +- author: Name of the organization(s) or person(s) that authored the package. diff --git a/about b/about index 411f3279..dc996eb6 100755 --- a/about +++ b/about @@ -1,19 +1,115 @@ #!/bin/bash # -# Copyright (c) 2015 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Copyright (c) 2018 nexB Inc. http://www.nexb.com/ - All rights reserved. # -# cd to the root directory -ABOUT_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -cd "$ABOUT_ROOT_DIR" +# A minimal shell wrapper to the CLI entry point of AboutCode -CONFIGURED_PYTHON=$ABOUT_ROOT_DIR/bin/python -if [ ! -f "$CONFIGURED_PYTHON" ]; then - echo "* Configuring AboutCode ..." - source $ABOUT_ROOT_DIR/configure -fi +################################################################################### +# from https://raw.githubusercontent.com/mkropat/sh-realpath/58c03982cfd8accbcf0c4426a4adf0f120a8b2bb/realpath.sh +# realpath emulation for portability on *nix +# this allow running scancode from aribtrary locations and from symlinks +# +# Copyright (c) 2014 Michael Kropat +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in +# all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +# THE SOFTWARE. + +realpath() { + canonicalize_path "$(resolve_symlinks "$1")" +} + +resolve_symlinks() { + _resolve_symlinks "$1" +} + +_resolve_symlinks() { + _assert_no_path_cycles "$@" || return + + local dir_context path + path=$(readlink -- "$1") + if [ $? -eq 0 ]; then + dir_context=$(dirname -- "$1") + _resolve_symlinks "$(_prepend_dir_context_if_necessary "$dir_context" "$path")" "$@" + else + printf '%s\n' "$1" + fi +} + +_prepend_dir_context_if_necessary() { + if [ "$1" = . ]; then + printf '%s\n' "$2" + else + _prepend_path_if_relative "$1" "$2" + fi +} + +_prepend_path_if_relative() { + case "$2" in + /* ) printf '%s\n' "$2" ;; + * ) printf '%s\n' "$1/$2" ;; + esac +} -source $ABOUT_ROOT_DIR/bin/activate +_assert_no_path_cycles() { + local target path + + target=$1 + shift + + for path in "$@"; do + if [ "$path" = "$target" ]; then + return 1 + fi + done +} + +canonicalize_path() { + if [ -d "$1" ]; then + _canonicalize_dir_path "$1" + else + _canonicalize_file_path "$1" + fi +} + +_canonicalize_dir_path() { + (cd "$1" 2>/dev/null && pwd -P) +} + +_canonicalize_file_path() { + local dir file + dir=$(dirname -- "$1") + file=$(basename -- "$1") + (cd "$dir" 2>/dev/null && printf '%s/%s\n' "$(pwd -P)" "$file") +} + +################################################################################### +# Now run AboutCode "about" proper + +ABOUTCODE_BIN="$( realpath "${BASH_SOURCE[0]}" )" +ABOUTCODE_ROOT_DIR="$( cd "$( dirname "${ABOUTCODE_BIN}" )" && pwd )" + + +ABOUTCODE_CONFIGURED_PYTHON="$ABOUTCODE_ROOT_DIR/bin/python" +if [ ! -f "$ABOUTCODE_CONFIGURED_PYTHON" ]; then + echo "* Configuring AboutCode for first use..." + CONFIGURE_QUIET=1 "$ABOUTCODE_ROOT_DIR/configure" etc/conf +fi -$ABOUT_ROOT_DIR/bin/about "$@" +"$ABOUTCODE_ROOT_DIR/bin/about" "$@" diff --git a/about.ABOUT b/about.ABOUT index fff20799..2f3debf7 100644 --- a/about.ABOUT +++ b/about.ABOUT @@ -2,8 +2,6 @@ about_resource: . name: AboutCode-toolkit about_resource_path: . -version: 3.0.0.dev6 - description: | AboutCode Toolkit is a tool to process ABOUT files. An ABOUT file provides a simple way to document the provenance (origin and license) @@ -12,9 +10,12 @@ description: | homepage_url: http://www.nexb.com/community.html -license: apache-2.0 -license_name: Apache 2.0 -license_file: apache-2.0.LICENSE +license_expression: apache-2.0 +licenses: + - key: apache-2.0 + name: Apache 2.0 + file: apache-2.0.LICENSE + copyright: Copyright (c) 2013-2017 nexB Inc. notice_file: NOTICE diff --git a/appveyor.yml b/appveyor.yml index ceb7ffed..29cb4f29 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -8,9 +8,3 @@ build: off test_script: - set - py.test -vvs tests - -on_success: - - "python etc/scripts/irc-notify.py aboutcode [{project_name}:{branch}] {short_commit}: \"{message}\" ({author}) {color_green}Succeeded,Details: {build_url},Commit: {commit_url}" - -on_failure: - - "python etc/scripts/irc-notify.py aboutcode [{project_name}:{branch}] {short_commit}: \"{message}\" ({author}) {color_red}Failed,Details: {build_url},Commit: {commit_url}" diff --git a/configure b/configure index 094a4682..ff8fbcfe 100755 --- a/configure +++ b/configure @@ -25,4 +25,8 @@ if [[ "$PYTHON_EXE" == "" ]]; then PYTHON_EXE=python fi + $PYTHON_EXE "$CONFIGURE_ROOT_DIR/etc/configure.py" $CFG_CMD_LINE_ARGS +if [ -f "$CONFIGURE_ROOT_DIR/bin/activate" ]; then + source $CONFIGURE_ROOT_DIR/bin/activate +fi diff --git a/configure.bat b/configure.bat index d1a9dc55..d9aa3cc6 100644 --- a/configure.bat +++ b/configure.bat @@ -18,7 +18,6 @@ set ABOUT_ROOT_DIR=%~dp0 @rem there is a space at the end of the set SCANCODE_CLI_ARGS= line ... @rem NEVER remove this! @rem otherwise, this script and scancode do not work. - set ABOUT_CLI_ARGS= @rem Collect/Slurp all command line arguments in a variable :collectarg @@ -42,7 +41,7 @@ if not exist "c:\python27\python.exe" ( echo( echo On Windows, AboutCode requires Python 2.7.x 32 bits to be installed first. echo( - echo Please download and install Python 2.7 ^(Windows x86 MSI installer^) version 2.7.10. + echo Please download and install Python 2.7 ^(Windows x86 MSI installer^) version 2.7.15. echo Install Python on the c: drive and use all default installer options. echo Do NOT install Python v3 or any 64 bits edition. echo Instead download Python from this url and see the README.rst file for more details: diff --git a/etc/configure.py b/etc/configure.py index 253f7e30..68e1954f 100644 --- a/etc/configure.py +++ b/etc/configure.py @@ -126,14 +126,22 @@ def clean(root_dir): Remove cleanable directories and files in root_dir. """ print('* Cleaning ...') - cleanable = '''build bin lib Lib include Include Scripts local - django_background_task.log - develop-eggs eggs parts .installed.cfg - .Python - .cache - pip-selfcheck.json - '''.split() - + cleanable = ''' + __pycache__ + .pytest_cache + build + bin + lib + Lib + include + Include + Scripts + local + .eggs + .cache + pip-selfcheck.json + src/aboutcode_toolkit.egg-info + '''.split() # also clean __pycache__ if any cleanable.extend(find_pycache(root_dir)) diff --git a/setup.cfg b/setup.cfg index c29a174d..fe7360f4 100644 --- a/setup.cfg +++ b/setup.cfg @@ -5,7 +5,7 @@ universal = 1 license_file = NOTICE [aliases] -release = clean --all sdist --formats=bztar,zip bdist_wheel +release = clean --all sdist bdist_wheel [tool:pytest] norecursedirs = diff --git a/setup.py b/setup.py index 676f878b..9dac5e60 100644 --- a/setup.py +++ b/setup.py @@ -24,7 +24,7 @@ def read(*names, **kwargs): setup( name='aboutcode-toolkit', - version='3.4.0.pre1', + version='4.0.0.pre1', license='Apache-2.0', description=( 'AboutCode-toolkit is a tool to document the provenance (origin and license) of ' @@ -32,7 +32,7 @@ def read(*names, **kwargs): 'Collect inventories, generate attribution documentation.' ), long_description=( - 'AttributeCode provides a simple way to document the' + 'AboutCode toolkit provides a simple way to document the' 'provenance (i.e. origin and license) of software components that' 'you use in your project. This documentation is stored in *.ABOUT' 'files, side-by-side with the documented code.' @@ -79,13 +79,15 @@ def read(*names, **kwargs): 'boolean.py >= 3.5, < 4.0', 'license_expression >= 0.94, < 1.0', + + 'attrs', ], extras_require={ ":python_version < '3.6'": ['backports.csv'], }, entry_points={ 'console_scripts': [ - 'about=attributecode.cmd:about', + 'about=aboutcode.cmd:about', ] }, ) diff --git a/src/attributecode/__init__.py b/src/aboutcode/__init__.py similarity index 58% rename from src/attributecode/__init__.py rename to src/aboutcode/__init__.py index 445e30a8..76f62b06 100644 --- a/src/attributecode/__init__.py +++ b/src/aboutcode/__init__.py @@ -18,10 +18,12 @@ from __future__ import print_function from __future__ import unicode_literals -from collections import namedtuple -import logging +from collections import OrderedDict import os +import attr +import saneyaml + try: # Python 2 unicode # NOQA @@ -29,11 +31,10 @@ # Python 3 unicode = str # NOQA -import saneyaml -__version__ = '3.4.0.pre1' +__version__ = '4.0.0.pre1' -__about_spec_version__ = '3.1' +__about_spec_version__ = '4.0' __copyright__ = """ Copyright (c) 2013-2018 nexB Inc. All rights reserved. http://dejacode.org @@ -49,63 +50,62 @@ """ -class Error(namedtuple('Error', ['severity', 'message'])): - """ - An Error data with a severity and message. - """ - def __new__(self, severity, message): - if message: - if isinstance(message, unicode): - message = self._clean_string(message) - else: - message = self._clean_string(unicode(repr(message), encoding='utf-8')) - message = message.strip('"') - - return super(Error, self).__new__( - Error, severity, message) +def message_converter(value): + if value: + if isinstance(value, unicode): + value = clean_string(value) + else: + value = clean_string(unicode(repr(value), encoding='utf-8')) + value = value.strip('"') + return value - def __repr__(self, *args, **kwargs): - sev, msg = self._get_values() - return 'Error(%(sev)s, %(msg)s)' % locals() - def __eq__(self, other): - return repr(self) == repr(other) +@attr.attributes(repr=False) +class Error(object): + """ + An Error data with a severity and message and an optional path attribute. + """ + severity = attr.attrib() + message = attr.attrib(converter=message_converter) + # relative POSIX path of the ABOUT file + path = attr.attrib(default=None) # , repr=False) - def _get_values(self): + def __repr__(self): sev = severities[self.severity] - msg = self._clean_string(repr(self.message)) - return sev, msg + msg = clean_string(repr(self.message)) + return 'Error(%(sev)s, %(msg)s)' % locals() def render(self): - sev, msg = self._get_values() + sev = severities[self.severity] + msg = clean_string(repr(self.message)) return '%(sev)s: %(msg)s' % locals() def to_dict(self, *args, **kwargs): """ Return an ordered dict of self. """ - return self._asdict() + return attr.asdict(self, dict_factory=OrderedDict) - @staticmethod - def _clean_string(s): - """ - Return a cleaned string for `s`, stripping eventual "u" prefixes - from unicode representations. - """ - if not s: - return s - if s.startswith(('u"', "u'")): - s = s.lstrip('u') - s = s.replace('[u"', '["') - s = s.replace("[u'", "['") - s = s.replace("(u'", "('") - s = s.replace("(u'", "('") - s = s.replace("{u'", "{'") - s = s.replace("{u'", "{'") - s = s.replace(" u'", " '") - s = s.replace(" u'", " '") - s = s.replace("\\\\", "\\") + +def clean_string(s): + """ + Return a cleaned string for `s`, stripping eventual "u" prefixes + from unicode representations. + """ + if not s: return s + if s.startswith(('u"', "u'")): + s = s.lstrip('u') + s = s.replace('[u"', '["') + s = s.replace("[u'", "['") + s = s.replace("(u'", "('") + s = s.replace("(u'", "('") + s = s.replace("{u'", "{'") + s = s.replace("{u'", "{'") + s = s.replace(" u'", " '") + s = s.replace(" u'", " '") + s = s.replace("\\\\", "\\") + return s # modeled after the logging levels diff --git a/src/attributecode/__main__.py b/src/aboutcode/__main__.py similarity index 96% rename from src/attributecode/__main__.py rename to src/aboutcode/__main__.py index 32a0ce70..18a6676c 100644 --- a/src/attributecode/__main__.py +++ b/src/aboutcode/__main__.py @@ -20,5 +20,5 @@ if __name__ == '__main__': # pragma: nocover - from attributecode import cmd + from aboutcode import cmd cmd.about() diff --git a/src/aboutcode/api.py b/src/aboutcode/api.py new file mode 100644 index 00000000..0a7fc3c1 --- /dev/null +++ b/src/aboutcode/api.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2013-2017 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +import json + +import click + +from aboutcode import ERROR +from aboutcode import Error +from aboutcode import util +from aboutcode import model +from aboutcode.util import python2 +from aboutcode import CRITICAL + +if python2: # pragma: nocover + from urllib2 import HTTPError # NOQA + from urllib import urlencode # NOQA + from urlparse import urljoin # NOQA + from urlparse import urlparse # NOQA + from urllib import quote # NOQA + from urllib2 import Request # NOQA + from urllib2 import urlopen # NOQA +else: # pragma: nocover + from urllib.error import HTTPError # NOQA + from urllib.parse import urlencode # NOQA + from urllib.parse import urljoin # NOQA + from urllib.parse import urlparse # NOQA + from urllib.parse import quote # NOQA + from urllib.request import Request # NOQA + from urllib.request import urlopen # NOQA + +from license_expression import Licensing + + +""" +API helpers +""" + + +# FIXME: args should start with license_key +def request_license_data(api_url, api_key, license_key): + """ + Return a tuple of (dictionary of license data, list of errors) given a + `license_key`. Send a request to `api_url` authenticating with `api_key`. + """ + headers = {'Authorization': 'Token %s' % api_key} + payload = {'api_key': api_key, 'key': license_key, 'format': 'json'} + + api_url = api_url.rstrip('/') + payload = urlencode(payload) + + full_url = '{api_url}/?{payload}'.format(**locals()) + # handle special characters in URL such as space etc. + quoted_url = quote(full_url, safe="%/:=&?~#+!$,;'@()*[]") + + license_data = {} + errors = [] + try: + request = Request(quoted_url, headers=headers) + response = urlopen(request) + response_content = response.read().decode('utf-8') + license_data = json.loads(response_content) + + if not license_data['results']: + msg = 'Invalid license key: %s' % license_key + errors.append(Error(ERROR, msg)) + + except HTTPError as http_e: + # some auth problem + if http_e.code == 403: + msg = (u"Authorization denied. Invalid '--api-key'. " + u"License generation is skipped.") + errors.append(Error(ERROR, msg)) + else: + # Since no api_url/api_key/network status have + # problem detected, it yields 'license' is the cause of + # this exception. + msg = 'Invalid license key: %s' % license_key + errors.append(Error(ERROR, msg)) + + except Exception as e: + errors.append(Error(ERROR, str(e))) + + finally: + if license_data.get('count') == 1: + license_data = license_data.get('results')[0] + else: + license_data = {} + + return license_data, errors + + +def get_license_details(api_url, api_key, license_key): + """ + Return a License object given a `license_key` using the `api_url` + authenticating with `api_key`. + """ + license_data, errors = request_license_data(api_url, api_key, license_key) + lic = None + key = license_data.get('key') + if key: + is_active = license_data.get('is_active', False) + if not is_active: + errors.append(Error(CRITICAL, 'License key is NOT active: {}'.format(license_key))) + + name = license_data.get('name') + text = license_data.get('full_text') + dje_domain = '{uri.scheme}://{uri.netloc}/'.format(uri=urlparse(api_url)) + dje_license_url = urljoin(dje_domain, 'urn/?urn=urn:dje:license:{license_key}') + url = dje_license_url.format(license_key=license_key) + + lic = model.License(key=key, name=name, text=text, url=url) + return lic, errors + + +def fetch_licenses(packages, api_url, api_key, verbose=False): + """ + Return a mapping of {license key: License object} given an `packages` list of + Package object and a list of Error. + """ + + errors = [] + + if have_network_connection(): + if not valid_api_url(api_url): + msg = "URL not reachable. Invalid '--api_url'. License retrieval is skipped." + errors.append(Error(ERROR, msg)) + else: + msg = 'Network problem. Please check your Internet connection. License retrieval is skipped.' + errors.append(Error(ERROR, msg)) + + msg = "Authorization denied. Invalid '--api_key'. License retrieval is skipped." + auth_error = Error(ERROR, msg) + + # collect unique license keys + license_keys = set() + licensing = Licensing() + for package in packages: + if not package.license_expression: + # TODO: we should have a check for this + continue + package_license_keys = licensing.license_keys( + package.license_expression, unique=True, simple=True) + license_keys.update(package_license_keys) + + licenses_by_key = {} + + # fetch license key proper + for license_key in sorted(license_keys): + # No need to go through fetching all the licensesif we detected invalid '--api_key' + if auth_error in errors: + break + license, errs = get_license_details(api_url, api_key, license_key) # NOQA + errors.extend(errs) + if license: + licenses_by_key[license_key] = license + + if verbose: + click.echo('Fetched license: {}'.format(license_key)) + + return licenses_by_key, util.unique(errors) + + +def valid_api_url(api_url): + try: + request = Request(api_url) + # This will always goes to exception as no key are provided. + # The purpose of this code is to validate the provided api_url is correct + urlopen(request) + except HTTPError as http_e: + # The 403 error code is refer to "Authentication credentials were not provided.". + # This is correct as no key are provided. + if http_e.code == 403: + return True + except: + # All other exceptions yield to invalid api_url + pass + return False + + +# FIXME: rename to is_online: BUT do we really need this at all???? +def have_network_connection(): + """ + Return True if an HTTP connection to some public web site is possible. + """ + import socket + if python2: + import httplib # NOQA + else: + import http.client as httplib # NOQA + + http_connection = httplib.HTTPConnection('enterprise.dejacode.com', timeout=10) # NOQA + try: + http_connection.connect() + except socket.error: + return False + else: + return True diff --git a/src/aboutcode/attrib.py b/src/aboutcode/attrib.py new file mode 100644 index 00000000..256c60c6 --- /dev/null +++ b/src/aboutcode/attrib.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +import datetime +import io +import os + +import jinja2 + +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode.licenses import COMMON_LICENSES +from aboutcode.attrib_util import get_template + + +# FIXME: the template dir should be outside the code tree +DEFAULT_TEMPLATE_FILE = os.path.join( + os.path.dirname(os.path.realpath(__file__)), 'templates', 'default_html.template') + + +def generate_attribution_doc( + packages, output_location, template_loc=DEFAULT_TEMPLATE_FILE, variables=None): + """ + Generate and save an attribution doc at `output_location` using an `packages` + list of Package objects, a `template_loc` template file location and a + `variables` optional dict of extra variables. + Return a list of Error objects if any. + """ + errors = [] + + with io.open(template_loc, encoding='utf-8') as inp: + template_text = inp.read() + + rendering_errors, rendered = create_attribution_text( + packages, template_text=template_text, variables=variables) + + errors.extend(rendering_errors) + + if rendered: + with io.open(output_location, 'w', encoding='utf-8') as of: + of.write(rendered) + + return errors + + +def create_attribution_text(packages, template_text, variables=None): + """ + Generate an attribution text from an `packages` list of Package objects, a + `template_text` template text and a `variables` optional dict of extra + variables. + + Return a list of errors and the attribution text (or None). + + TODO: document data available to the template and how to write custom templates. + """ + rendered = None + errors = [] + template = get_template(template_text) + + packages = sorted(packages) + + licenses_by_key = {} + for package in packages: + for license in package.licenses: # NOQA + licenses_by_key[license.key] = license + + # a sorted common licenses list in use for reporting + common_licenses_in_use = sorted( + lic for key, lic in licenses_by_key.items() if key in COMMON_LICENSES) + + try: + rendered = template.render( + # the current UTC time + utcnow=datetime.datetime.utcnow(), + # variables from CLI vartext option + variables=variables, + + # a list of all sorted packages objects + packages=packages, + + # sorted list of unique license objects + licenses=sorted(licenses_by_key.values()), + + # list of common licenses keys + common_licenses=COMMON_LICENSES, + # sorted list of common License object actually used across all packages + common_licenses_in_use=common_licenses_in_use, + + #################################################################### + # legacy data for backward compatibility + #################################################################### + # a list of all package objects: use packages instead + abouts=packages, + # prefer using variables + vartext_dict=variables, + ) + + except Exception as e: + import traceback + err = str(e) + '\n' + traceback.format_exc() + error = Error(CRITICAL, 'Template processing error: {}'.format(err)) + errors.append(error) + + return errors, rendered + + +def check_template(template_text): + """ + Check the syntax of a template. Return an error tuple (line number, + message) if the template is invalid or None if it is valid. + """ + try: + get_template(template_text) + except (jinja2.TemplateSyntaxError, jinja2.TemplateAssertionError) as e: + return e.lineno, e.message diff --git a/src/aboutcode/attrib_util.py b/src/aboutcode/attrib_util.py new file mode 100644 index 00000000..cf665905 --- /dev/null +++ b/src/aboutcode/attrib_util.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +from jinja2 import Environment +from jinja2.filters import environmentfilter +from jinja2.filters import make_attrgetter +from jinja2.filters import ignore_case +from jinja2.filters import FilterArgumentError + + +""" +Extra JINJA2 custom filters and other template utilities. +""" + + +def get_template(template_text): + """ + Return a template built from a text string. + Register custom templates as needed. + """ + env = Environment(autoescape=True) + # register our custom filters + env.filters.update(dict( + unique_together=unique_together, + multi_sort=multi_sort)) + return env.from_string(template_text) + + +@environmentfilter +def multi_sort(environment, value, reverse=False, case_sensitive=False, + attributes=None): + """ + Sort an iterable using an "attributes" list of attribute names available on + each iterable item. Sort ascending unless reverse is "true". Ignore the case + of strings unless "case_sensitive" is "true". + + .. sourcecode:: jinja + + {% for item in iterable|multi_sort(attributes=['date', 'name']) %} + ... + {% endfor %} + """ + if not attributes: + raise FilterArgumentError( + 'The multi_sort filter requires a list of attributes as argument, ' + 'such as in: ' + "for item in iterable|multi_sort(attributes=['date', 'name'])") + + # build a list of attribute getters, one for each attribute + do_ignore_case = ignore_case if not case_sensitive else None + attribute_getters = [] + for attribute in attributes: + ag = make_attrgetter(environment, attribute, postprocess=do_ignore_case) + attribute_getters.append(ag) + + # build a key function that has runs all attribute getters + def key(v): + return [a(v) for a in attribute_getters] + + return sorted(value, key=key, reverse=reverse) + + +@environmentfilter +def unique_together(environment, value, case_sensitive=False, attributes=None): + """ + Return a list of unique items from an iterable. Unicity is checked when + considering together all the values of an "attributes" list of attribute + names available on each iterable item.. The items order is preserved. Ignore + the case of strings unless "case_sensitive" is "true". + .. sourcecode:: jinja + + {% for item in iterable|unique_together(attributes=['date', 'name']) %} + ... + {% endfor %} + + """ + if not attributes: + raise FilterArgumentError( + 'The unique_together filter requires a list of attributes as argument, ' + 'such as in: ' + "{% for item in iterable|unique_together(attributes=['date', 'name']) %} ") + + # build a list of attribute getters, one for each attribute + do_ignore_case = ignore_case if not case_sensitive else None + attribute_getters = [] + for attribute in attributes: + ag = make_attrgetter(environment, attribute, postprocess=do_ignore_case) + attribute_getters.append(ag) + + # build a unique_key function that has runs all attribute getters + # and returns a hashable tuple + def unique_key(v): + return tuple(repr(a(v)) for a in attribute_getters) + + unique = [] + seen = set() + for item in value: + key = unique_key(item) + if key not in seen: + seen.add(key) + unique.append(item) + return unique diff --git a/src/attributecode/cmd.py b/src/aboutcode/cmd.py similarity index 53% rename from src/attributecode/cmd.py rename to src/aboutcode/cmd.py index 7af695b5..cdcee09d 100644 --- a/src/attributecode/cmd.py +++ b/src/aboutcode/cmd.py @@ -21,7 +21,6 @@ from collections import defaultdict from functools import partial import io -import logging import os import sys @@ -29,20 +28,22 @@ # silence unicode literals warnings click.disable_unicode_literals_warning = True -from attributecode import WARNING -from attributecode.util import unique +from aboutcode import __about_spec_version__ +from aboutcode import __version__ +from aboutcode import Error +from aboutcode import CRITICAL +from aboutcode import WARNING +from aboutcode import severities -from attributecode import __about_spec_version__ -from attributecode import __version__ -from attributecode import severities -from attributecode.attrib import check_template -from attributecode.attrib import DEFAULT_TEMPLATE_FILE -from attributecode.attrib import generate_and_save as generate_attribution_doc -from attributecode.gen import generate as generate_about_files -from attributecode.model import collect_inventory -from attributecode.model import write_output -from attributecode.util import extract_zip -from attributecode.util import filter_errors +from aboutcode.attrib import check_template +from aboutcode.attrib import DEFAULT_TEMPLATE_FILE +from aboutcode.attrib import generate_attribution_doc +from aboutcode.gen import generate_about_files +from aboutcode.inv import collect_inventory +from aboutcode.inv import save_as_json +from aboutcode.inv import save_as_csv +from aboutcode.util import extract_zip +from aboutcode.util import unique __copyright__ = """ @@ -115,8 +116,9 @@ def validate_key_values(ctx, param, value): kvals, errors = parse_key_values(value) if errors: + name = param.name ive = '\n'.join(sorted(' ' + x for x in errors)) - msg = ('Invalid {param} option(s):\n' + msg = ('Invalid {name} option(s):\n' '{ive}'.format(**locals())) raise click.UsageError(msg) return kvals @@ -132,12 +134,20 @@ def validate_extensions(ctx, param, value, extensions=tuple(('.csv', '.json',))) return value +def validate_api_url(ctx, param, value): + if value: + value = value.strip('/') + if not value.endswith('licenses'): + value = '/'.join([value, 'licenses']) + return value + + ###################################################################### # inventory subcommand ###################################################################### @about.command(cls=AboutCommand, - short_help='Collect the inventory of .ABOUT files to a CSV or JSON file.') + short_help='Collect an inventory of .ABOUT files in a CSV or JSON file.') @click.argument('location', required=True, @@ -157,9 +167,9 @@ def validate_extensions(ctx, param, value, extensions=tuple(('.csv', '.json',))) type=click.Choice(['json', 'csv']), help='Set OUTPUT inventory file format.') -@click.option('-q', '--quiet', +@click.option('-c', '--check-files', is_flag=True, - help='Do not print error or warning messages.') + help='Check that code and license files referenced in ABOUT files exist.') @click.option('--verbose', is_flag=True, @@ -167,7 +177,7 @@ def validate_extensions(ctx, param, value, extensions=tuple(('.csv', '.json',))) @click.help_option('-h', '--help') -def inventory(location, output, format, quiet, verbose): # NOQA +def inventory(location, output, format, check_files, verbose): # NOQA """ Collect the inventory of .ABOUT file data as CSV or JSON. @@ -175,21 +185,29 @@ def inventory(location, output, format, quiet, verbose): # NOQA OUTPUT: Path to the JSON or CSV inventory file to create. """ - if not quiet: - print_version() - click.echo('Collecting inventory from ABOUT files...') + print_version() + click.echo('Collecting inventory from ABOUT files...') # FIXME: do we really want to continue support zip as an input? + # accept zipped ABOUT files as input if location.lower().endswith('.zip'): - # accept zipped ABOUT files as input location = extract_zip(location) - errors, abouts = collect_inventory(location) - write_errors = write_output(abouts=abouts, location=output, format=format) + + errors, packages = collect_inventory(location, check_files=check_files) + + writers = { + 'json': save_as_json, + 'csv': save_as_csv, + } + writer = writers[format] + write_errors = writer(location=output, packages=packages) errors.extend(write_errors) - errors_count = report_errors(errors, quiet, verbose, log_file_loc=output + '-error.log') - if not quiet: - msg = 'Inventory collected in {output}.'.format(**locals()) - click.echo(msg) + log_file_loc = output + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + msg = 'Inventory collected in {output}' + if errors_count: + msg += ' with ERRORS\nSee log file: {log_file_loc}' + click.echo(msg.format(**locals())) sys.exit(errors_count) @@ -198,35 +216,29 @@ def inventory(location, output, format, quiet, verbose): # NOQA ###################################################################### @about.command(cls=AboutCommand, - short_help='Generate .ABOUT files from an inventory as CSV or JSON.') + short_help='Generate .ABOUT files from an CSV or JSON inventory.') @click.argument('location', required=True, metavar='LOCATION', type=click.Path( - exists=True, file_okay=True, dir_okay=True, readable=True, resolve_path=True)) + exists=True, file_okay=True, dir_okay=False, readable=True, resolve_path=True)) @click.argument('output', required=True, metavar='OUTPUT', type=click.Path(exists=True, file_okay=False, writable=True, resolve_path=True)) -# FIXME: the CLI UX should be improved with two separate options for API key and URL -@click.option('--fetch-license', - nargs=2, - type=str, - metavar='URL KEY', - help='Fetch license data and text files from a DejaCode License Library ' - 'API URL using the API KEY.') - @click.option('--reference', metavar='DIR', type=click.Path(exists=True, file_okay=False, readable=True, resolve_path=True), help='Path to a directory with reference license data and text files.') -@click.option('-q', '--quiet', +@click.option('--legacy-placement', is_flag=True, - help='Do not print error or warning messages.') + help='Use legacy .ABOUT file placement: when creating an ABOUT file to ' + 'document a directory, create the ABOUT file INSIDE the directory and not ' + 'side-by-side with the directory.') @click.option('--verbose', is_flag=True, @@ -234,7 +246,7 @@ def inventory(location, output, format, quiet, verbose): # NOQA @click.help_option('-h', '--help') -def gen(location, output, fetch_license, reference, quiet, verbose): +def gen(location, output, reference, legacy_placement, verbose): """ Generate .ABOUT files in OUTPUT from an inventory of .ABOUT files at LOCATION. @@ -242,25 +254,98 @@ def gen(location, output, fetch_license, reference, quiet, verbose): OUTPUT: Path to a directory where ABOUT files are generated. """ - if not quiet: - print_version() - click.echo('Generating .ABOUT files...') + print_version() + click.echo('Generating .ABOUT files...') if not location.endswith(('.csv', '.json',)): raise click.UsageError('ERROR: Invalid input file extension: must be one .csv or .json.') - errors, abouts = generate_about_files( - location=location, - base_dir=output, + errors, packages = generate_about_files( + inventory_location=location, + target_dir=output, reference_dir=reference, - fetch_license=fetch_license, - ) + legacy_placement=legacy_placement) - errors_count = report_errors(errors, quiet, verbose, log_file_loc=output + '-error.log') - if not quiet: - abouts_count = len(abouts) - msg = '{abouts_count} .ABOUT files generated in {output}.'.format(**locals()) - click.echo(msg) + log_file_loc = output + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + + packages_count = len(packages) + msg = '{packages_count} .ABOUT files generated in {output}' + if errors_count: + msg += ' with ERRORS\nSee log file: {log_file_loc}' + click.echo(msg.format(**locals())) + sys.exit(errors_count) + + +###################################################################### +# fetch-licenses subcommand +###################################################################### + +@about.command(cls=AboutCommand, + short_help='Fetch licenses from a remote DejaCode License Library API.', + name='fetch-licenses') + +@click.argument('location', + required=True, + metavar='LOCATION', + callback=partial(validate_extensions, extensions=('.csv',)), + type=click.Path( + exists=True, file_okay=True, dir_okay=False, readable=True, resolve_path=True)) + +@click.argument('output', + required=True, + metavar='OUTPUT', + type=click.Path(exists=True, dir_okay=True, file_okay=False, writable=True, resolve_path=True)) + +@click.option('--api-key', + metavar='API-KEY', + envvar='DEJACODE_API_KEY', + type=str, + help='DejaCode License Library API KEY.') + +@click.option('--api-url', + metavar='API-URL', + envvar='DEJACODE_API_URL', + callback=validate_api_url, + type=str, + help='DejaCode License Library API URL.') + +@click.option('--verbose', + is_flag=True, + help='Show all error and warning messages.') + +@click.help_option('-h', '--help') + +def fetch_licenses(location, output, api_key, api_url, verbose): # NOQA + """ +Load inventory from LOCATION then fetch license texts and data referenced in +this inventory license expressions from a remote DejaCode License Library API +and save the license texts and data in the OUTPUT directory. + +LOCATION: Path to a JSON or CSV inventory file. + +OUTPUT: Directory path where to save the fetched reference license data and texts. + """ + from aboutcode import api + from aboutcode import gen + + print_version() + click.echo('Fetching licenses...') + + errors, packages = gen.load_inventory(location) + + licenses_by_key, fetch_errors = api.fetch_licenses(packages, api_url, api_key, verbose) + errors.extend(fetch_errors) + + for license in licenses_by_key.values(): # NOQA + license.dump(output) + + log_file_loc = output + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + msg = 'Licenses saved to {output}' + if errors_count: + msg += ' with ERRORS\nSee log file: {log_file_loc}' + click.echo(msg.format(**locals())) sys.exit(errors_count) @@ -282,6 +367,16 @@ def validate_template(ctx, param, value): '{lineno}: "{message}"'.format(**locals())) return value +def display_template_help(ctx, param, value): + if not value or ctx.resilient_parsing: + return + base_dir = os.path.dirname(__file__) + with io.open(os.path.join(base_dir , 'template-help.txt'), encoding='utf-8') as th: + template_help = th.read() + + click.echo(template_help) + ctx.exit() + @about.command(cls=AboutCommand, short_help='Generate an attribution document from .ABOUT files.') @@ -297,22 +392,24 @@ def validate_template(ctx, param, value): metavar='OUTPUT', type=click.Path(exists=False, dir_okay=False, writable=True, resolve_path=True)) -@click.option('--template', +@click.option('-t', '--template', metavar='FILE', callback=validate_template, type=click.Path(exists=True, dir_okay=False, readable=True, resolve_path=True), help='Path to an optional custom attribution template to generate the ' 'attribution document. If not provided the default built-in template is used.') -@click.option('--vartext', +@click.option('-v', '--vartext', multiple=True, callback=validate_key_values, metavar='=', - help='Add variable text as key=value for use in a custom attribution template.') + help='Add variable text as key=value for use in a custom attribution template. ' + 'Can be used multiple times for multiple variable texts.') -@click.option('-q', '--quiet', - is_flag=True, - help='Do not print error or warning messages.') +@click.option('--help-template', + is_flag=True, is_eager=True, expose_value=False, + callback=display_template_help, + help='Show additional help to write custom attribution templates and exit.') @click.option('--verbose', is_flag=True, @@ -320,7 +417,7 @@ def validate_template(ctx, param, value): @click.help_option('-h', '--help') -def attrib(location, output, template, vartext, quiet, verbose): +def attrib(location, output, template, vartext, verbose): """ Generate an attribution document at OUTPUT using .ABOUT files at LOCATION. @@ -328,29 +425,45 @@ def attrib(location, output, template, vartext, quiet, verbose): OUTPUT: Path where to write the attribution document. """ - if not quiet: - print_version() - click.echo('Generating attribution...') - - # accept zipped ABOUT files as input - if location.lower().endswith('.zip'): - location = extract_zip(location) - - errors, abouts = collect_inventory(location) + print_version() + click.echo('Generating attribution...') - attrib_errors = generate_attribution_doc( - abouts=abouts, - output_location=output, - template_loc=template, - variables=vartext, - ) - errors.extend(attrib_errors) + errors = [] - errors_count = report_errors(errors, quiet, verbose, log_file_loc=output + '-error.log') + template_error = check_template(template) + if template_error: + lineno, message = template_error + errors.apend(Error( + CRITICAL, + 'Template validation error at line: {lineno}: "{message}"'.format(**locals()) + )) - if not quiet: - msg = 'Attribution generated in: {output}'.format(**locals()) - click.echo(msg) + else: + # FIXME: this is not a feature. Unzipping should be done by the users IMHO + # accept zipped ABOUT files as input + if location.lower().endswith('.zip'): + location = extract_zip(location) + + errors, packages = collect_inventory(location) + + # load all files + for package in packages: + package.load_files() + + attrib_errors = generate_attribution_doc( + packages=packages, + output_location=output, + template_loc=template, + variables=vartext, + ) + errors.extend(attrib_errors) + + log_file_loc = output + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + msg = 'Attribution generated in: {output}' + if errors_count: + msg += ' with ERRORS\nSee log file: {log_file_loc}' + click.echo(msg.format(**locals())) sys.exit(errors_count) @@ -361,8 +474,7 @@ def attrib(location, output, template, vartext, quiet, verbose): # FIXME: This is really only a dupe of the Inventory command @about.command(cls=AboutCommand, - short_help='Validate that the format of .ABOUT files is correct and report ' - 'errors and warnings.') + short_help='Validate the format of .ABOUT files.') @click.argument('location', required=True, @@ -370,13 +482,17 @@ def attrib(location, output, template, vartext, quiet, verbose): type=click.Path( exists=True, file_okay=True, dir_okay=True, readable=True, resolve_path=True)) +@click.option('-c', '--check-files', + is_flag=True, + help='Check that code and license files referenced in ABOUT files exist.') + @click.option('--verbose', is_flag=True, help='Show all error and warning messages.') @click.help_option('-h', '--help') -def check(location, verbose): +def check(location, check_files, verbose): """ Check .ABOUT file(s) at LOCATION for validity and print error messages. @@ -384,9 +500,9 @@ def check(location, verbose): """ print_version() click.echo('Checking ABOUT files...') - errors, _abouts = collect_inventory(location) - severe_errors_count = report_errors(errors, quiet=False, verbose=verbose) - sys.exit(severe_errors_count) + errors, _packages = collect_inventory(location, check_files=check_files) + errors_count = report_errors(errors, verbose) + sys.exit(errors_count) ###################################################################### @@ -396,13 +512,13 @@ def check(location, verbose): def print_config_help(ctx, param, value): if not value or ctx.resilient_parsing: return - from attributecode.transform import tranformer_config_help + from aboutcode.transform import tranformer_config_help click.echo(tranformer_config_help) ctx.exit() @about.command(cls=AboutCommand, - short_help='Transform a CSV by applying renamings, filters and checks.') + short_help='Transform a CSV by renaming and filtering columns.') @click.argument('location', required=True, @@ -427,43 +543,91 @@ def print_config_help(ctx, param, value): callback=print_config_help, help='Show configuration file format help and exit.') -@click.option('-q', '--quiet', - is_flag=True, - help='Do not print error or warning messages.') - @click.option('--verbose', is_flag=True, help='Show all error and warning messages.') @click.help_option('-h', '--help') -def transform(location, output, configuration, quiet, verbose): # NOQA +def transform(location, output, configuration, verbose): # NOQA """ -Transform the CSV file at LOCATION by applying renamings, filters and checks +Transform and validate the CSV file at LOCATION by renaming or deleting columns and write a new CSV to OUTPUT. LOCATION: Path to a CSV file. OUTPUT: Path to CSV inventory file to create. """ - from attributecode.transform import transform_csv_to_csv - from attributecode.transform import Transformer + from aboutcode.transform import transform_csv_to_csv + from aboutcode.transform import Transformer - if not quiet: - print_version() - click.echo('Transforming CSV...') + print_version() + click.echo('Transforming CSV...') if not configuration: - transformer = Transformer.default() + transformer = Transformer() else: transformer = Transformer.from_file(configuration) errors = transform_csv_to_csv(location, output, transformer) - errors_count = report_errors(errors, quiet, verbose) - if not quiet and not errors: - msg = 'Transformed CSV written to {output}.'.format(**locals()) - click.echo(msg) + log_file_loc = output + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + msg = 'Transformed CSV written to {output}' + if errors_count: + msg += ' with ERRORS\nSee log file: {log_file_loc}' + click.echo(msg.format(**locals())) + sys.exit(errors_count) + + +###################################################################### +# reformat subcommand +###################################################################### + +@about.command(cls=AboutCommand, + short_help='Reformat existing .ABOUT files in-place to the standard format and ordering.') + +@click.argument('location', + required=True, + metavar='LOCATION', + type=click.Path( + exists=True, file_okay=True, dir_okay=True, readable=True, resolve_path=True)) + +@click.option('--verbose', + is_flag=True, + help='Show all error and warning messages.') + +@click.help_option('-h', '--help') + +def reformat(location, verbose): # NOQA + """ + Reformat existing .ABOUT files in-place to the standard format and ordering. + Does nothing if there are errors. + +LOCATION: Path to an .ABOUT file or a directory with .ABOUT files. + """ + print_version() + click.echo('Collecting inventory from ABOUT files...') + + # FIXME: do we really want to continue support zip as an input? + # accept zipped ABOUT files as input + if location.lower().endswith('.zip'): + location = extract_zip(location) + + errors, packages = collect_inventory(location) + errors_count = 0 + if not errors: + click.echo('Saving reformatted ABOUT files...') + for package in packages: + package.dump(package.about_file_location) + click.echo('Saved {} reformatted ABOUT files.'.format(len(packages))) + + if errors: + click.echo('Errors found. Aborting.') + log_file_loc = location + '-error.log' + errors_count = report_errors(errors, verbose, log_file_loc=log_file_loc) + msg = 'See log file: {log_file_loc}' + click.echo(msg.format(**locals())) sys.exit(errors_count) @@ -471,31 +635,31 @@ def transform(location, output, configuration, quiet, verbose): # NOQA # Error management ###################################################################### -def report_errors(errors, quiet, verbose, log_file_loc=None): +def report_errors(errors, verbose, log_file_loc=None): """ - Report the `errors` list of Error objects to screen based on the `quiet` and - `verbose` flags. + Report the `errors` list of Error objects to screen based on the `verbose` + flag. If `log_file_loc` file location is provided also write a verbose log to this - file. - Return True if there were severe error reported. + file. Return True if there were severe error reported. """ errors = unique(errors) - messages, severe_errors_count = get_error_messages(errors, quiet, verbose) + messages, severe_errors_count = get_error_messages(errors, verbose) for msg in messages: click.echo(msg) if log_file_loc: - log_msgs, _ = get_error_messages(errors, quiet=False, verbose=True) - with io.open(log_file_loc, 'w', encoding='utf-8') as lf: - lf.write('\n'.join(log_msgs)) + log_msgs, _ = get_error_messages(errors, verbose=True) + if log_msgs: + with io.open(log_file_loc, 'w', encoding='utf-8') as lf: + lf.write('\n'.join(log_msgs)) return severe_errors_count -def get_error_messages(errors, quiet=False, verbose=False): +def get_error_messages(errors, verbose=False): """ Return a tuple of (list of error message strings to report, - severe_errors_count) given an `errors` list of Error objects and using the - `quiet` and `verbose` flags. + severe_errors_count) given an `errors` list of Error objects using the + `verbose` flags. """ errors = unique(errors) severe_errors = filter_errors(errors, WARNING) @@ -503,18 +667,25 @@ def get_error_messages(errors, quiet=False, verbose=False): messages = [] - if severe_errors and not quiet: + if severe_errors: error_msg = 'Command completed with {} errors or warnings.'.format(severe_errors_count) messages.append(error_msg) - for severity, message in errors: + for error in errors: + severity = error.severity + message = error.message + path = error.path sevcode = severities.get(severity) or 'UNKNOWN' - msg = '{sevcode}: {message}'.format(**locals()) - if not quiet: - if verbose: - messages .append(msg) - elif severity >= WARNING: - messages .append(msg) + + msg = '{sevcode}: ' + if path: + msg += 'in ABOUT file: "{path}": ' + msg += '{message}' + msg = msg.format(**locals()) + if verbose: + messages.append(msg) + elif severity >= WARNING: + messages .append(msg) return messages, severe_errors_count ###################################################################### @@ -524,33 +695,43 @@ def get_error_messages(errors, quiet=False, verbose=False): def parse_key_values(key_values): """ Given a list of "key=value" strings, return: - - a dict {key: [value, value, ...]} + - a dict {key: value} - a sorted list of unique error messages for invalid entries where there is - a missing a key or value. + a missing a key or value or duplicated key. """ if not key_values: return {}, [] errors = set() - parsed_key_values = defaultdict(list) + parsed_key_values = {} for key_value in key_values: key, _, value = key_value.partition('=') - key = key.strip().lower() + key = key.strip().strip('\'"').lower() if not key: errors.add('missing in "{key_value}".'.format(**locals())) continue + if key in parsed_key_values: + errors.add('duplicated already defined: "{key_value}".'.format(**locals())) + continue + value = value.strip() if not value: errors.add('missing in "{key_value}".'.format(**locals())) continue - values = parsed_key_values[key] - if value not in values: - parsed_key_values[key].append(value) + parsed_key_values[key] = value + + return parsed_key_values, sorted(errors) - return dict(parsed_key_values), sorted(errors) + +def filter_errors(errors, minimum_severity=WARNING): + """ + Return a list of unique `errors` Error object filtering errors that have a + severity below `minimum_severity`. + """ + return unique([e for e in errors if e.severity >= minimum_severity]) if __name__ == '__main__': diff --git a/src/aboutcode/gen.py b/src/aboutcode/gen.py new file mode 100644 index 00000000..a07dcd58 --- /dev/null +++ b/src/aboutcode/gen.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +from collections import OrderedDict +import io +import json +import posixpath + +from aboutcode import Error +from aboutcode import ERROR +from aboutcode import CRITICAL +from aboutcode import model +from aboutcode import util +from aboutcode.util import csv +from aboutcode.util import unique +from aboutcode.util import resource_name + + +def load_inventory(location, base_dir=None, legacy_placement=False): + """ + Load the inventory file at `location` as Package objects. + Use the `base_dir` to resolve the ABOUT file location. + Return a list of errors and a list of Package objects. + """ + packages = [] + + if location.endswith('.csv'): + inventory = load_csv(location) + elif location.endswith('.json'): + inventory = load_json(location) + else: + err = Error( + CRITICAL, + 'Unsupported inventory file type. Must be one of .csv or .json') + return [err], [] + + inventory = list(inventory) + + if not inventory: + err = Error(CRITICAL, 'Empty inventory.') + return [err], [] + + # various check prior to generation + + # validate field names using the first row fields as a sample + sample = dict(inventory[0]) + standard_fields, custom_fields = model.split_fields(sample) + fields_err = model.validate_field_names( + standard_fields.keys(), custom_fields.keys()) + if fields_err: + return fields_err, packages + + if base_dir: + base_dir = util.to_posix(base_dir) + + errors = [] + + for rn, entry in enumerate(inventory, 1): + entry = dict(entry) + + abr = entry.get('about_resource', '') + if not abr: + errors.append(Error(CRITICAL, 'Required field "about_resource" is missing in row: {}.'.format(rn))) + continue + + about_resource = util.to_posix(abr) + if abr != about_resource: + msg = ('Skipping invalid "about_resource". Path must be a POSIX path ' + 'using "/" (slash) as separator: "{}"'.format(abr)) + errors.append(Error(ERROR, msg)) + continue + file_name = resource_name(abr) + + abfp = entry.get('about_file_path', '') + if abfp and abfp != util.to_posix(abfp): + msg = ('Skipping invalid "about_file_path". Path must be a POSIX path ' + 'using "/" (slash) as separator: "{}"'.format(abr)) + errors.append(Error(ERROR, msg)) + continue + + if not abfp: + abr_is_dot = abr == '.' + if abr_is_dot: + msg = ('Skipping invalid "about_resource". Path cannot be a ' + 'single "." (period) without an "about_file_path"') + errors.append(Error(ERROR, msg)) + continue + + abfp = about_resource + + abfp = util.to_posix(abfp) + # Ensure there is no absolute directory path + about_file_path = abfp.strip('/') + + # Skip paths with lead and trailing spaces in directories or files segments + if has_spaces(about_file_path): + msg = ('Skipping invalid path to create an ABOUT file: a path segment ' + 'cannot start or end with a space: "{}"'.format(about_file_path)) + errors.append(Error(ERROR, msg)) + continue + + # Deal with legacy ABOUT file placement + if not legacy_placement: + # always use the file name as the about_resource to make this relative + # to the ABOUT file location + entry['about_resource'] = file_name + else: + # 1. we do not override the about_resource field to its filename and + # use instead the full "about_file_path" + # 2. if we have a directory, we use this placement for the ABOUT file: + # about_file_path/file_name : this will force the creation of the ABOUT + # file insude the documented directory + is_directory = abfp.endswith('/') + if is_directory: + about_file_path = posixpath.join(about_file_path, file_name) + + if not about_file_path.endswith('.ABOUT'): + about_file_path += '.ABOUT' + + entry['about_file_path'] = about_file_path + + if base_dir: + entry['about_file_location'] = posixpath.join(base_dir, about_file_path) + + try: + packages.append(model.Package.from_dict(entry)) + + except Exception as e: + if len(e.args) == 1 and isinstance(e.args[0], Error): + err = e.args[0] + msg = 'Cannot create .ABOUT file for: "{}":\n{}'.format( + about_file_path, err.message) + err = Error(CRITICAL, msg) + else: + import traceback + msg = 'Cannot create .ABOUT file for: "{}".\n{}\n{}'.format( + about_file_path, str(e) , traceback.format_exc()) + err = Error(CRITICAL, msg) + errors.append(err) + continue + + return unique(errors), packages + + +def has_spaces(path): + """ + Return True if any segments of the `path` string contains a leading or + trailing space. + """ + path = util.to_posix(path).strip('/') + return any(seg != seg.strip() for seg in path.split('/') if seg) + + +def load_csv(location): + """ + Read CSV at `location` and yield an ordered mapping for each row. + """ + with io.open(location, encoding='utf-8') as csvfile: + for row in csv.DictReader(csvfile): + yield row + + +def load_json(location): + """ + Read JSON file at `location` and return a list of ordered dicts, one for + each entry. + """ + # FIXME: IMHO we should know where the JSON is from and its shape + # FIXME use: object_pairs_hook=OrderedDict + with io.open(location, 'rb') as json_file: + results = json.load(json_file, object_pairs_hook=OrderedDict) + + # If the loaded JSON is not a list, + # - JSON output from AboutCode Manager: + # look for the "components" field as it is the field + # that contain everything the tool needs and ignore other fields. + # For instance, + # { + # "aboutcode_manager_notice":"xyz", + # "aboutcode_manager_version":"xxx", + # "components": + # [{ + # "license_expression":"apache-2.0", + # "copyright":"Copyright (c) 2017 nexB Inc.", + # "path":"ScanCode", + # ... + # }] + # } + # + # - JSON output from ScanCode: + # look for the "files" field as it is the field + # that contain everything the tool needs and ignore other fields: + # For instance, + # { + # "scancode_notice":"xyz", + # "scancode_version":"xxx", + # "files": + # [{ + # "path": "test", + # "type": "directory", + # "name": "test", + # ... + # }] + # } + # + # - JSON file that is not produced by scancode or aboutcode toolkit + # For instance, + # { + # "path": "test", + # "type": "directory", + # "name": "test", + # ... + # } + # FIXME: this is too clever and complex... IMHO we should not try to guess the format. + # instead a command line option should be provided explictly to say what is the format + if isinstance(results, list): + results = sorted(results) + else: + if u'aboutcode_manager_notice' in results: + results = results['components'] + elif u'scancode_notice' in results: + results = results['files'] + else: + results = [results] + return results + + +def generate_about_files(inventory_location, target_dir, reference_dir=None, + legacy_placement=False): + """ + Load ABOUT data from a CSV or JSON inventory at `inventory_location`. + Write .ABOUT files in the `target_dir` directory. + + If `reference_dir` is provided reuse and copy license and notice files + referenced in the inventory. + + Return a list errors and a list of Package objects. + """ + errors, packages = load_inventory( + inventory_location, base_dir=target_dir, legacy_placement=legacy_placement) + notices_by_filename = {} + licenses_by_key = {} + + if reference_dir: + notices_by_filename, licenses_by_key = model.get_reference_licenses(reference_dir) + + # TODO: validate inventory!!!!! to catch error before creating ABOUT files + + # update all licenses and notices + for package in packages: + # Fix the location to ensure this is a proper .ABOUT file + afl = package.about_file_location + if not afl: + pass + + if not afl.endswith('.ABOUT'): + afl = afl.rstrip('\\/').strip() + '.ABOUT' + + package.about_file_location = afl + + # used as a "prettier" display of .ABOUT file path + about_path = afl.replace(target_dir, '').strip('/') + + # Update the License objects of this Package using a mapping of reference licenses as {key: License} + for license in package.licenses: # NOQA + ref_lic = licenses_by_key.get(license.key) + if not ref_lic: + msg = ( + 'Cannot generate valid .ABOUT file for: "{}". ' + 'Reference license is missing: {}'.format(about_path, license.key)) + errors.append(Error(ERROR, msg)) + continue + + license.update(ref_lic) + + if package.notice_file: + notice_text = notices_by_filename.get(package.notice_file) + if not notice_text: + msg = ( + 'Cannot generate valid .ABOUT file for: "{}". ' + 'Empty or missing notice_file: {}'.format(about_path, package.notice_file)) + errors.append(Error(ERROR, msg)) + else: + package.notice_text = notice_text + + # create the files proper + package.dump(location=afl, with_files=True) + + return unique(errors), packages diff --git a/src/aboutcode/inv.py b/src/aboutcode/inv.py new file mode 100644 index 00000000..714ca64e --- /dev/null +++ b/src/aboutcode/inv.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- +# ============================================================================ +# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +import io +import json +import os +# FIXME: why posixpath??? +import posixpath +import traceback + +import attr + +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import util +from aboutcode.model import Package +from aboutcode.util import csv +from aboutcode.util import normalize +from aboutcode.util import python2 +from aboutcode.util import to_posix +from aboutcode.util import unique +from aboutcode.util import get_relative_path +from aboutcode.util import resource_name + + +""" +Collect and validate inventories of ABOUT files +""" + + +def collect_inventory(location, check_files=False): + """ + Collect any ABOUT files in the directory tree at `location` and return a + list of errors and a list of Package objects. + + If `check_files` is True, also check that files referenced in an ABOUT file + exist (about_resource, license and notice files, etc.) + """ + errors = [] + input_location = normalize(location) + about_locations = [] + try: + about_locations.extend(get_about_locations(input_location)) + except Exception as e: + errors.append(Error(CRITICAL, str(e) + '\n' + traceback.format_exc())) + + name_errors = util.check_file_names(about_locations) + errors.extend(name_errors) + + packages = [] + + if errors: + return sorted(unique(errors)), packages + + + is_file = os.path.isfile(input_location) + + for about_file_loc in about_locations : + package = None + try: + if is_file: + about_file_path = resource_name(input_location) + else: + about_file_path = get_relative_path(input_location, about_file_loc) + + package = Package.load(about_file_loc) + package.about_file_path = about_file_path + packages.append(package) + + if check_files: + package.check_files() + + # this could be a dict keys by path to keep per-path things? + errors.extend(package.errors) + + except Exception as exce: + if all(isinstance(e, Error) for e in exce.args): + for err in exce.args: + if not err.path: + err.path = about_file_path + errors.append(err) + else: + errors.append(Error( + CRITICAL, + str(exce) + '\n' + traceback.format_exc(), + path=about_file_path + )) + + if package: + # Insert path reference in every Package error + for err in package.errors: + if not err.path: + err.path = about_file_path + + return sorted(unique(errors)), packages + + +def is_about_file(path): + """ + Return True if the path represents a valid ABOUT file name. + """ + if path: + path = path.lower() + return path.endswith('.about') and path != '.about' + + +def get_locations(location): + """ + Yield posix locations of files given the `location` of a + a file or a directory tree containing ABOUT files. + File locations are normalized using posix path separators. + """ + assert os.path.exists(location) + location = normalize(location) + location = to_posix(location) + + if os.path.isfile(location): + yield location + else: + for name in os.listdir(location): + path = posixpath.join(location , name) + for f in get_locations(path): + yield f + + +def get_about_locations(location): + """ + Return a list of locations of ABOUT files given the `location` of a + a file or a directory tree containing ABOUT files. + File locations are normalized using posix path separators. + """ + for loc in get_locations(location): + if is_about_file(loc): + yield loc + + +def get_field_names(packages): + """ + Given a list of Package objects, return a list of any field names that exist + in any object, including custom fields. + """ + standard_seen = set() + custom_seen = set() + for a in packages: + standard, custom = a.fields() + standard_seen.update(standard) + custom_seen.update(custom) + + # resort standard fields in standard order + # which is a tad complex as this is a predefined order + standard_names = list(attr.fields_dict(Package).keys()) + standard = [] + for name in standard_names: + if name in standard_seen: + standard.append(name) + + return standard + sorted(custom_seen) + + +def save_as_json(location, packages): + """ + Write a JSON file at `location` given a list of Package objects. + Return a list of Error objects. + """ + + serialized = [a.to_dict(with_path=True, with_licenses=True) for a in packages] + + if python2: + with io.open(location, 'wb') as out: + out.write(json.dumps(serialized, indent=2)) + else: + with io.open(location, 'w', encoding='utf-8') as out: + out.write(json.dumps(serialized, indent=2)) + + return [] + + +def save_as_csv(location, packages): + """ + Write a CSV file at `location` given a list of Package objects. + Return a list of Error objects. + LEGACY: the licenses list of objects CANNOT be serialized to CSV + """ + serialized = [a.to_dict(with_path=True, with_licenses=False) for a in packages] + + field_names = get_field_names(packages) + + errors = [] + + with io.open(location, mode='w', encoding='utf-8') as output_file: + writer = csv.DictWriter(output_file, field_names) + writer.writeheader() + for row in serialized: + # FIXME: we should just crash instead IMHO + # See https://github.com/dejacode/about-code-tool/issues/167 + try: + writer.writerow(row) + except Exception as e: + msg = 'Generation skipped for {}: '.format(row) + str(e) + errors.append(Error(CRITICAL, msg)) + return errors diff --git a/src/attributecode/licenses.py b/src/aboutcode/licenses.py similarity index 94% rename from src/attributecode/licenses.py rename to src/aboutcode/licenses.py index 300fc8b6..356afe57 100644 --- a/src/attributecode/licenses.py +++ b/src/aboutcode/licenses.py @@ -19,8 +19,9 @@ from __future__ import unicode_literals # Common license keys -COMMON_LICENSES = ( +COMMON_LICENSES = set([ 'aes-128-3.0', + 'agpl-3.0', 'agpl-3.0-plus', 'apache-1.1', 'apache-2.0', @@ -44,7 +45,10 @@ 'cc-by-2.5', 'cc-by-sa-3.0', 'curl', + 'epl-1.0', + 'epl-2.0', 'freetype', + 'gpl-1.0', 'gpl-1.0-plus', 'gpl-2.0', 'gpl-2.0-bison', @@ -74,7 +78,9 @@ 'net-snmp', 'npl-1.1', 'ntpl', + 'openssl', 'openssl-ssleay', + 'python', 'ssleay-windows', 'rsa-md4', 'rsa-md5', @@ -86,4 +92,4 @@ 'uoi-ncsa', 'x11', 'zlib', -) +]) diff --git a/src/aboutcode/model.py b/src/aboutcode/model.py new file mode 100644 index 00000000..1286bbfb --- /dev/null +++ b/src/aboutcode/model.py @@ -0,0 +1,766 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- +# ============================================================================ +# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +""" +AboutCode toolkit is a tool to process ABOUT files. ABOUT files are +small text files that document the provenance (aka. the origin and +license) of software components as well as the essential obligation +such as attribution/credits and source code redistribution. See the +ABOUT spec at http://dejacode.org. + +AboutCode toolkit reads and validates ABOUT files and collect software +components inventories. +""" + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +from collections import OrderedDict +from functools import partial +import io +import os +import re + +import attr +import click +from license_expression import Licensing + +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import saneyaml +from aboutcode import util + +if util.python2: + str = unicode # NOQA + + +################################################################################ +# Validation and conversion utilities +################################################################################ + +def validate_custom_fields(about_obj, attribute, value): + """ + Check a mapping of custom_fields. Raise an Exception on errors. + """ + if not value: + return + + errors = [] + + if value and not isinstance(value, dict): + msg = ( + 'Custom fields must be a dictionary: %(value)r.') + raise Exception(Error(CRITICAL, msg % locals())) + + errors.extend(validate_custom_field_names(field_names=value.keys())) + + if errors: + raise Exception(*errors) + + if value and not isinstance(value, dict): + msg = ( + 'Custom fields must be a dictionary: %(value)r.') + raise Exception(Error(CRITICAL, msg % locals())) + + errors.extend(validate_custom_field_names(field_names=value.keys())) + + if errors: + raise Exception(*errors) + + +def convert_custom_fields(value): + if value: + value = {key: string_cleaner(value) for key, value in value.items()} + return value + + +def validate_custom_field_names(field_names): + """ + Check a list of custom field name and return a list of Error. + """ + if not field_names: + return [] + + errors = [] + is_valid_name = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*$').match + + for name in sorted(field_names): + # Check if names aresafe to use as an attribute name. + if not is_valid_name(name): + msg = ( + 'Custom field name: %(name)r contains illegal characters. ' + 'Only these characters are allowed: ' + 'ASCII letters, digits and "_" underscore. ' + 'The first character must be a letter.') + errors.append(Error(CRITICAL, msg % locals())) + if not name.lower() == name: + msg = 'Custom field name: %(name)r must be lowercase.' + errors.append(Error(CRITICAL, msg % locals())) + + return errors + + +booleans = { + 'yes': True, 'y': True, 'true': True, 'x': True, + 'no': False, 'n': False, 'false': False, } + + +def boolean_converter(value): + """ + Convert a yes/no value to a proper True/False boolean value. + """ + if value is True or value is False: + return value + + if isinstance(value, str): + value = value.lower().strip() + if not value: + value = False + elif value in booleans: + value = booleans[value] + return value + + +def validate_flag_field(about_obj, attribute, value): + """ + Check a boolean flag value for errors. Raise an Exception on errors. + """ + if value is True or value is False: + return + + name = attribute.name + msg = ( + 'Field name: %(name)r has an invalid flag value: ' + '%(value)r: should be one of yes or no or true or false.') + raise Exception(Error(CRITICAL, msg % locals())) + + +def copyright_converter(value): + value = string_cleaner(value) + if value: + value = '\n'.join(v.strip() for v in value.splitlines(False)).strip() + return value + + +def path_converter(value): + value = string_cleaner(value) + if value and isinstance(value, str): + value = util.to_posix(value).strip().strip('/') + return value + + +def about_resource_validator(about_obj, attribute, value): + if value and not isinstance(value, str): + msg = 'Required field "about_resource" must be a single string.' + raise Exception(Error(CRITICAL, msg)) + + if not value or not value.strip(): + msg = 'Required field "about_resource" is empty.' + raise Exception(Error(CRITICAL, msg)) + + +def license_expression_converter(value): + """ + Validate and normalize the license expression. + """ + value = string_cleaner(value) + if value: + licensing = Licensing() + expression = licensing.parse(value, simple=True) + value = str(expression) + return value + + +def string_cleaner(value): + if value and isinstance(value, str): + value = value.strip() + return value + + +def split_fields(data, skip_empty=False): + """ + Given a `data` mapping return two mappings: one with only standard fields + and one with custom fields. + """ + standard_fields = {} + custom_fields = {} + + standard_field_names = set(attr.fields_dict(Package).keys()) + + for key, value in data.items(): + if skip_empty and not value: + continue + if key in standard_field_names: + standard_fields[key] = value + else: + custom_fields[key] = value + return standard_fields, custom_fields + + +def validate_unique_names(field_names): + """ + Given a list of field names, validate their unicity and case. + Return a list of Error. + """ + errors = [] + keys = set(field_names) + keys_lower = set([k.lower() for k in keys]) + if len(keys) != len(keys_lower): + errors.append(Error(CRITICAL, 'Invalid fields: lowercased field names must be unique.')) + + if keys != keys_lower: + errors.append(Error(CRITICAL, 'Invalid fields: all field names must be lowercase.')) + + empty = False + for name in field_names: + if not name: + empty = True + break + if empty: + errors.append(Error(CRITICAL, 'Invalid empty field name.')) + return errors + + +def validate_field_names(standard_field_names, custom_field_names): + """ + Validate a `field_names` sequence of field names. Return a list of Error. + """ + errors = [] + standard_field_names = list(standard_field_names) + + if 'about_resource' not in standard_field_names: + errors.append(Error(CRITICAL, 'Required field "about_resource" is missing.')) + + custom_field_names = list(custom_field_names) + + uni_errors = validate_unique_names(standard_field_names + custom_field_names) + errors.extend(uni_errors) + + cf_errors = validate_custom_field_names(custom_field_names) + errors.extend(cf_errors) + + return errors + + +################################################################################ +# Models proper +################################################################################ + + +# shorthand for attributes +optional_attrib = partial(attr.attrib, default=None) +non_repr_attrib = partial(optional_attrib, repr=False) + +string_attrib = partial(non_repr_attrib, type=str, converter=string_cleaner) +path_attrib = partial(non_repr_attrib, type=str, converter=path_converter) +bool_attrib = partial(non_repr_attrib, default=False, type=bool, + validator=validate_flag_field, converter=boolean_converter) + + +@attr.attributes +class License(object): + """ + A license object + """ + # POSIX path relative to the ABOUT file location where the text file lives + key = string_attrib(repr=True) + name = string_attrib() + file = path_attrib(default=None) + url = string_attrib() + text = string_attrib(cmp=False) + + def __attrs_post_init__(self, *args, **kwargs): + if not self.file: + self.file = self.default_file_name + + @property + def default_file_name(self): + return self.key + '.LICENSE' + + def to_dict(self): + """ + Return an OrderedDict of license data (excluding texts). + Fields with empty values are not included. + """ + excluded = set(['text', ]) + + def valid_fields(attr, value): + return (value and attr.name not in excluded) + + return attr.asdict(self, filter=valid_fields, dict_factory=OrderedDict) + + def update(self, other_license): + """ + Update self "unset" fields with data from another License. + """ + assert isinstance(other_license, License) + assert other_license.key == self.key + self.name = self.name or other_license.name + self.url = self.url or other_license.url + self.file = self.file or other_license.file + self.text = self.text or other_license.text + + @classmethod + def load(cls, location): + """ + Return a License object built from the YAML file at `location`. + """ + with io.open(location, encoding='utf-8') as inp: + data = saneyaml.load(inp.read(), allow_duplicate_keys=False) + return cls.from_dict(data) + + @classmethod + def from_dict(cls, data): + """ + Return a License object built a `data` mapping. + """ + return License( + key=data['key'], + name=data.get('name'), + file=data.get('file'), + url=data.get('url')) + + def file_loc(self, base_dir): + fn = self.file or self.default_file_name + return os.path.join(base_dir, util.to_native(fn)) + + def load_text(self, base_dir): + """ + Load the license text found in `base_dir`. + """ + file_loc = self.file_loc(base_dir) + + # text can be garbage and not valid UTF + with io.open(file_loc, 'rb') as inp: + text = inp.read() + self.text = text.decode(encoding='utf-8', errors='replace') + + def dump(self, target_dir): + """ + Write this license as a .yml data file and a .LICENSE text file in + `target_dir`. + """ + data_loc = os.path.join(target_dir, self.key + '.yml') + with io.open(data_loc, 'w', encoding='utf-8') as out: + out.write(saneyaml.dump(self.to_dict())) + + # always write a text file even if this is an empty one + text = self.text or '' + if not text: + click.echo('WARNING: license text is empty for {}'.format(self.key)) + + file_loc = self.file_loc(target_dir) + with io.open(file_loc, 'w', encoding='utf-8') as out: + out.write(text) + + +def get_reference_licenses(reference_dir): + """ + Return reference licenses text and data loaded from `reference_dir`as a + tuple of two mappings: a mapping of notices as {notice_file: notice text} + and a mapping of {license key: License} loaded from a `reference_dir`. + + In the `reference_dir` there can be pairs of text and data files for a license key: + - a license text file must be named after its license key as `key.LICENSE` + - a license .yml YAML data file with license data to load as a License object. + All other files not part of a license files pair are treated as "notice files". + + For instance, we can have the files foo.LICENSE and foo.yml where foo.yml contains: + + key: foo + name: The Foo License + url: http://zddfsdfsd.com/FOO + """ + + notices_by_name = {} + licenses_by_key = {} + ref_files = os.listdir(reference_dir) + data_files = [f for f in ref_files if f.endswith('.yml')] + text_files = set([f for f in ref_files if not f.endswith('.yml')]) + + for data_file in data_files: + loc = os.path.join(reference_dir, data_file) + lic = License.load(loc) + licenses_by_key[lic.key] = lic + + if lic.file not in text_files: + click.echo( + 'ERROR: The reference license: {} does not have a ' + 'corresponding text file: {}'.format(lic.key, lic.file)) + else: + lic.load_text(reference_dir) + text_files.remove(lic.file) + + assert lic.text is not None, 'Incorrect reference license with no text: {}'.format(lic.key) + + # whatever is left are "notice" files + for notice_file in text_files: + loc = os.path.join(reference_dir, notice_file) + # text can be garbage and not valid UTF + with io.open(loc, 'rb') as inp: + text = inp.read() + text = text.decode(encoding='utf-8', errors='replace') + notices_by_name[notice_file] = text + + return notices_by_name, licenses_by_key + + +@attr.attributes(slots=True) +class Package(object): + """ + A package object + """ + + # the absolute location where the ABOUT file is stored + about_file_location = string_attrib(cmp=False) + + # a relative posix path where the ABOUT file is stored + about_file_path = string_attrib(cmp=False) + + # this is a path relative to the ABOUT file location + # everything else is optional + about_resource = path_attrib(repr=True, validator=about_resource_validator) + + name = string_attrib(repr=True) + version = string_attrib(repr=True) + description = string_attrib() + homepage_url = string_attrib() + download_url = string_attrib() + notes = string_attrib() + + copyright = string_attrib(converter=copyright_converter) + license_expression = string_attrib(converter=license_expression_converter) + + # boolean flags as yes/no + attribute = bool_attrib() + redistribute = bool_attrib() + modified = bool_attrib() + track_changes = bool_attrib() + internal_use_only = bool_attrib() + + # a list of License objects + licenses = non_repr_attrib(default=attr.Factory(list)) + + # path relative to the ABOUT file location + notice_file = path_attrib() + # the text loaded from notice_file + notice_text = string_attrib() + notice_url = string_attrib() + + # path relative to the ABOUT file location + changelog_file = path_attrib() + + owner = string_attrib() + owner_url = string_attrib() + + # SPDX-like VCS URL + vcs_url = string_attrib() + + md5 = string_attrib() + sha1 = string_attrib() + sha256 = string_attrib() + sha512 = string_attrib() + + spec_version = string_attrib() + + # custom files as name: value + custom_fields = non_repr_attrib( + default=attr.Factory(dict), validator=validate_custom_fields, converter=convert_custom_fields) + + # list of Error object + errors = non_repr_attrib(default=attr.Factory(list), cmp=False) + + def __attrs_post_init__(self, *args, **kwargs): + # populate licenses from expression + if self.license_expression and not self.licenses: + keys = Licensing().license_keys( + self.license_expression, unique=True, simple=True) + licenses = [License(key=key) for key in keys] + self.licenses = licenses + + def __getattr__(self, name): + """ + Make custom fields available as direct instance attributes. + """ + try: + return self.custom_fields[name] + except ValueError: + raise AttributeError(self.__class__.__name__+'{} is invalid.'.format(name)) + + @property + def base_dir(self): + return os.path.dirname(self.about_file_location) + + @classmethod + def from_dict(cls, data): + """ + Return a Package object built a `data` mapping. + """ + data = dict(data) + standard_fields, custom_fields = split_fields(data, skip_empty=True) + standard_fields.pop('errors', None) + + errors = validate_field_names(standard_fields.keys(), custom_fields.keys()) + if errors: + raise Exception(*errors) + + licenses = standard_fields.pop('licenses', []) or [] + licenses = [License.from_dict(l) for l in licenses] + return Package(licenses=licenses, custom_fields=custom_fields, **standard_fields) + + @classmethod + def load(cls, about_file_location): + """ + Return a Package object built from the YAML file at + `about_file_location` or None. Raise Exception on errors. + """ + # TODO: expand/resolve/abs/etc + about_file_location = util.to_posix(about_file_location) + + with io.open(about_file_location, encoding='utf-8') as inp: + text = inp.read() + + data = saneyaml.load(text, allow_duplicate_keys=False) + if not isinstance(data, dict): + raise Exception('Invalid ABOUT file: should be name/value pairs: {}'.format(about_file_location)) + data['about_file_location'] = about_file_location + return cls.from_dict(data) + + @classmethod + def loads(cls, text): + """ + Return a Package object built from a YAML `text` or None. + Raise Exception on errors. + """ + data = saneyaml.load(text, allow_duplicate_keys=False) + return cls.from_dict(data) + + # these fields are excluded from a to_dict() serialization + _excluded_fields = set([ + 'about_file_location', + 'errors', + 'custom_fields', + 'notice_text', + # this is for the licenses.text attribute + 'text', + ]) + + def to_dict(self, with_licenses=True, with_path=False, excluded_fields=_excluded_fields): + """ + Return an OrderedDict of Package data (excluding texts and ABOUT file path). + Fields with empty values are not included. + """ + excluded_fields = set(excluded_fields) + if not with_licenses: + excluded_fields.add('licenses') + if not with_path: + excluded_fields.add('about_file_path') + + def valid_fields(attr, value): + return (value and attr.name not in excluded_fields) + + data = attr.asdict(self, + recurse=True, filter=valid_fields, dict_factory=OrderedDict) + + # add custom fields + # note: we sort these fields by name + for key, value in sorted(self.custom_fields.items()): + if value: + data[key] = value + + return data + + def hashable(self): + """ + Return a hashable data representing this object and that is usable for + comparison and unicity checks. The about_resource filed is ignored and + not included. All texts are included if present. + """ + excluded_fields = set([ + 'about_resource', 'errors', + 'about_file_location', 'about_file_path', + ]) + return repr(tuple(self.to_dict(excluded_fields=excluded_fields).items())) + + def dumps(self): + """ + Return a YAML representation for this Package. + If `with_files` is True, also write any reference notice or license file. + """ + return saneyaml.dump(self.to_dict(), indent=2) + + def dump(self, location, with_files=False): + """ + Write this Package object to the .ABOUT file at `location`. + If `with_files` is True, also write any referenced notice or license file. + """ + location = util.to_native(location) + base_dir = os.path.dirname(location) + if not os.path.exists(base_dir): + os.makedirs(base_dir) + + with io.open(location, 'w', encoding='utf-8') as out: + out.write(self.dumps()) + + if with_files: + self.write_files(base_dir) + + @classmethod + def standard_fields(cls): + """ + Return a list of standard field names available in this class. + """ + return [f for f in attr.fields_dict(cls).keys() + if f not in cls._excluded_fields] + + def fields(self): + """ + Return a list of standard field names and a list of custom field names + in use (with a value set) in this object. + """ + + def valid_fields(attribute, value): + return (value and attribute.name not in self._excluded_fields) + + standard = attr.asdict( + self, recurse=False, filter=valid_fields, dict_factory=OrderedDict) + standard = list(standard.keys()) + + custom = [key for key, value in self.custom_fields.items() if value] + + return standard, custom + + def field_names(self): + """ + Return a list of all field names in use in this object. + """ + standard = list(attr.fields_dict(self.__class__).keys()) + custom = [k for k, v in self.custom_fields.items() if v] + return standard + custom + + def write_files(self, base_dir=None): + """ + Write all referenced license and notice files. + """ + base_dir = base_dir or self.base_dir + + def _write(text, target_loc): + if not target_loc: + return + + text = text or '' + parent = os.path.dirname(target_loc) + if not os.path.exists(parent): + os.makedirs(parent) + + with io.open(target_loc, 'w', encoding='utf-8') as out: + out.write(text) + + _write(self.notice_text, self.notice_file_loc(base_dir)) + + for license in self.licenses: # NOQA + _write(license.text, license.file_loc(base_dir)) + + def about_resource_loc(self, base_dir=None): + """ + Return the location to the about_resource. + """ + base_dir = base_dir or self.base_dir + return self.about_resource and os.path.join(base_dir, self.about_resource) + + def notice_file_loc(self, base_dir=None): + """ + Return the location to the notice_file or None. + """ + base_dir = base_dir or self.base_dir + return self.notice_file and os.path.join(base_dir, util.to_native(self.notice_file)) + + def changelog_file_loc(self, base_dir=None): + """ + Return the location to the changelog_file or None. + """ + base_dir = base_dir or self.base_dir + return self.changelog_file and os.path.join(base_dir, util.to_native(self.changelog_file)) + + def check_files(self, base_dir=None): + """ + Check that referenced files exist. Update and return self.errors. + """ + if self.about_file_location and not os.path.exists(self.about_file_location): + msg = 'ABOUT file: {} does not exists.'.format(self.about_file_location) + self.errors.append(Error(CRITICAL, msg)) + + base_dir = base_dir or self.base_dir + + if not os.path.exists(base_dir): + msg = 'Base directory: {} does not exists: unable to check files existence.'.format(base_dir) + self.errors.append(Error(CRITICAL, msg)) + return + + about_resource_loc = self.about_resource_loc(base_dir) + if about_resource_loc and not os.path.exists(about_resource_loc): + msg = 'File about_resource: "{}" does not exists'.format(self.about_resource) + self.errors.append(Error(CRITICAL, msg)) + + notice_file_loc = self.notice_file_loc(base_dir) + if notice_file_loc and not os.path.exists(notice_file_loc): + msg = 'File notice_file: "{}" does not exists'.format(self.notice_file) + self.errors.append(Error(CRITICAL, msg)) + + changelog_file_loc = self.changelog_file_loc(base_dir) + if changelog_file_loc and not os.path.exists(changelog_file_loc): + msg = 'File changelog_file: "{}" does not exists'.format(self.changelog_file) + self.errors.append(Error(CRITICAL, msg)) + + for license in self.licenses: # NOQA + license_file_loc = license.file_loc(base_dir) + if not os.path.exists(license_file_loc): + msg = 'License file: "{}" does not exists'.format(license.file) + self.errors.append(Error(CRITICAL, msg)) + + return self.errors + + def load_files(self, base_dir=None): + """ + Load all referenced license and notice texts. Return a list of errors. + """ + base_dir = base_dir or self.base_dir + errors = [] + + def _load_text(loc): + if loc: + text = None + try: + # text can be garbage and not valid UTF + with io.open(loc, 'rb') as inp: + text = inp.read() + text = text.decode(encoding='utf-8', errors='replace') + + except Exception as e: + msg = 'Unable to read text file: {}\n'.format(loc) + str(e) + errors.append(Error(CRITICAL, msg)) + return text + + text = _load_text(self.notice_file_loc(base_dir)) + if text: + self.notice_text = text + + for license in self.licenses: # NOQA + text = _load_text(license.file_loc(base_dir)) + if text: + license.text = text + + return errors diff --git a/src/aboutcode/template-help.txt b/src/aboutcode/template-help.txt new file mode 100644 index 00000000..18d00d69 --- /dev/null +++ b/src/aboutcode/template-help.txt @@ -0,0 +1 @@ +TODO: write me! \ No newline at end of file diff --git a/src/aboutcode/templates/default_html.template b/src/aboutcode/templates/default_html.template new file mode 100644 index 00000000..5206cb08 --- /dev/null +++ b/src/aboutcode/templates/default_html.template @@ -0,0 +1,61 @@ + + + + + + Open Source Software Information + + + +

OPEN SOURCE SOFTWARE INFORMATION

+
+

Licenses, acknowledgments and required copyright notices for open source packages:

+
+ +
+ {% for package in packages -%} +

{{ package.name }} {{ package.version or '' }}

+ {% endfor %} +
+
+ + + {% for package in packages -%} +
+

{{ package.name }} {{ package.version or '' }}

+ {% if package.license_expression -%} +

This package is licensed under: {{ package.license_expression }}

+ {%- endif %} + {% if package.copyright -%} + 
{{ package.copyright}}
+ {%- endif %} + {% if package.notice_text -%} + 
{{ package.notice_text}}
+ {%- endif %}{% for license in package.licenses -%} + {% if license.key in common_licenses -%} +

Full text of {{ license.key }} - {{ license.name }} is available at the end of this document.

+ {% else -%} +

{{ license.key }} - {{ license.name or ''}}

+ 

{{ license.text or ''}}

+ {%- endif %} + {%- endfor %} +
+ {% endfor %} + +
+ + +

Common Licenses in Use:

+ {% for license in common_licenses_in_use -%} +

{{ license.key }} - {{ license.name }}

+ 

{{ license.text }}

+ {%- endfor %} + +

End

+ This file was generated on: {{ utcnow }} (UTC) + + diff --git a/src/aboutcode/templates/list.csv b/src/aboutcode/templates/list.csv new file mode 100644 index 00000000..ff48b505 --- /dev/null +++ b/src/aboutcode/templates/list.csv @@ -0,0 +1,4 @@ +Name,Version,License,Homepage +{% for package in packages %} +"{{package.name}}","{{package.version}}","{{package.license_expression}}","{{package.homepage_url}}" +{% endfor %} diff --git a/src/attributecode/transform.py b/src/aboutcode/transform.py similarity index 74% rename from src/attributecode/transform.py rename to src/aboutcode/transform.py index ba9b9eb8..e84fc6be 100644 --- a/src/attributecode/transform.py +++ b/src/aboutcode/transform.py @@ -23,11 +23,12 @@ import attr -from attributecode import CRITICAL -from attributecode import Error -from attributecode import saneyaml -from attributecode.util import csv -from attributecode.util import python2 +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import saneyaml +from aboutcode.model import Package +from aboutcode.util import csv +from aboutcode.util import python2 if python2: # pragma: nocover @@ -72,7 +73,7 @@ def transform_data(rows, transformer): column_names = next(rows) column_names = transformer.clean_columns(column_names) - dupes = check_duplicate_columns(column_names) + dupes = get_duplicate_columns(column_names) if dupes: msg = 'Duplicated column name: {name}' @@ -84,9 +85,9 @@ def transform_data(rows, transformer): # convert to dicts using the renamed columns data = [OrderedDict(zip_longest(column_names, row)) for row in rows] - if transformer.column_filters: + if transformer.kept_columns: data = list(transformer.filter_columns(data)) - column_names = [c for c in column_names if c in transformer.column_filters] + column_names = [c for c in column_names if c in transformer.kept_columns] errors = transformer.check_required_columns(data) if errors: @@ -96,48 +97,60 @@ def transform_data(rows, transformer): tranformer_config_help = ''' -A transform configuration file is used to describe which transformations and -validations to apply to a source CSV file. This is a simple text file using YAML -format, using the same format as an .ABOUT file. -The attributes that can be set in a configuration file are: +A transform configuration file is used to describe which transformations and to +apply to a source CSV file. This configuration file a simple text file using +YAML format (the same format as an .ABOUT file). + +The settings that can be defines in a configuration file are: * column_renamings: -An optional map of source CSV column name to target CSV new column name that +An optional map of source CSV column names to target CSV new column names that is used to rename CSV columns. For instance with this configuration the columns "Directory/Location" will be renamed to "about_resource" and "foo" to "bar": + column_renamings: - 'Directory/Location' : about_resource - foo : bar + 'Directory/Location': about_resource + foo: bar The renaming is always applied first before other transforms and checks. All other column names referenced below are these that exist AFTER the renamings have been applied to the existing column names. + +* kept_columns: +An optional list of column names that should be kept in the transformed CSV. + +If this list is NOT provided, all the columns from the source CSV will be kept +(and added in generated .ABOUT files if the CSV is used for this later). + +If this list is provided, only the listed columns are kept in the transformed +CSV and all other columns are removed. + +For instance with this configuration the target CSV will only contains the "name" +and "version" columns and no other column: + + kept_columns: + - name + - version + + * required_columns: An optional list of required column names that must have a value, beyond the -standard columns names. If a source CSV does not have such a column or a row is -missing a value for a required column, an error is reported. +standard required columns names (e.g. about_resource and about file_path for +the gen command). + +If a source CSV does not have such a column or a row is missing a value for a +listed required column, an error is reported. This validation occurs after +processing the CVS for "kept_columns" For instance with this configuration an error will be reported if the columns "name" and "version" are missing or if any row does not have a value set for these columns: - required_columns: - - name - - version - -* column_filters: -An optional list of column names that should be kept in the transformed CSV. If -this list is provided, all the columns from the source CSV that should be kept -in the target CSV must be listed be even if they are standard or required -columns. If this list is not provided, all source CSV columns are kept in the -transformed target CSV. -For instance with this configuration the target CSV will only contains the "name" -and "version" columns and no other column: - column_filters: + required_columns: - name - version ''' @@ -149,7 +162,7 @@ class Transformer(object): column_renamings = attr.attrib(default=attr.Factory(dict)) required_columns = attr.attrib(default=attr.Factory(list)) - column_filters = attr.attrib(default=attr.Factory(list)) + kept_columns = attr.attrib(default=attr.Factory(list)) # a list of all the standard columns from AboutCode toolkit standard_columns = attr.attrib(default=attr.Factory(list), init=False) @@ -159,21 +172,8 @@ class Transformer(object): # called by attr after the __init__() def __attrs_post_init__(self, *args, **kwargs): - from attributecode.model import About - about = About() - self.essential_columns = list(about.required_fields) - self.standard_columns = [f.name for f in about.all_fields()] - - @classmethod - def default(cls): - """ - Return a default Transformer with built-in transforms. - """ - return cls( - column_renamings={}, - required_columns=[], - column_filters=[], - ) + self.essential_columns = ['about_resource'] + self.standard_columns = Package.standard_fields() @classmethod def from_file(cls, location): @@ -186,7 +186,7 @@ def from_file(cls, location): return cls( column_renamings=data.get('column_renamings', {}), required_columns=data.get('required_columns', []), - column_filters=data.get('column_filters', []), + kept_columns=data.get('kept_columns', []), ) def check_required_columns(self, data): @@ -196,15 +196,13 @@ def check_required_columns(self, data): """ errors = [] required = set(self.essential_columns + self.required_columns) - if not required: - return [] - for rn, item in enumerate(data): + for rn, item in enumerate(data, 1): missings = [rk for rk in required if not item.get(rk)] if not missings: continue - missings = ', '.join(missings) + missings = ', '.join(sorted(missings)) msg = 'Row {rn} is missing required values for columns: {missings}' errors.append(Error(CRITICAL, msg.format(**locals()))) return errors @@ -228,7 +226,7 @@ def apply_renamings(self, column_names): def clean_columns(self, column_names): """ - Apply standard cleanups to a list of columns and return these. + Apply standard cleanups to a list of column names and return these. """ if not column_names: return column_names @@ -236,17 +234,20 @@ def clean_columns(self, column_names): def filter_columns(self, data): """ - Yield transformed dicts from a `data` list of dicts keeping only - columns with a name in the `column_filters`of this Transformer. - Return the data unchanged if no `column_filters` exists. + Yield transformed mappings from a `data` list of mapping keeping only + columns with a name in the `kept_columns` of this Transformer. + Return the data unchanged if `kept_columns` does not exist or is empty. """ - column_filters = set(self.clean_columns(self.column_filters)) + kept_columns = set(self.clean_columns(self.kept_columns)) for entry in data: - items = ((k, v) for k, v in entry.items() if k in column_filters) - yield OrderedDict(items) + if kept_columns: + items = ((k, v) for k, v in entry.items() if k in kept_columns) + yield OrderedDict(items) + else: + yield entry -def check_duplicate_columns(column_names): +def get_duplicate_columns(column_names): """ Check that there are no duplicate in the `column_names` list of column name strings, ignoring case. Return a list of unique duplicated column names. @@ -259,7 +260,9 @@ def read_csv_rows(location): """ Yield rows (as a list of values) from a CSV file at `location`. """ - with io.open(location, encoding='utf-8') as csvfile: + # note: Excel can produce unreadable UTF files + with io.open(location, encoding='utf-8', errors='replace') as csvfile: + # note: we do not use a dict reader to check later for duplicate column names reader = csv.reader(csvfile) for row in reader: yield row @@ -270,7 +273,7 @@ def write_csv(location, data, column_names): # NOQA Write a CSV file at `location` the `data` list of ordered dicts using the `column_names`. """ - with io.open(location, 'w', encoding='utf-8', newline='\n') as csvfile: + with io.open(location, 'w', encoding='utf-8') as csvfile: writer = csv.DictWriter(csvfile, fieldnames=column_names) writer.writeheader() writer.writerows(data) diff --git a/src/aboutcode/util.py b/src/aboutcode/util.py new file mode 100644 index 00000000..519427b3 --- /dev/null +++ b/src/aboutcode/util.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- +# ============================================================================ +# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +from collections import OrderedDict +import os +import posixpath +import string +import sys + +from aboutcode import CRITICAL +from aboutcode import Error + + +python2 = sys.version_info[0] < 3 + +if python2: # pragma: nocover + from backports import csv # NOQA + # monkey patch backports.csv until bug is fixed + # https://github.com/ryanhiebert/backports.csv/issues/30 + csv.dict = OrderedDict +else: # pragma: nocover + import csv # NOQA + + +on_windows = 'win32' in sys.platform + + +def to_posix(path): + """ + Return a path using the posix path separator given a path that may contain + posix or windows separators, converting "\\" to "/". + NB: this path will still be valid in the windows explorer. It will be a + valid path everywhere in Python. It may not lways be valid for windows + command line operations. + """ + return path.replace('\\', '/') + + +def to_native(path): + """ + Return a path using the current OS path separator given a path that may + contain posix or windows separators, converting "/" to "\\" on windows + and "\\" to "/" on posix OSes. + """ + return path.replace('\\', os.path.sep).replace('/', os.path.sep) + + +valid_file_chars = string.digits + string.ascii_letters + '_-.' + ' ' + + +def invalid_chars(path): + """ + Return a list of invalid characters in the file name of `path`. + """ + path = to_posix(path) + rname = resource_name(path) + name = rname.lower() + return [c for c in name if c not in valid_file_chars] + +# FIXME: do not checl for invalida characters +def check_file_names(paths): + """ + Given a sequence of file paths, check that file names are valid and that + there are no case-insensitive duplicates in any given directories. + Return a list of errors. + + From spec : + A file name can contain only these US-ASCII characters: + - digits from 0 to 9 + - uppercase and lowercase letters from A to Z + - the _ underscore, - dash and . period signs. + From spec: + The case of a file name is not significant. On case-sensitive file + systems (such as Linux), a tool must raise an error if two ABOUT files + stored in the same directory have the same lowercase file name. + """ + # FIXME: this should be a defaultdicts that accumulates all duplicated paths + seen = {} + errors = [] + for orig_path in paths: + path = orig_path + invalid = invalid_chars(path) + if invalid: + invalid = ''.join(invalid) + msg = ('Invalid characters %(invalid)r in file name at: ' + '%(path)r' % locals()) + errors.append(Error(CRITICAL, msg)) + + path = to_posix(orig_path) + name = resource_name(path).lower() + parent = posixpath.dirname(path) + path = posixpath.join(parent, name) + path = posixpath.normpath(path) + path = posixpath.abspath(path) + existing = seen.get(path) + if existing: + msg = ('Duplicate files: %(orig_path)r and %(existing)r ' + 'have the same case-insensitive file name' % locals()) + errors.append(Error(CRITICAL, msg)) + else: + seen[path] = orig_path + return errors + + +def normalize(location): + """ + Return an absolute normalized location. + """ + location = os.path.expanduser(location) + location = os.path.expandvars(location) + location = os.path.normpath(location) + location = os.path.abspath(location) + return location + + +def get_relative_path(base_loc, full_loc): + """ + Return a posix path for a given full_loc location relative to a base_loc + location. + """ + def norm(p): + p = to_posix(p) + p = p.strip('/') + return posixpath.normpath(p) + + base = norm(base_loc) + full = norm(full_loc) + + assert full.startswith(base), ( + 'Cannot compute relative path: %(full_loc)r does not starts with %(base_loc)r' % locals()) + + assert full != base, ( + 'Cannot compute relative path: %(full_loc)r is the same as: %(base_loc)r' % locals()) + relative = full[len(base) + 1:] + + # We don't want to keep the first segment of the root of the returned path. + # See https://github.com/nexB/aboutcode/issues/276 + # relative = posixpath.join(base_name, relative) + return relative + + +def resource_name(path): + """ + Return the file or directory name from a path. + """ + path = path.strip() + path = to_posix(path) + path = path.rstrip('/') + _left, right = posixpath.split(path) + return right.strip() + + +def extract_zip(location): + """ + Extract a zip file at location in a temp directory and return the temporary + directory where the archive was extracted. + """ + import zipfile + import tempfile + + if not zipfile.is_zipfile(location): + raise Exception('Incorrect zip file %(location)r' % locals()) + + archive_base_name = os.path.basename(location).replace('.zip', '') + base_dir = tempfile.mkdtemp(prefix='aboutcode-toolkit-extract-') + target_dir = os.path.join(base_dir, archive_base_name) + + os.makedirs(target_dir) + + if target_dir.endswith(('\\', '/')): + target_dir = target_dir[:-1] + + with zipfile.ZipFile(location) as zipf: + for info in zipf.infolist(): + name = info.filename + content = zipf.read(name) + target = os.path.join(target_dir, name) + is_dir = target.endswith(('\\', '/')) + if is_dir: + target = target[:-1] + parent = os.path.dirname(target) + if on_windows: + target = target.replace('/', '\\') + parent = parent.replace('/', '\\') + if not os.path.exists(parent): + os.makedirs(parent) + if not content and is_dir: + if not os.path.exists(target): + os.makedirs(target) + if not os.path.exists(target): + with open(target, 'wb') as f: + f.write(content) + return target_dir + + +def unique(sequence): + """ + Return a list of unique items found in sequence. Preserve the original + sequence order. + For example: + >>> unique([1, 5, 3, 5]) + [1, 5, 3] + """ + deduped = [] + for item in sequence: + if item not in deduped: + deduped.append(item) + return deduped diff --git a/src/attributecode/api.py b/src/attributecode/api.py deleted file mode 100644 index 5eb4cef6..00000000 --- a/src/attributecode/api.py +++ /dev/null @@ -1,119 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf8 -*- - -# ============================================================================ -# Copyright (c) 2013-2017 nexB Inc. http://www.nexb.com/ - All rights reserved. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ - -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -import json - -from attributecode import ERROR -from attributecode import Error -from attributecode.util import python2 - - -if python2: # pragma: nocover - from urllib import quote # NOQA - from urllib import urlencode # NOQA - from urllib2 import HTTPError # NOQA - from urllib2 import Request # NOQA - from urllib2 import urlopen # NOQA -else: # pragma: nocover - from urllib.parse import quote # NOQA - from urllib.parse import urlencode # NOQA - from urllib.request import Request # NOQA - from urllib.request import urlopen # NOQA - from urllib.error import HTTPError # NOQA - - -""" -API call helpers -""" - - -# FIXME: args should start with license_key -def request_license_data(api_url, api_key, license_key): - """ - Return a tuple of (dictionary of license data, list of errors) given a - `license_key`. Send a request to `api_url` authenticating with `api_key`. - """ - headers = { - 'Authorization': 'Token %s' % api_key, - } - payload = { - 'api_key': api_key, - 'key': license_key, - 'format': 'json' - } - - api_url = api_url.rstrip('/') - payload = urlencode(payload) - - full_url = '%(api_url)s/?%(payload)s' % locals() - # handle special characters in URL such as space etc. - quoted_url = quote(full_url, safe="%/:=&?~#+!$,;'@()*[]") - - license_data = {} - errors = [] - try: - request = Request(quoted_url, headers=headers) - response = urlopen(request) - response_content = response.read().decode('utf-8') - # FIXME: this should be an ordered dict - license_data = json.loads(response_content) - if not license_data['results']: - msg = u"Invalid 'license': %s" % license_key - errors.append(Error(ERROR, msg)) - - except HTTPError as http_e: - # some auth problem - if http_e.code == 403: - msg = (u"Authorization denied. Invalid '--api_key'. " - u"License generation is skipped.") - errors.append(Error(ERROR, msg)) - else: - # Since no api_url/api_key/network status have - # problem detected, it yields 'license' is the cause of - # this exception. - msg = u"Invalid 'license': %s" % license_key - errors.append(Error(ERROR, msg)) - - except Exception as e: - errors.append(Error(ERROR, str(e))) - - finally: - if license_data.get('count') == 1: - license_data = license_data.get('results')[0] - else: - license_data = {} - - return license_data, errors - - -# FIXME: args should start with license_key -def get_license_details_from_api(api_url, api_key, license_key): - """ - Return a tuple of license data given a `license_key` using the `api_url` - authenticating with `api_key`. - The details are a tuple of (license_name, license_key, license_text, errors) - where errors is a list of strings. - Missing values are provided as empty strings. - """ - license_data, errors = request_license_data(api_url, api_key, license_key) - license_name = license_data.get('name', '') - license_text = license_data.get('full_text', '') - license_key = license_data.get('key', '') - return license_name, license_key, license_text, errors diff --git a/src/attributecode/attrib.py b/src/attributecode/attrib.py deleted file mode 100644 index bf59150a..00000000 --- a/src/attributecode/attrib.py +++ /dev/null @@ -1,204 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf8 -*- - -# ============================================================================ -# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ - -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -import collections -import datetime -import io -import os - -import jinja2 - -from attributecode import CRITICAL -from attributecode import ERROR -from attributecode import Error -from attributecode.licenses import COMMON_LICENSES -from attributecode.model import parse_license_expression -from attributecode.util import add_unc - - -# FIXME: the template dir should be outside the code tree -DEFAULT_TEMPLATE_FILE = os.path.join( - os.path.dirname(os.path.realpath(__file__)), 'templates', 'default_html.template') - - -def generate(abouts, template=None, variables=None): - """ - Generate an attribution text from an `abouts` list of About objects, a - `template` template text and a `variables` optional dict of extra - variables. - - Return a tuple of (error, attribution text) where error is an Error object - or None and attribution text is the generated text or None. - """ - rendered = None - error = None - template_error = check_template(template) - if template_error: - lineno, message = template_error - error = Error( - CRITICAL, - 'Template validation error at line: {lineno}: "{message}"'.format(**locals()) - ) - return error, None - - template = jinja2.Template(template) - - try: - captured_license = [] - license_key_and_context = {} - sorted_license_key_and_context = {} - license_file_name_and_key = {} - license_key_to_license_name = {} - license_name_to_license_key = {} - # FIXME: This need to be simplified - for about in abouts: - # about.license_file.value is a OrderDict with license_text_name as - # the key and the license text as the value - if about.license_file: - # We want to create a dictionary which have the license short name as - # the key and license text as the value - for license_text_name in about.license_file.value: - if not license_text_name in captured_license: - captured_license.append(license_text_name) - if license_text_name.endswith('.LICENSE'): - license_key = license_text_name.strip('.LICENSE') - else: - license_key = license_text_name - license_key_and_context[license_key] = about.license_file.value[license_text_name] - sorted_license_key_and_context = collections.OrderedDict(sorted(license_key_and_context.items())) - license_file_name_and_key[license_text_name] = license_key - - # Convert/map the key in license expression to license name - if about.license_expression.value and about.license_name.value: - special_char_in_expression, lic_list = parse_license_expression(about.license_expression.value) - lic_name_list = about.license_name.value - lic_name_expression_list = [] - - # The order of the license_name and key should be the same - # The length for both list should be the same - assert len(lic_name_list) == len(lic_list) - - # Map the license key to license name - index_for_license_name_list = 0 - for key in lic_list: - license_key_to_license_name[key] = lic_name_list[index_for_license_name_list] - license_name_to_license_key[lic_name_list[index_for_license_name_list]] = key - index_for_license_name_list = index_for_license_name_list + 1 - - # Create a license expression with license name instead of key - for segment in about.license_expression.value.split(): - if segment in license_key_to_license_name: - lic_name_expression_list.append(license_key_to_license_name[segment]) - else: - lic_name_expression_list.append(segment) - - # Join the license name expression into a single string - lic_name_expression = ' '.join(lic_name_expression_list) - - # Add the license name expression string into the about object - about.license_name_expression = lic_name_expression - - # Get the current UTC time - utcnow = datetime.datetime.utcnow() - rendered = template.render( - abouts=abouts, common_licenses=COMMON_LICENSES, - license_key_and_context=sorted_license_key_and_context, - license_file_name_and_key=license_file_name_and_key, - license_key_to_license_name=license_key_to_license_name, - license_name_to_license_key=license_name_to_license_key, - utcnow=utcnow, - variables=variables - ) - except Exception as e: - lineno = getattr(e, 'lineno', '') or '' - if lineno: - lineno = ' at line: {}'.format(lineno) - err = getattr(e, 'message', '') or '' - error = Error( - CRITICAL, - 'Template processing error {lineno}: {err}'.format(**locals()), - ) - return error, rendered - - -def check_template(template_string): - """ - Check the syntax of a template. Return an error tuple (line number, - message) if the template is invalid or None if it is valid. - """ - try: - jinja2.Template(template_string) - except (jinja2.TemplateSyntaxError, jinja2.TemplateAssertionError) as e: - return e.lineno, e.message - - -def generate_from_file(abouts, template_loc=DEFAULT_TEMPLATE_FILE, variables=None): - """ - Generate an attribution text from an `abouts` list of About objects, a - `template_loc` template file location and a `variables` optional - dict of extra variables. - - Return a tuple of (error, attribution text) where error is an Error object - or None and attribution text is the generated text or None. - """ - - template_loc = add_unc(template_loc) - with io.open(template_loc, encoding='utf-8') as tplf: - tpls = tplf.read() - return generate(abouts, template=tpls, variables=variables) - - -def generate_and_save(abouts, output_location, template_loc=None, variables=None): - """ - Generate an attribution text from an `abouts` list of About objects, a - `template_loc` template file location and a `variables` optional - dict of extra variables. Save the generated attribution text in the - `output_location` file. - Return a list of Error objects if any. - """ - errors = [] - - # Parse license_expression and save to the license list - for about in abouts: - if not about.license_expression.value: - continue - special_char_in_expression, lic_list = parse_license_expression(about.license_expression.value) - if special_char_in_expression: - msg = (u"The following character(s) cannot be in the license_expression: " + - str(special_char_in_expression)) - errors.append(Error(ERROR, msg)) - else: - about.license_key.value = lic_list - - rendering_error, rendered = generate_from_file( - abouts, - template_loc=template_loc, - variables=variables - ) - - if rendering_error: - errors.append(rendering_error) - - if rendered: - output_location = add_unc(output_location) - with io.open(output_location, 'w', encoding='utf-8') as of: - of.write(rendered) - - return errors diff --git a/src/attributecode/gen.py b/src/attributecode/gen.py deleted file mode 100644 index a3bd6fd1..00000000 --- a/src/attributecode/gen.py +++ /dev/null @@ -1,289 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf8 -*- - -# ============================================================================ -# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ - -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -import codecs -from collections import OrderedDict - -# FIXME: why posipath??? -from posixpath import basename -from posixpath import dirname -from posixpath import exists -from posixpath import join -from posixpath import normpath - -from attributecode import ERROR -from attributecode import CRITICAL -from attributecode import INFO -from attributecode import Error -from attributecode import model -from attributecode import util -from attributecode.util import add_unc -from attributecode.util import csv -from attributecode.util import to_posix -from attributecode.util import UNC_PREFIX_POSIX -from attributecode.util import unique - - -def check_duplicated_columns(location): - """ - Return a list of errors for duplicated column names in a CSV file - at location. - """ - location = add_unc(location) - # FIXME: why errors=ignore? - with codecs.open(location, 'rb', encoding='utf-8', errors='ignore') as csvfile: - reader = csv.reader(csvfile) - columns = next(reader) - columns = [col for col in columns] - - seen = set() - dupes = OrderedDict() - for col in columns: - c = col.lower() - if c in seen: - if c in dupes: - dupes[c].append(col) - else: - dupes[c] = [col] - seen.add(c.lower()) - - errors = [] - if dupes: - dup_msg = [] - for name, names in dupes.items(): - names = u', '.join(names) - msg = '%(name)s with %(names)s' % locals() - dup_msg.append(msg) - dup_msg = u', '.join(dup_msg) - msg = ('Duplicated column name(s): %(dup_msg)s\n' % locals() + - 'Please correct the input and re-run.') - errors.append(Error(ERROR, msg)) - return unique(errors) - - -def check_duplicated_about_file_path(inventory_dict): - """ - Return a list of errors for duplicated about_file_path in a CSV file at location. - """ - afp_list = [] - errors = [] - for component in inventory_dict: - # Ignore all the empty path - if component['about_file_path']: - if component['about_file_path'] in afp_list: - msg = ("The input has duplicated values in 'about_file_path' " - "field: " + component['about_file_path']) - errors.append(Error(CRITICAL, msg)) - else: - afp_list.append(component['about_file_path']) - return errors - - -# TODO: this should be either the CSV or the ABOUT files but not both??? -def load_inventory(location, base_dir, reference_dir=None): - """ - Load the inventory file at `location` for ABOUT and LICENSE files stored in - the `base_dir`. Return a list of errors and a list of About objects - validated against the `base_dir`. - - Optionally use `reference_dir` as the directory location of extra reference - license and notice files to reuse. - """ - errors = [] - abouts = [] - base_dir = util.to_posix(base_dir) - # FIXME: do not mix up CSV and JSON - if location.endswith('.csv'): - # FIXME: this should not be done here. - dup_cols_err = check_duplicated_columns(location) - if dup_cols_err: - errors.extend(dup_cols_err) - return errors, abouts - inventory = util.load_csv(location) - else: - inventory = util.load_json(location) - - try: - # FIXME: this should not be done here. - dup_about_paths_err = check_duplicated_about_file_path(inventory) - if dup_about_paths_err: - errors.extend(dup_about_paths_err) - return errors, abouts - except Exception as e: - # TODO: why catch ALL Exception - msg = "The essential field 'about_file_path' is not found in the " - errors.append(Error(CRITICAL, msg)) - return errors, abouts - - for i, fields in enumerate(inventory): - # check does the input contains the required fields - required_fields = model.About.required_fields - - for f in required_fields: - if f not in fields: - msg = "Required fiel: %(f)r not found in the " % locals() - errors.append(Error(ERROR, msg)) - return errors, abouts - afp = fields.get(model.About.ABOUT_FILE_PATH_ATTR) - - # FIXME: this should not be a failure condition - if not afp or not afp.strip(): - msg = 'Empty column: %(afp)r. Cannot generate .ABOUT file.' % locals() - errors.append(Error(ERROR, msg)) - continue - else: - afp = util.to_posix(afp) - loc = join(base_dir, afp) - about = model.About(about_file_path=afp) - about.location = loc - - ld_errors = about.load_dict( - fields, - base_dir, - running_inventory=False, - reference_dir=reference_dir, - ) - # 'about_resource' field will be generated during the process. - # No error need to be raise for the missing 'about_resource'. - for e in ld_errors: - if e.message == 'Field about_resource is required': - ld_errors.remove(e) - for e in ld_errors: - if not e in errors: - errors.extend(ld_errors) - abouts.append(about) - - return unique(errors), abouts - - -def generate(location, base_dir, reference_dir=None, fetch_license=False): - """ - Load ABOUT data from a CSV inventory at `location`. Write ABOUT files to - base_dir. Return errors and about objects. - """ - not_exist_errors = [] - api_url = '' - api_key = '' - gen_license = False - # FIXME: use two different arguments: key and url - # Check if the fetch_license contains valid argument - if fetch_license: - # Strip the ' and " for api_url, and api_key from input - api_url = fetch_license[0].strip("'").strip('"') - api_key = fetch_license[1].strip("'").strip('"') - gen_license = True - - # TODO: WHY use posix?? - bdir = to_posix(base_dir) - - errors, abouts = load_inventory( - location=location, - base_dir=bdir, - reference_dir=reference_dir - ) - - if gen_license: - license_dict, err = model.pre_process_and_fetch_license_dict(abouts, api_url, api_key) - if err: - for e in err: - # Avoid having same error multiple times - if not e in errors: - errors.append(e) - - for about in abouts: - if about.about_file_path.startswith('/'): - about.about_file_path = about.about_file_path.lstrip('/') - dump_loc = join(bdir, about.about_file_path.lstrip('/')) - - # The following code is to check if there is any directory ends with spaces - split_path = about.about_file_path.split('/') - dir_endswith_space = False - for segment in split_path: - if segment.endswith(' '): - msg = (u'File path : ' - u'%(dump_loc)s ' - u'contains directory name ends with spaces which is not ' - u'allowed. Generation skipped.' % locals()) - errors.append(Error(ERROR, msg)) - dir_endswith_space = True - break - if dir_endswith_space: - # Continue to work on the next about object - continue - - try: - # Generate value for 'about_resource' if it does not exist - if not about.about_resource.value: - about.about_resource.value = OrderedDict() - about_resource_value = '' - if about.about_file_path.endswith('/'): - about_resource_value = u'.' - else: - about_resource_value = basename(about.about_file_path) - about.about_resource.value[about_resource_value] = None - about.about_resource.present = True - # Check for the existence of the 'about_resource' - # If the input already have the 'about_resource' field, it will - # be validated when creating the about object - loc = util.to_posix(dump_loc) - about_file_loc = loc - path = join(dirname(util.to_posix(about_file_loc)), about_resource_value) - if not exists(path): - path = util.to_posix(path.strip(UNC_PREFIX_POSIX)) - path = normpath(path) - msg = (u'Field about_resource: ' - u'%(path)s ' - u'does not exist' % locals()) - not_exist_errors.append(msg) - - if gen_license: - # Write generated LICENSE file - license_key_name_context_url_list = about.dump_lic(dump_loc, license_dict) - if license_key_name_context_url_list: - # use value not "presence" - if not about.license_file.present: - about.license_file.value = OrderedDict() - for lic_key, lic_name, lic_context, lic_url in license_key_name_context_url_list: - gen_license_name = lic_key + u'.LICENSE' - about.license_file.value[gen_license_name] = lic_context - about.license_file.present = True - if not about.license_name.present: - about.license_name.value.append(lic_name) - if not about.license_url.present: - about.license_url.value.append(lic_url) - if about.license_url.value: - about.license_url.present = True - if about.license_name.value: - about.license_name.present = True - - about.dump(dump_loc) - - for e in not_exist_errors: - errors.append(Error(INFO, e)) - - except Exception as e: - # only keep the first 100 char of the exception - # TODO: truncated errors are likely making diagnotics harder - emsg = repr(e)[:100] - msg = (u'Failed to write .ABOUT file at : ' - u'%(dump_loc)s ' - u'with error: %(emsg)s' % locals()) - errors.append(Error(ERROR, msg)) - return unique(errors), abouts diff --git a/src/attributecode/model.py b/src/attributecode/model.py deleted file mode 100644 index a1bb9e63..00000000 --- a/src/attributecode/model.py +++ /dev/null @@ -1,1310 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf8 -*- -# ============================================================================ -# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ - -""" -AboutCode toolkit is a tool to process ABOUT files. ABOUT files are -small text files that document the provenance (aka. the origin and -license) of software components as well as the essential obligation -such as attribution/credits and source code redistribution. See the -ABOUT spec at http://dejacode.org. - -AboutCode toolkit reads and validates ABOUT files and collect software -components inventories. -""" - -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -from collections import OrderedDict -import io -import json -import os -# FIXME: why posixpath??? -import posixpath -import traceback - -from attributecode.util import python2 - -if python2: # pragma: nocover - from itertools import izip_longest as zip_longest # NOQA - from urlparse import urljoin, urlparse # NOQA - from urllib2 import urlopen, Request, HTTPError # NOQA -else: # pragma: nocover - basestring = str # NOQA - from itertools import zip_longest # NOQA - from urllib.parse import urljoin, urlparse # NOQA - from urllib.request import urlopen, Request # NOQA - from urllib.error import HTTPError # NOQA - -from license_expression import Licensing - -from attributecode import CRITICAL -from attributecode import ERROR -from attributecode import INFO -from attributecode import WARNING -from attributecode import api -from attributecode import Error -from attributecode import saneyaml -from attributecode import util -from attributecode.util import add_unc -from attributecode.util import copy_license_notice_files -from attributecode.util import csv -from attributecode.util import filter_errors -from attributecode.util import is_valid_name -from attributecode.util import on_windows -from attributecode.util import UNC_PREFIX -from attributecode.util import ungroup_licenses -from attributecode.util import unique - - -class Field(object): - """ - An ABOUT file field. The initial value is a string. Subclasses can and - will alter the value type as needed. - """ - - def __init__(self, name=None, value=None, required=False, present=False): - # normalized names are lowercased per specification - self.name = name - # save this and do not mutate it afterwards - if isinstance(value, basestring): - self.original_value = value - elif value: - self.original_value = repr(value) - else: - self.original_value = value - - # can become a string, list or OrderedDict() after validation - self.value = value or self.default_value() - - self.required = required - # True if the field is present in an About object - self.present = present - - self.errors = [] - - def default_value(self): - return '' - - def validate(self, *args, **kwargs): - """ - Validate and normalize thyself. Return a list of errors. - """ - errors = [] - name = self.name - - self.value = self.default_value() - if not self.present: - # required fields must be present - if self.required: - msg = u'Field %(name)s is required' - errors.append(Error(CRITICAL, msg % locals())) - return errors - else: - # present fields should have content ... - # The boolean value can be True, False and None - # The value True or False is the content of boolean fields - if not self.has_content: - # ... especially if required - if self.required: - msg = u'Field %(name)s is required and empty' - severity = CRITICAL - else: - severity = INFO - msg = u'Field %(name)s is present but empty.' - errors.append(Error(severity, msg % locals())) - else: - # present fields with content go through validation... - # first trim any trailing spaces on each line - if isinstance(self.original_value, basestring): - value = '\n'.join(s.rstrip() for s - in self.original_value.splitlines(False)) - # then strip leading and trailing spaces - value = value.strip() - else: - value = self.original_value - self.value = value - try: - validation_errors = self._validate(*args, **kwargs) - errors.extend(validation_errors) - except Exception as e: - emsg = repr(e) - msg = u'Error validating field %(name)s: %(value)r: %(emsg)r' - errors.append(Error(CRITICAL, msg % locals())) - raise - - # set or reset self - self.errors = errors - return errors - - def _validate(self, *args, **kwargs): - """ - Validate and normalize thyself. Return a list of errors. - Subclasses should override as needed. - """ - return [] - - def serialize(self): - """ - Return a unicode serialization of self in the ABOUT format. - """ - name = self.name - value = self.serialized_value() or u'' - if self.has_content or self.value: - value = value.splitlines(True) - # multi-line - if len(value) > 1: - # This code is used to read the YAML's multi-line format in - # ABOUT files - # (Test: test_loads_dumps_is_idempotent) - if value[0].strip() == u'|' or value[0].strip() == u'>': - value = u' '.join(value) - else: - # Insert '|' as the indicator for multi-line follow by a - # newline character - value.insert(0, u'|\n') - # insert 4 spaces for newline values - value = u' '.join(value) - else: - # FIXME: See https://github.com/nexB/aboutcode-toolkit/issues/323 - # The yaml.load() will throw error if the parsed value - # contains ': ' character. A work around is to put a pipe, '|' - # to indicate the whole value as a string - if value and ': ' in value[0]: - value.insert(0, u'|\n') - # insert 4 spaces for newline values - value = u' '.join(value) - else: - value = u''.join(value) - - serialized = u'%(name)s:' % locals() - if value: - serialized += ' ' + '%(value)s' % locals() - return serialized - - def serialized_value(self): - """ - Return a unicode serialization of self in the ABOUT format. - Does not include a white space for continuations. - """ - return self._serialized_value() or u'' - - @property - def has_content(self): - return self.original_value - - def __repr__(self): - name = self.name - value = self.value - required = self.required - has_content = self.has_content - present = self.present - r = ('Field(name=%(name)r, value=%(value)r, required=%(required)r, present=%(present)r)') - return r % locals() - - def __eq__(self, other): - """ - Equality based on string content value, ignoring spaces. - """ - return (isinstance(other, self.__class__) - and self.name == other.name - and self.value == other.value) - - -class StringField(Field): - """ - A field containing a string value possibly on multiple lines. - The validated value is a string. - """ - def _validate(self, *args, **kwargs): - errors = super(StringField, self)._validate(*args, ** kwargs) - return errors - - def _serialized_value(self): - return self.value if self.value else u'' - - def __eq__(self, other): - """ - Equality based on string content value, ignoring spaces - """ - if not (isinstance(other, self.__class__) - and self.name == other.name): - return False - - if self.value == other.value: - return True - - # compare values stripped from spaces. Empty and None are equal - if self.value: - sval = u''.join(self.value.split()) - if not sval: - sval = None - - if other.value: - oval = u''.join(other.value.split()) - if not oval: - oval = None - - if sval == oval: - return True - - -class SingleLineField(StringField): - """ - A field containing a string value on a single line. The validated value is - a string. - """ - def _validate(self, *args, **kwargs): - errors = super(SingleLineField, self)._validate(*args, ** kwargs) - if self.value and isinstance(self.value, basestring) and '\n' in self.value: - name = self.name - value = self.original_value - msg = (u'Field %(name)s: Cannot span multiple lines: %(value)s' - % locals()) - errors.append(Error(ERROR, msg)) - return errors - - -class ListField(StringField): - """ - A field containing a list of string values, one per line. The validated - value is a list. - """ - def default_value(self): - return [] - - def _validate(self, *args, **kwargs): - errors = super(ListField, self)._validate(*args, ** kwargs) - - # reset - self.value = [] - - if isinstance(self.original_value, basestring): - values = self.original_value.splitlines(False) - elif isinstance(self.original_value, list): - values = self.original_value - else: - values = [repr(self.original_value)] - - for val in values: - if isinstance(val, basestring): - val = val.strip() - if not val: - name = self.name - msg = (u'Field %(name)s: ignored empty list value' - % locals()) - errors.append(Error(INFO, msg)) - continue - # keep only unique and report error for duplicates - if val not in self.value: - self.value.append(val) - else: - name = self.name - msg = (u'Field %(name)s: ignored duplicated list value: ' - '%(val)r' % locals()) - errors.append(Error(WARNING, msg)) - return errors - - def _serialized_value(self): - return self.value if self.value else u'' - - def __eq__(self, other): - """ - Equality based on sort-insensitive values - """ - - if not (isinstance(other, self.__class__) - and self.name == other.name): - return False - - if self.value == other.value: - return True - - # compare values stripped from spaces. - sval = [] - if self.value and isinstance(self.value, list): - sval = sorted(self.value) - - oval = [] - if other.value and isinstance(other.value, list): - oval = sorted(other.value) - - if sval == oval: - return True - -class UrlListField(ListField): - """ - A URL field. The validated value is a list of URLs. - """ - def _validate(self, *args, **kwargs): - """ - Check that URLs are valid. Return a list of errors. - """ - errors = super(UrlListField, self)._validate(*args, ** kwargs) - name = self.name - val = self.value - for url in val: - if not self.is_valid_url(url): - msg = (u'Field %(name)s: Invalid URL: %(val)s' % locals()) - errors.append(Error(WARNING, msg)) - return errors - - @staticmethod - def is_valid_url(url): - """ - Return True if a URL is valid. - """ - scheme, netloc, _path, _p, _q, _frg = urlparse(url) - valid = scheme in ('http', 'https', 'ftp') and netloc - return valid - - -class UrlField(StringField): - """ - A URL field. The validated value is a URL. - """ - def _validate(self, *args, **kwargs): - """ - Check that URL is valid. Return a list of errors. - """ - errors = super(UrlField, self)._validate(*args, ** kwargs) - name = self.name - val = self.value - if not self.is_valid_url(val): - msg = (u'Field %(name)s: Invalid URL: %(val)s' % locals()) - errors.append(Error(WARNING, msg)) - return errors - - @staticmethod - def is_valid_url(url): - """ - Return True if a URL is valid. - """ - scheme, netloc, _path, _p, _q, _frg = urlparse(url) - valid = scheme in ('http', 'https', 'ftp') and netloc - return valid - - -class PathField(ListField): - """ - A field pointing to one or more paths relative to the ABOUT file location. - The validated value is an ordered dict of path->location or None. - The paths can also be resolved - """ - def default_value(self): - return {} - - def _validate(self, *args, **kwargs): - """ - Ensure that paths point to existing resources. Normalize to posix - paths. Return a list of errors. - - base_dir is the directory location of the ABOUT file used to resolve - relative paths to actual file locations. - """ - errors = super(PathField, self)._validate(*args, ** kwargs) - self.about_file_path = kwargs.get('about_file_path') - self.running_inventory = kwargs.get('running_inventory') - self.base_dir = kwargs.get('base_dir') - self.reference_dir = kwargs.get('reference_dir') - - if self.base_dir: - self.base_dir = util.to_posix(self.base_dir) - - name = self.name - - # FIXME: Why is the PathField an ordered dict? - # dict of normalized paths to a location or None - paths = OrderedDict() - - for path in self.value: - path = path.strip() - path = util.to_posix(path) - - # normalize eventual / to . - # and a succession of one or more ////// to . too - if path.strip() and not path.strip(posixpath.sep): - path = '.' - - # removing leading and trailing path separator - # path are always relative - path = path.strip(posixpath.sep) - - # the license files, if need to be copied, are located under the path - # set from the 'license-text-location' option, so the tool should check - # at the 'license-text-location' instead of the 'base_dir' - if not (self.base_dir or self.reference_dir): - msg = (u'Field %(name)s: Unable to verify path: %(path)s:' - u' No base directory provided' % locals()) - errors.append(Error(ERROR, msg)) - location = None - paths[path] = location - continue - - if self.reference_dir: - location = posixpath.join(self.reference_dir, path) - else: - # The 'about_resource' should be a joined path with - # the 'about_file_path' and the 'base_dir - if not self.running_inventory and self.about_file_path: - # Get the parent directory of the 'about_file_path' - afp_parent = posixpath.dirname(self.about_file_path) - - # Create a relative 'about_resource' path by joining the - # parent of the 'about_file_path' with the value of the - # 'about_resource' - arp = posixpath.join(afp_parent, path) - normalized_arp = posixpath.normpath(arp).strip(posixpath.sep) - location = posixpath.join(self.base_dir, normalized_arp) - else: - location = posixpath.join(self.base_dir, path) - - location = util.to_native(location) - location = os.path.abspath(os.path.normpath(location)) - location = util.to_posix(location) - location = add_unc(location) - - if not os.path.exists(location): - # We don't want to show the UNC_PREFIX in the error message - location = util.to_posix(location.strip(UNC_PREFIX)) - msg = (u'Field %(name)s: Path %(location)s not found' - % locals()) - # We want to show INFO error for 'about_resource' - if name == u'about_resource': - errors.append(Error(INFO, msg)) - else: - errors.append(Error(CRITICAL, msg)) - location = None - - paths[path] = location - - self.value = paths - return errors - - -class AboutResourceField(PathField): - """ - Special field for about_resource. self.resolved_paths contains a list of - the paths resolved relative to the about file path. - """ - def __init__(self, *args, ** kwargs): - super(AboutResourceField, self).__init__(*args, ** kwargs) - self.resolved_paths = [] - - def _validate(self, *args, **kwargs): - errors = super(AboutResourceField, self)._validate(*args, ** kwargs) - return errors - - -class FileTextField(PathField): - """ - A path field pointing to one or more text files such as license files. - The validated value is an ordered dict of path->Text or None if no - location or text could not be loaded. - """ - def _validate(self, *args, **kwargs): - """ - Load and validate the texts referenced by paths fields. Return a list - of errors. base_dir is the directory used to resolve a file location - from a path. - """ - - errors = super(FileTextField, self)._validate(*args, ** kwargs) - super(FileTextField, self)._validate(*args, ** kwargs) - - # a FileTextField is a PathField - # self.value is a paths to location ordered dict - # we will replace the location with the text content - name = self.name - for path, location in self.value.items(): - if not location: - # do not try to load if no location - # errors about non existing locations are PathField errors - # already collected. - continue - try: - # TODO: we have lots the location by replacing it with a text - location = add_unc(location) - with io.open(location, encoding='utf-8') as txt: - text = txt.read() - self.value[path] = text - except Exception as e: - # only keep the first 100 char of the exception - emsg = repr(e)[:100] - msg = (u'Field %(name)s: Failed to load text at path: ' - u'%(path)s ' - u'with error: %(emsg)s' % locals()) - errors.append(Error(ERROR, msg)) - # set or reset self - self.errors = errors - return errors - - -class BooleanField(SingleLineField): - """ - An flag field with a boolean value. Validated value is False, True or None. - """ - def default_value(self): - return None - - true_flags = ('yes', 'y', 'true', 'x') - false_flags = ('no', 'n', 'false') - flag_values = true_flags + false_flags - - def _validate(self, *args, **kwargs): - """ - Check that flag are valid. Convert flags to booleans. Default flag to - False. Return a list of errors. - """ - errors = super(BooleanField, self)._validate(*args, ** kwargs) - self.about_file_path = kwargs.get('about_file_path') - flag = self.get_flag(self.original_value) - if flag is False: - name = self.name - val = self.original_value - about_file_path = self.about_file_path - flag_values = self.flag_values - msg = (u'Path: %(about_file_path)s - Field %(name)s: Invalid flag value: %(val)r is not ' - u'one of: %(flag_values)s' % locals()) - errors.append(Error(ERROR, msg)) - self.value = None - elif flag is None: - name = self.name - msg = (u'Field %(name)s: field is empty. ' - u'Defaulting flag to no.' % locals()) - errors.append(Error(INFO, msg)) - self.value = None - else: - if flag == u'yes' or flag is True: - self.value = True - else: - self.value = False - return errors - - def get_flag(self, value): - """ - Return a normalized existing flag value if found in the list of - possible values or None if empty or False if not found or original value - if it is not a boolean value - """ - if value is None or value == '': - return None - - if isinstance(value, bool): - return value - else: - if isinstance(value, basestring): - value = value.strip() - if not value: - return None - - value = value.lower() - if value in self.flag_values: - if value in self.true_flags: - return u'yes' - else: - return u'no' - else: - return False - else: - return False - - @property - def has_content(self): - """ - Return true if it has content regardless of what value, False otherwise - """ - if self.original_value: - return True - return False - - def _serialized_value(self): - # default normalized values for serialization - if self.value: - return u'yes' - elif self.value is False: - return u'no' - else: - # self.value is None - # TODO: should we serialize to No for None??? - return u'' - - def __eq__(self, other): - """ - Boolean equality - """ - return (isinstance(other, self.__class__) - and self.name == other.name - and self.value == other.value) - - -def validate_fields(fields, about_file_path, running_inventory, base_dir, - reference_dir=None): - """ - Validate a sequence of Field objects. Return a list of errors. - Validation may update the Field objects as needed as a side effect. - """ - errors = [] - for f in fields: - val_err = f.validate( - base_dir=base_dir, - about_file_path=about_file_path, - running_inventory=running_inventory, - reference_dir=reference_dir, - ) - errors.extend(val_err) - return errors - - -def validate_field_name(name): - if not is_valid_name(name): - msg = ('Field name: %(name)r contains illegal name characters: ' - '0 to 9, a to z, A to Z and _.') - return Error(CRITICAL, msg % locals()) - - -class About(object): - """ - Represent an ABOUT file and functions to parse and validate a file. - """ - # special names, used only when serializing lists of ABOUT files to CSV or - # similar - - # name of the attribute containing the relative ABOUT file path - ABOUT_FILE_PATH_ATTR = 'about_file_path' - - # name of the attribute containing the resolved relative Resources paths - about_resource_path_attr = 'about_resource_path' - - # Required fields - required_fields = [ABOUT_FILE_PATH_ATTR, 'name'] - - def get_required_fields(self): - return [f for f in self.fields if f.required] - - def set_standard_fields(self): - """ - Create fields in an ordered dict to keep a standard ordering. We - could use a metaclass to track ordering django-like but this approach - is simpler. - """ - self.fields = OrderedDict([ - ('about_resource', AboutResourceField(required=True)), - ('name', SingleLineField(required=True)), - ('version', SingleLineField()), - - ('download_url', UrlField()), - ('description', StringField()), - ('homepage_url', UrlField()), - ('notes', StringField()), - - ('license_expression', StringField()), - ('license_key', ListField()), - ('license_name', ListField()), - ('license_file', FileTextField()), - ('license_url', UrlListField()), - ('copyright', StringField()), - ('notice_file', FileTextField()), - ('notice_url', UrlField()), - - ('redistribute', BooleanField()), - ('attribute', BooleanField()), - ('track_changes', BooleanField()), - ('modified', BooleanField()), - ('internal_use_only', BooleanField()), - - ('changelog_file', FileTextField()), - - ('owner', StringField()), - ('owner_url', UrlField()), - ('contact', StringField()), - ('author', StringField()), - ('author_file', FileTextField()), - - ('vcs_tool', SingleLineField()), - ('vcs_repository', SingleLineField()), - ('vcs_path', SingleLineField()), - ('vcs_tag', SingleLineField()), - ('vcs_branch', SingleLineField()), - ('vcs_revision', SingleLineField()), - - ('checksum_md5', SingleLineField()), - ('checksum_sha1', SingleLineField()), - ('checksum_sha256', SingleLineField()), - ('spec_version', SingleLineField()), - ]) - - for name, field in self.fields.items(): - # we could have a hack to get the actual field name - # but setting an attribute is explicit and cleaner - field.name = name - setattr(self, name, field) - - def __init__(self, location=None, about_file_path=None, strict=False): - """ - Create an instance. - If strict is True, raise an Exception on errors. Otherwise the errors - attribute contains the errors. - """ - self.set_standard_fields() - self.custom_fields = OrderedDict() - - self.errors = [] - - # about file path relative to the root of an inventory using posix - # path separators - self.about_file_path = about_file_path - - # os native absolute location, using posix path separators - self.location = location - self.base_dir = None - if self.location: - self.base_dir = os.path.dirname(location) - self.errors.extend(self.load(location)) - if strict and self.errors and filter_errors(self.errors): - msg = '\n'.join(map(str, self.errors)) - raise Exception(msg) - - def __repr__(self): - return repr(self.all_fields()) - - def __eq__(self, other): - """ - Equality based on fields and custom_fields., i.e. content. - """ - return (isinstance(other, self.__class__) - and self.fields == other.fields - and self.custom_fields == other.custom_fields) - - def all_fields(self): - """ - Return the list of all Field objects. - """ - return list(self.fields.values()) + list(self.custom_fields.values()) - - def as_dict(self): - """ - Return all the standard fields and customer-defined fields of this - About object in an ordered dict. - """ - data = OrderedDict() - data[self.ABOUT_FILE_PATH_ATTR] = self.about_file_path - with_values = ((fld.name, fld.serialized_value()) for fld in self.all_fields()) - non_empty = ((name, value) for name, value in with_values if value) - data.update(non_empty) - return data - - def hydrate(self, fields): - """ - Process an iterable of field (name, value) tuples. Update or create - Fields attributes and the fields and custom fields dictionaries. - Return a list of errors. - """ - errors = [] - seen_fields = OrderedDict() - - for name, value in fields: - orig_name = name - name = name.lower() - - # Some special attributes - if name == self.ABOUT_FILE_PATH_ATTR: - # this is a special attribute set directly on object - setattr(self, name, value) - continue - - if name == self.about_resource_path_attr: - # this is a special attribute, skip entirely - continue - - # A field that has been alredy processed ... and has a value - previous_value = seen_fields.get(name) - if previous_value: - if value != previous_value: - msg = (u'Field %(orig_name)s is a duplicate. ' - u'Original value: "%(previous_value)s" ' - u'replaced with: "%(value)s"') - errors.append(Error(WARNING, msg % locals())) - continue - - seen_fields[name] = value - - # A standard field (could be essential/required or not) - standard_field = self.fields.get(name) - if standard_field: - standard_field.original_value = value - standard_field.value = value - standard_field.present = True - continue - - # A custom field - # is the name valid? - illegal_name_error = validate_field_name(name) - if illegal_name_error: - errors.append(illegal_name_error) - continue - - msg = 'Field %(orig_name)s is a custom field.' - errors.append(Error(INFO, msg % locals())) - # is this a known one? - custom_field = self.custom_fields.get(name) - if custom_field: - # An known custom field - custom_field.original_value = value - custom_field.value = value - custom_field.present = True - else: - # A new, unknown custom field - # custom fields are always handled as StringFields - # FIXME: with yaml we could just set whatever is provided - custom_field = StringField(name=name, value=value, present=True) - self.custom_fields[name] = custom_field - # FIXME: why would this ever fail??? - try: - if name in dir(self): - raise Exception('Illegal field: %(name)r: %(value)r.' % locals()) - setattr(self, name, custom_field) - except: - msg = 'Internal error with custom field: %(name)r: %(value)r.' - errors.append(Error(CRITICAL, msg % locals())) - - return errors - - def process(self, fields, about_file_path, running_inventory=False, - base_dir=None, reference_dir=None): - """ - Validate and set as attributes on this About object a sequence of - `fields` name/value tuples. Return a list of errors. - """ - self.base_dir = base_dir - self.reference_dir = reference_dir - afp = self.about_file_path - - errors = self.hydrate(fields) - - # We want to copy the license_files before the validation - if reference_dir: - copy_license_notice_files( - fields, base_dir, reference_dir, afp) - - # TODO: why? we validate all fields, not only these hydrated - validation_errors = validate_fields( - self.all_fields(), - about_file_path, - running_inventory, - self.base_dir, - self.reference_dir) - errors.extend(validation_errors) - - return errors - - def load(self, location): - """ - Read, parse and process the ABOUT file at `location`. - Return a list of errors and update self with errors. - """ - self.location = location - loc = util.to_posix(location) - base_dir = posixpath.dirname(loc) - errors = [] - try: - loc = add_unc(loc) - with io.open(loc, encoding='utf-8') as txt: - input_text = txt.read() - # FIXME: this should be done in the commands, not here - """ - The running_inventory defines if the current process is 'inventory' or not. - This is used for the validation of the path of the 'about_resource'. - In the 'inventory' command, the code will use the parent of the about_file_path - location and join with the 'about_resource' for the validation. - On the other hand, in the 'gen' command, the code will use the - generated location (aka base_dir) along with the parent of the about_file_path - and then join with the 'about_resource' - """ - running_inventory = True - data = saneyaml.load(input_text, allow_duplicate_keys=False) - errs = self.load_dict(data, base_dir, running_inventory) - errors.extend(errs) - except Exception as e: - trace = traceback.format_exc() - msg = 'Cannot load invalid ABOUT file: %(location)r: %(e)r\n%(trace)s' - errors.append(Error(CRITICAL, msg % locals())) - - self.errors = errors - return errors - - # FIXME: should be a from_dict class factory instead - # FIXME: running_inventory: remove this : this should be done in the commands, not here - def load_dict(self, fields_dict, base_dir, running_inventory=False, reference_dir=None,): - """ - Load this About object file from a `fields_dict` name/value dict. - Return a list of errors. - """ - # do not keep empty - fields = list(fields_dict.items()) - - for key, value in fields: - if not value: - # never return empty or absent fieds - continue - - if key == u'licenses': - # FIXME: use a license object instead - lic_key, lic_name, lic_file, lic_url = ungroup_licenses(value) - if lic_key: - fields.append(('license_key', lic_key)) - if lic_name: - fields.append(('license_name', lic_name)) - if lic_file: - fields.append(('license_file', lic_file)) - if lic_url: - fields.append(('license_url', lic_url)) - # The licenses field has been ungrouped and can be removed. - # Otherwise, it will gives the following INFO level error - # 'Field licenses is a custom field.' - licenses_field = (key, value) - fields.remove(licenses_field) - errors = self.process( - fields=fields, - about_file_path=self.about_file_path, - running_inventory=running_inventory, - base_dir=base_dir, - reference_dir=reference_dir, - ) - self.errors = errors - return errors - - @classmethod - def from_dict(cls, about_data, base_dir=''): - """ - Return an About object loaded from a python dict. - """ - about = cls() - about.load_dict(about_data, base_dir=base_dir) - return about - - def dumps(self): - """ - Return self as a formatted ABOUT string. - """ - data = OrderedDict() - # Group the same license information (name, url, file) together - license_key = [] - license_name = [] - license_file = [] - license_url = [] - file_fields = ['about_resource', 'notice_file', 'changelog_file', 'author_file'] - bool_fields = ['redistribute', 'attribute', 'track_changes', 'modified'] - for field in self.all_fields(): - if not field.value: - continue - - if field.name == 'license_key' and field.value: - license_key = field.value - elif field.name == 'license_name' and field.value: - license_name = field.value - elif field.name == 'license_file' and field.value: - license_file = field.value.keys() - elif field.name == 'license_url' and field.value: - license_url = field.value - - # No multiple 'about_resource' reference supported. - # Take the first element (should only be one) in the list for the - # value of 'about_resource' - elif field.name in file_fields and field.value: - data[field.name] = list(field.value.keys())[0] - else: - if field.value or (field.name in bool_fields and not field.value == None): - data[field.name] = field.value - - # Group the same license information in a list - license_group = list(zip_longest(license_key, license_name, license_file, license_url)) - for lic_group in license_group: - lic_dict = OrderedDict() - if lic_group[0]: - lic_dict['key'] = lic_group[0] - if lic_group[1]: - lic_dict['name'] = lic_group[1] - if lic_group[2]: - lic_dict['file'] = lic_group[2] - if lic_group[3]: - lic_dict['url'] = lic_group[3] - data.setdefault('licenses', []).append(lic_dict) - - return saneyaml.dump(data) - - def dump(self, location): - """ - Write formatted ABOUT representation of self to location. - """ - loc = util.to_posix(location) - parent = posixpath.dirname(loc) - - if not posixpath.exists(parent): - os.makedirs(add_unc(parent)) - - about_file_path = loc - if not about_file_path.endswith('.ABOUT'): - # FIXME: we should not infer some location. - if about_file_path.endswith('/'): - about_file_path = util.to_posix( - os.path.join(parent, os.path.basename(parent))) - about_file_path += '.ABOUT' - - if on_windows: - about_file_path = add_unc(about_file_path) - - with io.open(about_file_path, mode='w', encoding='utf-8') as dumped: - dumped.write(self.dumps()) - - def dump_lic(self, location, license_dict): - """ - Write LICENSE files and return the a list of key, name, context and the url - as these information are needed for the ABOUT file - """ - license_name = license_context = license_url = '' - loc = util.to_posix(location) - parent = posixpath.dirname(loc) - license_key_name_context_url = [] - - if not posixpath.exists(parent): - os.makedirs(add_unc(parent)) - - if self.license_expression.present and not self.license_file.present: - special_char_in_expression, lic_list = parse_license_expression(self.license_expression.value) - self.license_key.value = lic_list - self.license_key.present = True - if not special_char_in_expression: - for lic_key in lic_list: - try: - if license_dict[lic_key]: - license_path = posixpath.join(parent, lic_key) - license_path += u'.LICENSE' - license_path = add_unc(license_path) - license_name, license_context, license_url = license_dict[lic_key] - license_info = (lic_key, license_name, license_context, license_url) - license_key_name_context_url.append(license_info) - with io.open(license_path, mode='w', encoding='utf-8', newline='\n') as lic: - lic.write(license_context) - except: - pass - return license_key_name_context_url - - -def collect_inventory(location): - """ - Collect ABOUT files at location and return a list of errors and a list of - About objects. - """ - errors = [] - input_location = util.get_absolute(location) - about_locations = list(util.get_about_locations(input_location)) - - name_errors = util.check_file_names(about_locations) - errors.extend(name_errors) - abouts = [] - for about_loc in about_locations: - about_file_path = util.get_relative_path(input_location, about_loc) - about = About(about_loc, about_file_path) - # Insert about_file_path reference to the error - for severity, message in about.errors: - msg = (about_file_path + ": " + message) - errors.append(Error(severity, msg)) - abouts.append(about) - - return unique(errors), abouts - - -def get_field_names(abouts): - """ - Given a list of About objects, return a list of any field names that exist - in any object, including custom fields. - """ - fields = [] - fields.append(About.ABOUT_FILE_PATH_ATTR) - - standard_fields = About().fields.keys() - standards = [] - for a in abouts: - for name, field in a.fields.items(): - if field.required: - if name not in standards: - standards.append(name) - else: - if field.present: - if name not in standards: - standards.append(name) - # resort standard fields in standard order - # which is a tad complex as this is a predefined order - sorted_std = [] - for fn in standard_fields: - if fn in standards: - sorted_std.append(fn) - fields.extend(sorted_std) - - customs = [] - for a in abouts: - for name, field in a.custom_fields.items(): - if field.has_content: - if name not in customs: - customs.append(name) - # always sort custom fields list by name - customs.sort() - fields.extend(customs) - - return fields - - -def about_object_to_list_of_dictionary(abouts): - """ - Convert About objects to a list of dictionaries - """ - serialized = [] - for about in abouts: - # TODO: this wholeblock should be under sd_dict() - ad = about.as_dict() - if 'about_file_path' in ad.keys(): - afp = ad['about_file_path'] - afp = '/' + afp if not afp.startswith('/') else afp - ad['about_file_path'] = afp - serialized.append(ad) - return serialized - - -def write_output(abouts, location, format): # NOQA - """ - Write a CSV/JSON file at location given a list of About objects. - Return a list of Error objects. - """ - about_dicts = about_object_to_list_of_dictionary(abouts) - location = add_unc(location) - if format == 'csv': - errors = save_as_csv(location, about_dicts, get_field_names(abouts)) - else: - errors = save_as_json(location, about_dicts) - return errors - - -def save_as_json(location, about_dicts): - mode = 'w' - if python2: - mode = 'wb' - with io.open(location, mode=mode) as output_file: - data = util.format_about_dict_for_json_output(about_dicts) - output_file.write(json.dumps(data, indent=2)) - return [] - - -def save_as_csv(location, about_dicts, field_names): - errors = [] - with io.open(location, mode='w', encoding='utf-8', newline='') as output_file: - writer = csv.DictWriter(output_file, field_names) - writer.writeheader() - csv_formatted_list = util.format_about_dict_for_csv_output(about_dicts) - for row in csv_formatted_list: - # See https://github.com/dejacode/about-code-tool/issues/167 - try: - writer.writerow(row) - except Exception as e: - msg = u'Generation skipped for ' + row['about_file_path'] + u' : ' + str(e) - errors.append(Error(CRITICAL, msg)) - return errors - - -def pre_process_and_fetch_license_dict(abouts, api_url, api_key): - """ - Modify a list of About data dictionaries by adding license information - fetched from the DejaCode API. - """ - dje_uri = urlparse(api_url) - domain = '{uri.scheme}://{uri.netloc}/'.format(uri=dje_uri) - dje_lic_urn = urljoin(domain, 'urn/?urn=urn:dje:license:') - key_text_dict = {} - captured_license = [] - errors = [] - if util.have_network_connection(): - if not valid_api_url(api_url): - msg = u"URL not reachable. Invalid '--api_url'. License generation is skipped." - errors.append(Error(ERROR, msg)) - else: - msg = u'Network problem. Please check your Internet connection. License generation is skipped.' - errors.append(Error(ERROR, msg)) - for about in abouts: - # No need to go through all the about objects for license extraction if we detected - # invalid '--api_key' - auth_error = Error(ERROR, u"Authorization denied. Invalid '--api_key'. License generation is skipped.") - if auth_error in errors: - break - if about.license_expression.present: - special_char_in_expression, lic_list = parse_license_expression(about.license_expression.value) - if special_char_in_expression: - msg = (u"The following character(s) cannot be in the licesne_expression: " + - str(special_char_in_expression)) - errors.append(Error(ERROR, msg)) - else: - for lic_key in lic_list: - if not lic_key in captured_license: - detail_list = [] - license_name, license_key, license_text, errs = api.get_license_details_from_api(api_url, api_key, lic_key) - for e in errs: - if e not in errors: - errors.append(e) - if license_key: - captured_license.append(lic_key) - dje_lic_url = dje_lic_urn + license_key - detail_list.append(license_name) - detail_list.append(license_text) - detail_list.append(dje_lic_url) - key_text_dict[license_key] = detail_list - return key_text_dict, errors - - -def parse_license_expression(lic_expression): - licensing = Licensing() - lic_list = [] - special_char = special_char_in_license_expresion(lic_expression) - if not special_char: - # Parse the license expression and save it into a list - lic_list = licensing.license_keys(lic_expression) - return special_char, lic_list - - -def special_char_in_license_expresion(lic_expression): - not_support_char = [ - '!', '@', '#', '$', '%', '^', '&', '*', '=', '{', '}', - '|', '[', ']', '\\', ':', ';', '<', '>', '?', ',', '/'] - special_character = [] - for char in not_support_char: - if char in lic_expression: - special_character.append(char) - return special_character - - -def valid_api_url(api_url): - try: - request = Request(api_url) - # This will always goes to exception as no key are provided. - # The purpose of this code is to validate the provided api_url is correct - urlopen(request) - except HTTPError as http_e: - # The 403 error code is refer to "Authentication credentials were not provided.". - # This is correct as no key are provided. - if http_e.code == 403: - return True - except: - # All other exceptions yield to invalid api_url - pass - return False diff --git a/src/attributecode/templates/default_html.template b/src/attributecode/templates/default_html.template deleted file mode 100644 index 386d20c7..00000000 --- a/src/attributecode/templates/default_html.template +++ /dev/null @@ -1,87 +0,0 @@ - - - - - Open Source Software Information - - - -

OPEN SOURCE SOFTWARE INFORMATION

-
-

Licenses, acknowledgments and required copyright notices for - open source components:

-
- -
- {% for about_object in abouts %} -

{{ about_object.name.value }}{% if about_object.version.value %} {{ about_object.version.value }}{% endif %}

- {% endfor %} -
- -
- - - {% for about_object in abouts %} -
-

{{ about_object.name.value }} - {% if about_object.version.value %}{{ about_object.version.value }}{% endif %} -

- {% if about_object.license_expression.value %} -

This component is licensed under - {{ about_object.license_expression.value }} - {% endif %} - {% if about_object.copyright.value %} - 

{{about_object.copyright.value}}
- {% endif %} - {% if about_object.notice_file.value %} - {% for notice in about_object.notice_file.value %} - 
{{ about_object.notice_file.value[notice] }}
- {% endfor %} - {% endif %} - {% if about_object.license_key.value %} - {% for license_key in about_object.license_key.value %} - {% if license_key in common_licenses %} -

Full text of - - {{ license_key }} - - is available at the end of this document.

- {% endif %} - {% endfor %} - {% if about_object.license_file.value %} - {% for lic_file_name in about_object.license_file.value %} - {% if not license_file_name_and_key[lic_file_name] in common_licenses %} - 
{{ about_object.license_file.value[lic_file_name] | e}}
- {% endif %} - {% endfor %} - {% endif %} - {% else %} - {% if about_object.license_file.value %} - {% for lic_file_name in about_object.license_file.value %} - 
{{ about_object.license_file.value[lic_file_name] | e}}
- {% endfor %} - {% endif %} - {% endif %} -
- {% endfor %} - -
- -

Common Licenses Used in This Product

- - {% for key in license_key_and_context %} - {% if key in common_licenses %} -

{{ key }}

- 
{{ license_key_and_context[key]|e }}
- {% endif %} - {% endfor %} - -

End

- This file was generated on: {{ utcnow }} (UTC) - - - diff --git a/src/attributecode/templates/default_json.template b/src/attributecode/templates/default_json.template deleted file mode 100644 index dc19a5a6..00000000 --- a/src/attributecode/templates/default_json.template +++ /dev/null @@ -1,18 +0,0 @@ -{ - "ossAttribution": { - "title": "Open Source Software Information", - "entries": [ - {% for about_object in abouts %} - { - "name": "{{ about_object.name.value }}"{% if about_object.version.value or about_object.license_expression.value-%},{%- endif %} - {% if about_object.version.value -%} - "version": "{{ about_object.version.value }}"{% if about_object.license_expressio.value-%},{%- endif %} - {%- endif %} - {% if about_object.license_expression.value -%} - "license_expression": "{{ about_object.license_expression.value }}" - {%- endif %} - }{% if not loop.last -%},{%- endif %} - {%- endfor %} - ] - } -} \ No newline at end of file diff --git a/src/attributecode/templates/list.csv b/src/attributecode/templates/list.csv deleted file mode 100644 index b576a9b4..00000000 --- a/src/attributecode/templates/list.csv +++ /dev/null @@ -1,4 +0,0 @@ -Name,Version,DejaCode License,Homepage -{% for about in abouts %} -"{{about.name}}","{{about.version}}","{{about.license_name}}","{{about.homepage_url}}" -{% endfor %} diff --git a/src/attributecode/util.py b/src/attributecode/util.py deleted file mode 100644 index 4a18109b..00000000 --- a/src/attributecode/util.py +++ /dev/null @@ -1,558 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf8 -*- -# ============================================================================ -# Copyright (c) 2013-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ============================================================================ - -from __future__ import absolute_import -from __future__ import print_function -from __future__ import unicode_literals - -import codecs -from collections import OrderedDict -import json -import ntpath -import os -import posixpath -import re -import shutil -import string -import sys - -from attributecode import CRITICAL -from attributecode import WARNING -from attributecode import Error - - -python2 = sys.version_info[0] < 3 - -if python2: # pragma: nocover - from itertools import izip_longest as zip_longest # NOQA -else: # pragma: nocover - from itertools import zip_longest # NOQA - -if python2: # pragma: nocover - from backports import csv # NOQA - # monkey patch backports.csv until bug is fixed - # https://github.com/ryanhiebert/backports.csv/issues/30 - csv.dict = OrderedDict -else: # pragma: nocover - import csv # NOQA - - -on_windows = 'win32' in sys.platform - - -def to_posix(path): - """ - Return a path using the posix path separator given a path that may contain - posix or windows separators, converting "\\" to "/". NB: this path will - still be valid in the windows explorer (except for a UNC or share name). It - will be a valid path everywhere in Python. It will not be valid for windows - command line operations. - """ - return path.replace(ntpath.sep, posixpath.sep) - - -UNC_PREFIX = u'\\\\?\\' -UNC_PREFIX_POSIX = to_posix(UNC_PREFIX) -UNC_PREFIXES = (UNC_PREFIX_POSIX, UNC_PREFIX,) - -valid_file_chars = string.digits + string.ascii_letters + '_-.' + ' ' - - -def invalid_chars(path): - """ - Return a list of invalid characters in the file name of `path`. - """ - path = to_posix(path) - rname = resource_name(path) - name = rname.lower() - return [c for c in name if c not in valid_file_chars] - - -def check_file_names(paths): - """ - Given a sequence of file paths, check that file names are valid and that - there are no case-insensitive duplicates in any given directories. - Return a list of errors. - - From spec : - A file name can contain only these US-ASCII characters: - - digits from 0 to 9 - - uppercase and lowercase letters from A to Z - - the _ underscore, - dash and . period signs. - From spec: - The case of a file name is not significant. On case-sensitive file - systems (such as Linux), a tool must raise an error if two ABOUT files - stored in the same directory have the same lowercase file name. - """ - # FIXME: this should be a defaultdicts that accumulates all duplicated paths - seen = {} - errors = [] - for orig_path in paths: - path = orig_path - invalid = invalid_chars(path) - if invalid: - invalid = ''.join(invalid) - msg = ('Invalid characters %(invalid)r in file name at: ' - '%(path)r' % locals()) - errors.append(Error(CRITICAL, msg)) - - path = to_posix(orig_path) - name = resource_name(path).lower() - parent = posixpath.dirname(path) - path = posixpath.join(parent, name) - path = posixpath.normpath(path) - path = posixpath.abspath(path) - existing = seen.get(path) - if existing: - msg = ('Duplicate files: %(orig_path)r and %(existing)r ' - 'have the same case-insensitive file name' % locals()) - errors.append(Error(CRITICAL, msg)) - else: - seen[path] = orig_path - return errors - - -# TODO: rename to normalize_path -def get_absolute(location): - """ - Return an absolute normalized location. - """ - location = os.path.expanduser(location) - location = os.path.expandvars(location) - location = os.path.normpath(location) - location = os.path.abspath(location) - return location - - -def get_locations(location): - """ - Return a list of locations of files given the `location` of a - a file or a directory tree containing ABOUT files. - File locations are normalized using posix path separators. - """ - location = add_unc(location) - location = get_absolute(location) - assert os.path.exists(location) - - if os.path.isfile(location): - yield location - else: - for base_dir, _, files in os.walk(location): - for name in files: - bd = to_posix(base_dir) - yield posixpath.join(bd, name) - - -def get_about_locations(location): - """ - Return a list of locations of ABOUT files given the `location` of a - a file or a directory tree containing ABOUT files. - File locations are normalized using posix path separators. - """ - for loc in get_locations(location): - if is_about_file(loc): - yield loc - - -def get_relative_path(base_loc, full_loc): - """ - Return a posix path for a given full location relative to a base location. - The first segment of the different between full_loc and base_loc will become - the first segment of the returned path. - """ - def norm(p): - if p.startswith(UNC_PREFIX) or p.startswith(to_posix(UNC_PREFIX)): - p = p.strip(UNC_PREFIX).strip(to_posix(UNC_PREFIX)) - p = to_posix(p) - p = p.strip(posixpath.sep) - p = posixpath.normpath(p) - return p - - base = norm(base_loc) - path = norm(full_loc) - - assert path.startswith(base), ('Cannot compute relative path: ' - '%(path)r does not start with %(base)r' - % locals()) - base_name = resource_name(base) - no_dir = base == base_name - same_loc = base == path - if same_loc: - # this is the case of a single file or single dir - if no_dir: - # we have no dir: the full path is the same as the resource name - relative = base_name - else: - # we have at least one dir - parent_dir = posixpath.dirname(base) - parent_dir = resource_name(parent_dir) - relative = posixpath.join(parent_dir, base_name) - else: - relative = path[len(base) + 1:] - # We don't want to keep the first segment of the root of the returned path. - # See https://github.com/nexB/attributecode/issues/276 - # relative = posixpath.join(base_name, relative) - return relative - - -def to_native(path): - """ - Return a path using the current OS path separator given a path that may - contain posix or windows separators, converting "/" to "\\" on windows - and "\\" to "/" on posix OSes. - """ - path = path.replace(ntpath.sep, os.path.sep) - path = path.replace(posixpath.sep, os.path.sep) - return path - - -def is_about_file(path): - """ - Return True if the path represents a valid ABOUT file name. - """ - if path: - path = path.lower() - return path.endswith('.about') and path != '.about' - - -def resource_name(path): - """ - Return the file or directory name from a path. - """ - path = path.strip() - path = to_posix(path) - path = path.rstrip(posixpath.sep) - _left, right = posixpath.split(path) - return right.strip() - - - -def load_csv(location): - """ - Read CSV at `location`, return a list of ordered dictionaries, one - for each row. - """ - results = [] - # FIXME: why ignore encoding errors here? - with codecs.open(location, mode='rb', encoding='utf-8', - errors='ignore') as csvfile: - for row in csv.DictReader(csvfile): - # convert all the column keys to lower case - updated_row = OrderedDict( - [(key.lower(), value) for key, value in row.items()] - ) - results.append(updated_row) - return results - - -def load_json(location): - """ - Read JSON file at `location` and return a list of ordered dicts, one for - each entry. - """ - # FIXME: IMHO we should know where the JSON is from and its shape - # FIXME use: object_pairs_hook=OrderedDict - with open(location) as json_file: - results = json.load(json_file) - - # If the loaded JSON is not a list, - # - JSON output from AboutCode Manager: - # look for the "components" field as it is the field - # that contain everything the tool needs and ignore other fields. - # For instance, - # { - # "aboutcode_manager_notice":"xyz", - # "aboutcode_manager_version":"xxx", - # "components": - # [{ - # "license_expression":"apache-2.0", - # "copyright":"Copyright (c) 2017 nexB Inc.", - # "path":"ScanCode", - # ... - # }] - # } - # - # - JSON output from ScanCode: - # look for the "files" field as it is the field - # that contain everything the tool needs and ignore other fields: - # For instance, - # { - # "scancode_notice":"xyz", - # "scancode_version":"xxx", - # "files": - # [{ - # "path": "test", - # "type": "directory", - # "name": "test", - # ... - # }] - # } - # - # - JSON file that is not produced by scancode or aboutcode toolkit - # For instance, - # { - # "path": "test", - # "type": "directory", - # "name": "test", - # ... - # } - # FIXME: this is too clever and complex... IMHO we should not try to guess the format. - # instead a command line option should be provided explictly to say what is the format - if isinstance(results, list): - results = sorted(results) - else: - if u'aboutcode_manager_notice' in results: - results = results['components'] - elif u'scancode_notice' in results: - results = results['files'] - else: - results = [results] - return results - - -# FIXME: rename to is_online: BUT do we really need this at all???? -def have_network_connection(): - """ - Return True if an HTTP connection to some public web site is possible. - """ - import socket - if python2: - import httplib # NOQA - else: - import http.client as httplib # NOQA - - http_connection = httplib.HTTPConnection('dejacode.org', timeout=10) # NOQA - try: - http_connection.connect() - except socket.error: - return False - else: - return True - - -def extract_zip(location): - """ - Extract a zip file at location in a temp directory and return the temporary - directory where the archive was extracted. - """ - import zipfile - import tempfile - - if not zipfile.is_zipfile(location): - raise Exception('Incorrect zip file %(location)r' % locals()) - - archive_base_name = os.path.basename(location).replace('.zip', '') - base_dir = tempfile.mkdtemp(prefix='aboutcode-toolkit-extract-') - target_dir = os.path.join(base_dir, archive_base_name) - target_dir = add_unc(target_dir) - os.makedirs(target_dir) - - if target_dir.endswith((ntpath.sep, posixpath.sep)): - target_dir = target_dir[:-1] - - with zipfile.ZipFile(location) as zipf: - for info in zipf.infolist(): - name = info.filename - content = zipf.read(name) - target = os.path.join(target_dir, name) - is_dir = target.endswith((ntpath.sep, posixpath.sep)) - if is_dir: - target = target[:-1] - parent = os.path.dirname(target) - if on_windows: - target = target.replace(posixpath.sep, ntpath.sep) - parent = parent.replace(posixpath.sep, ntpath.sep) - if not os.path.exists(parent): - os.makedirs(add_unc(parent)) - if not content and is_dir: - if not os.path.exists(target): - os.makedirs(add_unc(target)) - if not os.path.exists(target): - with open(target, 'wb') as f: - f.write(content) - return target_dir - - -def add_unc(location): - """ - Convert a `location` to an absolute Window UNC path to support long paths on - Windows. Return the location unchanged if not on Windows. See - https://msdn.microsoft.com/en-us/library/aa365247.aspx - """ - if on_windows and not location.startswith(UNC_PREFIX): - if location.startswith(UNC_PREFIX_POSIX): - return UNC_PREFIX + os.path.abspath(location.strip(UNC_PREFIX_POSIX)) - return UNC_PREFIX + os.path.abspath(location) - return location - - -# FIXME: add docstring -def copy_license_notice_files(fields, base_dir, reference_dir, afp): - """ - Given a list of (key, value) `fields` tuples and a `base_dir` where ABOUT - files and their companion LICENSe are store, and an extra `reference_dir` - where reference license an notice files are stored and the `afp` - about_file_path value, this function will copy to the base_dir the - license_file or notice_file if found in the reference_dir - - """ - lic_name = '' - for key, value in fields: - if key == 'license_file' or key == 'notice_file': - lic_name = value - - from_lic_path = posixpath.join(to_posix(reference_dir), lic_name) - about_file_dir = os.path.dirname(to_posix(afp)).lstrip('/') - to_lic_path = posixpath.join(to_posix(base_dir), about_file_dir) - - if on_windows: - from_lic_path = add_unc(from_lic_path) - to_lic_path = add_unc(to_lic_path) - - # Strip the white spaces - from_lic_path = from_lic_path.strip() - to_lic_path = to_lic_path.strip() - - # Errors will be captured when doing the validation - if not posixpath.exists(from_lic_path): - continue - - if not posixpath.exists(to_lic_path): - os.makedirs(to_lic_path) - try: - shutil.copy2(from_lic_path, to_lic_path) - except Exception as e: - print(repr(e)) - print('Cannot copy file at %(from_lic_path)r.' % locals()) - - -# FIXME: we should use a license object instead -def ungroup_licenses(licenses): - """ - Ungroup multiple licenses information - """ - lic_key = [] - lic_name = [] - lic_file = [] - lic_url = [] - for lic in licenses: - if 'key' in lic: - lic_key.append(lic['key']) - if 'name' in lic: - lic_name.append(lic['name']) - if 'file' in lic: - lic_file.append(lic['file']) - if 'url' in lic: - lic_url.append(lic['url']) - return lic_key, lic_name, lic_file, lic_url - - -# FIXME: add docstring -def format_about_dict_for_csv_output(about_dictionary_list): - csv_formatted_list = [] - file_fields = ['license_file', 'notice_file', 'changelog_file', 'author_file'] - for element in about_dictionary_list: - row_list = OrderedDict() - for key in element: - if element[key]: - if isinstance(element[key], list): - row_list[key] = u'\n'.join((element[key])) - elif key == u'about_resource' or key in file_fields: - row_list[key] = u'\n'.join((element[key].keys())) - else: - row_list[key] = element[key] - csv_formatted_list.append(row_list) - return csv_formatted_list - - -# FIXME: add docstring -def format_about_dict_for_json_output(about_dictionary_list): - licenses = ['license_key', 'license_name', 'license_file', 'license_url'] - file_fields = ['notice_file', 'changelog_file', 'author_file'] - json_formatted_list = [] - for element in about_dictionary_list: - row_list = OrderedDict() - # FIXME: aboid using parallel list... use an object instead - license_key = [] - license_name = [] - license_file = [] - license_url = [] - - for key in element: - if element[key]: - # The 'about_resource' is an ordered dict - if key == 'about_resource': - row_list[key] = list(element[key].keys())[0] - elif key in licenses: - if key == 'license_key': - license_key = element[key] - elif key == 'license_name': - license_name = element[key] - elif key == 'license_file': - license_file = element[key].keys() - elif key == 'license_url': - license_url = element[key] - elif key in file_fields: - row_list[key] = element[key].keys() - else: - row_list[key] = element[key] - - # Group the same license information in a list - license_group = list(zip_longest(license_key, license_name, license_file, license_url)) - if license_group: - licenses_list = [] - for lic_group in license_group: - lic_dict = OrderedDict() - if lic_group[0]: - lic_dict['key'] = lic_group[0] - if lic_group[1]: - lic_dict['name'] = lic_group[1] - if lic_group[2]: - lic_dict['file'] = lic_group[2] - if lic_group[3]: - lic_dict['url'] = lic_group[3] - licenses_list.append(lic_dict) - row_list['licenses'] = licenses_list - json_formatted_list.append(row_list) - return json_formatted_list - - -def unique(sequence): - """ - Return a list of unique items found in sequence. Preserve the original - sequence order. - For example: - >>> unique([1, 5, 3, 5]) - [1, 5, 3] - """ - deduped = [] - for item in sequence: - if item not in deduped: - deduped.append(item) - return deduped - - -def filter_errors(errors, minimum_severity=WARNING): - """ - Return a list of unique `errors` Error object filtering errors that have a - severity below `minimum_severity`. - """ - return unique([e for e in errors if e.severity >= minimum_severity]) - - -""" -Return True if a string s name is safe to use as an attribute name. -""" -is_valid_name = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*$').match diff --git a/tests/test_api.py b/tests/test_api.py index 60764879..522167a1 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -22,9 +22,10 @@ import mock -from attributecode import api -from attributecode import ERROR -from attributecode import Error +from aboutcode import api +from aboutcode import ERROR +from aboutcode import Error +from aboutcode.model import License class FakeResponse(object): @@ -49,14 +50,15 @@ def test_api_get_license_details_from_api(self, request_license_data): errors = [] request_license_data.return_value = license_data, errors - expected = ( - 'Apache License 2.0', - 'apache-2.0', - 'Apache License Version 2.0 ...', - []) - result = api.get_license_details_from_api( - api_url='api_url', api_key='api_key', license_key='license_key') - assert expected == result + expected = License( + key='apache-2.0', + name='Apache License 2.0', + text='Apache License Version 2.0 ...', + url=u'http://fake.url/urn/?urn=urn:dje:license:license_key') + + result, errors = api.get_license_details( + api_url='http://fake.url/', api_key='api_key', license_key='license_key') + assert expected.to_dict() == result.to_dict() @mock.patch.object(api, 'urlopen') def test_api_request_license_data_with_result(self, mock_data): @@ -78,5 +80,27 @@ def test_api_request_license_data_without_result(self, mock_data): mock_data.return_value = FakeResponse(response_content) license_data = api.request_license_data( api_url='http://fake.url/', api_key='api_key', license_key='apache-2.0') - expected = ({}, [Error(ERROR, "Invalid 'license': apache-2.0")]) + expected = ({}, [Error(ERROR, "Invalid license key: apache-2.0")]) assert expected == license_data + + + @mock.patch.object(api, 'urlopen') + def test_valid_api_url(self, mock_data): + mock_data.return_value = '' + assert api.valid_api_url('non_valid_url') is False + + @mock.patch('aboutcode.api.have_network_connection') + @mock.patch('aboutcode.api.valid_api_url') + def test_fetch_licenses(self, have_network_connection, valid_api_url): + have_network_connection.return_value = True + + valid_api_url.return_value = False + error_msg = ( + 'Network problem. Please check your Internet connection. ' + 'License retrieval is skipped.') + expected = ({}, [Error(ERROR, error_msg)]) + assert api.fetch_licenses([], '', '') == expected + + valid_api_url.return_value = True + expected = ({}, []) + assert api.fetch_licenses([], '', '') == expected diff --git a/tests/test_attrib.py b/tests/test_attrib.py index 9742c317..118abc94 100644 --- a/tests/test_attrib.py +++ b/tests/test_attrib.py @@ -22,41 +22,41 @@ import os import unittest -from testing_utils import get_test_loc +from aboutcode import attrib +from aboutcode import inv -from attributecode import attrib -from attributecode import model +from testing_utils import get_test_loc class TemplateTest(unittest.TestCase): - def test_check_template_simple_valid_returns_None(self): + def test_check_template_with_simple_valid_template_returns_None(self): expected = None assert expected == attrib.check_template('template_string') - def test_check_template_complex_valid_returns_None(self): + def test_check_template_with_complex_valid_template_returns_None(self): template = ''' - {% for about in abouts -%} - {{ about.name.value }}: {{ about.version.value }} - {% for res in about.about_resource.value -%} + {% for package in packages -%} + {{ package.name.value }}: {{ package.version.value }} + {% for res in package.about_resource.value -%} resource: {{ res }} {% endfor -%} {% endfor -%}''' expected = None assert expected == attrib.check_template(template) - def test_check_template_complex_invalid_returns_error(self): + def test_check_template_with_complex_invalid_template_returns_error(self): template = ''' - {% for about in abouts -%} - {{ about.name.value }}: {{ about.version.value }} - {% for res in about.about_ressdsdsdsdsdsdource.value -%} + {% for package in packages -%} + {{ package.name.value }}: {{ package.version.value }} + {% for res in package.about_ressdsdsdsdsdsdource.value -%} resource: {{] res }} {% endfor -%} {% endfor -%}''' expected = (5, "unexpected ']'") assert expected == attrib.check_template(template) - def test_check_template_invalid_return_error_lineno_and_message(self): + def test_check_template_with_invalid_template_return_error_lineno_and_message(self): expected = 1, "unexpected end of template, expected 'end of print statement'." assert expected == attrib.check_template('{{template_string') @@ -66,41 +66,46 @@ def test_check_template_all_builtin_templates_are_valid(self): template_loc = os.path.join(builtin_templates_dir, template) with io.open(template_loc, 'r', encoding='utf-8') as tmpl: template = tmpl.read() - try: - assert None == attrib.check_template(template) - except: - raise Exception(template_loc) + assert None == attrib.check_template(template) class GenerateTest(unittest.TestCase): - def test_generate_from_collected_inventory_wih_custom_temaplte(self): + def test_generate_from_collected_inventory_wih_custom_template(self): test_file = get_test_loc('test_attrib/gen_simple/attrib.ABOUT') - errors, abouts = model.collect_inventory(test_file) - assert not errors + errors, packages = inv.collect_inventory(test_file) + assert [] == errors test_template = get_test_loc('test_attrib/gen_simple/test.template') with open(test_template) as tmpl: - template = tmpl.read() + template_text = tmpl.read() expected = ( 'Apache HTTP Server: 2.4.3\n' 'resource: httpd-2.4.3.tar.gz\n') - error, result = attrib.generate(abouts, template) + error, result = attrib.create_attribution_text(packages, template_text) assert expected == result assert not error - def test_generate_with_default_template(self): - test_file = get_test_loc('test_attrib/gen_default_template/attrib.ABOUT') - errors, abouts = model.collect_inventory(test_file) + def test_generate_with_default_template(self, regen=False): + test_file = get_test_loc('test_attrib/gen_default_template') + errors, packages = inv.collect_inventory(test_file) assert not errors - error, result = attrib.generate_from_file(abouts) + test_template = attrib.DEFAULT_TEMPLATE_FILE + with open(test_template) as tmpl: + template_text = tmpl.read() + + error, result = attrib.create_attribution_text(packages, template_text) assert not error expected_file = get_test_loc( 'test_attrib/gen_default_template/expected_default_attrib.html') + if regen: + with io.open(expected_file, 'w') as out: + out.write(result) + with open(expected_file) as exp: expected = exp.read() diff --git a/tests/test_cmd.py b/tests/test_cmd.py index 36ae8c88..1c961315 100644 --- a/tests/test_cmd.py +++ b/tests/test_cmd.py @@ -21,14 +21,14 @@ import io import unittest -from attributecode import CRITICAL -from attributecode import DEBUG -from attributecode import ERROR -from attributecode import INFO -from attributecode import NOTSET -from attributecode import WARNING -from attributecode import cmd -from attributecode import Error +from aboutcode import CRITICAL +from aboutcode import DEBUG +from aboutcode import ERROR +from aboutcode import INFO +from aboutcode import NOTSET +from aboutcode import WARNING +from aboutcode import cmd +from aboutcode import Error from testing_utils import run_about_command_test_click from testing_utils import get_test_loc @@ -47,7 +47,7 @@ def test_report_errors(capsys): Error(DEBUG, 'msg4'), Error(NOTSET, 'msg4'), ] - ec = cmd.report_errors(errors, quiet=False, verbose=True, log_file_loc=None) + ec = cmd.report_errors(errors, verbose=True, log_file_loc=None) assert 3 == ec out, err = capsys.readouterr() expected_out = [ @@ -71,7 +71,7 @@ def test_report_errors_without_verbose(capsys): Error(DEBUG, 'msg4'), Error(NOTSET, 'msg4'), ] - ec = cmd.report_errors(errors, quiet=False, verbose=False, log_file_loc=None) + ec = cmd.report_errors(errors, verbose=False, log_file_loc=None) assert 3 == ec out, err = capsys.readouterr() expected_out = [ @@ -84,39 +84,6 @@ def test_report_errors_without_verbose(capsys): assert expected_out == out.splitlines(False) -def test_report_errors_with_quiet_ignores_verbose_flag(capsys): - errors = [ - Error(CRITICAL, 'msg1'), - Error(ERROR, 'msg2'), - Error(INFO, 'msg3'), - Error(WARNING, 'msg4'), - Error(DEBUG, 'msg4'), - Error(NOTSET, 'msg4'), - Error(WARNING, 'msg4'), - ] - severe_errors_count = cmd.report_errors(errors, quiet=True, verbose=True) - assert severe_errors_count == 3 - out, err = capsys.readouterr() - assert '' == out - assert '' == err - - -def test_report_errors_with_quiet_ignores_verbose_flag2(capsys): - errors = [ - Error(CRITICAL, 'msg1'), - Error(ERROR, 'msg2'), - Error(INFO, 'msg3'), - Error(WARNING, 'msg4'), - Error(DEBUG, 'msg4'), - Error(NOTSET, 'msg4'), - Error(WARNING, 'msg4'), - ] - severe_errors_count = cmd.report_errors(errors, quiet=True, verbose=False) - assert severe_errors_count == 3 - out, err = capsys.readouterr() - assert '' == out - assert '' == err - def test_report_errors_with_verbose_flag(capsys): errors = [ Error(CRITICAL, 'msg1'), @@ -127,7 +94,7 @@ def test_report_errors_with_verbose_flag(capsys): Error(NOTSET, 'msg4'), Error(WARNING, 'msg4'), ] - severe_errors_count = cmd.report_errors(errors, quiet=False, verbose=True) + severe_errors_count = cmd.report_errors(errors, verbose=True) assert severe_errors_count == 3 out, err = capsys.readouterr() expected_out = [ @@ -155,7 +122,7 @@ def test_report_errors_can_write_to_logfile(): ] result_file = get_temp_file() - _ec = cmd.report_errors(errors, quiet=False, verbose=True, + _ec = cmd.report_errors(errors, verbose=True, log_file_loc=result_file) with io.open(result_file, 'r', encoding='utf-8') as rf: result = rf.read() @@ -183,7 +150,7 @@ def test_report_errors_does_not_report_duplicate_errors(capsys): Error(WARNING, 'msg4'), Error(CRITICAL, 'msg1'), ] - severe_errors_count = cmd.report_errors(errors, quiet=True, verbose=True) + severe_errors_count = cmd.report_errors(errors, verbose=True) assert severe_errors_count == 3 @@ -208,22 +175,6 @@ def test_get_error_messages(): assert expected == emsgs -def test_get_error_messages_quiet(): - errors = [ - Error(CRITICAL, 'msg1'), - Error(ERROR, 'msg2'), - Error(INFO, 'msg3'), - Error(WARNING, 'msg4'), - Error(DEBUG, 'msg4'), - Error(NOTSET, 'msg4'), - ] - - emsgs, ec = cmd.get_error_messages(errors, quiet=True) - assert 3 == ec - expected = [] - assert expected == emsgs - - def test_get_error_messages_verbose(): errors = [ Error(CRITICAL, 'msg1'), @@ -307,12 +258,14 @@ def test_parse_key_values_simple(self): 'keY=bar', ] expected = { - 'key': ['value', 'bar'], - 'this': ['THat'] + 'key': 'value', + 'this': 'THat' } keyvals, errors = cmd.parse_key_values(test) assert expected == keyvals - assert not errors + + expected_errs = ['duplicated already defined: "keY=bar".'] + assert expected_errs == errors def test_parse_key_values_with_errors(self): @@ -323,7 +276,7 @@ def test_parse_key_values_with_errors(self): 'FOO=bar' ] expected = { - 'foo': ['bar'], + 'foo': 'bar', } keyvals, errors = cmd.parse_key_values(test) assert expected == keyvals @@ -374,6 +327,23 @@ def test_about_gen_help_text(): 'test_cmd/help/about_gen_help.txt', regen=False) +def test_about_fetch_licenses_help_text(): + check_about_stdout( + ['fetch-licenses', '--help'], + 'test_cmd/help/about_fetch_licenses_help.txt', regen=False) + + +def test_about_transform_help_text(): + check_about_stdout( + ['transform', '--help'], + 'test_cmd/help/about_transform_help.txt', regen=False) + + +def test_about_transform_expanded_help_text(): + check_about_stdout( + ['transform', '--help-format'], + 'test_cmd/help/about_transform_config_help.txt', regen=False) + def test_about_check_help_text(): check_about_stdout( ['check', '--help'], @@ -385,6 +355,10 @@ def test_about_attrib_help_text(): ['attrib', '--help'], 'test_cmd/help/about_attrib_help.txt', regen=False) +def test_about_reformat_help_text(): + check_about_stdout( + ['reformat', '--help'], + 'test_cmd/help/about_reformat_help.txt', regen=False) def test_about_command_fails_with_an_unknown_subcommand(): test_dir = get_temp_dir() @@ -414,15 +388,3 @@ def test_about_transform_command_can_run_minimally_without_error(): test_file = get_test_loc('test_cmd/transform.csv') result = get_temp_file('file_name.csv') run_about_command_test_click(['transform', test_file, result]) - - -def test_about_transform_help_text(): - check_about_stdout( - ['transform', '--help'], - 'test_cmd/help/about_transform_help.txt', regen=False) - - -def test_about_transform_expanded_help_text(): - check_about_stdout( - ['transform', '--help-format'], - 'test_cmd/help/about_transform_config_help.txt', regen=False) diff --git a/tests/test_gen.py b/tests/test_gen.py index 583cd0eb..efee150e 100644 --- a/tests/test_gen.py +++ b/tests/test_gen.py @@ -19,183 +19,381 @@ from __future__ import unicode_literals from collections import OrderedDict +import os import unittest +from aboutcode import ERROR +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import gen +from aboutcode import model + +from testing_utils import check_json from testing_utils import get_temp_dir from testing_utils import get_test_loc -from attributecode import ERROR -from attributecode import INFO -from attributecode import CRITICAL -from attributecode import Error -from attributecode import gen -from unittest.case import skip - class GenTest(unittest.TestCase): - def test_check_duplicated_columns(self): - test_file = get_test_loc('test_gen/dup_keys.csv') - expected = [Error(ERROR, 'Duplicated column name(s): copyright with copyright\nPlease correct the input and re-run.')] - result = gen.check_duplicated_columns(test_file) - assert expected == result - def test_check_duplicated_columns_handles_lower_upper_case(self): - test_file = get_test_loc('test_gen/dup_keys_with_diff_case.csv') - expected = [Error(ERROR, 'Duplicated column name(s): copyright with Copyright\nPlease correct the input and re-run.')] - result = gen.check_duplicated_columns(test_file) + def test_load_inventory_base(self): + location = get_test_loc('test_gen/inv_simple.csv') + target_dir = get_temp_dir() + errors, packages = gen.load_inventory(location, target_dir) + + expected_errors = [] + assert expected_errors == errors + + expected = [OrderedDict([ + ('about_file_path', u'inv/about.zip.ABOUT'), + ('about_resource', u'about.zip'), + ('name', u'AboutCode'), + ('version', u'0.11.0'), + ('description', u'multi\nline'), + (u'custom1', u'multi\nline') + ])] + result = [a.to_dict(with_path=True) for a in packages] assert expected == result - def test_check_duplicated_about_file_path(self): - test_dict = [ - {'about_file_path': '/test/test.c', 'version': '1.03', 'name': 'test.c'}, - {'about_file_path': '/test/abc/', 'version': '1.0', 'name': 'abc'}, - {'about_file_path': '/test/test.c', 'version': '1.04', 'name': 'test1.c'}] + def test_load_inventory_with_errors(self): + location = get_test_loc('test_gen/inv4.csv') + target_dir = get_temp_dir() + errors, packages = gen.load_inventory(location, target_dir) + expected_errors = [ + Error(CRITICAL, 'Invalid fields: all field names must be lowercase.'), + Error(CRITICAL, 'Custom field name: \'Confirmed Copyright\' contains illegal characters. ' + 'Only these characters are allowed: ASCII letters, digits and "_" underscore. ' + 'The first character must be a letter.'), + Error(CRITICAL, "Custom field name: 'Confirmed Copyright' must be lowercase."), + Error(CRITICAL, "Custom field name: 'Resource' must be lowercase.") + ] + assert expected_errors == errors + assert [] == packages + + def test_generate_about_files_fails_to_generate_if_one_dir_endswith_space(self): + location = get_test_loc('test_gen/inventory/complex/about_file_path_dir_endswith_space.csv') + target_dir = get_temp_dir() + errors, _packages = gen.generate_about_files(location, target_dir) + expected = [ + Error(ERROR, 'Skipping invalid path to create an ABOUT file: a path segment ' + 'cannot start or end with a space: "about /about.ABOUT"')] + assert expected == errors + + def test_generate_about_files_with_about_file_path_as_directory_generate_about_file_name(self): + location = get_test_loc('test_gen/inv2.csv') + target_dir = get_temp_dir() + errors, packages = gen.generate_about_files(location, target_dir) + expected = [] + assert expected == errors + + expected = [OrderedDict([('about_resource', u'.'), ('name', u'AboutCode'), ('version', u'0.11.0')])] + assert expected == [a.to_dict() for a in packages] + + generated_about_loc = packages[0].about_file_location + assert generated_about_loc.endswith('ABOUT') + about_file = model.Package.load(generated_about_loc) + expected = OrderedDict([('about_resource', u'.'), ('name', u'AboutCode'), ('version', u'0.11.0')]) + assert expected == about_file.to_dict() + + def test_generate_about_files_is_empty_and_has_errors_if_about_resource_reference_missing(self): + location = get_test_loc('test_gen/inv3.csv') + target_dir = get_temp_dir() + + errors, packages = gen.generate_about_files(location, target_dir) + expected = [ + Error(CRITICAL, 'Required field "about_resource" is missing.') + ] + assert expected == errors + assert [] == packages + + def test_generate_about_files_is_partial_and_has_errors_if_some_about_resource_reference_missing(self): + location = get_test_loc('test_gen/inv_with_some_about_resource_missing.csv') + target_dir = get_temp_dir() + + errors, packages = gen.generate_about_files(location, target_dir) expected = [ Error(CRITICAL, - "The input has duplicated values in 'about_file_path' field: /test/test.c")] - result = gen.check_duplicated_about_file_path(test_dict) + 'Required field "about_resource" is missing in row: 1.') +# 'Cannot create .ABOUT file for: "inv/test.tar.gz.ABOUT".\n' +# 'Required field "about_resource" is missing.') + ] + assert expected == errors + expected = [OrderedDict([('about_resource', u'My.gz'), ('name', u'AboutCode'), ('version', u'0.11.0')])] + assert expected == [a.to_dict() for a in packages] + assert 1 == len(os.listdir(target_dir)) + + def test_generate_about_files_simple(self): + location = get_test_loc('test_gen/inv_simple.csv') + target_dir = get_temp_dir() + + errors, packages = gen.generate_about_files(location, target_dir) + assert [] == errors + + result = [a.to_dict(with_path=True) for a in packages] + expected = [OrderedDict([ + ('about_file_path', 'inv/about.zip.ABOUT'), + ('about_resource', 'about.zip'), + ('name', 'AboutCode'), + ('version', '0.11.0'), + ('description', 'multi\nline'), + ('custom1', 'multi\nline')]) + ] + assert expected == result + + generated = os.listdir(os.path.join(target_dir, 'inv')) + expected = ['about.zip.ABOUT'] + assert expected == generated + + def test_generate_about_files_with_dot_about_resource_return_errors(self): + location = get_test_loc('test_gen/inv_with_dot.csv') + target_dir = get_temp_dir() + + errors, packages = gen.generate_about_files(location, target_dir) + expected = [ + Error(ERROR, 'Skipping invalid "about_resource". Path cannot be a ' + 'single "." (period) without an "about_file_path"') + ] + assert expected == errors + + assert [] == packages + + generated = os.listdir(target_dir) + assert [] == generated + + def test_generate_about_files_simple_with_afp(self): + location = get_test_loc('test_gen/inv_with_afp.csv') + target_dir = get_temp_dir() + + errors, packages = gen.generate_about_files(location, target_dir) + assert [] == errors + + result = [a.to_dict() for a in packages] + expected = [OrderedDict([ + ('about_resource', u'.'), + ('name', u'AboutCode'), + ('version', u'0.11.0'), + ('description', u'multi\nline'), + (u'custom1', u'multi\nline')]) + ] assert expected == result - def test_load_inventory(self): - location = get_test_loc('test_gen/inv.csv') - base_dir = get_temp_dir() - errors, abouts = gen.load_inventory(location, base_dir) + generated = os.listdir(os.path.join(target_dir, 'inv')) + expected = ['this.ABOUT'] + assert expected == generated + + def test_generate_about_files_reuses_licenses_from_reference_dir(self): + inventory_location = get_test_loc('test_gen/inv-with-complex-expression-and-notice.csv') + target_dir = get_temp_dir() + reference_dir = get_test_loc('test_gen/reference') + + errors, packages = gen.generate_about_files(inventory_location, target_dir, reference_dir) + expected_errors = [] + assert expected_errors == errors + + result = [a.to_dict() for a in packages] + expected = get_test_loc('test_gen/reference-expected.json') + check_json(expected, result) + + generated_files = os.listdir(os.path.join(target_dir, 'inv')) + expected = [ + 'bsd-new.LICENSE', + 'gpl-2.0.LICENSE', + 'mit.LICENSE', + 'that.ABOUT', + 'that.NOTICE', + 'this.ABOUT', + 'this.NOTICE', + ] + assert expected == sorted(generated_files) + + def test_generate_about_files_does_not_require_about_file_path(self): + inventory_location = get_test_loc('test_gen/inv_no_afp.csv') + target_dir = get_temp_dir() + reference_dir = get_test_loc('test_gen/reference') + errors, packages = gen.generate_about_files(inventory_location, target_dir, reference_dir) + expected_errors = [] + assert expected_errors == errors + + result = [a.to_dict() for a in packages] + expected = get_test_loc('test_gen/inv_no_afp-expected.json') + check_json(expected, result) + + generated_files1 = os.listdir(os.path.join(target_dir, 'this')) + expected = ['aboutcode.ABOUT', 'bsd-new.LICENSE'] + assert expected == sorted(generated_files1) + + generated_files2 = os.listdir(os.path.join(target_dir, 'that')) + expected = ['bsd-new.LICENSE', 'commons-log.jar.ABOUT', 'mit.LICENSE'] + assert expected == sorted(generated_files2) + + def test_generate_about_files_skip_files_with_spaces(self): + inventory_location = get_test_loc('test_gen/inv_with_spaces.csv') + target_dir = get_temp_dir() + reference_dir = get_test_loc('test_gen/reference') + + errors, packages = gen.generate_about_files(inventory_location, target_dir, reference_dir) expected_errors = [ - Error(INFO, 'Field custom1 is a custom field.'), - Error(INFO, 'Field about_resource: Path') + Error(ERROR, 'Skipping invalid path to create an ABOUT file: a path segment cannot start or end with a space: "that/ commons-log.jar"'), + Error(ERROR, 'Skipping invalid path to create an ABOUT file: a path segment cannot start or end with a space: "that/commons-log.jar "'), + Error(ERROR, 'Skipping invalid path to create an ABOUT file: a path segment cannot start or end with a space: "that /commons-log.jar"'), + Error(ERROR, 'Skipping invalid path to create an ABOUT file: a path segment cannot start or end with a space: " that/commons-log.jar"') ] - for exp, err in zip(expected_errors, errors): - assert exp.severity == err.severity - assert err.message.startswith(exp.message) - - expected = ( -'''about_resource: . -name: AboutCode -version: 0.11.0 -description: | - multi - line -custom1: | - multi - line -''' - ) - result = [a.dumps() for a in abouts] - assert expected == result[0] + assert expected_errors == errors + assert not packages - def test_load_inventory_with_errors(self): - location = get_test_loc('test_gen/inv4.csv') - base_dir = get_temp_dir() - errors, abouts = gen.load_inventory(location, base_dir) + def test_generate_about_files_skip_files_with_non_posix_about_resource(self): + inventory_location = get_test_loc('test_gen/inv_with_not_posix.csv') + target_dir = get_temp_dir() + reference_dir = get_test_loc('test_gen/reference') + errors, packages = gen.generate_about_files(inventory_location, target_dir, reference_dir) expected_errors = [ - Error(CRITICAL, "Field name: 'confirmed copyright' contains illegal name characters: 0 to 9, a to z, A to Z and _."), - Error(INFO, 'Field resource is a custom field.'), - Error(INFO, 'Field test is a custom field.'), - Error(INFO, 'Field about_resource: Path') + Error(ERROR, 'Skipping invalid "about_resource". Path must be a POSIX path ' + 'using "/" (slash) as separator: "this\\aboutcode"') + ] + assert expected_errors == errors + assert not packages + + +class TestJson(unittest.TestCase): + + def test_load_json(self): + test_file = get_test_loc('test_gen/json/expected.json') + expected = [OrderedDict([ + ('about_file_path', '/load/this.ABOUT'), + ('about_resource', '.'), + ('name', 'AboutCode'), + ('version', '0.11.0')]) ] - # assert [] == errors - for exp, err in zip(expected_errors, errors): - assert exp.severity == err.severity - assert err.message.startswith(exp.message) - - expected = ( - 'about_resource: .\n' - 'name: AboutCode\n' - 'version: 0.11.0\n' - 'description: |\n' - ' multi\n' - ' line\n' - # 'confirmed copyright: Copyright (c) nexB, Inc.\n' - 'resource: this.ABOUT\n' - 'test: This is a test\n' - ) - result = [a.dumps() for a in abouts] - assert expected == result[0] - - def test_generation_dir_endswith_space(self): - location = get_test_loc('test_gen/inventory/complex/about_file_path_dir_endswith_space.csv') - base_dir = get_temp_dir() - errors, _abouts = gen.generate(location, base_dir) - expected_errors_msg1 = 'contains directory name ends with spaces which is not allowed. Generation skipped.' - expected_errors_msg2 = 'Field about_resource' - assert errors - assert len(errors) == 2 - assert expected_errors_msg1 in errors[0].message or expected_errors_msg1 in errors[1].message - assert expected_errors_msg2 in errors[0].message or expected_errors_msg2 in errors[1].message - - def test_generation_with_no_about_resource(self): - location = get_test_loc('test_gen/inv2.csv') - base_dir = get_temp_dir() - errors, abouts = gen.generate(location, base_dir) - expected = OrderedDict([('.', None)]) - assert abouts[0].about_resource.value == expected - assert len(errors) == 1 + result = gen.load_json(test_file) + assert expected == result - def test_generation_with_no_about_resource_reference(self): - location = get_test_loc('test_gen/inv3.csv') - base_dir = get_temp_dir() + def test_load_json2(self): + test_file = get_test_loc('test_gen/json/expected_need_mapping.json') + expected = [dict(OrderedDict([ + ('about_file', '/load/this.ABOUT'), + ('about_resource', '.'), + ('version', '0.11.0'), + ('name', 'AboutCode'), + ]) + )] + result = gen.load_json(test_file) + assert expected == result + + def test_load_non_list_json(self): + test_file = get_test_loc('test_gen/json/not_a_list_need_mapping.json') + # FIXME: why this dict nesting?? + expected = [dict(OrderedDict([ + ('about_resource', '.'), + ('name', 'AboutCode'), + ('path', '/load/this.ABOUT'), + ('version', '0.11.0'), + ]) + )] + result = gen.load_json(test_file) + assert expected == result + + def test_load_non_list_json2(self): + test_file = get_test_loc('test_gen/json/not_a_list.json') + expected = [OrderedDict([ + ('about_file_path', '/load/this.ABOUT'), + ('about_resource', '.'), + ('name', 'AboutCode'), + ('version', '0.11.0'), + ]) + ] + result = gen.load_json(test_file) + assert expected == result - errors, abouts = gen.generate(location, base_dir) - expected = OrderedDict([('test.tar.gz', None)]) + def test_load_json_from_abc_mgr(self): + test_file = get_test_loc('test_gen/json/aboutcode_manager_exported.json') + expected = [dict(OrderedDict([ + ('license_expression', 'apache-2.0'), + ('copyright', 'Copyright (c) 2017 nexB Inc.'), + ('licenses', [{'key':'apache-2.0'}]), + ('copyrights', [{'statements':['Copyright (c) 2017 nexB Inc.']}]), + ('path', 'ScanCode'), + ('review_status', 'Analyzed'), + ('name', 'ScanCode'), + ('version', '2.2.1'), + ('owner', 'nexB Inc.'), + ('code_type', 'Source'), + ('is_modified', False), + ('is_deployed', False), + ('feature', ''), + ('purpose', ''), + ('homepage_url', None), + ('download_url', None), + ('license_url', None), + ('notice_url', None), + ('programming_language', 'Python'), + ('notes', ''), + ('fileId', 8458), + ]))] + result = gen.load_json(test_file) + assert expected == result - assert abouts[0].about_resource.value == expected - assert len(errors) == 1 - msg = 'Field about_resource' - assert msg in errors[0].message + def test_load_json_from_scancode(self): + test_file = get_test_loc('test_gen/json/scancode_info.json') + expected = [dict(OrderedDict([ + ('type', 'file'), + ('name', 'Api.java'), + ('path', 'Api.java'), + ('base_name', 'Api'), + ('extension', '.java'), + ('size', 5074), + ('date', '2017-07-15'), + ('sha1', 'c3a48ec7e684a35417241dd59507ec61702c508c'), + ('md5', '326fb262bbb9c2ce32179f0450e24601'), + ('mime_type', 'text/plain'), + ('file_type', 'ASCII text'), + ('programming_language', 'Java'), + ('is_binary', False), + ('is_text', True), + ('is_archive', False), + ('is_media', False), + ('is_source', True), + ('is_script', False), + ('files_count', 0), + ('dirs_count', 0), + ('size_count', 0), + ('scan_errors', []), + ]))] + result = gen.load_json(test_file) + assert expected == result - def test_generation_with_no_about_resource_reference_no_resource_validation(self): - location = get_test_loc('test_gen/inv3.csv') - base_dir = get_temp_dir() - - errors, abouts = gen.generate(location, base_dir) - expected = OrderedDict([('test.tar.gz', None)]) - - assert abouts[0].about_resource.value == expected - assert len(errors) == 1 - - def test_generate(self): - location = get_test_loc('test_gen/inv.csv') - base_dir = get_temp_dir() - - errors, abouts = gen.generate(location, base_dir) - msg1 = 'Field custom1 is a custom field.' - msg2 = 'Field about_resource' - - assert msg1 in errors[0].message - assert msg2 in errors[1].message - - result = [a.dumps() for a in abouts][0] - expected = ( -'''about_resource: . -name: AboutCode -version: 0.11.0 -description: | - multi - line -custom1: | - multi - line -''' - ) + +class TestCsv(unittest.TestCase): + + def test_load_csv_without_mapping(self): + test_file = get_test_loc('test_gen/csv/about.csv') + expected = [OrderedDict([ + ('about_file', 'about.ABOUT'), + ('about_resource', '.'), + ('name', 'ABOUT tool'), + ('version', '0.8.1') + ])] + result = list(gen.load_csv(test_file)) + assert expected == result + + def test_load_csv_load_rows(self): + test_file = get_test_loc('test_gen/csv/about.csv') + expected = [OrderedDict([ + ('about_file', 'about.ABOUT'), + ('about_resource', '.'), + ('name', 'ABOUT tool'), + ('version', '0.8.1') + ])] + result = list(gen.load_csv(test_file)) assert expected == result - @skip('FIXME: this test is making a failed, live API call') - def test_generate_not_overwrite_original_license_file(self): - location = get_test_loc('test_gen/inv5.csv') - base_dir = get_temp_dir() - reference_dir = None - fetch_license = ['url', 'lic_key'] - - _errors, abouts = gen.generate( - location, base_dir, reference_dir, fetch_license) - - result = [a.dumps()for a in abouts][0] - expected = ( - 'about_resource: .\n' - 'name: AboutCode\n' - 'version: 0.11.0\n' - 'licenses:\n' - ' - file: this.LICENSE\n') + def test_load_csv_does_not_convert_column_names_to_lowercase(self): + test_file = get_test_loc('test_gen/csv/about_key_with_upper_case.csv') + expected = [OrderedDict([ + ('about_file', 'about.ABOUT'), + ('about_resource', '.'), + ('nAme', 'ABOUT tool'), + ('Version', '0.8.1') + ])] + result = list(gen.load_csv(test_file)) assert expected == result diff --git a/tests/test_inv.py b/tests/test_inv.py new file mode 100644 index 00000000..1b5e9d98 --- /dev/null +++ b/tests/test_inv.py @@ -0,0 +1,369 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2014-2018 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +from collections import OrderedDict +import io +import shutil +import sys +import unittest +from unittest.case import skipIf + +from aboutcode import CRITICAL +from aboutcode import INFO +from aboutcode import Error +from aboutcode import inv +from aboutcode import model +from aboutcode.util import csv +from aboutcode.util import on_windows +from aboutcode.util import to_posix + +from testing_utils import check_json +from testing_utils import extract_test_loc +from testing_utils import get_temp_file +from testing_utils import get_test_loc + +try: + # Python 2 + unicode # NOQA +except NameError: # pragma: nocover + # Python 3 + unicode = str # NOQA + + +py3 = sys.version_info[0] == 3 + + +def load_csv(location): + """ + Read CSV at `location` and yield an ordered mapping for each row. + """ + with io.open(location, encoding='utf-8') as csvfile: + for row in csv.DictReader(csvfile): + yield row + + +def check_csv(expected, result, regen=False): + """ + Assert that the contents of two CSV files locations `expected` and + `result` are equal. + """ + if regen: + shutil.copyfile(result, expected) + expected = sorted([sorted(d.items()) for d in load_csv(expected)]) + result = [d.items() for d in load_csv(result)] + result = sorted(sorted(items) for items in result) + + assert expected == result + + +def get_test_content(test_location): + """ + Read file at test_location and return a unicode string. + """ + return get_unicode_content(get_test_loc(test_location)) + + +def get_unicode_content(location): + """ + Read file at location and return a unicode string. + """ + with io.open(location, encoding='utf-8') as doc: + return doc.read() + + +def fix_location(packages, test_dir): + """ + Fix the package.about_file_location by removing the `test_dir` from the path. + """ + for a in packages: + loc = a.about_file_location.replace(test_dir, '').strip('/\\') + a.about_file_location = to_posix(loc) + + +class InventoryTest(unittest.TestCase): + + def test_collect_inventory_return_errors(self): + test_loc = get_test_loc('test_inv/collect_inventory_errors') + errors, _packages = inv.collect_inventory(test_loc) + expected_errors = [] + assert expected_errors == errors + + @skipIf(on_windows and not py3, 'Windows support for long path requires https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation') + def test_collect_inventory_with_long_path(self): + test_loc = extract_test_loc('test_inv/longpath.zip') + _errors, packages = inv.collect_inventory(test_loc) + + expected_paths = [ + 'longpath/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/non-supported_date_format.ABOUT', + 'longpath/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/supported_date_format.ABOUT' + ] + fix_location(packages, test_loc) + + assert sorted(expected_paths) == sorted([a.about_file_location for a in packages]) + + expected_name = ['distribute', 'date_test'] + result_name = [a.name for a in packages] + assert sorted(expected_name) == sorted(result_name) + + def test_collect_inventory_can_collect_a_single_file(self): + test_loc = get_test_loc('test_inv/single_file/django_snippets_2413.ABOUT') + errors, packages = inv.collect_inventory(test_loc) + expected = [] + assert expected == errors + expected_loc = get_test_loc('test_inv/single_file/django_snippets_2413.ABOUT-expected.json') + result = [a.to_dict(with_path=True) for a in packages] + check_json(expected_loc, result, regen=False) + + def test_collect_inventory_return_no_warnings_and_model_can_use_relative_paths(self): + test_loc = get_test_loc('test_inv/rel/allAboutInOneDir') + errors, _packages = inv.collect_inventory(test_loc) + expected_errors = [] + result = [(e.severity, e.message) for e in errors if e.severity > INFO] + assert expected_errors == result + + def test_collect_inventory_populate_about_file_path(self): + test_loc = get_test_loc('test_inv/complete') + errors, packages = inv.collect_inventory(test_loc) + expected = [] + assert expected == errors + + expected = get_test_loc('test_inv/complete-expected.json') + result = [a.to_dict(with_path=True) for a in packages] + check_json(expected, result) + + def test_collect_inventory_with_multi_line(self): + test_loc = get_test_loc('test_inv/multi_line_license_expression.ABOUT') + errors, packages = inv.collect_inventory(test_loc) + assert [] == errors + expected = [ + 'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:mit', + 'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:apache-2.0'] + results = [l.url for l in packages[0].licenses] + assert expected == results + + assert 'mit OR apache-2.0' == packages[0].license_expression + + def test_collect_inventory_always_collects_custom_fields(self): + test_loc = get_test_loc('test_inv/custom_fields.ABOUT') + errors, packages = inv.collect_inventory(test_loc) + expected = [] + assert expected == errors + assert {'custom_mapping': 'test', 'resource': '.'} == packages[0].custom_fields + + def test_collect_inventory_does_not_raise_error_and_maintains_order_on_custom_fields(self): + test_loc = get_test_loc('test_inv/custom_fields2.ABOUT') + errors, packages = inv.collect_inventory(test_loc) + expected_errors = [] + assert expected_errors == errors + + expected = [OrderedDict([ + ('about_resource', u'.'), + ('name', u'test'), + (u'custom_mapping', u'test'), + (u'resource', u'.')])] + assert expected == [a.to_dict() for a in packages] + + def test_collect_inventory_works_with_relative_paths(self): + # FIXME: This test need to be run under src/aboutcode/ + # or otherwise it will fail as the test depends on the launching + # location + test_loc = get_test_loc('test_inv/relative') + # Use '.' as the indication of the current directory + test_loc1 = test_loc + '/./' + # Use '..' to go back to the parent directory + test_loc2 = test_loc + '/../relative' + errors1, packages1 = inv.collect_inventory(test_loc1) + assert [] == errors1 + expected = get_test_loc('test_inv/relative-1-expected.json') + result = [a.to_dict() for a in packages1] + check_json(expected, result) + + errors2, packages2 = inv.collect_inventory(test_loc2) + assert [] == errors2 + expected = get_test_loc('test_inv/relative-2-expected.json') + result = [a.to_dict() for a in packages2] + check_json(expected, result) + + def test_collect_inventory_basic_from_directory(self): + test_dir = get_test_loc('test_inv/basic') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + + inv.save_as_csv(result_file, packages) + assert [] == errors + + expected = get_test_loc('test_inv/basic/expected.csv') + check_csv(expected, result_file) + + def test_collect_inventory_with_about_resource_path_from_directory(self): + test_dir = get_test_loc('test_inv/basic_with_about_resource_path') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + + inv.save_as_csv(result_file, packages) + expected_errors = [] + assert expected_errors == errors + expected = get_test_loc('test_inv/basic_with_about_resource_path/expected.csv') + check_csv(expected, result_file) + + def test_collect_inventory_is_empty_when_about_resource_is_missing(self): + test_dir = get_test_loc('test_inv/no_about_resource_key') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + + inv.save_as_csv(result_file, packages) + + expected_errors = [ + Error(CRITICAL, + 'Required field "about_resource" is missing.', + path='about/about.ABOUT')] + assert expected_errors == errors + + expected = get_test_loc('test_inv/no_about_resource_key/expected.csv') + check_csv(expected, result_file, regen=False) + + def test_collect_inventory_contains_only_about_with_about_resource(self): + test_dir = get_test_loc('test_inv/some_missing_about_resource') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + fix_location(packages, test_dir) + + inv.save_as_csv(result_file, packages) + + expected_errors = [ + Error(CRITICAL, + 'Required field "about_resource" is missing.', + path='about/about.ABOUT')] + assert expected_errors == errors + + expected = get_test_loc('test_inv/some_missing_about_resource/expected.csv') + check_csv(expected, result_file, regen=False) + + def test_collect_inventory_complex_from_directory(self): + test_dir = get_test_loc('test_inv/complex') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + + inv.save_as_csv(result_file, packages) + + assert all(e.severity == INFO for e in errors) + + expected = get_test_loc('test_inv/complex/expected.csv') + check_csv(expected, result_file) + + def test_collect_inventory_does_not_damage_line_endings(self): + test_dir = get_test_loc('test_inv/crlf') + result_file = get_temp_file() + errors, packages = inv.collect_inventory(test_dir) + errors2 = inv.save_as_csv(result_file, packages) + errors.extend(errors2) + + assert all(e.severity == INFO for e in errors) + + expected = get_test_loc('test_inv/crlf/expected.csv') + check_csv(expected, result_file) + + def test_write_output_csv(self): + test_file = get_test_loc('test_inv/this.ABOUT') + package = model.Package.load(test_file) + result_file = get_temp_file() + inv.save_as_csv(result_file, [package]) + expected = get_test_loc('test_inv/expected.csv') + check_csv(expected, result_file) + + def test_write_output_json(self): + test_file = get_test_loc('test_inv/this.ABOUT') + package = model.Package.load(about_file_location=test_file) + result_file = get_temp_file() + inv.save_as_json(result_file, [package]) + expected = get_test_loc('test_inv/expected.json') + check_json(expected, result_file) + + def test_is_about_file(self): + assert inv.is_about_file('test.About') + assert inv.is_about_file('test2.aboUT') + assert not inv.is_about_file('no_about_ext.something') + assert not inv.is_about_file('about') + assert not inv.is_about_file('about.txt') + + def test_is_about_file_is_false_if_only_bare_extension(self): + assert not inv.is_about_file('.ABOUT') + + +class TestGetLocations(unittest.TestCase): + + def test_get_locations(self): + test_dir = get_test_loc('test_inv/about_locations') + expected = sorted([ + 'file with_spaces.ABOUT', + 'file1', + 'file2', + 'dir1/file2', + 'dir1/file2.aBout', + 'dir1/dir2/file1.about', + 'dir2/file1']) + + result = sorted(inv.get_locations(test_dir)) + result = [l.partition('/about_locations/')[-1] for l in result] + assert expected == result + + def test_get_about_locations(self): + test_dir = get_test_loc('test_inv/about_locations') + expected = sorted([ + 'file with_spaces.ABOUT', + 'dir1/file2.aBout', + 'dir1/dir2/file1.about', + ]) + + result = sorted(inv.get_about_locations(test_dir)) + result = [l.partition('/about_locations/')[-1] for l in result] + assert expected == result + + def test_get_locations_can_yield_a_single_file(self): + test_file = get_test_loc('test_inv/about_locations/file with_spaces.ABOUT') + result = list(inv.get_locations(test_file)) + assert 1 == len(result) + + def test_get_about_locations_for_about(self): + location = get_test_loc('test_inv/get_about_locations') + result = list(inv.get_about_locations(location)) + expected = 'get_about_locations/about.ABOUT' + assert result[0].endswith(expected) + + @skipIf(on_windows and not py3, 'Windows support for long path requires https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation') + def test_get_locations_with_very_long_path(self): + longpath = ( + 'longpath' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' + ) + test_loc = extract_test_loc('test_inv/locations/longpath.zip') + result = list(inv.get_locations(test_loc)) + assert any(longpath in r for r in result) diff --git a/tests/test_model.py b/tests/test_model.py index 15de984f..858469bc 100644 --- a/tests/test_model.py +++ b/tests/test_model.py @@ -20,27 +20,19 @@ from collections import OrderedDict import io -import json -import posixpath -import shutil import unittest -import mock import saneyaml -from attributecode import CRITICAL -from attributecode import ERROR -from attributecode import INFO -from attributecode import WARNING -from attributecode import Error -from attributecode import model -from attributecode.util import add_unc -from attributecode.util import load_csv -from attributecode.util import to_posix - -from testing_utils import extract_test_loc -from testing_utils import get_temp_file +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import model +from aboutcode.util import python2 +from aboutcode.util import unique + from testing_utils import get_test_loc +from testing_utils import check_json + try: # Python 2 @@ -50,47 +42,6 @@ unicode = str # NOQA -def check_csv(expected, result, regen=False, fix_cell_linesep=False): - """ - Assert that the contents of two CSV files locations `expected` and - `result` are equal. - """ - if regen: - shutil.copyfile(result, expected) - expected = sorted([sorted(d.items()) for d in load_csv(expected)]) - result = [d.items() for d in load_csv(result)] - if fix_cell_linesep: - result = [list(fix_crlf(items)) for items in result] - result = sorted(sorted(items) for items in result) - - assert expected == result - - -def fix_crlf(items): - """ - Hackish... somehow the CVS returned on Windows is sometimes using a backward - linesep convention: - instead of LF inside cells and CRLF at EOL, - they use CRLF everywhere. - This is fixing this until we find can why - """ - for key, value in items: - if isinstance(value, unicode) and '\r\n' in value: - value = value.replace('\r\n', '\n') - yield key, value - - -def check_json(expected, result): - """ - Assert that the contents of two JSON files are equal. - """ - with open(expected) as e: - expected = json.load(e, object_pairs_hook=OrderedDict) - with open(result) as r: - result = json.load(r, object_pairs_hook=OrderedDict) - assert expected == result - - def get_test_content(test_location): """ Read file at test_location and return a unicode string. @@ -106,144 +57,9 @@ def get_unicode_content(location): return doc.read() -class FieldTest(unittest.TestCase): - def test_Field_init(self): - model.Field() - model.StringField() - model.ListField() - model.UrlField() - model.BooleanField() - model.PathField() - model.FileTextField() - - def test_empty_Field_has_no_content(self): - field = model.Field() - assert not field.has_content - - def test_empty_Field_has_default_value(self): - field = model.Field() - assert '' == field.value - - def test_PathField_check_location(self): - test_file = 'license.LICENSE' - field = model.PathField(name='f', value=test_file, present=True) - base_dir = get_test_loc('test_model/base_dir') - - errors = field.validate(base_dir=base_dir) - expected_errrors = [] - assert expected_errrors == errors - - result = field.value[test_file] - expected = add_unc(posixpath.join(to_posix(base_dir), test_file)) - assert expected == result - - def test_PathField_check_missing_location(self): - test_file = 'does.not.exist' - field = model.PathField(name='f', value=test_file, present=True) - base_dir = get_test_loc('test_model/base_dir') - errors = field.validate(base_dir=base_dir) - - file_path = posixpath.join(base_dir, test_file) - err_msg = 'Field f: Path %s not found' % file_path - - expected_errors = [ - Error(CRITICAL, err_msg)] - assert expected_errors == errors - - result = field.value[test_file] - assert None == result - - def test_TextField_loads_file(self): - field = model.FileTextField( - name='f', value='license.LICENSE', present=True) - - base_dir = get_test_loc('test_model/base_dir') - errors = field.validate(base_dir=base_dir) - assert [] == errors - - expected = {'license.LICENSE': 'some license text'} - assert expected == field.value - - def test_UrlField_is_valid_url(self): - assert model.UrlField.is_valid_url('http://www.google.com') - - def test_UrlField_is_valid_url_not_starting_with_www(self): - assert model.UrlField.is_valid_url('https://nexb.com') - assert model.UrlField.is_valid_url('http://archive.apache.org/dist/httpcomponents/commons-httpclient/2.0/source/commons-httpclient-2.0-alpha2-src.tar.gz') - assert model.UrlField.is_valid_url('http://de.wikipedia.org/wiki/Elf (Begriffsklärung)') - assert model.UrlField.is_valid_url('http://nothing_here.com') - - def test_UrlField_is_valid_url_no_schemes(self): - assert not model.UrlField.is_valid_url('google.com') - assert not model.UrlField.is_valid_url('www.google.com') - assert not model.UrlField.is_valid_url('') - - def test_UrlField_is_valid_url_not_ends_with_com(self): - assert model.UrlField.is_valid_url('http://www.google') - - def test_UrlField_is_valid_url_ends_with_slash(self): - assert model.UrlField.is_valid_url('http://www.google.co.uk/') - - def test_UrlField_is_valid_url_empty_URL(self): - assert not model.UrlField.is_valid_url('http:') - - def check_validate(self, field_class, value, expected, expected_errors): - """ - Check field values after validation - """ - field = field_class(name='s', value=value, present=True) - # check that validate can be applied multiple times without side effects - for _ in range(2): - errors = field.validate() - assert expected_errors == errors - assert expected == field.value - - def test_StringField_validate_trailing_spaces_are_removed(self): - field_class = model.StringField - value = 'trailin spaces ' - expected = 'trailin spaces' - self.check_validate(field_class, value, expected, expected_errors=[]) - - def test_ListField_contains_list_after_validate(self): - value = 'string' - field_class = model.ListField - expected = [value] - self.check_validate(field_class, value, expected, expected_errors=[]) - - def test_ListField_contains_stripped_strings_after_validate(self): - value = '''first line - second line ''' - field_class = model.ListField - expected = ['first line', 'second line'] - self.check_validate(field_class, value, expected, expected_errors=[]) - - def test_PathField_contains_stripped_strings_after_validate(self): - value = '''first line - second line ''' - field_class = model.ListField - expected = ['first line', 'second line'] - self.check_validate(field_class, value, expected, expected_errors=[]) - - def test_PathField_contains_dict_after_validate(self): - value = 'string' - field_class = model.PathField - expected = OrderedDict([('string', None)]) - expected_errors = [ - Error(ERROR, 'Field s: Unable to verify path: string: No base directory provided') - ] - self.check_validate(field_class, value, expected, expected_errors) - - def test_SingleLineField_has_errors_if_multiline(self): - value = '''line1 - line2''' - field_class = model.SingleLineField - expected = value - expected_errors = [Error(ERROR, 'Field s: Cannot span multiple lines: line1\n line2')] - self.check_validate(field_class, value, expected, expected_errors) - - class YamlParseTest(unittest.TestCase): maxDiff = None + def test_saneyaml_load_can_parse_simple_fields(self): test = get_test_content('test_model/parse/basic.about') result = saneyaml.load(test) @@ -328,7 +144,7 @@ def test_saneyaml_load_accepts_unicode_keys_and_values(self): ('name', 'name'), ('about_resource', '.'), ('owner', 'Matías Aguirre'), - (u'Matías', u'unicode field name') + (u'matías', u'unicode field name') ] assert expected == list(result.items()) @@ -365,323 +181,273 @@ def test_saneyaml_loads_blank_lines_and_lines_without_no_colon(self): except Exception: pass -class AboutTest(unittest.TestCase): - def test_About_load_ignores_original_field_order_and_uses_standard_predefined_order(self): +class PackageTest(unittest.TestCase): + + def test_Package_load_ignores_original_field_order_and_uses_standard_predefined_order(self): # fields in this file are not in the standard order test_file = get_test_loc('test_model/parse/ordered_fields.ABOUT') - a = model.About(test_file) + a = model.Package.load(test_file) + a.about_file_path = 'this.ABOUT' + assert [] == a.errors - expected = ['about_resource', 'name', 'version', 'download_url'] - result = [f.name for f in a.all_fields() if f.present] - assert expected == result + expected_std = ['about_file_path', 'about_resource', 'name', 'version', 'download_url'] + expected_cust = sorted(['other', 'that']) + standard, custom = a.fields() + assert expected_std == standard + assert expected_cust == sorted(custom) - def test_About_duplicate_field_names_are_detected_with_different_case(self): - # This test is failing because the YAML does not keep the order when - # loads the test files. For instance, it treat the 'About_Resource' as the - # first element and therefore the dup key is 'about_resource'. + def test_Package_duplicate_field_names_are_detected_with_different_case(self): test_file = get_test_loc('test_model/parse/dupe_field_name.ABOUT') - a = model.About(test_file) - expected = [ - Error(WARNING, 'Field About_Resource is a duplicate. Original value: "." replaced with: "new value"'), - Error(WARNING, 'Field Name is a duplicate. Original value: "old" replaced with: "new"') - ] - - - result = a.errors - assert sorted(expected) == sorted(result) - - def test_About_duplicate_field_names_are_not_reported_if_same_value(self): - # This test is failing because the YAML does not keep the order when - # loads the test files. For instance, it treat the 'About_Resource' as the - # first element and therefore the dup key is 'about_resource'. + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, 'Invalid fields: lowercased field names must be unique.'), + Error(CRITICAL, 'Invalid fields: all field names must be lowercase.'), + Error(CRITICAL, "Custom field name: 'About_Resource' must be lowercase."), + Error(CRITICAL, "Custom field name: 'Name' must be lowercase.")) + assert expected == e.args + + def test_Package_duplicate_field_names_are_not_reported_if_same_value(self): test_file = get_test_loc('test_model/parse/dupe_field_name_no_new_value.ABOUT') - a = model.About(test_file) - expected = [ -] - result = a.errors - assert sorted(expected) == sorted(result) - - def check_About_hydrate(self, about, fields): - expected = set([ - 'name', - 'homepage_url', - 'download_url', - 'version', - 'copyright', - 'date', - 'license_spdx', - 'license_text_file', - 'notice_file', - 'about_resource']) - - expected_errors = [ - Error(INFO, 'Field date is a custom field.'), - Error(INFO, 'Field license_spdx is a custom field.'), - Error(INFO, 'Field license_text_file is a custom field.')] - - errors = about.hydrate(fields) - - assert expected_errors == errors - - result = set([f.name for f in about.all_fields() if f.present]) - assert expected == result - - def test_About_hydrate_normalize_field_names_to_lowercase(self): - test_content = get_test_content('test_gen/parser_tests/upper_field_names.ABOUT') - fields = saneyaml.load(test_content).items() - a = model.About() - for _ in range(3): - self.check_About_hydrate(a, fields) - - def test_About_with_existing_about_resource_has_no_error(self): - test_file = get_test_loc('test_gen/parser_tests/about_resource_field.ABOUT') - a = model.About(test_file) + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, 'Invalid fields: lowercased field names must be unique.'), + Error(CRITICAL, 'Invalid fields: all field names must be lowercase.'), + Error(CRITICAL, "Custom field name: 'About_Resource' must be lowercase."), + Error(CRITICAL, "Custom field name: 'Name' must be lowercase."), + ) + assert expected == e.args + + def test_Package_fails_if_field_names_are_not_lowercase(self): + test_file = get_test_loc('test_model/parser_tests/upper_field_names.ABOUT') + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, 'Invalid fields: all field names must be lowercase.'), + Error(CRITICAL, "Custom field name: 'homepage_URL' must be lowercase."), + ) + assert expected == e.args + + def test_Package_with_existing_about_resource_has_no_error(self): + test_file = get_test_loc('test_model/parser_tests/about_resource_field.ABOUT') + a = model.Package.load(test_file) assert [] == a.errors - result = a.about_resource.value['about_resource.c'] - # this means we have a location - self.assertNotEqual([], result) - - def test_About_has_errors_when_about_resource_is_missing(self): - test_file = get_test_loc('test_gen/parser_tests/.ABOUT') - a = model.About(test_file) - expected = [Error(CRITICAL, 'Field about_resource is required')] - result = a.errors - assert expected == result + assert a.about_resource - def test_About_has_errors_when_about_resource_does_not_exist(self): - test_file = get_test_loc('test_gen/parser_tests/missing_about_ref.ABOUT') - file_path = posixpath.join(posixpath.dirname(test_file), 'about_file_missing.c') - a = model.About(test_file) - err_msg = 'Field about_resource: Path %s not found' % file_path - expected = [Error(INFO, err_msg)] - result = a.errors - assert expected == result + def test_Package_has_errors_when_about_resource_does_not_exist(self): + test_file = get_test_loc('test_model/parser_tests/missing_about_ref.ABOUT') + package = model.Package.load(test_file) + package.check_files() - def test_About_has_errors_when_missing_required_fields_are_missing(self): - test_file = get_test_loc('test_model/parse/missing_required.ABOUT') - a = model.About(test_file) expected = [ - Error(CRITICAL, 'Field about_resource is required'), - Error(CRITICAL, 'Field name is required'), + Error(CRITICAL, 'File about_resource: "about_file_missing.c" does not exists') ] - result = a.errors - assert expected == result + assert expected == package.errors + + def test_Package_raise_exception_when_missing_required_fields_are_missing(self): + test_file = get_test_loc('test_model/parse/missing_required.ABOUT') - def test_About_has_errors_when_required_fields_are_empty(self): + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, 'Required field "about_resource" is missing.'), + ) + assert expected == e.args + + def test_Package_raise_exception_when_required_fields_are_empty(self): test_file = get_test_loc('test_model/parse/empty_required.ABOUT') - a = model.About(test_file) - expected = [ - Error(CRITICAL, 'Field about_resource is required and empty'), - Error(CRITICAL, 'Field name is required and empty'), - ] - result = a.errors - assert expected == result + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, 'Required field "about_resource" is missing.'), + ) + assert expected == e.args - def test_About_has_errors_with_empty_notice_file_field(self): + def test_Package_has_no_errors_with_empty_notice_file_field(self): test_file = get_test_loc('test_model/parse/empty_notice_field.about') - a = model.About(test_file) - expected = [ - Error(INFO, 'Field notice_file is present but empty.')] + a = model.Package.load(test_file) + expected = [] result = a.errors assert expected == result - def test_About_custom_fields_are_never_ignored(self): + def test_Package_custom_fields_are_never_ignored_unless_empty(self): test_file = get_test_loc('test_model/custom_fields/custom_fields.about') - a = model.About(test_file) - result = [(n, f.value) for n, f in a.custom_fields.items()] - expected = [ - (u'single_line', u'README STUFF'), - (u'multi_line', u'line1\nline2'), - (u'other', u'sasasas'), - (u'empty', u'') - ] + a = model.Package.load(test_file) + expected = { + 'single_line': 'README STUFF', + 'multi_line': 'line1\nline2', + 'other': 'sasasas', + } - assert expected == result + assert expected == a.custom_fields - def test_About_custom_fields_are_not_ignored_and_order_is_preserved(self): + def test_Package_custom_fields_order_is_ignored_and_order_is_not_preserved(self): test_file = get_test_loc('test_model/custom_fields/custom_fields.about') - a = model.About(test_file) - result = [(n, f.value) for n, f in a.custom_fields.items()] + a = model.Package.load(test_file) + result = sorted(a.custom_fields.items()) expected = [ - (u'single_line', u'README STUFF'), (u'multi_line', u'line1\nline2'), (u'other', u'sasasas'), - (u'empty', u'') + (u'single_line', u'README STUFF'), ] assert sorted(expected) == sorted(result) - def test_About_has_errors_for_illegal_custom_field_name(self): + def test_Package_custom_fields_are_available_as_direct_instance_attributes(self): + test_file = get_test_loc('test_model/custom_fields/custom_fields.about') + a = model.Package.load(test_file) + assert 'sasasas' == a.other + + def test_Package_has_info_for_custom_field_name(self): test_file = get_test_loc('test_model/parse/illegal_custom_field.about') - a = model.About(test_file) - expected_errors = [ - Error(INFO, 'Field hydrate is a custom field.'), - Error(CRITICAL, "Internal error with custom field: 'hydrate': 'illegal name'.") - ] - assert expected_errors == a.errors - assert not hasattr(getattr(a, 'hydrate'), 'value') - field = list(a.custom_fields.values())[0] - assert 'hydrate' == field.name - assert 'illegal name' == field.value + package = model.Package.load(test_file) + assert [] == package.errors - def test_About_file_fields_are_empty_if_present_and_path_missing(self): + def test_Package_check_files_collect_errors_if_path_missing(self): test_file = get_test_loc('test_model/parse/missing_notice_license_files.ABOUT') - a = model.About(test_file) + package = model.Package.load(test_file) - file_path1 = posixpath.join(posixpath.dirname(test_file), 'test.LICENSE') - file_path2 = posixpath.join(posixpath.dirname(test_file), 'test.NOTICE') + package.check_files() - err_msg1 = Error(CRITICAL, 'Field license_file: Path %s not found' % file_path1) - err_msg2 = Error(CRITICAL, 'Field notice_file: Path %s not found' % file_path2) - - expected_errors = [err_msg1, err_msg2] - assert expected_errors == a.errors + expected_errors = [ + Error(CRITICAL, 'File notice_file: "test.NOTICE" does not exists'), + Error(CRITICAL, 'License file: "test.LICENSE" does not exists'), + ] + assert expected_errors == package.errors - assert {'test.LICENSE': None} == a.license_file.value - assert {'test.NOTICE': None} == a.notice_file.value + assert 'test.NOTICE' == package.notice_file - def test_About_notice_and_license_text_are_loaded_from_file(self): + def test_Package_notice_and_license_text_are_loaded_from_file(self): test_file = get_test_loc('test_model/parse/license_file_notice_file.ABOUT') - a = model.About(test_file) + a = model.Package.load(test_file) + a.load_files() expected = '''Tester holds the copyright for test component. Tester relinquishes copyright of this software and releases the component to Public Domain. * Email Test@tester.com for any questions''' - - result = a.license_file.value['license_text.LICENSE'] - assert expected == result + assert expected == a.licenses[0].text expected = '''Test component is released to Public Domain.''' - result = a.notice_file.value['notice_text.NOTICE'] - assert expected == result + assert expected == a.notice_text - def test_About_license_and_notice_text_are_empty_if_field_missing(self): + def test_Package_license_and_notice_text_are_empty_if_field_missing(self): test_file = get_test_loc('test_model/parse/no_file_fields.ABOUT') - a = model.About(test_file) + a = model.Package.load(test_file) assert [] == a.errors - assert {} == a.license_file.value - assert {} == a.notice_file.value + assert not a.notice_file + assert [] == a.licenses - def test_About_rejects_non_ascii_names_and_accepts_unicode_values(self): + def test_Package_cannot_be_created_with_non_ascii_custom_field_names(self): test_file = get_test_loc('test_model/parse/non_ascii_field_name_value.about') - a = model.About(test_file) - expected = [ - Error(CRITICAL, "Field name: 'mat\xedas' contains illegal name characters: 0 to 9, a to z, A to Z and _.") - ] - assert expected == a.errors - def test_About_invalid_boolean_value(self): - test_file = get_test_loc('test_model/parse/invalid_boolean.about') - a = model.About(test_file) - expected_msg = "Field modified: Invalid flag value: 'blah'" - assert expected_msg in a.errors[0].message - def test_About_contains_about_file_path(self): + var = 'mat\\xedas' if python2 else 'matías' + msg = ('Custom field name: \'{}\' contains illegal characters. ' + 'Only these characters are allowed: ASCII letters, digits and "_" underscore. ' + 'The first character must be a letter.').format(var) + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = (Error(CRITICAL, msg),) + assert expected == e.args + + def test_Package_cannot_be_created_with_invalid_boolean_value(self): + test_file = get_test_loc('test_model/parse/invalid_boolean.about') + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected = ( + Error(CRITICAL, "Field name: 'modified' has an invalid flag value: " + "'blah': should be one of yes or no or true or false."), + ) + assert expected == e.args + + def test_Package_contains_about_file_path(self): test_file = get_test_loc('test_model/serialize/about.ABOUT') # TODO: I am not sure this override of the about_file_path makes sense - a = model.About(test_file, about_file_path='complete/about.ABOUT') + a = model.Package.load(test_file) assert [] == a.errors - expected = 'complete/about.ABOUT' - result = a.about_file_path - assert expected == result - def test_About_equals(self): + expected = 'test_model/serialize/about.ABOUT' + assert a.about_file_location.endswith(expected) + + def test_Package_equals(self): test_file = get_test_loc('test_model/equal/complete/about.ABOUT') - a = model.About(test_file, about_file_path='complete/about.ABOUT') - b = model.About(test_file, about_file_path='complete/about.ABOUT') + a = model.Package.load(test_file) + b = model.Package.load(test_file) assert a == b - def test_About_are_not_equal_with_small_text_differences(self): - test_file = get_test_loc('test_model/equal/complete2/about.ABOUT') - a = model.About(test_file, about_file_path='complete2/about.ABOUT') - test_file2 = get_test_loc('test_model/equal/complete/about.ABOUT') - b = model.About(test_file2, about_file_path='complete/about.ABOUT') - assert a.dumps() != b.dumps() - assert a != b + def test_Package_are_not_equal_with_small_text_differences(self): + test_file = get_test_loc('test_model/equal/complete/about.ABOUT') + package = model.Package.load(about_file_location=test_file) - def test_get_field_names_only_returns_non_empties(self): - a = model.About() - a.custom_fields['f'] = model.StringField( - name='f', value='1', present=True) - b = model.About() - b.custom_fields['g'] = model.StringField( - name='g', value='1', present=True) - abouts = [a, b] - # ensure that custom fields and about file path are collected - # and that all fields are in the correct order - expected = [ - model.About.ABOUT_FILE_PATH_ATTR, - 'about_resource', 'name', 'f', 'g' - ] - result = model.get_field_names(abouts) - assert expected == result + test_file2 = get_test_loc('test_model/equal/complete2/about.ABOUT') + package2 = model.Package.load(test_file2) - def test_get_field_names_does_not_return_duplicates_custom_fields(self): - a = model.About() - a.custom_fields['f'] = model.StringField(name='f', value='1', - present=True) - a.custom_fields['cf'] = model.StringField(name='cf', value='1', - present=True) - b = model.About() - b.custom_fields['g'] = model.StringField(name='g', value='1', - present=True) - b.custom_fields['cf'] = model.StringField(name='cf', value='2', - present=True) - abouts = [a, b] - # ensure that custom fields and about file path are collected - # and that all fields are in the correct order - # FIXME: this is not USED - expected = [ - 'about_file_path', - 'about_resource', - 'name', - 'cf', - 'f', - 'g', - ] - result = model.get_field_names(abouts) - assert expected == result + assert package.dumps() != package2.dumps() + assert package != package2 + + def test_get_field_names_only_returns_non_empties(self): + a = model.Package(about_resource='.', notice_file='sadasdasd') + a.custom_fields['f'] = '1' + expected = ['about_resource', 'notice_file'], ['f'] + assert expected == a.fields() class SerializationTest(unittest.TestCase): - def test_About_dumps(self): + + def test_Package_dumps(self): test_file = get_test_loc('test_model/dumps/about.ABOUT') - a = model.About(test_file) + a = model.Package.load(test_file) assert [] == a.errors expected = '''about_resource: . name: AboutCode version: 0.11.0 description: | - AboutCode is a tool - to process ABOUT files. - An ABOUT file is a file. + AboutCode is a tool + to process ABOUT files. + An ABOUT file is a file. homepage_url: http://dejacode.org -license_expression: apache-2.0 copyright: Copyright (c) 2013-2014 nexB Inc. +license_expression: apache-2.0 +licenses: + - file: apache-2.0.LICENSE + key: apache-2.0 notice_file: NOTICE owner: nexB Inc. -author: Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez vcs_tool: git vcs_repository: https://github.com/dejacode/about-code-tool.git -licenses: - - key: apache-2.0 - file: apache-2.0.LICENSE +author: + - Jillian Daguil + - Chin Yeung Li + - Philippe Ombredanne + - Thomas Druez ''' - result = a.dumps() - assert expected == result - def test_About_dumps_does_all_non_empty_present_fields(self): + expected = model.Package.loads(expected) + result = model.Package.loads(a.dumps()) + assert expected.to_dict() == result.to_dict() + + def test_Package_dumps_does_all_non_empty_present_fields(self): test_file = get_test_loc('test_model/parse/complete2/about.ABOUT') - a = model.About(test_file) - expected_error = [ - Error(INFO, 'Field custom1 is a custom field.'), - Error(INFO, 'Field custom2 is a custom field.'), - Error(INFO, 'Field custom2 is present but empty.') - ] - assert sorted(expected_error) == sorted(a.errors) + a = model.Package.load(test_file) + assert [] == a.errors expected = '''about_resource: . name: AboutCode @@ -693,35 +459,23 @@ def test_About_dumps_does_all_non_empty_present_fields(self): result = a.dumps() assert expected == result - def test_About_dumps_with_different_boolean_value(self): + def test_Package_is_not_created_with_invalid_flag_value(self): test_file = get_test_loc('test_model/parse/complete2/about2.ABOUT') - a = model.About(test_file) - expected_error_msg = "Field track_changes: Invalid flag value: 'blah' is not one of" - assert len(a.errors) == 1 - assert expected_error_msg in a.errors[0].message - - expected = '''about_resource: . - -name: AboutCode -version: 0.11.0 - -redistribute: no -attribute: yes -modified: yes -''' - - result = a.dumps() - assert set(expected) == set(result) - - def test_About_dumps_all_non_empty_fields(self): + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except Exception as e: + expected_error = ( + Error(CRITICAL, + "Field name: 'track_changes' has an invalid flag value: 'blah':" + " should be one of yes or no or true or false."), + ) + assert expected_error == e.args + + def test_Package_dumps_all_non_empty_fields(self): test_file = get_test_loc('test_model/parse/complete2/about.ABOUT') - a = model.About(test_file) - expected_error = [ - Error(INFO, 'Field custom1 is a custom field.'), - Error(INFO, 'Field custom2 is a custom field.'), - Error(INFO, 'Field custom2 is present but empty.') - ] - assert sorted(expected_error) == sorted(a.errors) + a = model.Package.load(test_file) + assert [] == a.errors expected = '''about_resource: . name: AboutCode @@ -733,54 +487,42 @@ def test_About_dumps_all_non_empty_fields(self): result = a.dumps() assert expected == result - def test_About_as_dict_contains_special_paths(self): + def test_Package_to_dict_contains_special_paths(self): test_file = get_test_loc('test_model/special/about.ABOUT') - a = model.About(test_file, about_file_path='complete/about.ABOUT') - expected_errors = [] - assert expected_errors == a.errors - as_dict = a.as_dict() - expected = 'complete/about.ABOUT' - result = as_dict[model.About.ABOUT_FILE_PATH_ATTR] - assert expected == result + a = model.Package.load(test_file) + assert [] == sorted(a.errors) + + expected = get_test_loc('test_model/special/about-expected.json') + result = a.to_dict() + check_json(expected, result) def test_load_dump_is_idempotent(self): test_file = get_test_loc('test_model/this.ABOUT') - a = model.About() - a.load(test_file) - dumped_file = get_temp_file('that.ABOUT') - a.dump(dumped_file) - + a = model.Package.load(test_file) expected = get_unicode_content(test_file).splitlines() - result = get_unicode_content(dumped_file).splitlines() + result = a.dumps().splitlines() assert expected == result def test_load_can_load_unicode(self): test_file = get_test_loc('test_model/unicode/nose-selecttests.ABOUT') - a = model.About() - a.load(test_file) - file_path = posixpath.join(posixpath.dirname(test_file), 'nose-selecttests-0.3.zip') - err_msg = 'Field about_resource: Path %s not found' % file_path + a = model.Package.load(test_file) + a.check_files() errors = [ - Error(INFO, 'Field dje_license is a custom field.'), - Error(INFO, 'Field license_text_file is a custom field.'), - Error(INFO, 'Field scm_tool is a custom field.'), - Error(INFO, 'Field scm_repository is a custom field.'), - Error(INFO, 'Field test is a custom field.'), - Error(INFO, err_msg)] + Error(CRITICAL, 'File about_resource: "nose-selecttests-0.3.zip" does not exists'), + ] - assert errors == a.errors - assert 'Copyright (c) 2012, Domen Kožar' == a.copyright.value + assert errors == unique(a.errors) + assert 'Copyright (c) 2012, Domen Kožar' == a.copyright - def test_load_has_errors_for_non_unicode(self): + def test_load_raise_exception_for_non_unicode(self): test_file = get_test_loc('test_model/unicode/not-unicode.ABOUT') - a = model.About() - a.load(test_file) - err = a.errors[0] - assert CRITICAL == err.severity - assert 'Cannot load invalid ABOUT file' in err.message - assert 'UnicodeDecodeError' in err.message - - def test_as_dict_load_dict_ignores_empties(self): + try: + model.Package.load(test_file) + self.fail('Exception not raised') + except UnicodeDecodeError: + pass + + def test_to_dict_load_dict_ignores_empties(self): test = { 'about_resource': '.', 'author': '', @@ -792,28 +534,28 @@ def test_as_dict_load_dict_ignores_empties(self): 'name': 'AboutCode', 'owner': 'nexB Inc.'} - expected = { - 'about_file_path': None, - 'about_resource': OrderedDict([('.', None)]), - 'copyright': 'Copyright (c) 2013-2014 nexB Inc.', - 'custom1': 'some custom', - 'description': 'AboutCode is a tool\nfor files.', - 'license_expression': 'apache-2.0', - 'name': 'AboutCode', - 'owner': 'nexB Inc.'} + expected = OrderedDict([ + ('about_resource', u'.'), + ('name', u'AboutCode'), + ('description', u'AboutCode is a tool\nfor files.'), + ('copyright', u'Copyright (c) 2013-2014 nexB Inc.'), + ('license_expression', u'apache-2.0'), + ('licenses', [OrderedDict([ + ('key', u'apache-2.0'), + ('file', u'apache-2.0.LICENSE'), + ])]), + ('owner', u'nexB Inc.'), + (u'custom1', u'some custom')] + ) - a = model.About() - base_dir = 'some_dir' - a.load_dict(test, base_dir) - as_dict = a.as_dict() - # FIXME: why converting back to dict? - assert expected == dict(as_dict) + a = model.Package.from_dict(test) + assert expected == a.to_dict() def test_load_dict_as_dict_is_idempotent_ignoring_special(self): test = { - 'about_resource': ['.'], + 'about_resource': '.', 'attribute': 'yes', - 'author': 'Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez', + 'author': ['Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez'], 'copyright': 'Copyright (c) 2013-2014 nexB Inc.', 'description': 'AboutCode is a tool to process ABOUT files. An ABOUT file is a file.', 'homepage_url': 'http://dejacode.org', @@ -823,293 +565,56 @@ def test_load_dict_as_dict_is_idempotent_ignoring_special(self): 'vcs_repository': 'https://github.com/dejacode/about-code-tool.git', 'vcs_tool': 'git', 'version': '0.11.0'} - a = model.About() - base_dir = 'some_dir' - a.load_dict(test, base_dir) - as_dict = a.as_dict() + + a = model.Package.from_dict(test) expected = { - 'about_file_path': None, - 'about_resource': OrderedDict([('.', None)]), - 'attribute': 'yes', - 'author': 'Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez', + 'about_resource': '.', + 'attribute': True, + 'author': ['Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez'], 'copyright': 'Copyright (c) 2013-2014 nexB Inc.', 'description': 'AboutCode is a tool to process ABOUT files. An ABOUT file is a file.', 'homepage_url': 'http://dejacode.org', 'license_expression': 'apache-2.0', + 'licenses': [OrderedDict([ + ('key', u'apache-2.0'), + ('file', u'apache-2.0.LICENSE'), + ])], 'name': 'AboutCode', 'owner': 'nexB Inc.', 'vcs_repository': 'https://github.com/dejacode/about-code-tool.git', 'vcs_tool': 'git', 'version': '0.11.0'} - assert expected == dict(as_dict) + assert expected == dict(a.to_dict()) def test_about_model_class_from_dict_constructor(self): - about_data = { - 'about_resource': ['.'], - 'attribute': 'yes', - 'author': 'Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez', - 'copyright': 'Copyright (c) 2013-2014 nexB Inc.', - 'description': 'AboutCode is a tool to process ABOUT files. An ABOUT file is a file.', - 'homepage_url': 'http://dejacode.org', - 'license_expression': 'apache-2.0', - 'name': 'AboutCode', - 'owner': 'nexB Inc.', - 'vcs_repository': 'https://github.com/dejacode/about-code-tool.git', - 'vcs_tool': 'git', - 'version': '0.11.0', - } - - about = model.About.from_dict(about_data) - assert isinstance(about, model.About) - - about_data.update({ - 'about_file_path': None, - 'about_resource': OrderedDict([('.', None)]), - }) - assert about_data == about.as_dict() - - def test_write_output_csv(self): - path = 'test_model/this.ABOUT' - test_file = get_test_loc(path) - abouts = model.About(location=test_file, about_file_path=path) - - result = get_temp_file() - model.write_output([abouts], result, format='csv') - - expected = get_test_loc('test_model/expected.csv') - check_csv(expected, result) - - def test_write_output_json(self): - path = 'test_model/this.ABOUT' - test_file = get_test_loc(path) - abouts = model.About(location=test_file, about_file_path=path) - - result = get_temp_file() - model.write_output([abouts], result, format='json') - - expected = get_test_loc('test_model/expected.json') - check_json(expected, result) - - -class CollectorTest(unittest.TestCase): - - def test_collect_inventory_return_errors(self): - test_loc = get_test_loc('test_model/collect_inventory_errors') - errors, _abouts = model.collect_inventory(test_loc) - file_path1 = posixpath.join(test_loc, 'distribute_setup.py') - file_path2 = posixpath.join(test_loc, 'date_test.py') - - err_msg1 = 'non-supported_date_format.ABOUT: Field about_resource: Path %s not found' % file_path1 - err_msg2 = 'supported_date_format.ABOUT: Field about_resource: Path %s not found' % file_path2 - expected_errors = [ - Error(INFO, 'non-supported_date_format.ABOUT: Field date is a custom field.'), - Error(INFO, 'supported_date_format.ABOUT: Field date is a custom field.'), - Error(INFO, err_msg1), - Error(INFO, err_msg2)] - assert sorted(expected_errors) == sorted(errors) - - def test_collect_inventory_with_long_path(self): - test_loc = extract_test_loc('test_model/longpath.zip') - _errors, abouts = model.collect_inventory(test_loc) - assert 2 == len(abouts) - - expected_paths = ( - 'longpath/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/non-supported_date_format.ABOUT', - 'longpath/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/supported_date_format.ABOUT' - ) - results = [a.about_file_path for a in abouts] - assert all(r.endswith(expected_paths) for r in results) - - expected_name = ['distribute', 'date_test'] - result_name = [a.name.value for a in abouts] - assert sorted(expected_name) == sorted(result_name) - - def test_collect_inventory_can_collect_a_single_file(self): - test_loc = get_test_loc('test_model/single_file/django_snippets_2413.ABOUT') - _errors, abouts = model.collect_inventory(test_loc) - assert 1 == len(abouts) - expected = ['single_file/django_snippets_2413.ABOUT'] - result = [a.about_file_path for a in abouts] - assert expected == result - - def test_collect_inventory_return_no_warnings_and_model_can_uuse_relative_paths(self): - test_loc = get_test_loc('test_model/rel/allAboutInOneDir') - errors, _abouts = model.collect_inventory(test_loc) - expected_errors = [] - result = [(level, e) for level, e in errors if level > INFO] - assert expected_errors == result - - def test_collect_inventory_populate_about_file_path(self): - test_loc = get_test_loc('test_model/inventory/complete') - errors, abouts = model.collect_inventory(test_loc) - assert [] == errors - expected = 'about.ABOUT' - result = abouts[0].about_file_path - assert expected == result - - def test_collect_inventory_with_multi_line(self): - test_loc = get_test_loc('test_model/parse/multi_line_license_expresion.ABOUT') - errors, abouts = model.collect_inventory(test_loc) - assert [] == errors - expected_lic_url = [ - 'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:mit', - 'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:apache-2.0'] - returned_lic_url = abouts[0].license_url.value - assert expected_lic_url == returned_lic_url - - def test_collect_inventory_with_license_expression(self): - test_loc = get_test_loc('test_model/parse/multi_line_license_expresion.ABOUT') - errors, abouts = model.collect_inventory(test_loc) - assert [] == errors - expected_lic = 'mit or apache-2.0' - returned_lic = abouts[0].license_expression.value - assert expected_lic == returned_lic - - def test_collect_inventory_always_collects_custom_fieldsg(self): - test_loc = get_test_loc('test_model/inventory/custom_fields.ABOUT') - errors, abouts = model.collect_inventory(test_loc) - expected_msg1 = 'Field resource is a custom field' - assert len(errors) == 2 - assert expected_msg1 in errors[0].message - # The not supported 'resource' value is collected - assert abouts[0].resource.value - - def test_collect_inventory_does_not_raise_error_and_maintains_order_on_custom_fields(self): - test_loc = get_test_loc('test_model/inventory/custom_fields2.ABOUT') - errors, abouts = model.collect_inventory(test_loc) - expected_errors = [ - Error(INFO, 'inventory/custom_fields2.ABOUT: Field resource is a custom field.'), - Error(INFO, 'inventory/custom_fields2.ABOUT: Field custom_mapping is a custom field.') - ] - assert expected_errors == errors - expected = [u'about_resource: .\nname: test\nresource: .\ncustom_mapping: test\n'] - assert expected == [a.dumps() for a in abouts] - - def test_parse_license_expression(self): - spec_char, returned_lic = model.parse_license_expression('mit or apache-2.0') - expected_lic = ['mit', 'apache-2.0'] - expected_spec_char = [] - assert expected_lic == returned_lic - assert expected_spec_char == spec_char - - def test_parse_license_expression_with_special_chara(self): - spec_char, returned_lic = model.parse_license_expression('mit, apache-2.0') - expected_lic = [] - expected_spec_char = [','] - assert expected_lic == returned_lic - assert expected_spec_char == spec_char - - def test_collect_inventory_works_with_relative_paths(self): - # FIXME: This test need to be run under src/attributecode/ - # or otherwise it will fail as the test depends on the launching - # location - test_loc = get_test_loc('test_model/inventory/relative') - # Use '.' as the indication of the current directory - test_loc1 = test_loc + '/./' - # Use '..' to go back to the parent directory - test_loc2 = test_loc + '/../relative' - errors1, abouts1 = model.collect_inventory(test_loc1) - errors2, abouts2 = model.collect_inventory(test_loc2) - assert [] == errors1 - assert [] == errors2 - expected = 'about.ABOUT' - result1 = abouts1[0].about_file_path - result2 = abouts2[0].about_file_path - assert expected == result1 - assert expected == result2 - - def test_collect_inventory_basic_from_directory(self): - location = get_test_loc('test_model/inventory/basic') - result = get_temp_file() - errors, abouts = model.collect_inventory(location) - - model.write_output(abouts, result, format='csv') - - expected_errors = [] - assert expected_errors == errors - - expected = get_test_loc('test_model/inventory/basic/expected.csv') - check_csv(expected, result) - - def test_collect_inventory_with_about_resource_path_from_directory(self): - location = get_test_loc('test_model/inventory/basic_with_about_resource_path') - result = get_temp_file() - errors, abouts = model.collect_inventory(location) - - model.write_output(abouts, result, format='csv') - - expected_errors = [] - assert expected_errors == errors - - expected = get_test_loc('test_model/inventory/basic_with_about_resource_path/expected.csv') - check_csv(expected, result) - - def test_collect_inventory_with_no_about_resource_from_directory(self): - location = get_test_loc('test_model/inventory/no_about_resource_key') - result = get_temp_file() - errors, abouts = model.collect_inventory(location) - - model.write_output(abouts, result, format='csv') - - expected_errors = [Error(CRITICAL, 'about/about.ABOUT: Field about_resource is required')] - assert expected_errors == errors - - expected = get_test_loc('test_model/inventory/no_about_resource_key/expected.csv') - check_csv(expected, result) - - def test_collect_inventory_complex_from_directory(self): - location = get_test_loc('test_model/inventory/complex') - result = get_temp_file() - errors, abouts = model.collect_inventory(location) - - model.write_output(abouts, result, format='csv') - - assert all(e.severity == INFO for e in errors) - - expected = get_test_loc('test_model/inventory/complex/expected.csv') - check_csv(expected, result, fix_cell_linesep=True, regen=False) - - def test_collect_inventory_does_not_convert_lf_to_crlf_from_directory(self): - location = get_test_loc('test_model/crlf/about.ABOUT') - result = get_temp_file() - errors, abouts = model.collect_inventory(location) - errors2 = model.write_output(abouts, result, format='csv') - errors.extend(errors2) - assert all(e.severity == INFO for e in errors) - - expected = get_test_loc('test_model/crlf/expected.csv') - check_csv(expected, result, fix_cell_linesep=True, regen=False) - - -class FetchLicenseTest(unittest.TestCase): - - @mock.patch.object(model, 'urlopen') - def test_valid_api_url(self, mock_data): - mock_data.return_value = '' - assert model.valid_api_url('non_valid_url') is False - - @mock.patch('attributecode.util.have_network_connection') - @mock.patch('attributecode.model.valid_api_url') - def test_pre_process_and_fetch_license_dict(self, have_network_connection, valid_api_url): - have_network_connection.return_value = True - - valid_api_url.return_value = False - error_msg = ( - 'Network problem. Please check your Internet connection. ' - 'License generation is skipped.') - expected = ({}, [Error(ERROR, error_msg)]) - assert model.pre_process_and_fetch_license_dict([], '', '') == expected - - valid_api_url.return_value = True - expected = ({}, []) - assert model.pre_process_and_fetch_license_dict([], '', '') == expected + data = OrderedDict([ + ('about_resource', '.'), + ('name', 'AboutCode'), + ('version', '0.11.0'), + ('description', 'AboutCode is a tool to process ABOUT files. An ABOUT file is a file.'), + ('homepage_url', 'http://dejacode.org'), + ('copyright', 'Copyright (c) 2013-2014 nexB Inc.'), + ('license_expression', 'apache-2.0'), + ('attribute', True), + ('licenses', [OrderedDict([ + ('key', 'apache-2.0'), + ('file', 'apache-2.0.LICENSE'), + ])]), + ('owner', 'nexB Inc.'), + ('author', ['Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez']), + ('vcs_repository', 'https://github.com/dejacode/about-code-tool.git'), + ('vcs_tool', 'git'), + ]) + package = model.Package.from_dict(data) + assert data.items() == package.to_dict().items() + + +class ReferenceTest(unittest.TestCase): + + def test_get_reference_licenses_can_load_non_utf_files(self): + test_dir = get_test_loc('test_model/reference') + notices_by_name, licenses_by_key = model.get_reference_licenses(test_dir) + assert ['bad.NOTICE'] == list(notices_by_name.keys()) + assert ['weird'] == list(licenses_by_key.keys()) diff --git a/tests/test_transform.py b/tests/test_transform.py new file mode 100644 index 00000000..ed8ee052 --- /dev/null +++ b/tests/test_transform.py @@ -0,0 +1,63 @@ +#!/usr/bin/env python +# -*- coding: utf8 -*- + +# ============================================================================ +# Copyright (c) 2014-2017 nexB Inc. http://www.nexb.com/ - All rights reserved. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import unicode_literals + +import unittest + +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import transform + +from testing_utils import get_test_loc + + +class TransformTest(unittest.TestCase): + + def test_read_csv_rows_can_read_invalid_utf8(self): + test_file = get_test_loc('test_transform/mojibake.csv') + list(transform.read_csv_rows(test_file)) + + def test_get_duplicate_columns(self): + column_names = 'a', 'b', 'a' + result = transform.get_duplicate_columns(column_names) + assert ['a'] == result + + def test_check_required_columns_always_include_defaults(self): + test_data = [ + dict(about_resource='' , + name='Utilities' , + version='0.11.0' , + foo='bar', + baz='val'), + dict( + about_resource='tarball.tgz', + name='Core', + version='1', + foo='', + baz='') + ] + + required_columns = ['name', 'version', 'foo', 'required'] + transformer = transform.Transformer(required_columns=required_columns) + errors = transformer.check_required_columns(test_data) + expected = [ + Error(CRITICAL, 'Row 1 is missing required values for columns: about_resource, required'), + Error(CRITICAL, 'Row 2 is missing required values for columns: foo, required')] + + assert expected == errors diff --git a/tests/test_util.py b/tests/test_util.py index c1cad7cf..48dd2cac 100644 --- a/tests/test_util.py +++ b/tests/test_util.py @@ -18,22 +18,19 @@ from __future__ import print_function from __future__ import unicode_literals -from collections import OrderedDict import string import unittest import saneyaml -from testing_utils import extract_test_loc -from testing_utils import get_test_loc +from aboutcode import CRITICAL +from aboutcode import Error +from aboutcode import model +from aboutcode import util + from testing_utils import on_posix from testing_utils import on_windows -from attributecode import CRITICAL -from attributecode import Error -from attributecode import model -from attributecode import util - class TestResourcePaths(unittest.TestCase): @@ -207,22 +204,11 @@ def test_check_file_names_with_invalid_chars_return_errors(self): assert expected[0].message == result[0].message assert expected == result - def test_is_about_file(self): - assert util.is_about_file('test.About') - assert util.is_about_file('test2.aboUT') - assert not util.is_about_file('no_about_ext.something') - assert not util.is_about_file('about') - assert not util.is_about_file('about.txt') - - def test_is_about_file_is_false_if_only_bare_extension(self): - assert not util.is_about_file('.ABOUT') - def test_get_relative_path(self): test = [('/some/path', '/some/path/file', 'file'), ('path', '/path/file', 'file'), ('/path', '/path/file', 'file'), ('/path/', '/path/file/', 'file'), - ('/path/', 'path/', 'path'), ('/p1/p2/p3', '/p1/p2//p3/file', 'file'), (r'c:\some/path', 'c:/some/path/file', 'file'), (r'c:\\some\\path\\', 'c:/some/path/file', 'file'), @@ -231,252 +217,12 @@ def test_get_relative_path(self): result = util.get_relative_path(base_loc, full_loc) assert expected == result - def test_get_relative_path_with_same_path_twice(self): - test = [('/some/path/file', 'path/file'), - ('/path/file', 'path/file'), - ('/path/file/', 'path/file'), - ('path/', 'path'), - ('/p1/p2//p3/file', 'p3/file'), - ('c:/some/path/file', 'path/file'), - (r'c:\\some\\path\\file', 'path/file'), - ] - for loc, expected in test: - result = util.get_relative_path(loc, loc) - assert expected == result - - -class TestGetLocations(unittest.TestCase): - - def test_get_locations(self): - test_dir = get_test_loc('test_util/about_locations') - expected = sorted([ - 'file with_spaces.ABOUT', - 'file1', - 'file2', - 'dir1/file2', - 'dir1/file2.aBout', - 'dir1/dir2/file1.about', - 'dir2/file1']) - - result = sorted(util.get_locations(test_dir)) - result = [l.partition('/about_locations/')[-1] for l in result] - assert expected == result - - def test_get_about_locations(self): - test_dir = get_test_loc('test_util/about_locations') - expected = sorted([ - 'file with_spaces.ABOUT', - 'dir1/file2.aBout', - 'dir1/dir2/file1.about', - ]) - - result = sorted(util.get_about_locations(test_dir)) - result = [l.partition('/about_locations/')[-1] for l in result] - assert expected == result - - def test_get_locations_can_yield_a_single_file(self): - test_file = get_test_loc('test_util/about_locations/file with_spaces.ABOUT') - result = list(util.get_locations(test_file)) - assert 1 == len(result) - - def test_get_about_locations_for_about(self): - location = get_test_loc('test_util/get_about_locations') - result = list(util.get_about_locations(location)) - expected = 'get_about_locations/about.ABOUT' - assert result[0].endswith(expected) - - # FIXME: these are not very long/deep paths - def test_get_locations_with_very_long_path(self): - longpath = ( - 'longpath' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - '/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1/longpath1' - ) - test_loc = extract_test_loc('test_util/longpath.zip') - result = list(util.get_locations(test_loc)) - assert any(longpath in r for r in result) - - -class TestCsv(unittest.TestCase): - - def test_load_csv_without_mapping(self): - test_file = get_test_loc('test_util/csv/about.csv') - expected = [OrderedDict([ - ('about_file', 'about.ABOUT'), - ('about_resource', '.'), - ('name', 'ABOUT tool'), - ('version', '0.8.1')]) - ] - result = util.load_csv(test_file) - assert expected == result - - def test_load_csv_load_rows(self): - test_file = get_test_loc('test_util/csv/about.csv') - expected = [OrderedDict([ - ('about_file', 'about.ABOUT'), - ('about_resource', '.'), - ('name', 'ABOUT tool'), - ('version', '0.8.1')]) - ] - result = util.load_csv(test_file) - assert expected == result - - def test_load_csv_does_convert_column_names_to_lowercase(self): - test_file = get_test_loc('test_util/csv/about_key_with_upper_case.csv') - expected = [OrderedDict( - [('about_file', 'about.ABOUT'), - ('about_resource', '.'), - ('name', 'ABOUT tool'), - ('version', '0.8.1')]) - ] - result = util.load_csv(test_file) - assert expected == result - - def test_format_about_dict_for_csv_output(self): - about = [OrderedDict([ - (u'about_file_path', u'/input/about1.ABOUT'), - (u'about_resource', [u'test.c']), - (u'name', u'AboutCode-toolkit'), - (u'license_expression', u'mit AND bsd-new'), - (u'license_key', [u'mit', u'bsd-new'])])] - - expected = [OrderedDict([ - (u'about_file_path', u'/input/about1.ABOUT'), - (u'about_resource', u'test.c'), - (u'name', u'AboutCode-toolkit'), - (u'license_expression', u'mit AND bsd-new'), - (u'license_key', u'mit\nbsd-new')])] - - output = util.format_about_dict_for_csv_output(about) - assert output == expected - - -class TestJson(unittest.TestCase): - - def test_load_json(self): - test_file = get_test_loc('test_util/json/expected.json') - expected = [OrderedDict([ - ('about_file_path', '/load/this.ABOUT'), - ('about_resource', '.'), - ('name', 'AboutCode'), - ('version', '0.11.0')]) - ] - result = util.load_json(test_file) - assert expected == result - - def test_load_json2(self): - test_file = get_test_loc('test_util/json/expected_need_mapping.json') - expected = [dict(OrderedDict([ - ('about_file', '/load/this.ABOUT'), - ('about_resource', '.'), - ('version', '0.11.0'), - ('name', 'AboutCode'), - ]) - )] - result = util.load_json(test_file) - assert expected == result - - def test_load_non_list_json(self): - test_file = get_test_loc('test_util/json/not_a_list_need_mapping.json') - # FIXME: why this dict nesting?? - expected = [dict(OrderedDict([ - ('about_resource', '.'), - ('name', 'AboutCode'), - ('path', '/load/this.ABOUT'), - ('version', '0.11.0'), - ]) - )] - result = util.load_json(test_file) - assert expected == result - - def test_load_non_list_json2(self): - test_file = get_test_loc('test_util/json/not_a_list.json') - expected = [OrderedDict([ - ('about_file_path', '/load/this.ABOUT'), - ('version', '0.11.0'), - ('about_resource', '.'), - ('name', 'AboutCode'), - ]) - ] - result = util.load_json(test_file) - assert expected == result - - def test_load_json_from_abc_mgr(self): - test_file = get_test_loc('test_util/json/aboutcode_manager_exported.json') - expected = [dict(OrderedDict([ - ('license_expression', 'apache-2.0'), - ('copyright', 'Copyright (c) 2017 nexB Inc.'), - ('licenses', [{'key':'apache-2.0'}]), - ('copyrights', [{'statements':['Copyright (c) 2017 nexB Inc.']}]), - ('path', 'ScanCode'), - ('review_status', 'Analyzed'), - ('name', 'ScanCode'), - ('version', '2.2.1'), - ('owner', 'nexB Inc.'), - ('code_type', 'Source'), - ('is_modified', False), - ('is_deployed', False), - ('feature', ''), - ('purpose', ''), - ('homepage_url', None), - ('download_url', None), - ('license_url', None), - ('notice_url', None), - ('programming_language', 'Python'), - ('notes', ''), - ('fileId', 8458), - ]))] - result = util.load_json(test_file) - assert expected == result - - def test_load_json_from_scancode(self): - test_file = get_test_loc('test_util/json/scancode_info.json') - expected = [dict(OrderedDict([ - ('type', 'file'), - ('name', 'Api.java'), - ('path', 'Api.java'), - ('base_name', 'Api'), - ('extension', '.java'), - ('size', 5074), - ('date', '2017-07-15'), - ('sha1', 'c3a48ec7e684a35417241dd59507ec61702c508c'), - ('md5', '326fb262bbb9c2ce32179f0450e24601'), - ('mime_type', 'text/plain'), - ('file_type', 'ASCII text'), - ('programming_language', 'Java'), - ('is_binary', False), - ('is_text', True), - ('is_archive', False), - ('is_media', False), - ('is_source', True), - ('is_script', False), - ('files_count', 0), - ('dirs_count', 0), - ('size_count', 0), - ('scan_errors', []), - ]))] - result = util.load_json(test_file) - assert expected == result - - def test_format_about_dict_for_json_output(self): - about = [OrderedDict([ - (u'about_file_path', u'/input/about1.ABOUT'), - (u'about_resource', OrderedDict([(u'test.c', None)])), - (u'name', u'AboutCode-toolkit'), - (u'license_key', [u'mit', u'bsd-new'])])] - - expected = [OrderedDict([ - (u'about_file_path', u'/input/about1.ABOUT'), - (u'about_resource', u'test.c'), - (u'name', u'AboutCode-toolkit'), - (u'licenses', [ - OrderedDict([(u'key', u'mit')]), - OrderedDict([(u'key', u'bsd-new')])])])] - - output = util.format_about_dict_for_json_output(about) - assert output == expected + def test_get_relative_path_with_same_path_raise_exception(self): + try: + util.get_relative_path('/some/path/file', '/some/path/file') + self.fail('Exception not raised') + except AssertionError as e: + assert 'is the same as' in str(e) class TestMiscUtils(unittest.TestCase): @@ -542,39 +288,13 @@ def test_load_yaml_about_file_with_multiline(self): # notes: exceptio is rasied only for the first dupe assert 'Duplicate key in YAML source: owner' == str(e) - def test_ungroup_licenses(self): - about = [ - OrderedDict([ - (u'key', u'mit'), - (u'name', u'MIT License'), - (u'file', u'mit.LICENSE'), - (u'url', u'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:mit')]), - OrderedDict([ - (u'key', u'bsd-new'), - (u'name', u'BSD-3-Clause'), - (u'file', u'bsd-new.LICENSE'), - (u'url', u'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:bsd-new')]) - ] - expected_lic_key = [u'mit', u'bsd-new'] - expected_lic_name = [u'MIT License', u'BSD-3-Clause'] - expected_lic_file = [u'mit.LICENSE', u'bsd-new.LICENSE'] - expected_lic_url = [ - u'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:mit', - u'https://enterprise.dejacode.com/urn/?urn=urn:dje:license:bsd-new'] - lic_key, lic_name, lic_file, lic_url = util.ungroup_licenses(about) - assert expected_lic_key == lic_key - assert expected_lic_name == lic_name - assert expected_lic_file == lic_file - assert expected_lic_url == lic_url - def test_unique_does_deduplicate_and_keep_ordering(self): items = ['a', 'b', 'd', 'b', 'c', 'a'] expected = ['a', 'b', 'd', 'c'] results = util.unique(items) assert expected == results - def test_unique_can_handle_About_object(self): - base_dir = 'some_dir' + def test_unique_can_handle_Package_object(self): test = { 'about_resource': '.', 'author': '', @@ -587,16 +307,10 @@ def test_unique_can_handle_About_object(self): 'owner': 'nexB Inc.' } - a = model.About() - a.load_dict(test, base_dir) - - c = model.About() - c.load_dict(test, base_dir) - - b = model.About() - test.update(dict(about_resource='asdasdasd')) - b.load_dict(test, base_dir) + a = model.Package.from_dict(test) + c = model.Package.from_dict(test) + b = model.Package.from_dict(test) - abouts = [a, b] - results = util.unique(abouts) + packages = [a, b, c] + results = util.unique(packages) assert [a] == results diff --git a/tests/testdata/test_attrib/gen_default_template/attrib2.ABOUT b/tests/testdata/test_attrib/gen_default_template/attrib2.ABOUT new file mode 100644 index 00000000..ccf8df45 --- /dev/null +++ b/tests/testdata/test_attrib/gen_default_template/attrib2.ABOUT @@ -0,0 +1,2 @@ +about_resource: . +name: Apache Server diff --git a/tests/testdata/test_attrib/gen_default_template/expected_default_attrib.html b/tests/testdata/test_attrib/gen_default_template/expected_default_attrib.html index 3fba0f1f..dec734a1 100644 --- a/tests/testdata/test_attrib/gen_default_template/expected_default_attrib.html +++ b/tests/testdata/test_attrib/gen_default_template/expected_default_attrib.html @@ -1,6 +1,7 @@ - + +