Enhance `transform` to also work with JSON input

[chinyeungli]

The utility transform can only work with CSV file. Perhaps we should enhance it to also work with JSON file.

[srthkdb]

Hey! I am Sarthak Dubey, a 2nd-year computer science student from IIT Kanpur. I'd love to work on the project 'Improve AboutCode ToolKit with New and Enhanced Features' for gsoc'20. Can I take up this task for submitting a small patch as advised on the aboutcode gsoc home page?

[chinyeungli]

@srthkdb welcome to work on the issue. I think the current code has simple json format covered.
i.e. the transform should already covered json files generated from AboutCode tk.
Our next goal is to have it also cover the output from scancode.

[srthkdb]

@chinyeungli I think we'll have to find a way to figure out that the input json file is an output of scancode. For this, we can check the "tool_name" field in "headers". We also need to skip checking for essentials fields(about_resource, name), or do a different check. I think apart from this, we need to extract fields from "headers" and "files", and apply transformations. Am I going in the right direction, and is there anything else that needs to be done?

[srthkdb]

@chinyeungli I cannot figure out how to make required_columns and column_filters work for scancode inputs.
This is what I've written so far.
Added this to transform_json():

if(data["headers"][0]["tool_name"] == "scancode-toolkit"):
        new_data, errors = process_json_keys_scancode(data, renamings, transformer)

Wrote this function for scancode inputs

def process_json_keys_scancode(data, renamings, transformer):
    o_dict = OrderedDict()
    o_dict_headers_list = []
    o_dict_files_list = []
    new_data = []
    
    for item in data["headers"]:
        o_dict_headers = OrderedDict()
        for k in item.keys():
            if k in renamings.keys():
                for r_key in renamings.keys():
                    if k == r_key:
                        o_dict_headers[renamings[r_key]] = item[k]
            else:
                o_dict_headers[k] = item[k]
        o_dict_headers_list.append(o_dict_headers)


    for item in data["files"]:
        o_dict_files = OrderedDict()
        for k in item.keys():
            if k in renamings.keys():
                for r_key in renamings.keys():
                    if k == r_key:
                        o_dict_files[renamings[r_key]] = item[k]
            else:
                o_dict_files[k] = item[k]
        o_dict_files_list.append(o_dict_files)


    for k in data.keys():
        if k in renamings.keys():
            for r_key in renamings.keys():
                if k == r_key:
                    o_dict[renamings[r_key]] = data[k]
        else:
            o_dict[k] = data[k]
    
    if("files" in renamings.keys()):
        o_dict[renamings["files"]] = o_dict_files_list
    else:
        o_dict["files"] = o_dict_files_list
    if("headers" in renamings.keys()):
        o_dict[renamings["headers"]] = o_dict_headers_list
    else:
        o_dict["headers"] = o_dict_headers_list
    new_data = [o_dict]

    if transformer.column_filters:
        new_data = list(transformer.filter_columns(new_data))
    else:
        new_data = list(new_data)
    
    errors = transformer.check_required_columns(new_data, isFromScancode=True)

    return new_data, errors

Changed check_required_columns() a little:

def check_required_columns(self, data, isFromScancode=False):
        """
        Return a list of Error for a `data` list of ordered dict where a
        dict is missing a value for a required column name.
        """
        errors = []
        if(isFromScancode):
            required = set(self.required_columns)
        else:
            required = set(self.essential_columns + self.required_columns)
        if not required:
            return []

        for rn, item in enumerate(data):
            missings = [rk for rk in required if not item.get(rk)]
            if not missings:
                continue

            missings = ', '.join(missings)
            msg = 'Row {rn} is missing required values for columns: {missings}'
            errors.append(Error(CRITICAL, msg.format(**locals())))
        return errors

[srthkdb]

This works for column_renamings, but for column_filtering and required_columns, I either need to somehow know which fields are inside "headers" and "files", or I can use the same list of columns for filtering for all fields outside and inside "headers" and "files". For the former approach, I can keep an exhaustive list of field names for all three, but that won't work once the fields are renamed. For the latter, the user will need to pass all child fields along with parent fields, for example, if I need "files", I will all need to pass all it's child fields which is impractical. I have also not included child fields of licenses for column_renaming, I'll do it once you approve the above code, or I'll change it according to your advice.

[srthkdb]

There is one more thing I would like to ask regarding writing project proposal. Is the list of issues in URLs of "Improve AboutCode ToolKit with New and Enhanced Features" sufficient or are they just examples and I need to find more enhancements?

[chinyeungli]

@srthkdb I think you are in the right direction, and thanks for the work.
First of all, I think we need to spend some times on the objective/design.
i.e. for the JSON input, do we take all JSON format file as input, OR should we be very strict to only include JSON files generated from AboutCode TK and ScanCode TK?

What do you think/suggest?

For the column_filtering and required_columns, those will be applied after the renaming. Therefore, I don't think we need to do anything on those.

For the proposal, those URLs are the main focus. More enhancements are more than welcome. For instance, we also need to rename the "column" word in the code. This transform was originally design for CSV, so the word "column" is fine, but since we are also aming to deal with JSON, the "column" wording may not make sense and need to be updated etc...

Suggestions and thoughts are open and very welcome.

[srthkdb]

@chinyeungli Thanks for your suggestions, I have worked on them. Please see gitter https://gitter.im/aboutcode-org/aboutcode-toolkit