Required phrases cleanup before dataset construction

[Kaushik-Kumar-CEG]

Found 606 {{ }} required phrase marker changes to be made across 10,601 annotated rules that will add noise to NER training data if left as it is

Check	Count
Marker cuts a word boundary	1
HTML entity inside marker	2
Trailing punctuation inside marker	166
Leading/trailing space inside marker	312
URL/filepath in marker to be changed	125

Full report: https://gist.github.com/Kaushik-Kumar-CEG/cc4b89a2253f818bb749c45a4026eea1

Will submit changes as a PR

[armijnhemel]

Although I think that the required phrases need a lot of work I personally don't think that what you are proposing is the right approach (but will defer to @AyanSinhaMahapatra and @pombredanne ). There are people using this data for different purposes and they might need the original data (I am one of them). Changing the data so it suits your needs (which I am concluding from your work) would interfere with my use case.

When reading the gist it seems that you are already anticipating how the clean up works. For example, you say:

The last character inside }} is punctuation (. , ; :). The tokenizer strips these before matching, so they add no value but create noisy entity boundaries for NER training

Doing the clean up beforehand is a bad idea, because if the clean up method changes, then the rules might no longer be useful, as they have been pre-cleaned for a different clean up method. I think that the rules should always stay as close to the original source as possible, including punctuation, and so on.

And this is also why #4953 is important.

When you say:

There is a space after {{ or before }}. The tokenizer ignores these, so they don't help matching but shift BIO label boundaries in NER training data

I would suggest to clean up the data before feeding it into your machine learning, instead of cleaning up the rule data beforehand.

{{ }} marker splits a word

It doesn't split a word. What happened is @pombredanne having fat fingers and inserting the letter i where it shouldn't have been inserted. This is a genuine bug which was introduced in e50169f and should be addressed. Good find.

These rules already have a license name marker, so the filepath marker is redundant. Removing it will not affect matching since the license name marker is already present

No, these are not redundant. I very often see the sequence matcher (which should be made optional) report the wrong license. The more required phrases, even if it is a file path, the better.

The same for this:

These rules already have a license name marker, so the URL marker is redundant. The URL adds no safety, only fragility

The URLs add so much more safety to protect against the sequence matcher (which should be made optional).

Finally:

URLs and filepaths are references, not license names. Labeling them as entities trains the model to extract URLs instead of license text, causing false positives at inference

If this is the case, then there is something wrong on your end. As said in the beginning mangling the data so it fits your purpose is not the solution. I think you should add an extra step that takes the original data, cleans it up and then feeds it to your program. Because you have identified the files that are problematic for you it seems you have already done that work :-)

[AyanSinhaMahapatra]

I agree with everything you said @armijnhemel 🙇 Thanks for the detailed comment.

Re-iterating some points made here as they are important:

• the rule data should be present as close to the original example, and should not be cleaned up.
• I would suggest to clean up the data before feeding it into your machine learning, instead of cleaning up the rule data beforehand.
• URL/Filepath in markers, these are necessary and not to be removed. But actually we need to also train the model to consider and mark this as required phrases. (This is one of the improvement points I made during your initial presentation @Kaushik-Kumar-CEG )

[Kaushik-Kumar-CEG]

Thanks for the detailed feedback @armijnhemel and @AyanSinhaMahapatra, I appreciate it

I'll keep the rule data as it is and handle all normalization in my preprocessing pipeline before training

To clarify my reasoning, I wasn't proposing to change any rule text, only the position of {{ }} boundaries. The rules are already inconsistent in how these are placed, for example {{Apache License.}} and {{Apache License}}. both exist across different rules, same with {{ MIT }} vs {{MIT}}. These weren't deliberate choices, just different people annotating at different times. Since the tokenizer strips punctuation and whitespace anyway, both forms produce identical required phrase tokens and identical matching behavior. I thought standardizing them would be a harmless cleanup as they are functionally the same. But I understand the concern about keeping data stable, so I'll handle it on my end instead

Regarding URLs/filepaths - understood, I'll train the model to recognize these as valid required phrases too

[armijnhemel]

To clarify my reasoning, I wasn't proposing to change any rule text, only the position of {{ }} boundaries. The rules are already inconsistent in how these are placed, for example {{Apache License.}} and {{Apache License}}. both exist across different rules, same with {{ MIT }} vs {{MIT}}. These weren't deliberate choices, just different people annotating at different times. Since the tokenizer strips punctuation and whitespace anyway, both forms produce identical required phrase tokens and identical matching behavior. I thought standardizing them would be a harmless cleanup as they are functionally the same.

It is a valid point and yes, there is a lot of inconsistency. Currently there are no clear guidelines (or any guidelines) or best practices when it comes to writing these rules. This is definitely something that should be addressed in the future. I am not opposed to moving the boundaries a bit, but not without having established clear guidelines first.

[AyanSinhaMahapatra]

or example {{Apache License.}} and {{Apache License}}. both exist across different rules, same with {{ MIT }} vs {{MIT}}.

This is fine to have a standardized way of having these required phrase boundaries, I also think it's better to have an automatic linter/script for this in addition to having a guidelines to better enforce and apply these rules.

Let's add these guidelines to this PR where I'm updating the licenses/rule related documentation:

• Improve how-to-guides related to license data updates #4789

[pombredanne]

I agree with the comments here: changing rules is not welcomed, but you can likely apply any kind of preprocessing you like in your code, since you are already able to find these issues.

@Kaushik-Kumar-CEG BTW, https://gist.github.com/Kaushik-Kumar-CEG/cc4b89a2253f818bb749c45a4026eea1 looks like AI-generated to me.

• please put the reporting code that you created or generated somewhere and link it here
• please disclose the AI tool you have used too

[Kaushik-Kumar-CEG]

@pombredanne Understood. I ran analysis scripts on the rule files and collected raw data

I used Claude to help format the findings into the clean report. I'll disclose AI usage going forward