Repository navigation
Regex for detecting language codes incorrect #43
Description
System Info
Ubuntu, PHP 8.1.2
PHP Version
8.1.2
Environment/Platform
- Command-line application
- Web application
- Serverless
- Other (please specify)
Description
In the Codewithkyrian\Transformers\PretrainedTokenizers\NllbTokenizer class this regex is used to detect language codes: /^[a-z]{3}_[A-Z]{3}$/
However some models, like Xenova/nllb-200-distilled-600M use a format like eng_Latn (full list)
I would suggest something like /^[a-z]{3}_[a-zA-Z]{3,4}$/
Is there a big penalty to false positives here? Is this check required?
Reproduction
$trans = pipeline('translation', 'Xenova/nllb-200-distilled-600M');
$trans('Translation test', srcLang: 'eng_Latn', tgtLang: 'deu_Latn');
Activity
CodeWithKyrian commentedon Jul 15, 2024
You're right, the current regex does not accommodate the formats provided by this particular model. Didn't get to test with it so thank you for bringing this to my attention.
In this context, I don't see any significant penalty for false positives, so sure, your suggested regex
/^[a-z]{3}_[a-zA-Z]{3,4}$/would be more inclusive for different language code formats.I appreciate your contribution and will incorporate this improvement. Thank you!
Thorry84 commentedon Jul 16, 2024
Thanks so much for your work! <3