Scan Fails with Timeout for Specific Binaries Despite Extended Timeout Setting

[rihanasahul]

Description:

Hi team,

I'm encountering a timeout error when scanning a few binary files for licenses using ScanCode. Here's what's happening:

• Some binaries fail with a timeout error (120s).

• I increased the timeout value to 1 hour, but ScanCode still waits for the full hour and then throws the same timeout error.

• Interestingly, other binaries (even larger ones) scan successfully all together in under 40 minutes.

• For example, a failed binary of size ~103 MB was tried individually, but still failed after 1 hour with a timeout.

This behavior seems inconsistent, and I’m wondering what might be causing it or how to debug further.

I’m happy to provide more general information or logs if needed.

Thanks in advance for your help!

[jloehel]

@rihanasahul I am seeing some similar behavior. When I use the copyright plugin. In example for dictionary.bin in corefx: https://github.com/mono/corefx/blob/da5dd2beafb947325e3948ae54844797724bb6d4/src/Native/AnyOS/brotli/common/dictionary.bin.

@rihanasahul For debugging you can set these env variables:

• SCANCODE_DEBUG_COPYRIGHT_DEEP
• SCANCODE_DEBUG_COPYRIGHT_TOKEN

to True.

For me I could trace it down to one matcher in the lexer of pygmars (https://github.com/aboutcode-org/pygmars/blob/main/src/pygmars/lex.py#L153C20-L153C27.):

Label: 'YR'

Matcher: '^([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\
\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0-9]|(20\
[0-3][0-9][\\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\
\\\\&nbsp)*)+(((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9][0-9][\
\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-])+20[0-\
3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*)|(((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\\
]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\
\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*)\
(([6-9][0-9]|[0-][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*)*))*$'

Token Value: 201420152016201720182019202020212022202320242025202620272028202920302031203220332034203520362037201320122011201020092008200720062005200420032002200120001999199819971996199519941993199219911990198919881987198619851984198319821981198019791978197719761975197419731972197119701969196819671966196519641963196219611960195919581957195619551954195319521951195010001024139400009999comomasesteestaperotodohacecadaanobiendiaasividacasootroforosolootracualdijosidograntipotemadebealgoqueestonadatrespococasabajotodasinoaguapuesunosantediceluisellamayozonaamorpisoobraclicellodioshoracasizanaomrarutanepootiznodotozheonikhNaeebymyVysovyvoNoobPoliniRFNeMytyOnimdaZaDaNuObteIzeinummTyuzhfy'nmm`kl'wrdyf~hwlmlkwlhbslnhy'yqdhlthmbhlwlyblybkshym'mntbylnHbhmmshwshfirstvideolightworldmediawhitecloseblackrightsmallbooksplacemusicfieldorderpointvalueleveltableboardhousegroupworksyearsstatetodaywaterstartstyledeathpowerphonenighterrorinputabouttermstitletoolseventlocaltimeslargewordsgamesshortspacefocusclearmodelblockguideradiosharewomenagainmoneyimagenamesyounglineslatercolorgreenfront&watchforcepricerulesbeginaftervisitissueareasbelowindextotalhourslabelprintpressbuiltlinksspeedstudytradefoundsenseundershownformsrangeaddedstillmovedtakenaboveflashfixedoftenotherviewschecklegalriveritemsquickshapehumanexistgoingmoviethirdbasicpeacestagewidthloginideaswrotepagesusersdrivestorebreaksouthvoicesitesmonthwherebuildwhichearthforumthreesportpartyClicklowerlivesclasslayerentrystoryusagesoundcourtyour

That is this pattern:

scancode-toolkit/src/cluecode/copyrights.py

Lines 2149 to 2155 in 4b57a7f

	(r'^' + _PUNCT + _YEAR_OR_YEAR_YEAR_WITH_PUNCT + '+' +
	'(' +
	_YEAR_OR_YEAR_YEAR_WITH_PUNCT +
	'|' +
	_YEAR_THEN_YEAR_SHORT +
	')*' + '$', 'YR'),

@pombredanne Any chance to improve the pattern? The issue is the counting of years in the beginning.

>>> def time_regex(test_string):
...     pattern = re.compile('^([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\.\
/:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0\
-9]|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\\
}\\~i]|\\\\&nbsp)*)+(((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9\
][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-]\
)+20[0-3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*)|(((19[6-9][0-9]|20[0-3][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\\
[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*|((19[6-9][0-9][\\.,\\-]_)+[6-9][0-9]|(19[6-9][0-9][\\.,\\-])+[0-9]|(20[0-3][0-9][\\.,\\-])+[0-2][0-9]|(20[0-3][0-9][\\.,\\-])+[0-9]|(20[0-3]\
[0-9][\\.,\\-])+20[0-3][0-9]|(20[0-3][0-9][\\.,\\-])+20[0-3]x|(20[0-3][0-9][\\.,\\-])+20[0-3][0-9]a)([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&\
nbsp)*)(([6-9][0-9]|[0-][0-9])([!"\\#\\$%\\&\'\\(\\)\\*\\+,\\-\\./:;<=>\\?@\\[\\\\\\]\\^_`\\{\\|\\}\\~i]|\\\\&nbsp)*)*))*$')
...     pattern.match(test_string)
...
>>> def timer(func, string):
...     t = timeit.Timer(setup='from __main__ import '+ func, stmt=func+'("'+ string +'")')
...     print(func.ljust(16)+':', t.timeit(number=20000))
...
>>> timer('time_regex', "comomasesteestaperotodohacecadaanobiendiaasividacasootroforosolootracualdijosidograntipotemadebealgoqueestonadatrespococasabajotodasinoaguapuesunosantedice\
luisellamayozonaamorpisoobraclicellodioshoracasizanaomrarutanepootiznodotozheonikhNaeebymyVysovyvoNoobPoliniRFNeMytyOnimdaZaDaNuObteIzeinummTyuzhfy'nmm`kl'wrdyf~hwlmlkwlhbslnhy'yqd\
hlthmbhlwlyblybkshym'mntbylnHbhmmshwshfirstvideolightworldmediawhitecloseblackrightsmallbooksplacemusicfieldorderpointvalueleveltableboardhousegroupworksyearsstatetodaywaterstartst\
yledeathpowerphonenighterrorinputabouttermstitletoolseventlocaltimeslargewordsgamesshortspacefocusclearmodelblockguideradiosharewomenagainmoneyimagenamesyounglineslatercolorgreenfr\
ont&watchforcepricerulesbeginaftervisitissueareasbelowindextotalhourslabelprintpressbuiltlinksspeedstudytradefoundsenseundershownformsrangeaddedstillmovedtakenaboveflashfixedofteno\
therviewschecklegalriveritemsquickshapehumanexistgoingmoviethirdbasicpeacestagewidthloginideaswrotepagesusersdrivestorebreaksouthvoicesitesmonthwherebuildwhichearthforumthreesportp\
artyClicklowerlivesclasslayerentrystoryusagesoundcourtyour")
time_regex      : 0.02591296799982956
>>> timer('time_regex', '20142015201620172018201920202021202220232024202520262027202820292030203120322033203420352036203720132012201120102009200820072006200520042003200220012000199\
9199819971996199519941993199219911990198919881987198619851984198319821981198019791978197719761975197419731972197119701969196819671966196519641963196219611960')
time_regex      : 0.19425528799911262
>>> timer('time_regex', '20142015201620172018201920202021202220232024202520262027202820292030203120322033203420352036203720132012201120102009200820072006200520042003200220012000199\
91998199719961995199419931992199119901989198819871986198519841983198219811980197919781977197619751974197319721971197019691968196719661965196419631962196119601959')
>>> # Runs and runs and runs and causes the timeout

[pombredanne]

@jloehel @rihanasahul Thanks for the report. We should not time out and we need to fix this!

@jloehel do you know how to fix this?

[jloehel]

Hi @pombredanne, you are welcome. I am not sure yet. It's a massive regex. Can you please point me to some test cases the regex needs to fulfill.

[CRaNkXD]

I have probably found another file with the same issue:
https://github.com/dotnet/runtime/blob/v8.0.15/src/native/external/brotli/common/dictionary.bin
Line 30 is the problematic one.