Skip to content

Report progress when doing matching in d2d step #1298

Description

@pombredanne

When doing a JS and Java D2d on a large codebase, there is either little or no progress provided

2024-06-27 03:27:44.41 Step [match_directories_to_purldb] starting
2024-06-27 03:27:45.65 Matching 78,856 directories from to/ in PurlDB
2024-06-27 03:41:29.99 Progress: 10% (7886/78856) ETA: 7419 seconds (2.1 hours)
2024-06-27 04:00:04.20 Progress: 20% (15772/78856) ETA: 7754 seconds (2.2 hours)
2024-06-27 04:10:07.26 Progress: 30% (23657/78856) ETA: 5930 seconds (1.6 hours)
2024-06-27 04:30:06.73 Progress: 40% (31543/78856) ETA: 5612 seconds (1.6 hours)
2024-06-27 04:45:46.57 Progress: 50% (39428/78856) ETA: 4681 seconds (1.3 hours)
2024-06-27 05:03:54.35 Progress: 60% (47314/78856) ETA: 3846 seconds (1.1 hours)
2024-06-27 05:24:31.20 Progress: 70% (55200/78856) ETA: 3002 seconds (50.0 minutes)
2024-06-27 05:45:52.57 Progress: 80% (63085/78856) ETA: 2072 seconds (34.5 minutes)
2024-06-27 06:05:29.40 Progress: 90% (70971/78856) ETA: 1052 seconds (17.5 minutes)
2024-06-27 06:25:34.22 Progress: 100% (78856/78856)
2024-06-27 06:25:34.41 16,889 directories matched in PurlDB
2024-06-27 06:25:34.42 Step [match_directories_to_purldb] completed in 10670 seconds (3.0 hours)

for these I would like to see progress displayed at a fixed time interval, rather than based percentage completion.

and also

2024-06-27 06:25:34.42 Step [match_resources_to_purldb] starting
2024-06-27 06:25:36.44 Matching 201,361 .map, .js, .mjs, .ts, .d.ts, .jsx, .tsx, .css, .scss, .less, .sass, .soy, .class resources in PurlDB, using SHA1

for these I would like to see some progress displayed at a fixed time interval, with percentage completion. Nothing is displayed at all

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions