Generate kinds concurrently or in parallel

Currently we generate each kind one after the other, which can take awhile (we're approaching 5 min in Gecko now). Instead, we should generate the tasks for kinds in parallel.

We'll have to take kind-dependencies into account and come up with a way to synchronize workers such that we only generate a kind one all of its dependencies have also been generated.

Do we have any idea how much this would speed up Gecko taskgraph generation (I presume that would be the longest one).

No I'm not sure :(. It would be a lot of work to get the full benefit out of asyncio (e.g, we'd need to move from requests to aiohttp, use async file system access, figure out how to yield control when accessing the build system, etc). All this work could happen incrementally over time, but it's hard to say what the end result would be.

@ahal

I love async and I could see the benefits of it in scriptworker for instance. Like you raised in the team's meeting @ahal, going async may raise the entry barrier even higher. A year ago, I checked how many people contributed to the taskcluster folder in Gecko. The numbers were amazing: 300 contributors in total and about 30 core contributors.

However, outside of Gecko, the numbers aren't that high. I get the feeling Releng owns many taskgraph configuration. If next year, we want to be more hands-off, I don't think we should go async yet.

For reference, the Android decision task takes 5 minutes too. It's mainly because we shell out to time-consuming process (bug 1788606). Going async wouldn't help this case.

It might still be possible to support both async and sync transforms. The problem is that async transforms return async generators, which can't be iterated over in a synchronous transform.

We could either say "no sync transforms allowed after any async ones" (I don't love this). Or we could convert the async generator to a list prior to passing it into a sync transform. This would negate the benefits of using a generator, but maybe that's a fair trade-off.

If we supported both, I'd envision all core transforms would be async (so contribution barrier would be higher there still), but then consumers could choose whether to bother with async or not.

It's also worth noting that we're not talking hugely more difficult here. Basically just s/def/async def and s/for/async for.

But I agree we can sit on this for now.

@djmitche

It might still be possible to support both async and sync transforms. The problem is that async transforms return async generators, which can't be iterated over in a synchronous transform.

We could either say "no sync transforms allowed after any async ones" (I don't love this). Or we could convert the async generator to a list prior to passing it into a sync transform. This would negate the benefits of using a generator, but maybe that's a fair trade-off.

I think @djmitche took an approach of creating synchronous wrappers to all outward-facing async functions, e.g.

async def do_stuff_async(*args, **kwargs):
  ...

def do_stuff_sync(*args, **kwargs):
    return asyncio.run(do_stuff_async(*args, **kwargs)

which allows for all of the library to be async, but for downstream consumers to use functions synchronously.

I also played around with generating kinds in parallel rather than concurrently:
https://phabricator.services.mozilla.com/D239805

It didn't seem to improve things at all. I'm not sure if this is overhead from serializing / deserializing the loaded_tasks, or process overhead or what. I even made an optimization to only pass in kind_dependencies_tasks rather than all tasks down to the worker processes. But this only got us back to par.

Ok, I found a bottleneck in my logic, after fixing that I can shave off about ~10s of generation time on my machine. That's about a 23% improvement with ./mach taskgraph full --fast and a 10% improvement with ./mach taskgraph full. This number makes sense to me because tests are by far the long poll here, so we can only improve by however long it takes to generate non-test tasks, which are typically quite fast.

Still, a 10s improvement is nothing to sneeze at for the developer sitting at their terminal waiting for their ./mach try to finish...

Not to move tho goalposts, but this also makes me wonder if there are ways we can (safely) concurrently process entries in an individual kind? Offhand, I don't think there are any dependencies or data passed from one task in a kind to the next? (There's a ton of tasks in Gecko's test kind, so that would presumably be pretty impactful if it's possible.)

I bet there are test harnesses/flavors that don't depend on each other that might be able to be split out. There might be side effects to this, but hopefully detectable via a taskgraph diff (and maybe a search of the test code to make sure there aren't any unexpected hardcodes?)

Though I'm now thinking mach might have some test kind hardcodes... maybe create a TEST_KINDS tuple that specifies which kinds are tests.

Is the bottleneck IO or CPU?

IO, reading the moz.build/test manifest files to generate a list of all tests and waiting for the ML service to say which tests to run are the two main things. Neither of these things are enabled with --fast, so they don't affect people waiting at the cli too much.

I think the holy grail here is going full asyncio and using Python 3.13t (free threaded aka no GIL). That's probably still a few years away from being stable enough for our use case (but maybe worth trying just to see?). But then we wouldn't need to worry about explicitly parallelizing things within a kind. It would just be automagic.

So personally I think steering towards that is probably the best bet rather than bothering to parallelize within a kind.. Though if we were going to parallelize within a kind, doing so across each tasks-from (e.g taskcluster/kinds/test/mochitest.yml) would be an obvious hook point to do it.

Oh and I should link the profile that started me reviving this patch in the first place (thanks to florian):
https://share.firefox.dev/4ihfT0q

There's some other low hanging fruit like using a faster JSON library and optimizing the perf tasks.

I meant splitting test into multiple kinds, but that might also take a lot of effort.

I meant splitting test into multiple kinds, but that might also take a lot of effort.

I'm not sure that would be necessary. I don't think it's even possible to have dependencies between individual tasks in a kind. (

taskgraph/src/taskgraph/generator.py

Lines 76 to 89 in 63ec732

    
           tasks = [ 
        
               Task( 
        
                   self.name, 
        
                   label=task_dict["label"], 
        
                   description=task_dict["description"], 
        
                   attributes=task_dict["attributes"], 
        
                   task=task_dict["task"], 
        
                   optimization=task_dict.get("optimization"), 
        
                   dependencies=task_dict.get("dependencies"), 
        
                   soft_dependencies=task_dict.get("soft-dependencies"), 
        
                   if_dependencies=task_dict.get("if-dependencies"), 
        
               ) 
        
               for task_dict in transforms(trans_config, inputs) 
        
           ]

is where we iterate over those; unless trans_config or inputs changes while iterating, each task will get the same inputs regardless of the order things are called in. There's also no sorting of these tasks first, which is further evidence there's no dependencies.)

I meant splitting test into multiple kinds, but that might also take a lot of effort.

I'm not sure that would be necessary. I don't think it's even possible to have dependencies between individual tasks in a kind. (

taskgraph/src/taskgraph/generator.py

Lines 76 to 89 in 63ec732
tasks = [
Task(
self.name,
label=task_dict["label"],
description=task_dict["description"],
attributes=task_dict["attributes"],
task=task_dict["task"],
optimization=task_dict.get("optimization"),
dependencies=task_dict.get("dependencies"),
soft_dependencies=task_dict.get("soft-dependencies"),
if_dependencies=task_dict.get("if-dependencies"),
)
for task_dict in transforms(trans_config, inputs)
]
is where we iterate over those; unless trans_config or inputs changes while iterating, each task will get the same inputs regardless of the order things are called in. There's also no sorting of these tasks first, which is further evidence there's no dependencies.)

I split this part off into #716. If I understand the profile linked above, we spent ~40% of ./mach taskgraph time in transforms. Parallelizing those calls within a kind across cores seems likely to help, especially considering well over half of the tasks in Gecko are in one kind at the moment.

IO, reading the moz.build/test manifest files to generate a list of all tests

Assuming we load distinct manifests for different tests, #716 ought to help with this part, and a very quick dump of manifests by test tasks suggests that this is at least partly the case.

(AFAICT all of this loading is done by calls to active_tests, which is called by the test transforms.)

@ahal

This was achieved in #738, which was largely built on top of #5 (thanks @ahal!)

And since it was mentioned here...

I meant splitting test into multiple kinds, but that might also take a lot of effort.

I ended up filing https://bugzilla.mozilla.org/show_bug.cgi?id=1982403 about this, and https://bugzilla.mozilla.org/show_bug.cgi?id=1982404 as a first step to move one type of test out.

multiprocessing was backed out in #746 due to not working properly on Windows (see the ticket for the gory details).

Based on that experience, I don't think any solution that involves extra processes would work out without major refactoring or hacks. If I understand the lay of the land correctly, that suggests we should look at using multiple threads, and adopting free-threaded python when possible to make use of multiple cores.

As a quick experiment, I used the work from #738 and replaced the ProcessPoolExecutor with a ThreadPoolExecutor and ran it with 3.13+3.13 freethreaded, thinking that it might be a cheaper way to get some wins. The results were not encouraging:

Python	taskgraph	result
3.13	15.0.1	1:15:73
3.13 free threaded	15.0.1	1:16:56
3.13	15.0.1 w/ ThreadPoolExecutor	1:18:09
3.13 free threaded	15.0.1 w/ ThreadPoolExecutor	1:15:91

...so no easy wins, at least with the current free threaded python.

Multiprocess support ended up being relanded in #765.

I've also been looking at multithreading a bit more. With 3.14 free-threaded python perf improved significantly, but it's still not on par with multiprocess.

Navigation Menu

Repository navigation

Generate kinds concurrently or in parallel #5

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	tasks = [
	Task(
	self.name,
	label=task_dict["label"],
	description=task_dict["description"],
	attributes=task_dict["attributes"],
	task=task_dict["task"],
	optimization=task_dict.get("optimization"),
	dependencies=task_dict.get("dependencies"),
	soft_dependencies=task_dict.get("soft-dependencies"),
	if_dependencies=task_dict.get("if-dependencies"),
	)
	for task_dict in transforms(trans_config, inputs)
	]

Generate kinds concurrently or in parallel #5

Description

Activity

bhearsum commented on Nov 22, 2022

ahal commented on Nov 22, 2022

JohanLorenzo commented on Nov 23, 2022

ahal commented on Nov 23, 2022

ahal commented on Nov 23, 2022

escapewindow commented on Nov 23, 2022

ahal commented on Feb 26, 2025

ahal commented on Feb 26, 2025

bhearsum commented on Feb 27, 2025

escapewindow commented on Feb 27, 2025

djmitche commented on Feb 27, 2025

ahal commented on Feb 27, 2025

ahal commented on Feb 27, 2025

escapewindow commented on Feb 27, 2025

bhearsum commented on Feb 27, 2025

bhearsum commented on Jul 2, 2025

bhearsum commented on Jul 2, 2025

bhearsum commented on Aug 12, 2025

bhearsum commented on Aug 14, 2025

bhearsum commented on Aug 14, 2025

bhearsum commented on Oct 21, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions