Repository navigation
fix: reduce lock contention in distributor channels - #10026
Merged
Merged
Conversation
Reduce lock contention in distributor channels via: - use atomic counters instead of "counter behind mutex" where appropriate - use less state - only lock when needed - move "wake" operation out of lock scopes (they are eventual operations anyways and many wake operations results in "futex wake" operations -- i.e. a syscall -- which you should avoid while holding the lock)
Contributor
Author
|
/benchmark |
Benchmark resultsBenchmarks comparing 5820507 (main) and b0f2504 (PR) |
Contributor
Author
|
/benchmark |
Benchmark resultsBenchmarks comparing 03d8ba1 (main) and b0f2504 (PR) |
Contributor
Author
|
Benchmarks are mostly noise, but locally in a server-like setting with multiple consumers and a limited number of tokio threads, I can totally see both a reduces lock contention (based on futex-waits) and higher performance. |
crepererum
marked this pull request as ready for review
April 11, 2024 08:46
3 of 9 tasks
Dandandan
previously approved these changes
Apr 11, 2024
Dandandan
left a comment
Contributor
There was a problem hiding this comment.
LGTM. I wonder if benchmarks show a difference on higher cpu count (instead of GH runners)?
Contributor
I will run them on my 8 core benchmark machine to see if I see a difference |
alamb
previously approved these changes
Apr 11, 2024
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me, thank you @crepererum and @Dandandan
I read the code carefully, and I think it looks good, though there is one area related to a potential race condition that I think would be worth reviewing
Also, given the trickiness of this code (with several inter-related mutexes) I think encapsulating the functionality into methods on Channel / ChannelState would make the flow easier to follow and verify. However, this is a personal preference and I don't think it is required.
I'll post my performance results here when I have them
|
|
||
| // does ANY receiver need data? | ||
| // if so, allow sender to create another | ||
| if this.gate.empty_channels.load(Ordering::SeqCst) == 0 { |
Contributor
There was a problem hiding this comment.
Is accessing this field before taking the send_wakers a race condition?
I think the answer is no, as self.channel.state.lock() is preventing empty_channels from changing between this line and the guard
However, if this is the case wouldn't it be clearer to not use an atomic usize here and just hang the count on channel state?
Contributor
Author
There was a problem hiding this comment.
However, if this is the case wouldn't it be clearer to not use an atomic usize here and just hang the count on channel state?
Reading this size should be cheap (=lockless). If you place that behind a mutex, all threads will have a single hot-spot critical section.
| n_senders: 1, | ||
| recv_alive: true, | ||
| recv_wakers: Vec::default(), | ||
| Arc::new(Channel { |
Contributor
There was a problem hiding this comment.
It would help me with readability / encapsulation to make this Channel::new() or Channel::default() so it is easier to follow the locking logic
Comment on lines
+235
to
+241
| let to_wake = guard_channel_state | ||
| .recv_wakers | ||
| .as_mut() | ||
| .expect("not closed"); | ||
| let mut tmp = Vec::with_capacity(to_wake.capacity()); | ||
| std::mem::swap(to_wake, &mut tmp); | ||
| tmp |
Contributor
There was a problem hiding this comment.
This might be more readable if it was in a function like
guard_channel_state.take_wakers()Or something - that would also give you a place to document what was happening more
Comment on lines
+171
to
+175
| if state | ||
| .data | ||
| .as_ref() | ||
| .map(|data| data.is_empty()) | ||
| .unwrap_or_default() |
Contributor
There was a problem hiding this comment.
Could we put this into ChannelState as a function with a more readable name? Like
if state.channel_gone() {
...
Contributor
|
Benchmark results. The only one that looks potentially problematic is clickbench 28 -- I'll rerun and see if I can reproduce the results |
Contributor
|
🤔 it seems like q28 is really quite a bit slower |
alamb
dismissed
their stale review
April 11, 2024 15:08
We need to investigate the performance regression
Dandandan
self-requested a review
April 12, 2024 06:35
Contributor
Author
|
I've pushed another change that makes the code prettier and also fixes a potential "Time-of-check to time-of-use" race condition. With that, I disagree on benchmark results: Or in other words: I think it's noise. You may wonder why my perf results are so bad: I'm testing on a Laptop but locked the frequency down to 1.5GHz to avoid thermal throttling, otherwise the numbers would be all over the place (and would also depend on previous runs, uptime and other factors too much). |
Contributor
Thanks @crepererum -- I am rerunning on my test machine (a VM on gcp) |
Contributor
|
In case anyone is interested, here is q28: SELECT
REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k,
AVG(length("Referer")) AS l,
COUNT(*) AS c,
MIN("Referer")
FROM 'hits.parquet'
WHERE "Referer" <> ''
GROUP BY k
HAVING COUNT(*) > 100000
ORDER BY l DESC
LIMIT 25;My unscientific run on my laptop without control also showed no difference with the current ode However, when I ran on my 8 core (relatively old) GCP VM I still see the difference. I am now trying to reproduce standalone and will profile |
Contributor
|
Sorry for the delay, I spent time trying to reproduce performance difference in this PR Reproducer
cat q28.sql
SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\\.)?([^/]+)/.*$', '\\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer")
FROM 'hits.parquet' WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;TestsBuild datafusion-cli with cd datafusion-cli
cargo build --release
cp target/release/datafusion-cli ~/Downloads/datafusion-cli-distributorAlso build it with merge-base for i in `seq 1 5`; do ./datafusion-cli-distributor -f q28.sql | grep Elapsed ; done
Elapsed 6.576 seconds.
Elapsed 6.591 seconds.
Elapsed 6.499 seconds.
Elapsed 6.599 seconds.
Elapsed 6.515 seconds. for i in `seq 1 5`; do ./datafusion-cli-merge-base -f q28.sql | grep Elapsed ; done
Elapsed 6.756 seconds.
Elapsed 6.761 seconds.
Elapsed 6.633 seconds.
Elapsed 6.565 seconds.
Elapsed 6.565 seconds.If anything this looks better for the distributor branch. I'll see if I can reproduce the results on the gcp machine |
Contributor
Author
|
If it helps, I can rebase this PR as well. |
alamb
approved these changes
Apr 25, 2024
alamb
left a comment
Contributor
There was a problem hiding this comment.
I re-ran the tests manually and can't see any significant difference
alamb@aal-dev:~$ for i in `seq 1 5`; do ./datafusion-cli-merge-base -f q28.sql | grep Elapsed ; done
Elapsed 28.206 seconds.
Elapsed 27.259 seconds.
Elapsed 27.456 seconds.
Elapsed 27.371 seconds.
Elapsed 27.130 seconds.
alamb@aal-dev:~$ ln -s arrow-datafusion11/datafusion-cli/target/release/datafusion-cli ./datafusion-cli-distributor
ln: failed to create symbolic link './datafusion-cli-distributor': File exists
alamb@aal-dev:~$ for i in `seq 1 5`; do ./datafusion-cli-distributor -f q28.sql | grep Elapsed ; done
Elapsed 27.382 seconds.
Elapsed 27.145 seconds.
Elapsed 27.302 seconds.
Elapsed 27.031 seconds.
Elapsed 27.338 seconds.Thus I think we should proceed with merging this PR and we can adjust it later if needed
Contributor
|
I merged this branch up from main, and plan to merge it once the CI checks pass |
ccciudatu
pushed a commit
to hstack/datafusion
that referenced
this pull request
Apr 26, 2024
* fix: lock contention in distributor channels Reduce lock contention in distributor channels via: - use atomic counters instead of "counter behind mutex" where appropriate - use less state - only lock when needed - move "wake" operation out of lock scopes (they are eventual operations anyways and many wake operations results in "futex wake" operations -- i.e. a syscall -- which you should avoid while holding the lock) * refactor: add more docs and tests for distributor channels --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
-
Rationale for this change
Reduce lock contention in distributor channels. Found while investigating runtime performance using
tokio2chrome(eBPF version).What changes are included in this PR?
Are these changes tested?
Existing tests pass. Benchmark didn't detect a regression.
Are there any user-facing changes?
Faster DataFusion.