peci: eSPI host stops responding

Model: CONFIG_PECI_OVER_ESPI=y
EC version: dd555b9
OS: Pop!_OS 22.04
Kernel: 6.2.7-060207-generic

Cases:

Unplugging a display connected to the TBT port causes PECI GetTemp() timeouts
Suspend/resume causes PECI GetTemp() timeouts

Steps to reproduce

Plug in a display to the TBT port
Power on system, booting to OS
Unplug the display

Expected behavior

PECI commands resume normal operation
CPU temp reporting is accurate

Actual behavior

PECI commands begin timing out and do not recover
CPU temp reported is always 0

Additional info

This specifically affects the PECI-over-eSPI implementation (CONFIG_PECI_OVER_ESPI=y).

Acknowledging the completion in #368 fixed it from hanging on shutdown.

Presumably, this could happen because of:

invalid/insufficient checks in peci_available()
missing the check to see if a transaction is currently in progress in peci_get_temp()
an issue specifically with eSPI usage?

Workaround when it gets in this state:

Completely power off system (not restart) and unplug from AC power

Blocks: #370

PECI over eSPI is now used by

addw4
lemp13
lemp13-b
oryp11
oryp12

While GetTemp is the most likely PECI call to fail, #525 shows that WrPkgConfig also can fail because of this.

Is there something I can do to help this move forward? I can reproduce it with my HW setup pretty easily so I can provide any additional debug logs if needed.

Any chance this will be fixed soon? Something we can help with? It is a really bad experience having a nice and expensive laptop such as the lemp13 that makes more noise than my washing machine.

@jaime-ez

@jaime-ez at this point I've workarounded this by implementing dbus service that switches fan to manual and a sliding bar changing fan speed manually for gnome-shell 🙈 far from ideal but I can share this if you're interested after some cleanup

Not nice, not packaged but I've added some instructions:

https://github.com/nextsux/ec/tree/manual-fan-control

for me, powering down, removing AC, and then powering back up fixes the problem. Annoying to do on a regular basis but not that big a deal.

@nextsux

@nextsux thank you sir! I will try your solution.

If folks at system76 don't solve it soon they should at least link to your solution in their docs https://support.system76.com/articles/fan-noise

Powering down the computer every time I change workstation is unacceptable.

@ywwg

@ywwg well.. I move with lemp13 daily. I usually have LOTS of work opened. And powering it off is really annoying. So I just live with my tool and manually change the fan speed. But I would say - totally sub-optimal.

I have full all the tools for de-bricking my laptop so I can (and I'm willing to) test any debugging version of EC firmware. I've also tried to study PECI specs so I can move this forward. But no luck. Without insights from @crawfxrd I'm completely lost here 😭 I can imagine there must be lots of things in his hands as EC seems to be one man show. But this is what the community is for. If not directly fixing, we can do testing, trying debug versions, provide logs. Also this is not a random issue. I can reproduce it in 100% of cases.

Btw whats interesting - I can reproduce it only using Philips display. My other displays do not cause this. But in this Philips display it's enough to go to a meeting. Display suspends and I came back to the laptop, that's burning hot.

Which brings me to - @ywwg no, it IS a big deal. I can not afford to reboot the laptop every time I leave it alone for just a few minutes and display suspends. (And no, thanks to some stupid policy, that display can not be set to disable this powersaving. I can set this on laptop. But then when I connect it at home, it will again - be shining into the night whole night there).

@ywwg well.. I move with lemp13 daily. I usually have LOTS of work opened. And powering it off is really annoying. So I just live with my tool and manually change the fan speed. But I would say - totally sub-optimal.

I have full all the tools for de-bricking my laptop so I can (and I'm willing to) test any debugging version of EC firmware. I've also tried to study PECI specs so I can move this forward. But no luck. Without insights from @crawfxrd I'm completely lost here 😭 I can imagine there must be lots of things in his hands as EC seems to be one man show. But this is what the community is for. If not directly fixing, we can do testing, trying debug versions, provide logs. Also this is not a random issue. I can reproduce it in 100% of cases.

Btw whats interesting - I can reproduce it only using Philips display. My other displays do not cause this. But in this Philips display it's enough to go to a meeting. Display suspends and I came back to the laptop, that's burning hot.

Just to add a data point, I have the issue with a Samsung display

Which brings me to - @ywwg no, it IS a big deal. I can not afford to reboot the laptop every time I leave it alone for just a few minutes and display suspends. (And no, thanks to some stupid policy, that display can not be set to disable this powersaving. I can set this on laptop. But then when I connect it at home, it will again - be shining into the night whole night there).

Hi, apologies if this is not the right place to make this comment, but today I updated to firmware 2025-07-24_c242738 on my lemp13-b and my fan just stopped working, take a look at the psensor screenshot, CPU temp is not even reported:

Has this issue been reported? Is there a workaround?

Has this issue been reported? Is there a workaround?

Try powering down, removing AC connection, wait, power back up

Thanks! I did it once after the firmware update, but apparently it takes 2 times. Now psensor shows the temp and the fan starts over 65 c

Any updates on this issue from system76? My RAM died and I'm beginning to think it might be related to overheating. We need a proper fix

A temporary workaround I use when the bug happens is to force the fan to rotate at full speed. You can do that hitting Fn+1. This is useful when the laptop is burning and I can’t restart it right now.

ec: Pin to PECI-over-eSPI recovery fix branch

@0xb0

I built an instrumented EC ROM that dumps the eSPI OOB engine registers on every peci_get_temp / peci_wr_pkg_config response timeout, to capture what state the engine is actually in during the permanent stall on lemp13-b. Reproduced with a USB-C/TBT DP monitor suspend+wake (100% reproducible here). Sharing because it narrows the root cause and, I believe, rules out the EC-side approach.

What the engine looks like during the permanent stall

Across 130+ consecutive response timeout lines the dump never changes:

peci oob state: C2CAC0=13 USCTRL0=0 OSCTRL0=0 OSCTRL4=6

(C2CAC0 = ESC2CAC0 @0x13, USCTRL0 = ESUCTRL0 @0xb0, OSCTRL0/OSCTRL4 = ESOCTRL0/ESOCTRL4.)

ESC2CAC0 = 0x13: OOB channel ENABLE (b0) and READY (b1) both set — channel is up.
ESUCTRL0 = 0x00: ENABLE / GO / CH_DISABLED / DONE / BUSY all clear — the upstream engine is idle and clean, not stranded and not busy.
ESOCTRL0 = 0x00: PUT_OOB STATUS = 0 — no downstream response has arrived.

Transient vs permanent are register-identical

The self-recovering "temp reads 0 for a few seconds then comes back" case shows the same C2CAC0=13 USCTRL0=0 OSCTRL0=0; one transient sample even caught OSCTRL0=0x80 (STATUS=1 — a response landing right at the ~10 ms timeout edge), occasionally with a short len 6 < 7. So from the EC's registers there is no difference between "will recover" and "stuck forever," and there is no EC-side state left to reset: the EC sends the request (upstream DONE fires, no upstream error), the channel stays enabled+ready, and the PCH simply stops sending the downstream PUT_OOB PECI response. Transient = PCH answers late; permanent = PCH never answers.

What I think this means

It lines up with the earlier note here that SMFI host commands also fail intermittently right after the TBT event and then recover, while only the OOB channel never re-establishes — i.e. a TBT/DP disruption briefly perturbs the whole eSPI host interface and the OOB responder doesn't come back.
Regarding peci: Fix eSPI host becoming unresponsive #530: since ESUCTRL0 already reads 0 in this state (not BUSY, DONE not stuck), "wait for upstream busy" and clearing ESUCTRL0 DONE act on an engine that is already idle, so I don't expect them to change the permanent stall — though they may still be correct hardening for the transaction start. I flashed an equivalent EC-side recovery and it did not change the symptom, which is consistent with this.
That points the root cause host-side: the PCH PECI-over-eSPI responder / coreboot eSPI OOB channel configuration / an Intel erratum around PECI-over-eSPI + TBT-DP hotplug. Consistent with addw4 (no TBT) not showing the bug.

Offer

I can reproduce this 100% (USB-C/TBT DP monitor suspend→wake), I have the de-brick tooling, and I'm happy to flash and run any instrumented EC build — e.g. dumping additional OOB/eSPI status, or trying an OOB channel re-arm sequence — to pin down where the responder wedges. If it's more useful on the coreboot/FSP side, tell me what to capture.

@nextsux

Another lemp13 data point, plus one finding that may be worth fixing independently of #530.

System: lemp13, firmware 2025-07-24_c242738 (EC a832be53), Arch Linux, kernel 7.2.x. I log system76_acpi temp1/pwm1/fan1 and coretemp package temp every 30s.

1. On current firmware the failure mode is fan off

When GetTemp starts timing out, my fan sits at 0% / 0 RPM while the package is at 90–104°C. Last episode it stayed that way for 18 hours, with only PROCHOT holding the CPU back. firmware-open#649 (same firmware) reports the same thing.

That looks like a lost fail-safe rather than part of the eSPI problem itself:

Before ceba69d7 ("Refactor reading thermals, updating fan duty"), peci_get_fan_duty() did this on a GetTemp error:
```
// Default to 50% if there is an error
peci_temp = 0;
duty = PWM_DUTY(50);
```
Since the refactor, peci_read_temp() stores 0 on error (src/app/main/peci.c, still the case on master 8faf1c2c), and fan_update_target() passes peci_temp straight into the fan curve (fan/step.c, fan/interp.c). 0°C is below the first curve point, so the result is 0% duty.

The constant 5923 RPM in #529 (firmware 2024-07-08) is exactly what my fan does at 50% duty, so that build probably still had the old default.

Restoring a fail-safe when PECI is available but GetTemp fails (the old 50%, or holding the last good reading) would make this failure noisy instead of hot, even before the root cause is found.

2. It can start with no suspend and no hotplug

2026-09-27, on AC, under sustained load: EC read 67°C at 16:24:52 and 0°C from 16:25:22 onward. The last resume was at 10:39:40, and the kernel logged nothing (no PM, USB-C/TBT, DRM or ACPI messages) between 16:20 and 16:26. Earlier episodes came after a few days with several S3 cycles. So on this unit it doesn't need a display unplug or a resume at the moment it fails.

3. Which resets clear it

Action	Clears it?
Warm reboot	No (EC still 0°C on the fresh boot)
Shutdown (S5) + power on, AC left plugged in, no button hold	Yes
AC unplug + power button hold	Yes

So on my unit a plain shutdown is enough and the EC doesn't need to lose power. That seems to point at the PCH / eSPI link re-init rather than state inside the EC.

4. Userspace can still run the fan while it's stuck

The EC keeps processing SMFI commands in this state. CMD_FAN_SET_MODE (PWM) plus CMD_FAN_SET_PWM, driven from coretemp, works: with the EC reading 0°C, 50% duty gave 5923 RPM and the package dropped 70→63°C in 15s. I'm running a small root daemon that takes over only while temp1 reads 0 and hands back FAN_MODE_AUTO as soon as the EC reads a real temperature again. It's similar in spirit to the manual-control branch @nextsux shared above.

Another lemp13-b on 2025-07-24_c242738 (EC a832be53), Pop!_OS 24.04, kernel 7.0.11. Found in this state today at ~127 h uptime: EC temp1 0 °C, fan 0 RPM, package 72–82 °C. Onset wasn't logged; this boot included at least one S3 cycle and a USB-C monitor/hub, so I can't say which one triggered it.

Confirming the SMFI path on this EC: system76_ectool built from a832be53, fan_mode pwm then fan_pwm 1 128 took the fan from 0 to ~5500 RPM in ~9 s and the package from 72 to 63 °C, with temp1 still reading 0. Two things that tripped me up: fan indices start at 1 (fan_pwm 0 … returns Protocol(1)), and fan_pwm is refused until the mode is already pwm.

+1 to restoring a fail-safe duty on GetTemp error, as described above. It turns a silent thermal throttle into a loud fan, whatever the eSPI root cause turns out to be, and it's the only mitigation that doesn't need root tooling on the host.

Another lemp13-b data point, on Debian 13 (kernel 7.1.13). On this unit a full cold reset did not clear it.

History

On firmware 2024-07-08_926f73d the fan sat at a constant ~5900 RPM (pwm1=128) while idle at ~51 °C, with system76_acpi temp1 = 0. That matches the old 50% fail-safe described above.
Updated to 2025-07-24_c242738 (EC a832be53) with system76-firmware-cli. On the first boot after flashing, temp1 was still 0, the fan was at 0 RPM, the package was at 83–87 °C idle, and it reached 103 °C after a few seconds of all-core load with the fan still at 0.

Resets tried

Action	Cleared?
Firmware flash (EC reflashed)	No
Warm reboot	No
Shutdown + AC unplugged + power button held 30 s + power on	No, temp1 = 0 again straight after boot

An external 1080p monitor is connected via USB-C (DRM connector DP-1) and was attached across all of these boots, which may be why it never recovers here.

Workaround, confirmed working
Same as above: system76_ectool built from a832be53 (the current master tool uses different command numbers, so fan_mode pwm returns Protocol(1) against this EC), then fan_mode pwm and fan_pwm 1 <duty>, driven from x86_pkg_temp. Under all-core load the fan reached ~9500 RPM and the package settled around 80 °C. My daemon only takes over while temp1 reads 0 and hands back FAN_MODE_AUTO once the EC reads a real temperature.

+1 for restoring a fail-safe duty when GetTemp fails. Silent 100 °C is a much worse failure than a loud fan.

Navigation Menu

Repository navigation

peci: eSPI host stops responding #369

Steps to reproduce

Expected behavior

Actual behavior

Additional info

1 remaining item

1. On current firmware the failure mode is fan off

2. It can start with no suspend and no hotplug

3. Which resets clear it

4. Userspace can still run the fan while it's stuck

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

peci: eSPI host stops responding #369

Description

Steps to reproduce

Expected behavior

Actual behavior

Additional info

Activity

crawfxrd commented on Apr 10, 2025

nextsux commented on May 18, 2025

jaime-ez commented on Aug 6, 2025

nextsux commented on Aug 6, 2025

ywwg commented on Aug 6, 2025

jaime-ez commented on Aug 6, 2025

nextsux commented on Aug 6, 2025

jaime-ez commented on Aug 6, 2025

1 remaining item

jaime-ez commented on Oct 28, 2025

ywwg commented on Oct 28, 2025

jaime-ez commented on Oct 28, 2025

heydemo commented on Jun 12, 2026

jaime-ez commented on Jul 28, 2026

liZe commented on Jul 28, 2026

jaime-ez commented on Jul 31, 2026

nextsux commented on Sep 15, 2026

Tuareg0xFFFF commented on Sep 28, 2026

1. On current firmware the failure mode is fan off

2. It can start with no suspend and no hotplug

3. Which resets clear it

4. Userspace can still run the fan while it's stuck

sam-at-dao commented on Oct 10, 2026

Wasylq commented on Nov 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions