Repository navigation
tailscaled is unsupervised: a daemon panic leaves /etc/resolv.conf pointing at MagicDNS and breaks all DNS in the container #61
Description
What happened
tailscaled (1.102.3, installed by this feature) panicked inside a long-running dev container and was never restarted. Because it crashed rather than shut down, it did not restore /etc/resolv.conf, which it had pointed at MagicDNS. The container was left with:
nameserver 100.100.100.100
nameserver fd7a:115c:a1e0::53
and no daemon to answer it, so every DNS lookup in the container black-holed: getent hosts registry.npmjs.org failed, npm/pnpm failed with EAI_AGAIN, and curl hung for its full timeout. The container itself stayed up and looked healthy, so this went unnoticed for hours.
From /var/log/tailscaled.log:
health(warnable=derp-timed-out): error: Tailscale hasn't heard from the '<region>' relay
server in 5m51s. The server might be temporarily unavailable, or your Internet
connection might be down.
panic: ipnlocal: watchdog timeout
goroutine 240320 [running]:
tailscale.com/ipn/ipnlocal.(*LocalBackend).reportDeadlock(0x...?)
tailscale.com@v1.102.3/ipn/ipnlocal/watchdog.go:146 +0xb2
created by time.goFunc
time/sleep.go:215 +0x2d
Two containers on the same host, both using this feature, panicked within the same minute — one panic each, both preceded by the same DERP stall. So whatever the underlying deadlock is, it is reachable from an ordinary network blip and can hit every container at once.
Why this is a feature-level problem
The panic itself belongs to tailscale/tailscale, but the blast radius is this feature's, for two reasons:
- Nothing supervises the daemon.
tailscaled-devcontainer-startrunsexec setsid /usr/local/sbin/tailscaledin a backgrounded subshell and never looks at it again, andtailscaled-entrypointcalls that script once beforeexec "$@". Iftailscaledexits for any reason, it stays exited for the life of the container. - The crash takes DNS down with it, and the fix is already on disk. The feature's own backup,
/etc/resolv.pre-tailscale-backup.conf, was intact and correct the whole time. Nothing consults it unless tailscaled shuts down cleanly.
The result is the worst failure shape: a container whose networking is silently broken in a way that looks like a registry outage, a proxy problem, or a broken image.
Suggestions
- Supervise
tailscaled— a restart loop in the start script, or run it under whatever init the image has. A crash then costs seconds of connectivity instead of the rest of the container's life. - Failing that, make the state self-healing: on start (and ideally periodically), if
tailscaledis not running,/etc/resolv.confnames100.100.100.100, and/etc/resolv.pre-tailscale-backup.confexists, restore the backup. That alone converts "all DNS is dead" into "the tailnet is down", which is both recoverable and diagnosable.
Possibly the same root cause as #57 (rebuild leaves DNS broken, backup missing) and #11.
Environment
- Feature:
ghcr.io/tailscale/codespace/tailscale - tailscale/tailscaled 1.102.3 (commit 9329c3677031109ff6d0b80abee0cddc8f35ff6f)
- Base image: Ubuntu 22.04.5 LTS, Docker (not Codespaces), containers up 7-14 days
Activity
bradfitz commentedon Sep 16, 2026
This is pretty painful-to-read slop and is already addressed by a half dozen features (use systemd-resolved to never hit this problem, this is why the tailscaled.service does
ExecStopPost=/usr/sbin/tailscaled --cleanup, this is why we have the dnsresolvecache / dnsfallback as a last resort, to boot up and fix things...)The real interesting part here would be why it crashed at all. That deadlock information would be the interesting part.
I guess in human speak, your request is that the codespace container should have a wrapper process around it to restore /etc/resolv.conf in case crashes (deadlocks in this case) so codespaces can continue to ... do something?
But really I care more about that truncated deadlock stack. Still have it?
nocnokneo commentedon Sep 16, 2026
Part of this might be a misunderstanding of the goal of this project. I know the repo is called "codespace" but the about line says "Tailscale for devcontainers and codespaces" so I assumed that it's intended to support devcontainers more generally which is how I'm using it (directly with the devcontainer CLI). I have a simple container that execs
sh. I could run dbus and resolved myself under something that keeps them alive, but I don't think I should have to. Besides, someone running something like Alpine wouldn't have a reasonable path to doing something like that.Having Tailscale running inside my container enables access to specific resources on our tailnet, but the container is largely still functional with basic internet connectivity, so yes, something that restores
/etc/resolv.confwould be useful (easier to recover and diagnose). But really, I would prefer if tailscaled were restarted (at least as first-order recovery attempt).container2-tailscaled.log
container1-tailscaled.log
bradfitz commentedon Sep 16, 2026
Yeah, most of the features I mentioned aren't really relevant in the codespaces context.
In any case, the real problem is it crashing at all. A secondary dream is changing the codespaces environment to not use /etc/resolv.conf as a resource that multiple containers need to fight over, but we can tackle that later.
From your stacks, it appears that the logging system is blocked and we're trying to log while holding a mutex and therefore things get wedged and the watchdog kicks in.
When does this happen? I'm curious what's making logging freeze like that.
I suspect the VM/container is getting frozen and then upon resume, the watchdog is confused and using the wrong clock and the logging isn't actually frozen... just by wall clock it seems frozen. Looking into that now.
nocnokneo commentedon Sep 16, 2026
Yep, just took a look through the system logs. There was an OOM storm last night between 21:04:33 and 21:15 (someone was likely doing some R&D work; this is a shared machine). I didn't see any indication that tailscale was OOM-killed though so theoretically should have been able to survive if the watchdog didn't get confused and panic.
The devcontainer is running on a bare metal Linux machine, not a VM.