Repository navigation
Forwarder eventually stops sending logs #448
Description
We've tested the new forwarder functionality to replace the old seqfwd app. After running correctly for a few days, we've observed several instances of this message in logs across several boxes (all running as a service on Windows):
{
"@t":"2026-03-05T20:45:30.6719054Z",
"@m":"Forwarding log shipper failed and exited",
"@i":"1efb7888",
"@l":"Fatal",
"@x":"System.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\\0000000000000001.clef' is denied.\r\n at System.IO.FileSystem.DeleteFile(String fullPath)\r\n at SeqCli.Forwarder.Filesystem.System.SystemStoreDirectory.TryDelete(String name) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Filesystem\\System\\SystemStoreDirectory.cs:line 96\r\n at SeqCli.Forwarder.Storage.BufferReader.AdvanceTo(BufferPosition newReaderHead) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Storage\\BufferReader.cs:line 248\r\n at SeqCli.Forwarder.Channel.ForwardingChannel.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Channel\\ForwardingChannel.cs:line 115",
"@tr":"39d89a8ca8d77ce084232c6be16c588b",
"@sp":"8e85c1e1a8c13011",
"SourceContext":"SeqCli.Forwarder.Channel.ForwardingChannel",
"RequestId":"0HNJQ5TSLESBP:00000001",
"RequestPath":"/api/events/raw",
"ConnectionId":"0HNJQ5TSLESBP",
"MachineName":"xxx",
"Application":"SeqCli Forwarder"
}Our configuration is pretty simple - the only interesting bit is that we've set useApiKeyForwarding to true in the json:
{
"connection": {
"serverUrl": "https://xxx",
"apiKey": null,
"pooledConnectionLifetimeMilliseconds": null,
"eventSizeLimitBytes": 262144,
"batchSizeLimitBytes": 10485760
},
"output": {
"disableColor": false,
"forceColor": false
},
"forwarder": {
"storage": {
"targetChunkSizeBytes": 52428800,
"maxChunks": null
},
"diagnostics": {
"internalLoggingLevel": "Information",
"internalLogServerUri": null,
"internalLogServerApiKey": null,
"exposeIngestionLog": false,
"ingestionLogShowDetail": false
},
"api": {
"listenUri": "http://127.0.0.1:15341"
},
"useApiKeyForwarding": true
},
"encryption": {
"encryptor": null,
"encryptorArgs": null,
"decryptor": null,
"decryptorArgs": null
},
"profiles": {}
}Looks like this may happen once one of the clef buffers hits the targetChunkSizeBytes limit? Restarting the service will run through one chunk and then fail to delete it. Maybe a leaked file handle or something?
Activity
liammclennan commentedon Mar 9, 2026
Hi @chuseman
Have you tried excluding 'C:\ProgramData\Seqcli' from your antivirus scanning?
chuseman commentedon Mar 10, 2026
No, we don't have any exclusions setup. We run Crowdstrike and that's corporate controlled so it would take some doing to get an exclusion added.
Is the behavior here different from the old seqfwd service? That ran fine on the same servers -- we've had to roll back to it for now.
nblumhardt commentedon Mar 10, 2026
The old Seq Forwarder (essentially) used a single file that it would lock at start-up, so there would be minimal opportunities for AV scanners etc. to jump in and cause locking problems. The new version uses a set of rolling log files, so would be more susceptible to this kind of outside interference.
There's also a chance it's a lurking bug we haven't yet spotted. You might be able to get some more info on which process is using the lock via
resmonor similar.chuseman commentedon Mar 12, 2026
I've ran
procmonto try to see what's going on. Here's how I tested:procmoncapture, filtered tofd5a6b94b1bf49df954745f8262a5c2ddirectory.fd5a6b94b1bf49df954745f8262a5c2ddirectory only contains the following files:0000000000000000.bookmark0000000000000022.clef,api.keyseqcliservice, wait for logs to report that it's running; ie:{"@t":"2026-03-12T20:48:49.7310130Z","@m":"Opening local buffer in \"C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\"","@i":"5fca42e5","StorePath":"C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d","SourceContext":"SeqCli.Forwarder.Channel.ForwardingAuthenticationStrategy","MachineName":"xxx","Application":"SeqCli Forwarder"}seqcli{"@t":"2026-03-12T20:53:17.0857294Z","@m":"Forwarding log shipper failed and exited","@i":"1efb7888","@l":"Fatal","@x":"System.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\\0000000000000022.clef' is denied.\r\n at System.IO.FileSystem.DeleteFile(String fullPath)\r\n at SeqCli.Forwarder.Filesystem.System.SystemStoreDirectory.TryDelete(String name) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Filesystem\\System\\SystemStoreDirectory.cs:line 96\r\n at SeqCli.Forwarder.Storage.BufferReader.AdvanceTo(BufferPosition newReaderHead) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Storage\\BufferReader.cs:line 248\r\n at SeqCli.Forwarder.Channel.ForwardingChannel.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Channel\\ForwardingChannel.cs:line 115","SourceContext":"SeqCli.Forwarder.Channel.ForwardingChannel","MachineName":"xxx","Application":"SeqCli Forwarder"}procmoncapture.Here's what the
procmoncapture says about0000000000000022.clefaround the time of the delete (which is after0000000000000023.clefgets created and0000000000000000.bookmarkis updated):procmondoesn't see any other process interested in0000000000000022.clefother thanseqcli.exe.Do you think this rules out Crowdstrike/antivirus scanning? Or do I still need to try to push for an exclusion of this directory to test it?
nblumhardt commentedon Mar 12, 2026
Thanks for the update, Chris, that's good info to have. We'll review it and let you know later today.
Best regarsd,
Nick
liammclennan commentedon Mar 13, 2026
Hi @chuseman,
We are still looking into this. At this stage it looks most likely to be a bug in seqcli.
liammclennan commentedon Mar 17, 2026
Hi @chuseman
We have not yet been able to reproduce the issue you've described. I think we will need some more detail of your setup. Can you please send a diagnostic report to support@datalust.co and we will continue the conversation there?
liammclennan commentedon Mar 17, 2026
@chuseman skip the diagnostic report. That won't help as you are probably running
seqclion a different computer. Send a message to support@datalust.co and we can pick up the conversation there.nblumhardt commentedon Mar 19, 2026
One quick follow-up note; although we haven't been able to trigger the issue with them, we have found a couple of suspicious bits of code that should lead to the issue you're reporting. (Why they don't might be down to Windows versions, etc.).
I'll send a patch for
devthat addresses them sometime in the next week; without us being able to repro it's not guaranteed it'll sort the issue out, but should be worth trying.Thanks again for the report and all the investigation @chuseman!
KodrAus commentedon Mar 30, 2026
Hi @chuseman 👋
I’m currently looking at this over in #451. Would you be happy to pull a preview build in your environment when it’s ready to confirm it’s all fixed?
chuseman commentedon Mar 30, 2026
Sure @KodrAus I can test it out - just let me know where to grab the bits!
nblumhardt commentedon Mar 31, 2026
Thanks @chuseman!
2025.2.02489-dev-02489is now on the releases page. Keen to hear how you go.chuseman commentedon Mar 31, 2026
Good news! This appears to fully fix the issue we were running into.
Thanks for the fix!
KodrAus commentedon Mar 31, 2026
Thanks for the update @chuseman. I'll keep an eye on the issue so please do let us know if you run into anything else 👍