-
Bug
-
Resolution: Unresolved
-
Critical
-
rhel-9.2.0
-
None
-
Yes
-
Critical
-
rhel-virt-windows
-
8
-
False
-
False
-
-
None
-
None
-
None
-
None
-
Unspecified
-
Unspecified
-
Unspecified
-
-
All
*Describe the bug*
New virtio drivers (0.1.285, 0.1.292), i.e. all builds with 1bbc422 commit [Address possible memory management issues when receiving interrupts for already completed requests] contain serious bug, especially present in combination of Windows Server 2025 and SQL Server.
Under massive load, race conditions occur, and such messages can be found in SQL Server Log / Windows Application Log (information only, not marked as warnings/errors - the more it's tricky):
`A read of the file '.mdf' at offset 0x00003897472000 succeeded after failing 1 time(s) with error: **incorrect checksum* (expected: 0xad4c6778; actual: 0xad4c6778)` - see that actual/expected are the same, i.e. a storage problem (read retry)
`A read of the file '.mdf' at offset 0x00003897470000 succeeded after failing 1 time(s) with error: **incorrect pageid* (expected 1:29669944; actual 1:29669944)` - see that actual/expected are the same, i.e. a storage problem (read retry)
Dozens of such messages can be found every day (under real SQL/IO load), and after 2-3 days, SQL Server service hangs. It's extremely dangerous in production environment. *It's all about Windows Server 2025 platform.* SQL Server is just trying to explain what's happening. Please note the *actual/expected values are the same.* No DB corruption (DBCC is clear). So this is the *general read-retry issue on WS2025* with newer virtio drivers. Version 0.1.271 is the last working out-of-the-box (without 1bbc422 commit). ~According to other (Proxmox) users, no problems on WS2022 or older (with 0.1.285).~
I have deeply analyzed and isolated this problem. I've tested my own vioscsi driver build with 1bbc422 revert (based on Oct 31 master) and all is running well, no errors so far (in production system for more than 4 days). I'm sure about the root cause and resolution/temporary mitigation.
*To Reproduce*
According to my findings, this is what is happening:
Storport in Windows Server 2025 changed towards better performance. It's quite clear that it scales much better on multi-core systems, so while in WS2022 there was a lower risk that the driver requests would be parallelized, the opposite is true for WS2025. In other words, on WS2025’s Storport, the timing mix can cause a "rare" mis-match or stale completion → one read returns the wrong bytes (page checksum/pageid “wrong”), SQL Server retries immediately, the next read hits the right buffer and “succeeds after failing 1 time”.
*Expected behavior*
There should be driver stability on all platforms.
Instead, check these two lines in 1bbc422: [non-atomic increment](https://github.com/virtio-win/kvm-guest-drivers-windows/commit/1bbc422#diff-b8f1732536a5020c949209736b628a0af83022fbbded42c0eaef7bc3149dd6c3R845-R846)
```
srbExt->id = adaptExt->last_srb_id;
adaptExt->last_srb_id++;
```
At least to me, it's clear that the assignment and increment are not one atomic call (no lock, no interlocked op), so if occasionally this call is processed per-partes, the read attempt have to fail.
On modern Storport (as in Windows Server 2025), StartIo can run on multiple CPUs/queues much more aggressively than older builds, so two threads can actually read the same last_srb_id and hand out duplicate IDs. When the host later completes those I/Os, the driver can mismatch which SRB it completes or fail to find one (“No SRB found for ID”), causing a transient bad read that SQL Server retries — exactly the “succeeded after failing 1 time” messages mentioned above.
So the fix can be something like this (GPT-5 Thinking-generated, no my own suggestion):
```
— a/vioscsi/vioscsi.h
+++ b/vioscsi/vioscsi.h
@@
- ULONG_PTR last_srb_id;
+ volatile LONG64 last_srb_id; // atomically incremented
— a/vioscsi/vioscsi.c
+++ b/vioscsi/vioscsi.c
@@ VioScsiFindAdapter(...)
- adaptExt->last_srb_id = 1;
+ adaptExt->last_srb_id = 1;
@@ VioScsiStartIo(...)
- srbExt->id = adaptExt->last_srb_id;
- adaptExt->last_srb_id++;
- if (adaptExt->last_srb_id == 0 || (adaptExt->tmf_cmd.SrbExtension &&
- adaptExt->last_srb_id == (ULONG_PTR)&adaptExt->tmf_cmd.SrbExtension->cmd))
{
- adaptExt->last_srb_id++;
- }
+ ULONG_PTR id;
{ +#if defined(_WIN64) + id = (ULONG_PTR)InterlockedIncrement64(&adaptExt->last_srb_id); +#else + id = (ULONG_PTR)InterlockedIncrement((volatile LONG*)&adaptExt->last_srb_id); +#endif + }
+ dowhile (id == 0 ||
+ (adaptExt->tmf_cmd.SrbExtension &&
+ id == (ULONG_PTR)&adaptExt->tmf_cmd.SrbExtension->cmd));
+ srbExt->id = id;
```
The `Interlocked` family is specifically recommended/allowed in Storport miniports and provides the needed memory barriers: [MicrosoftDocs - Interlocked Operations in Storport Miniport Drivers](https://github.com/MicrosoftDocs/windows-driver-docs/blob/staging/windows-driver-docs-pr/storage/interlocked-operations-in-storport-miniport-drivers.md)
*Host:*
- Disto: Proxmox 8.4
*VM:*
- Windows Server 2025
- Which driver has a problem: 0.1.285+ (0.1.271 is OK)
- Working driver: just revert the 1bbc422
*Additional context*
Please note that I'm aware of @vrozenfe 's revert (a6d690a) of @benyamin-codez 's commit (15e64ac) of "NO-SDV [vioscsi] Reduce spinlock management complexity" and I can confirm this is completely unrelated to the issue mentioned above (just to avoid confusion regarding other recent vioscsi commits/reverts).
FYI @MartinDrab