Limit of shadow copies over SMB

The integration of shadow copies into the Windows Explorer significantly eases the recovery of mistakenly deleted or overwritten files, especially since this feature is also available for shared storages over SMB with a potentially large number of users. The vfs_shadow_copy2 [1] module adds the ability to Samba to expose snapshots to Windows clients and comprises an elegant way to expose ZFS snapshots directly to a Windows user. Great! you might think, as ZFS has no problem with a large number of snapshots [2], this seemingly gives us the power to establish long term and fine grained filesystem level versioning of our data.

Over the past years I have installed half a dozen FreeNAS based network storage systems, utilizing ZFS with periodic snapshots and Samba with vfs_shadow_copy2 enabled to expose snapshots to Windows clients. This has worked out perfectly at the beginning, but Microsoft wouldn’t be Microsoft if there wasn’t a surprise, and over time this once noble feature began to vanish. At first sight, there was no sign of any obvious error and quickly testing the very same FreeNAS version in a virtual machine instantly brought back the shadow copies, so it couldn’t be a sole matter of the FreeNAS server or the Windows client. After playing around with different configurations for a while it turned out that it constantly worked in my test setup, as it did in any new setup in the past, but no matter what I tried, I couldn’t revive shadow copies on the established servers.

As google didn’t bring up anything similar to this issue I started digging into this. First, increasing the log verbosity of Samba showed the following:

../source3/smbd/smb2_server.c:1782(smbd_smb2_request_verify_creditcharge) CreditCharge too low, given 1, needed 2

The SMB protocol uses credits to rate limit traffic and prevent a denial-of-service. This is accomplished by granting and consuming credits to/from the clients [3]. Each client request consumes at least one credit and the server grants at least one credit with every response to the client [4]. The number of credits needed for a read request depends on the size of the response and is calculated as follows ( 1 + (Length – 1) / 65536 ) [5], whereas multi-credit or LargeMTU requests are possible since SMB 2.1 [6]. At this point, I started to get an idea about what was going on here. To get the last pieces together I started dumping the TCP packets involved in the request using Wireshark.

smb2_bug_traffic

Packets 9 and 11 show the first half of the communication, the client asking for snapshots, offering a 16 byte response buffer, which the server uses to let the client know the total number of available snapshots and the response size needed to transfer a complete list of snapshots [7].

smb2_bug_response

The client then correctly requests the list of snapshots, announcing the requested data space Max IOCTL Out Size, but only offering one credit. Since the expected data size is greater than 65536, one credit is not sufficient for this operation and the server correctly therefore responds with STATUS_INVALID_PARAMETER [7]. This explains the log entry observed earlier.

smb2_bug_error

Even if you disable the DisableLargeMTU Setting [8] the client still doesn’t offer more than one credit for a FSCTL_SRV_ENUMERATE_SNAPSHOTS operation and I couldn’t find any other option so far to teach Windows to receive a snapshot lists larger than one packet. This leads in fact to an effective limit of roughly 1300 snapshots, the number that fits into a single packet. I couldn’t find any answer to the question why Windows isn’t able to handle larger snapshot lists so far. I guess this can at least be considered as unwanted behavior or worse simply be a bug in Windows.

[1] https://www.samba.org/samba/samba/docs/man/manpages/vfs_shadow_copy2.8.html
[2] https://en.wikipedia.org/wiki/ZFS
[3] https://msdn.microsoft.com/en-us/library/cc246719.aspx
[4] https://msdn.microsoft.com/en-us/library/cc246704.aspx
[5] https://msdn.microsoft.com/en-us/library/cc246638.aspx
[6] https://blogs.msdn.microsoft.com/openspecification/2009/06/22/smb-2-1-multi-credit-large-mtu-operations/
[7] https://msdn.microsoft.com/en-us/library/cc246447.aspx
[8] https://msdn.microsoft.com/en-us/library/windows/hardware/dn567661(v=vs.85).aspx

This article is licensed under CC BY-NC-SA 4.0