08-19-2019 07:58 AM
Update as of 8/19/2019.
The current ETA for MR640 (and the SBM fix) has been changed from August to open. However, I anticipate it being release in the next few weeks. I’ll continue to check the status on this and send you an update every week or so.
08-30-2019 07:56 AM
@stevecookMOL 1903 is started to end up on machines here, and this is quickly becoming a rather large problem for us. I really don't want to disable v3 completely, but running out of options.
No change yet in the current status for release. It is still in testing and does not have a firm ETA.
09-06-2019 07:36 AM
My Storage admin was looking over the notes for the latest Unity release (OE 126.96.36.199.5.116) and told me this:
"Not fixed in the current release. Still references the workaround."
Any updates from MS on this - or still DELL\EMC issue??
09-09-2019 07:46 AM - edited 09-09-2019 07:47 AM
Not sure why most people are trying to blame EMC / FluidFS for the SMB implementation.
I think Microsoft should be the one to fix the issue.
SMB 3.1.1 is not new, our EMC NAS has been on SMB 3.1.1 for years, and on previous Windows 10 releases (1703, 1709, 1803, 1809), it is able using SMB 3.1.1 to connect to our EMC NAS servers. Why the same SMB 3.1.1 stopped working in 1903?
Try to have a Windows 10 PC with build of (1703, 1709, 1803, 1809) connected with your EMC NAS. and run Get-SMBConnection, it should prove that the connection is utilizing SMB 3.1.1. There is no issue on EMC's SMB implementation.
09-09-2019 02:24 PM - edited 09-09-2019 02:25 PM
@IronRolia Because Dell was not following the SMB specification. An SMB server is not supposed to fail when sent unsupported capabilities during NEGOTIATE phase, regardless of the maximum dialect support. 1903 added a Compression flag to SMB capabilities (which was also publicly documented so that vendors could support it if they wished) and if not supporting compression, the Dell device is supposed to respond that it doesn't support that flag, not error and tear down session.
This is why Dell already fixed their other product with this symptom. Other manufacturers were unaffected by the addition of SMB Compression support in 1903 because they followed the spec.
Note: I own SMB and its specification.
09-10-2019 06:34 AM - edited 09-10-2019 07:28 AM
Thanks for the reply.
My question is if DELL did not follow the SMB specification, how would the SMB 3.1.1 dialect / protocol worked on previous Windows 10 releases with EMC NAS?
If I run GET-SMBConnection when connected to our EMC NAS, on previous Windows 10 releases, it clearly shows the connection are utilization SMB 3.1.1 protocol.
Do you have prove of any other vendors that actually use SMB 3.1.1 on their NAS storage that work with Windows 10 1903?
Are you suggesting that Microsoft also did not follow the SMB specs on their previous Win 10 releases, and now they decided to "fix" it in release 1903?
I really hope Microsoft and DELL can work together to investigate the issue collaboratively and have this issue fixed.
09-10-2019 08:37 AM - edited 09-10-2019 09:00 AM
FWIW the latest from Dell.
No official ETA yet for MR640 but I think that a release sometime this month is a strong possibility.
@IronRolia Neds' explanation says that the 3.1.1 spec if properly supported can have features that a vendor may not support, but that vendor device is still supposed to work. MS added the compression piece in 1903 and if the vendor was using the spec properly they would still work, and just not actually have the compression part. The Dell system is failing because it does not properly handle the added features as the 3.1.1 spec says it should.
Netapp does not have this issue, nor does a small Qnap device I run. I think Dell has already fixed this on their Unity line (perhaps not from reading above, I don't have one of those systems), but those of us stuck using the FluidFS system which they have already stopped selling don't yet have a fix. Our system is covered under warranty until March 2022 so i'm certainly expecting a fix.
09-10-2019 10:56 AM
Thanks for the clarification, I didn't catch the fine print of compression feature in 1903's SMB 3.1.1.
I really hope Microsoft and DELL can jointly work on this and come up with a fix soon.
Before a fix become available, I hope Microsoft can also provide some workaround solution like replacing the SMB dlls or some registry changes etc.
I personally have tried to replacing below DLLs from previous version of Windows 10 releases, but didn't work. Hope @Ned Pyle can point out a better workaround solutions before we can get a permanent fix from DELL.
09-10-2019 11:02 AM
@IronRolia I've been telling users to rollback versions for now, as the Dell fix is to completely disable smb 3 on the NAS end. Not sure on what level you are dealing with the problem, but we've also blocked the 1903 rollout for now. I don't love the solution by any means, but just dealing with what we've got to work with.
09-10-2019 11:08 AM
@IronRolia Yep, we rang them up right away :). They already fixed another line of products, I'm sure they are diligently working on this. The only workarounds right now are to disable SMB 3.1.1 within the NAS so that the client doesn't try to negotiate compression capabilities - this sacrifices a lot of other things, obviously. I think some other folks in this thread have talked about how to do this, I have no idea :\
09-10-2019 11:13 AM - edited 09-10-2019 11:19 AM
09-10-2019 11:17 AM
09-10-2019 02:03 PM - edited 09-10-2019 02:04 PM
Thanks for the instruction.
We have over thousand users sharing the the NAS, only few PCs upgraded to 1903, will revert back to 1809. Besides on 1809, we are able to take advantage of SMB 3.1.1, disabling SMB v3 is not going to work for us.
09-23-2019 04:44 AM
@Sybuxbeat me to it, hope to get this installed this week. If anyone has has loaded it already and had any comments it would be appreciated. Presumably it fixes the 1903 issue, but I haven't been able to get a copy of the release notes yet.
09-23-2019 06:13 AM
Pretty minimal changes,
Fixed Issues in FluidFS Version 6.0.400016
The following issues were fixed in FluidFS 6.0.400016:
NAS Volumes, Shares,
After installing Windows 10 update 1903, SMB shares are not accessible using the UNC path.
System Functionality The system incorrectly reports that duplicate IP addresses are in use for nodes on the cluster.
The system incorrectly reports battery failures during the battery calibration process
09-24-2019 01:20 AM
@tomatthe I installed the new version on our DR FluidFS this morning. It all went smoothly and it now works with SMB3 and Windows 10 1903!
I'm going to give it a few more days testing before installing it on the production servers but it looks ok so far.
09-25-2019 05:05 AM
@DarrenMiller Still trying to get Dell to review the diags I sent them and get me the actual firmware.
FWIW a friend of mine who also runs this system sent me the info below. Hopefully unlikely anyone else will run into this, but sounds like a bad situation.
The problem that occurred on our end occurred during the initial diagnostics process. We were working with the individual assigned to help get us prepped for the update, and so we pulled the ISO, and were running the initial tests. Part of that process – they had asked us to update the support password. When we did, the process tossed an error. They didn’t think anything about it. However, that process created a seed that didn’t exist, and the system literally hung during the diagnostic process trying to sync that file between the two nodes. It was difficult to diagnosis and we had to have someone from the team that actually wrote the queuing system for the product determine the problem and write a small utility to stop the cascade of badness. We were actually at a point of essentially re-imaging both nodes because no one could figure out all Thursday and Friday what the hell was going on.
Once that was fixed, the firmware update took 25-30 minutes – tops."
09-26-2019 07:48 AM
Having the same issue with Unity 300.
This Dell EMC Article worked for me.
09-26-2019 08:20 AM
@DarrenMiller finally got this installed our on DR side yesterday. Did Dell request you rolling restart both controllers on the FS prior to updating, and then again request you restart both controllers after updating firmware? They asked that I do that, then send them the gen diags after doing all of that. The 2nd set of rolling restarts did create some problems.
They are also asking for access to the storage arrays, which makes me a bit hesitant to think this firmware is production ready. That being said I haven't seen any issues with it so far, and 1903 does work properly.
09-26-2019 11:32 AM
@tomatthe They requested rolling reboots before the upgrade and logs, but not rolling reboots after.
I'll will try rolling reboots now the upgrade is done to check it doesn't cause us any issues. We normally have to do rolling reboots once a month anyway as the controllers regularly run out of cache (long standing issue, not related to this).
I was a bit surprised by the extra steps - I've never had to do that before so it does seem they are being extra cautious with this one.
10-10-2019 05:12 AM
@DarrenMiller They eventually said the luns were dropped during the restarts because there was a raid rebalance going at that time. I'm not sure why that would cause it to drop connections, but that's what I was told. I had replaced a drive in one of the sc2080's under the NAS head a few days before doing the restarts. From the end user side I don't believe there is any way to actually see that the system is still balancing data after replacing a disk.
The amount of disk these arrays have dropped is at about 15% in 2.5 years of use which is about 10% worse then any other array I run fwiw.