The story about disappearing files

Sep 082014

Last week a local customer reported strange problems with his EMC Isilon storage. For example sometimes when they copy a file from Mac A to their central storage they can’t see it on Mac B. Only Macs are affected by this strange behaviour – all their PCs work great. I was happy when they booked an on-site appointment to investigate the problems further.

I started the investigation by talking to all the people there and writing down all the issues.
After summarising I found out that most of the issues where caused by the fact that they mixed SMB and NFS.
I discussed this with the customer and he happily agreed to switch all machines to SMB.

Isilon

However we still had one issue to solve: Some files (those with umlauts in their filename) where only visible over NFS. The following blog post is a summary of my on-site procedure and my findings:

1.) File Creating with Umlauts over NFS3

If you create a file with umlauts in it over NFS3 (tested with MS Word) it can’t be opened over SMB (“Das Programm kann nicht gefunden werden”). It still works over NFS. You are unable to delete the EAs (._ Files) over SMB (“No such file or directory”).

After removing the EA files over NFS, MS Word launched but complained about an illegal filename. I was furthermore unable to read the file (“no such file or directory”) using cat – however it was still working with NFS.

2.) File Creating with Umlauts over SMB

Same problem as above! The file could not be accessed using NFS – everything working as expected over SMB. Furthermore SMB supports alternate data streams -> EAs get lost between protocols. It this case it is somewhat good that they are enabled as it would break QT7 otherwise.

3.) File Creating without Umlauts

Everything works fine if there is no umlaut in the filename.

4.) Files are not shown within Finder

In Terminal you can see them using ls. This is related to the EA ._ metadata files! If a file has EA’s they disappear if Finder is unable to access those. This is the reason why some movies with umlauts in their name are hidden. If you delete the ._ files they reappear – but are still inaccessible.

5.) Word sometimes unable to save files with Umlauts?

“Word kann dieses Dokument aufgrund eines Bennenungs- oder Berechtigungsfehlerd nicht auf dem Zielvolume schreiben”

6.) Verification

Based on that knowledge I used the following procedure to locate the problem: I created a file with QT X (test.mov) on the storage. Then I duplicated it and renamed to either “NFS aaaÜ.mov” and “SMB aaaÜ.mov” over the corresponding protocol. Thereby two files were created.

Then I ran the following hexdump commands:

NFS Test File

MBP:Test3 user$ ls /Volumes/Broadcast/Test/Test3/NFS*|hexdump -C #over SMB
00000000  2f 56 6f 6c 75 6d 65 73  2f 42 72 6f 61 64 63 61  |/Volumes/Broadca|
00000010  73 74 2f 54 65 73 74 2f  54 65 73 74 33 2f 4e 46  |st/Test/Test3/NF|
00000020  53 20 61 61 61 55 cc 88  2e 6d 6f 76 0a           |S aaaU...mov.|
0000002d
MBP:Test3 user$ ls /Volumes/broadcast-1/Test/Test3/NFS*|hexdump -C #over NFS
00000000  2f 56 6f 6c 75 6d 65 73  2f 62 72 6f 61 64 63 61  |/Volumes/broadca|
00000010  73 74 2d 31 2f 54 65 73  74 2f 54 65 73 74 33 2f  |st-1/Test/Test3/|
00000020  4e 46 53 20 61 61 61 55  cc 88 2e 6d 6f 76 0a     |NFS aaaU...mov.|
0000002f

SMB Test File

MBP:Test3 user$ ls /Volumes/Broadcast/Test/Test3/SMB*|hexdump -C #over SMB
00000000  2f 56 6f 6c 75 6d 65 73  2f 42 72 6f 61 64 63 61  |/Volumes/Broadca|
00000010  73 74 2f 54 65 73 74 2f  54 65 73 74 33 2f 53 4d  |st/Test/Test3/SM|
00000020  42 20 61 61 61*55 cc 88* 2e 6d 6f 76 0a           |B aaaU...mov.|
0000002d
MBP:Test3 user$ ls /Volumes/broadcast-1/Test/Test3/SMB*|hexdump -C #over NFS
00000000  2f 56 6f 6c 75 6d 65 73  2f 62 72 6f 61 64 63 61  |/Volumes/broadca|
00000010  73 74 2d 31 2f 54 65 73  74 2f 54 65 73 74 33 2f  |st-1/Test/Test3/|
00000020  53 4d 42 20 61 61 61*c3  9c*2e 6d 6f 76 0a        |SMB aaa...mov.|
0000002e

Thereby I found out, that there is a different filename reported if you are using SMB. I marked the corresponding changes with an *. What that means it, that there are character encodings issues.

7.) The Issue: NFS

To test if the issue was related to their Isilon I repeated the test on a Debian VM. It shows the same strange issues. Thereby I conclude that the issue is caused by OS X’s NFS client and Finder. A possible way to reproduce this is to rename a file using terminal:

mv "/Volumes/NFSServer/testfile.mov" "/Volumes/NFSServer/testfileäöü.mov"

The expected behaviour is, that the file testfile.mov got renamed to testfileäöü.mov. While exactly that happened, the file got inaccessible. You cannot open it anymore.

9.) Next Steps

To fix this issues we recommend the following next steps:

Switch all machines to SMB – Thereby pretty much all problems should be fixed automatically.
To finalise the migration we have to fix the remaining issues:
- We need a script that deletes all ._ EA files
- Than we have to check if we can access all files containing umlauts. If not we have to rename them to work again (rename to some temporary name over NFS and rename back using SMB) This is the hard part, as we have to preserve the umlauts. Thereby we may be able to avoid the need to relink all assets.

If you have the same problem and need help see the About Me page for contact details.

#bogner.sh