Recently a new customer reported sporadic AFP service crashes when sharing folders located on a Xsan volume.
First I want to give you a short overview
The client uses Mac OS X 10.6.8 in combination with Xsan 2.2.1. They have 2 MDCs and several NAS servers. The clients can access the Xsan volumes only through the NAS servers using AFP.
While this works great most of the time sometimes the AFP services crashes. After that the Mac clients don’t see the sharepoints any more. On the NAS server responsible for this share the SAN volume is still mounted but when trying to access it using Finder a crash occurs. It’s interesting that you are still able to browse the folders using ls in Terminal.app.
A temporary fix is to kill the AppleFileServer process!
After investigating further we discovered the following error messages:
5/21/12 1:23:05 PMmds[868] (Error) Server: Peer checkin failed -- no store for path '/Volumes/XSAN/Share' 5/21/12 1:23:05 PMcom.apple.AppleFileServer[837]MDSChannelPeerCreate: (os/kern) invalid argument
My partial explanation
These log messages are well known and documented on several webpages (1, 2). They are caused by a somehow inaccessible .fseventsd folder on the specific sharepoint. While this can be repaired quite easy on “normal” sharepoints it’s not that easy on Xsan volumes.
I have not been able to find the exact cause but I think it’s something like this:
- A client connects to the AFP service.
- The AFP server tries to connect to the Spotlight service of the specific volume
- Spotlight queries the fseventsd subsystem of OS X
- Something breaks while talking to fseventsd
A workaround
The problem can be “fixed” by disabling the MDS service. This can be archived either by unloading the launchd plist file or removing it entirely.
Temporary disable MDS (until reboot or reload)
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
Disable MDS
sudo rm /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
Attention: Stopping the MDS service will completely disable Spotlight