Debug Virtual File System

If a user experiences system crashes when using censhare Client with the VFS enabled, this may be an indication of an error in the kernel extension (not necessarily censhare however). Because such errors are difficult to analyze, macOS provides the ability to intercept the panic to generate so-called core dumps as soon as such a system crash occurs.

A core dump provides an image of the computer state at the time of the crash. It collects information about system extensions loaded (such as the VFS) and values ​​from the memory and the register content of the processor. Based on such a dump, it can be traced, at which position in the program code the system crash occurred.

Since these core dumps are produced at a time when the operating system is already in an unstable state, the dump files are not stored on the computer itself. This is a precautionary measure of the operating system because, among other things, it can not be guaranteed that such a write operation would not damage the local hard drive.

For this reason, you have to set up a dedicated core dump server to collect the dump files of affected computers, the so-called core dump clients.

Use a core dump server only in trusted networks and only on servers where you can trust all users Core dumps are transmitted unencrypted from a core dump client to the core dump server Since core dumps can contain sensitive data, make sure that they are protected against unauthorized access.

Set up core dump services

The core dump service consists of the following services:

• core dump server
• censhare crash server with censhare Crash Server app
• core dump clients with censhare Crash Reporter app

With the data collected by these services, you can analyze the problem of the crash closely and isolate the cause in more detail.

System requirements

For the core dump server, you need a macOS with the following characteristics:

• at least macOS 10.5

• enough memory (a single-core dump is several hundred MB in size)

• The server must be accessible on a 24/7 basis and own a static IPv4 address.

• This IP address must be visible to all core dump clients, no NAT and or firewall between client and server, etc. is allowed.

For the core dump client Macs the following conditions apply:

• At least macOS 10.7 (Lion)

• a recent censhare client with VFS enabled

• The clients are in the same network as the server, no NAT and no firewall between client and server, etc. is allowed.

Set up the server

The server consists of two parts. The core dump server and the censhare Crash Server app.

Set up a core dump server

To set up a core dump server, proceed as follows:

1. Create the directory /PanicDumps. In this directory, the received core dumps are stored.

   CODE

   sudo mkdir /PanicDumps 
   sudo chown root:wheel /PanicDumps 
   sudo chmod 1777 /PanicDumps

2. Enable the core dump server using the tool launchctl. With the argument -w, the core dump server is restarted automatically if you reboot the computer.

   CODE

   sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist

3. You can check whether the core dump server is running using a launchctl command:

   CODE

   sudo launchctl list | grep kdump

   As long as com.apple.kdumpd is displayed in the console output, the server is up and running. If, however, there is no output, the server has stopped.

Set up the censhare Crash Server app

The censhare Crash Server app must be running on the same computer on which the core dump server is set up.

1. Download the censhare Crash Server

2. Start the server by double-clicking the icon.

You do not need to configure the server. It has no graphical user interface. Its task is to receive panic reports of core dump clients and to store them in the / PanicDumps directory. Please make sure that the app is launched automatically each time after a system reboot.

Set up the clients

To enable the Macs on which the censhare client is running to submit core dumps to the server, configured the clients accordingly. Also, install the censhare Crash Reporter app on all your computers.

Set up the core dump clients

1. Download Kernel Core Dump Service from Technical Note TN2118 - Kernel Core Dumps.

2. Open the AppleScript Editor (located in Applications/Utilities/).

3. Open the Kernel Core Dump Service.app in the Editor.

4. Replace the IP 192.168.213.23 entry of the attribute panicd_ip with the IP of your own core dump server.

   CODE

   [...] 
   do shell script "nvram boot-args="debug=0x0544 _panicd_ip=192.168.213.23"" with administrator privileges 
   [...]

5. Save the modified app.

6. Copy the app on each computer that is to be used as a core dump client.

7. Run the script, click Enable and then restart the Mac.

Set up and use the censhare Crash Reporter app

This app should be executed each time after a system crash has occurred and the system is rebooted. The app asks you several questions and sends the response with additional information to the censhare crash server. However, first, the Reporter app must be adjusted to the IP of the censhare crash server.

1. Download the app censhare Crash Reporter from Technical Note TN2118 - Kernel Core Dumps.

2. Right-click the app in the Finder and select Show Package Contents from the context menu.

3. Open the file Info.plist from inside the directory Contents with a text editor.

4. Replace the IP 192.168.213.23 in the server-ip attribute with the IP of your core dump server.

   CODE

   <key>JVMArguments</key>
   <array>
   <string>--server-ip=192.168.213.23</string>
   <string>--server-port=9999</string>
   <string>--question=Your name</string>
   <string>--question=How many (render-)clients ran in parallel</string>
   <string>--question=Which client version(s) did you use</string>
   <string>--question=What have you done just before the system crashed</string>
   </array>

   The values of the attributes reflect the questions asked by the app. You can add any number of additional attributes of this value, as more questions are desired or needed

5. Save the modified app.

6. Install the app on each computer that is to be used as a core dump client.

7. Run the app every time when after a system crash and the system is rebooted.

Test the core dump services

To test if the individual parts of the core dump services operate as expected, proceed as follows. The test is performed on a computer that is configured as a core dump client.

1. Quit all running applications, and save open data.

2. Run the following command in the terminal. Attention, this will lead to a system crash:

   sudo dtrace -w -n "BEGIN{ panic();}"

   Your screen should now appear similar to the following output (white text on black lines):

   

   
   

3. Wait until the end of data transmission which is done when the screen shows "Waiting for remote debugger connection".

   Then restart your computer.

4. After restart, run the app censhare Crash Reporter. Answer all questions (for testing functionality) and then click Send Report.

5. Then connect to the core dump server.

6. Open the directory /PanicDumps. You see two files. One file is the core dump, the other file is the panic report that was generated by the crash reporter.

   If one of the two files does not exist, please check if all servers are running and whether the write permissions are set correctly on the directory PanicDump. Try using the command touch /PanicDumps/test.file to create an empty file. If this fails, the permissions are not correct.

The transmission of core dumps by a core dump client is only possible over the installed Ethernet interface of your Mac. Wireless transfer via the AirPort interface is not supported by Apple. That means device units without a wired Ethernet interface (the MacBook Air) cannot send core dumps unless you try a Thunderbolt / Gigabit Ethernet adapter (not tested). The use of the censhare Crash Reporter app is not affected by this limitation and should therefore be used in any case.

For additional information about Apple's core dump client/server, see Technical Note TN2118 - Kernel Core Dumps.