macOS – machine2machine kernel debugging

Hey folks,

due to recent macOS research I had to setup kernel debugging again. I have already done a lot of research in that area so I was quite familiar with the procedure. Nevertheless every time when you have to deal with a new version of an operating system you’ve been playing around before, there are some unexpected problems which will make this kind of setup a pain in the ass!

Personally I prefer to jump directly into new research and solve problems step by step but sometimes that can be very frustrating, especially when preperation tasks and setup is tricky.

Physical vs. VMware

Usually I try to solve most of my task using virtual machines. I use VMware Workstation Pro a lot and so far I was happy with it. On macOS of course I use VMware Fusion which is the nicer VMware product in my opinion. It is faster, doesn’t hang a lot and I love the quick VM switching via pseudo “swipping” using the keyboard.

Installing macOS inside VMware Fusion is simple. Installing inside VMware Workstation is already a bit more complicated. Most likely you know that you will have to patch VMware Workstation with the latest version of “VMware Unlocker“. Here I really want to thank the author of the tool – you helped a lot of people with it AND I never had a single problem using the unlocker.

Beside the fact that this is prohibted by VMware the unlocker introduces an updating issue in VMware Workstation. Whenever a new update was applied the unlocker needs to run again. A small issue I have to admit, but if you forget that you might experience trouble. Keep that in mind.

Additionally and limited to macOS VMs you will have to manually patch you .vmx file. There is an issue related to SMC which can be solved as follows:

Adding the both lines to your .vmx file will allow you either run a fresh macOS installation from an .iso file or a preinstalled macOS VM created by VMware Fusion.

All of the mentioned issues are annoying but not difficult to deal with. So far so good. When you want to debug a kernel, then you want control the system like this:

SoftICE – The best debugger ever!

My eyes still start glowing when I see this little window. I miss the old days when SoftICE was still running (have to admit that SoftICE architecture was not the best, what at the end was one reason why development was stopped…). And one of the reason why reversers did love it was that you could press CTRL+D at any time to fireup the debugger. The system did freeze completely and you would eventually become a debugging god with full control in your operating system universe ….;-)

Well these days are over but I still love take over the control completely when I debug. Especially I’m missing CTRL+D feature.

The good news here is that you can have that feeling again when you debug the macOS kernel. And this basically is why you cannot do that the vm2vm way.

Let me quickly wrap-up how this works:

Debuggee-System

  1. Disable rootless/SIP protection. At the time of writing this article you can do it by booting into the recovery mode (Control-Key+R), then opening a terminal to issue the following command:
  2. Set the nvram boot-args accordingly to your needs (for initial testing I highly recommend to use debug=0x5). Remember, if you did skip step 1 you won’t be able to correctly modify the nvram. A sample for setting the nvram boot-args would look like that:
  3. Reboot machine
  4. The machine will freeze at a point and tell you that it is waiting for a kernel remote debugger to attach. Additionally it will print the Debuggee-System’s IP and MAC address. Save both in your notes, you will need it later.

Debugger-System

  1. Install XCode + XCode commandline tools if not yet done.
  2. Install Kernel Debugging Kit matching the exact OS version on the Debuggee-System (this is highly recommended if you want to have kernel symbols available during your debugging session)
  3. Set a static ARP entry to ensure proper network communication. The command to add an entry is:

    It might happen that after the command was executed you will see a message that will tell you that a previous entry was deleted or maybe not even existing. Just ignore that.
  4. Open a terminal and start the debugger:
  5. Once you are in the lldb shell you are ready to debug. Now you need the IP address and MAC address of the debuggee. Using the following command you will be able to connect/attach:

There you go! Hopefully there was no problem and you have now a working remote connection and thus being able to debug debuggee system’s kernel.

This scenario obviously applies on when you are in the comfortable situation of having two physical macOS machines at hand. Lets assume you type c+enter to let the debuggee system continue running for a second.

How do you gonna add a breakpoint or examine a memory location ?

That is very NMI enters the stage. WikiPedia offers the following brief description of what NMI is:

In computing, a non-maskable interrupt (NMI) is a hardware interrupt that standard interrupt-masking techniques in the system cannot ignore. It typically occurs to signal attention for non-recoverable hardware errors. (Some NMIs may be masked, but only by using proprietary methods specific to the particular NMI.)

An NMI is often used when response time is critical or when an interrupt should never be disabled during normal system operation. Such uses include reporting non-recoverable hardware errors, system debugging and profiling, and handling of special cases like system resets.

Modern computer architectures typically use NMIs to handle non-recoverable errors which need immediate attention. Therefore, such interrupts should not be masked in the normal operation of the system. These errors include non-recoverable internal system chipset errors, corruption in system memory such as parity and ECC errors, and data corruption detected on system and peripheral buses.

If you are interested in more info please have a look here: https://en.wikipedia.org/wiki/Non-maskable_interrupt

Lovely right !? Doesn’t some too good but here it is crucial to have working NMIs on a system so you can trigger the debugger to break at any time. Think of it like:

  • You press a specific combination of keys
  • That will force the debuggee system to trigger an NMI
  • And therefore the debugger will be notified to reactivate the debugging session right away

As you will understand without the capability to issue an NMI there will be no CTRL+D SoftICE like feature (basically that is not completely true as I am ignoring debugStub implementation in VMware Fusion and Workstation, but that is a different story).

Practically in my case running the latest version of macOS 10.12.3 at this time the magic key combination will be Left CommandKey + Right CommandKey + PowerButton. In the moment you press the those buttons together your debugger will jump in and you are in control again.

Virtually no NMI available !?

Basically that is the reason why it doesn’t seem to work in VMware Workstation/Pro. It seems that there is no mechanism that is  capable of sending an NMI which will recognized by the debugger system.

How did I test that ?

I tried more or less the above depicted method to setup kernel debugging. The first significant difference was that even with boot-args=”debug=0x1 -v” or boot-args=”debug=0x5 -v” set the debuggee system did not stop at boot time to wait for the kernel debugger.

From what I can say this is connected to the NMI issue. Interestingly I remember a former setup where this was working but for VMware Fusion. Fusion and Workstation are for sure not completely equal but they share a lot of common code.

Knowing that I guess there is a way to trigger such a NMI but as of yet nobody has found the way to do so. Unfortunately VMware is not gonna helpful here 🙂 If I can spend some more time I will definitively look deeper here. Having the same setup with two virtual system would be awesome.

Conclusion

My recommendation here is whenever you want to start immediately debugging then go for the physical setup. It will save you a lot of time!

Another reason to choose the physical setup is that there might be bugs which you cannot replicate inside a virtual machine because they are hardware dependent. Just recently I had exactly that.

I hope this little article helped to better understand issue with kernel debugging on macOS on virtual system as well as how a quick’n’dirty-setup could look like. If there are any questions related to the topic feel free to drop an email. mitp0sh@mitp0sh.de

 

 

いらっしゃいませ!

Hello Passenger!

After a long period of idle, I finally decided to build-up my broken blog from scratch again. This time I will try to approach from a more minimalistic perspective and hope to provide decent content for you as my limited time will allow me.

The most important reason why I want to contribute something to our scene is that I really miss the old days when reverse engineers shared most of their knowledge. Sadly, I have to say that this seems no longer be the case. A pretty obvious reason in my opinion is that nowadays our jobs became kind of an elitist thingy and the involed money makes some of us feel to guard and protect their knowledge from being stolen.

Personally, I doubt that this is a good and healthy philosophy. Quite the contrary is the case I think: The more we shield knowledge from others the less we will generate innovation.

The one and only thing I can say for sure is that I wouldn’t be where I am today if not a bunch of very talented and friendly individuals would have taught me how to do the things right. Thank you very much!

There is a lot stuff I want to talk/write about so please stay tuned 😉

Nuff said, I hope you will find some of the contributions interesting and even if the opposite is the case, something was learned too!

Cheers,

mitp0sh of PDX

### In memory of Francesco Vianello aka +Fravia ###