How we ported Linux to the M1

1. Apple special sauce: The M1 Processor

When Apple released their desktop products with the M1 processor in November 2020, quite a few people in the tech community were surprised by the excellent performance of these systems. But those who have been following the development of Apple phone chipsets closely knew that the evolutionary path Apple followed would result in a powerful 64-bit Arm processor.

At Corellium, we've been tracking the Apple mobile ecosystem since iPhone 6, released in 2014 with two 64-bit cores. Since then, Apple has been focusing their energy on building faster chips, preferring to improve single-threaded performance over throwing more cores on the chip. This approach was enabled by their in-house hardware design team, and resulted in unique parts with a broad feature set, leading the industry in terms of architectural features.

It also made Apple silicon rather distinct from all other 64-bit Arm hardware in terms of both CPU core and peripherals. Our Corellium virtualization platform has been providing security researchers with unparalleled insight into how operating systems and programs work on Apple Arm processors. But in the process of developing our virtualization system, we also gain knowledge about the hardware we are modeling, and this knowledge can be best refined by testing it against real hardware - which we have only been able to do with the emergence of checkm8, an exploit that let us load programs onto Apple smartphones. This led directly to the Sandcastle project, where we built a kernel port to the A10 processor in early 2020.

So when Apple decided to allow installing custom kernels on the Macs with M1 processor, we were very happy to try building another Linux port to further our understanding of the hardware platform. As we were creating a model of the processor for our security research product, we were working on the Linux port in parallel.

2. Starting the port for Linux on M1

Many components of the M1 are shared with Apple mobile SoCs, which gave us a good running start. But when writing Linux drivers, it became very apparent how non-standard Apple SoCs really are. Our virtual environment is extremely flexible in terms of models it can accommodate; but on the Linux side, the 64-bit Arm world has largely settled on a well-defined set of building blocks and firmware interfaces - nearly none of which were used on the M1.

To start with, Apple CPUs boot the operating system kernel in a different way. The bootloader, traditionally called iBoot, loads an executable object file in a format called Mach-O, optionally compressed and wrapped in a signed ASN.1 based wrapper format called IMG4. For comparison, normal Linux on 64-bit Arm starts as a flat binary image (optionally compressed and put in one of the few container formats), or a Windows-style "PE" executable on UEFI platforms.

But the real surprises start when further CPU cores are brought up. On other 64-bit Arm systems, this is done by calling the firmware through an interface called PSCI (a few systems use poll-tables, but the firmware is still responsible for them). But on M1, CPU cores start at an address specified by a MMIO register (set to a specific offset within the kernel image, then locked, by the bootloader), and simply begin running the kernel.

If that wasn't enough, Apple designed their own interrupt controller, the Apple Interrupt Controller (AIC), not compatible with either of the major Arm GIC standards. And not only that: the timer interrupts - normally connected to a regular per-CPU interrupt on Arm - are instead routed to the FIQ, an abstruse architectural feature, seen more frequently in the old 32-bit Arm days. Naturally, Linux kernel did not support delivering any interrupts via the FIQ path, so we had to add that.

When you try to get multiple processors in a system to talk to each other, you have to provide a set of inter-processor interrupts (IPIs). On older Apple SoCs, those were handled similarly to IRQs, by executing MMIO accesses to the AIC. But on newer ones, Apple uses a set of processor core registers to dispatch and acknowledge IPIs, and they are - again - delivered as FIQs. So the FIQ support was really quite important. Fortunately, our work on virtual models in our security research product has prepared us for this.

After working out a few more hardware quirks, and adding a pre-loader that acts as a wrapper for Linux and provides a trampoline for starting processor cores, we could set a framebuffer and were greeted with the sight of eight penguins representing the eight cores of the M1.

linux-m1

3. Need input! Connecting the USB port

Unfortunately, since we do not have a UART cable for the M1 Macs, we had to find another way to add a keyboard (and maybe even a mouse). There are fundamentally three paths to do that on the M1 Mac Mini: the built-in USB host in the M1 chip (serves the Thunderbolt/USB ports), the xHCI USB host on PCIe (serves the type A ports) and Bluetooth.

While we won't get into the details of Apple Bluetooth, we'll note it uses a non-standard PCIe-based protocol that is supported in our virtualization product, and would require not only bringing up PCIe ports on the M1 chip, but also writing a custom kernel driver for this protocol. That made it seem like the worst choice for getting this done quickly.

This means we had a choice between bringing up PCIe and using the standard kernel xHCI driver, or bringing up the built-in USB controller. Apple has been using the Synopsys DWC3 dual-role USB controller for a while in their chips, and it has a Linux kernel driver. Unfortunately, Apple is also in the habit of adding custom logic around the controller, so this ended up being a fair bit of work.

Both the PCIe and the built-in DWC3 USB controller on M1 use IOMMUs, called DARTs. Apple has been refining their DART design in a consistent, evolutionary way, resulting in an excellent, full-featured IOMMU. The last version even has support for sub-page memory protection, rarely seen elsewhere. (We had a blog post on IOMMUs and other similar devices last month.)

To actually connect the USB port inside the M1 to the USB type-C connectors on the back of the Mac Mini, we had to interact with a chip on I2C (which means GPIO and I2C drivers) which has customized firmware. We've seen the protocol for these while building our virtualized models; nothing is a big surprise if you have a bird's eye view of the system.

After a few days of figuring out the details of USB, we were finally able to connect an external USB hub and connect a keyboard, mouse and a Flash drive, opening the possibility for running a normal desktop Linux distribution.

Tutorial for USB Boot (Works on Mac Mini/Pro/Air)

1. Download the Ubuntu rootfs

The first step to booting Linux on your Mac M1 is to download the Ubuntu POC rootfs available here. We used a Raspberry Pi image because it was a live USB boot image, so we only had to make minor modifications to boot it.

2. Extract the image

You will need a minimum 16G external USB drive. Extract the image by typing:

tar -xjvf ubuntu-20.10-preinstalled-desktop-arm64+raspi.img.bz2

Then, use disk utility to locate the name of the external disk. Finally, copy the image to the USB drive using:

sudo dd if=ubuntu-20.10-preinstalled-desktop-arm64+raspi.img of=/dev/rYOURUSBDISK bs=1m

Once you complete the dd, you will need to copy the WiFi firmware across to the exfat partition. You can do that by typing:

cp -RLav /usr/share/firmware/wifi /Volumes/system-boot

3. Connect to the Mac

Connect your USB drive to the Mac M1 using a dongle via the USB C port. The USB A ports are not currently supported.

4. Boot into 1TR

To boot into 1TR (the one true recovery OS), turn off your Mac M1 and then hold Power until you see "loading options". Once it loads, you can select the terminal option from the menu bar at the top.

5. Install the Custom Kernel

The next step is to install the custom kernel. We have made a script that makes this step easier for you. You can run it by typing:

bash -c "$(curl -fsSL https://downloads.corellium.info/linuxusbboot.sh)"

The script will prompt you for your username and password. One you see it print "Kernel installed" it's safe to type reboot.

Once you're booted, you'll be prompted for a login. The username is "pi" and the password is "raspberry." The root password is also "raspberry."

7. Reverting to MacOS

To revert to booting MacOS, in 1TR open terminal and type bputil -n

------

Tutorial for NVME Boot ( Works on Mac Mini/Pro/Air )

1. Download the Ubuntu rootfs

2. Create Partition and Flash Image

Warning! Don't attempt this step if you aren't sure what you're doing. If done incorrectly, this can harm your device.

Open Disk util, then click the drive at the top on the left and select partitions. In the partition information, select the + and add a exfat partition of at least 16G. Take note of the disk id and the partition offset. Once you have added the partition, make sure to unmount that partition. You can now dd the image to that new partition. To do that you need to first extract the image:

tar -xjvf ubuntu-20.10-preinstalled-desktop-arm64+raspi-nvme.img.tar.bz2

Then dd to the specific partition you created.

sudo dd if=ubuntu-20.10-preinstalled-desktop-arm64+raspi-nvme.img of=/dev/rYOURDISK bs=1m

Next, create another exfat partition with the size of 200M and call it EFIESP. Then once its finished formatting, you can copy the Wifi firmware to that EFIESP partition by typing:

cp -RLav /usr/share/firmware/wifi /Volumes/EFIESP

3. Boot into 1TR

To boot into 1TR (the one true recovery OS), turn off your Mac M1 and then hold Power until you see "loading options". Once it loads, you can select the terminal option from the menu bar at the top.

4. Install the Custom Kernel

The next step is to install the custom kernel. We have made a script that makes this step easier for you. You can run it by typing:

bash -c "$(curl -fsSL https://downloads.corellium.info/linuxnvmeboot.sh)"

The script will prompt you for your username and password. One you see it print "Kernel installed" it's safe to type reboot.

Once you're booted, you'll be prompted for a login. The username is "pi" and the password is "raspberry." The root password is also "raspberry."

6. Reverting to MacOS

To revert to booting MacOS, in 1TR open terminal and type bputil -n

------

If you're interested in supporting our work on open source projects like these, please consider donating on our behalf to the EFF, who work tirelessly to defend security researchers and protect the digital rights of users and developers. You should also consider supporting the work being done by the folks over at Asahi Linux.

We'd like to extend a very special thanks to the engineers behind PongoOS for contributing their expertise and collaboration. We're looking forward to updating with a version that uses PongoOS as the bootloader!

Set up a meeting with our experts to discuss how Corellium can advance your security research.