How to Debug the Kernel

One of the major benefits of using a virtual environment is the ability to inspect and modify the state of the whole system under user control. The primary interface to these features is through the TCP-based GDB remote protocol compatible stub.

Important Notes on the Kernel Debug Stub​

• The kernel debug stub presents the multiple CPUs in the system as 'threads' of the process being debugged. For instance, on an iPhone 6, there are two threads: 1 and 2, corresponding to CPU 0 and CPU 1. (GDB protocol does not allow for a thread called 0.)
• Only all-stop mode is supported: if one CPU stops, the others stop as well.
• Single-stepping is supported by using AArch64 single-stepping features. In the case of vCont packets, if a CPU single-steps and other CPUs do not have actions specified, they do not perform a step.
• Hardware breakpoints and watchpoints are supported. (Up to 4 of each at a time.) Software breakpoint packets issued to the stub are converted into hardware breakpoints.
• Memory addresses from the debugger are passed through virtual-to-physical address translation - this is necessary to make it work. Also, only actual RAM is visible to the debugger; access to MMIO regions is ignored and returns 0.
• Only one concurrent debug stub connection is supported per VM.
• High latency can cause the VM to visually “freeze” or “stutter” as the underlying kernel is setting breakpoints and communicating with your GDB. If you use features like watch / awatch / rwatch with conditions, every breakpoint is sent to your local machine, the condition is calculated, and then resumed if it was meant to ignore. This is how GDB functions, but it is typically not noticeable for local kernel debugging. On the local GDB end, nothing will visually indicate that this is happening unless the condition is met, in which case you’ll receive a prompt.

Initial Setup​

1. Set up your virtual device.
2. Connect to the VPN if you are using the cloud product.
3. Grab the Corellium vmlinux file with symbols appropriate for your device:
   • Android 7 and 8 vmlinux-3.18
   • Android 9 and 10 vmlinux-4.4
   • Android 11 vmlinux-5.4
   • Android 12 vmlinux-5.10
   • Android 13 vmmlinux-5.15

Connecting to the Kernel Debug Stub​

Using GDB / LLDB​

To connect using GDB / LLDB, check out our article on GDB / LLDB.

To Connect to the Stub​

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link.

Example 

(gdb) target remote 10.11.1.1:44219

Remote debugging using 10.11.1.1:44219

warning: No executable has been specified and target does not support

determining executable automatically. Try using the "file" command.

0x00000008030e40c8 in ?? ()

To Switch CPUs (in this case, to CPU 1)

(gdb)thread 2

[Switching to thread 2 (Thread 2)]

#0 0x00000008030e40c8 in ?? ()

To Access Monitor Commands

(gdb) monitor sr ttbr1_el1=0x0000000034d4593d
CPU 1, ttbr1_el1 := 0x0000000034d4593d (before: 0x0000000000000000)

Otherwise, use regular GDB commands to control the debug stub.

To Connect to the Stub Without the Binary​

Note: The address and port provided here are for example purposes only. You will need to use the address and port for your particular virtual device. You can find the address and port for your device at the end of the "kernel gdb" link.

Example 

$ lldb

(lldb) gdb-remote 10.11.1.1:44219

Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B

Load Address: 0xfffffff007004000

WARNING: Unable to locate kernel binary on the debugger system.

Process 1 stopped

* thread #1, stop reason = signal SIGINT

    frame #0: 0xfffffff0071a513c

->  0xfffffff0071a513c: bl 0xfffffff00709fc00

    0xfffffff0071a5140: mrs x8, TPIDR_EL1

    0xfffffff0071a5144: ldr x8, [x8, #0x428]

    0xfffffff0071a5148: str xzr, [x8, #0xf0]

Target 0: (No executable module.) stopped.

(lldb)

To Connect to the Stub with the Binary​

Example

$ lldb ~/Documents/Work/kerneli61125

(lldb) gdb-remote 10.11.1.31:39535

Kernel UUID: B99BA98C-3AAA-30E9-B733-07845A9EC56B

Load Address: 0xfffffff007004000

Kernel slid 0x0 in memory.

Loaded kernel file /Users/planetbeing/Documents/Work/kerneli61125

Loading 140 kext modules warning: Can't find binary/dSYM for com.apple.kec.corecrypto (A6668145-C49A-3D84-96E6-80AE4AA9E4B4)

.warning: Can't find binary/dSYM for com.apple.kec.Libm (51AFA03E-8041-3D11-BD40-A6D1AED1C667)

.warning: Can't find binary/dSYM for com.apple.kec.pthread (75DF2E44-845A-3C15-987F-D53AC36CFD72)

.warning: Can't find binary/dSYM for com.apple.driver.usb.cdc (0B64EFC9-CF03-37AF-8C86-4A0C419BC87B)

.warning: Can't find binary/dSYM for com.apple.driver.AppleUSBDeviceMux (84B68AED-C037-3166-BA42-906FE26FA76F)

. done.

Process 1 stopped

* thread #1, stop reason = signal SIGINT

    frame #0: 0xfffffff0071a513c
kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125 + 296
kerneli61125`___lldb_unnamed_symbol1665$$kerneli61125:

-> 0xfffffff0071a513c <+296>: bl 0xfffffff00709fc00 ; ___lldb_unnamed_symbol102$$kerneli61125
   
0xfffffff0071a5140 <+300>: mrs x8, TPIDR_EL1

    0xfffffff0071a5144 <+304>: ldr x8, [x8, #0x428]

    0xfffffff0071a5148 <+308>: str xzr, [x8, #0xf0]

Target 0: (kerneli61125) stopped.

(lldb)

The GDB stub represents CPU cores as threads.

(lldb) thread list

Process 1 stopped

* thread #1: tid = 0x0001, 0xfffffff0071a513c, stop reason = signal SIGINT

  thread #2: tid = 0x0002, 0xfffffff0071a513c
(lldb) thread select 2

* thread #2

    frame #0: 0xfffffff0071a513c

->  0xfffffff0071a513c: bl 0xfffffff00709fc00

    0xfffffff0071a5140: mrs x8, TPIDR_EL1

    0xfffffff0071a5144: ldr x8, [x8, #0x428]

    0xfffffff0071a5148: str xzr, [x8, #0xf0]

(lldb)

While LLDB is specified in the UI, users are free to use GDB as well.

$ gdb

(gdb) target remote 10.11.1.1:4000

Remote debugging using 10.11.1.1:4000

0xffff80001117094c in ?? ()

(gdb) info reg

x0 0xe0 224

x1 0x11c430 1164336

x2 0x0 0

x3 0x4000000000000000 4611686018427387904

x4 0x0 0

x5 0xffff8000dde56000 -140733765558272

x6 0x80 128

...

(gdb) x/10i $pc

=> 0xffff80001117094c: hint #0x1d

   0xffff800011170950: ret

   0xffff800011170954: hint #0x19

   0xffff800011170958: stp x29, x30, [sp,#-16]!

   0xffff80001117095c: mov x29, sp

   0xffff800011170960: bl 0xffff800011170940

   0xffff800011170964: ldp x29, x30, [sp],#16

   0xffff800011170968: hint #0x1d

   0xffff80001117096c: ret

   0xffff800011170970: hint #0x19

You can use the regular LLDB commands to control the debug stub.

Your debugger will work as if it was attached to a hardware debugger (think OpenOCD).

Using IDA​

The following instructions are for IDA 7.0 versions.

1. Select Debugger | Switch debugger... from the main menu, then pick Remote GDB debugger in the dialog box.

2. Then, again from the main menu, select Debugger then Debugger options.... Click the Set specific options button and make sure the Use stepping support checkbox is checked.

3. Finally, select Debugger | Process options... from the main menu, enter the stub's address in the Hostname and Port fields. After this setup, which is saved in the IDA database, select Debugger | Attach to process... to attach to the running device.

To access monitor commands from IDA, locate the GDB command line bar at the bottom of the window (just above the status bar, next to a GDB button). Enter the monitor commands there, without the word "monitor" itself. For instance, instead of monitor sr, simply write sr and press Enter. The output will appear in IDA's text output window above.

Looking to debug the kernel on a virtual device? Book a meeting to deep dive into TCP-based GDB remote protocol usage, handle multiple CPUs, and manage breakpoints with precision.

Advance Your Mobile Security Research with Corellium

Experience Corellium’s groundbreaking virtualization technology for mobile devices and discover never-before-possible mobile vulnerability and threat research for iOS and Android phones. Book a meeting today to explore how our platform can optimize mobile security research and malware analysis.