My Adventures in Kernel Module Compilation

My Adventures in Kernel Module Compilation using NixOS, debian

The article that got me into this

The actual story about how I tried to compile my kernel module. I came across this article on MuppetLabs.com where Brian Raiter describes how he tries to reduce the binary size.

Well, I read a little, in the first few sections paragraphs he gives an example on how to build your own simple kernel module, and I actually wanted to try making a kernel module for quite a long time. Developing a Kernel module is quite different to the regular development that you might do in C, and I’m not that proficient in C anyway. However, I decided to give it a try on my Mac. Usually, I run Linux through OrbStack, and that time, I also tried to do it that way.

Trying on NixOS with AI’s help

First, I started NixOS, and made the prompt for Claude AI about what I need to do on NixOS to create an environment for kernel development. It gladly obliged, and I tried to run the code it provided - however, there was a problem with the Kernel that is used in Orbstack, because, well they ship their own kernel for reasons.

{ pkgs ? import <nixpkgs> {} }:

let
  # Get the current kernel version
  kernelVersion = builtins.readFile (builtins.toFile "kernel-version" (builtins.readFile "/proc/sys/kernel/osrelease"));
  kernelVersionClean = builtins.replaceStrings ["."] ["_"] (builtins.substring 0 (builtins.match "([0-9]+\.[0-9]+).*" kernelVersion)[0].length kernelVersion);
  
  # Get kernel packages for current version
  linuxPackagesAttr = "linuxPackages_" + kernelVersionClean;
  kernelPackages = pkgs.${linuxPackagesAttr} or pkgs.linuxPackages;
  kernel = kernelPackages.kernel;
  kernelHeaders = kernel.dev;
  
  # Create a basic Makefile if one doesn't exist
  makefile = pkgs.writeTextFile {
    name = "Makefile";
    text = ''
      obj-m += hello.o

      all:
      	make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) modules

      clean:
      	make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) clean

      install:
      	make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) modules_install
    '';
  };
in
  pkgs.mkShell {
    buildInputs = with pkgs; [
      # Basic build tools
      gnumake
      gcc
      binutils
      
      # Kernel-specific dependencies
      kernelHeaders
      pkg-config
      
      # Useful utilities
      kmod # For insmod, rmmod, etc.
    ];
    
    shellHook = ''
      echo "Linux Kernel Module Development Environment"
      echo "Kernel version: ${kernel.modDirVersion}"
      
      # Create a Makefile if one doesn't exist yet
      if [ ! -f Makefile ]; then
        cp ${makefile} Makefile
        echo "Created a basic Makefile for kernel module"
      fi
   
      echo ""
      echo "Usage:"
      echo "  make             - Build the kernel module"
      echo "  make clean       - Clean build files"
      echo "  sudo insmod hello.ko - Load the module (after building)"
      echo "  sudo rmmod hello - Unload the module"
      echo "  dmesg | tail     - View kernel messages"
      echo ""
    '';
  }

That is, I guess they compile their own kernel for every Linux that they ship with, probably for some better compatibility with macOS. In general, I tried to mess around with this a bit, but it didn’t work out.

Switching to Debian

Then, I decided to try Debian, choosing Debian Trixie for the endeavour — meaning 13th version of Debian. Using a Debian Trixie machine, I managed to set up the environment fairly easily, but then when I tried to install Linux headers which are required for Kernel module complication (because kernel module is compiled for a specific kernel) - like in the article, using uname -r for package discovery. Alas, that didn’t work - again, because Orbstack ships with its own kernel, headers for which are of course not present in the package registry. I began searching for where to get the headers, and also I’ve emailed Orbstack Support and wrote to Discord - but I didn’t receive any response at that time. After some time, I finally found the linux kernel headers section on their website, alongside some information. Consequently, I tried to compile with those headers - but there were a lot of missing pieces, and I was forced to take some pieces (i.e. Makefiles and some other things) from the aptitude-provided headers on my Debian Trixie. After I smashed it all together, and compiled it all with headers from kernel 6.12 - it turned out that Debian Trixie was already using kernel 6.13 - so I got some error like Invalid format or something like that.

All in all, I thought that maybe the reason for that is the kernel version mismatch (6.12 vs 6.13) - and tried to find 6.13 headers in Aptitude, which I found in Debian experimental branch - which I had to add to apt.sources the new experimental endpoint.

After that, I managed to install 6.13 kernel from experimental branch, installed the headers from there one more time, gathered it all together with Orbstack’s headers one more time, and tried to compile again - it failed again with the same error, even though the kernel seemed like it was the same one, maybe the custom kernel from Orbstack has some changes that don’t allow the thing to work.

Switching to Bare Metal

In short, I tossed it in the garbage, and decided to continue using the bare-metal NixOS server that I have - it required some patching of the shell.nix file that was generated by AI, which successfully worked for me, and was quite painless. I managed to compile & run hello.c module from the aforementioned article.

The only major thing that I had to change is that my NixOS configuration is done via flakes - and I wanted to use the same Kernel that is used on the server, to avoid downloading a separate one, which meant I needed to take my flake.lock with the exact version of nixpkgs that is used to build the machine configuration. Here’s the nixpkgs.nix used to make it work:

# A nixpkgs instance that is grabbed from the pinned nixpkgs commit in the lock file
# This is useful to avoid using channels when using legacy nix commands
let lock = (builtins.fromJSON (builtins.readFile ./flake.lock)).nodes.nixpkgs-unstable.locked;
in
import (fetchTarball {
  url = "https://github.com/nixos/nixpkgs/archive/${lock.rev}.tar.gz";
  sha256 = lock.narHash;
})

and with that, I managed to run it with old shell interface (nix-shell), and so everything worked wonderfully - and can be reproduced rather easily.

Here’s the final shell.nix that I used:

{ pkgs ? (import ./nixpkgs.nix) { } }:

let
#  kernel = pkgs.linuxPackages_latest;
 # kernelHeaders = kernel.dev;
  kernel = pkgs.linuxPackages_latest.kernel;
  kernelHeaders = kernel.dev;
  
  # Create a basic Makefile if one doesn't exist
  makefile = pkgs.writeTextFile {
    name = "Makefile";
    text = ''
      obj-m += comfile.o

      all:
        make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) modules

      clean:
        make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) clean

      install:
        make -C ${kernelHeaders}/lib/modules/${kernel.modDirVersion}/build M=$(PWD) modules_install
    '';
  };
in
  pkgs.mkShell {
    buildInputs = with pkgs; [
      # Basic build tools
      gnumake
      gcc
      binutils

      # Kernel-specific dependencies
      kernelHeaders
      pkg-config

      # Useful utilities
      kmod # For insmod, rmmod, etc.
      nasm
    ];
    
    shellHook = ''
      echo "Linux Kernel Module Development Environment"
      echo "Kernel version: ${kernel.modDirVersion}"
      # Create a Makefile if one doesn't exist yet
      if [ ! -f Makefile ]; then
        cp ${makefile} Makefile
        echo "Created a basic Makefile for kernel module"
      fi

      echo ""
      echo "Usage:"
      echo "  make             - Build the kernel module"
      echo "  make clean       - Clean build files"
      echo "  sudo insmod hello.ko - Load the module (after building)"
      echo "  sudo rmmod hello - Unload the module"
      echo "  dmesg | tail     - View kernel messages"
      echo ""
    '';
  }