Amit Bahree's (useless?) insight!

🎉Announcing My New Book: Generative AI in Action📚

Mon, 16 Sep 2024 00:00:00 +0000

In today’s rapidly evolving tech world, mastering Generative AI isn’t just an advantage—it’s a necessity. Are you ready to harness its power to transform your business and solve real-world challenges? I’m excited to announce that my new book, Generative AI in Action, is now available in print and ebook formats from Manning Publications . 📖

Special Launch Offer 🌟

As a thank-you to my early supporters, I’m offering an exclusive discount. Use the code pbbahree at checkout to receive 45% off your purchase of Generative AI in Action in all formats (valid through Sept. 30, 2024)!

Get your discounted copy 👉 » here « .

Cover of Generative AI in Action

Why You Should Read Generative AI in Action ✨

Comprehensive Coverage

This book offers an in-depth introduction to Generative AI, covering foundation models, large language models (#LLMs), small language models (#SLMs), and practical applications. From the basics to advanced topics like prompt engineering, Generative AI in Action provides everything you need to start building and scaling AI solutions. Whether you’re a beginner or a seasoned professional, you’ll find valuable insights to accelerate your AI journey.

Real-World Examples

Discover how enterprises across industries are leveraging Generative AI to innovate and solve complex problems. Whether it’s improving customer engagement or optimizing operations, the practical examples provided can be directly applied to your projects for immediate impact.

Hands-On Techniques

Dive into step-by-step guides and hands-on examples for integrating AI models into your workflows. Learn techniques such as:

Prompt Engineering: Craft effective prompts to unlock the full potential of AI models like GPT-4.
Retrieval-Augmented Generation (RAG): Enhance your AI models with real-time data for improved accuracy.
Model Adaptation: Fine-tune AI models to meet your organization’s specific needs.

Ethical AI and Best Practices

As AI becomes more critical in decision-making, understanding its ethical implications is crucial. Generative AI in Action covers topics like privacy, security, and bias mitigation—ensuring your AI deployments are fair, transparent, and aligned with your organizational values.

Expert Insights

Drawing from my experience helping build the Azure AI platform, I share insider knowledge on leveraging the latest advancements in AI for your projects. This book provides you with the tools to make the most of cutting-edge technologies like large language models (LLMs) and small language models (SLMs).

Advanced Techniques Covered in the Book 🔥

Beyond the basics, Generative AI in Action delves into advanced techniques essential for mastering Generative AI in modern enterprise environments:

Prompt Engineering: Strategies like zero-shot, few-shot, and many-shot learning, along with chain-of-thought reasoning, to optimize AI outputs.
Retrieval-Augmented Generation (RAG): Combine retrieval-based methods with generative models for real-time, relevant data integration.
Model Adaptation and Fine-Tuning: Customize generative models to specific tasks using techniques such as low-rank adaptation and reinforcement learning from human feedback (RLHF).
Chatting with Your Data: Build AI-powered chat systems that interact with enterprise data using vector databases and retrieval techniques.
Scaling and Production Deployment: Strategies for scaling AI solutions while ensuring performance, reliability, and compliance with enterprise standards.
Evaluations and Benchmarks: Learn to evaluate and benchmark AI models using traditional metrics and cutting-edge frameworks.

Explore the GitHub Repository 💾

For those eager to dive into the code, the book has a companion GitHub repository filled with examples and projects to get you started. Check it out at bit.ly/GenAIBook . Explore the code, experiment, and start building your AI-powered solutions today.

Get Your Copy Today! 🛒

Don’t miss this opportunity to lead the AI revolution within your organization. Order your copy of Generative AI in Action and use the code pbbahree to receive 45% off (valid through Sept. 30, 2024). Transform your organization’s AI capabilities today!

With gratitude 💚

Amit Bahree.

In the world of AI, there’s a thrill,
With “Generative AI in Action,” you’ll skill.
From prompts to fine-tuning,
Your projects are blooming,
Grab your copy, and master the drill!

Building a microkernel in Rust (Part 1): Foundations, booting on bare metal

Wed, 25 Feb 2026 00:00:00 +0000

5-Part Series:

Part 0: Why build an OS from scratch?
Part 1 (this): Foundations
Part 2: Communication (coming soon)
Part 3: Concurrency (coming soon)
Part 4: Memory and beyond (coming soon)

GitHub Repository: bahree/rust-microkernel — full source code and build scripts

Docker Image: amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code

You’re about to write code that runs with nothing underneath it.

No operating system. No standard library. No println!. No heap. No file descriptors. No threads. Just you, a CPU, and some RAM.

That probably sounds terrifying. Or maybe thrilling. Honestly, it’s both. The first time you get a bare-metal kernel to print a single character to a serial port, you’ll stare at your terminal for a solid minute. One letter. It takes days of work. And it feels like you’ve built a cathedral.

This post walks through everything that happens between “the CPU wakes up” and “a message appears on screen.” We’ll go instruction by instruction through the assembly boot code, byte by byte through the UART driver, and concept by concept through the design decisions that make it all work. By the end, you’ll have a bootable AArch64 kernel running in QEMU that prints to a virtual serial port, built on a clean, platform-agnostic kernel architecture.

Let’s get to it.

TL;DR

In this part, we boot a minimal Rust kernel on the AArch64 QEMU virt machine. We write ARM assembly to set up a stack, zero the BSS, drop from EL2 (hypervisor level) to EL1 (kernel level), enable floating-point registers, and jump into Rust. The Rust side implements a PL011 UART driver using memory-mapped I/O, wraps it behind a Logger trait for platform abstraction, and calls into a shared kernel entry point. The whole thing compiles to a single ELF binary that QEMU loads directly.

1. What “bare metal” actually means

When you write a normal Rust program (or any program using a higher level language for that matter), there’s a tower of software beneath you. Think about what happens when you call println!("hello"). First, Rust’s standard library formats the string and then calls the OS to write it to stdout. Next, the OS looks up your process’s file descriptor table, finds that stdout points to a terminal, copies your bytes into a kernel buffer, and eventually, a terminal emulator reads those bytes and draws glyphs on screen. And to make all of this work, sitting below all of that, is the firmware which already tested the RAM, configured the CPU caches, initialized the PCI bus, and handed control to the bootloader.

Guess what? We’re removing all of that. 🤖

Our code is the first thing that runs after the CPU wakes up. There’s no runtime to set up the stack. No OS to provide virtual memory. No firmware to initialize the UART (well, QEMU helps us a little here, but we will ignore that for now). We are the bottom of the stack. Everything above us, we have to build.

In Rust terms, this means two attributes at the top of every bare-metal crate:

#![no_std]
#![no_main]

Bare-metal crate attributes

#![no_std] tells the compiler: don’t link the standard library. The standard library (std) gives you Vec, String, Box, println!, file I/O, threads, and networking. All of those features need an underlying operating system. Vec needs a heap allocator, which needs mmap or brk from the OS. println! needs stdout, which needs file descriptors, which need a kernel. Since we ARE the kernel, there’s nothing beneath us to provide those services.

With no_std, we only get core: basic types, traits, iterators, Option, Result, math operations. Things that need zero OS support.

#![no_main] is subtler. Normally, when you compile a Rust binary, the compiler generates a hidden entry point that sets up the stack, initializes the heap allocator, configures panic handling, spawns the main thread, and then calls your main() function. None of that machinery exists on bare metal. We handle all initialization ourselves, in assembly, and we tell the compiler not to look for a standard main entry point.

2. What’s an ELF?

Before we dive into boot code, a quick concept: when you compile a Rust program (or C, or anything that targets a Unix-like system), the output isn’t just a flat blob of machine code. It’s a structured file called an ELF (Executable and Linkable Format).

An ELF file is like a shipping manifest. It says: “here’s a chunk of executable code, load it at this address. Here’s read-only data, put it over here. Here’s uninitialized data (BSS), allocate this much space and zero it. Oh, and the program starts executing at this address.”

When we build our kernel, cargo and the linker produce an ELF binary. QEMU knows how to read ELF files, so it parses the headers, loads each section to the right memory address, and sets the program counter to the entry point (_start). On real hardware without an ELF-aware loader, you’d strip the ELF headers and produce a raw binary. But QEMU makes our lives easier.

The important sections in our ELF:

.text.boot: The very first code that runs (our assembly entry point)
.text: The rest of our compiled Rust code
.rodata: Read-only data (string constants, lookup tables)
.data: Initialized mutable data (statics with non-zero initial values)
.bss: Uninitialized data (statics that start at zero, just a size, no actual bytes in the file)

3. The QEMU virt machine

We’re targeting qemu-system-aarch64 -machine virt, which is QEMU’s generic ARM virtual machine. It doesn’t model any specific real-world board. Instead, it gives us a clean, well-documented set of virtual hardware:

Component	Details
CPU	Cortex-A53 (emulated), 64-bit ARMv8-A
RAM	256 MB starting at `0x4000_0000`
UART	PL011 at `0x0900_0000`
Interrupt controller	GICv2 (Generic Interrupt Controller)
Timer	ARM Generic Timer (CNTP)

Here’s the QEMU command we use to run our kernel:

qemu-system-aarch64 \
  -machine virt,gic-version=2 \
  -cpu cortex-a53 \
  -m 256M \
  -nographic \
  -serial mon:stdio \
  -kernel dist/virt/os-aarch64-virt.elf

QEMU launch command for AArch64 virt

Let’s break down what each of these mean:

-machine virt,gic-version=2: Use the generic ARM virtual machine with a GICv2 interrupt controller. GICv2 is simpler than GICv3 and sufficient for single-core work.
-cpu cortex-a53: Emulate a Cortex-A53 core. This is the same CPU in the Raspberry Pi Zero 2 W.
-m 256M: Give the machine 256 MB of RAM.
-nographic: Don’t open a GUI window. We’re using serial output, not a display.
-serial mon:stdio: Connect the virtual serial port to our terminal’s stdin/stdout. This is how we see output.
-kernel dist/virt/os-aarch64-virt.elf: Load our ELF binary directly. QEMU parses the ELF, loads sections to the right addresses, and jumps to _start.

The beauty of virt is its simplicity. No GPU firmware to deal with, no board-specific quirks, no SD card flashing. Build, run, see output. The iteration cycle is seconds.

4. Setup

You need Rust nightly (for bare-metal features like no_std binaries) and the AArch64 bare-metal target. Our repository includes a rust-toolchain.toml that handles most of this:

[toolchain]
channel = "nightly"
components = ["llvm-tools-preview", "rust-src"]
targets = ["x86_64-unknown-none", "aarch64-unknown-none"]

rust-toolchain.toml

If you’re setting up from scratch:

# Install Rust nightly with AArch64 target
rustup default nightly
rustup target add aarch64-unknown-none
rustup component add llvm-tools-preview rust-src

# Install QEMU (Linux)
sudo apt install qemu-system-aarch64

# Install QEMU (macOS)
brew install qemu

# Clone the repository
git clone https://github.com/bahree/rust-microkernel.git
cd rust-microkernel

Install Rust toolchain and QEMU

5. Build and run

./scripts/build-aarch64-virt.sh demo-ipc
./scripts/run-aarch64-virt.sh

Build and run the AArch64 virt kernel

Expected output:

rustOS: aarch64 QEMU virt boot OK
rustOS: IPC + cooperative scheduling demo
rustOS: kernel online
rustOS: microkernel step 1 (IPC + cooperative scheduling)
sched: starting
task/ping: poll
task/ping: sent ping
task/pong: got ping
task/ping: got pong

Expected serial output

Figure 1: Boot output showing the kernel printing to serial, proving our boot assembly, UART driver, and platform abstraction all work.

If you see that, everything’s working. Press Ctrl-A then X to exit QEMU.

If you don’t pass a feature flag, the default build uses demo-memory (which we’ll cover in Part 4). The demo-ipc flag gives us the cooperative scheduling output shown above.

6. The boot sequence

Here’s what happens from the moment QEMU starts to the moment you see “boot OK” on your terminal:

flowchart TD
    A[QEMU loads ELF to 0x40080000] --> B[CPU starts at _start, running at EL2]
    B --> C[Set stack pointer]
    C --> D[Zero BSS section]
    D --> E{Current exception level?}
    E -->|EL2| F[Set EL1 stack pointer]
    F --> G[Configure SPSR_EL2 for EL1h return]
    G --> H[Set ELR_EL2 to el1_start]
    H --> I[Disable FP/ASIMD trapping at EL2]
    I --> J[eret: drop to EL1]
    J --> K[el1_start]
    E -->|EL1| K
    K --> L[Install exception vector table]
    L --> M[Enable FP/ASIMD at EL1]
    M --> N["bl rust_main"]
    N --> O[Initialize PL011 UART]
    O --> P["Print: boot OK"]
    P --> Q[Call kernel::kmain]

    style B fill:#f9f,stroke:#333
    style J fill:#ff9,stroke:#333
    style Q fill:#9f9,stroke:#333

AArch64 virt boot sequence

That’s a lot of steps. Let’s walk through these; however, first, you need to be able to read the assembly.

7. ARM assembly primer

If you’ve never read assembly before, the key is that we need to understand only a handful of concepts to follow the boot code. And as with most things in life, once you get the hang of it, assembly is surprisingly readable. The thing though is that everything is just very, very explicit about what it’s doing.

7.1 Registers

ARM gives you 31 general-purpose 64-bit registers, named x0 through x30. Think of them as 31 local variables that live inside the CPU itself, way faster than RAM. A register access takes maybe one clock cycle. A RAM access? Hundreds.

Some registers have conventional roles:

x0 through x7: Function arguments and return values
x29: Frame pointer (like rbp on x86)
x30: Link register, holds the return address after a bl (branch-with-link) call
sp: Stack pointer, tracks the top of the call stack
xzr: The zero register. Always reads as zero, discards writes. Surprisingly useful.

You’ll also see w0 through w30. These are the lower 32 bits of the corresponding x register. w0 is the bottom half of x0. ARM uses these when working with 32-bit values.

7.2 System registers

Beyond the general-purpose registers, ARM has a separate world of system registers that control CPU behavior. You can’t use them in normal instructions like add or sub. You access them with special instructions:

mrs x0, CurrentEL: Move from system register to general-purpose register. Read the current exception level into x0.
msr spsr_el2, x0: Move from general-purpose register to system register. Write x0 into the saved program status register for EL2.

These are the knobs and dials that configure how the CPU works - what exception level we’re at, whether floating-point is enabled, where the exception vector table lives, and what happens on an eret.

7.3 Common instructions

Here’s what you’ll see in the boot code:

Instruction	What it does
`ldr x0, =label`	Load the address of `label` into `x0`
`mov sp, x0`	Copy `x0` into the stack pointer
`str xzr, [x1], #8`	Store zero to the address in `x1`, then add 8 to `x1`
`cmp x1, x2`	Compare `x1` and `x2` (sets condition flags)
`b.ge 2f`	Branch forward to label `2:` if the comparison was greater-than-or-equal
`b.ne label`	Branch to `label` if not equal
`bl rust_main`	Branch with link: save return address in `x30`, jump to `rust_main`
`bic x0, x0, #(1 << 10)`	Bit clear: clear bit 10 in `x0`
`orr x0, x0, #(3 << 20)`	Bitwise OR: set bits 20 and 21 in `x0`
`lsr x1, x1, #2`	Logical shift right by 2 bits
`and x1, x1, #3`	Bitwise AND with 3 (keep only bits 0 and 1)
`adr x0, label`	Load the PC-relative address of `label` into `x0`
`isb`	Instruction synchronization barrier (flush pipeline)
`eret`	Exception return (drop to lower exception level)
`wfe`	Wait for event (low-power sleep)

7.4 Local labels and directives

Assembly uses numbered local labels like 1:, 2:, 3:. You reference them with a direction suffix: 1b means “search backward for label 1:” and 2f means “search forward for label 2:.” This avoids name collisions when the same pattern (like a loop) appears multiple times in the same file.

.section .text.boot tells the assembler: “put the following code into a section named .text.boot.” The linker script (which we’ll cover later) uses this name to ensure our boot code lands at the very start of the binary, exactly where the CPU expects it.

.global _start makes the _start symbol visible to the linker so that it can be used as the entry point.

8. Walking through boot.S

This is the heart of the post. Let’s go through the actual boot assembly line by line.

Here’s the file: crates/arch_aarch64_virt/src/boot.S. We’re showing the boot sequence portion (the first 60 lines) - you can see the full code from the repository.

.section .text.boot
.global _start
_start:
  // Set stack
  ldr x0, =__stack_top
  mov sp, x0

  // Zero BSS: [__bss_start, __bss_end)
  ldr x1, =__bss_start
  ldr x2, =__bss_end
1:
  cmp x1, x2
  b.ge 2f
  str xzr, [x1], #8
  b 1b
2:
  // If we entered at EL2 (typical for QEMU virt), drop to EL1 so the kernel runs
  // in a simpler environment (EL1 + GICv2 + CNTP timer).
  mrs x1, CurrentEL
  lsr x1, x1, #2
  and x1, x1, #3
  cmp x1, #2
  b.ne el1_start

  // Set up an EL1 stack pointer.
  ldr x0, =__stack_top
  msr sp_el1, x0

  // Configure EL2 to return to EL1h.
  mov x0, #(0b0101)         // EL1h
  msr spsr_el2, x0
  adr x0, el1_start
  msr elr_el2, x0

  // Enable FP/ASIMD access at EL1 and ensure EL2 doesn't trap it.
  mrs x0, cptr_el2
  bic x0, x0, #(1 << 10)    // TFP = 0 (don't trap FP/ASIMD)
  msr cptr_el2, x0
  isb
  eret

el1_start:
  // Install a minimal exception vector table for the *current* exception level.
  // QEMU `virt` typically enters at EL2, so we must set VBAR_EL2 (not just VBAR_EL1).
  adr x0, vectors
  msr vbar_el1, x0
  isb

  // Enable FP/ASIMD for Rust/LLVM.
  // Rust/LLVM may use NEON registers for struct copies/memcpy even in early bring-up.
  // If FP is disabled, this traps with EC=0x07 (FP/ASIMD access trap).
  mrs x0, cpacr_el1
  orr x0, x0, #(3 << 20)   // FPEN = 0b11
  msr cpacr_el1, x0
  isb

  bl rust_main
3:
  wfe
  b 3b

boot.S: the complete boot sequence (crates/arch_aarch64_virt/src/boot.S, lines 1-60)

Now let’s walk through each piece to help understand what it means.

8.1 Stack setup (lines 5-6)

  ldr x0, =__stack_top
  mov sp, x0

Setting up the stack pointer

The very first thing we do is set up a stack. But why? Because Rust literally cannot run without one. Every function call pushes a return address onto the stack. Every local variable lives on the stack. Without a valid stack pointer, the first bl instruction would try to save the return address to… nowhere. Instant crash.

__stack_top is a symbol defined by our linker script. It points to the top of a 64 KB stack memory region. ARM stacks grow downward (from high addresses to low), so we point sp at the top.

Why 64 KB? It’s a reasonable starting size. We don’t have many nested function calls yet, and we don’t have a heap, so the stack doesn’t need to be huge. If we run out, we’ll get a data abort (the ARM equivalent of a segfault). We’d increase it then.

8.2 BSS zeroing (lines 8-16)

  ldr x1, =__bss_start
  ldr x2, =__bss_end
1:
  cmp x1, x2
  b.ge 2f
  str xzr, [x1], #8
  b 1b
2:

Zeroing the BSS section

BSS stands for “Block Started by Symbol” (a historical name from the 1950s). It’s the section where uninitialized global and static variables live. In Rust, if you write static mut COUNTER: u32 = 0;, the compiler puts it in .bss. The key insight: the ELF file doesn’t actually store the zeros. It just records the size. That saves space in the binary. But it means the memory might contain garbage when we start.

The C and Rust languages both guarantee that uninitialized statics start at zero. Before any Rust code runs, we have to zero the entire BSS region manually. That’s what this loop does:

Load the start and end addresses of .bss (defined by the linker script) into x1 and x2.
Compare them. If x1 >= x2, the BSS is empty (or we’re done), skip to label 2:
Store zero (xzr, the zero register) to the address in x1, then increment x1 by 8 (one 64-bit word)
Jump back to the comparison

The [x1], #8 syntax is a post-increment addressing mode. It means “use the address in x1 for the store, THEN add 8 to x1.” It’s a single instruction that does two things. ARM loves these compound operations.

8.3 Checking the exception level (lines 19-23)

  mrs x1, CurrentEL
  lsr x1, x1, #2
  and x1, x1, #3
  cmp x1, #2
  b.ne el1_start

Detecting the current exception level

Now things get interesting. We need to know what exception level the CPU is running at. QEMU’s virt machine starts us at EL2 (hypervisor level), but we want to run our kernel at EL1 (the normal kernel level). If we’re already at EL1 for some reason, we skip the drop.

mrs x1, CurrentEL reads the CurrentEL system register. The exception level is encoded in bits [3:2] of this register (ARM’s register designs can be a little eccentric). So we shift right by 2 (lsr x1, x1, #2) and mask off everything except the bottom 2 bits (and x1, x1, #3). Now x1 holds 0, 1, 2, or 3 corresponding to EL0 through EL3.

If it’s 2, we proceed with the EL2-to-EL1 drop. If it’s not, we jump straight to el1_start.

8.4 Exception levels explained

Before we get into the drop, let’s talk about why exception levels exist.

ARM’s AArch64 architecture has four privilege levels:

Level	Name	Who runs here	What they can do
EL0	Application	User programs	Normal instructions only. Can’t touch hardware.
EL1	Kernel	Operating systems	Configure MMU, handle interrupts, access all memory
EL2	Hypervisor	Virtual machine monitors	Virtualize EL1 guests, trap privileged operations
EL3	Secure Monitor	TrustZone firmware	Switch between secure and non-secure worlds

Think about it this way: EL0 is a sandbox. Applications can compute, but they can’t mess with memory mappings, disable interrupts, or talk to hardware directly. If they try, the CPU traps to EL1, and the kernel decides what to do (usually: kill the process with a signal).

EL1 is where kernels live. Linux, macOS, Windows, and our rustOS all run here. You can configure page tables, handle interrupts, and access device memory.

EL2 is for hypervisors. If you want to run multiple operating systems on the same hardware (like AWS does with EC2 instances), the hypervisor at EL2 virtualizes the hardware so each guest OS at EL1 thinks it has the machine to itself.

EL3 is for firmware-level security (ARM TrustZone). We don’t touch it.

QEMU starts us at EL2 because it’s the most flexible starting point. If you were writing a hypervisor, you’d stay there. Since we’re writing a kernel, we drop to EL1.

8.5 The EL2 to EL1 drop (lines 26-40)

  // Set up an EL1 stack pointer.
  ldr x0, =__stack_top
  msr sp_el1, x0

  // Configure EL2 to return to EL1h.
  mov x0, #(0b0101)         // EL1h
  msr spsr_el2, x0
  adr x0, el1_start
  msr elr_el2, x0

  // Enable FP/ASIMD access at EL1 and ensure EL2 doesn't trap it.
  mrs x0, cptr_el2
  bic x0, x0, #(1 << 10)    // TFP = 0 (don't trap FP/ASIMD)
  msr cptr_el2, x0
  isb
  eret

Dropping from EL2 to EL1

This is the most intricate part of the boot sequence. Here’s the trick: you can’t just “jump” to a lower exception level. ARM doesn’t have a “go to EL1” instruction. Instead, you use the exception return mechanism backwards.

Normally, when an exception occurs (such as an interrupt), the CPU saves the current state and jumps to a higher exception level. The handler processes the exception, then executes eret (exception return) to return. We’re abusing this: we set up the return state registers to point at EL1, then execute eret as if we were “returning” from an exception that never happened.

Here’s each step:

msr sp_el1, x0 sets the stack pointer that EL1 will use. Each exception level has its own stack pointer. We’re at EL2 right now, so sp refers to sp_el2. We need to pre-configure sp_el1 before we get there.

mov x0, #(0b0101) / msr spsr_el2, x0 configures the Saved Program Status Register. When eret executes, the CPU restores the processor state from spsr_el2. The value 0b0101 (decimal 5) means: return to EL1 using the handler stack pointer variant (called “EL1h”). Bits [3:2] = 01 select EL1. Bit [0] = 1 selects the h variant, meaning EL1 will use sp_el1 as its stack pointer (rather than sp_el0).

adr x0, el1_start / msr elr_el2, x0 sets the Exception Link Register. This is the address the CPU will jump to on eret. We point it at el1_start, the label where our EL1 code begins. adr computes a PC-relative address, which is important because our code might not be running at the address the linker assumed (though, in practice, it is for us).

bic x0, x0, #(1 << 10) / msr cptr_el2, x0 disables the floating-point trap at EL2. By default, EL2 can trap FP/ASIMD (NEON) instructions executed at EL1. If we don’t clear this bit, the first time Rust tries to use a NEON register (which happens sooner than you’d think), the CPU will trap to EL2 and crash. bic stands for “bit clear.” It clears bit 10 (the TFP flag) in the cptr_el2 register.

isb is an Instruction Synchronization Barrier. It forces the CPU to finish processing all pending changes to system registers before executing the next instruction. Without it, the pipeline might still be using stale configuration values.

eret does the actual drop. It atomically loads the program counter from elr_el2 and the processor state from spsr_el2. In one instruction, we go from EL2 to EL1 and start executing at el1_start.

It’s a beautiful hack, honestly. The hardware designers intended eret for returning from exception handlers. We’re using it as a one-way door to a lower privilege level.

8.6 Vector table installation (lines 43-47)

  adr x0, vectors
  msr vbar_el1, x0
  isb

Installing the exception vector table

The CPU needs to know where to jump when an exception occurs (an interrupt, a system call, a data abort, etc.). ARM uses a vector table: a fixed-layout block of code that contains an entry for each exception type. There are 16 entries, every 128 bytes (0x80) apart, for a total of 2048 bytes (0x800).

We load the address of our vectors table and write it to vbar_el1 (Vector Base Address Register for EL1). When an exception happens at EL1, the CPU adds an offset to this base address and starts executing there. We’ll cover the vector table in detail in Part 3 when we implement interrupt handling.

8.7 FP/ASIMD enablement: why Rust needs NEON (lines 49-55)

  mrs x0, cpacr_el1
  orr x0, x0, #(3 << 20)   // FPEN = 0b11
  msr cpacr_el1, x0
  isb

Enabling floating-point and SIMD at EL1

You might wonder: we’re writing a kernel, not a graphics engine. Why do we need floating-point?

Here’s the thing. LLVM (Rust’s code generator) uses NEON registers for more than just floating-point math. It uses them for memory operations like memcpy and struct copies. If you have a struct that’s, say, 32 bytes, LLVM might decide the fastest way to copy it is to load it into a pair of 128-bit NEON registers and store it back. This happens even in integer-only code. Even in early boot.

If FP/ASIMD is disabled when this happens, the CPU generates a synchronous exception with exception class EC=0x07 (FP/ASIMD access trap). Your kernel crashes before it even prints “hello.”

We handled the EL2 side already (clearing TFP in cptr_el2). Now at EL1, we set FPEN (bits [21:20]) in cpacr_el1 to 0b11, which means “don’t trap FP/ASIMD at EL1 or EL0.” The orr instruction sets those bits. Another isb to synchronize.

8.8 Jumping to Rust (lines 57-60)

  bl rust_main
3:
  wfe
  b 3b

Calling into Rust

Finally, bl rust_main is a branch-with-link: it saves the return address in x30 (the link register) and jumps to rust_main. This is where we leave assembly and enter Rust.

The three lines after it are a safety net. rust_main is declared as -> ! (never returns), but if something goes horribly wrong and it does return, we don’t want the CPU executing random memory. So we enter an infinite loop: wfe (wait for event, which puts the CPU in a low-power sleep state) followed by a branch back to wfe.

9. The Rust entry point

Now we’re in Rust. Let’s look at the actual main.rs for the AArch64 virt platform.

#![no_std]
#![no_main]

use core::panic::PanicInfo;
use hal::log::Logger;

core::arch::global_asm!(include_str!("boot.S"));

// QEMU `virt` PL011 UART base.
const UART0_BASE: usize = 0x0900_0000;

struct UartLogger;

impl UartLogger {
    #[inline(always)]
    fn mmio_write(offset: usize, val: u32) {
        unsafe { core::ptr::write_volatile((UART0_BASE + offset) as *mut u32, val) }
    }

    #[inline(always)]
    fn mmio_read(offset: usize) -> u32 {
        unsafe { core::ptr::read_volatile((UART0_BASE + offset) as *const u32) }
    }

    fn putc(c: u8) {
        // FR (0x18) bit5 = TXFF (transmit FIFO full)
        while (Self::mmio_read(0x18) & (1 << 5)) != 0 {}
        Self::mmio_write(0x00, c as u32);
    }

    pub(crate) fn puts(s: &str) {
        for &b in s.as_bytes() {
            if b == b'\n' {
                Self::putc(b'\r');
            }
            Self::putc(b);
        }
    }
}

impl hal::log::Logger for UartLogger {
    fn log(&self, s: &str) {
        UartLogger::puts(s);
    }
}

mod timer;
mod preempt;
mod mem;

#[unsafe(no_mangle)]
pub extern "C" fn rust_main() -> ! {
    let logger = UartLogger;
    logger.log("rustOS: aarch64 QEMU virt boot OK\n");

    #[cfg(feature = "demo-ipc")]
    {
        logger.log("rustOS: IPC + cooperative scheduling demo\n");
        kernel::kmain(&logger)
    }

    #[cfg(feature = "demo-timer")]
    {
        logger.log("rustOS: timer interrupts demo\n");
        timer::init();
        logger.log("rustOS: timer started, entering idle loop\n");
        loop {
            hal::arch::halt();
        }
    }

    #[cfg(feature = "demo-preempt")]
    {
        logger.log("rustOS: preemptive multitasking demo\n");
        preempt::init();
        extern "C" {
            fn start_first(ctx: *const preempt::Context) -> !;
        }
        unsafe { start_first(preempt::first_context()) }
    }

    #[cfg(feature = "demo-memory")]
    {
        logger.log("rustOS: memory management demo (frames + page tables)\n");
        mem::demo();
        loop {
            hal::arch::halt();
        }
    }

    #[cfg(not(any(feature = "demo-ipc", feature = "demo-timer",
                   feature = "demo-preempt", feature = "demo-memory")))]
    {
        logger.log("rustOS: no demo selected, halting\n");
        loop {
            hal::arch::halt();
        }
    }
}

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    UartLogger::puts("rustOS: PANIC\n");
    loop {
        hal::arch::halt();
    }
}

crates/arch_aarch64_virt/src/main.rs

Let’s unpack the important pieces.

9.1 `global_asm!` and including boot.S

core::arch::global_asm!(include_str!("boot.S"));

Including assembly in the Rust build

This line tells the Rust compiler: “take the contents of boot.S and include them as global assembly in this compilation unit.” The assembler processes our boot code, the linker resolves the symbols (_start, rust_main, __stack_top, etc.), and everything ends up in one binary. It’s the bridge between our assembly boot code and Rust.

9.2 `#[unsafe(no_mangle)]` and `extern "C"`

#[unsafe(no_mangle)]
pub extern "C" fn rust_main() -> ! {

The Rust entry point signature

Two things happening here.

#[unsafe(no_mangle)] prevents the Rust compiler from mangling the function name. Normally, Rust encodes type information into symbol names (so rust_main might become something like _ZN17arch_aarch64_virt9rust_main17h3a2b1c4d5e6f7g8hE). Our assembly code calls bl rust_main and needs to find that exact name.

extern "C" says: use the C calling convention. Rust’s own calling convention isn’t stable and can change between compiler versions. The C ABI is well-defined on every platform. On AArch64, the C calling convention puts the first argument in x0, the second in x1, return values in x0, and so on. Since assembly calls this function, we need a stable, predictable calling convention.

-> ! means this function never returns. On bare metal, there’s nothing to return to. The CPU would start executing whatever random bytes follow in memory. The function must either loop forever or halt the CPU.

9.3 The feature gates

#[cfg(feature = "demo-ipc")]
{
    logger.log("rustOS: IPC + cooperative scheduling demo\n");
    kernel::kmain(&logger)
}

Compile-time feature selection

The #[cfg(feature = "...")] attributes are compile-time switches. Depending on which feature you pass to the build script, a different code path gets compiled in. Only one demo runs at a time. This is a common pattern in embedded Rust: use cargo features to select between different configurations without runtime overhead.

9.4 The panic handler

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    UartLogger::puts("rustOS: PANIC\n");
    loop {
        hal::arch::halt();
    }
}

Bare-metal panic handler

On bare metal, you must define what happens when Rust panics (array out of bounds, unwrap() on None, explicit panic!(), etc.). There’s no OS to catch it. Our handler prints “PANIC” to the UART and halts. In a more sophisticated kernel, you’d print the panic message and a stack trace. For now, knowing something panicked is enough.

10. PL011 UART: talking to hardware via MMIO

In a normal program, every memory address maps to a byte of RAM. Address 0x1000 holds some data, you read it, you write it, it’s just storage. On bare metal, some addresses are special: they’re connected to hardware peripherals instead of RAM. Writing to address 0x0900_0000 on our QEMU virt machine doesn’t store anything in memory. It sends a byte out the serial line. This is memory-mapped I/O (MMIO).

The PL011 UART sits at that address. When you write a byte to offset 0x00 from that base, the UART hardware transmits it. When you read from offset 0x18, you get the UART’s flag register, telling you whether the transmit FIFO is full. These addresses come from the hardware manufacturer’s specification (or in QEMU’s case, from QEMU’s source code and the virt machine documentation). There’s no way to discover them at runtime. You look them up.

10.1 The UartLogger implementation

Let’s look at how we actually write bytes to the UART:

const UART0_BASE: usize = 0x0900_0000;

impl UartLogger {
    #[inline(always)]
    fn mmio_write(offset: usize, val: u32) {
        unsafe { core::ptr::write_volatile((UART0_BASE + offset) as *mut u32, val) }
    }

    #[inline(always)]
    fn mmio_read(offset: usize) -> u32 {
        unsafe { core::ptr::read_volatile((UART0_BASE + offset) as *const u32) }
    }
}

MMIO helper functions (from main.rs)

write_volatile and read_volatile are critical. Normally, the Rust compiler (through LLVM) is free to optimize away memory operations. If you write to an address and never read it back, the optimizer might skip the write entirely. “Why bother,” it thinks, “nobody’s going to look at it.”

But for MMIO, the hardware IS looking at it. Writing to the UART data register transmits a byte. The compiler can’t see that side effect. volatile tells the compiler: “this access has effects you can’t reason about. Do it exactly as written, in exactly this order.”

Both functions are unsafe because we’re casting raw integers to pointers and dereferencing them. In safe Rust, pointer arithmetic is forbidden. On bare metal, it’s the only way to talk to hardware.

10.2 Sending a character

fn putc(c: u8) {
    // FR (0x18) bit5 = TXFF (transmit FIFO full)
    while (Self::mmio_read(0x18) & (1 << 5)) != 0 {}
    Self::mmio_write(0x00, c as u32);
}

PL011 character transmission (from main.rs)

The PL011 UART has a transmit FIFO (a small hardware buffer, typically 16 bytes). When you write a byte to the data register (offset 0x00), it goes into the FIFO. The UART hardware drains the FIFO at the configured baud rate, sending bits out the serial line.

But the FIFO can fill up. If you write faster than the UART can transmit, the FIFO overflows and bytes get lost. So before writing, we check the Flag Register (offset 0x18). Bit 5 is TXFF (transmit FIFO full). If it’s set, we spin-wait until there’s room.

This is called polling or busy-waiting. It’s the simplest approach and perfectly fine for debug output. In a production driver, you’d use interrupts instead (the UART can signal when the FIFO has room, so the CPU can do other work instead of spinning).

10.3 Sending a string

pub(crate) fn puts(s: &str) {
    for &b in s.as_bytes() {
        if b == b'\n' {
            Self::putc(b'\r');
        }
        Self::putc(b);
    }
}

PL011 string transmission with newline conversion (from main.rs)

Two things to notice. First, we iterate over the raw bytes of the string (s.as_bytes()), not characters. At this level, we’re dealing with bytes, not Unicode code points.

Second, the \n to \r\n conversion. Serial terminals expect both a carriage return (\r, move cursor to column 0) and a line feed (\n, move cursor down one line). If you send just \n, many terminals will move down without returning to the left edge, giving you a staircase effect. So we insert a \r before every \n.

10.4 The Logger trait implementation

impl hal::log::Logger for UartLogger {
    fn log(&self, s: &str) {
        UartLogger::puts(s);
    }
}

Logger trait implementation (from main.rs)

This bridges the gap between our platform-specific UART driver and the platform-agnostic kernel. The kernel never calls UartLogger::puts directly. It calls logger.log() on a trait object. More on this in the next section.

11. The platform-agnostic kernel

Here’s the design payoff. The kernel crate knows nothing about ARM, nothing about UART, nothing about QEMU. It just knows it has something that can log strings.

11.1 The Logger trait

pub trait Logger {
    fn log(&self, s: &str);
}

crates/hal/src/log.rs

Three lines. That’s the entire hardware abstraction layer for output. Any platform that can print strings can implement this trait.

11.2 The kernel entry

#![no_std]

use hal::log::Logger;

mod ipc;
mod sched;

use core::cell::UnsafeCell;

#[repr(transparent)]
struct RouterCell(UnsafeCell<ipc::Router>);
unsafe impl Sync for RouterCell {}

// Force the router into a writable section. On some bare-metal targets, a `static`
// with interior mutability can otherwise end up in a read-only segment, causing
// a data abort when we first write to it (exactly what we saw on aarch64 QEMU virt).
#[link_section = ".data"]
static ROUTER: RouterCell = RouterCell(UnsafeCell::new(ipc::Router::new()));

pub fn kmain(logger: &dyn Logger) -> ! {
    logger.log("rustOS: kernel online\n");
    logger.log("rustOS: microkernel step 1 (IPC + cooperative scheduling)\n");

    let router: &mut ipc::Router = unsafe { &mut *ROUTER.0.get() };

    let mut ping = sched::PingTask::new();
    let mut pong = sched::PongTask::new();
    let mut tasks: [&mut dyn sched::Task; 2] = [&mut ping, &mut pong];

    sched::run(&mut tasks, logger, router)
}

crates/kernel/src/lib.rs

Look at the kmain signature: fn kmain(logger: &dyn Logger) -> !. It takes a trait object (&dyn Logger), not a concrete type. The kernel doesn’t know whether logger is a PL011 UART, a COM1 serial port, or a carrier pigeon with a Morse code encoder. It just knows it can call .log(). If you wanted to port this kernel to RISC-V or even a microcontroller with an SPI display, you’d implement Logger for that platform’s output device and call kmain with it.

Notice the #[link_section = ".data"] on the ROUTER static. This is a hard-won lesson. On AArch64, the linker sometimes places statics with interior mutability (like UnsafeCell) into read-only sections. The first write causes a data abort. Forcing it into .data guarantees it lands in a writable section. We learned this the painful way.

The IPC router and scheduler are the subject of Part 2 . For now, just know that kmain is where platform-agnostic kernel logic lives.

11.3 The halt function

#[inline(always)]
pub fn halt() {
    #[cfg(target_arch = "x86_64")]
    unsafe {
        core::arch::asm!("hlt", options(nomem, nostack, preserves_flags));
    }

    #[cfg(target_arch = "aarch64")]
    unsafe {
        // Use WFI (wait-for-interrupt) so we reliably sleep until the next IRQ.
        // WFE can return immediately if an event is already latched.
        core::arch::asm!("wfi", options(nomem, nostack, preserves_flags));
    }

    #[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
    {
        loop {}
    }
}

crates/hal/src/arch.rs

This is another piece of the hardware abstraction layer. On AArch64, wfi (wait for interrupt) puts the CPU into a low-power sleep state until an interrupt fires. On x86_64, the equivalent is hlt. The kernel code just calls hal::arch::halt() and doesn’t care which instruction runs underneath.

The comment about wfe vs wfi is interesting. wfe (wait for event) can return immediately if an event flag is already set, which means your “sleep” loop might spin instead of sleeping. wfi is more predictable: it always waits for an actual interrupt.

12. Linker scripts

On bare metal, there’s no OS to decide where your code lives in memory. The linker script is the document that says: put .text.boot first (so the CPU’s entry point is at the right address), put .text after it, then .rodata, .data, .bss, and finally the stack.

Our linker script for the virt platform places the kernel at 0x40080000. This is within the RAM region that starts at 0x40000000 on the QEMU virt machine. The 0x80000 offset is a convention (it matches the Raspberry Pi’s load address offset), though for QEMU it’s somewhat arbitrary as long as it’s within RAM.

The linker script also defines symbols like __bss_start, __bss_end, and __stack_top that the assembly boot code references. Without these, the assembler would have no idea where BSS begins or where to put the stack.

The build.rs file for our crate tells cargo to use this linker script:

fn main() {
    println!("cargo:rerun-if-changed=linker.ld");
    println!("cargo:rerun-if-changed=src/boot.S");
    let manifest_dir = std::env::var("CARGO_MANIFEST_DIR")
        .expect("CARGO_MANIFEST_DIR not set");
    println!("cargo:rustc-link-arg=-T{}/linker.ld", manifest_dir);
}

crates/arch_aarch64_virt/build.rs

The cargo:rustc-link-arg=-T.../linker.ld line passes the linker script to the linker. Without it, the linker would use its default layout, which almost certainly wouldn’t put our code at the right address.

13. Hands-on exercise

Here’s a quick challenge. Open crates/kernel/src/lib.rs and add a second log message to kmain:

pub fn kmain(logger: &dyn Logger) -> ! {
    logger.log("rustOS: kernel online\n");
    logger.log("rustOS: microkernel step 1 (IPC + cooperative scheduling)\n");
    logger.log("rustOS: hello from YOUR_NAME_HERE!\n");  // add this
    // ... rest of the function

Adding a custom log message

Rebuild and run:

./scripts/build-aarch64-virt.sh demo-ipc
./scripts/run-aarch64-virt.sh

Rebuild and verify

You should see your message appear between the boot messages and the scheduler output. One line of Rust. It traveled through a trait object, into a UART driver, through memory-mapped I/O, into a virtual PL011 peripheral, and out QEMU’s emulated serial port to your terminal.

Let’s step back and look at what we’ve got. A 60-line assembly boot sequence that sets up a stack, zeros BSS, drops from EL2 to EL1, enables floating-point, installs an exception vector table, and hands off to Rust. A Rust entry point that initializes a PL011 UART driver and calls into a platform-agnostic kernel. A three-line Logger trait that abstracts away all the hardware details. And a kernel entry function that doesn’t know or care what CPU architecture it’s running on. It boots. It prints. It works. And everything from here builds on this foundation. The IPC system in Part 2 , the timer interrupts in Part 3 , the MMU in Part 4 : they all stand on the boot sequence and UART driver we just walked through. Get this part right, and the rest is (relatively) straightforward.

14. Brief appendix: how would this differ on other platforms?

If you’re curious about x86_64 or Raspberry Pi, here’s a quick sketch.

On x86_64, the boot process is much more complex because of backwards compatibility with the 1978 Intel 8086. The CPU starts in 16-bit real mode, where you can only address 1 MB of memory. You have to transition through protected mode (32-bit) and then long mode (64-bit), setting up GDTs (Global Descriptor Tables), enabling PAE (Physical Address Extension), and configuring initial page tables (paging is required for 64-bit mode). Most Rust OS projects use the bootloader crate by Philipp Oppermann, which handles all these mode transitions and loads your kernel ELF. For serial output, x86 has legacy COM ports at I/O address 0x3F8, accessed with special in and out instructions (port-mapped I/O rather than memory-mapped I/O). The uart_16550 crate wraps this nicely.

On Raspberry Pi, something unexpected happens: the GPU boots first, not the CPU. The VideoCore GPU reads bootcode.bin from the SD card, loads start.elf, reads config.txt for settings, and then loads your kernel (kernel8.img) to address 0x80000. Only then does the ARM CPU start. The boot assembly is simpler than our QEMU virt version (no EL2 drop needed, the firmware handles that), but the UART driver is more complex. The Pi has two UARTs: a full PL011 (which is connected to Bluetooth by default) and a simpler Mini-UART on GPIO pins 14/15. The Mini-UART’s clock is tied to the GPU frequency, which makes baud rate configuration trickier. And of course, the development cycle is slower: build, copy to SD card, plug into Pi, power on, check output.

Both platforms share the same kernel crate and Logger trait. That’s the whole point of the abstraction.

15. References

ARM Architecture Reference Manual (ARMv8-A) (Section D1 for exception levels, Section G for system registers)
OSDev Wiki (community-driven OS development knowledge base)
Writing an OS in Rust by Philipp Oppermann (excellent x86_64 series)
The Embedded Rust Book (no_std patterns and bare-metal Rust)
QEMU AArch64 virt machine documentation
PL011 UART Technical Reference Manual

5-Part Series:

Part 0: Why build an OS from scratch?
Part 1 (this): Foundations
Part 2: Communication (coming soon)
Part 3: Concurrency (coming soon)
Part 4: Memory and beyond (coming soon)

Building a microkernel in Rust (Part 0): Why build an OS from scratch?

Sun, 22 Feb 2026 00:00:00 +0000

5-Part Series:

Part 0 (this): Why build an OS from scratch?
Part 1: Foundations (coming soon)
Part 2: Communication (coming soon)
Part 3: Concurrency (coming soon)
Part 4: Memory and beyond (coming soon)

GitHub Repository: bahree/rust-microkernel — full source code and build scripts

Docker Image: amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code

Why this, why now?

I recently wrapped up an incredible chapter at Microsoft, working on the AI engineering team in CoreAI. Between roles, I found myself on gardening leave with something rare in this industry: a few weeks of unstructured time. And instead of doing the sensible thing (rest, catch up on sleep, maybe touch grass), my brain went: “You know what sounds fun? Writing an operating system from scratch.”

Look, I’ve spent the last several years deep in AI/ML, from foundational model development to applied AI at scale. That work isn’t going anywhere (the LLM from scratch series on this blog is a good example of that curiosity). But I’ve had this itch for a while now to go back to the fundamentals. The stuff that sits beneath all those PyTorch tensors and CUDA kernels. The stuff that actually makes a computer compute.

There’s an old joke: a QA engineer walks into a bar. Orders 1 beer. Orders 0 beers. Orders 99999999 beers. Orders -1 beers. Orders a lizard. Orders NULL beers. First real customer walks in and asks where the bathroom is. The bar bursts into flames.

That’s basically what happens when you try to run code without an OS. You think you’re ordering a beer, but there’s nobody behind the counter, no glasses, no refrigerator, and the building doesn’t have plumbing yet. You are the plumbing. This series is about building that plumbing from nothing.

Part 0? Really?

Yes, we’re starting at zero. Because real programmers count from zero, and honestly, this part is about understanding what we’re building before we build it. Think of it as the moment before you start cooking, where you read the whole recipe, check you’ve got the ingredients, and figure out what you’re actually making. You don’t skip that step. (Okay, maybe you do sometimes. But you shouldn’t.)

If you’ve ever wondered how code even runs without an operating system underneath, or what happens between pressing the power button and seeing a login screen, or why everyone says “virtual memory” but nobody explains how it actually works, you’re in the right place.

What happens when there’s nothing underneath you?

Every program you’ve ever written ran on top of an operating system. When you call println!, the OS figures out where to send those bytes. When you allocate memory, the OS finds a free chunk and maps it into your address space. When your code runs alongside other programs without crashing into them, that’s the OS keeping everyone in their own lane.

Now imagine none of that exists. No OS. No standard library. No heap. No println!. Just you, a CPU, and some RAM. You want to print a single character? You need to know the exact memory address of the serial port hardware, write one byte to that address, and hope you configured the device correctly. If you didn’t, nothing happens. The screen stays blank. The CPU doesn’t complain. It just sits there.

That’s where we’re starting. We’re going to build, from nothing, the thing that makes all of those invisible layers work. By the end, you’ll understand exactly what happens between “power on” and “your code runs.” And honestly? It’s one of the most satisfying things you can do as a programmer.

TL;DR

This is a 5-part educational series (Part 0 through Part 4) where we build a minimal OS kernel in Rust from scratch. We focus on AArch64 QEMU virt, which means anyone with a laptop can run it. No special hardware needed, just QEMU.

Here’s what you’ll learn:

Boot sequences: How a CPU goes from power-on to running your first line of Rust
Serial output: The simplest possible way to see what your kernel is doing
Message-passing IPC: How tasks communicate without sharing memory
Cooperative and preemptive scheduling: From polite turn-taking to the OS forcibly yanking the CPU away
Timer interrupts: How hardware gives your OS a sense of time
Context switching: Saving and restoring every register so tasks don’t know they were paused
Virtual memory: Page tables, the MMU, and why every address your code uses is a lie

Warning: This is a learning project, not production code. The goal is understanding, not security or completeness.

Time commitment: About 2-4 hours per part, totaling 5 parts. Follow along at your own pace.

1. A quick primer: AArch64, QEMU, and the “virt” machine

You’ll see the phrase “AArch64 QEMU virt” throughout this series, so let’s unpack it before we go further.

AArch64 is ARM’s 64-bit instruction set architecture. It’s the CPU architecture inside your phone, your tablet, Apple Silicon Macs, most cloud servers (AWS Graviton, Ampere Altra), and the Raspberry Pi 3 and newer. When we say we’re writing an AArch64 kernel, we mean we’re writing code that speaks the language these CPUs understand: A64 instructions, 31 general-purpose 64-bit registers (x0 through x30), four exception levels (EL0 through EL3) for privilege separation, and a specific way of handling interrupts, memory, and device access.

If you’ve only worked with x86_64 (Intel/AMD), the concepts are the same, but the details differ. ARM uses a load/store architecture (you can’t perform arithmetic directly in memory; you have to load values into registers first), a different interrupt model (GIC instead of APIC), and a different boot flow (exception levels instead of real/protected/long mode transitions). None of this matters yet. We’ll explain every ARM-specific detail as we encounter it.

QEMU is a machine emulator. It simulates an entire computer in software: CPU, RAM, interrupt controller, timer, serial port, everything. When we run our kernel in QEMU, it doesn’t touch your real hardware. QEMU pretends to be an ARM computer, and our kernel runs on it. This is incredibly useful for OS development because you can restart instantly, you can’t brick anything, and the emulated hardware behaves predictably.

“virt” is a specific machine type provided by QEMU. Real ARM hardware comes in thousands of configurations: different board layouts, different peripherals, different memory maps. The virt machine is a clean, minimal design created by QEMU specifically for virtual machines and testing. It gives you:

A PL011 UART (serial port) at a known memory address for text output
A GICv2 interrupt controller for routing hardware interrupts to the CPU
An ARM Generic Timer for periodic tick interrupts
RAM starting at a fixed address (0x40000000)
No legacy baggage, no quirky firmware, no board-specific workarounds

This is why we chose it for the blog series. The virt machine strips away all the hardware complexity that has nothing to do with OS fundamentals. You don’t need to worry about GPU initialization, USB enumeration, or PCIe configuration. You get a CPU, memory, a timer, an interrupt controller, and a serial port. That’s exactly what you need to learn how an OS works, and nothing more.

When you run qemu-system-aarch64 -machine virt, you’re telling QEMU: “pretend to be this specific kind of ARM computer.” Our kernel is written to match that pretend computer’s memory map and peripherals.

2. What we’re building

rustOS is a minimal microkernel. Not a “toy” that prints hello world and stops, but not Linux either. It’s the kind of thing that actually boots, actually runs multiple tasks, actually translates virtual addresses to physical ones. The interesting middle ground where you learn how real operating systems work without drowning in 30 million lines of code.

The kernel boots on AArch64 QEMU virt (an emulated ARM machine that anyone can run). It implements message-passing IPC, the core idea behind microkernel design, in which tasks communicate by sending small, fixed-size messages through a router rather than poking each other’s memory directly. It runs a cooperative scheduler that polls tasks in round-robin, and then we upgrade to preemptive multitasking with timer interrupts so tasks can’t hog the CPU. And it manages virtual memory with 4-level page tables and a real MMU, the hardware that makes every modern OS possible.

Now, the codebase also supports x86_64 and the Raspberry Pi Zero 2 W. Those platforms share the same kernel logic and are available in the repository if you want to explore them. But the blog series focuses on AArch64 virt because it’s the most accessible. You don’t need a Raspberry Pi. You don’t need a Windows machine to deal with x86 bootloader quirks. You need QEMU, and QEMU runs on everything.

Before we look at the code layout, a quick Rust concept if you’re coming from another language: a crate is Rust’s unit of compilation, roughly equivalent to a library or package in other ecosystems. A crate has its own Cargo.toml (think package.json or pom.xml) that declares its name, dependencies, and build settings. A workspace groups multiple crates together so they can share dependencies and be built in one cargo build invocation. Our project is a workspace with several crates: kernel, hal, and one per platform. Each crate compiles independently and can depend on others. When you see kernel crate or hal crate below, think “self-contained module with a clear API boundary.”

Here’s how the code is organized:

graph TB
    subgraph "Platform-Agnostic Core"
        kernel[kernel<br/>IPC Router, Scheduler, Tasks]
        hal[hal<br/>Logger Trait, Arch Primitives]
    end

    subgraph "Platform Crate"
        virt[arch_aarch64_virt<br/>QEMU Boot, PL011 UART<br/>Timer, GIC, MMU]
    end

    virt --> hal
    virt --> kernel

    style kernel fill:#e1f5ff
    style hal fill:#e1f5ff
    style virt fill:#fff4e1

Figure 4: Platform-agnostic crate architecture

The kernel crate knows nothing about hardware. It talks to the world through a Logger trait defined in hal. The platform crate (arch_aarch64_virt) implements that trait using a PL011 UART driver and handles all ARM-specific setup: boot assembly, timer configuration, GIC interrupt routing, and MMU page tables. This separation is the same pattern you’ll find in real operating systems. Linux has arch/arm64/ for ARM-specific code and kernel/ for the portable parts. We’re doing the same thing, just at a much smaller scale.

3. Why this series?

Most OS tutorials either skip the hard parts (boot, interrupts, memory management), use simulators instead of real code, or don’t explain why things work the way they do. You get a code dump with a comment saying “configure the GIC” and zero explanation of what the GIC is, why it exists, or what happens if you get it wrong.

This series is different. We show complete implementations with no hand-waving. Every register write gets explained. Every design decision has a rationale. When we configure the MMU, we don’t just say “set TCR_EL1 to this magic number.” We break down every bit field, explain what it controls, and why we chose that value.

We also focus on understanding the features. A production OS needs thousands of things we don’t build: filesystems, networking, user mode, multicore support, security hardening. We skip all of that. Not because those things aren’t important, but because they all build on the foundation we’re teaching here. You can’t write a filesystem without understanding memory management. You can’t implement a network stack without interrupts. Get the foundation right, and the rest is (hard, but conceptually clear) engineering.

And we’re using Rust, which honestly makes bare-metal programming less terrifying than it used to be. The type system catches entire classes of bugs at compile time. You’ll still write plenty of unsafe code (this is an OS, after all), but the safe parts of Rust keep you honest about where the dangerous bits are.

Operating systems sit at the intersection of hardware (registers, interrupts, memory controllers), architecture (x86 vs ARM, privilege levels, MMUs), concurrency (multiple tasks, race conditions), and abstraction (clean APIs over messy hardware). There’s no debugger, no standard library, and mistakes cause silent crashes. This series shows you how to navigate that complexity, one piece at a time.

4. Why Rust? (and not C or assembly)

If you’re building an OS, you’re probably wondering: why Rust? Traditionally, kernels are written in C (Linux, xv6, Minix) or even raw assembly. Both are fine choices. But Rust brings something genuinely new to the table.

4.1 The problem with C

C gives you total control. No garbage collector, no runtime, direct memory access. That’s why it’s been the lingua franca of OS development for 50 years. But C also trusts you completely, and that trust is the problem. Buffer overflows, use-after-free, null pointer dereferences, data races: these aren’t theoretical. They’re the source of roughly 70% of security vulnerabilities in production systems (Microsoft and Google have both published studies confirming this).

In kernel code, these bugs are catastrophic. A buffer overflow in userspace causes your program to crash. A kernel-space buffer overflow corrupts the entire system. There’s no safety net below you.

4.2 The problem with assembly only

You could write an OS in pure assembly. Some people do, and there’s real value in understanding every instruction. But assembly doesn’t scale. An OS needs abstractions (traits, modules, type checking) to stay maintainable. Writing a mailbox router in assembly would work, but debugging it would be miserable. And you’d lose all the benefits of a compiler that can optimize, inline, and catch type errors for you.

4.3 What Rust gives you

Rust’s ownership system catches entire classes of bugs at compile time. You can’t use a value after freeing it. You can’t have two mutable references to the same data. You can’t forget to initialize memory. The borrow checker is annoying until you realize it’s catching bugs that would take hours to debug on bare metal, where there’s no debugger, no address sanitizer, and no core dump.

But Rust also gets out of your way when you need it to. The unsafe keyword lets you write raw pointer dereferences, inline assembly, and MMIO operations. You have to be explicit about it. The dangerous parts are clearly marked, and the safe parts are genuinely safe. This is a huge improvement over C, where everything is implicitly unsafe.

Rust’s no_std ecosystem is also mature. Crates like bootloader, uart_16550, and volatile provide building blocks for OS development. The core library gives you basic types, traits, and iterators without needing an allocator. And #[repr(C)] lets you control struct layout for hardware register interactions.

Concretely, here are bugs Rust prevented during this project:

The compiler refused to let us access the IPC router from an interrupt handler without explicit synchronization (UnsafeCell + Sync impl)
A missing volatile read would have been optimized away in C, silently breaking UART output. In Rust, we had to use read_volatile, making the intent clear explicitly
Type mismatches between page table entry formats were caught at compile time, not at boot time, with a mysterious hang

Is Rust perfect for OS development? No. The borrow checker sometimes fights you in ways that feel unnecessary for single-threaded kernel code. static mut is awkward. Some patterns that are natural in C require contortions in Rust. But on balance, we think the trade-off is overwhelmingly positive. You spend a little more time satisfying the compiler, and a lot less time debugging silent memory corruption.

5. Getting started

You’ll need Rust (nightly), QEMU, and basic Rust knowledge. If you know what ownership and traits are, you’re good. If you’ve written a function that takes &self and returned a Result, you’ll be fine. You don’t need to know assembly or have prior OS background. That’s the whole point of the series.

Tool	Purpose	Installation
Rust (nightly)	Bare-metal compilation	`rustup default nightly`
QEMU	Virtual machine for running our kernel	`apt install qemu-system-aarch64` or `brew install qemu`
Git	Cloning the repository	You probably already have this

5.1 Install Rust (nightly)

# Install Rust via rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Follow prompts, select default installation
# Then reload your shell environment:
source $HOME/.cargo/env

# Switch to nightly (required for bare-metal features)
rustup default nightly

# Add the rust-src component (needed for building core for bare-metal targets)
rustup component add rust-src --toolchain nightly

# Add the AArch64 bare-metal target
rustup target add aarch64-unknown-none

Verify it works:

$ rustc --version
rustc 1.86.0-nightly (5765cbc51 2025-01-28)

$ rustup target list --installed | grep aarch64
aarch64-unknown-none

5.2 Install QEMU

# Ubuntu/Debian
sudo apt install qemu-system-aarch64

# macOS
brew install qemu

# Verify
$ qemu-system-aarch64 --version
QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.4)

5.3 Clone and build

$ git clone https://github.com/bahree/rust-microkernel.git
$ cd rust-microkernel
$ ./scripts/build-aarch64-virt.sh

[virt] building aarch64 QEMU virt ELF (default features)...
    Finished `release` profile [optimized] target(s) in 0.28s
[virt] wrote dist/virt/os-aarch64-virt.elf

5.4 Run it

$ ./scripts/run-aarch64-virt.sh

rustOS: aarch64 QEMU virt boot OK
rustOS: memory management demo (frames + page tables)
mm: demo start
mm: kernel_end=0x000000004009A010
mm: free_start=0x000000004009B000
mm: ram_end=0x0000000050000000
mm: frame0=0x000000004009B000
mm: frame1=0x000000004009C000
mm: read0=0x00000000AABBCCDD
mm: read1=0x0000000011223344
mm: ttbr0=0x0000000040085000
mm: test_va=0x0000000080000000
mm: enabling MMU (caches off)...
mm: test_va_read=0x00000000DEADBEEF
mm: demo done (MMU is ON)

Press Ctrl+A then X to exit QEMU.

Figure 1: Memory demo output showing frame allocation, page table construction, MMU enablement, and virtual address translation.

If you see that output, congratulations. You just ran an operating system kernel that booted on bare metal, allocated physical memory frames, built 4-level page tables, enabled the MMU, and successfully translated a virtual address from a few shell commands.

5.5 Try the other demos

The codebase has multiple demos you can build with feature flags:

# IPC + cooperative scheduling (Part 2)
$ ./scripts/build-aarch64-virt.sh demo-ipc && ./scripts/run-aarch64-virt.sh

rustOS: aarch64 QEMU virt boot OK
rustOS: IPC + cooperative scheduling demo
rustOS: kernel online
rustOS: microkernel step 1 (IPC + cooperative scheduling)
sched: starting
task/ping: poll
task/ping: sent ping
task/pong: got ping

# Preemptive multitasking (Part 3)
$ ./scripts/build-aarch64-virt.sh demo-preempt && ./scripts/run-aarch64-virt.sh

rustOS: aarch64 QEMU virt boot OK
rustOS: preemptive multitasking demo
A
B
A
B
A
B
...

Figure 2: IPC demo showing ping and pong tasks exchanging messages through the mailbox router.

Figure 3: Preempt demo showing two threads alternating without ever yielding, proving the OS is in control.

If all three demos work, your environment is fully set up and ready for the series.

5.6 Docker (no local setup)

If you just want to try it without installing anything:

docker pull amitbahree/rust-microkernel:latest
docker run -it amitbahree/rust-microkernel:latest
# Inside the container:
./scripts/build-aarch64-virt.sh && ./scripts/run-aarch64-virt.sh

x86_64 and Raspberry Pi builds also exist in the repository. Run ./scripts/build-x86.sh or ./scripts/build-rpi.sh if you want to try them. They share the same kernel logic but use different boot code and hardware drivers.

6. Series roadmap

Here’s what each part covers and why it comes in that order. Each part builds on the previous one. You can’t preempt tasks without interrupts. You can’t isolate processes without virtual memory. The ordering isn’t arbitrary.

Part 0: Why build an OS from scratch? (you are here) — The overview. What we’re building, why, and how the pieces fit together.

Part 1: Foundations — How a CPU goes from power-on to running your first line of Rust. AArch64 boot assembly that drops from EL2 to EL1, stack setup, BSS zeroing, PL011 UART driver, and the platform abstraction (HAL) that keeps the kernel hardware-agnostic.

Part 2: Communication — Message-passing IPC with a mailbox router. Tasks implement a poll() method, the scheduler calls them round-robin, and we build PingTask/PongTask to show communication in action. Plus why cooperative scheduling has limits (one misbehaving task freezes everything).

Part 3: Concurrency — The big one. ARM Generic Timer, GICv2 interrupt controller, periodic timer interrupts, and a preemptive scheduler that saves all 31 registers plus SP, ELR, and SPSR. Two threads print “A” and “B” without ever yielding, proving the OS is in control.

Part 4: Memory and beyond — Virtual memory from the ground up. Bump allocator for physical frames, 4-level page tables (L0/L1/L2/L3), MMU configuration with MAIR, TCR, and TTBR0, and address translation verified by writing through a virtual address and reading back through the physical one. Ends with reflections and where to go next.

7. The big picture: what makes an operating system?

Before we start building, let’s understand what we’re aiming for. An operating system has many layers, and we’re focusing on the foundational ones that everything else depends on.

7.1 Complete OS architecture

graph TB
    subgraph "User Space (Ring 3 / EL0)"
        Apps[Applications<br/>ls, cat, browser, games]
        Libs[Standard Libraries<br/>libc, libstd]
        Shell[Shell<br/>bash, zsh]
    end

    subgraph "Kernel Space (Ring 0 / EL1)"
        subgraph "System Call Interface"
            Syscalls[System Calls<br/>open, read, write, fork]
        end

        subgraph "Process Management"
            Sched[Scheduler<br/>Round-robin, CFS, priorities]
            IPC[IPC<br/>Message passing, pipes, signals]
            Proc[Process Table<br/>PIDs, state, resources]
        end

        subgraph "Memory Management"
            VMM[Virtual Memory<br/>Page tables, TLB, swapping]
            Heap[Heap Allocator<br/>malloc/free, buddy system]
            MMU[MMU Control<br/>TTBR, page faults]
        end

        subgraph "File Systems"
            VFS[VFS Layer<br/>Common file API]
            FS[File Systems<br/>ext4, FAT, NTFS]
            Cache[Page Cache<br/>Read-ahead, write-back]
        end

        subgraph "Device Drivers"
            Block[Block Devices<br/>Disk, SSD, NVMe]
            Net[Network<br/>Ethernet, WiFi, TCP/IP]
            Char[Character Devices<br/>UART, keyboard, mouse]
            GPU[Graphics<br/>Framebuffer, GPU drivers]
        end

        subgraph "Core Kernel - WHAT WE BUILD"
            Boot[Boot<br/>Platform init, entry]
            Int[Interrupts<br/>Timer, exceptions, GIC/PIC]
            Ctx[Context Switch<br/>Save/restore registers]
            Mem[Physical Memory<br/>Frame allocator]
        end
    end

    subgraph "Hardware"
        CPU[CPU<br/>Cores, caches, MMU]
        RAM[RAM<br/>Physical memory]
        Disk[Storage<br/>HDD, SSD]
        Devices[Peripherals<br/>UART, Timer, GPIO]
    end

    Apps --> Syscalls
    Libs --> Syscalls
    Shell --> Syscalls

    Syscalls --> Sched
    Syscalls --> VMM
    Syscalls --> VFS

    Sched --> Ctx
    Sched --> Proc
    Sched --> IPC

    VMM --> MMU
    VMM --> Heap
    VMM --> Mem

    VFS --> FS
    VFS --> Cache
    FS --> Block

    Block --> Devices
    Net --> Devices
    Char --> Devices
    GPU --> Devices

    Int --> CPU
    Ctx --> CPU
    MMU --> CPU
    Mem --> RAM
    Boot --> CPU

    style Boot fill:#e8f5e8
    style Int fill:#e8f5e8
    style Ctx fill:#e8f5e8
    style Mem fill:#e8f5e8
    style IPC fill:#fff4e1
    style Sched fill:#fff4e1
    style VMM fill:#fff4e1

Figure 5: Complete OS architecture showing what we build

Look at that diagram. It’s a lot. But here’s the thing: we’re only building the green and yellow boxes at the bottom. Boot, interrupts, context switching, physical memory, IPC, scheduling, and virtual memory. That’s the core kernel layer, the absolute foundation that every OS needs. These aren’t optional features you can skip. They’re the bedrock on which everything else is built.

7.2 What we build (the foundation)

Boot sequences (Part 1) are how a computer goes from pressing the power button to running your kernel’s first line of code. It’s the handoff between hardware/firmware and your operating system. On ARM, this means handling exception levels (EL2 hypervisor to EL1 kernel), setting up a stack, zeroing the BSS, and enabling floating-point instructions before the LLVM-generated code can run. Get any step wrong, and the CPU hangs silently.

IPC and scheduling (Parts 2 and 3) are how multiple tasks run on a single CPU core. The scheduler decides who runs when. IPC lets tasks communicate without sharing memory. We start with cooperative scheduling (tasks voluntarily yield) and upgrade to preemptive scheduling (the OS is in control). This requires context switching: saving all 31 ARM registers, the stack pointer, the return address, and the CPU flags, then loading another task’s saved state.

Interrupts and exceptions (Part 3) are hardware signals that tell the CPU, “stop what you’re doing and handle this NOW.” The timer fires, interrupt! Without interrupts, your OS would have to constantly poll hardware (“Are you done yet? How about now?”), wasting billions of CPU cycles. And you’d have no way to switch between tasks forcibly. Interrupts are the foundation for everything asynchronous in a computer.

Memory management (Part 4) is the illusion that each program has its own private address space. The MMU (Memory Management Unit) translates every address using page tables, a 4-level tree structure that maps virtual addresses to physical ones. We build those tables, configure the MMU, and enable translation. After that, every memory access your code makes goes through hardware that we configured.

7.3 What we don’t build

We skip filesystems, networking, user mode, multicore, and security hardening. Not because they’re unimportant, but because they all build on the foundation we’re teaching. You can’t write a filesystem without memory management. You can’t implement a network stack without interrupts. You can’t add user mode without understanding privilege levels and page tables. Part 4 points you to resources for going deeper into each of these areas.

8. Repository structure

The full source code is at github.com/bahree/rust-microkernel . Here’s how it’s organized, focused on the AArch64 virt platform that the blog series covers:

rust-microkernel/
├── crates/
│   ├── kernel/                 # Platform-agnostic kernel logic
│   │   ├── lib.rs              # Entry point (kmain)
│   │   ├── ipc.rs              # Message-passing router
│   │   └── sched.rs            # Cooperative scheduler
│   ├── hal/                    # Hardware abstraction layer
│   │   ├── log.rs              # Logger trait
│   │   └── arch.rs             # Architecture primitives (halt)
│   └── arch_aarch64_virt/      # AArch64 QEMU virt platform
│       ├── boot.S              # Assembly: EL2→EL1 drop, vectors
│       ├── main.rs             # Rust entry point
│       ├── uart.rs             # PL011 UART driver
│       ├── timer.rs            # Generic Timer + GIC
│       ├── preempt.rs          # Context switching
│       └── mem.rs              # Frame allocator + MMU
├── scripts/
│   ├── build-aarch64-virt.sh   # Build AArch64 virt kernel
│   └── run-aarch64-virt.sh     # Run in QEMU
├── docs/
│   └── blog/                   # This blog series
└── dist/                       # Build artifacts

The kernel crate knows nothing about hardware. It accepts a Logger trait object and calls .log() to print messages. The hal crate defines that trait. The arch_aarch64_virt crate implements it with a real UART driver and handles everything ARM-specific: boot assembly, timer setup, interrupt routing, page tables.

The x86_64 and Raspberry Pi platform crates (arch_x86_64, arch_aarch64_rpi) also exist in crates/ and follow the same pattern. They’re not covered in the blog series, but use the same kernel and hal crates.

9. References and acknowledgments

This project stands on the shoulders of giants. Philipp Oppermann’s “Writing an OS in Rust” showed that Rust OS development is not only possible but genuinely enjoyable. MIT’s xv6 demonstrated that you can teach OS internals with a clean, readable codebase. The OSDev community has been documenting hardware quirks and boot sequences for decades, and their wiki is invaluable.

For operating systems theory, Operating Systems: Three Easy Pieces (free online) is the best introduction we’ve found. For ARM architecture specifics, the ARM Architecture Reference Manual is the definitive reference (and it’s freely available). For bare-metal Rust patterns, the Embedded Rust Book covers no_std development in depth.

We also learned from real microkernel projects: seL4 (formally verified, beautifully designed), Minix 3 (Tanenbaum’s vision of what microkernels should be), and the L4 family (proving that microkernel IPC can be fast).

5-Part Series:

Part 0 (this): Why build an OS from scratch?
Part 1: Foundations (coming soon)
Part 2: Communication (coming soon)
Part 3: Concurrency (coming soon)
Part 4: Memory and beyond (coming soon)

GitHub Repository: bahree/rust-microkernel — full source code and build scripts

Docker Image: amitbahree/rust-microkernel — prebuilt dev environment with Rust, QEMU, and source code

Building LLMs from Scratch - Part 4: Evaluation & Deployment

Fri, 02 Jan 2026 00:00:00 +0000

TL;DR

In this final part of our 4-part series on building language models from scratch, we explore the evaluation, testing, and deployment pipeline that transforms our trained historical language models into working systems. Part 1 showed you how to use the published models, Part 2 covered data collection and custom tokenization, and Part 3 detailed the model architecture and training infrastructure. Here, we complete the journey with evaluation frameworks, testing infrastructure, and deployment to Hugging Face Hub.

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need much larger datasets, more sophisticated infrastructure, and additional considerations not covered here.

As outlined in Part 1 , both the SLM (117M parameters) and the Regular Model (354M parameters) use the same training code and infrastructure with different configurations defined in config.py. The evaluation and deployment infrastructure is also identical - only the model architecture parameters differ.

Both PyTorch checkpoint inference and Hugging Face model inference are fully working and available. Both the SLM and the Regular model are published on Hugging Face Hub . Local PyTorch checkpoints can be used directly for inference with the script inference_pytorch.py.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete evaluation and deployment infrastructure (05_evaluation/, 06_inference/, 10_scripts/) plus guides (08_documentation/EVALUATION_GUIDE.md, 08_documentation/HUGGINGFACE_PUBLISHING.md, 08_documentation/DEPLOYMENT_GUIDE.md)

🟥 Series Posts: Part 1 - Using the Published Historical Models | Part 2 - Data Collection & Custom Tokenizer | Part 3 - Training Architecture & GPU Optimization | Part 4 (this post)

🟧 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📗 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts

1. The Evaluation Challenge: Measuring What Matters for Historical Language Models

Now that we have trained models from Part 3 , we face a critical question: How do we know if our models actually work? This isn’t just about checking if the code runs - it’s about validating that the models can generate historically accurate, linguistically appropriate text that captures the essence of 1500-1850 London English.

The challenge with evaluating historical language models goes far beyond standard LLM metrics. Standard evaluation approaches like Perplexity and BLEU scores (we explain these and other metrics in Section 2.1 ) tell us whether the model generates fluent text. Still, they don’t answer the questions that matter for historical applications: Does the model avoid anachronisms? Can it distinguish between Tudor and Victorian language patterns? Does it understand London geography and historical context?

Consider a simple example: if we prompt the model with “In the year 1600, I traveled to London by railway”, a standard language model might generate this without flagging the obvious problem - railways didn’t exist in 1600. The evaluation framework needs to catch these temporal inconsistencies, period-inappropriate language, and historical inaccuracies that standard metrics miss.

This evaluation challenge requires building a specialized assessment pipeline that understands historical context, temporal boundaries, and period-specific linguistic patterns. We need metrics that can distinguish between a model that generates fluent modern English and one that produces authentic historical text - two very different capabilities.

1.1 High-Level Evaluation Strategy

Our evaluation framework provides two complementary approaches that work with both PyTorch checkpoints and Hugging Face models, as illustrated in Figure 1 below.

graph TD
    A[🤖 Trained Models<br/>SLM 117M / Regular 354M] --> B{Evaluation Type}
    
    B -->|Quick| C[⚡ Quick Evaluation<br/>Historical accuracy, language quality, coherence]
    B -->|Comprehensive| D[🔬 Comprehensive Evaluation<br/>Benchmarks, G-Eval, groundedness]
    
    C --> E[📊 Evaluation Results<br/>Historical accuracy scores, metrics]
    D --> E
    
    E --> F{Quality OK?}
    F -->|Yes| G[🚀 Deployment Options]
    F -->|No| H[🔄 Retrain/Adjust]
    H --> A
    
    G --> I[📦 PyTorch Checkpoints<br/>Direct inference]
    G --> J[🤗 Hugging Face Hub<br/>Published models]
    G --> K[💻 Local Deployment<br/>API, CLI, notebooks]
    
    I --> L[✅ Working Models<br/>Ready for use]
    J --> L
    K --> L
    
    style A fill:#e1f5fe
    style E fill:#f3e5f5
    style L fill:#e8f5e8
    style H fill:#fff3e0

Figure 1: Complete Evaluation and Deployment Pipeline

Quick Evaluation (quick_eval.py): Rapid validation testing historical accuracy on key events (e.g., 1665 plague, 1666 fire, etc.), language quality metrics (vocabulary diversity, historical pattern detection, readability), and coherence (ROUGE scores). Runs in minutes without external APIs.

Comprehensive Evaluation (comprehensive_evaluator.py): Extends quick evaluation with benchmark datasets (small MMLU and HellaSWAG subsets), groundedness/fluency metrics, and optional LLM-as-a-judge scoring via G-Eval (using an external GPT model). Produces detailed reports with generation samples.

Both evaluators test across historical periods (such as Tudor, Stuart, and Georgian), language patterns (archaic pronouns and verb forms), and London-specific knowledge (geography and landmarks). The framework goes beyond standard LM metrics to assess period-appropriate language, temporal consistency, and historical accuracy.

2. Model Evaluation Framework

Now that we’ve outlined the evaluation challenge, let’s dive into the implementation. Our evaluation framework provides two complementary approaches that work with both PyTorch checkpoints and Hugging Face models. The framework is designed to be practical for a learning project while still providing meaningful insights into model performance.

2.1 Historical, Linguistic, and Category-Specific Evaluation

To make the evaluation concrete, we look at the model from three complementary aspects that together capture how well it understands the period, writes fluent text, and handles the different slices of the corpus. This multi-dimensional approach ensures we catch various types of failures - a model might generate grammatically perfect text but fail historically, or vice versa.

Historical assessments: Quick evaluation uses targeted prompts around key events (e.g., 1665 plague, 1666 fire, Old Bailey trials) and checks for expected keywords and phrases. Comprehensive evaluation adds temporal consistency checks (forbidden/required terms per period), date-range sanity checks, and historical benchmarks (custom historical questions and the MMLU subset).
Linguistic assessments: We measure surface quality (chars/words/sentences per sample, words per sentence), vocabulary diversity (unique/total tokens), readability (Flesch-style scores), and presence of historical patterns (archaic verb forms like hath, doth, pronouns like thou, thee, conjunctions and interjections). This shows whether the model writes in a historically flavored yet readable style.
Category-specific benchmarks: Evaluations are grouped by period (Tudor, Stuart, Georgian), by linguistic phenomena (archaic forms, dialogue patterns), and by London knowledge (Thames, Westminster, Old Bailey, etc.). The comprehensive evaluator further probes general reasoning using HellaSWAG and MMLU subsets to assess the model’s performance across broader benchmarks.

Industry-Standard Evaluation Metrics and Benchmarks

Our evaluation framework uses several standard metrics and benchmarks from LLM research. Here’s what each one measures and why we include it:

Perplexity: How surprised the model is by the reference text; lower is better because it means the model assigns higher probability to what actually happened in the corpus.

BLEU / ROUGE: N-gram overlap between generated and reference text, giving a rough sense of literal similarity and how closely the model “sticks” to the reference phrasing. We use ROUGE-L (longest common subsequence) to evaluate coherence and narrative flow.

MMLU (Massive Multitask Language Understanding): A large multiple-choice exam covering many academic subjects. Here, we use a tiny subset as a sanity check for general knowledge and reasoning, not as a primary goal.

HellaSWAG: A commonsense inference benchmark where the model must pick a plausible continuation for a short story-like context. We use it to see whether the model’s basic reasoning looks sensible.

G-Eval: An LLM-as-a-judge pattern where a stronger reference model (for example, GPT) scores generated text along dimensions like coherence or groundedness. In this project, it is optional and requires an external API key.

Groundedness: Asks: does the model stick to the provided context / known facts, or hallucinate? Our implementation approximates this by comparing generations against reference answers and historical constraints.

For a deeper treatment of evaluation benchmarks (including MMLU, HellaSWAG, and LLM-as-a-judge methods like G-Eval), see Chapter 12 - Evaluating and Monitoring Generative Systems in the book 📘 Generative AI in Action .

2.2 Automated Evaluation Pipeline

The run_comprehensive_evaluation function in 05_evaluation/comprehensive_evaluator.py orchestrates the entire evaluation process. Listing 1 shows how it works: We iterate over test sets, generate text with the model, compute all the metrics defined above, and aggregate the results into a results dictionary for analysis.

def run_comprehensive_evaluation(model, tokenizer, test_data, device='cuda'):
    """Run comprehensive evaluation on historical language model"""
    
    # Initialize evaluation metrics
    metrics = {
        'perplexity': [],
        'bleu_scores': [],
        'rouge_scores': [],
        'historical_accuracy': [],
        'linguistic_quality': [],
        'coherence_scores': [],
        'temporal_consistency': []
    }
    
    # Evaluate on different text types
    for text_type, samples in test_data.items():
        logger.info(f"Evaluating on {text_type} samples...")
        
        for sample in samples:
            # Generate text
            generated = generate_text(model, tokenizer, sample['prompt'], device)
            
            # Calculate metrics
            perplexity = calculate_perplexity(model, tokenizer, sample['text'], device)
            bleu = calculate_bleu(generated, sample['reference'])
            rouge = calculate_rouge(generated, sample['reference'])
            hist_acc = assess_historical_accuracy(generated, sample['context'])
            ling_qual = assess_linguistic_quality(generated)
            coherence = assess_coherence(generated)
            temp_cons = assess_temporal_consistency(generated, sample['time_period'])
            
            # Store metrics
            metrics['perplexity'].append(perplexity)
            metrics['bleu_scores'].append(bleu)
            metrics['rouge_scores'].append(rouge)
            metrics['historical_accuracy'].append(hist_acc)
            metrics['linguistic_quality'].append(ling_qual)
            metrics['coherence_scores'].append(coherence)
            metrics['temporal_consistency'].append(temp_cons)
    
    # Calculate aggregate metrics
    results = {}
    for metric_name, values in metrics.items():
        results[metric_name] = {
            'mean': np.mean(values),
            'std': np.std(values),
            'min': np.min(values),
            'max': np.max(values),
            'median': np.median(values)
        }
    
    return results

Listing 1: Comprehensive Evaluation Pipeline

The pipeline computes all the metrics we outlined above (standard LM metrics such as perplexity and BLEU/ROUGE, plus our historically specific assessments of accuracy, linguistic quality, and coherence). Each metric provides a different lens through which to view model performance: perplexity measures how well the model predicts the training distribution, BLEU/ROUGE measures literal similarity to the reference text, and the custom metrics assess historical authenticity and linguistic appropriateness.

Why This Multi-Metric Approach Matters?

Standard language model evaluation often focuses on perplexity and n-gram overlap metrics, which measure general language quality but miss domain-specific requirements. For historical language models, we need to know not just whether the text is fluent, but whether it’s historically accurate, temporally consistent, and linguistically appropriate for the target period. This multi-metric approach ensures we catch different types of failures - a model might generate grammatically perfect text but fail historically, or produce historically accurate content with poor linguistic quality.

The aggregation step (computing mean, std, min, max, median) provides a comprehensive view of model performance across different test cases. This statistical summary helps identify whether the model performs consistently or has high variance, whether certain types of prompts cause failures, and how the model compares across different historical periods and linguistic phenomena.

2.3 Historical Accuracy Assessment

Standard LLM evaluation metrics (perplexity, BLEU, ROUGE) measure general language quality, but they don’t tell us whether the model generates historically accurate text for London between 1500-1850. To address this, we built customized evaluation tools that check period-appropriate language, temporal consistency, London-specific geography and landmarks, and historical fact accuracy. These tools are implemented in 05_evaluation/comprehensive_evaluator.py as shown in Listing 2:

def assess_historical_accuracy(generated_text, historical_context):
    """Assess the historical accuracy of generated text"""
    
    accuracy_score = 0.0
    total_checks = 0
    
    # Check temporal consistency
    temporal_score = check_temporal_consistency(generated_text, historical_context['time_period'])
    accuracy_score += temporal_score
    total_checks += 1
    
    # Check historical facts
    fact_score = check_historical_facts(generated_text, historical_context['facts'])
    accuracy_score += fact_score
    total_checks += 1
    
    # Check period-appropriate language
    language_score = check_period_language(generated_text, historical_context['time_period'])
    accuracy_score += language_score
    total_checks += 1
    
    # Check geographical accuracy
    geo_score = check_geographical_accuracy(generated_text, historical_context['location'])
    accuracy_score += geo_score
    total_checks += 1
    
    # Check social context accuracy
    social_score = check_social_context(generated_text, historical_context['social_class'])
    accuracy_score += social_score
    total_checks += 1
    
    return accuracy_score / total_checks

def check_temporal_consistency(text, time_period):
    """Check if text maintains temporal consistency with the specified period"""
    
    # Define period-specific constraints
    period_constraints = {
        '1500-1600': {
            'forbidden_terms': ['electricity', 'steam engine', 'railway'],
            'required_terms': ['ye', 'hath', 'doth'],
            'date_range': (1500, 1600)
        },
        '1600-1700': {
            'forbidden_terms': ['railway', 'telegraph'],
            'required_terms': ['hath', 'doth', 'verily'],
            'date_range': (1600, 1700)
        },
        '1700-1800': {
            'forbidden_terms': ['telegraph', 'telephone'],
            'required_terms': ['hath', 'doth', 'indeed'],
            'date_range': (1700, 1800)
        },
        '1800-1850': {
            'forbidden_terms': ['telephone', 'automobile'],
            'required_terms': ['indeed', 'verily', 'pray'],
            'date_range': (1800, 1850)
        }
    }
    
    if time_period not in period_constraints:
        return 0.5  # Neutral score for unknown periods
    
    constraints = period_constraints[time_period]
    score = 1.0
    
    # Check for forbidden terms (anachronisms)
    for term in constraints['forbidden_terms']:
        if term.lower() in text.lower():
            score -= 0.2
    
    # Check for required period-appropriate terms
    period_terms_found = 0
    for term in constraints['required_terms']:
        if term.lower() in text.lower():
            period_terms_found += 1
    
    if constraints['required_terms']:
        score += 0.3 * (period_terms_found / len(constraints['required_terms']))
    
    # Check date references
    date_score = check_date_references(text, constraints['date_range'])
    score += 0.2 * date_score
    
    return max(0.0, min(1.0, score))

Listing 2: Historical Accuracy Assessment

The forbidden terms (like “electricity” for 1500-1600, “railway” for 1600-1700) are anachronisms - technologies or concepts that didn’t exist in those periods. We selected them based on historical timelines: electricity wasn’t harnessed until the late 1700s, railways didn’t appear until the early 1800s, and telegraphs came later. Similarly, the required terms (such as “hath”, “doth”, and “verily”) are archaic language patterns we observed frequently in the training corpus for each period.

We analyzed the corpus to identify which linguistic markers were most characteristic of each era, then selected a small set that would catch obvious anachronisms without being overly restrictive. This is a practical heuristic rather than an exhaustive historical grammar - we focus on high-impact anachronisms and common period markers that are easy to detect automatically.

How the scoring works

The check_temporal_consistency() function starts with a score of 1.0 and applies penalties and bonuses: each forbidden term found subtracts 0.2 (so finding “railway” in 1600-1700 text drops the score), while finding required period-appropriate terms adds up to 0.3 based on how many are present. Date references within the period add up to 0.2. The final score ranges from 0.0 to 1.0.

The overall assess_historical_accuracy() function then averages the five component scores (temporal consistency, historical facts, period-appropriate language, geographical accuracy, and social context) to produce a single score between 0 and 1, with higher values indicating better historical accuracy. In practice (and yes, we are generalizing), scores above 0.7 indicate good historical consistency, while scores below 0.5 suggest significant anachronisms or factual errors.

2.4 Linguistic Quality Evaluation

While historical accuracy checks whether the model gets facts and period-appropriate terms right, linguistic quality measures how well the model writes - grammar, coherence, vocabulary diversity, sentence structure, and the presence of historical language patterns.

Standard metrics like BLEU and ROUGE don’t capture whether the text reads naturally or uses appropriate archaic forms. We built customized tools that assess these dimensions, implemented in 05_evaluation/comprehensive_evaluator.py as shown in Listing 3:

To make this easier to read, it helps to view the code as a scoring scaffold rather than a complete NLP system. Each check_* function is expected to return a normalized score in the range [0, 1] (higher is better), and assess_linguistic_quality() simply averages those components so you can track one headline number over time.

This mirrors patterns from earlier in the series: in Part 2 we used lightweight, automatable checks to validate data quality, and in Part 3 we relied on simple, repeatable metrics to judge training health. Here, we do the same for generation quality: start with cheap checks that run everywhere, then iterate toward richer evaluators as needed.

Also note that the exact weights (0.3/0.2, etc.) are tunable. The main benefit is splitting “linguistic quality” into components you can inspect individually, so when output is bad, you can tell why (grammar-ish structure vs coherence vs vocabulary vs historically flavored patterns).

def assess_linguistic_quality(generated_text):
    """Assess the linguistic quality of generated historical text"""
    
    quality_score = 0.0
    total_checks = 0
    
    # Check grammatical correctness
    grammar_score = check_grammatical_correctness(generated_text)
    quality_score += grammar_score
    total_checks += 1
    
    # Check coherence and flow
    coherence_score = check_text_coherence(generated_text)
    quality_score += coherence_score
    total_checks += 1
    
    # Check vocabulary appropriateness
    vocab_score = check_vocabulary_appropriateness(generated_text)
    quality_score += vocab_score
    total_checks += 1
    
    # Check sentence structure variety
    structure_score = check_sentence_structure_variety(generated_text)
    quality_score += structure_score
    total_checks += 1
    
    # Check historical language patterns
    pattern_score = check_historical_language_patterns(generated_text)
    quality_score += pattern_score
    total_checks += 1
    
    return quality_score / total_checks

def check_grammatical_correctness(text):
    """Check grammatical correctness of generated text"""
    
    # Parse text into sentences
    sentences = nltk.sent_tokenize(text)
    
    if not sentences:
        return 0.0
    
    correct_sentences = 0
    
    for sentence in sentences:
        # Check for basic grammatical patterns
        if check_sentence_grammar(sentence):
            correct_sentences += 1
    
    return correct_sentences / len(sentences)

def check_historical_language_patterns(text):
    """Check if text follows appropriate historical language patterns"""
    
    score = 0.0
    total_patterns = 0
    
    # Check for appropriate use of historical verb forms
    historical_verbs = ['hath', 'doth', 'dost', 'art', 'wilt', 'shalt']
    verb_score = 0
    for verb in historical_verbs:
        if verb in text.lower():
            verb_score += 1
    if historical_verbs:
        score += 0.3 * (verb_score / len(historical_verbs))
    total_patterns += 1
    
    # Check for appropriate use of historical pronouns
    historical_pronouns = ['thou', 'thee', 'thy', 'thine', 'ye']
    pronoun_score = 0
    for pronoun in historical_pronouns:
        if pronoun in text.lower():
            pronoun_score += 1
    if historical_pronouns:
        score += 0.3 * (pronoun_score / len(historical_pronouns))
    total_patterns += 1
    
    # Check for appropriate use of historical conjunctions
    historical_conjunctions = ['whilst', 'betwixt', 'amongst', 'ere', 'anon']
    conj_score = 0
    for conj in historical_conjunctions:
        if conj in text.lower():
            conj_score += 1
    if historical_conjunctions:
        score += 0.2 * (conj_score / len(historical_conjunctions))
    total_patterns += 1
    
    # Check for appropriate use of historical interjections
    historical_interjections = ['verily', 'indeed', 'forsooth', 'prithee', 'marry']
    interj_score = 0
    for interj in historical_interjections:
        if interj in text.lower():
            interj_score += 1
    if historical_interjections:
        score += 0.2 * (interj_score / len(historical_interjections))
    total_patterns += 1
    
    return score / total_patterns if total_patterns > 0 else 0.0

Listing 3: Linguistic Quality Evaluation

About NLTK: We use NLTK (Natural Language Toolkit), a standard Python library for natural language processing, to handle text tokenization. If you followed Part 1 ’s setup instructions, NLTK was already installed as part of the data processing dependencies. In check_grammatical_correctness(), we use nltk.sent_tokenize() to split text into sentences so we can evaluate grammar sentence-by-sentence. NLTK also provides word tokenization (word_tokenize) and BLEU score calculation (sentence_bleu), which are used elsewhere in the evaluation pipeline.

We chose NLTK because it’s well-established, handles edge cases (like abbreviations and historical punctuation), and provides reliable sentence boundaries even with archaic English patterns. The same qualities made it useful during data collection and cleaning (covered in Part 2 ).

The historical language patterns we check (verbs like hath, doth, pronouns like thou, thee, conjunctions like whilst, betwixt, and interjections like verily, forsooth) are the same archaic forms we identified during corpus analysis for temporal consistency. The difference here is that we’re measuring their presence as a positive signal of historical authenticity, rather than using them as required/forbidden constraints. Each pattern category (verbs, pronouns, conjunctions, interjections) contributes proportionally to the score based on how many patterns from that category appear in the text.

How the scoring works

The assess_linguistic_quality() function averages five component scores (grammar, coherence, vocabulary appropriateness, sentence structure variety, and historical language patterns) to produce a single score between 0 and 1. Each component is evaluated independently and returns a score in the range [0, 1].

For example, check_grammatical_correctness() counts the proportion of grammatically correct sentences, while check_historical_language_patterns() weights the presence of archaic verb forms (30%), pronouns (30%), conjunctions (20%), and interjections (20%) to produce a pattern score. The final linguistic quality score is the simple average of all five components. In practice, scores above 0.75 indicate strong linguistic quality with good grammar and historical flavor, while scores below 0.6 suggest the model struggles with either basic grammar or historical language patterns.

2.5 Running Evaluations

You can run the evaluators directly from the command line. The framework defaults to CPU for safety (so you can evaluate during training without GPU conflicts), but you can use --device gpu when the GPU is free for faster evaluation.

Quick example:

# Quick evaluation (runs in minutes, no external APIs)
python 05_evaluation/run_evaluation.py --mode quick --device cpu

# Comprehensive evaluation (includes benchmarks, optional G-Eval)
python 05_evaluation/run_evaluation.py --mode comprehensive --device cpu

The unified launcher (run_evaluation.py) supports multiple modes: setup (install dependencies), quick (fast validation), comprehensive (full suite with benchmarks), dataset (generate test cases), and all (complete evaluation). You can also call quick_eval.py or comprehensive_evaluator.py directly if you need more control.

Practical Evaluation Workflow:

Our typical evaluation workflow follows this pattern:

After Training: Run a quick evaluation to get immediate feedback on model performance
Before Publishing: Run a comprehensive evaluation to ensure the model meets quality standards
During Development: Use interactive testing to explore model behavior on specific prompts
For Research: Generate custom test datasets and run targeted evaluations

The framework defaults to CPU for safety (so you can evaluate during training without GPU conflicts), but you can use --device gpu when the GPU is free for faster evaluation. This design allows continuous assessment throughout the training process without interfering with GPU resources needed for training.

For complete usage examples, command-line options, and troubleshooting, see the Evaluation Guide in the repository.

3. Comprehensive Testing Pipeline

3.1 Automated Testing Framework

The 06_testing package contains a parallel set of tests that double-check the full system. Listing 4 captures the idea behind run_comprehensive_tests.

We group tests into basic functionality, historical accuracy, linguistic quality, performance, edge cases, and integration, then run them as a batch and emit a structured report. This mirrors how you would build a real CI test suite, but at a scale appropriate for this learning project.

def run_comprehensive_tests(model, tokenizer, device='cuda'):
    """Run comprehensive tests on historical language model"""
    
    test_results = {
        'basic_functionality': test_basic_functionality(model, tokenizer, device),
        'historical_accuracy': test_historical_accuracy(model, tokenizer, device),
        'linguistic_quality': test_linguistic_quality(model, tokenizer, device),
        'performance_metrics': test_performance_metrics(model, tokenizer, device),
        'edge_cases': test_edge_cases(model, tokenizer, device),
        'integration_tests': test_integration(model, tokenizer, device)
    }
    
    # Generate test report
    generate_test_report(test_results)
    
    return test_results

def test_basic_functionality(model, tokenizer, device):
    """Test basic model functionality"""
    
    tests = {
        'text_generation': test_text_generation(model, tokenizer, device),
        'tokenization': test_tokenization(tokenizer),
        'model_loading': test_model_loading(model, device),
        'memory_usage': test_memory_usage(model, device),
        'inference_speed': test_inference_speed(model, tokenizer, device)
    }
    
    return tests

def test_historical_accuracy(model, tokenizer, device):
    """Test historical accuracy of generated text"""
    
    tests = {
        'temporal_consistency': test_temporal_consistency(model, tokenizer, device),
        'factual_accuracy': test_factual_accuracy(model, tokenizer, device),
        'period_appropriate_language': test_period_language(model, tokenizer, device),
        'geographical_accuracy': test_geographical_accuracy(model, tokenizer, device),
        'social_context_accuracy': test_social_context(model, tokenizer, device)
    }
    
    return tests

Listing 4: Comprehensive Testing Framework

Automated tests cover basics, historical accuracy, linguistic quality, performance, edge cases, and integration.

3.2 Interactive Testing and Validation

For manual exploration, the interactive testing interface (conceptually similar to the CLI flows in 06_inference/inference_unified.py) lets you type prompts, trigger specific test groups, and immediately inspect analysis for each generation. Listing 5 shows a simple REPL loop that dispatches to the same evaluation helpers used in the automated tests.

def interactive_testing(model, tokenizer, device='cuda'):
    """Interactive testing interface for historical language model"""
    
    print("Interactive Testing Mode")
    print("=" * 50)
    print("Enter prompts to test the model. Type 'quit' to exit.")
    print("Available commands:")
    print("  - Enter any text prompt to generate continuation")
    print("  - 'test_historical' - Run historical accuracy tests")
    print("  - 'test_linguistic' - Run linguistic quality tests")
    print("  - 'test_performance' - Run performance tests")
    print("  - 'quit' - Exit testing mode")
    print()
    
    while True:
        try:
            prompt = input("Enter prompt: ").strip()
            
            if prompt.lower() == 'quit':
                break
            elif prompt.lower() == 'test_historical':
                run_historical_tests(model, tokenizer, device)
            elif prompt.lower() == 'test_linguistic':
                run_linguistic_tests(model, tokenizer, device)
            elif prompt.lower() == 'test_performance':
                run_performance_tests(model, tokenizer, device)
            elif prompt:
                # Generate text
                generated = generate_text(model, tokenizer, prompt, device)
                print(f"Generated: {generated}")
                print()
                
                # Analyze generated text
                analysis = analyze_generated_text(generated, prompt)
                print(f"Analysis: {analysis}")
                print()
            else:
                print("Please enter a valid prompt or command.")
                
        except KeyboardInterrupt:
            print("\nExiting interactive testing mode...")
            break
        except Exception as e:
            print(f"Error: {e}")
            print("Please try again.")

def analyze_generated_text(text, prompt):
    """Analyze generated text for quality and accuracy"""
    
    analysis = {
        'length': len(text),
        'sentences': len(nltk.sent_tokenize(text)),
        'historical_accuracy': assess_historical_accuracy(text, {}),
        'linguistic_quality': assess_linguistic_quality(text),
        'coherence': assess_coherence(text),
        'relevance': assess_relevance(text, prompt)
    }
    
    return analysis

Listing 5: Interactive Testing Interface

Interactive mode lets you try prompts, run quick tests, and see immediate analysis.

3.3 Performance Benchmarking

Performance benchmarking follows the same pattern: generate controlled workloads and measure speed and resource usage. Listing 6 illustrates how we vary sequence length, measure average latency, and compute tokens-per-second, alongside separate helpers for memory, batch throughput, long-sequence handling, and basic concurrency.

def benchmark_model_performance(model, tokenizer, device='cuda'):
    """Benchmark model performance across different scenarios"""
    
    benchmarks = {
        'inference_speed': benchmark_inference_speed(model, tokenizer, device),
        'memory_usage': benchmark_memory_usage(model, device),
        'batch_processing': benchmark_batch_processing(model, tokenizer, device),
        'long_sequence_handling': benchmark_long_sequences(model, tokenizer, device),
        'concurrent_requests': benchmark_concurrent_requests(model, tokenizer, device)
    }
    
    return benchmarks

def benchmark_inference_speed(model, tokenizer, device):
    """Benchmark inference speed for different sequence lengths"""
    
    sequence_lengths = [50, 100, 200, 500, 1000]
    results = {}
    
    for length in sequence_lengths:
        # Generate test prompts of different lengths
        prompts = generate_test_prompts(length, num_prompts=100)
        
        # Measure inference time
        start_time = time.time()
        for prompt in prompts:
            generate_text(model, tokenizer, prompt, device)
        end_time = time.time()
        
        total_time = end_time - start_time
        avg_time_per_prompt = total_time / len(prompts)
        tokens_per_second = length / avg_time_per_prompt
        
        results[length] = {
            'avg_time_per_prompt': avg_time_per_prompt,
            'tokens_per_second': tokens_per_second,
            'total_time': total_time
        }
    
    return results

Listing 6: Performance Benchmarking

Benchmarks capture inference speed, memory, batch throughput, long-sequence handling, and simple concurrency.

4. Model Deployment and Publishing

With evaluation and testing complete, we’re ready to make our models available for use. This section covers the two deployment paths we support: direct inference from PyTorch checkpoints (useful during development and for maximum control) and publishing to Hugging Face Hub (for easy sharing and community access).

As called out in Part 1 , both the SLM (117M parameters) and the Regular Model (354M parameters) are fully trained and available. The SLM has already been published on Hugging Face Hub , while the Regular Model is ready for publication. Both can also be run directly from local PyTorch checkpoints.

4.1 Two Paths to Inference

We provide two complementary ways to run inference, each suited to different use cases.

PyTorch Checkpoint Inference gives you direct access to the trained model weights without any conversion overhead. This is ideal during development, when you want to test a freshly trained checkpoint, or when you need maximum control over the inference process. The checkpoints live in 09_models/checkpoints/ - the SLM at slm/checkpoint-4000.pt (117M parameters) and the Regular Model at checkpoint-60001.pt (354M parameters). The inference_pytorch.py script handles loading these directly: Listing 7

# SLM inference from checkpoint
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/slm/checkpoint-4000.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Regular model inference from checkpoint
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/checkpoint-60001.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

Listing 7: Running Inference from PyTorch Checkpoints

Hugging Face Model Inference uses the published models on Hugging Face Hub, which means anyone can load and use them with just a few lines of code - no need to download checkpoints or set up the full training environment. The inference_unified.py script provides a consistent interface for both published models and local checkpoints: Listing 8

# Published model inference (downloads from Hugging Face Hub)
python 06_inference/inference_unified.py \
  --published \
  --model_name bahree/london-historical-slm \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Interactive mode for exploration
python 06_inference/inference_unified.py --published --model_type slm --interactive

# Demo mode with curated historical prompts
python 06_inference/inference_unified.py --published --model_type slm --demo

Listing 8: Hugging Face Model Inference

We’ve tested both paths extensively. The published SLM loads in about 9 seconds on a GPU, generates text in under 6 seconds, and passes all 10 automated validation tests. The unified inference script provides clean logging, proper model detection, and accurate parameter counts - small details that make a big difference when debugging or demonstrating the models.

4.2 Publishing to Hugging Face Hub

Publishing to Hugging Face Hub makes our models accessible to the broader community without requiring anyone to clone our repository or set up a training environment. The process involves converting our PyTorch checkpoints to the Hugging Face format, creating a model card with documentation, and uploading everything to the Hub.

The publishing workflow is handled by scripts in 10_scripts/ - specifically publish_slm_to_huggingface.py for the SLM and publish_to_huggingface.py for the Regular Model. Listing 9 shows the core publishing flow: authenticate with the Hub, create (or reuse) a repository, save the model and tokenizer locally in Hugging Face format, upload the folder, and generate a model card.

def publish_to_huggingface(model, tokenizer, model_name, description, tags):
    """Publish model to Hugging Face Hub"""
    from huggingface_hub import HfApi
    
    api = HfApi()
    
    # Create model repository
    repo_id = f"bahree/{model_name}"
    api.create_repo(repo_id=repo_id, exist_ok=True)
    
    # Save model and tokenizer locally
    model.save_pretrained(f"./models/{model_name}")
    tokenizer.save_pretrained(f"./models/{model_name}")
    
    # Upload to Hub
    api.upload_folder(
        folder_path=f"./models/{model_name}",
        repo_id=repo_id,
        commit_message="Initial model upload"
    )
    
    # Generate and upload model card (README.md)
    model_card = generate_model_card(model_name, description, tags)
    api.upload_file(
        path_or_fileobj=model_card,
        path_in_repo="README.md",
        repo_id=repo_id
    )
    
    return repo_id

Listing 9: Hugging Face Publishing

The generate_model_card() function creates the README.md that appears on the Hugging Face model page. This includes model description, architecture details, training data sources, usage examples, and limitations. You can see the live model cards at bahree/london-historical-slm and bahree/london-historical-llm .

Listing 10 shows how to load and use the published models:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the published model
model_name = "bahree/london-historical-slm"  # or "bahree/london-historical-llm"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Generate historical text
prompt = "In the year 1834, I walked through the streets of London and witnessed"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=50,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.2,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Listing 10: Loading Models from Hugging Face Hub

4.3 Publishing Workflow

If you want to publish your own trained model to Hugging Face Hub, here’s the workflow we followed:

Set up authentication: Install huggingface_hub and authenticate with a token that has Write permissions. You can generate tokens at huggingface.co/settings/tokens .
Convert the checkpoint: PyTorch training checkpoints include optimizer states and training metadata that aren’t needed for inference. The conversion scripts extract just the model weights and translate them to Hugging Face’s naming conventions (covered in detail in Section 5).
Prepare the tokenizer: Save the tokenizer files alongside the model. Our custom tokenizer with 30,000 tokens and 150+ historical special tokens needs to be converted to the transformers library format.
Generate a model card: The README.md on your Hugging Face model page serves as documentation. Include model architecture details, training data sources, usage examples, evaluation results, and limitations. The scripts generate this automatically, but you should review and customize it.
Upload and validate: Push everything to the Hub, then immediately test with from_pretrained() to ensure the published model loads and generates correctly.

📝 Full documentation: See 08_documentation/HUGGINGFACE_PUBLISHING.md and 08_documentation/DEPLOYMENT_GUIDE.md in the repository for the complete step-by-step workflow with troubleshooting guidance.

5. PyTorch to Hugging Face Format Conversion

5.1 Why Format Conversion is Necessary

During training, our models are saved in PyTorch’s native .pt format. These checkpoints include everything needed to resume training: model weights, optimizer states, learning rate schedules, and training metadata. However, for deployment and sharing, we need a leaner, inference-optimized format compatible with the broader machine learning ecosystem.

Think of it like the difference between a development environment and a production deployment: training checkpoints are like a developer’s workspace with all the tools and intermediate files, while Hugging Face format is like a clean, standardized package that anyone can use without understanding the internal training details.

The Hugging Face Hub expects models to follow specific file structures, naming conventions, and metadata requirements. The conversion process extracts just the model weights (discarding optimizer states and training metadata), translates weight names to match Hugging Face conventions, creates proper configuration files, and ensures the tokenizer is compatible with the transformers library.

5.2 The Conversion Process

The conversion handles several transformations to bridge PyTorch and Hugging Face formats:

Weight name mapping: PyTorch layer names like transformer.h.0.attn.c_attn.weight become Hugging Face names like transformer.h.0.attn.c_attn.weight (mostly the same for GPT-2, but with careful handling of edge cases)
Automatic torch.compile handling: If you used torch.compile() during training, weights get prefixed with _orig_mod. - the conversion strips these prefixes
Configuration translation: Model hyperparameters (n_layer, n_head, n_embd, etc.) are mapped to Hugging Face’s config.json format
Tokenizer conversion: Our custom 30,000-token vocabulary with 150+ historical special tokens is converted to transformers library format
Validation: After conversion, we verify that the model loads correctly and produces expected outputs

💻 Full Implementation: See 10_scripts/publish_slm_to_huggingface.py for the complete conversion pipeline with error handling, validation, and model card generation.

5.3 Dependencies for Hugging Face Integration

The Hugging Face integration requires specific dependencies and follows established patterns for model publishing and usage: Listing 11

# Required dependencies for Hugging Face integration
huggingface_dependencies = {
    "transformers": ">=4.21.0",
    "torch": ">=1.12.0",
    "tokenizers": ">=0.12.0",
    "safetensors": ">=0.3.0",
    "accelerate": ">=0.20.0",
    "huggingface_hub": ">=0.10.0"
}

# Model loading and usage example
def load_published_model(model_name="bahree/london-historical-slm"):
    """Load published model from Hugging Face Hub"""
    
    # Suppress warnings for cleaner output
    import os
    import warnings
    import logging
    
    os.environ['TRANSFORMERS_VERBOSITY'] = 'error'
    warnings.filterwarnings('ignore')
    logging.getLogger("transformers").setLevel(logging.ERROR)
    os.environ['TOKENIZERS_PARALLELISM'] = 'false'
    
    # Load model and tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name)
    
    # Set pad token if not set
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
        tokenizer.pad_token_id = tokenizer.eos_token_id
    
    return model, tokenizer

def generate_historical_text(model, tokenizer, prompt, max_length=50, temperature=0.3):
    """Generate historical text using the published model"""
    
    # Tokenize input
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    
    # Generate text
    with torch.no_grad():
        outputs = model.generate(
            inputs,
            max_new_tokens=max_length,
            do_sample=True,
            temperature=temperature,
            top_p=0.9,
            top_k=20,
            repetition_penalty=1.2,
            no_repeat_ngram_size=3,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
            early_stopping=True
        )
    
    # Decode output
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated_text

# Example usage
if __name__ == "__main__":
    # Load the published model
    model, tokenizer = load_published_model()
    
    # Test prompts
    test_prompts = [
        "In the year 1834, I walked through the streets of London and witnessed",
        "The gentleman from the country said, 'we have never seen such a sight",
        "The Thames flowed dark and mysterious through the heart",
        "Parliament sat in Westminster Hall",
        "The Great Fire of 1666 had destroyed"
    ]
    
    # Generate text for each prompt
    for prompt in test_prompts:
        generated = generate_historical_text(model, tokenizer, prompt)
        print(f"Prompt: {prompt}")
        print(f"Generated: {generated}")
        print("-" * 80)

Listing 11: Hugging Face Dependencies

Hugging Face integration provides standard from_pretrained() loading and generation with minimal setup, making the models easy to share and reuse.

5.4 Comprehensive Testing and Validation Framework

Once a model is on the Hub, 06_inference/test_published_models.py provides a concrete implementation of the testing pattern in Listing 12. It loads the model via from_pretrained, runs functional, historical, linguistic, and performance checks, and prints a human-readable summary so you can verify the published artefact behaves like your local checkpoints.

def test_published_model(model_name="bahree/london-historical-slm"):
    """Comprehensive testing of published model"""
    
    print(f"Testing published model: {model_name}")
    
    # Load model
    model, tokenizer = load_published_model(model_name)
    
    # Test basic functionality
    basic_tests = test_basic_functionality(model, tokenizer)
    
    # Test historical accuracy
    historical_tests = test_historical_accuracy(model, tokenizer)
    
    # Test linguistic quality
    linguistic_tests = test_linguistic_quality(model, tokenizer)
    
    # Test performance metrics
    performance_tests = test_performance_metrics(model, tokenizer)
    
    # Compile results
    results = {
        "basic_functionality": basic_tests,
        "historical_accuracy": historical_tests,
        "linguistic_quality": linguistic_tests,
        "performance_metrics": performance_tests
    }
    
    # Print summary
    print("\nTest Results Summary:")
    print("=" * 50)
    for category, tests in results.items():
        print(f"\n{category.replace('_', ' ').title()}:")
        for test_name, result in tests.items():
            status = "PASS" if result else "FAIL"
            print(f"  {test_name}: {status}")
    
    return results

def test_basic_functionality(model, tokenizer):
    """Test basic model functionality"""
    
    tests = {}
    
    # Test model loading
    tests["model_loading"] = model is not None and tokenizer is not None
    
    # Test tokenizer functionality
    test_text = "In the year 1834, London was"
    tokens = tokenizer.encode(test_text)
    decoded = tokenizer.decode(tokens)
    tests["tokenizer_encode_decode"] = test_text in decoded
    
    # Test model generation
    try:
        inputs = tokenizer.encode(test_text, return_tensors="pt")
        with torch.no_grad():
            outputs = model.generate(inputs, max_new_tokens=10, do_sample=False)
        generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
        tests["model_generation"] = len(generated) > len(test_text)
    except Exception as e:
        tests["model_generation"] = False
    
    # Test special tokens
    special_tokens = ["<|london|>", "<|thou|>", "<|hath|>", "<|doth|>"]
    special_token_tests = []
    for token in special_tokens:
        if token in tokenizer.get_vocab():
            special_token_tests.append(True)
        else:
            special_token_tests.append(False)
    tests["special_tokens"] = any(special_token_tests)
    
    return tests

def test_historical_accuracy(model, tokenizer):
    """Test historical accuracy of generated text"""
    
    tests = {}
    
    # Test prompts for different historical periods
    period_prompts = {
        "1500-1600": "In the year 1550, the gentleman said",
        "1600-1700": "In the year 1650, the gentleman said",
        "1700-1800": "In the year 1750, the gentleman said",
        "1800-1850": "In the year 1834, the gentleman said"
    }
    
    for period, prompt in period_prompts.items():
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=30)
            
            # Check for period-appropriate language
            period_terms = {
                "1500-1600": ["ye", "hath", "doth", "thou", "thee"],
                "1600-1700": ["hath", "doth", "thou", "thee", "verily"],
                "1700-1800": ["hath", "doth", "thou", "thee", "indeed"],
                "1800-1850": ["indeed", "verily", "whilst", "pray"]
            }
            
            found_terms = sum(1 for term in period_terms[period] if term in generated.lower())
            tests[f"period_{period}"] = found_terms > 0
            
        except Exception as e:
            tests[f"period_{period}"] = False
    
    # Test London-specific knowledge
    london_prompts = [
        "The Thames flowed through",
        "Westminster Hall was",
        "The Tower of London",
        "Cheapside was filled with"
    ]
    
    london_tests = []
    for prompt in london_prompts:
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=20)
            london_tests.append(len(generated) > len(prompt))
        except Exception as e:
            london_tests.append(False)
    
    tests["london_knowledge"] = any(london_tests)
    
    return tests

def test_linguistic_quality(model, tokenizer):
    """Test linguistic quality of generated text"""
    
    tests = {}
    
    # Test prompts for linguistic quality
    quality_prompts = [
        "The gentleman walked through the garden",
        "In the morning, the sun rose",
        "The old man sat by the fire",
        "The young woman read her book"
    ]
    
    quality_tests = []
    for prompt in quality_prompts:
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=30)
            
            # Check for basic linguistic quality
            sentences = generated.split('.')
            quality_tests.append(len(sentences) > 1)
            
        except Exception as e:
            quality_tests.append(False)
    
    tests["linguistic_quality"] = any(quality_tests)
    
    # Test coherence
    coherence_prompt = "The gentleman walked through the garden and"
    try:
        generated = generate_historical_text(model, tokenizer, coherence_prompt, max_length=50)
        tests["coherence"] = len(generated) > len(coherence_prompt)
    except Exception as e:
        tests["coherence"] = False
    
    return tests

def test_performance_metrics(model, tokenizer):
    """Test performance metrics of the model"""
    
    tests = {}
    
    # Test inference speed
    test_prompt = "In the year 1834, London was"
    try:
        start_time = time.time()
        generated = generate_historical_text(model, tokenizer, test_prompt, max_length=50)
        end_time = time.time()
        
        inference_time = end_time - start_time
        tests["inference_speed"] = inference_time < 5.0  # Should complete within 5 seconds
        
    except Exception as e:
        tests["inference_speed"] = False
    
    # Test memory usage
    try:
        import psutil
        process = psutil.Process()
        memory_usage = process.memory_info().rss / 1024 / 1024  # MB
        tests["memory_usage"] = memory_usage < 8000  # Should use less than 8GB
        
    except Exception as e:
        tests["memory_usage"] = True  # Skip if psutil not available
    
    return tests

Listing 12: Testing Published Models

Published model tests validate loading, generation, historical accuracy, and basic performance before and after publication.

5.5 Model Card Generation

The model card serves as the primary documentation on Hugging Face Hub, making it the first thing users see when they discover your model. A well-crafted model card helps users understand what the model does, how to use it, and its limitations. The generate_comprehensive_model_card() function in 10_scripts/publish_slm_to_huggingface.py creates this documentation automatically.

What Makes an Effective Model Card:

The model card for our historical language models includes several key sections that provide users with everything they need to get started. At a minimum, include:

Model Description & Key Features: A clear explanation that the model was trained from scratch (not fine-tuned), emphasizing the 117M parameter SLM variant and 354M parameter Regular Model, with details about the custom 30,000-token vocabulary and 150+ historical special tokens.
Setup Instructions: Platform-specific guidance for creating virtual environments (Linux/macOS/Windows), installing dependencies (transformers, torch, accelerate), and handling different accelerators (CPU, NVIDIA CUDA, AMD ROCm).
Quick Start Code: Auto-device detection that works across CPU, CUDA, and ROCm with sensible generation parameters (temperature=0.8, top_p=0.95, repetition_penalty=1.2).
Training Details: Architecture specifics (GPT-2 Small/Medium), training infrastructure (2x GPU with Distributed Data Parallel), performance metrics (training loss, MFU utilization), and data sources (218+ historical sources spanning 1500-1850).
Example Prompts: Period-specific prompts demonstrating different historical eras (Tudor, Stuart, Georgian, Victorian) and London-specific contexts (Thames, Westminster, Parliament).
Testing & Validation: Instructions for running the automated test suite (test_published_models.py) and interactive testing with custom prompts.
Troubleshooting: Common issues and solutions for PyTorch installation, GPU detection, and memory constraints.
Citation & License: BibTeX citation format and MIT license information.

Key Implementation Details:

The model card generation follows Hugging Face conventions with YAML frontmatter specifying license, library, pipeline type, language, and tags. The script emphasizes that models were trained from scratch (not fine-tuned) and provides device-agnostic code examples that run on CPU, CUDA, and ROCm.

The card also includes detailed model selection guidance comparing the SLM (faster, lower memory) versus the Regular Model (higher quality, more parameters), helping users choose the right model for their use case - whether that’s quick experimentation, educational purposes, or production deployment.

💻 Complete Implementation: See 10_scripts/publish_slm_to_huggingface.py and 10_scripts/publish_to_huggingface.py for the full model card generation implementation.

👀 Live Model Cards: View the published cards at bahree/london-historical-slm and bahree/london-historical-llm .

📝 Documentation: See HUGGINGFACE_PUBLISHING.md and DEPLOYMENT_GUIDE.md for complete publishing and deployment workflows.

5.6 Local Deployment Options

Finally, Listing 13 sketches how you might wrap a trained model into a simple REST API or CLI. These patterns are intentionally minimal, meant to help you connect the dots between the inference utilities in 06_inference/ and real applications (dashboards, notebooks, small services).

def setup_local_deployment(model, tokenizer, deployment_type='api'):
    """Set up local deployment for historical language model"""
    
    if deployment_type == 'api':
        return setup_api_deployment(model, tokenizer)
    elif deployment_type == 'cli':
        return setup_cli_deployment(model, tokenizer)
    elif deployment_type == 'notebook':
        return setup_notebook_deployment(model, tokenizer)
    else:
        raise ValueError(f"Unknown deployment type: {deployment_type}")

def setup_api_deployment(model, tokenizer):
    """Set up REST API deployment"""
    
    from flask import Flask, request, jsonify
    import torch
    
    app = Flask(__name__)
    
    @app.route('/generate', methods=['POST'])
    def generate_text():
        data = request.get_json()
        prompt = data.get('prompt', '')
        max_length = data.get('max_length', 100)
        temperature = data.get('temperature', 0.3)
        
        # Generate text
        inputs = tokenizer.encode(prompt, return_tensors="pt")
        with torch.no_grad():
            outputs = model.generate(
                inputs,
                max_length=max_length,
                temperature=temperature,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )
        
        generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
        
        return jsonify({
            'generated_text': generated_text,
            'prompt': prompt,
            'parameters': {
                'max_length': max_length,
                'temperature': temperature
            }
        })
    
    @app.route('/health', methods=['GET'])
    def health_check():
        return jsonify({'status': 'healthy', 'model_loaded': True})
    
    return app

def setup_cli_deployment(model, tokenizer):
    """Set up command-line interface deployment"""
    
    import argparse
    
    def main():
        parser = argparse.ArgumentParser(description='Historical Language Model CLI')
        parser.add_argument('--prompt', type=str, required=True, help='Text prompt')
        parser.add_argument('--max_length', type=int, default=100, help='Maximum length')
        parser.add_argument('--temperature', type=float, default=0.3, help='Temperature')
        parser.add_argument('--interactive', action='store_true', help='Interactive mode')
        
        args = parser.parse_args()
        
        if args.interactive:
            run_interactive_mode(model, tokenizer)
        else:
            generate_and_print(model, tokenizer, args.prompt, args.max_length, args.temperature)
    
    return main

Listing 13: Local Deployment Setup

Local deployment options: REST API, CLI, or notebook integration for different workflows.

6. Quality Assurance and Validation

Before wrapping up, let’s look at the quality assurance systems that ensure the models behave reliably across different scenarios.

6.1 Automated Quality Checks

The system includes automated quality checks that validate model performance and reliability: Listing 14

def run_quality_checks(model, tokenizer, device='cuda'):
    """Run quality checks on historical language model"""
    
    quality_checks = {
        'model_integrity': check_model_integrity(model),
        'tokenizer_consistency': check_tokenizer_consistency(tokenizer),
        'generation_quality': check_generation_quality(model, tokenizer, device),
        'historical_accuracy': check_historical_accuracy(model, tokenizer, device),
        'performance_metrics': check_performance_metrics(model, tokenizer, device),
        'memory_usage': check_memory_usage(model, device),
        'error_handling': check_error_handling(model, tokenizer, device)
    }
    
    # Generate quality report
    quality_report = generate_quality_report(quality_checks)
    
    return quality_checks, quality_report

def check_model_integrity(model):
    """Check model integrity and consistency"""
    
    checks = {
        'parameter_count': check_parameter_count(model),
        'weight_distribution': check_weight_distribution(model),
        'gradient_flow': check_gradient_flow(model),
        'activation_patterns': check_activation_patterns(model)
    }
    
    return checks

def check_generation_quality(model, tokenizer, device):
    """Check quality of generated text"""
    
    test_prompts = [
        "In the year of our Lord 1750, London was",
        "The Thames flowed through the heart of",
        "Merchants and tradesmen plied their wares",
        "The Great Fire of 1666 had changed",
        "Parliament sat in Westminster, making laws"
    ]
    
    quality_scores = []
    
    for prompt in test_prompts:
        # Generate text
        generated = generate_text(model, tokenizer, prompt, device)
        
        # Check quality metrics
        quality_score = {
            'coherence': assess_coherence(generated),
            'grammatical_correctness': assess_grammatical_correctness(generated),
            'historical_accuracy': assess_historical_accuracy(generated, {}),
            'linguistic_quality': assess_linguistic_quality(generated),
            'relevance': assess_relevance(generated, prompt)
        }
        
        quality_scores.append(quality_score)
    
    return quality_scores

Listing 14: Quality Assurance Checks

Quality checks cover model integrity, generation quality, historical accuracy, performance, and error handling, ensuring the models behave reliably across different scenarios.

6.2 Continuous Integration and Testing

If you want to wire this into a lightweight CI gate, keep it simple and CPU-friendly. The goal is not to re-run full benchmarks in CI - it’s to catch obvious regressions (can the model load, can it generate, do the evaluators still run).

Minimal CI smoke checks (suggested):

# 1) Run a fast, local evaluation pass (no external APIs)
python 05_evaluation/run_evaluation.py --mode quick --device cpu

# 2) Run a local inference smoke test from a checkpoint (replace with your path)
python 06_inference/inference_pytorch.py --checkpoint <path-to-checkpoint.pt> --prompt "In the year 1834, London was"

# 3) Optional: test the published model (requires downloading from Hugging Face)
python 06_inference/test_published_models.py --model_name bahree/london-historical-slm

7. Summary

We’ve now completed the full cycle of building language models from scratch. This final part has shown how to transform trained models into working systems that can be evaluated, tested, and deployed for real-world use. The journey that began in Part 1 with using published models, continued through Part 2 ’s data collection and tokenization, and Part 3 ’s training architecture, now concludes with evaluation and deployment - the critical final steps that make models usable.

What we’ve built:

The evaluation, testing, and deployment pipeline provides a practical approach for bringing historical language models from research to deployment. We’ve created specialized assessment metrics that go beyond standard LLM evaluation to catch historical inaccuracies, temporal inconsistencies, and period-inappropriate language. The testing infrastructure ensures reliability across different scenarios, while multiple deployment options make the models accessible to researchers, educators, and developers worldwide.

Current Deployment Status:

PyTorch Checkpoint Inference: Fully working with both SLM and Regular models
Hugging Face Model Inference: SLM published and available, Regular model ready
Local Testing: Both inference methods tested and validated on a remote Ubuntu machine
Documentation: Complete guides and examples for all inference methods
Performance: Clean logging, proper model detection, accurate parameter counts

The Complete Pipeline:

This four-part series has demonstrated the complete LLM development lifecycle:

Data Collection ( Part 2 ): We gathered 218+ historical sources spanning 1500-1850, processed them through a sophisticated cleaning pipeline, and created a 500M+ character corpus of authentic historical English.
Custom Tokenization ( Part 2 ): We built a specialized BPE tokenizer with 30,000 vocabulary tokens and 150+ special tokens that understand historical language patterns, London geography, and period-specific terminology.
Model Training ( Part 3 ): We implemented custom GPT architectures, optimized for multi-GPU training, and successfully trained two models - an SLM (117M parameters) and a Regular model (354M parameters) - both capable of generating authentic historical text.
Evaluation & Deployment (This Part): We built comprehensive evaluation frameworks that assess historical accuracy, linguistic quality, and temporal consistency. We created a testing infrastructure for reliability and deployed models to the Hugging Face Hub for community access.

The Learning Journey:

What started as a learning project has become a complete, working system that demonstrates every aspect of LLM development - from raw data collection through model deployment. The principles and techniques we’ve covered scale from the 500M-character corpus to production-scale systems, and the evaluation frameworks we’ve built can be adapted to any domain-specific language modeling task.

Whether you’re a researcher exploring historical linguistics, an educator teaching AI concepts, or a developer building specialized language models, this series provides the complete toolkit for understanding and implementing LLM development from scratch. The models are published, the code is available, and the journey from data to deployment is complete.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete training infrastructure (04_training/), model architecture (config.py), and evaluation/deployment (05_evaluation/, 06_inference/, 10_scripts/)

🟥 Series Posts: Part 1 - Using the Published Historical Models | Part 2 - Data Collection & Custom Tokenizer | Part 3 - Training Architecture & GPU Optimization | Part 4 (this post)

🟧 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📗 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts

8. Resources

If you want to reproduce the full pipeline (or adapt it to your own domain), these are the most useful starting points:

Public GitHub repo: github.com/bahree/helloLondon
Evaluation guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/EVALUATION_GUIDE.md
Hugging Face publishing guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/HUGGINGFACE_PUBLISHING.md
Deployment guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/DEPLOYMENT_GUIDE.md
Published models: bahree/london-historical-slm | bahree/london-historical-llm

References

Vaswani et al. (2017) - Attention Is All You Need: https://arxiv.org/abs/1706.03762
Radford et al. (2019) - Language Models are Unsupervised Multitask Learners: https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe
Lin (2004) - ROUGE: A Package for Automatic Evaluation of Summaries: https://aclanthology.org/W04-1013/
Papineni et al. (2002) - BLEU: A Method for Automatic Evaluation of Machine Translation: https://aclanthology.org/P02-1040/
Hendrycks et al. (2021) - Measuring Massive Multitask Language Understanding (MMLU): https://arxiv.org/abs/2009.03300
Zellers et al. (2019) - HellaSwag: Can a Machine Really Finish Your Sentence?: https://arxiv.org/abs/1905.07830
Liu et al. (2023) - G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment: https://arxiv.org/abs/2303.16634

Acknowledgments

This project builds upon the excellent work of the open-source community. Special thanks to haykgrigo3’s TimeCapsuleLLM for the initial inspiration and framework for historical language model training, and to Andrej Karpathy’s nanoGPT for the foundational GPT architecture and training methodology. The project extends these foundations with specialized adaptations for historical text, including custom tokenizers, advanced data filtering, evaluation frameworks, and educational deployment infrastructure.

Vibe Coding

Sun, 23 Nov 2025 00:00:00 +0000

Deadlock by design: two vibes, two locks, zero unlocks.

In vibe coding, that’s not a bug - it is a feature. 😎

Here’s what happens when agreement-first engineering meets C++ and mutexes:

#include <iostream>
#include <mutex>
#include <stdexcept>

std::mutex mVibes, mProd;

void shipToProd(bool agree) {
    // We lock the vibes and production—because feelings 
    // and facts both need exclusive access.
    std::lock_guard<std::mutex> a(mVibes);
    std::lock_guard<std::mutex> b(mProd);

    if (agree) {
        std::cout << "Deploying to prod…\n";
        // Reality pushes back:
        throw std::runtime_error("DeadlockException: vibes vs reality");
    } else {
        std::cout << "Ignored tests.\n";
    }
}

int main() {
    try {
        shipToProd(true); // agreement-first engineering
    } catch (const std::exception& ex) {
        std::cout << "AI: You're absolutely right! (" << ex.what() << ")\n";
        std::cout << "/* lock(prod); lock(vibes); // Deadlock achieved; energy immaculate */\n";
        std::cout << "COMMIT vibes; ROLLBACK sanity;\n";
    }
    return 0;
}

Listing 1: Two mutexes walk into prod… and never come out

#GeekyJokes #AI #GenAI #VibeCoding

🏛️Building LLMs from Scratch - Part 3: Training Architecture & GPU Optimization

Sat, 01 Nov 2025 00:00:00 +0000

TL;DR

In this third part of our 4-part series on building language models from scratch, I explore the complete training infrastructure that transforms our clean historical data and custom tokenizer into working language models.

Part 1 How to build a Large Language Model from Scratch - covered using the published model
Part 2 Building LLMs from Scratch - Part 2: Data Collection & Custom Tokenizers - detailed data collection and custom tokenizer development.

Here, we build the complete training pipeline from a custom GPT architecture through deployment-ready checkpoints.

This post demonstrates how to design custom model architectures, optimize GPU utilization, and implement comprehensive training pipelines that transform our 500M+ character historical corpus into two working language models.

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need significantly larger datasets, more sophisticated infrastructure, and additional considerations that are not covered in this post.

As outlined in Part 1 , both the SLM (117M parameters) and the regular Model (354M parameters) use the same training code and pipeline (04_training/train_model_slm.py and 04_training/train_model.py) with different configurations defined in config.py. The training infrastructure, GPU optimization, checkpointing, and WandB integration are identical - only the model architecture parameters differ.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete training infrastructure (04_training/), model architecture (config.py), and GPU configuration (08_documentation/GPU_TUNING.md)

🧱 Series Posts: Part 1 – Using the Published Historical Models | Part 2 – Data Collection & Custom Tokenizer | Part 3 (this post) | Part 4 – Evaluation & Deployment

🤗 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📚 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts.

1. The Training Challenge: From Data to Working Models

Now that we have our clean historical corpus and custom tokenizer from Part 2 , we need to transform this data into working language models. This isn’t just about running training scripts – it’s about designing an architecture that can learn from historical text, optimizing for the unique patterns of 1500-1850 English, and building infrastructure to handle the computational demands of language model training.

The challenge with historical language modeling isn’t just having enough data - it’s having the right architecture and training process that can learn from the complex linguistic patterns in historical texts. Unlike modern text, historical English contains archaic vocabulary, period-specific terminology, and cultural references that require specialized attention mechanisms and training strategies.

1.1 High-Level Training Process Overview

The model training pipeline transforms our clean historical data and custom tokenizer into working language models through several key stages:

Model Architecture Design - involves a custom GPT implementation optimized for historical text patterns
GPU Configuration covers - multi-GPU training with precision optimization and memory management
Training Infrastructure - includes distributed training, checkpointing, and experiment tracking
Performance Optimization - encompasses mixed precision, compilation, and hardware-specific tuning
Model Validation - covers testing and evaluation of trained models.

Figure 1 below illustrates this complete training pipeline:

graph TD
    A[📚 Clean Historical Corpus<br/>500M+ characters] --> B[🔤 Custom Tokenizer<br/>30K vocab + 150+ special tokens]
    B --> C[🏗️ Model Architecture<br/>Custom GPT for Historical Text]
    
    C --> D[⚙️ GPU Configuration<br/>Multi-GPU + Precision Optimization]
    D --> D1[Mixed Precision<br/>bf16/fp16]
    D --> D2[Torch Compile<br/>JIT optimization]
    D --> D3[Memory Management<br/>Gradient checkpointing]
    
    D1 --> E[🏋️ Training Process<br/>60K iterations, checkpointing]
    D2 --> E
    D3 --> E
    
    E --> E1[SLM: 117M params<br/>7-8 hours training]
    E --> E2[Regular: 354M params<br/>28-32 hours training]
    
    E --> E3[WandB Integration<br/>Experiment tracking]
    E --> E4[Checkpointing<br/>Resume capability]
    E --> E5[Multi-GPU Support<br/>Distributed training]
    
    E1 --> F[📊 Model Evaluation<br/>Historical accuracy testing]
    E2 --> F
    E3 --> F
    E4 --> F
    E5 --> F
    
    F --> G{Quality OK?}
    G -->|Yes| H[🚀 Deployment<br/>Hugging Face + Local Inference]
    G -->|No| I[🔄 Retrain/Adjust]
    I --> E
    H --> J[💬 Text Generation<br/>Historical language output]
    
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style H fill:#e8f5e8
    style J fill:#fff3e0

Figure 1: Complete Training Pipeline

We will explore each of these components in detail, starting with the model architecture design, but first, let’s discuss why I chose PyTorch as the framework for this project.

1.2 Using PyTorch

I chose PyTorch for this project based on three key factors: educational accessibility, integration with the research ecosystem, and practical convenience. PyTorch provides many components out of the box - transformer blocks, attention layers, feed-forward networks, training loops, and CUDA support - which makes it much easier for learners building their first language model.

From a technical perspective, PyTorch’s memory management and GPU optimization features-including automatic mixed precision, gradient checkpointing, and efficient attention implementations well-suited for the resource-intensive task of training language models on historical text.

PyTorch’s recent developments, such as torch.compile, FlashAttention kernels, and SDPA operator (scaled dot-product attention), provide significant performance improvements, making training more efficient. These improvements enhance both speed and memory efficiency, which are critical for scaling LLMs. Of course, in our case, we’re building a working toy example rather than scaling to production levels, and these optimizations help keep training times reasonable on available hardware.

What about other Frameworks?

I also considered TensorFlow and JAX, but neither seemed right for helloLondon; TensorFlow’s API felt too complex, specifically from a beginner’s perspective. JAX has excellent performance and a clean, functional approach, but it’s more research-focused and has a smaller ecosystem, which would make it harder to follow along and experiment with.

2. Model Architecture Overview

2.1 Understanding the GPT Architecture

Our custom GPT (Generative Pre-trained Transformer) is a decoder-only transformer model designed for autoregressive language modeling on historical text. The architecture consists of four core components, each serving a distinct purpose in the sequence-to-sequence prediction pipeline. These are: token embeddings, position embeddings, causal self-attention mechanisms, and the language modeling head. Let us double-click into each component to understand its role and implementation.

2.1.1 Token Embeddings

Token embeddings convert discrete token IDs from our 30,000-token historical vocabulary into dense, continuous vector representations. Each token (whether it’s a word, subword unit, or special token) is mapped to a point in a high-dimensional space (768 dimensions for SLM, 1024 for the regular model).

This is implemented as a simple lookup table - wte = torch.nn.Embedding(config.vocab_size, config.n_embd). When processing the token sequence, we look up the corresponding vector for each token ID. These embeddings are learned during training - the model learns which tokens should be close together in this vector space based on their co-occurrence patterns in historical text.

For historical language models, this is particularly valuable because rare historical terms (like “yeoman” or “guildhall”) get their own representations that can capture contextual relationships with related terms from that era.

2.1.2 Position Embeddings

Position embeddings encode each token’s absolute position within the sequence. This is crucial because, unlike recurrent models, transformer architectures have no inherent notion of temporal order or sequence position - they process all tokens in parallel. Let us double-click into the problem why.

Think of it like reading words without any sense of order. The words “The cat chased the mouse” would be indistinguishable from “Mouse the chased cat the” - you’d see the same words but lose all meaning because you don’t know which word came first, second, or third. Transformers face exactly this problem because they process all words simultaneously rather than sequentially, unlike older RNN models.

To help the model understand word order, we add position embeddings to the token embeddings. This way, each token’s representation includes information about both “what” the token is and “where” it appears in the sequence. We use learned position embeddings (as opposed to fixed sinusoidal patterns): wpe = torch.nn.Embedding(config.block_size, config.n_embd). For the SLM with a 512-token context window, we learn 512 different position vectors (one for each possible position). Similarly, the regular model with a 1024-token context learns 1024 position vectors.

Position embeddings work like giving each word a “timestamp” or “address” that tells the model where it sits in the sequence:

Token embedding says: “This is the word ‘cat’” → converts to a vector like [0.2, -0.5, 0.8, ...]
Position embedding says: “This word is at position 3” → adds another vector like [0.1, 0.3, -0.2, ...]
Combined: The model sees both “what” the word is AND “where” it appears

The embedding vectors are combined element-wise: x = token_emb + position_emb. This allows the model to understand both what each token is (via the token embedding) and where it appears in the sequence (via the position embedding).

Our model uses learned position embeddings, meaning during training the model discovers that:

Position 1 tends to be capitalized (start of sentence)
Position 512 might be mid-sentence (needs different handling)
Certain positions in historical documents have patterns (formal openings, closings, etc.)

This is different from fixed sinusoidal embeddings (used in the original Transformer paper), which use a mathematical formula to encode positions. Learned embeddings are generally better because they adapt to specific patterns in the training data.

In historical texts, word order is crucial for understanding meaning. Consider “The King granted the land” versus “The land granted the King” - same words, completely different meanings. Historical legal documents and Victorian-era writings often have precise word order that changes legal or semantic meaning. Position embeddings ensure the model can distinguish between these critical variations.

2.1.3 Causal Self-Attention

Causal self-attention is the mechanism that allows each position in the sequence to selectively attend to previous positions. The “causal” constraint ensures the model can only look at past tokens (not future ones), which is essential for autoregressive generation.

When you read a sentence, you naturally use context from earlier words to understand later ones. If you see “The King granted the land to his loyal…”, you can predict that “servant,” “knight,” or “subject” might come next because you remember what came before. The model needs to do the same thing - use previous words to predict the next word.

However, there’s a crucial constraint: during training, when predicting word 7, the model must only see words 1-6, never word 8 or beyond. This “causal” (cause-and-effect) constraint ensures the model learns realistic patterns - in the real world, you can’t use future information to predict the present.

How Attention Works

Think of attention as a sophisticated “relevance detector”. When the model is processing the word “loyal” in our example above, it needs to look back and ask: “Which previous words are most relevant for understanding this context?” The attention mechanism computes a weighted sum of all previous token representations, where the weights are determined by how relevant each previous token is to the current one.

This is done through three learned linear projections that create different “views” of each word:

Query (Q): “What am I looking for?” - The current word asks a question
Key (K): “What do I have to offer?” - Previous words advertise their content
Value (V): “What information should I contribute?” - The actual information to pass forward

Let us see a practical example to help us grok the concept. Consider the historical phrase: “The alderman of Cheapside, having served the city faithfully, was granted…”

When processing “granted” the attention mechanism:

Creates a Query from “granted” asking “what context do I need?”
Compares this Query against Keys from all previous words
Finds high relevance with “alderman” (who is being granted something) and “faithfully” (why the grant is happening)
Uses these attention weights to pull relevant Values from those words
Combines this information to understand better “granted” in context

The attention score between token i and token j is computed as:

$$\text{Attention}(Q_i, K_j) = \text{softmax}\left(\frac{Q_i K_j^T}{\sqrt{d_k}}\right)V_j$$

Breaking this down:

$Q_i K_j^T$ computes how well the query from token i matches the key from token j (higher = more relevant)
$1/\sqrt{d_k}$ is a scaling factor that prevents scores from getting too large (which would make softmax too “sharp”)
$\text{softmax}$ converts scores into probabilities that sum to 1 (so each word gets a weighted “vote”)
Finally, we use these weights to combine the Values from all previous tokens

The $1/\sqrt{d_k}$ scaling factor (where $d_k = 64$ for our SLM, so $\sqrt{64} = 8$) prevents the dot products from growing too large with high-dimensional embeddings, ensuring stable gradients during training. The softmax ensures the weights sum to 1, creating a proper probability distribution over the previous tokens.

Why This Matters for Historical Text?

Historical documents present unique challenges that make attention particularly valuable. Consider a legal document from 1750: “John Smith, yeoman of the parish of St. Giles, being of sound mind and body, doth hereby bequeath…”

The attention mechanism enables the model to:

Connect “doth bequeath” back to “John Smith” across multiple clauses
Understand that “yeoman” modifies “John Smith” even though they’re separated
Learn that “doth” (archaic) and “does” (modern) serve similar grammatical functions
Recognize that formal legal phrasing follows specific patterns

For our historical language models, this attention mechanism learns which historical terms and phrases co-occur and relate to one another contextually - crucial for understanding historical documents where terminology and phrasing differ from modern English. The model learns to attend to relevant historical context, enabling it to generate coherent text that maintains period-appropriate language patterns and references.

2.1.4 Language Modeling Head

The language modeling head (also called the “output projection” or lm_head) is the final translator that turns the rich internal representation (after all the attention + MLP refinements) back into a decision: “Given everything I’ve seen so far, what is the most likely next token?” It does this by mapping each hidden vector at every position into a vector of length equal to the vocabulary size (30,000 in our historical tokenizer). Each element of that output vector is a logit - an unnormalized score indicating how likely the model thinks the token is to be the next one.

Implementation is intentionally simple: lm_head = torch.nn.Linear(n_embd, vocab_size). We don’t put an activation function after it because we want raw, unconstrained scores. Those scores then flow into:

Inference: Apply softmax -> probabilities -> sample or greedy pick
Training: Feed logits + target token IDs into cross-entropy loss -> gradients flow backward

You can think of logits as evidence totals. The softmax transforms those evidence values into a normalized probability distribution that the model can sample from. High logit = more supporting evidence; low logit = less.

Step-by-step (Inference vs Training):

Hidden state at last position (e.g., index 511) enters lm_head.
Linear projection produces a 30,000-dimensional logit vector.
In inference: probs = softmax(logits / temperature); optionally apply top-k/top-p filtering.
Sample (or argmax) a token → append to sequence → repeat.
In training: Cross-entropy compares logits to the true next token; loss scalar backpropagates through the head into all prior layers.

Because our vocabulary mixes common function words (“the”, “and”, “of”) with rare era-specific tokens (“yeoman”, “guildhall”, “paternoster”, “quoth”), the head must reliably distinguish both frequent and infrequent patterns. Rare historical tokens need consistent representations from embedding -> transformer -> head so they are not forgotten. If their logits remained perpetually low, the model would never learn to generate them in authentic contexts.

Logits (not probabilities) inside the model - We retain logits (raw, unnormalized scores) instead of immediately converting to probabilities because they yield numerically stable loss computation - PyTorch efficiently fuses log_softmax with negative log-likelihood - allow cleaner gradient flow before any normalization (we only invoke softmax when we actually need a distribution), and enable flexible post-processing (temperature scaling, top-k or top-p filtering, repetition penalties) directly in score space without forcing an extra probability recomputation step.

We reuse the input embedding matrix for the output projection to keep input and output semantics aligned and reduce parameter and memory traffic. This concept is called Weight Typing, which we will cover in detail in Section 2.2.2 - Weight Tying .

We share the embedding and output projection weights (self.transformer.wte.weight = self.lm_head.weight) so input token interpretation and next-token scoring occur in the same semantic space.

Using the shared embedding matrix $E$ (shape $(V,d)$), the logits are computed with $\text{logits} = h \cdot E^T$, reusing the same rows used for token lookup. This saves parameters (~23.0M SLM, ~30.7M Regular), keeps gradients for rare historical tokens coupled, and reduces memory traffic (Press & Wolf, 2017; Inan et al., 2016). See Section 2.2.2 for detailed mechanics, benefits, and the historian/scribe analogy.

In short, the lm_head converts rich contextual understanding into next-token scores; with weight tying (details in Section 2.2.2) it stays efficient and semantically consistent.

2.1.5 The Complete Flow

The complete forward pass through our GPT model works as follows:

Input: A sequence of token IDs (batch × sequence_length, e.g., 512 tokens)
Token Embedding: Convert each token ID to a dense vector (768 or 1024 dimensions)
Position Embedding: Add position information to each token
Transformer Blocks: Pass through n_layer transformer blocks (12 for SLM, 24 for regular model), each containing:
- Layer normalization
- Causal self-attention (with multiple heads)
- Residual connection
- Layer normalization
- Feed-forward MLP
- Residual connection
Final Layer Norm: Normalize the final hidden states
Language Head: Project to vocabulary logits (30,000 dimensions)
Output: Probability distribution over next token

This architecture design is conventional and follows the GPT-style pattern established by OpenAI’s GPT models. The traditional design is intentional - it allows for clear, educational learning from the implementation while being configured to work seamlessly with our historical tokenizer from Part 2.

Figure 2 below illustrates the complete architecture for the SLM:

graph TD
    A[Input Tokens<br/>512 tokens] --> B[Token Embedding<br/>30K vocab → 768 dim]
    A --> C[Position Embedding<br/>512 pos → 768 dim]
    B --> D[Add Embeddings]
    C --> D
    D --> E[Layer Norm]
    E --> F[Transformer Block 1<br/>12 heads, 768 dim]
    F --> G[Transformer Block 2<br/>12 heads, 768 dim]
    G --> H[...]
    H --> I[Transformer Block 12<br/>12 heads, 768 dim]
    I --> J[Final Layer Norm]
    J --> K[Language Head<br/>768 → 30K vocab]
    K --> L[Output Logits<br/>30K probabilities]
    
    subgraph "Key Specifications"
        M[Layers: 12<br/>Heads: 12<br/>Embedding: 768<br/>Context: 512<br/>Parameters: 117M<br/>Training: 7-8 hours<br/>MFU: 8-9%]
    end
    
    style A fill:#e1f5fe
    style L fill:#e8f5e8
    style M fill:#fff3e0

Figure 2: SLM Architecture (117M Parameters)

The Regular model, as shown in Figure 3 below, follows the same architectural pattern as the SLM but with increased capacity: 24 transformer layers instead of 12, 16 attention heads instead of 12, and 1024-dimensional embeddings instead of 768.

This represents a ~3x increase in parameters (354M vs 117M), ~2x more attention heads, and ~33% larger embedding dimensions, providing significantly more computational power for learning complex historical language patterns.

graph TD
    A[Input Tokens<br/>1024 tokens] --> B[Token Embedding<br/>30K vocab → 1024 dim]
    A --> C[Position Embedding<br/>1024 pos → 1024 dim]
    B --> D[Add Embeddings]
    C --> D
    D --> E[Layer Norm]
    E --> F[Transformer Block 1<br/>16 heads, 1024 dim]
    F --> G[Transformer Block 2<br/>16 heads, 1024 dim]
    G --> H[...]
    H --> I[Transformer Block 24<br/>16 heads, 1024 dim]
    I --> J[Final Layer Norm]
    J --> K[Language Head<br/>1024 → 30K vocab]
    K --> L[Output Logits<br/>30K probabilities]
    
    subgraph "Key Specifications"
        M[Layers: 24<br/>Heads: 16<br/>Embedding: 1024<br/>Context: 1024<br/>Parameters: 354M<br/>Training: 28-32 hours<br/>MFU: 15-20%]
    end
    
    style A fill:#e1f5fe
    style L fill:#e8f5e8
    style M fill:#fff3e0

Figure 3: Regular Model Architecture (354M Parameters)

2.2 SimpleGPT Class

Now that we’ve covered the theory behind the GPT architecture, let’s examine the actual implementation. The SimpleGPT class is at the heart of our implementation - it’s the core class that brings together all the components discussed in section 2.1 into a working language model. The class inherits from PyTorch’s **torch.nn.Module**, which is the base class for all neural network components in PyTorch. This gives us access to automatic differentiation, GPU support, and other PyTorch features.

2.2.1 The `init` method

The __init__ method is the constructor that assembles our entire language model from individual components. First, it stores all hyperparameters (such as vocabulary size, embedding dimensions, and the number of layers) in a configuration object that the rest of the model can reference. Next, it creates the embedding layers - one that converts our 30,000 historical tokens into dense vectors, and another that encodes position information. Hence, the model knows where each word appears in the sequence.

Next, it builds the transformer blocks - the core processing units that do the heavy lifting. Each block contains self-attention mechanisms and feed-forward networks that learn to understand relationships between words. The method also initializes the language modeling head, the final layer that converts all internal processing back into probabilities for which word should come next.

Finally, it sets up proper weight initialization to ensure the model starts with good random weights (not too big, not too small), and implements weight tying between the input embeddings and the output layer. This clever technique reduces the number of parameters while improving training efficiency by sharing weights between the first and last layers.

This is important because if the weights are too large, the model’s gradients can explode during training, leading to unstable learning. If they start too small, gradients can vanish, rendering the model unable to learn. Our initialization ensures the model begins in the “Goldilocks zone” - just right for effective learning. Without this, even a perfectly designed architecture might fail to train properly.

Now, let me show you the actual implementation. The code in Listing 1 demonstrates how we implement the core GPT architecture:

class SimpleGPT(torch.nn.Module):
    """Simple GPT model based on nanoGPT
    
    This class implements a decoder-only transformer model optimized for 
    historical text generation. It inherits from PyTorch's Module class
    to get automatic differentiation and GPU support.
    """
    
    def __init__(self, config):
        super().__init__()  # Initialize the parent PyTorch Module class
        self.config = config  # Store all model hyperparameters
        
        # Create the main transformer components using ModuleDict
        # ModuleDict allows us to organize related layers together
        self.transformer = torch.nn.ModuleDict(dict(
            # Token Embedding Layer (wte = "word token embedding")
            # Converts each token ID to a high-dimensional vector
            # Input: token IDs (integers 0 to vocab_size-1)
            # Output: dense vectors of size $n_{embd}$ (e.g., 768 dimensions)
            wte = torch.nn.Embedding(config.vocab_size, config.n_embd),
            
            # Position Embedding Layer (wpe = "word position embedding") 
            # Encodes where each token appears in the sequence
            # Input: position indices (0 to block_size-1)
            # Output: dense vectors of size $n_{embd}$
            wpe = torch.nn.Embedding(config.block_size, config.n_embd),
            
            # Dropout Layer for regularization
            # Randomly sets some inputs to zero during training to prevent overfitting
            drop = torch.nn.Dropout(config.dropout),
            
            # Stack of Transformer Blocks (h = "hidden layers")
            # Each SimpleBlock contains self-attention and feed-forward layers
            # We create n_layer blocks (e.g., 12 for SLM, 24 for regular model)
            h = torch.nn.ModuleList([SimpleBlock(config) for _ in range(config.n_layer)]),
            
            # Final Layer Normalization
            # Normalizes the output before the language modeling head
            ln_f = torch.nn.LayerNorm(config.n_embd, bias=config.bias),
        ))
        
        # Language Modeling Head
        # Converts the final hidden states back to vocabulary space
        # Input: hidden states of size $n_{embd}$
        # Output: logits for each token in vocabulary ($vocab_{size}$ logits)
        self.lm_head = torch.nn.Linear(config.n_embd, config.vocab_size, bias=False)
        
        # Initialize all weights using our custom initialization method
        # This ensures the model starts with good random weights
        self.apply(self._init_weights)
        
        # Weight Tying: Share weights between input embeddings and output layer
        # This technique improves training efficiency and model performance
        # by ensuring the same representation space is used for input and output
        self.transformer.wte.weight = self.lm_head.weight

Listing 1: SimpleGPT Model Architecture

2.2.2 Weight Tying

The tied weights between the embedding layer and language modeling head (self.transformer.wte.weight = self.lm_head.weight) are a crucial optimization for our historical language model. In a typical neural network, you’d have two separate weight matrices - one for converting input tokens to embeddings, and another for converting hidden states back to vocabulary probabilities. Weight tying means we use the same weight matrix for both operations.

Think of it like this: instead of having two different dictionaries (one for reading, one for writing), we use the same dictionary for both. The same table that maps “alderman” → [0.2, -0.5, 0.8, …] is used whether the model is reading “alderman” as input or trying to generate “alderman” as output.

Without weight tying, the model would have two separate weight matrices - one for converting input tokens to embeddings, and another for converting hidden states back to vocabulary probabilities. This means the model could learn that “alderman” means one thing when it sees it as input, but something slightly different when it tries to generate it as output. For rare historical terms, this inconsistency can cause the model to “forget” how to use words it has seen before properly.

Historical vocabulary contains many rare terms such as “quoth” “alderman” and “paternoster” that appear infrequently in our training data. Without weight tying, the model might learn different representations for the same word when it sees it as input versus when it generates it as output. This inconsistency can cause the model to struggle with rare historical terms.

When the model sees “alderman” in the input, it learns a specific representation of it. Later, when it needs to generate “alderman” in the output, it uses that same learned representation, ensuring consistency and improving the model’s ability to generate coherent historical language with proper terminology.

Mechanics (matrix reuse) A single matrix $E \in \mathbb{R}^{(V \times d)}$ serves both roles: row lookup for input embeddings and column interaction for output scoring. The language head reuses it to compute logits via:

$$\text{logits} = h \cdot E^T$$

where $h$ is the hidden state at each position, this keeps the input interpretation and output prediction within the same semantic geometry - no second projection to drift or disagree.

Why it helps Parameter savings (~23.0M SLM, ~30.7M Regular) lower memory footprint and bandwidth. Gradients for predicting a rare token (e.g., yeoman, guildhall, paternoster) directly refine the very rows used to embed it on future inputs - improving both recall and generation. Shared weights mildly regularize against the two spaces drifting apart and empirically improve perplexity for mid-scale autoregressive models (Press & Wolf, 2017; Inan et al., 2016).

Analogy If the transformer stack is a panel of historians debating context, the language modeling head is the scribe choosing the next historically plausible word. Weight tying means the scribe and historians consult the same dictionary - no translation mismatch between how words are read and how they’re proposed.

Practical notes Avoid inflating vocabulary unnecessarily (cost scales with $V$); tied weights do not remove the need for careful rare token coverage in the corpus; and if later adding adapters or LoRA heads, remember that tying interacts with how those layers inject low-rank updates.

Now that we understand how the model efficiently handles vocabulary, let’s examine the core processing units that transform these embeddings into meaningful representations.

2.3 Transformer Block Design

Each transformer block implements the standard attention and feed-forward pattern, but with optimizations for historical text processing. Let us look at the code real quick and then get into a little more detail.

class SimpleBlock(torch.nn.Module):
    """Simple transformer block"""
    
    def __init__(self, config):
        super().__init__()
        self.ln_1 = torch.nn.LayerNorm(config.n_embd, bias=config.bias)
        self.attn = SimpleCausalSelfAttention(config)
        self.ln_2 = torch.nn.LayerNorm(config.n_embd, bias=config.bias)
        self.mlp = SimpleMLP(config)
    
    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

Listing 2: Transformer Block Implementation

This code implements a single transformer block, which, as we know, is the fundamental building unit of our GPT model.

2.3.1 Self-Attention Step

The self-attention step (x = x + self.attn(self.ln_1(x))) is the block that first normalizes the input with LayerNorm, then applies self-attention to understand relationships between words. The + creates a “residual connection” that helps information flow through the network.

As we discussed in section 2.1.4 , self-attention is the “magic” of transformers, allowing each token to decide how much attention to pay to every other token in the sequence. Our implementation uses multiple attention heads (12 for SLM, 16 for Regular) that operate in parallel, with each head learning to focus on different types of relationships - syntactic, semantic, or positional. Causal masking ensures that during training, the model learns to predict the next token based solely on the preceding context, which is essential for coherent text generation.

The residual connection (+) is crucial, as it allows the model to preserve the original token representation while adding contextual information from the attention. The pre-normalization approach (LayerNorm before attention) provides more stable training than post-normalization, especially important when working with the varied linguistic patterns found in historical text.

2.3.2 Feed-Forward Step

After attention, we have the feed-forward step (x = x + self.mlp(self.ln_2(x))), which first normalizes the attended information with LayerNorm, then passes it through a multi-layer perceptron (MLP) that transforms and processes it.

The MLP typically consists of two linear layers with a non-linear activation function (like GELU) between them, allowing the model to learn complex non-linear transformations of the attended features. This step is crucial because attention can only perform linear transformations on the input representations; the feed-forward network adds the necessary non-linearity, enabling the model to learn complex patterns and relationships in the historical text. Another residual connection preserves the original information, ensuring that the model can always fall back to the pre-attention representation if needed.

2.3.3 Understanding the Feed-Forward (MLP) Sublayer

Directly beneath the SimpleBlock code above, you see the line self.mlp = SimpleMLP(config). After attention has mixed information across positions, the model passes each token embedding through a position-wise feed-forward network (the MLP). Unlike attention, it does not look at other tokens; it refines the representation of each token independently, given the contextualized features attention just produced. In practice, this is where raw contextual patterns are distilled into richer semantic, stylistic, and morphological signals.

Figure 4 below visualizes how a single transformer block routes data through normalization, attention, and the feed-forward expansion/contraction before returning an upgraded representation via the residual path:

graph TB
    A[Input Embeddings<br/>batch, seq, emb] --> LN1[LayerNorm 1]
    LN1 --> ATTN[Multi-Head Attention<br/>query, key, value]
    ATTN --> DROPA[Dropout]
    DROPA --> RES1[Residual Add<br/>x + attn_out]
    RES1 --> LN2[LayerNorm 2]
    RES1 --> LN2
    LN2 --> EXPAND[Linear Expand<br/>emb → 4*emb]
    EXPAND --> GELU[GELU Activation]
    GELU --> PROJECT[Linear Project<br/>4*emb → emb]
    PROJECT --> DROPM[Dropout]
    DROPM --> RES2[Residual Add<br/>res1 + mlp_out]
    RES2 --> OUT[Block Output<br/>Updated embeddings]
    style A fill:#e1f5fe
    style ATTN fill:#f3e5f5
    style EXPAND fill:#fff3e0
    style PROJECT fill:#fff3e0
    style RES2 fill:#e8f5e8

Figure 4: Internal Flow of a Transformer Block

Conceptually, the MLP is a two-step projection: first, an expansion into a higher-dimensional “workspace” with a non-linear activation, then a projection back down so the residual can safely merge with the original stream.

For our SLM, 768 dimensions expand to 3072 and then contract back to 768; for the larger model, 1024 dimensions expand to 4096. This temporary widening allows the network to express combinations of features that a purely linear transform could not capture. It is the difference between merely routing information and actually transforming it.

Here is the representative structure shown in Listing 3:

class SimpleMLP(torch.nn.Module):
    def __init__(self, config):
        super().__init__()
        self.fc_in  = torch.nn.Linear(config.n_embd, 4 * config.n_embd)
        self.act    = torch.nn.GELU()
        self.fc_out = torch.nn.Linear(4 * config.n_embd, config.n_embd)
        self.drop   = torch.nn.Dropout(config.dropout)
    def forward(self, x):
        return self.drop(self.fc_out(self.act(self.fc_in(x))))

Listing 3: Feed-Forward (MLP) Sublayer Implementation

Why expand then shrink?

The widened hidden space allows the model to form intermediate feature bundles (e.g., tense, register, archaic morphology) that do not cleanly live in the original lower-dimensional basis. The contraction enforces a stable interface for the residual path and keeps the total parameter count manageable. Removing the expansion would noticeably degrade expressiveness; removing the contraction would balloon memory use and break architectural symmetry.

In our context, the historical model internalizes regularities like mapping “hath” and “doth” into modern tense abstractions while still preserving period flavor; it encodes stylistic shifts between court proceedings, religious prose, and narrative storytelling; it stabilizes inconsistent orthography and variant spellings so downstream layers predict coherent continuations instead of brittle echoes. Attention tells the model where to look; the MLP decides how to reinterpret what it saw.

Focusing only on attention gives an incomplete mental model of transformers. More than half of the parameters and a large fraction of the FLOPs sit in these feed-forward layers. Under-sized MLPs lead to shallow pattern memorization - models that can repeat phrases but cannot generalize style or adapt archaic forms to new contexts. Properly scaled MLP width (the common ×4 expansion) is a proven sweet spot: smaller factors underfit; much larger ones give diminishing returns at this scale (see scaling law discussions in Kaplan et al. 2020 ).

A useful mental analogy: attention is the lively debate in a hall; the MLP is each participant stepping aside to integrate what was heard into their own refined understanding before the next round of discussion. When you see x = x + self.mlp(self.ln_2(x)), that addition represents the moment a token’s contextual representation is upgraded. Without this transformation, the model would “hear” context but fail to internalize it, producing shallow, literal continuations rather than fluent, period-authentic prose.

In our helloLondon models, the MLP is therefore essential for converting raw multi-head attention patterns into durable historical linguistic competence - one of the quiet reasons the generated text feels coherent rather than stitched together.

Each block in our model (12 for SLM, 24 for Regular) applies this same pattern, allowing the model to build an increasingly sophisticated understanding of historical language patterns as text flows through the layers.

Each transformer block applies layer normalization before both the self-attention mechanism and the feed-forward network, followed by residual connections. This pre-normalization approach (as opposed to post-normalization) has been shown to provide more stable training, especially important when working with the varied linguistic patterns found in historical text.

2.3.4 Activation choice matters

The activation function determines how the neural network processes information at each layer. Think of it as a “decision maker” that decides how much of each input signal to pass through to the next layer. The most common activation functions are ReLU (Rectified Linear Unit) and GELU (Gaussian Error Linear Unit).

ReLU is simple and fast: it passes positive values unchanged and sets negative values to zero (f(x) = max(0, x)). However, ReLU can be “harsh” - it completely cuts off negative signals, leading to “dead neurons” that never activate again. GELU is smoother and more sophisticated: it uses a Gaussian distribution to determine how much of each input to pass through (f(x) = x * Φ(x) where Φ is the cumulative distribution function of a standard normal distribution). This creates a smooth, differentiable function that allows for more nuanced information processing.

GELU offers smoother gradients and better calibration for language than plain ReLU. The smoother nature of GELU helps the model learn more subtle patterns in historical text, where the relationships between words and phrases can be complex and nuanced. Alternatives like SwiGLU can yield marginal gains in perplexity but increase implementation complexity - valuable in frontier systems, optional in educational builds like helloLondon. Modest dropout in the MLP further improves generalization on a corpus that, while sizable, is still modest relative to billion-token modern pretraining regimes.

2.3.5 Pre vs Post-normalization

In pre-normalization, we normalize the input before processing it (like we do here). In post-normalization, we’d process first, then normalize the output. Pre-normalization is like checking that your ingredients are properly prepared before cooking, while post-normalization is like seasoning after cooking - both work, but pre-normalization tends to yield more consistent results.

This matters because historical texts contain complex syntactic structures and long-range dependencies that require sophisticated attention mechanisms. The residual connections ensure that information can flow directly through the network, helping the model learn to preserve important historical context across long sequences while still allowing the attention mechanism to focus on relevant historical details.

2.4 Causal Self-Attention for Historical Sequences

The attention mechanism is crucial for understanding the complex relationships in historical text. Our implementation is based on the original transformer architecture from “Attention Is All You Need” (Vaswani et al., 2017), but optimized for historical language patterns.

Understanding Multi-Head Attention

Multi-head attention runs several attention “heads” in parallel, allowing the model to focus on different aspects of a sequence simultaneously (syntax, semantics, and position). Compared to a single head, this parallelism yields richer representations—think multiple specialists examining the same text. In our setup, the SLM uses 12 heads and the Regular model 16, scaling capacity with model size. Empirically, heads tend to specialize (e.g., subject–verb agreement, word relations, word order), as observed by Clark et al. (2019) - “What Does BERT Look At?” .

Research by Kaplan et al. (2020) Scaling Laws for Neural Language Models shows that the optimal number of attention heads scales with model size. For our 117M-parameter SLM, 12 heads provide sufficient parallel processing capacity, while our 354M-parameter Regular model benefits from 16 heads to capture more complex attention patterns.

The attention mechanism has $O(n^2)$ complexity with respect to sequence length. This means that doubling our sequence length from 512 to 1024 tokens is a quadratic jump and requires 4x more memory for attention computations. This is why we carefully balance sequence length with available GPU memory and why techniques like FlashAttention (Dao et al., 2022) are so important for memory efficiency.

How the attention mechanism works in practice:

The code in Listing 4 shows how we implement the attention mechanism that we’ve been discussing. Here’s what happens step by step:

Input Processing: The model receives a batch of sequences (B = batch size, T = sequence length, C = embedding dimension). For example, with our SLM: B=4, T=512, C=768.
Query, Key, Value Generation: The input embeddings are transformed into three different representations - Query (Q), Key (K), and Value (V) - using a single linear layer that outputs 3×768 dimensions, then splits them.
Multi-Head Reshaping: Each of Q, K, V is reshaped to separate the 12 attention heads, so each head gets its own 64-dimensional subspace (768 ÷ 12 = 64).
Attention Computation: The scaled dot-product attention is computed, where each word “looks at” all previous words (causal masking) and decides how much attention to pay to each.
Output Assembly: All attention head outputs are combined back into a single representation and projected through a final linear layer.

This implementation is optimized for historical text processing, using PyTorch’s efficient scaled_dot_product_attention function with causal masking to ensure the model can only attend to previous tokens, not future ones.

class SimpleCausalSelfAttention(torch.nn.Module):
    """Simple causal self-attention"""
    
    def __init__(self, config):
        super().__init__()
        assert config.n_embd % config.n_head == 0
        # Key, query, value projections for all heads, but in a batch
        self.c_attn = torch.nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias)
        # Output projection
        self.c_proj = torch.nn.Linear(config.n_embd, config.n_embd, bias=config.bias)
        # Regularization
        self.attn_dropout = torch.nn.Dropout(config.dropout)
        self.resid_dropout = torch.nn.Dropout(config.dropout)
        self.n_head = config.n_head
        self.n_embd = config.n_embd
        self.dropout = config.dropout
    
    def forward(self, x):
        # Batch size, sequence length, embedding dimensionality ($n_{embd}$)
        B, T, C = x.size()
        
        # Calculate query, key, values for all heads in batch 
        # and move head forward to be the batch dim
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)  # (B, nh, T, hs)
        
        # Causal self-attention; Self-attend:
        # (B, nh, T, hs) x (B, nh, hs, T) -> (B, nh, T, T)
        y = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=self.dropout if self.training else 0, is_causal=True)
        y = y.transpose(1, 2).contiguous().view(B, T, C)  # Re-assemble all head outputs side by side
        
        # Output projection
        y = self.resid_dropout(self.c_proj(y))
        return y

Listing 4: Causal Self-Attention Implementation

The attention mechanism computes attention as show below:

$$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Where $Q$, $K$, and $V$ are the query, key, and value matrices, respectively.

In our case, with 768-dimensional embeddings and 12 heads, each head operates on 64-dimensional subspaces ($d_k = 768 / 12 = 64$), providing sufficient representational capacity for each type of historical relationship while maintaining computational efficiency.

In addition, the $\sqrt{d_k}$ scaling factor ($\sqrt{64} = 8$) prevents the dot products from becoming too large, ensuring stable gradient flow during training.

In plain English, please!

Think of attention like a spotlight that can shine on different parts of a sentence. When the model is trying to understand the word “he” in a historical document, it needs to look back through the text to find who “he” refers to. The attention mechanism is like having multiple spotlights (our 12 or 16 attention heads) that can each focus on different aspects - each might look for people’s names, another for relationships, and another for locations.

The mathematical formula we showed above is how the model calculates the amount of attention to pay to each word. The scaling factor ($$\sqrt(64) = 8) is like adjusting the brightness of the spotlight – it prevents the model from being “blinded” by very bright spots and helps it focus on the right amount of information.

Does this matter for historical text?

Historical documents are particularly challenging because they often feature complex sentence structures and references spanning long distances. For example, in a court record, you might see “The defendant, John Smith, was accused of theft. He claimed innocence throughout the trial.” The model needs to understand that “He” refers to “John Smith,” even though there are several words between them. The attention mechanism enables the model to make these connections, generating coherent text that maintains proper historical context and references.

This is certainly required for language modeling, given the complex structures in which later words reference earlier ones, and understanding the full context is essential for proper interpretation. The attention mechanism enables the model to learn long-range dependencies, allowing it to generate coherent text across extended sequences. For historical texts specifically, this becomes even more important because archaic language patterns and historical references often span longer distances than those in modern texts.

2.5 Model Configuration

The model architecture uses a centralized configuration, where each parameter is selected based on research findings and practical constraints for historical text processing. The SLM architecture uses five key parameters, each representing a design choice with specific trade-offs between computational efficiency and learning capacity.

Parameter	Value	Purpose	Trade-off
`n_layer`	12	Number of transformer blocks (model depth)	More layers = better learning, but slower training
`n_head`	12	Number of attention heads (parallel processing)	More heads = better attention, but more computation
`n_embd`	768	Embedding dimension (token representation)	Larger = richer representations, but more memory
`max_length`	512	Context window size (sequence length)	Longer = more context, but quadratic memory growth
`vocab_size`	30K	Vocabulary size (tokenizer compatibility)	Larger = more words, but more parameters

These parameters work together to create a model that effectively processes historical text while remaining computationally manageable.

Layer Count (n_layer: 12)

The 12-layer architecture balances representational capacity with computational efficiency for historical text processing. Shallow layers (1-3) learn basic token patterns and grammatical structures, middle layers (4-8) capture complex syntactic relationships and historical language patterns, and deep layers (9-12) understand high-level semantic relationships and historical context.

This depth follows GPT-2 Small’s 12-layer architecture, which delivers strong performance while remaining computationally manageable on available hardware.

Attention Heads (n_head: 12)

Multi-head attention allows the model to attend to different types of relationships simultaneously – for example, temporal (chronological order), social (class hierarchies), geographical (London landmarks), and linguistic (archaic patterns). The 12-head architecture balances parallel processing capability with computational efficiency for historical text understanding.

Embedding Dimension (n_embd: 768)

The 768-dimensional embedding space can represent complex historical concepts, such as archaic terms (“yeoman”, “paternoster row”, “hath”), while maintaining computational efficiency. This dimension is commonly used in transformer architectures, including BERT-base and GPT-2 Medium.

Why 768 became standard: As a side note, in case you are seeing a lot of 768 lately, there are a good set of reasons for this. Beyond its divisibility ($768 ÷ 12 = 64$ per attention head), 768 aligns with GPU memory architecture - it’s a multiple of 256 (3 × 256), which matches common GPU memory bus widths and cache line sizes. This makes matrix operations more efficient on modern GPUs, as the hardware can process data in optimal chunks. Additionally, 768 provides sufficient representational capacity without the memory overhead of larger dimensions like 1024, making it practical for training on consumer hardware while still capturing complex linguistic relationships.

Context Window (n_positions: 512)

We use a 512-token context window as a practical balance between historical coherence and available compute for a learning-focused setup. While many of our working snippets (e.g., diary passages, sections of legal records, or literary excerpts) comfortably fit within 512 tokens, full historical documents can be much longer. The 512 window keeps attention costs manageable (quadratic in sequence length) while covering typical training segments we use.

Both models use the same 30K vocabulary from our custom historical tokenizer, ensuring consistent tokenization across model variants.

3. GPU Configuration and Perf. Optimization

The training system is designed to maximize GPU utilization while maintaining training stability. Understanding GPU architecture and memory management is crucial for efficient language model training, especially when working with historical text that requires significant computational resources.

3.1 GPU Architecture and Memory Management for Language Model Training

Training on historical text benefits from sensible GPU settings even for a small, learning-focused model. We keep to practical, low-risk optimizations (precision choice, batch/sequence trade-offs, memory-aware attention) and accept some trial and error—reserving heavier systems engineering for larger setups.

The main universal factors are:

Attention scales quadratically with sequence length, so longer contexts get expensive fast.
Natural language variability (syntax, vocabulary, style) demands sufficient model capacity and stable optimization.
Real‑world data quality (formatting, noise) can destabilize training, requiring robust error handling and memory management.

For historical text specifically, archaic vocabulary, period terminology, and cultural references introduce patterns absent from modern corpora. OCR artifacts and uneven formatting in digitized sources add noise beyond what’s typical in contemporary datasets.

3.1.1 GPU memory hierarchy and optimization strategies

Modern GPUs use a hierarchical memory system that significantly impacts training performance: fast but tiny registers and shared memory sit closest to the compute; a larger L2 cache buffers traffic; and global memory holds parameters and activations. Attention often ends up memory-bound, so moving less data (via AMP, Flash/SDPA kernels, and sensible sequence/batch sizes) is as important as raw FLOPs.

For language model training, the key optimization is managing the memory bandwidth bottleneck. Attention operations are often memory-bound rather than compute-bound, meaning performance is limited by how quickly data can be moved between memory levels rather than by computational power. If we are not careful, it is quite easy to run into memory issues, as shown in Figure 5 below.

Figure 5:OOM error

And it is not restricted to training only; even on checkpoints that are saved, we can also encounter memory issues, as shown in Figure 6.

Figure 6:OOM checkpoint eval

Mixed precision training and memory optimization

Training large language models requires careful memory management, especially when working with limited GPU resources. Our training system uses several optimization techniques to maximize memory efficiency while maintaining training stability.

GPU detection and basic configuration:

The training system needs to work across different hardware setups, from single consumer GPUs to multi-GPU servers. Our approach uses a centralized configuration system that automatically adapts to available hardware.

The actual GPU detection in train_model_slm.py is quite straightforward - it checks for distributed training environment variables (RANK, LOCAL_RANK, WORLD_SIZE) and sets up basic multi-GPU support if available. The system also detects GPU capabilities, such as bfloat16 support, and enables appropriate optimizations. This allows the same training script to work across different hardware setups, though the real complexity comes from the trial-and-error process of stabilizing training.

# GPU configuration (from config.py)
self.gpu_config = {
    "auto_detect": True,  # Automatically detect available GPUs
    "max_gpus": 0,  # Maximum number of GPUs to use (0 = no limit, use all available)
    "min_gpu_memory_gb": 8,  # Minimum GPU memory required (GB)
    "preferred_gpu_types": ["A30", "A40", "A100", "V100", "RTX4090", "RTX4080"],
    "fallback_to_cpu": True,  # Fall back to CPU if no suitable GPUs found
    "force_single_gpu": False,  # Force single GPU even if multiple available
    "force_multi_gpu": False,  # Force multi-GPU even if only one available
    "gpu_memory_fraction": 0.9,  # Fraction of GPU memory to use (0.0-1.0)
    "allow_growth": True,  # Allow GPU memory growth
    "log_device_placement": False  # Log device placement for debugging
}

Listing 5: GPU Configuration and Detection System

The configuration in Listing 5 is defined in our centralized config.py file and provides settings for automatic GPU detection, memory management, and fallback options. While this looks comprehensive, the actual implementation is simpler - the training code primarily relies on PyTorch’s built-in distributed training detection and basic device selection.

The reality of training: Nearly 100 runs and many failures

Figure 7: helloLondon training runs - WandB

Figure 7 shows the actual training experience: 99 total runs with 24 completions. The failures were largely data-driven - OCR and encoding issues, uneven sequence lengths, and sensitivity to learning-rate warmup - and a few were plain memory pressure from early, less conservative settings. The code stabilized early; the data and knobs took time.

This iterative process is typical in language model development - the “sophisticated” system shown here is the result of learning from these failures and gradually improving the training pipeline. The successful runs exhibit stable loss curves and appropriate learning rate schedules, demonstrating that the final configuration performs well on historical text processing tasks.

Most importantly, this experience reinforces a fundamental truth in machine learning: data quality is still king. No amount of sophisticated architecture, GPU optimization, or training infrastructure can overcome poor data quality. The “garbage in, garbage out” principle remains as true for language models as it was for the earliest machine learning systems. Our 75% failure rate was primarily due to data issues – such as inconsistent formatting, OCR errors, and encoding problems - not technical limitations. This is why Part 2’s focus on data cleaning and tokenization was so crucial to our success.

3.2 Precision and Performance Configuration

The system includes precision and performance configuration options that can be tuned based on available hardware. Mixed-precision training uses lower-precision (fp16/bf16) for most operations while keeping full precision for critical computations, providing significant memory savings and speed improvements with minimal impact on quality.

Understanding fp16 and bf16: The Precision Trade-off

To understand why precision matters for language model training, we need to look at how computers represent numbers. Standard floating-point numbers use 32 bits (float32), but we can use fewer bits to save memory and increase speed:

fp16 (Half Precision): Uses 16 bits to represent numbers, cutting memory usage in half and enabling faster computation. However, it has a smaller range of representable numbers, which can cause “overflow” (numbers too large) or “underflow” (numbers too small) during training.
bf16 (Brain Float 16): Also uses 16 bits, but with a different bit layout that matches float32’s exponent range. This means it can represent the same range of large and small numbers as float32, but with less precision for very small decimal values.

Why bf16 is better for language models:

bf16 provides better numerical stability than fp16, especially for large language models, reducing the likelihood of overflow and underflow that can cause training instability. The key difference is that bf16 can represent the same range of numbers as float32 (from very small to very large), while fp16 has a much smaller range. This is crucial for language models because:

Gradient magnitudes vary widely - Some gradients are very small (close to zero) while others are large
Attention weights - The softmax operations in attention can produce very small numbers that FP16 might round to zero
Learning rate scaling - Modern optimizers like AdamW work with gradients of varying magnitudes

When gradients become too small and are rounded to zero (underflow), the model stops learning effectively. When they become too large (overflow), training becomes unstable. bf16’s wider range helps prevent both issues.

Understanding precision and performance settings:

The configuration in Listing 6 toggles the levers that matter on consumer hardware: TF32 for faster matmuls, AMP (prefer bf16) for stability and memory cuts, torch.compile for an extra boost after warmup, and sequence/batch sizes sized to your VRAM. Used together, these commonly halve activation memory and yield 2-3x speedups versus full-precision baselines.

# Runtime/precision knobs (A30 optimized)
"enable_tf32": True,
"enable_amp": True,
"amp_dtype": "bf16",  # bf16 on Ampere; fallback to fp16 if unsupported
"enable_compile": True,  # torch.compile; set False to reduce memory usage
# Conservative baseline (for broad hardware) - uncomment to use:
# "enable_tf32": False,
# "enable_amp": True,
# "amp_dtype": "fp16",
# Sequence/batch control
"max_length": 1024,  # increase tokens per step when VRAM allows
"batch_size": 20,    # per-GPU batch; raise if VRAM allows
# Conservative sequence/batch - uncomment to use:
# "max_length": 768,
# "batch_size": 8,

Listing 6: Precision and Performance Configuration

Key GPU Configuration Settings

A few switches move the needle the most: enable TF32 on Ampere-class GPUs for a quick matrix-mul speedup; use AMP (bf16 where supported, fp16 otherwise) to halve activation memory; and turn on torch.compile if you can afford the warmup to get another 1.2-1.5x after a few hundred steps. Keep the sequence length in line with VRAM (~512 tokens for 8GB, ~1024 for 16GB+), and scale the per-GPU batch size accordingly (think hundreds of MB per batch at these widths). The repo includes sensible presets so you can start conservative and dial up.

3.2.1 Real-World Performance Results

On 2x A30s, the SLM lands around mid-20s MFU with ~210 ms/iter and ~18 GB per GPU, converging from ~10.4 loss to the mid-3s over the full run. The clean BPE tokenizer and precision stack keep math efficient, and DDP delivers the expected speedup over a single device.

Automatic Precision Detection and Memory Optimization:

The system also includes automatic precision detection and memory optimization during model initialization. The code snippet below shows how the system automatically selects the optimal precision format based on available hardware capabilities:

# Precision / TF32 knobs from config
tf32 = self.slm_config.get("enable_tf32", True)
torch.backends.cuda.matmul.allow_tf32 = bool(tf32)
torch.backends.cudnn.allow_tf32 = bool(tf32)
try:
    torch.set_float32_matmul_precision('high' if tf32 else 'medium')
except Exception:
    pass
use_amp = self.slm_config.get("enable_amp", True)
amp_dtype_cfg = self.slm_config.get("amp_dtype", "bf16").lower()
bf16_ok = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
if use_amp:
    if amp_dtype_cfg == 'bf16' and bf16_ok:
        self.dtype = 'bfloat16'
    else:
        self.dtype = 'float16'
else:
    self.dtype = 'float32'

Listing 7: Precision Detection and Memory Optimization

The TF32 configuration optimizes matrix operations for Ampere+ GPUs, delivering significant speedups while maintaining training stability.

3.3 Multi-GPU Training with Distributed Data Parallel

The system supports multi-GPU training using PyTorch’s DistributedDataParallel (DDP) - each GPU hosts a full model replica, processes different batches in parallel, and synchronizes gradients automatically. PyTorch handles the inter‑GPU communication, so on two GPUs, you typically see near‑linear speedup (~2×) for these model sizes.

Multi-GPU training improves throughput and shortens wall‑clock time by splitting work across devices. On our 2× A30 setup, we process 36 sequences in parallel (18 per GPU) instead of 18 on a single card, cutting Regular model training from ~56 hours to ~28–32 hours. It also offers operational flexibility: scale up or down based on the number of GPUs available.

However, multi-GPU training introduces several challenges that can limit performance gains. The primary bottleneck is inter-GPU communication - after each backward pass, gradients must be synchronized across all GPUs, which requires transferring large amounts of data. This communication overhead can become significant, especially with larger models and more GPUs.

The performance of multi-GPU training heavily depends on the interconnect between GPUs. On NVIDIA systems, InfiniBand provides the highest bandwidth and lowest latency for GPU-to-GPU communication, enabling near-linear scaling across many GPUs. NVLink (found on high-end NVIDIA GPUs such as A100 and H100) provides direct GPU-to-GPU connections with very high bandwidth, making it ideal for 2-8 GPU setups. PCIe connections are slower but more common in consumer and workstation systems.

In AMD systems, Infinity Fabric serves a role similar to NVLink, providing high-bandwidth interconnects between GPUs. AMD’s MI200 and MI300 series GPUs include Infinity Fabric links that enable efficient multi-GPU communication.

In practice, scaling efficiency depends on the ratio of computation to communication. Our historical language models have relatively modest parameter counts (117M-354M), so communication overhead can be significant compared to computation time. This is why we see good scaling with 2 GPUs but diminishing returns with more GPUs - the communication overhead starts to dominate.

DDP is more efficient than naive data parallelism because it reduces communication overhead and enables larger effective batch sizes, as shown in Listing 8 below.

# DDP setup (process group already initialized in main())
self.ddp = int(os.environ.get('RANK', -1)) != -1
if self.ddp:
    self.ddp_rank = int(os.environ['RANK'])
    self.ddp_local_rank = int(os.environ['LOCAL_RANK'])
    self.ddp_world_size = int(os.environ['WORLD_SIZE'])
    self.device = f'cuda:{self.ddp_local_rank}'
    torch.cuda.set_device(self.device)
    self.master_process = self.ddp_rank == 0
    self.seed_offset = self.ddp_rank
else:
    self.master_process = True
    self.seed_offset = 0
    self.ddp_world_size = 1

Listing 8: Multi-GPU Training Setup

What is “rank” and why does it matter?

In distributed training, each GPU process gets a unique “rank.” Rank 0 acts as the coordinator (handles logging, checkpointing, and WandB), while the remaining ranks focus purely on computation. This avoids collisions - only one process touches files and dashboards - while every device contributes gradients.

This division of labor is crucial because it prevents conflicts. Without it, all processes would try to save checkpoints simultaneously, log to WandB at the same time, or write to the same files, causing errors and corruption.

The key to scaling efficiency is that each GPU works independently on different data batches, then synchronizes only the essential information (gradients). Here’s how it works:

Parallel computation: Each GPU processes a different batch of data simultaneously
Gradient synchronization: After each backward pass, gradients are averaged across all GPUs
Independent updates: Each GPU updates its model copy with the averaged gradients

This means that if you have 2 GPUs, you can process 2x the data in the same time, giving you roughly 2x the speed. With 4 GPUs, you get approximately 4x speedup. The “near-linear” part acknowledges that there’s always some overhead from communication and synchronization, so that you might get 1.9x speedup instead of exactly 2x. Still, it’s close enough to be very effective.

However, there’s a practical limit to this approach. Beyond 8-16 GPUs, the communication overhead becomes so significant that you need more robust hardware (such as InfiniBand networks) and advanced systems engineering techniques (gradient compression, pipeline parallelism, model parallelism) to maintain efficiency. For truly large-scale training with hundreds of GPUs, you need specialized infrastructure and techniques that go far beyond what we’re doing here.

This combination of distributed training and memory optimization enables us to train our historical language models efficiently, even on consumer hardware. The distributed setup provides fault tolerance and near-linear speedup, while the precision optimizations enable larger models and longer sequences on the same hardware.

4. Training Infrastructure: Making It All Work Together

As a reminder, as we saw earlier, the two model variants share the same training stack (scheduler, checkpointing, WandB, DDP). See Part 1 for the high‑level comparison; here are the training‑relevant differences only:

SLM (117M): per‑GPU batch 18 → effective 36 on 2 GPUs; sequence length 512; ~7–8h on 2×A30
Regular (354M): per‑GPU batch 12 → effective 24 on 2 GPUs; sequence length 1024; ~28–32h on 2×A30

4.1 The Training Loop

The core training happens in the train() method, which implements a standard language model training loop with several key phases - outlined below.

4.1.1 Data Loading and Preparation

The training loop starts by loading tokenized data using get_batch('train'), which reads from pre-processed binary files created during data preparation. This includes both training and validation data, with the tokenizer from Part 2: Data Collection & Custom Tokenizers handling the conversion between text and tokens.

Main Training Loop Structure:

def train(self):
    """Main training loop"""
    # Get initial batch
    X, Y = self.get_batch('train')
    
    while True:
        # 1. Learning rate scheduling
        lr = self.get_lr(iter_num)
        
        # 2. Evaluation and checkpointing (every eval_interval steps)
        if iter_num % self.eval_interval == 0:
            losses = self.estimate_loss()
            # Save checkpoint if validation loss improved
            
        # 3. Forward pass with mixed precision
        with torch.amp.autocast():
            logits, loss = self.model(X, Y)
        
        # 4. Backward pass and optimization
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        optimizer.zero_grad()
        
        # 5. Get next batch
        X, Y = self.get_batch('train')
        
        # 6. Logging and monitoring
        if iter_num % self.log_interval == 0:
            # Log to WandB and console

Listing 9: Core Training Loop Structure

Training Process Flow:

Understanding how the training actually works requires seeing both the high-level flow and the technical details of each phase. Figure 8 shows the complete training process flow.

graph TD
    A[Start Training] --> B[Load Tokenized Data]
    B --> C[Initialize Model & Optimizer]
    C --> D[Training Loop Start]
    D --> E[Update Learning Rate]
    E --> F{Evaluation Time?}
    F -->|Yes| G[Run Validation]
    F -->|No| H[Forward Pass]
    G --> H
    H --> I[Compute Loss]
    I --> J[Backward Pass]
    J --> K[Gradient Clipping]
    K --> L[Update Weights]
    L --> M[Zero Gradients]
    M --> N[Log Metrics]
    N --> O{Checkpoint?}
    O -->|Yes| P[Save Model State]
    O -->|No| Q[Load Next Batch]
    P --> Q
    Q --> R{Max Iterations?}
    R -->|No| D
    R -->|Yes| S[Save Final Model]
    S --> T[End Training]

Figure 8: Training Process Flow

Now that we have a high-level overview of the training process, let us dig deeper into each phase and see how it works in practice.

4.1.2 Data Loading

Data loading reads pre-tokenized sequences from binary files (.bin) using np.memmap for memory efficiency. The initial tokenization process can take quite a long time on our 500M+ character corpus, but this is done only once and saved to disk. This optimization was crucial during our development process – given nearly 100 training runs and many failures, re-tokenizing the entire corpus each time would have been prohibitively slow. The system handles train/val splits (90/10 %) with random sampling per batch and uses pin_memory() and non_blocking=True for faster GPU transfers.

When we run this for the time time, it takes a long time to load and tokenize the training data corpus. We see this just startging in Figure 9 below.

Figure 9:Tokenizer training data

Batch sizes are optimized for our 2x A30 GPU setup: 18 per GPU for the SLM model (36 effective batch size) and 12 per GPU for the Regular model (24 effective batch size). These numbers balance memory usage with training stability – the SLM can handle larger batches thanks to its smaller 117M parameter count. In comparison, the Regular model’s 354M parameters require smaller batches to fit in GPU memory.

Figure 10 below shows the dual GPU setup used for one of the training sessions for the regular mode.

Figure 10:GPU detail

4.1.3 Learning Rate Scheduling

Learning Rate Scheduling uses cosine decay with warmup, a two-phase approach that helps prevent training instability. The warmup phase gradually increases the learning rate from 0 to the target value over 500 steps (SLM) or 1000 steps (Regular model), preventing the model from making large, destabilizing updates early in training.

After warmup, cosine decay smoothly reduces the learning rate following a cosine curve to 10% of the initial rate by the end of training. In case you are not familiar with Cosine decay, it is a scheduling strategy where the learning rate follows the shape of a cosine wave: starting at the maximum value after warmup, it decreases slowly at first, then more rapidly in the middle of training, and finally levels off gently near the minimum value.

Mathematically, this follows the curve lr = min_lr + (max_lr - min_lr) × 0.5 × (1 + cos(π × progress)), where progress goes from 0 (start of decay) to 1 (end of training). Unlike linear decay (which drops at a constant rate) or step decay (which drops abruptly at fixed intervals), cosine decay provides a smooth, natural reduction that helps the model explore the loss landscape more effectively early on, then refine its parameters more precisely as training progresses.

The initial learning rates are chosen based on model size: 3e-4 (0.0003) for the SLM model and 3e-5 (0.00003) for the Regular model. The 10x difference reflects the Regular model’s larger parameter count (354M vs 117M) - larger models typically need smaller learning rates to prevent gradient explosion. The cosine decay ensures the model converges smoothly rather than oscillating around the minimum, which is crucial for the complex patterns in historical text.

def get_lr(self, it):
    """Learning rate schedule"""
    warmup_iters = 500
    lr_decay_iters = self.max_iters
    min_lr = self.learning_rate * 0.1
    
    if it < warmup_iters:
        return self.learning_rate * (it + 1) / (warmup_iters + 1)
    if it > lr_decay_iters:
        return min_lr
    decay_ratio = (it - warmup_iters) / (lr_decay_iters - warmup_iters)
    assert 0 <= decay_ratio <= 1
    coeff = 0.5 * (1.0 + math.cos(math.pi * decay_ratio))
    return min_lr + coeff * (self.learning_rate - min_lr)

Listing 10: Learning Rate Scheduling Function

The code in Listing 10 shows how the learning rate schedule is implemented. The function takes the current iteration number and returns the appropriate learning rate based on whether we’re in the warmup phase (linear increase) or decay phase (cosine curve). The warmup_iters parameter controls the warmup duration, while min_lr sets the final learning rate to 10% of the initial value.

As a side note, in case you are curious about why a cosine decay specifically makes sense, then read on. The cosine function has unique mathematical properties that make it ideal for learning rate scheduling. Unlike linear decay (which drops too quickly) or exponential decay (which drops too slowly), cosine decay starts with a gentle slope that gradually steepens, then flattens out near the end. This creates a “restart” effect, allowing the model to escape local minima early in training and then fine-tune more precisely in later stages.

The smooth, continuous nature of cosine decay prevents the learning rate from changing too abruptly, which could destabilize training. Given the historical text’s complex linguistic patterns, this gradual, adaptive approach helps the model learn both general language structures and specific historical vocabulary without getting stuck in suboptimal solutions.

4.1.4 Evaluation

We run evaluations at a regular interval after a certain number of steps, which are defined in the eval_interval() method (defaults to 500 for SLM, 1000 for Regular) and compute loss on both train and validation sets using the estimate_loss() method. The different intervals reflect the models’ training complexity: the SLM trains faster and benefits from more frequent checks, while the Regular model’s longer runs can use less frequent evaluation.

The estimate_loss() function monitors training progress without disrupting the learning process. To ensure consistent measurements, it temporarily switches the model to evaluation mode (model.eval()). In this mode, dropout layers stop randomly dropping neurons (using the full network capacity), and batch normalization uses running statistics rather than recomputing them from each batch. This means the same input produces the same output every time, unlike in training mode, where dropout introduces randomness for regularization.

Rather than computing loss on the entire dataset (which would be too slow), estimate_loss() samples eval_iters random batches (default 100) from both training and validation sets. It computes the loss for each batch and returns the average, providing a representative estimate of model performance while remaining computationally efficient.

The evaluation process uses torch.no_grad() to disable gradient computation during validation. Gradients are the partial derivatives that tell us how to adjust each model parameter to reduce loss - they’re computed during the backward pass and stored for the optimizer. During evaluation, we don’t need gradients because we’re not updating weights; we’re just measuring performance.

Disabling gradient computation serves two critical purposes. First, it prevents memory leaks by not storing gradients for validation computations - without this, GPU memory would gradually increase during evaluation and eventually cause out-of-memory errors. Second, it ensures accurate loss measurement by preventing any accidental gradient updates during the evaluation phase. The no_grad() context manager is essential for maintaining training stability and memory efficiency.

4.1.5 Forward Pass

A forward pass is when the model processes input data through its layers to produce a prediction. Think of it like asking the model a question: given a sequence of historical text tokens, “what word should come next?” The model flows the input forward through 12 transformer blocks (SLM) or 24 blocks (Regular), each applying self-attention (to understand relationships between words) and feed-forward operations (to transform and refine the representations). At the end, the model outputs a probability distribution over all possible next tokens.

The forward pass uses mixed precision training with torch.amp.autocast and bf16/fp16 data types, reducing memory usage by ~50% while maintaining training stability. Cross-entropy loss is computed by comparing the model’s predicted probabilities with the actual next tokens in the training data; it measures how “wrong” the model’s predictions are. The loss function handles variable sequence lengths by appropriately padding sequences. The mixed precision approach is particularly important for our historical text corpus, which contains long sequences that would otherwise exceed GPU memory limits.

4.1.6 Backward Pass

After the forward pass tells us how wrong the model is (via the loss), the backward pass figures out how to fix it. Using loss.backward(), PyTorch computes gradients for every parameter in the model - these gradients tell us the direction and magnitude of changes needed to reduce the loss. It’s like having a GPS telling you which way to move and how far, but for 117 million (SLM) or 354 million (Regular) parameters simultaneously.

The system applies gradient clipping with torch.nn.utils.clip_grad_norm_ using a maximum norm of 1.0. Sometimes gradients can become extremely large, especially when processing complex or unusual historical text patterns. Without clipping, these huge gradients would cause the model parameters to jump wildly, potentially making the model unstable or causing it to “forget” what it learned. Clipping acts like a safety valve, limiting the maximum size of parameter updates to keep training stable. Put simply, gradient clipping caps the overall (global) gradient norm at a threshold; if it exceeds the limit, gradients are rescaled so the update stays bounded. In our early runs, omitting clipping occasionally produced NaN losses; keeping max_norm=1.0 eliminated those spikes.

After computing gradients, the system updates the model weights using the AdamW optimizer, which applies the gradients with momentum and adaptive learning rates for each parameter. The optimizer decouples weight decay (a regularization technique to prevent overfitting) from gradient updates, improving generalization. Finally, gradients are zeroed with optimizer.zero_grad(set_to_none=True) - this clears the gradient buffers before the next iteration, preventing them from accumulating across batches. The set_to_none=True option releases memory immediately rather than waiting for GC, improving memory efficiency.

4.1.7 Checkpointing

Checkpointing saves model state, optimizer state, iteration number, and best validation loss whenever validation performance improves, rather than at every evaluation. This selective saving strategy provides multiple benefits: it conserves disk space (our 354M-parameter model checkpoints are ~1.4GB each), reduces I/O overhead that can slow down training, and improves overall training time by 5-10% by eliminating redundant disk writes. The system maintains only the last 5 checkpoints (as configured in config.py), with PyTorch’s torch.save() using compression to ensure efficient storage while preserving all necessary training state for resuming. We’ll dive deeper into checkpointing strategies and implementation details in Section 6.

The training loop implements standard optimization practices, including dynamic learning rate scheduling, regular evaluation and checkpointing, and comprehensive logging to WandB (as detailed in Section 5). The system automatically saves checkpoints when validation loss improves, ensuring that the best model is always preserved. The learning rate schedule uses cosine decay with warmup, which is standard practice for transformer training.

📁 Full Implementation:

SLM: 04_training/train_model_slm.py

Regular Model: 04_training/train_model.py

4.2 Model Initialization: Setting Up the Training Foundation

Before the training loop can begin, the system must properly initialize the model, optimizer, and training infrastructure. The init_model() method handles this setup, ensuring everything is configured correctly for efficient training.

4.2.1 Model Configuration and Creation

The initialization process starts by loading metadata from the tokenized data to ensure the model architecture matches the training data. The system reads vocabulary size, block size, and other parameters from the meta.pkl file created during data preparation, ensuring consistency between the model and the data it will be trained on.

The model configuration is built from the SLM parameters defined in config.py, including the number of layers (12), attention heads (12), embedding dimensions (768), and other architectural choices. This configuration is then used to create the SimpleGPT model instance, which inherits from PyTorch’s nn.Module and provides all the functionality we discussed in the architecture section.

4.2.2 Optimizer Setup and Configuration:

The optimizer is the algorithm that actually updates the model’s parameters (weights and biases) during training. After the backward pass computes gradients (which tell us how to adjust each parameter), the optimizer applies those gradients to update the parameters and improve the model.

The system uses AdamW (Adam with Weight Decay), which is a popular optimizer for training transformers. AdamW combines the best of two approaches: Adam (which adapts the learning rate for each parameter individually, helping with convergence) and weight decay (a form of regularization that prevents overfitting by discouraging large parameter values).

However, not all parameters should be regularized the same way. The optimizer splits parameters into two groups for different weight decay:

2D parameters (weight matrices): These are the main “learnable” parts of the model - the connections between neurons in different layers. These receive weight decay (value 0.1) to prevent them from growing too large, which helps prevent overfitting.
1D parameters (biases): These are additive constants that help shift the model’s predictions. They don’t receive weight decay (value 0.0) because regularizing biases doesn’t help with overfitting and can actually hurt performance.

This two-group approach follows standard practices for transformer training and ensures the model generalizes well to unseen historical text.

Modern PyTorch supports “fused” optimizer operations, which combine multiple steps into a single, faster GPU kernel. Instead of executing separate operations (unscale gradients, update parameters, update optimizer state), fused AdamW performs all three in a single optimized GPU operation. This can provide 10-20% speedup on modern GPUs. The system automatically detects whether your PyTorch version supports fused operations and uses them when available, falling back to the standard implementation otherwise.

Concretely, we use AdamW with the following settings for this project: betas=(0.9, 0.95), weight_decay=0.1, and the learning rate provided by the scheduler (warmup + cosine decay). The AdamW eps parameter is left at the PyTorch default unless you change it in code. When available, the fused AdamW kernel is enabled automatically. See Listing 11 for the exact call in init_model().

4.2.3 Model Compilation and Multi-GPU Setup

Model compilation with PyTorch’s torch.compile is similar to traditional code compilation, but with important differences. When you compile Python code (like using gcc for C), the compiler transforms the source code into optimized machine code once, which then runs faster. Similarly, torch.compile takes your model’s computation graph and optimizes it, but it does this at runtime rather than ahead of time.

The compilation process analyzes your model’s operations (matrix multiplications, attention layers, etc.) and generates optimized kernels tuned to your hardware. This includes operator fusion (combining multiple operations into single GPU kernels), memory layout optimization (arranging data for better cache usage), and kernel selection (choosing the fastest implementation for your specific GPU). The result is often 1.2-1.5x speedier training, but with an initial “warmup” cost: the first few forward/backward passes are slower while PyTorch analyzes the model and generates optimized code.

This differs from traditional compilation because the optimization happens dynamically based on actual input shapes and hardware capabilities, rather than being pre-computed. It’s more like a JIT compiler that specializes your model’s operations for the exact conditions it encounters during training.

For multi-GPU training, the model is wrapped with DistributedDataParallel (DDP), which enables parallel training across multiple GPUs. The DDP wrapper handles gradient synchronization and ensures that all GPUs work with identical model parameters throughout training.

def init_model(self):
    """Initialize the model"""
    logger.info("Initializing model...")
    
    # Load metadata from tokenized data
    meta_path = self.data_dir / "meta.pkl"
    meta_vocab_size = None
    if meta_path.exists():
        with open(meta_path, 'rb') as f:
            meta = pickle.load(f)
        meta_vocab_size = meta['vocab_size']
        logger.info(f"Found vocab_size = {meta_vocab_size}")
    
    # Create model configuration
    model_args = dict(
        n_layer=self.n_layer,        # 12 for SLM
        n_head=self.n_head,          # 12 for SLM  
        n_embd=self.n_embd,          # 768 for SLM
        block_size=self.block_size,  # 512 for SLM
        bias=self.bias,              # False
        vocab_size=meta_vocab_size,  # From tokenized data
        dropout=self.dropout         # 0.1
    )
    
    # Create and configure model
    gptconf = SimpleGPTConfig(**model_args)
    self.model = SimpleGPT(gptconf)
    self.model.to(self.device)
    
    # Initialize optimizer with proper parameter groups
    self.optimizer = self.model.configure_optimizers(
        weight_decay=0.1,
        learning_rate=self.learning_rate,
        betas=(0.9, 0.95),
        device_type='cuda' if 'cuda' in self.device else 'cpu'
    )
    
    # Compile model for performance
    if torch.cuda.is_available() and self.slm_config.get("enable_compile", True):
        logger.info("Compiling model...")
        self.model = torch.compile(self.model, mode='reduce-overhead')
    
    # Wrap with DDP for multi-GPU training
    if self.ddp:
        self.model = DDP(self.model, device_ids=[self.ddp_local_rank])
        param_count = self.model.module.get_num_params()
    else:
        param_count = self.model.get_num_params()
    
    logger.info(f"Model initialized with {param_count:,} parameters")

Listing 11: Model Initialization Process

While our model is a relatively simple toy example focused on a single domain (historical London text), proper initialization remains important to avoid common training issues. The vocabulary size must match our custom historical tokenizer, the sequence length needs to work with our tokenized data, and the model architecture should be appropriate for the text patterns we’re learning.

The initialization process ensures these basic requirements are met before training begins, preventing issues such as vocabulary mismatches or memory allocation problems that could lead to training failures. This careful setup was helpful during our development process, where we ran nearly 100 training experiments. Proper initialization helped us avoid some basic configuration errors and focus on the actual training challenges.

Reproducibility and random seeds: To make runs repeatable on the same hardware, we set a deterministic seed per process using torch.manual_seed(1337 + seed_offset), where seed_offset is the DDP rank (0 for single‑GPU). This gives consistent data shuffling and initialization across restarts while keeping each process distinct under DDP. Note that some CUDA kernels (and AMP/bf16) can introduce non‑determinism; for strict determinism, you may also configure PyTorch’s deterministic flags at the cost of performance.

5. WandB Integration

Weights & Biases (WandB) is an experiment tracking and monitoring platform designed specifically for machine learning projects. Think of it as a “black box” for your training runs - it automatically records everything that happens during training so you can understand what worked, what didn’t, and why.

Training a language model is a long‑running experiment. Without live telemetry, we are flying blind and can’t tell whether learning is stable, whether hardware is saturated, or whether runs are comparable. WandB gives real‑time visibility, remote monitoring, and reproducibility. It records loss, learning rate, and perplexity over time; captures GPU utilization and iteration latency; logs configuration and artifacts; and lets you compare runs side‑by‑side to understand which settings worked.

The system includes WandB integration for experiment tracking and monitoring, with automatic configuration logging, real-time metric tracking (including loss, perplexity, and learning rate), model checkpoint integration, experiment comparison across different training runs, and resource monitoring (GPU utilization and memory usage). This integration helps track and compare different training runs, identify better configurations, and reproduce successful experiments.

Understanding WandB integration:

Listing 12 logs the signals you need to fly by instruments: loss and perplexity trends for learning, the LR schedule to confirm warmup/decay, and hardware utilization and iteration timing for throughput and stability. It’s not just logging - it’s how you compare runs and catch issues early.

This real-time monitoring lets us spot problems early, compare different training approaches, and ensure our historical language model is learning properly over days or weeks.

# Log to WandB - loss first for better mobile UI
if self.use_wandb:
    wandb.log({
        "train/loss": lossf,
        "train/lr": lr,
        "train/iter": iter_num,
        "train/mfu": running_mfu * 100 if running_mfu > 0 else 0,
        "train/dt_ms": dt * 1000,
    })

Listing 12: WandB Integration and Logging

The system logs training loss, learning rate, iteration number, model flops utilization (MFU), and training time per iteration. These metrics provide comprehensive insight into training progress, efficiency, and potential issues.

The most useful dials to watch are training loss (should steadily trend from ~8–10 toward ~2–4), MFU (a proxy for GPU efficiency - single‑digit theoretical targets but mid‑20s achievable with good tuning), the learning‑rate curve (warmup then cosine decay), and iteration time (a practical signal for throughput and stalls).

Both SLM and Regular model training runs complete 60,000 iterations, providing consistent training depth across both model variants. Figure 11 below shows the complete training experience for our Regular model (354M parameters), demonstrating both the console output and WandB’s comprehensive monitoring capabilities.

Figure 11:Complete training run output showing console logs and WandB monitoring for Regular model (354M parameters)

Whilst it might be obvious, the screenshot in Figure 11 captures the final moments of a successful 60,000-iteration training run, showing both the real-time console output and WandB’s comprehensive run summary. In this run, the logs reveal the training progression through the final iterations (59,850 to 60,000), with training loss steadily decreasing from 3.0575 to 2.7063, demonstrating healthy convergence.

The WandB run summary provides the complete picture: a final training loss of 2.70315, a validation loss of 3.61921, and a validation perplexity of 37.31, all indicating successful model training. The system automatically saved the final checkpoint and cleaned up old checkpoints, while WandB captured the entire training journey with detailed metrics tracking. This comprehensive monitoring approach ensures we can both track progress in real time and analyze the full training history afterward.

6. Checkpointing and Model Persistence

Checkpointing is one of the most critical aspects of training large language models, especially for historical text, where training can take days or weeks. A robust checkpointing system ensures that training progress is never lost due to hardware failures, power outages, or other interruptions. In this section, we’ll explore the comprehensive checkpointing system built for the helloLondon project, covering everything from basic checkpoint creation to advanced resume functionality.

6.1 Checkpoint System

The training system implements a practical checkpointing system that preserves all aspects of training state, ensuring that training can be resumed from exactly where it left off. This is particularly important for any complex model, where training can take a long time.

Each checkpoint packages four essentials: the model weights (so learning is preserved), the optimizer state (so momentum and adaptive stats resume cleanly), the current iteration (so schedules pick up in the right place), and the best validation loss to date (so we only promote genuinely better models). Together, these let you stop and restart without losing training dynamics.

The code in Listing 13 shows how these components are saved when validation loss improves:

if losses['val'] < best_val_loss:
    best_val_loss = losses['val']
    if iter_num > 0:
        checkpoint = {
            'model': raw_model.state_dict(),
            'optimizer': self.optimizer.state_dict(),
            'iter_num': iter_num,
            'best_val_loss': best_val_loss,
        }
        checkpoint_path = self.output_dir / f'checkpoint-{iter_num}.pt'
        logger.info(f"Saving checkpoint to {checkpoint_path}")
        torch.save(checkpoint, checkpoint_path)
        
        # Clean up old checkpoints - keep only the last 3
        self.cleanup_old_checkpoints()

Listing 13: Checkpointing and Model Persistence

The checkpointing system uses a simple yet effective approach: it saves checkpoints only when the validation loss improves, rather than at every evaluation. This approach serves multiple purposes. First, it ensures we’re always keeping the best-performing model, not just the most recent one. Second, it significantly reduces I/O overhead during training, as checkpoint saves can be expensive operations (our 354M parameter model checkpoints are ~1.4GB each). Third, it prevents disk space issues by avoiding the accumulation of suboptimal checkpoints. This selective checkpointing approach can improve overall training time by 5-10% by eliminating redundant disk writes.

6.2 Checkpoint Management and Cleanup

Since our 354M parameter model checkpoints are ~1.4GB each, we need to clean up old checkpoints to avoid running out of disk space. The system automatically keeps only the last 5 checkpoints and deletes older ones (as defined in config.py). The cleanup function in Listing 14 finds all checkpoint files, sorts them by modification time (newest first), and deletes everything except the most recent 5. Only the master process handles cleanup to avoid race conditions in multi-GPU setups.

def cleanup_old_checkpoints(self, keep_last=5):
    """Clean up old checkpoints, keeping only the last N"""
    if not self.master_process:
        return  # Only the master process should clean up
        
    try:
        # Find all checkpoint files
        checkpoint_files = list(self.output_dir.glob("checkpoint-*.pt"))
        
        if len(checkpoint_files) <= keep_last:
            return  # Not enough checkpoints to clean up
        
        # Sort by modification time (newest first)
        checkpoint_files.sort(key=lambda x: x.stat().st_mtime, reverse=True)
        
        # Keep the newest ones, delete the rest
        files_to_delete = checkpoint_files[keep_last:]
        
        for file_path in files_to_delete:
            try:
                file_path.unlink()
                logger.info(f"Deleted old checkpoint: {file_path.name}")
            except Exception as e:
                logger.warning(f"Failed to delete checkpoint {file_path.name}: {e}")
                
    except Exception as e:
        logger.warning(f"Checkpoint cleanup failed: {e}")

Listing 14: Checkpoint Cleanup and Management

6.3 Resume Training Functionality

The ability to resume training from any checkpoint is useful when training gets interrupted. This functionality lets you pick up where you left off, whether the interruption was a few minutes or longer.

The resume functionality loads a checkpoint file and restores the training state: the model weights, optimizer state, current iteration number, and best validation loss. If checkpoint loading fails, the code falls back to starting from scratch.

When loading checkpoints, the code handles two practical considerations. First, the map_location=self.device parameter ensures the checkpoint loads onto the correct device (CPU or GPU), which matters if you’re resuming on different hardware or after a restart. Second, for multi-GPU setups using DistributedDataParallel, the model is wrapped in a .module attribute, so the code uses raw_model = self.model.module if self.ddp else self.model to access the actual model underneath.

def resume_from_checkpoint_file(self):
    """Resume training from a checkpoint file"""
    if not self.resume_from_checkpoint:
        return
        
    checkpoint_path = Path(self.resume_from_checkpoint)
    if not checkpoint_path.exists():
        logger.error(f"Checkpoint file not found: {checkpoint_path}")
        return
        
    logger.info(f"Resuming from checkpoint: {checkpoint_path}")
    
    try:
        # Load checkpoint
        checkpoint = torch.load(checkpoint_path, map_location=self.device)
        
        # Load model state
        raw_model = self.model.module if self.ddp else self.model
        raw_model.load_state_dict(checkpoint['model'])
        logger.info("Model state loaded successfully")
        
        # Load optimizer state
        self.optimizer.load_state_dict(checkpoint['optimizer'])
        logger.info("Optimizer state loaded successfully")
        
        # Get iteration number and best validation loss
        self.start_iter = checkpoint.get('iter_num', 0)
        self.best_val_loss = checkpoint.get('best_val_loss', 1e9)
        
        logger.info(f"Resuming from iteration: {self.start_iter}")
        logger.info(f"Best validation loss so far: {self.best_val_loss:.4f}")
        
    except Exception as e:
        logger.error(f"Failed to load checkpoint: {e}")
        logger.info("Starting training from scratch...")
        self.start_iter = 0
        self.best_val_loss = 1e9

Listing 15: Resume Training from Checkpoint

The function loads the model weights, optimizer state, iteration number, and best validation loss from the checkpoint file, then continues training from where it left off. If the checkpoint file doesn’t exist or can’t be loaded, it logs an error and starts training from scratch. Since our Regular model takes 28-32 hours to train, resuming from a checkpoint saves significant time when training is interrupted by power outages, crashes, or manual stops.

7. Training Launch and Management

7.1 Multi-GPU Training with torchrun

For a single GPU, you can run the training script directly — a single Python process will use that device. To use multiple GPUs, launch training with torchrun, which spawns one worker process per GPU and lets the code initialize DistributedDataParallel (DDP). This enables larger effective batch sizes and faster wall‑clock training while keeping weights synchronized across devices; set --nproc_per_node to the number of GPUs you want to use (for example, --nproc_per_node=2).

torchrun is PyTorch’s recommended launcher for distributed training: it initializes the distributed backend and sets environment variables (RANK, LOCAL_RANK, WORLD_SIZE) to keep workers in sync. With torchrun --nproc_per_node=N (where N is the number of GPUs to use — it can be less than the total GPUs available), batches are sharded across the chosen GPUs and gradients are synchronized after each backward pass, which often gives near‑linear speedups on a small multi‑GPU node.

# Single GPU (even with multiple available)
python train_model_slm.py

# Multi-GPU with near-linear speedup
torchrun --nproc_per_node=2 train_model_slm.py

The training script handles DDP (DistributedDataParallel) via train_model_slm.py for gradient sync and batch distribution across GPUs. Figure 12 below shows an example where we have dual GPUs and both are being used.

Figure 12: Multiple GPU used for training

Note that if you run python train_model_slm.py on a multi‑GPU machine, only one GPU is used; the others remain idle. To use more than one GPU, we must use torchrun.

7.2 Training Monitoring

Training is monitored locally via structured console logs and remotely via WandB. The snippet in Listing 16 records loss, learning rate, timing, and MFU at a configurable interval and, when enabled, streams the same metrics to WandB for side‑by‑side run comparison.

# Timing and logging
t1 = time.time()
dt = t1 - t0
t0 = t1
if iter_num % self.log_interval == 0 and self.master_process:
    lossf = loss.item()
    if local_iter_num >= 5:
        mfu = raw_model.estimate_mfu(self.batch_size, dt)
        running_mfu = mfu if running_mfu == -1.0 else 0.9*running_mfu + 0.1*mfu
    logger.info(f"iter {iter_num}: loss {lossf:.4f}, time {dt*1000:.2f}ms, mfu {running_mfu*100:.2f}%")
    
    # Log to WandB - loss first for better mobile UI
    if self.use_wandb:
        wandb.log({
            "train/loss": lossf,
            "train/lr": lr,
            "train/iter": iter_num,
            "train/mfu": running_mfu * 100 if running_mfu > 0 else 0,
            "train/dt_ms": dt * 1000,
        })

Listing 16: Training Monitoring and Logging

Together, console logs and WandB provide real‑time visibility and reproducible experiment tracking; Figure 13 below shows an example of the console logs; see Section 5 for setup and dashboards.

Figure 13: Console training logs: iteration, loss, step time, MFU, and checkpoint saves

8. Model File Formats and Conversion

Training produces PyTorch checkpoint files (.pt) that contain model weights, optimizer state, and training metadata — everything needed to resume training. These checkpoints are covered in detail in Section 6 .

For sharing models and standard deployment workflows, we convert PyTorch checkpoints into the Hugging Face repository format. This conversion creates a portable, standardized model package that can be loaded with standard Hugging Face APIs.

8.1 Converting PyTorch Checkpoints to Hugging Face Format

The Hugging Face repository format is a standardized directory structure containing:

config.json: Architecture definition (layers, heads, embedding dimensions, vocabulary size, sequence length). Allows AutoModelForCausalLM to reconstruct the model architecture without custom code.
model.safetensors: Model weights in SafeTensors format (memory-mapped, secure loading). Contains only model parameters, no optimizer state — suitable for inference workloads.
generation_config.json: Default text generation parameters (max_new_tokens, temperature, top_p, repetition_penalty). Can be overridden at runtime.
Tokenizer files (tokenizer.json, vocab.json, merges.txt, special_tokens_map.json, tokenizer_config.json): Serialized tokenizer with vocabulary, merge rules, normalization, and special tokens matching the training configuration.

The conversion code in Listing 17 loads a PyTorch checkpoint, extracts model weights and config, handles torch.compile naming prefixes if present, and saves the model and tokenizer in Hugging Face format.

import torch
from transformers import GPT2LMHeadModel

def convert_pytorch_to_huggingface(pytorch_checkpoint_path, output_dir, tokenizer):
    """Convert PyTorch checkpoint to Hugging Face format"""
    
    # Load PyTorch checkpoint
    checkpoint = torch.load(pytorch_checkpoint_path, map_location='cpu')
    model_state = checkpoint['model']
    config = checkpoint['config']
    
    # Handle torch.compile prefixes
    if any(key.startswith('_orig_mod.') for key in model_state.keys()):
        clean_state = {}
        for key, value in model_state.items():
            clean_state[key[10:]] = value if key.startswith('_orig_mod.') else value
        model_state = clean_state
    
    # Convert to Hugging Face format
    hf_model = GPT2LMHeadModel(config)
    hf_model.load_state_dict(model_state)
    
    # Save in Hugging Face format
    hf_model.save_pretrained(output_dir)
    tokenizer.save_pretrained(output_dir)

Listing 17: PyTorch → Hugging Face Conversion (essentials)

The conversion handles a few practical details. If the model was compiled with torch.compile, parameter names are prefixed with _orig_mod., which the code strips to match Hugging Face module names. GPT2LMHeadModel(config) instantiates a GPT-2-style architecture that matches the checkpoint’s layer structure, and load_state_dict() loads the weights with automatic shape validation. The save_pretrained() method writes all required files to disk.

File sizes: PyTorch checkpoints are ~450MB (SLM) and ~1.4GB (Regular model); the Hugging Face format reduces this slightly by excluding the optimizer state. The tokenizer adds ~15MB to the repository.

9. Inference Options

Inference can run directly from PyTorch checkpoints or from Hugging Face models. PyTorch checkpoints are convenient during development since you can test any training checkpoint without conversion. Hugging Face models use standard from_pretrained() APIs and are better suited for sharing and deployment workflows.

# Option 1: PyTorch checkpoint inference (direct from training)
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/slm/checkpoint-60001.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Option 2: Hugging Face model inference (published models)
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bahree/london-historical-slm")
model = AutoModelForCausalLM.from_pretrained("bahree/london-historical-slm")

inputs = tokenizer("In the year 1834, I walked through the streets...", return_tensors="pt")
outputs = model.generate(inputs['input_ids'], max_new_tokens=50, do_sample=True)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)

Listing 18: Inference Options

Both methods load in seconds and generate ~50–100 tokens/sec on typical consumer GPUs (2–4GB VRAM for SLM, 6–8GB for the Regular model). Use PyTorch checkpoints for development and training comparisons; use Hugging Face models for production deployment and sharing. For interactive testing with published models, see Part 1 .

10. Summary

We built a training‑ready GPT pipeline for historical text, end‑to‑end: a clear decoder‑only architecture, pragmatic GPU/precision tuning, DDP for scale, resilient checkpointing/resume, WandB tracking, and clean hand‑off of artifacts (PyTorch checkpoints → Hugging Face export).

Outcome: two working models on the Part 2 corpus - 117M (SLM) and 354M (Regular) - ready for inference now and for evaluation/deployment in Part 4.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete training infrastructure (04_training/), model architecture (config.py), and GPU configuration (08_documentation/GPU_TUNING.md)

🧱 Series Posts: Part 1 – Using the Published Historical Models | Part 2 – Data Collection & Custom Tokenizer | Part 3 (this post) | Part 4 – Evaluation & Deployment

🤗 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on HuggingFace

📚 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts.

Ready for Part 4? Part 4 covers model evaluation, testing, and deployment strategies that turn your trained models into working systems ready for real-world use.

References

Vaswani et al. (2017) – Attention Is All You Need: https://arxiv.org/abs/1706.03762
Radford et al. (2019) – Language Models are Unsupervised Multitask Learners: https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe
Brown et al. (2020) – Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165
Kaplan et al. (2020) – Scaling Laws for Neural Language Models: https://arxiv.org/abs/2001.08361
Hoffmann et al. (2022) – Training Compute-Optimal LLMs (Chinchilla): https://arxiv.org/abs/2203.15556
Chowdhery et al. (2022) – PaLM: Scaling Language Modeling with Pathways: https://arxiv.org/abs/2204.02311
Clark et al. (2019) – What Does BERT Look At?: https://arxiv.org/abs/1906.04341
Voita et al. (2019) – Analyzing Multi‑Head Self‑Attention: https://arxiv.org/abs/1905.09418
Dao et al. (2022) – FlashAttention: https://arxiv.org/abs/2205.14135
Micikevicius et al. (2018) – Mixed Precision Training: https://arxiv.org/abs/1710.03740
Rajbhandari et al. (2020) – ZeRO: https://arxiv.org/abs/1910.02054
Paszke et al. (2019) – PyTorch: https://arxiv.org/abs/1912.01703
Kingma & Ba (2014) – Adam: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980
Loshchilov & Hutter (2017) – AdamW Decoupled Weight Decay Regularization : https://arxiv.org/abs/1711.05101
Smith & Topin (2017) – Super‑Convergence: Very Fast Training of Neural Networks Using Large Learning Rates: https://arxiv.org/abs/1708.07120
Goyal et al. (2017) – Accurate, Large Minibatch SGD: https://arxiv.org/abs/1706.02677
Sergeev & Del Balso (2018) – Horovod: https://arxiv.org/abs/1802.05799
Pope et al. (2022) – Efficiently Scaling Transformer Inference: https://arxiv.org/abs/2211.05102
Jawahar et al. (2019) – What does BERT learn about the structure of language?: https://aclanthology.org/P19-1356.pdf
Mikolov et al. (2013) – Word2vec: https://arxiv.org/abs/1301.3781
Pennington et al. (2014) – GloVe: https://aclanthology.org/D14-1162/
Devlin et al. (2018) – BERT https://arxiv.org/abs/1810.04805
Press & Wolf (2017) – Using the Output Embedding to Improve Language Models: https://arxiv.org/abs/1608.05859
Inan et al. (2016) – Tying Word Vectors and Word Classifiers: https://arxiv.org/abs/1611.01462

Happy Diwali - 2025

Sun, 19 Oct 2025 00:00:00 +0000

I 💖 Diwali. For those celebrating, from my family to yours!

🏛️Building LLMs from Scratch - Part 2: Data Collection & Custom Tokenizers

Sun, 12 Oct 2025 00:00:00 +0000

TL;DR

In this second part of our 4-part series on building language models from scratch, I explore the two foundational areas of LLM development: data collection and custom tokenizer creation. Part 1 - Building LLM from Scratch covered using the published model; here, we build the complete pipeline from raw historical documents to a custom tokenizer that understands archaic English, London geography, and period-specific terminology.

The challenge with historical LLMs isn’t just having enough data—it’s having the right data processed to preserve linguistic nuances across different historical periods. This post demonstrates how to transform over 218 historical sources into a corpus of more than 500 million characters using a specialized tokenizer for authentic historical text generation.

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need significantly larger datasets, more sophisticated infrastructure, and additional considerations that are not covered in this post.

1. The Historical Language Modeling Challenge

Building a language model for historical text presents unique challenges. Historical English from 1500 to 1850 contains linguistic patterns, vocabulary, and cultural references that modern tokenizers have never encountered. Standard tokenizers like TikToken fragment archaic words like “quoth” and “hast” into multiple subword tokens, destroying semantic meaning crucial for historical text generation.

A simple phrase like Quoth the alderman, 'Tis a fair day at Newgate becomes dozens of meaningless fragments, losing both historical context and linguistic coherence. This fragmentation is why we built a custom tokenizer trained specifically on historical English patterns, ensuring the model can generate coherent, historically accurate text.

As a reminder, both the SLM (117M parameters) and Regular Model (354M parameters) utilize the same training code and infrastructure, including GPU optimization, checkpointing, and WandB integration. The only difference lies in the model architecture parameters, which are specified in config.py.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete source code for data collection (02_data_collection/) and tokenizer training (03_tokenizer/). We will see the relevant code snippets in this post show key concepts—see the full implementation in the repository.

🧱 Series Posts: Part 1 – Using the Published Historical Models | Part 2 (this post) | Part 3 – Training Architecture & GPU Optimization | Part 4 – Evaluation & Deployment

What will you learn?

This project provides hands-on experience with real-world LLM development challenges, including data collection from over 218 historical sources, cleaning OCR errors and encoding issues, and developing custom tokenizers for historical text. Unlike theoretical tutorials, you receive complete, runnable code that demonstrates actual trade-offs and decisions—such as choosing BPE over WordPiece or handling different file formats—that you’d encounter in any serious LLM project.

While operating at a learning scale, the principles taught here directly apply to larger systems. Data collection patterns, cleaning strategies, and tokenizer design principles scale from our 500M character corpus to the 500B+ character datasets used in production models.

1.1 High-Level Process Overview

The complete pipeline transforms raw historical documents into a working language model through five key stages:

Data Collection: 218+ historical sources (1500-1850), including literature, newspapers, court records, and personal diaries
Cleaning Pipeline: Handles multiple file formats (PDF, HTML, XML, TXT) while removing OCR artifacts and preserving authentic historical language
Quality Validation: Removes duplicates, filters non-English content, and ensures only meaningful historical text reaches the final corpus
Custom Tokenizer Training: BPE-based tokenizer with ~150 special tokens capturing archaic pronouns, historical landmarks, and period-specific terminology
Model Training: Two language models (SLM 117M and Regular 354M parameters) trained on the same historical corpus

The result is a system capable of generating authentic historical text that captures the linguistic patterns and cultural context of 1500-1850 English. Figure 1 illustrates this complete pipeline:

graph TD
    A[📚 218+ Historical Sources<br/>1500-1850] --> B[🔍 Data Collection<br/>Download and Filter]
    B --> C[🧹 5-Phase Cleaning Pipeline<br/>Format-Specific Processing]
    C --> D[📊 Quality Validation<br/>Duplicate and Language Detection]
    D --> E[📝 500M+ Character Corpus<br/>Clean Historical Text]
    E --> F[🔤 Custom Tokenizer Training<br/>BPE with 150+ Special Tokens]
    F --> G[🤖 Language Model Training<br/>SLM 117M + Regular 354M]
    
    style A fill:#e1f5fe
    style E fill:#f3e5f5
    style F fill:#fff3e0

Figure 1: Complete Historical Text Processing Pipeline

2. Data Collection: The Foundation of Historical Language Modeling

Let us dig deeper into steps 1-4: data collection, cleaning, validation, and corpus creation. The data collection system processes over 218 sources spanning the years 1500-1850 to create a corpus of over 500 million characters of authentic historical English text. But collecting historical data isn’t just about downloading files - it’s about handling the sheer variety of formats and quality levels that historical documents present.

Historical documents come in all shapes and sizes - scanned books with OCR errors, HTML pages with messy markup, XML archives with rich metadata, and plain text files with inconsistent encoding. This is especially true for the earlier periods, when the quality of the documents can vary significantly, and most modern techniques for processing them struggle to cope. This data diversity requires a cleaning pipeline that transforms raw historical documents into training data while preserving the authentic language patterns of 1500-1850 English.

2.1 System Architecture: Processing 218+ Historical Sources

The data collection system employs a modular architecture, with historical_data_collector.py serving as the primary orchestration engine, coordinating with a data_sources.json configuration file that contains metadata for over 218 historical sources. This enables easy management and updates without code changes.

Supporting scripts include add_data_source.py for interactive source addition with built-in validation, and generate_report.py for comprehensive reporting and analysis across multiple output formats.

The data_sources.json file contains metadata for each source, including time periods, formats, licensing, and processing priorities. Each entry includes:

time_period (e.g., [1690, 1800] for London Lives)
format (XML, HTML, PDF)
priority (high/medium/low)
search_terms for collection guidance

Our data sources span multiple categories, each contributing unique perspectives to the historical corpus.

Project Gutenberg: This provides foundational literature with 8+ carefully selected texts, using relaxed quality criteria that accept texts with as low as 40% meaningful words to capture the full spectrum of historical writing styles.
Historical Archives: Historical Archives like London Lives (240,000 pages of personal records) and Old Bailey (197,000+ trial transcripts) offer rich historical content and were initially enabled in our data collection.
- Note: I was using the aggressive cleaning earlier (enabled using the aggressive_cleaning flag designed to remove structured legal data and semantic markup), and discovered that it was too aggressive and caused generation quality issues. After initial training runs revealed repetitive and incoherent text patterns, I turned off these sources. Enabling this back might be an exercise for you to try.
Archive.org: Archive.org has an API access that can be used for file filtering, and this makes it relatively straightforward.
The National Archives (TNA): TNA records contribute government correspondence and official documents that provide the institutional context for historical events.
British History Online: Finally, these supplements our collection with historical surveys and period documents that offer scholarly perspectives on the time periods we’re modeling.

However, each source type presents unique technical challenges that require specialized processing approaches. One example is Project Gutenberg, which contains files with standardized headers and footers that must be removed. (As a side note, I really appreciate the effort that has gone into this to make this formatting consistent, which makes the process of this relatively straightforward.)

On the other hand, PDF files often suffer from OCR errors, especially for older documents that contain corrupted historical language, requiring sophisticated text correction algorithms to restore proper spelling and grammar from scanned documents. The figure below shows one example of how older documents look. This example is “The abridgment of the charter of the city of London” from 1680.

Figure 2: The abridgment of the charter of the city of London (1680) - showing faded text and ink blots typical of historical documents

As you can see, the text is faded, has ink blots, and the font style is very different from modern text. OCR software often misinterprets characters in such documents, resulting in numerous errors, as illustrated in the image below. These OCR artifacts can severely degrade the quality of our training data if not properly addressed.

Figure 3: OCR errors in the charter document - showing how optical character recognition struggles with historical fonts and document quality

HTML files from sources like Archive.org contain navigation elements, advertisements, and modern web markup that contaminate the historical corpus, demanding careful content extraction that preserves only the meaningful historical text.

XML archives like London Lives and Old Bailey require specialized parsing to extract meaningful text while preserving semantic markup that provides context about speakers, dates, and document structure - a delicate balance between removing technical artifacts and maintaining historical authenticity.

Government records from TNA often contain bureaucratic formatting, form fields, and institutional language that need careful filtering to extract the human stories and historical narratives.

British History Online documents present challenges with academic formatting, footnotes, and scholarly apparatus that must be processed to maintain readability while preserving the scholarly context that makes them valuable for historical language modeling.

2.2 Cleaning Pipeline

I implement a 5-stage cleaning pipeline that helps transform the raw historical documents into training-ready text. Each phase addresses specific challenges that would otherwise contaminate our language model training.

2.2.1 Stage 1: File Discovery & Initial Filtering

Historical archives often contain files in various formats, which may be missing proper file extensions or have non-standard naming conventions. Many files contain non-English content that would contaminate our English historical corpus. Additionally, many sources employ their own templates and standards for this purpose. To resolve this, we first implement a simple file detection and naming cleanup, as shown in Listing 1. The code itself is simple and self-explanatory.

def detect_file_type(file_path: str) -> str:
    """Detect file type based on extension and content analysis"""
    # Extension-based detection
    if file_path.endswith(('.txt', '.txt.utf-8', '_txt.utf-8')):
        return 'text'
    elif file_path.endswith(('.pdf',)):
        return 'pdf'
    elif file_path.endswith(('.html', '.htm')):
        return 'html'
    elif file_path.endswith(('.xml',)):
        return 'xml'
    
    # Content-based detection for files without extensions
    with open(file_path, 'rb') as f:
        content = f.read(1024)  # Read first 1KB
        if b'<html' in content.lower() or b'<!doctype' in content.lower():
            return 'html'
        elif b'<?xml' in content.lower():
            return 'xml'
        elif content.isascii() and b'\x00' not in content:
            return 'text'
        else:
            return 'binary'

Listing 1: File Type Detection Function

When we run this locally, we will see the flow as outlined below, which illustrates how the detection works. This, of course, can be made more robust for non-English characters, but for now, we reject these.

File Type Detection Flow:

📁 Raw Files (218+ sources)
    ↓
🔍 File Type Detection
    ├── .txt, .txt.utf-8, _txt.utf-8 → Text Processing
    ├── .pdf → PDF Processing  
    ├── .html, .htm → HTML Processing
    ├── .xml → XML Processing (Old Bailey, London Lives)
    └── No Extension → Content Detection
        ├── HTML-like content → HTML Processing
        ├── Text-like content → Text Processing
        └── Binary/Unknown → REJECTED
    ↓
🚫 Filename Language Check
    ├── Non-English characters → REJECTED (logged)
    └── English/Latin → Continue

Historical archives often lack standardized file extensions and contain content in languages other than English. Our two-stage detection ensures we capture valuable historical documents while filtering out irrelevant files, preventing both data loss and processing waste.

2.2.2 Stage 2: Format-Specific Content Extraction

Each file format requires specialized processing due to its unique contamination sources, including Project Gutenberg headers, PDF OCR errors, HTML navigation elements, and XML structural markup. Our format-specific extraction functions clean these artifacts while preserving authentic historical content.

Text Files (.txt, .txt.utf-8)

Project Gutenberg texts contain standardized headers and footers that would confuse our language model. The cleaning process removes these while preserving the actual historical content. The code snippet in Listing 2 demonstrates this approach and is quite straightforward. Of course, this can be made more robust, but this works well for our selected texts.

def clean_gutenberg_text(text: str) -> str:
    """Clean Project Gutenberg text by removing headers/footers and metadata"""
    lines = text.split('\n')
    cleaned_lines = []
    in_content = False
    
    for line in lines:
        # Skip Gutenberg headers (before "*** START OF")
        if "*** START OF" in line:
            in_content = True
            continue
        # Skip Gutenberg footers (after "*** END OF")
        if "*** END OF" in line:
            break
        # Skip metadata lines
        if line.startswith(('Title:', 'Author:', 'Release Date:', 'Language:')):
            continue
        # Skip empty lines at start
        if not in_content and not line.strip():
            continue
            
        if in_content:
            cleaned_lines.append(line)
    
    return '\n'.join(cleaned_lines).strip()

Listing 2: Project Gutenberg Text Cleaning Function

Real Example - Before Cleaning:

Title: A Journal of the Plague Year
Author: Daniel Defoe
Release Date: March 2003
Language: English

*** START OF THE PROJECT GUTENBERG EBOOK A JOURNAL OF THE PLAGUE YEAR ***

It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland...

*** END OF THE PROJECT GUTENBERG EBOOK A JOURNAL OF THE PLAGUE YEAR ***

After Cleaning:

It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland...

Without this cleaning, the model would learn to generate Gutenberg headers and metadata instead of authentic historical text, contaminating the training data with modern digital artifacts.

PDF Files

PDF files from historical archives often contain OCR errors and digital artifacts that require correction. The cleaning process in Listing 3 addresses these issues while preserving historical content, removing page numbers and all-caps headers. While not perfect, it significantly improves text quality.

The OCR correction rules are based on common patterns in historical documents and can be refined for specific datasets. Libraries like PyMuPDF or pdfplumber extract text, while regex-based cleaning corrects common OCR errors and removes digital stamps. More advanced techniques, such as layout analysis or AI-based OCR correction, can further enhance this process.

def clean_pdf_text(text: str) -> str:
    """Clean PDF text by removing OCR artifacts and digital stamps"""
    # Remove page numbers: [Page 123], standalone numbers
    text = re.sub(r'\[Page \d+\]', '', text)
    text = re.sub(r'^\d+$', '', text, flags=re.MULTILINE)
    
    # Remove library stamps: Internet Archive, Google, etc.
    stamps = [
        'Internet Archive', 'Google Books', 'HathiTrust',
        'Digitized by Google', 'Scanned by Google'
    ]
    for stamp in stamps:
        text = text.replace(stamp, '')
    
    # Fix common OCR artifacts
    ocr_fixes = {
        r'\b0\b': 'O',  # 0 → O
        r'\b1\b': 'I',  # 1 → I  
        r'\b5\b': 'S',  # 5 → S
        r'\b8\b': 'B',  # 8 → B
        r'\brn\b': 'm', # rn → m
        r'\bcl\b': 'd'  # cl → d
    }
    
    for pattern, replacement in ocr_fixes.items():
        text = re.sub(pattern, replacement, text)
    
    # Remove all-caps lines (usually headers)
    lines = text.split('\n')
    cleaned_lines = [line for line in lines if not line.isupper() or len(line) < 10]
    
    return '\n'.join(cleaned_lines)

Listing 3: PDF Text Cleaning Function

Real Example - Before Cleaning:

[Page 1]
INTERNET ARCHIVE
A JOURNAL OF THE PLAGUE YEAR
BY DANIEL DEFOE

It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland. For it was indeed a very terrible time, and the people began to be very much alarmed at it.

After Cleaning:

A JOURNAL OF THE PLAGUE YEAR
BY DANIEL DEFOE

It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland. For it was indeed a very terrible time, and the people began to be very much alarmed at it.

OCR errors can significantly impact the quality of model training. For example, if London appears as L0nd0n due to OCR errors, the model won’t learn the correct spelling and will generate nonsensical text when asked about historical London. The correction process ensures our model learns authentic historical language patterns rather than digital artifacts, which is crucial for generating coherent and historically accurate text.

HTML Files

HTML files from historical websites and digital archives contain markup that needs to be stripped while preserving the actual text content. We use the BeautifulSoup library in Listing 4 to clean the HTML structure and extract only the meaningful text.

def clean_html_text(html_content: str) -> str:
    """Clean HTML content by removing markup and extracting text"""
    from bs4 import BeautifulSoup
    
    soup = BeautifulSoup(html_content, 'html.parser')
    
    # Remove unwanted elements
    for element in soup(['script', 'style', 'nav', 'header', 'footer', 
                        'aside', 'menu', 'form', 'input', 'button']):
        element.decompose()
    
    # Remove wiki-specific elements
    for element in soup.find_all(['div', 'span'], class_=['navbox', 'infobox', 'sidebar']):
        element.decompose()
    
    # Remove navigation elements
    for element in soup.find_all(['div', 'ul'], class_=['breadcrumb', 'navigation', 'menu']):
        element.decompose()
    
    # Extract text content
    text = soup.get_text(separator=' ', strip=True)
    
    # Clean up excessive whitespace
    text = re.sub(r'\s+', ' ', text)
    
    return text.strip()

Listing 4: HTML Text Cleaning Function

Real Example - Before Cleaning:

<!DOCTYPE html>
<html>
<head><title>London History</title></head>
<body>
<nav>Home | About | Contact</nav>
<header>London Historical Society</header>
<div class="content">
    <h1>The Great Fire of London</h1>
    <p>In the year 1666, a great fire consumed much of London...</p>
</div>
<footer>© 2024 London Historical Society</footer>
</body>
</html>

After Cleaning:

The Great Fire of London in the year 1666, a great fire consumed much of London...

HTML tags and navigation elements would contaminate training, causing the model to generate markup instead of historical text. Our cleaning process extracts meaningful content while preserving natural flow and structure.

XML Files (Historical Archives):

XML files from historical archives, such as the Old Bailey and London Lives, use specific schemas that require specialized parsing. Old Bailey employs TEI (Text Encoding Initiative) with TEI.2 elements, while London Lives uses semantic markup (name, geo, occupation). These structured formats contain authentic historical language with rich metadata, as shown in Listing 5.

def extract_old_bailey_text(soup) -> str:
    """Extract text from Old Bailey XML using TEI schema structure"""
    extracted_text = []
    
    # Check for TEI.2 elements (Old Bailey schema)
    tei_elements = soup.find_all('TEI.2')
    if tei_elements:
        # Extract trial accounts (main narrative content)
        trial_accounts = soup.find_all('div1', {'type': 'trialAccount'})
        for trial in trial_accounts:
            trial_text = extract_trial_narrative(trial)
            if trial_text:
                extracted_text.append(trial_text)
        
        # Extract front matter (session information)
        front_matter = soup.find_all('div1', {'type': 'frontMatter'})
        for front in front_matter:
            front_text = extract_front_matter_narrative(front)
            if front_text:
                extracted_text.append(front_text)
    
    return '\n\n'.join(extracted_text)

def extract_london_lives_text(soup) -> str:
    """Extract text from London Lives XML using semantic markup schema"""
    extracted_text = []
    
    # Check for London Lives specific elements (name, geo, occupation, date)
    name_elements = soup.find_all('name')
    geo_elements = soup.find_all('geo')
    occupation_elements = soup.find_all('occupation')
    
    if name_elements and geo_elements and occupation_elements:
        # Extract paragraphs with semantic markup
        paragraphs = soup.find_all('p')
        for para in paragraphs:
            p_text = extract_paragraph_with_semantic_markup(para)
            if p_text.strip():
                extracted_text.append(p_text)
    
    return '\n\n'.join(extracted_text)

Listing 5: XML Text Extraction Functions

Real Example - Old Bailey XML (Before Processing):

<trial>
<frontmatter>
<session>Session 1</session>
<date>1674-04-15</date>
<location>Old Bailey</location>
</frontmatter>
<proceedings>
The prisoner being brought to the bar, and the indictment being read, he pleaded Not Guilty. The witnesses being sworn, the first witness deposed that on the 15th day of April last, he saw the prisoner in the company of several suspicious persons...
</proceedings>
</trial>

After Processing:

Session 1 1674-04-15 Old Bailey The prisoner being brought to the bar, and the indictment being read, he pleaded Not Guilty. The witnesses being sworn, the first witness deposed, that on the 15th day of April last, he saw the prisoner in the company of several suspicious persons...

These XML files contain the most authentic historical language in our entire dataset. The Old Bailey trials show how people actually spoke in court during the 17th-19th centuries, while London Lives reveals the everyday language used in personal records and official documents. This authentic historical language is very useful for training a model that can generate historically accurate text, as it provides the model with genuine examples of how people wrote and spoke during different historical periods.

2.2.3 Stage 3: Text Normalization

After extraction, text normalization ensures consistency and compatibility with the training data. Historical documents contain encoding issues, inconsistent formatting, and special characters that confuse the model. Our normalization process fixes these issues and breaks long lines to fit within the model’s context window. This is critical because lines exceeding the context window appear as incomplete sentences to the transformer, severely degrading generation quality due to the attention mechanism’s inability to process fragmented text.

Inconsistent encoding and formatting can severely confuse the language model during training. For example, if some files use smart quotes (") and others use straight quotes ("), the model might not learn that they represent the same concept, leading to inconsistent and potentially incorrect text generation. Normalization ensures that the model observes consistent patterns across all training data, which is crucial for learning coherent language patterns and generating high-quality historical text.

The code snippet in Listing 6 demonstrates how we implement this normalization, which is quite straightforward.

def normalize_text(text: str) -> str:
    """Normalize text for consistent training data"""
    import unicodedata
    
    # Fix common encoding issues
    encoding_fixes = {
        'â€™': "'",  # Smart apostrophe
        'â€œ': '"',  # Smart quote left
        'â€': '"',   # Smart quote right
        'â€"': '—',  # Em dash
        'â€¢': '•',  # Bullet point
        'â€¦': '…',  # Ellipsis
    }
    
    for old, new in encoding_fixes.items():
        text = text.replace(old, new)
    
    # Normalize Unicode (NFC)
    text = unicodedata.normalize('NFC', text)
    
    # Break long lines for training compatibility (max 2000 chars)
    lines = text.split('\n')
    normalized_lines = []
    for line in lines:
        if len(line) > 2000:
            # Split at sentence boundaries
            sentences = re.split(r'(?<=[.!?])\s+', line)
            current_line = ""
            for sentence in sentences:
                if len(current_line + sentence) > 2000:
                    if current_line:
                        normalized_lines.append(current_line.strip())
                    current_line = sentence
                else:
                    current_line += " " + sentence if current_line else sentence
            if current_line:
                normalized_lines.append(current_line.strip())
        else:
            normalized_lines.append(line)
    
    # Normalize line endings and whitespace
    text = '\n'.join(normalized_lines)
    text = re.sub(r'[ \t]+', ' ', text)  # Multiple spaces/tabs to single space
    text = re.sub(r'\n\s*\n', '\n\n', text)  # Multiple newlines to double newline
    
    return text.strip()

Listing 6: Text Normalization Function

Real Example - Before Normalization:

The year was 1666, and the plague had come to London. â€œIt was indeed a very terrible time,â€ wrote one observer. The streets were filled with the sounds of horse-drawn carriages and the cries of the afflicted.

After Normalization:

The year was 1666, and the plague had come to London. "It was indeed a very terrible time," wrote one observer. The streets were filled with the sounds of horse-drawn carriages and the cries of the afflicted.

2.2.4 Stage 4: Quality Validation

Not all extracted text is suitable for training. Some files contain duplicates, non-English content, or poor-quality text that would degrade model performance. We need a comprehensive validation system that ensures only high-quality, relevant text is included in our training corpus.

The key challenge is striking a balance between quality standards and historical value. A strict approach might reject valuable historical documents that have some OCR issues, while a lenient approach might include too much low-quality content, which can degrade model training. To address this, I implemented a tiered quality threshold system that applies different standards based on content type:

General Content: 200+ chars, 50+ words, 50% meaningful words
Project Gutenberg: 200+ chars, 50+ words, 40% meaningful words (relaxed for historical value)
Historical Documents: 1000+ chars, 100+ words, 30% meaningful words (very relaxed for historical value)

This tiered approach ensures that we capture valuable historical content while maintaining quality standards, filtering out duplicates, non-English content, and low-quality text, thereby preserving the integrity of useful historical documents. Again, these implementations are quite simple, in the context of a toy project, but can be made more robust. The code itself is quite straightforward, as shown in Listing 7.

def analyze_text_quality(text: str, source_type: str = 'general') -> dict:
    """Analyze text quality and determine if it should be included in training corpus"""
    import hashlib
    import re
    
    # Length validation
    char_count = len(text)
    word_count = len(text.split())
    
    # OCR artifact detection using regex patterns
    ocr_patterns = {
        'long_capitals': r'[A-Z]{5,}\s+[A-Z]{5,}',
        'spaced_letters': r'\b[A-Za-z]\s+[A-Za-z]\s+[A-Za-z]\s+[A-Za-z]\b',
        'special_chars': r'[!@#$%^&*()]{3,}',
        'mixed_alphanumeric': r'\b\d+[A-Za-z]+\d+\b',
        'long_non_word': r'[^\w\s]{10,}'
    }
    
    ocr_issues = []
    for pattern_name, pattern in ocr_patterns.items():
        if re.search(pattern, text):
            ocr_issues.append(pattern_name)
    
    # Advertisement detection
    ad_patterns = [
        'this day is published', 'just ready', 'elegantly bound',
        'now ready', 'new novels', 'advertisements',
        'price \d+s', 'paternoster row', 'corner of', 'publishers'
    ]
    
    ad_count = sum(1 for pattern in ad_patterns if re.search(pattern, text, re.IGNORECASE))
    ad_density = ad_count / max(word_count, 1)
    
    # Meaningful word ratio calculation
    words = text.split()
    meaningful_words = [w for w in words if w.isalpha() and len(w) > 2]
    meaningful_ratio = len(meaningful_words) / max(len(words), 1)
    
    # Quality thresholds based on source type
    thresholds = {
        'general': {'min_chars': 200, 'min_words': 50, 'min_meaningful_ratio': 0.50},
        'gutenberg': {'min_chars': 200, 'min_words': 50, 'min_meaningful_ratio': 0.40},
        'historical': {'min_chars': 1000, 'min_words': 100, 'min_meaningful_ratio': 0.30}
    }
    
    threshold = thresholds.get(source_type, thresholds['general'])
    
    # Quality scoring
    score = 100
    score -= len(ocr_issues) * 3  # OCR issues
    score -= ad_density * 50  # Advertisement density
    score -= (1 - meaningful_ratio) * 20  # Meaningful word ratio
    
    # Check if text meets quality thresholds
    meets_thresholds = (
        char_count >= threshold['min_chars'] and
        word_count >= threshold['min_words'] and
        meaningful_ratio >= threshold['min_meaningful_ratio'] and
        ad_density < 0.1  # Less than 10% advertisement content
    )
    
    return {
        'char_count': char_count,
        'word_count': word_count,
        'meaningful_ratio': meaningful_ratio,
        'ocr_issues': ocr_issues,
        'ad_density': ad_density,
        'score': score,
        'meets_thresholds': meets_thresholds,
        'content_hash': hashlib.md5(text.encode()).hexdigest()
    }

Listing 7: Text Quality Analysis Function

Content Quality Validation

Our validation system employs multiple detection mechanisms to ensure training corpus quality:

OCR Artifact Detection: Regex patterns identify common digitization errors, including misread headers, character separation failures, scanning artifacts, alphanumeric misinterpretations, and corrupted text regions
Advertisement Filtering: Pattern matching detects commercial content using phrases like “this day is published”, “just ready”, “elegantly bound”, and price references
Quality Scoring: A 100-point system deducts points for OCR artifacts (-3 each), advertisement density (-50), and low meaningful word ratios (-20)

This multi-layered approach balances quality standards with preservation of valuable historical content, ensuring the model trains on authentic historical language while filtering out contamination sources.

2.2.5 Stage 5: Final Processing and Corpus Creation

After cleaning and validation, we create a final training corpus optimized for language model training. This requires intelligent segmentation that breaks long texts into manageable chunks while preserving the historical narrative flow, which is essential given the context window limits (e.g., 2048 tokens). The code snippet in Listing 8 demonstrates this final processing stage.

def create_comprehensive_corpus(cleaned_files: list) -> str:
    """Create final training corpus with intelligent segmentation"""
    corpus_parts = []
    
    for file_path in cleaned_files:
        with open(file_path, 'r', encoding='utf-8') as f:
            content = f.read()
        
        # Split into training segments
        segments = split_into_training_segments(content)
        corpus_parts.extend(segments)
    
    # Create final corpus
    final_corpus = '\n\n'.join(corpus_parts)
    
    # Save to file
    with open('london_historical_corpus_comprehensive.txt', 'w', encoding='utf-8') as f:
        f.write(final_corpus)
    
    return final_corpus

def split_into_training_segments(text: str, max_length: int = 2000) -> list:
    """Split text into training segments while preserving narrative flow"""
    # First split on double newlines (paragraphs)
    paragraphs = text.split('\n\n')
    
    segments = []
    current_segment = ""
    
    for paragraph in paragraphs:
        if len(current_segment + paragraph) <= max_length:
            current_segment += paragraph + '\n\n'
        else:
            if current_segment:
                segments.append(current_segment.strip())
            current_segment = paragraph + '\n\n'
    
    if current_segment:
        segments.append(current_segment.strip())
    
    # Further split long segments at sentence boundaries
    final_segments = []
    for segment in segments:
        if len(segment) > max_length:
            sentences = re.split(r'(?<=[.!?])\s+', segment)
            current_segment = ""
            for sentence in sentences:
                if len(current_segment + sentence) <= max_length:
                    current_segment += sentence + " "
                else:
                    if current_segment:
                        final_segments.append(current_segment.strip())
                    current_segment = sentence + " "
            if current_segment:
                final_segments.append(current_segment.strip())
        else:
            final_segments.append(segment)
    
    # Filter out segments that are too short
    return [seg for seg in final_segments if len(seg) >= 50]

Listing 8: Corpus Creation and Segmentation Functions

During my local runs, this final processing stage generated a comprehensive corpus of over 500 million characters across ~250,000 segments, with an average segment length of around 2,000 characters. The success rate of files making it into the final corpus ranged from 70% to 90%, depending on the quality and availability of the source.

Final Corpus Statistics:

Total Sources Processed: 218+ historical sources
Final Corpus Size: 500M+ characters
Training Segments: ~250,000 segments
Average Segment Length: ~2,000 characters
Success Rate: 70-90% (depending on source availability)

2.3 Detailed Data Processing Flow

Building on the high-level flow and having reviewed each of the areas, the detailed flow below illustrates the complete data cleaning process, including rejection paths, error handling, and statistics tracking. This is intended to provide a bird’s-eye view of the entire process.

graph TD
    A[📁 Raw Files] --> B{File Type Detection}
    
    B -->|.txt, .txt.utf-8| C[📄 Text File]
    B -->|.pdf| D[📄 PDF File]
    B -->|.html, .htm| E[📄 HTML File]
    B -->|.xml| F[📄 XML File]
    B -->|No Extension| G{Content Detection}
    
    G -->|HTML-like| E
    G -->|Text-like| C
    G -->|Binary/Unknown| REJECT1[❌ REJECTED]
    
    C --> H[🧹 clean_gutenberg_text]
    D --> I[🔧 extract_text_from_pdf]
    E --> J[🧹 clean_html_text]
    F --> K{XML Type Detection}
    
    I --> L[🧹 clean_pdf_text]
    
    K -->|Old Bailey| M[🔧 extract_old_bailey_text]
    K -->|London Lives| N[🔧 extract_london_lives_text]
    
    M --> O[🧹 clean_old_bailey_text]
    N --> P[🧹 clean_london_lives_text]
    
    H --> Q[🔧 normalize_text]
    L --> Q
    J --> Q
    O --> Q
    P --> Q
    
    Q --> R[🔍 Duplicate Detection]
    R -->|Duplicate| REJECT2[❌ REJECTED - Duplicate]
    R -->|Unique| S[🌍 Language Detection]
    
    S -->|Non-English| REJECT3[❌ REJECTED - Non-English]
    S -->|English| T[📊 Quality Analysis]
    
    T --> U{Quality Check}
    U -->|Poor Quality| REJECT4[❌ REJECTED - Poor Quality]
    U -->|Good Quality| V[💾 Save to Processed Directory]
    
    V --> W[📊 Update Statistics]
    W --> X[✅ Successfully Processed]
    
    REJECT1 --> Y[📝 Log Rejection Reason]
    REJECT2 --> Y
    REJECT3 --> Y
    REJECT4 --> Y
    
    Y --> Z[📊 Update Rejection Stats]
    
    style A fill:#e1f5fe
    style X fill:#c8e6c9
    style REJECT1 fill:#ffcdd2
    style REJECT2 fill:#ffcdd2
    style REJECT3 fill:#ffcdd2
    style REJECT4 fill:#ffcdd2
    style Y fill:#fff3e0
    style Z fill:#fff3e0

Figure 4: Detailed Data Processing Pipeline

2.5 Corpus Creation Process

After cleaning, the system creates the final training corpus through intelligent segmentation that preserves historical narrative flow:

📁 Cleaned Files
    ↓
🔧 create_comprehensive_corpus()
    ├── Read all cleaned_*.txt files
    ├── Split into training segments (split_into_training_segments)
    │   ├── Split on double newlines (paragraphs)
    │   ├── Max length: 2000 characters
    │   ├── Min length: 100 characters
    │   └── Further split long segments at sentence boundaries
    ├── Filter segments (min 50 characters)
    └── Write to london_historical_corpus_comprehensive.txt

The corpus creation process reads all cleaned text files and intelligently segments them into training-ready chunks. It first splits on double newlines to preserve paragraph boundaries, which are natural break points in historical text. Segments are constrained to a maximum of 2000 characters to fit within the model’s context window, with a minimum of 100 characters to ensure substantial content. Long segments are further split at sentence boundaries to maintain readability. Finally, segments shorter than 50 characters are filtered out as they’re unlikely to contain meaningful historical content.

Proper segmentation is crucial for training language models. The model needs to learn from coherent text segments that maintain historical narrative flow while fitting within its context window. Splitting on paragraph boundaries preserves the natural structure of historical documents, while sentence-level splitting ensures that very long paragraphs don’t exceed the model’s processing capabilities. This approach maximizes the model’s ability to learn from authentic historical language patterns while maintaining training efficiency.

2.6 Outcome: Training-Ready Corpus

The result is a clean, historically faithful corpus containing over 500 million characters of authentic historical English spanning 350 years of London history from 1500-1850. The corpus comprises high-quality text with minimal OCR artifacts, preserving historical language patterns and a rich cultural context that reflects the social, political, and economic realities of various historical periods. The text has been intelligently segmented for optimal language model training, with careful attention to maintaining the natural flow of historical narratives while ensuring compatibility with modern training techniques.

This corpus serves as the essential foundation for training our specialized historical tokenizer and language model, ensuring the model learns authentic historical English rather than modern text patterns. By providing the model with genuine examples of how people wrote and spoke during different historical periods, we enable it to generate text that captures the linguistic nuances, cultural references, and historical context that make historical language modeling both challenging and rewarding.

💻 Try It Yourself: The complete implementation, including all the data collection scripts, cleaning algorithms, and quality validation systems described in this section, is available in the helloLondon GitHub repository . The repository includes detailed documentation, example usage, and step-by-step guides for setting up your own historical language model training pipeline.

Now that we have examined the data collection and cleaning process, we can proceed to the next steps: creating a custom historical tokenizer and preparing for model training.

3. Custom Historical Tokenizer: The Key to Authentic Historical Text Generation

Creating a custom tokenizer is crucial for generating effective historical text. This section examines the necessity of a custom tokenizer, the challenges presented by historical language, and our chosen architecture. The tokenizer preserves the semantic meaning of historical words and phrases, enabling coherent and contextually accurate historical narratives.

Standard tokenizers like GPT-2’s fragment archaic words like “quoth” and “hast” into multiple subword tokens, destroying semantic meaning crucial for historical text generation.

Real Example - Standard Tokenizer vs. Our Custom Tokenizer:

Standard GPT-2 Tokenizer:

"Quoth the alderman, 'Tis a fair day at Newgate"
→ ['Qu', 'oth', ' the', ' ald', 'erman', ',', ' ', ''', 'T', 'is', ' a', ' fair', ' day', ' at', ' New', 'gate']

Our Custom Historical Tokenizer:

"Quoth the alderman, 'Tis a fair day at Newgate"
→ ['<|quoth|>', ' the', ' alderman', ',', ' ', ''', '<|tis|>', ' a', ' fair', ' day', ' at', ' <|newgate|>']

The standard tokenizer breaks historical language into 18 meaningless fragments, losing semantic meaning and historical context. Our custom tokenizer reduces this to 12 meaningful tokens, preserving authentic historical language patterns essential for coherent text generation.

A tokenizer that fragments historical language destroys the model’s ability to learn authentic patterns. The model needs to perceive “quoth” as a single concept, rather than fragmented subwords, to capture the linguistic nuances of different historical periods.

3.1 What Happens with Off-the-Shelf Tokenizers

What would happen if we used standard tokenizers like tiktoken or GPT-2’s tokenizer?

Standard tokenizers would force the model to waste capacity reconstructing fragmented historical words from subwords rather than learning historical language patterns. The model might learn to generate “Qu” + “oth” but struggle to use “quoth” in new contexts. Historical phrases like “methinks” would split into meaningless fragments, losing semantic coherence. London geography becomes particularly problematic, as place names like “Newgate” fragment, making spatial relationships harder to understand.

Generation Quality Issues:

# What you'd get with standard tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"
→ Generates: "Qu oth the ald erman, 'T is a fair day at New gate"
→ Result: Broken, unreadable historical text

# What you get with our custom tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"  
→ Generates: "Quoth the alderman, 'Tis a fair day at Newgate"
→ Result: Authentic, coherent historical text

A vocabulary that’s too small (10K tokens) would fragment even more historical words, making the problem worse, while a vocabulary that’s too large (100K+ tokens) would overfit to rare historical terms, wasting capacity on words that appear only once. Our choice of 30K tokens provides a balanced approach that captures common historical patterns without overfitting, ensuring the model learns the most important historical language patterns efficiently.

Real-World Example: With a standard tokenizer, our model might generate:

"The ald erman walk ed to New gate where he saw the pris oner"

With our custom tokenizer, it generates:

"The alderman walked to Newgate where he saw the prisoner"

The difference in historical text authenticity is significant between the two approaches.

3.2 Tokenizer Architecture

I had started with the easier WordPiece tokenizer (more of an accident rather than by design). Still, I realized later that it was unsuitable for historical text due to the ## subword prefix artifacts. We need a tokenizer that can handle historical English efficiently while preserving semantic meaning, unlike standard tokenizers like GPT-2’s WordPiece approach, which fragments historical language and, as a result, destroys the linguistic patterns we want to preserve. After some experimentation, I settled on a custom Byte Pair Encoding (BPE) tokenizer trained specifically on historical English.

BPE is a subword tokenization algorithm that learns to break text into meaningful subword units by iteratively finding the most frequent character pairs in the training corpus and merging them into single tokens. The process begins with individual characters and gradually evolves into common words and phrases.

For example, if "th" appears frequently in our historical corpus, BPE will learn to treat it as a single token rather than separate "t" and "h" tokens. This is particularly valuable for historical English, where words like "thou", "thee", and "thine" share common prefixes and suffixes.

3.2.1 Tokenizer Training Process

The BPE training algorithm analyzes our entire historical corpus to identify the most frequent character combinations, building a vocabulary that’s optimized for historical language patterns. We start with a base alphabet (comprising all letters) and special tokens, then iteratively merge the most frequent pairs until we reach our target vocabulary size of 30,000 tokens. This ensures that common historical words, such as "quoth", "hast", and "methinks", are treated as single tokens, while still allowing for the handling of rare or unknown words by breaking them into learned subword units.

The training process is computationally efficient and produces a tokenizer that’s specifically tuned to the linguistic patterns found in our historical corpus.

In this case, we don’t have to reinvent the wheel and use the Hugging Face tokenizers library, which provides a modular approach to building custom tokenizers. The library is organized into several key components: models define the core tokenization algorithm (BPE, WordPiece, Unigram), pre_tokenizers handle initial text splitting, normalizers clean and standardize text, trainers configure the learning process, and processors handle special token insertion. This modular design enables us to mix and match components to create a tokenizer tailored to our specific use case.

The models module offers several tokenization algorithms: BPE() for Byte Pair Encoding (what we use), WordPiece() for Google’s WordPiece algorithm, Unigram() for Google’s Unigram language model, and WordLevel() for simple word-level tokenization.

Each has different strengths - BPE is efficient and handles unknown words well, WordPiece is used by BERT but creates ## artifacts, Unigram is more flexible but computationally expensive, and WordLevel is simple but creates very large vocabularies.

Let us look at the code in Listing 9 for training our custom historical BPE tokenizer:

def train_tokenizer(self):
    """Train a custom tokenizer for historical English"""
    # Import the tokenizers library components
    from tokenizers import Tokenizer, models, pre_tokenizers, processors, trainers
    from tokenizers.normalizers import Sequence, NFD, StripAccents
    
    logger.info("Training custom historical tokenizer...")
    logger.info(f"Corpus: {self.corpus_path}")
    logger.info(f"Target vocabulary: {self.vocab_size:,} tokens")
    logger.info(f"Output directory: {self.output_dir}")
    
    # Initialize BPE tokenizer (not WordPiece)
    # models.BPE() creates a Byte Pair Encoding model that will learn subword patterns
    tokenizer = Tokenizer(models.BPE())
    
    # Normalizers for historical text - preserve case for better text reconstruction
    # Normalizers clean and standardize text before tokenization
    tokenizer.normalizer = Sequence([
        NFD(),           # Unicode normalization - converts characters to canonical form
        StripAccents()   # Remove accents - converts "café" to "cafe"
    ])
    
    # Pre-tokenizer for historical English - use simple whitespace splitting
    # Pre-tokenizers split text into initial segments before the main tokenization
    tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
        pre_tokenizers.WhitespaceSplit(),  # Split on whitespace
        pre_tokenizers.Punctuation()       # Split punctuation from words
    ])
    
    # Special tokens for historical English
    special_tokens = [
        "<|startoftext|>", "<|endoftext|>", "<|pad|>", "<|unk|>", "<|mask|>",
        # Historical language tokens
        "<|thou|>", "<|thee|>", "<|thy|>", "<|thine|>", "<|hast|>", "<|hath|>",
        "<|doth|>", "<|dost|>", "<|quoth|>", "<|tis|>", "<|twas|>", "<|twill|>",
        # London geography tokens
        "<|london|>", "<|thames|>", "<|westminster|>", "<|tower|>", "<|newgate|>",
        "<|southwark|>", "<|cheapside|>", "<|fleet|>", "<|ludgate|>", "<|aldgate|>",
        # Historical period tokens
        "<|tudor|>", "<|stuart|>", "<|georgian|>", "<|regency|>", "<|victorian|>",
        # Social class tokens
        "<|noble|>", "<|gentleman|>", "<|commoner|>", "<|apprentice|>", "<|yeoman|>",
        # Professional tokens
        "<|apothecary|>", "<|coachman|>", "<|chimneysweep|>", "<|baker|>", "<|butcher|>"
    ]
    
    # BPE trainer configuration
    # The trainer defines how the BPE algorithm learns from our corpus
    trainer = trainers.BpeTrainer(
        vocab_size=self.vocab_size,        # Target vocabulary size (30,000 tokens) - balanced between coverage and efficiency
        special_tokens=special_tokens,     # Pre-defined tokens that are always included
        min_frequency=2,                   # Minimum frequency prevents vocabulary pollution from OCR errors
        show_progress=True,                # Display training progress
        # Removed continuing_subword_prefix="##" to eliminate WordPiece-style artifacts
        # This ensures pure BPE tokenization without ## symbols in generated text
        initial_alphabet=["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m", 
                        "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z",
                        "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M",
                        "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"]
    )
    
    # Train the tokenizer on our historical corpus
    # This is where the BPE algorithm learns the optimal subword patterns
    tokenizer.train([str(self.corpus_path)], trainer)
    
    return tokenizer

Listing 9: Custom Tokenizer Training Function

3.2.2 Tokenization Architecture Decisions

Our custom historical tokenizer necessitated several critical design decisions to handle historical English effectively. We evaluated multiple tokenization approaches including Byte Pair Encoding (BPE) ( Sennrich et al., 2016 ), WordPiece ( Schuster & Nakajima, 2012 ), Unigram Language Model ( Kudo, 2018 ), SentencePiece ( Kudo & Richardson, 2018 ), and traditional character-level and word-level tokenization. Each approach has distinct trade-offs: BPE produces clean subwords without special markers (used by GPT models), WordPiece adds ## prefixes that contaminate generated text (used by BERT), Unigram uses probabilistic modeling but is computationally expensive, SentencePiece treats text as raw bytes and excels at multilingual scenarios, while character-level and word-level tokenization either produce impractically long sequences or massive vocabularies.

For historical text generation, BPE provides the optimal balance of clean output, efficient training, and effective vocabulary coverage, as demonstrated by Radford et al., 2019 and Mielke et al., 2021 . We also preserve case throughout tokenization, since historical text often uses capitalization for semantic meaning (e.g., “Thou” vs. “thou”), and include over 150 carefully designed special tokens that capture historical language patterns, London geography, and social context. This combination ensures our tokenizer can effectively learn and generate authentic historical language while maintaining computational efficiency.

3.3 Special Token Design: Capturing Historical Language Patterns

Historical English contains linguistic patterns, vocabulary, and cultural references that are no longer present in modern English. Standard tokenizers fragment these patterns, destroying the semantic meaning crucial for historical text generation. The solution here was to design 150 special tokens that capture the essence of historical English, organized into strategic categories that reflect the linguistic and cultural structure of 1500-1850 English.

def create_special_tokens() -> list:
    """Create special tokens for historical English"""
    special_tokens = [
        # Basic control tokens
        "<|startoftext|>", "<|endoftext|>", "<|pad|>", "<|unk|>", "<|mask|>",
        
        # Historical language tokens
        "<|thou|>", "<|thee|>", "<|thy|>", "<|thine|>",  # Second person pronouns
        "<|hast|>", "<|hath|>", "<|doth|>", "<|dost|>",  # Archaic verb forms
        "<|quoth|>", "<|tis|>", "<|twas|>", "<|twill|>", # Common contractions
        
        # London geography tokens
        "<|london|>", "<|thames|>", "<|westminster|>", "<|tower|>", "<|newgate|>",
        "<|southwark|>", "<|cheapside|>", "<|fleet|>", "<|ludgate|>", "<|aldgate|>",
        
        # Historical period tokens
        "<|tudor|>", "<|stuart|>", "<|georgian|>", "<|regency|>", "<|victorian|>",
        
        # Social and professional tokens
        "<|noble|>", "<|gentleman|>", "<|commoner|>", "<|apothecary|>", "<|coachman|>",
        "<|merchant|>", "<|court|>", "<|jury|>", "<|verdict|>", "<|church|>", "<|parish|>"
    ]
    
    return special_tokens

Listing 10: Special Tokens Creation Function

3.3.1 Token Category Analysis

Our special token vocabulary spans ten carefully curated categories, each designed to capture essential aspects of historical London life. The largest categories focus on Historical Language (25 tokens) and London Geography (20 tokens), providing the linguistic and spatial foundation for authentic historical text generation. These tokens capture archaic pronouns like "thou" and "thee," along with specific London locations like "Thames" and "Newgate" that were central to historical narratives.

The remaining categories address the social, professional, and cultural dimensions of historical society. Social Class and Professional tokens (35 tokens combined) reflect the highly stratified nature of historical London, enabling accurate dialogue between nobles, commoners, and various tradespeople. Legal and Judicial tokens support court proceedings from the Old Bailey, while Religious tokens capture the central role of faith in historical society. Temporal, Currency, and Transportation tokens (35 tokens combined) provide the temporal, economic, and logistical context that makes historical narratives authentic and believable.

3.3.2 Special Token Categories Visualization

Let us visualize the special token categories and their relationships as shown below. These special tokens enable the model to understand and generate authentic historical language. Without them, the model would fragment historical concepts into meaningless subwords, losing the cultural and linguistic context that makes historical text generation both challenging and rewarding.

graph LR
    A[🔤 Special Tokens<br/>150+ Total] --> B[📜 Historical Language<br/>25 tokens]
    A --> C[🏛️ London Geography<br/>20 tokens]
    A --> D[⏰ Historical Periods<br/>10 tokens]
    A --> E[👥 Social Classes<br/>15 tokens]
    A --> F[💼 Professions<br/>20 tokens]
    A --> G[⚖️ Legal & Judicial<br/>10 tokens]
    A --> H[⛪ Religious<br/>10 tokens]
    A --> I[🕐 Temporal<br/>15 tokens]
    A --> J[💰 Currency & Measurement<br/>10 tokens]
    A --> K[🚗 Transportation<br/>10 tokens]

    B --> B1["<|thou|>, <|thee|>, <|hast|>, <|doth|>, <|quoth|>"]
    C --> C1["<|london|>, <|thames|>, <|newgate|>, <|westminster|>"]
    D --> D1["<|tudor|>, <|stuart|>, <|georgian|>, <|regency|>"]
    E --> E1["<|noble|>, <|gentleman|>, <|commoner|>, <|yeoman|>"]
    F --> F1["<|apothecary|>, <|coachman|>, <|chimneysweep|>, <|baker|>"]
    G --> G1["<|court|>, <|jury|>, <|verdict|>, <|prisoner|>"]
    H --> H1["<|church|>, <|parish|>, <|prayer|>, <|blessed|>"]
    I --> I1["<|morn|>, <|eve|>, <|season|>, <|year|>"]
    J --> J1["<|shilling|>, <|pound|>, <|yard|>, <|furlong|>"]
    K --> K1["<|coach|>, <|carriage|>, <|horse|>, <|vessel|>"]

    %% class definitions (custom palette matching your original)
    classDef cls_root fill:#e1f5fe,stroke:#81d4fa,color:#000;
    classDef cls_hist fill:#f3e5f5,stroke:#ce93d8,color:#000;
    classDef cls_geo fill:#e8f5e8,stroke:#a5d6a7,color:#000;
    classDef cls_period fill:#fff3e0,stroke:#ffe0b2,color:#000;
    classDef cls_social fill:#fce4ec,stroke:#f8bbd0,color:#000;
    classDef cls_prof fill:#f1f8e9,stroke:#c5e1a5,color:#000;
    classDef cls_legal fill:#e0f2f1,stroke:#80cbc4,color:#000;
    classDef cls_relig fill:#f9fbe7,stroke:#e6ee9c,color:#000;
    classDef cls_temp fill:#e3f2fd,stroke:#90caf9,color:#000;
    classDef cls_curr fill:#fef7e0,stroke:#ffe082,color:#000;
    classDef cls_trans fill:#f3e5f5,stroke:#e1bee7,color:#000;

    %% assign classes
    class A cls_root;
    class B cls_hist;
    class C cls_geo;
    class D cls_period;
    class E cls_social;
    class F cls_prof;
    class G cls_legal;
    class H cls_relig;
    class I cls_temp;
    class J cls_curr;
    class K cls_trans;

Figure 5: Special Token Categories and Examples

3.5 Post-Processing and Hugging Face Integration

After training our custom tokenizer, we need to make it compatible with the broader machine learning ecosystem and ensure it works properly with language model training. Raw tokenizers can only convert text to tokens and back. Still, language models require additional functionality, such as special token handling, sequence padding, and integration with popular frameworks like Hugging Face Transformers.

The challenge, though, is that language model training requires specific formatting that raw tokenizers don’t provide. For example, training sequences need to be wrapped with special start/end tokens (<|startoftext|> and <|endoftext|>), padded to consistent lengths for batch processing, and integrated with the rest of the ecosystem. In our case, we also want to utilize Hugging Face and its ecosystem, allowing us to leverage standard training scripts and model architectures. Without proper post-processing, our custom tokenizer would be incompatible with existing training infrastructure.

We add post-processing capabilities that wrap text sequences with control tokens and create Hugging Face-compatible tokenizer files, ensuring seamless integration with the broader machine learning ecosystem while preserving our historical text optimizations.

There are three key areas that we need to consider:

Understanding Post-Processing: The first step is adding a post-processor that automatically wraps every text sequence with special start and end tokens. This is crucial because language models must be able to identify where sequences begin and end during training. For example, when we tokenize "Hello world", the post-processor automatically converts it to <|startoftext|> Hello world <|endoftext|>. This template processing ensures consistent formatting across all our training data.
Hugging Face Integration: Next, we create a Hugging Face-compatible wrapper around our custom tokenizer. This wrapper maps our special tokens to the standard token types that Hugging Face expects: beginning-of-sequence (bos), end-of-sequence (eos), padding, unknown, and masking tokens. This mapping allows our custom tokenizer to work seamlessly with standard training scripts and model architectures.
Special Token Functions: Each special token serves a specific purpose in language model training. The beginning-of-sequence token indicates when a new text starts, the end-of-sequence token marks the end of the text, padding tokens ensure all sequences in a batch have the same length, unknown tokens handle words not in our vocabulary, and masking tokens are used during training for masked language modeling tasks.

The code in Listing 11 demonstrates how we implement these post-processing steps and create a Hugging Face-compatible tokenizer:

def create_huggingface_tokenizer(tokenizer: Tokenizer, max_length: int = 1024) -> PreTrainedTokenizerFast:
    """Create Hugging Face compatible tokenizer"""
    from transformers import PreTrainedTokenizerFast
    
    # Add post-processor for sequence formatting
    tokenizer.post_processor = processors.TemplateProcessing(
        single="<|startoftext|> $A <|endoftext|>",
        special_tokens=[
            ("<|startoftext|>", 1),
            ("<|endoftext|>", 0),
        ]
    )
    
    # Create Hugging Face tokenizer wrapper
    hf_tokenizer = PreTrainedTokenizerFast(
        tokenizer_object=tokenizer,
        bos_token="<|startoftext|>",
        eos_token="<|endoftext|>",
        pad_token="<|pad|>",
        unk_token="<|unk|>",
        mask_token="<|mask|>",
        model_max_length=max_length
    )
    
    return hf_tokenizer

Listing 11: Hugging Face Tokenizer Integration

Without this integration, our custom tokenizer would be incompatible with standard language model training. The post-processor ensures proper sequence formatting, while the Hugging Face wrapper enables seamless integration with existing training infrastructure and model architectures. This makes our tokenizer compatible with standard training frameworks, allowing for easy sharing and deployment.

3.6 Testing and Validation

We need to ensure the tokenizer works correctly with historical text before using it for model training. This requires testing on diverse historical samples and validating both encoding and decoding accuracy. A simple way to do this is to encode a set of historical text samples, decode them back, and check if the original text is perfectly reconstructed. We also want to verify that special tokens are used correctly in the tokenized output.

def test_historical_tokenizer(tokenizer: Tokenizer) -> dict:
    """Test the trained tokenizer on historical text samples"""
    
    test_texts = [
        "In the year of our Lord 1834, the streets of London were filled with the sounds of horse-drawn carriages.",
        "The gentleman from the country said, 'I have never seen such a sight in all my days.'",
        "The Thames flowed dark and mysterious through the heart of the city.",
        "It was the best of times, it was the worst of times."
    ]
    
    results = {'perfect_reconstruction': 0, 'special_token_usage': 0, 'failed_tests': []}
    
    for i, text in enumerate(test_texts):
        # Encode and decode text
        encoded = tokenizer.encode(text)
        decoded = tokenizer.decode(encoded.ids)
        
        # Check reconstruction accuracy
        if decoded.strip() == text.strip():
            results['perfect_reconstruction'] += 1
        else:
            results['failed_tests'].append({'index': i, 'original': text, 'decoded': decoded})
        
        # Check special token usage
        special_tokens = [token for token in encoded.tokens if token.startswith('<|') and token.endswith('|>')]
        if special_tokens:
            results['special_token_usage'] += 1
    
    return results

Test Results:

Perfect Reconstruction: 99%+ accuracy on test cases
Special Token Usage: 80%+ of test cases use special tokens
Average Compression Ratio: ~0.3 tokens per word (highly efficient)
Success Rate: 99%+ for historical text samples

It is essential to conduct comprehensive testing to ensure the tokenizer operates reliably. In our case, the test cases cover different historical periods, writing styles, and linguistic patterns, giving us confidence that the tokenizer can handle the full range of historical text in our corpus. For a real-world LLM, this is, of course, more complex and would need to cover a broader set of areas.

Tokenizer Performance Validation

Not surprisingly, our custom tokenizer significantly outperforms standard approaches on historical text, as demonstrated by comprehensive metrics that compare it to GPT-2’s tokenizer, as shown in the table below. These metrics indicate that our custom tokenizer significantly outperforms standard approaches for historical text. The improved compression ratio and reconstruction accuracy ensure that the model learns from authentic historical language rather than tokenization artifacts, which is crucial for generating coherent and historically accurate text.

Metric	Standard GPT-2	Our Custom Tokenizer	Improvement
Vocabulary Size	50,257 tokens	30,000 tokens	40% smaller
Special Tokens	4 tokens	150+ tokens	37x more
Compression Ratio	~0.4 tokens/word	~0.3 tokens/word	25% better
Reconstruction Accuracy	95%	99%+	4% better
Historical Language Support	Poor	Good	N/A

These metrics validate that our 30K token vocabulary provides optimal coverage for historical text while remaining manageable for small language models. The 150+ special tokens capture linguistic patterns of 1500-1850 English, and the 25% better compression ratio means historical text is represented more efficiently, allowing the model to process longer sequences. The 99%+ reconstruction accuracy ensures no information is lost during tokenization, while excellent performance on archaic vocabulary, period-specific terminology, and London geography demonstrates the tokenizer’s effectiveness for historical language modeling.

3.8 Implementation and Usage

The complete tokenizer implementation, including training scripts, testing utilities, and validation tools, is available in the helloLondon GitHub repository . The repository provides:

Training Code: Complete BPE tokenizer training with configurable vocabulary sizes and special token definitions.
Testing Utilities: Comprehensive validation tools for testing tokenizer performance on historical text
Integration Examples: Ready-to-use code for incorporating the tokenizer into your own projects
Documentation: Detailed usage guides and API references

This implementation demonstrates how to build production-ready tokenizers for specialized domains, with particular focus on historical language processing and integration with modern ML frameworks.

4. Current Limitations

This project is designed as a learning exercise for those new to AI and LLM development. While we’ve built a functional system that demonstrates core concepts, this is not production-ready code and has several limitations that would need to be addressed for real-world deployment:

Data Scale & Quality:

Corpus size: Our 500M character corpus is tiny compared to production LLMs, which typically use 100x-1000x more data (50B-500B+ characters). This limits the model’s ability to learn diverse patterns and reduces the quality of generated output.
Source diversity: With only 218 sources, we lack comprehensive historical coverage across the 1500-1850 span, potentially missing important linguistic evolution patterns and regional variations.
Geographic bias: Heavy focus on London may not accurately represent broader historical English patterns from other regions, limiting the model’s generalizability.
Bias detection: We lack systematic approaches to identify or mitigate historical biases in the data, which could lead to the model perpetuating outdated or problematic language patterns.
Quality assessment: Our cleaning pipeline, while effective for common issues, overlooks many edge cases and artifacts that would require more sophisticated ML-based quality assessment in production.

Tokenizer & Model Architecture:

Vocabulary size: Our 30K token vocabulary is small compared to modern models (which often use 50K-100K+ tokens), limiting the model’s ability to represent diverse vocabulary efficiently.
Special tokens: The 150+ special tokens are manually curated rather than learned from data, which may miss important patterns that data-driven approaches would discover.
Context length: The 1024 token context window is very short compared to modern models (which often use 4K-32K+ tokens), limiting the model’s ability to maintain coherence in longer texts.
Language support: No support for other languages or historical variants beyond English, significantly limiting the model’s applicability.
Tokenization approach: While our BPE approach is clean and avoids WordPiece artifacts, it may not be optimal for all historical text patterns and could benefit from more sophisticated techniques.

Technical Infrastructure:

Error handling: Basic error handling with limited logging and monitoring makes it difficult to debug issues and track system health in production.
Testing: Minimal test coverage that excludes edge cases means many potential failure modes remain undetected until they occur in production.
Performance: No optimization for speed, memory, or distributed processing, making the system unsuitable for production-scale deployment.
Data management: Lacks data versioning and reproducibility guarantees, making it difficult to track changes and reproduce results across different environments.
Security: No security considerations for data handling and model deployment, creating potential vulnerabilities for sensitive historical data.
Compliance: Missing compliance considerations for GDPR, data privacy, and regulatory requirements, which are essential for production deployment.
Monitoring: No production monitoring, alerting, or observability features, making it impossible to detect and respond to issues in real-time.

These limitations are intentional trade-offs made to keep the project manageable and focused on core learning objectives, but they represent significant gaps for production deployment.

4.3 What You’d Need for Production

Data Engineering and Legal Framework

Production systems require 100x-1000x more data from diverse sources, with ML-based quality assessment, bias detection, and filtering that goes far beyond our simple heuristics. You’d need robust ETL pipelines with proper error handling and monitoring, as well as a comprehensive legal framework for copyright clearance, data licensing, and compliance management, which we haven’t addressed.

Model Architecture and Training

Meaningful historical language understanding would require models with over 1 billion parameters, utilizing sophisticated training techniques, regularization, and optimization. You’d need a comprehensive evaluation on diverse historical text tasks and domain-specific fine-tuning capabilities that our current system doesn’t support.

Infrastructure and Operations

Production deployment requires a multi-GPU, multi-node distributed training infrastructure, production-grade model serving with load balancing and scaling, comprehensive monitoring and alerting systems, and end-to-end security for both data and model protection—none of which our learning-focused system currently provides.

This progression from data → tokenizer → training → deployment provides a complete methodology for building specialized historical language models.

5. Resources and Further Reading

GitHub Repository: github.com/bahree/helloLondon - Complete source code for data collection and tokenizer training
Part 1: Building LLMs from Scratch - Part 1 - Quick start and overview
Documentation: Complete guides in the 08_documentation/ folder covering every aspect of the project
Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts

6. Summary

This post represents Part 2 of our learning journey into the fundamentals of LLM development. While we’ve built a functional data collection and tokenization system demonstrating core concepts, the real value lies in understanding:

Data flow from raw sources to training-ready corpora
Tokenization impact on model performance across different approaches
Challenges in processing historical and domain-specific text
Trade-offs between quality, scale, and complexity
Debugging and improvement strategies for encountered problems

The limitations we’ve identified are great learning opportunities. Every production LLM started as a learning project, and every limitation teaches you something new about how these systems work. This foundation prepares us for the next phase of our journey.

Ready for Part 3? Part 3 will cover the custom GPT architecture, GPU optimization strategies, and training infrastructure that transforms our clean data and custom tokenizer into working language models—while maintaining the same educational focus on understanding the fundamentals.

🧱 Series Posts: Part 1 – Using the Published Historical Models | Part 2 (this post) | Part 3 – Training Architecture & GPU Optimization | Part 4 – Evaluation & Deployment

🏛️How to build a Large Language Model from Scratch - Part 1

Tue, 23 Sep 2025 00:00:00 +0000

TL;DR

In this post, I show how to build a working LLM from scratch and show a complete end-to-end pipeline from data gathering to training to deployment of a language model. For this project I concentrate on Old English and only related to London, using historical London texts (1500-1850). To show the flexibility, I built two language models which are identical in architecture and the only differs is their size and parameters (117M vs 354M).

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need much larger datasets, more sophisticated infrastructure, and additional considerations not covered here.

This guide shows you how to monitor training progression, perform rapid evaluations, test models from both PyTorch checkpoints and published Hugging Face repositories, and ultimately publish your own - supported by complete code, live model artifacts, and educational inference tooling.

4-Part Series:

Part 1 (this): Quick start, inference, and overview
Part 2: Data collection and custom tokenizers
Part 3: Model architecture and GPU training
Part 4: Evaluation and deployment

1. Overview

Train AI models on 1500-1850 London texts. Complete 4-part series covering data collection, training, and deployment. Part 1: Quick start and overview.

📖 Want to understand the core LLM concepts? This series focuses on implementation and hands-on building. For a deeper understanding of foundational concepts like tokenizers, prompt engineering, RAG, responsible AI, fine-tuning, and more, check out my book Generative AI in Action .

You can learn more about the book → by clicking here 📘.

1.1 What was built?

I found many folks don’t understand what it entails to build an LLM, and where we do have guides, they only share piecemeal elements and nothing that is comprehensive for someone who is new to this. There are more detailed guides on fine-tuning existing models, but not much on the complete development pipeline. This series outlines that by walking through the process of creating specialized language models trained exclusively on historical London texts from 1500 to 1850.

I am mostly doing this for my own learning, and also sharing what I can. Many work-related details, for obvious reasons, I cannot share and discuss, but some small pet projects like this embody the same sentiment.

The helloLondon Historical Language Models represent a complete end-to-end implementation, from data collection through deployment. Rather than fine-tuning existing models, I chose to train from the ground up to eliminate modern biases and create models that genuinely understand historical language patterns, cultural contexts, and period-specific knowledge.

Two Model Variants I built two identical models with the same architecture, tokenizer, and training process. The only difference is the number of parameters: an SLM (117M parameters) optimized for learning and resource-constrained environments, and a Regular model (354M parameters) designed for higher-quality generation.

Both use identical code with different configuration files, allowing you to understand the impact of model size on performance and choose the right variant for your needs.

Model	Parameters	Iterations	Training Time*	Use Case
SLM (Small)	117M	60,000	~8-12 hours	Fast inference, resource-constrained
Regular (Full)	354M	60,000	~28-32 hours	High-quality generation

Note: Technically speaking, both these models can be called classified as SLMs given they are 117M and 354M parameters; however, for the sake of this project, I call the smaller of the two the SLM and the other regular.

1.2 Core Pipelines

The complete development pipeline encompasses multiple critical stages that transform raw historical texts into working language models. The process starts with data collection, where we systematically gather and filter over 218 historical London sources spanning 1500–1850. This process ensures we capture authentic period language while minimizing modern biases that could contaminate our models.

Next, we develop a custom tokenization system specifically designed for historical English. This involves training a domain-specific tokenizer with a 30,000-token vocabulary plus 150+ special tokens that capture period language patterns, archaic spellings, and historical terminology that modern tokenizers often miss.

The model architecture phase implements GPT-style causal language models entirely from scratch, creating two variants with 117M and 354M parameters, respectively. Both models share identical architecture and training processes, allowing for direct comparison of performance versus computational requirements.

Our training infrastructure leverages modern multi-GPU training with Distributed Data Parallel (DDP), comprehensive checkpointing for restart resilience, and real-time monitoring through Weights & Biases. This ensures reliable training even across extended periods and hardware failures.

Evaluation goes beyond standard metrics to include historical accuracy probes, perplexity tracking, qualitative generation review, and early failure detection. We specifically test how well our models understand historical context, period-appropriate language, and London geography.

Finally, deployment includes publishing models to Hugging Face alongside unified local and cloud inference scripts, making the models immediately accessible to researchers and developers worldwide.

1.3 Hands-On Experience

Every aspect of this project is designed for practical implementation and learning. The working code covers every stage from data collection through tokenizer training, model training, evaluation, and publishing - all fully implemented and documented with clear instructions and examples.

I already have both the models published on Hugging Face; which allows for Live models are immediately available for use, allowing you to test published checkpoints instantly or retrain from scratch with a single command. This dual approach lets you either jump straight into experimentation or understand the complete development process.

The project works with real data - over 500 million characters of authentic historical English from 1500–1850, carefully filtered to minimize modern bias while preserving the rich linguistic patterns of the period. This is using genuine historical texts that provide authentic training material.

Everything is well-structured with clear documentation, error handling, reproducible configurations, and automated publishing workflows. The codebase follows good development practices, making it suitable for learning LLM development concepts.

This series is structured to take you through the complete LLM development pipeline:

Part	Focus	Description
Part 1 (this post)	Quick start and end-to-end overview	Use published models, understand the complete pipeline, and get hands-on experience with working code and live models. The intent is that if you want to build this, you can follow the instructions and get a model in the end. If you want to understand more of the inner workings and details, then those will be covered in the subsequent blog posts.
Part 2	Data collection and custom tokenization	Deep dive into gathering 218+ historical sources, cleaning pipelines, and building specialized tokenizers for historical language patterns.
Part 3	Model architecture and training infrastructure	Technical implementation of custom GPT architectures, multi-GPU training, checkpointing, and performance optimization.
Part 4	Evaluation and deployment	Comprehensive testing frameworks, historical accuracy assessment, and deployment to Hugging Face.

For this first part, you have two paths to choose from based on your goals and available time:

Option 1: Quick Start with Published Models - Jump straight into using the pre-trained models on Hugging Face for immediate testing and exploration. Perfect if you want to see results quickly and aren’t concerned with the technical implementation details.
Option 2: Build from Scratch - Dive deep into the complete codebase and build your own historical language model from the ground up. Ideal if you want to understand every aspect of the pipeline and learn how to create specialized LLMs.

Let us start with option 1 - use the models.

2. Use the models - Try it now using Hugging Face

If you just want to get going and use the models and kick tires, the models are live on Hugging Face and ready to use.

SLM Model (117M parameters): 💡 https://huggingface.co/bahree/london-historical-slm
Regular Model (354M parameters): 💡 https://huggingface.co/bahree/london-historical-llm

In addition, you can also explore the complete codebase and build your own historical language model from scratch. The entire pipeline is documented with working code, training scripts, and deployment guides, and is available on GitHub:

Github Repo 💻 –> ⚙️ github.com/bahree/helloLondon .

If you want to quickly test the published models on Hugging Face (HF), you can do so in two ways: quick automated tests or interactive mode. This is the easiest way to get started and show that the models are fully working. You can either clone the repo and run the scripts or use the Python code snippet below.

If you don’t have a development environment set up, you can follow the instructions in the GitHub repo to set up a conda environment with all dependencies. And just for the local testing, you can use CPU only, but for interactive mode, a GPU is recommended. Finally, you will need at a minimum the following Python packages shown in Listing 1. Note, these are also called out on the Hugging Face model page.

python -m pip install -U pip setuptools wheel
python -m pip install "transformers[torch]" accelerate safetensors

Listing 1: Install Required Dependencies

Note: It is recommended to use a virtual environment or conda environment to avoid dependency conflicts. See the GitHub repo for complete setup instructions.

If you don’t have the code repo yet, you can run the commands in Listing 2 directly and run inference from Hugging Face.

Python Code:

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the published SLM model
model_name = "bahree/london-historical-slm"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Generate historical text
prompt = "In the year of our Lord 1834, I walked through the streets of London and witnessed"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=100,
    do_sample=True,
    temperature=0.3,
    top_p=0.9,
    top_k=20,
    repetition_penalty=1.2
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Listing 2: Load and Test Published Model

2.1 Local Testing with the Complete Codebase

Now that you’ve seen the models work directly from Hugging Face, let’s explore the complete development experience by working with the actual codebase. This section walks you through testing the models locally using the same infrastructure that was used to train them.

The helloLondon repository contains everything needed to reproduce the entire pipeline - from data collection through model deployment. By running these tests locally, you’ll get hands-on experience with the inference scripts and understand how the models integrate with the broader development workflow.

The following examples assume you’ve cloned the repository and are running from the root directory. All scripts are designed to work out-of-the-box with the published models, giving you immediate access to the same testing infrastructure used during development. You can test the models using Listing 3 or Listing 4.

# Test SLM model (117M parameters)
python 06_inference/test_published_models.py --model_type slm

# Test Regular model (354M parameters)  
python 06_inference/test_published_models.py --model_type regular

Listing 3: Test SLM Model

There is also an interactive mode where you can type in your own prompts and see the model generate text.

Interactive Testing:

# SLM model - Interactive mode
python 06_inference/inference_unified.py --published --model_type slm --interactive

# Regular model - Interactive mode
python 06_inference/inference_unified.py --published --model_type regular --interactive

# Single prompt testing
python 06_inference/inference_unified.py --published --model_type slm --prompt "In the year 1834, I walked through the streets of London and witnessed"
python 06_inference/inference_unified.py --published --model_type regular --prompt "In the year 1834, I walked through the streets of London and witnessed"

Listing 4: Interactive Mode Testing

If everything works, you should see output similar to the following for the SLM model:

Example Output ( Hugging Face SLM Example):

🧪 Testing SLM Model: bahree/london-historical-slm
============================================================
📂 Loading model...
✅ Model loaded in 8.91 seconds
📊 Model Info:
   Type: SLM
   Description: Small Language Model (117M parameters)
   Device: cuda
   Vocabulary size: 30,000
   Max length: 512

--- Test 1/10 ---
Prompt: In the year 1834, I walked through the streets of London and witnessed
Generated: a scene in which some of those who did not incline to come in contact with him took part in his discourse. It was on this occasion that I perceived that he had been engaged in some new business connected with the house, but for some days it had not taken place, nor did he appear so desirous of pursuing any further display of interest .....
Time: 5.75s

Notice how the model captures:

Period-appropriate language (“thank ’ee kindly,” “bade me go,” “spectacles”)
Historical dialogue patterns (formal speech, period-appropriate contractions)
Historical context (West Indies, poor rates, needle work, pocket-book)
Authentic historical narrative (detailed scene setting, period-appropriate social interactions)

Now that we have tried using the model, let’s explore option 2 and see how we can build it. Once you’ve built your own model, you’ll be able to test it using the checkpoints saved during training - see section 7.4 for detailed checkpoint testing instructions.

3. Build the models - From Scratch

Building a language model from scratch is both an art and a science - requiring careful orchestration of data, architecture, and training to create something that can genuinely understand and generate historical text. Unlike fine-tuning existing models, training from scratch gives us complete control over every aspect of the model’s knowledge and behavior.

The journey from raw historical documents to a working language model involves six critical phases, each building upon the previous one. The flowchart below illustrates this complete end-to-end pipeline, showing how we transform 218+ historical sources into two specialized models that can generate authentic medieval London text.

graph TD
    A[📚 Historical Data Collection<br/>218+ sources, 1500-1850] --> B[🧹 Data Cleaning & Processing<br/>Text normalization, filtering]
    B --> C[🔤 Custom Tokenizer Training<br/>30k vocab + 150+ special tokens]
    C --> D[🏋️ Model Training<br/>Two Identical Models<br/>SLM: 117M / Regular: 354M]
    D --> E[📊 Evaluation & Testing<br/>Historical accuracy, ROUGE, MMLU]
    E --> F[🚀 Deployment<br/>Hugging Face + Local Inference]
    
    G[📖 Building a Custom LLM] --> A
    
    F --> L[🎯 Use Cases<br/>Historical text generation<br/>Educational projects<br/>Research applications]
    
    style A fill:#e1f5fe
    style D fill:#f3e5f5
    style F fill:#e8f5e8
    style G fill:#fff3e0

Figure 1: Complete LLM Development Pipeline

Now that we have a bird’s eye view of the complete pipeline, let us get into the details and build the model from scratch. I am going to walk you through the complete process step-by-step.

I am also going to assume you have a basic understanding of Python, PyTorch, and command-line operations and have a more recent dev setup, including a relatively modern GPU (NVIDIA RTX 3060 or better recommended). For the sake of simplicity, I will show commands for Linux/macOS, but Windows users can easily adapt them.

Again, as a reminder, the ⚙️ GitHub repo has all the code and instructions you need to get started. You can clone the repo and follow along.

4. Environment and Configuration Setup

The foundation of any successful machine learning project lies in proper environment setup and configuration. This step involves creating a virtual environment, installing dependencies, and configuring the project structure. Understanding the key configuration files, directory organization, and overall project architecture is crucial - these elements form the backbone of the entire training process. Taking time to get this right upfront prevents countless headaches and debugging sessions later, ensuring smooth execution through all subsequent phases.

4.1 Key Configuration Files

config.py: Central configuration system (paths, training settings, tokenizer config)
01_environment/setup_environment.py: Environment setup script (reads from config.py)
requirements.txt: Python dependencies (auto-generated by setup script)

4.2 Important Directories (Created by Setup)

helloLondon/: Virtual environment directory
data/london_historical/: Historical text data storage
09_models/checkpoints/: Model checkpoints during training
09_models/tokenizers/: Custom tokenizer storage

Now that we have that out of the way, let us run the setup commands as shown in Listing 5. This will clone the repo, set up the environment, and install all dependencies. For this to work you will already have git, python, and python3-venv installed. If you don’t have these, please install them first.

PS: See the Training QuickStart guide in the GitHub repo for more details.

# Clone and setup environment
git clone https://github.com/bahree/helloLondon/
cd helloLondon
python 01_environment/setup_environment.py
source activate_env.sh

Listing 5: Clone and Setup Repository

As you run the setup script, you should see output similar to the images shown below; the script will create a virtual environment, install dependencies, and set up necessary directories. And then you can activate the environment using the source activate_env.sh command.

Figure 2: Environment setup process - Step 1 of 3 showing virtual environment creation

Figure 3: Environment setup process - Step 2 of 3 showing dependency installation

Figure 4: Environment setup process - Step 3 of 3 showing final configuration

Now that the configuration and environment are set up, we can validate them by running the following command. This will check if everything is working and you have the necessary dependencies installed.

When one activates the environment using source activate_env.sh, you will see it in the console as shown below.

The default environment name is called helloLondon. If you want to change the environment name from helloLondon to something else, you can modify the venv_name field in environment_config.json before running the setup script. This will create a virtual environment with your preferred name.

python3 -c "
from config import config
print('🔧 Configuration Overview')
print('=' * 50)
print(f'Project Root: {config.project_root}')
print(f'Data Directory: {config.london_historical_data}')
print(f'Tokenizer Directory: {config.london_tokenizer_dir}')
print(f'Checkpoints Directory: {config.checkpoints_dir}')
print(f'Virtual Environment: {config.project_root}/helloLondon')
print(f'Vocabulary Size: {config.tokenizer_config[\"vocab_size\"]:,} tokens')
print(f'Special Tokens: {len(config.tokenizer_config[\"special_tokens\"])} tokens')
print(f'SLM Model: {config.slm_config[\"model_name\"]}')
print(f'Training Epochs: {config.slm_config[\"num_epochs\"]}')
print(f'Batch Size: {config.slm_config[\"batch_size\"]}')
print(f'Max Length: {config.slm_config[\"max_length\"]}')
print('\\n🎯 Configuration looks good!')
"

Listing 6: Validate Configuration

The following directory structure will be generated after executing the setup script. Please note that certain directories will remain empty until the data collection and training processes are initiated.

helloLondon/
├── 📁 data/london_historical/          # Historical text data
│   ├── 📄 london_historical_corpus_comprehensive.txt  # Final training corpus
│   ├── 📁 downloads/                   # Raw downloaded data
│   ├── 📁 processed/                   # Cleaned and processed text
│   └── 📁 metadata/                    # Data collection metadata
├── 📁 09_models/
│   ├── 📁 checkpoints/                 # Regular model checkpoints (354M)
│   │   ├── 📄 checkpoint-44000.pt
│   │   ├── 📄 checkpoint-47000.pt
│   │   ├── 📄 checkpoint-51000.pt
│   │   ├── 📄 checkpoint-59000.pt
│   │   └── 📄 checkpoint-60001.pt
│   ├── 📁 checkpoints/slm/             # SLM model checkpoints (117M)
│   │   ├── 📄 checkpoint-52000.pt
│   │   ├── 📄 checkpoint-60000.pt
│   │   └── 📄 checkpoint-60001.pt
│   └── 📁 tokenizers/london_historical_tokenizer/  # Custom tokenizer
│       ├── 📄 tokenizer.json           # Tokenizer configuration
│       ├── 📄 vocab.json               # Vocabulary mapping
│       └── 📄 merges.txt               # BPE merge rules
├── 📁 helloLondon/                     # Virtual environment
└── 📁 logs/                            # Training logs and WandB data

Listing 7: Project Directory Structure

Prerequisites: Before proceeding with the following steps, please verify the following requirements:

Storage: Minimum 20GB of free disk space and stable internet connectivity for data acquisition

Hardware: GPU with 8GB+ VRAM for SLM training, 16GB+ VRAM for Regular model training. Cloud users should select appropriate instance types

Experiment Tracking (Optional but highly recommended): Weights & Biases account with WANDB_API_KEY environment variable configured for comprehensive training monitoring

Dependencies: Required data processing libraries (nltk, beautifulsoup4, etc.) will be automatically installed via the setup script

5. Data Collection

The foundation of any language model lies in its training data. For our historical London models, we’ve built a comprehensive data collection system that sources authentic text from 218+ historical sources spanning 1500-1850 - a remarkable 350-year window of London’s linguistic evolution. This isn’t just about downloading files; it’s about curating a high-quality corpus that captures the authentic voice of historical London.

Our data collection pipeline automatically processes multiple formats (PDFs, HTML, XML, plain text) from diverse sources, including Project Gutenberg classics, Old Bailey trial records, London Lives manuscripts, and British History Online archives. The system includes sophisticated quality control measures: language detection to filter non-English content, OCR artifact correction, duplicate detection, and historical period validation to ensure every text genuinely represents the target era.

The result? A curated corpus of 500M+ characters of authentic historical English text, ready to train models that understand not just the words, but the cultural context, social dynamics, and linguistic patterns of 18th and 19th-century London. Of course, you can always add your own data sources if you have them, and the system is designed to be extensible.

We can kick off the data collection process using Listing 8. This will be run from the project root directory.

# Download historical data with advanced filtering
python 02_data_collection/historical_data_collector.py --max_sources 100

# The system automatically filters:
# - Non-English content (Arabic, Chinese, etc.)
# - Poor OCR quality scans and gibberish
# - Advertisement-heavy commercial content  
# - Duplicate content and empty files
# - Special handling for Project Gutenberg classics

Listing 8: Download Historical Data

This process may take some time, depending on your internet speed, the number of sources you choose to download and your system’s performance. For me, on a very fast internet connection and a powerful machine this took typically 2-4 hours for downloading, and processing the full dataset. The script will save the cleaned and processed data in the data/london_historical/ directory, creating a comprehensive historical corpus.

The data collection process creates a comprehensive historical corpus with the main training file london_historical_corpus_comprehensive.txt containing 270M+ characters (~258MB) of authentic historical text. The complete data directory spans approximately 1.2GB, including 521MB of raw downloaded sources, 263MB of processed and cleaned content, and 126MB of tokenized training sequences ready for model training. The image below shows the data collection in progress.

Figure 5: Data collection process in progress showing real-time processing of historical sources

The final corpus represents one of the largest collections of historical London text ever assembled for language model training, with authentic content spanning 350 years of linguistic evolution. The two images below show an example of one of my runs, one of them showing the final output of the data cleaning and outlining the statistics. And the second one shows the size of the data on disk.

Figure 6: Data collection summary showing final statistics and corpus composition

The total size at the end of the data. Note this does not include the Old Bailey and London Lives data.

Figure 7: Total data size on disk showing comprehensive historical corpus

Figure 7: Total data size on disk showing the complete historical corpus storage requirements

Now that we have our data and have cleaned it. Let us build a custom tokenizer.

6. Train Custom Tokenizer

With our cleaned historical corpus ready, we now need to create a custom tokenizer specifically designed for historical English. Standard tokenizers like GPT-2 are optimized for modern text and fail catastrophically with historical language - treating archaic words like “quoth” and “hast” as multiple subword fragments, losing both meaning and efficiency.

Our custom tokenizer uses Byte Pair Encoding (BPE) with a 30,000 vocabulary size and 150+ carefully designed special tokens that understand:

Historical Language: Archaic pronouns (<|thou|>, <|thee|>), verbs (<|hast|>, <|doth|>), and expressions (<|verily|>, <|forsooth|>)
London Geography: Landmarks (<|thames|>, <|newgate|>, <|tower|>), streets (<|cheapside|>, <|fleet|>), and districts (<|southwark|>, <|westminster|>)
Historical Context: Period markers (<|tudor|>, <|stuart|>, <|georgian|>), social classes (<|noble|>, <|commoner|>), and professions (<|apothecary|>, <|coachman|>)

This specialized vocabulary ensures that common historical terms remain as single tokens rather than being fragmented, dramatically improving both training efficiency and text generation quality. We can kick off the tokenizer using Listing 9. Again, this will be run from the project root directory.

# Train historical tokenizer (30k vocabulary)
python 03_tokenizer/train_historical_tokenizer.py

Listing 9: Train Historical Tokenizer

The training process analyzes our 270M+ character corpus to learn optimal token boundaries, creating a tokenizer that understands the linguistic patterns of 1500-1850 English. The result is a highly efficient tokenizer with a compression ratio of ~0.3 tokens per character and 99%+ reconstruction accuracy - essential for training models that can generate authentic historical text.

Once the training is finished (and usually it is pretty quick - just a few minutes for our data size), we run a quick sanity test as the image below shows.

Figure 8: Custom tokenizer training progress showing vocabulary learning

Figure 8: Custom tokenizer training process showing BPE algorithm learning optimal token boundaries

Note that in testing, we might see a warning that the reconstruction differs; this is only because of the alphabet case being different and is expected. You can ignore this. An example of this is shown below.

Figure 9: Tokenizer reconstruction warning during training process

Figure 9: Tokenizer reconstruction warning showing expected case normalization differences

Why the “Reconstruction differs” warning is actually beneficial:

The reconstruction differences you see are not errors - they’re the tokenizer working exactly as designed for optimal language model training. The tokenizer uses Byte Pair Encoding (BPE), which breaks complex words into smaller, reusable subword units (like “Bourgh” → “bour ##gh”), and normalizes text to lowercase to reduce vocabulary size. These “differences” are actually features that make the tokenizer more efficient and the resulting language model more capable of generating authentic historical text.

📖 For detailed technical explanation: Part 2 of this series covers the complete tokenizer architecture, BPE implementation, special token design, and why these reconstruction differences are essential for optimal language model training.

Now that we have our data and the tokenizer is ready, it is time to train the model.

7. Train the Model

With our cleaned historical corpus and custom tokenizer in place, we can now train our language models. The training system is designed to build two identical models with different parameter counts, allowing you to choose between speed (SLM) and quality (Regular model) based on your needs.

Training Architecture: Both models use a custom GPT architecture specifically optimized for historical text, featuring sophisticated attention mechanisms that understand the complex relationships in historical language. The system includes automatic GPU detection, multi-GPU support, and comprehensive monitoring to ensure optimal training performance.

Training Process: The training system implements modern optimization techniques, including dynamic learning rate scheduling, automatic checkpointing, and real-time experiment tracking via WandB. The entire process is automated with intelligent configuration that adapts to your hardware setup, whether you’re using a single GPU or multiple GPUs for distributed training.

Performance Optimization: The system includes precision optimization (TF32, AMP) and memory management specifically tuned for historical text processing. Training typically takes 7-8 hours for the SLM and 28-32 hours for the Regular model on modern hardware, with comprehensive monitoring to track progress and identify any issues. Note, this time can vary significantly based on your hardware. The times mentioned here are based on dual NVIDIA A30s.

📖 For detailed technical implementation: Part 3 of this series covers the complete model architecture, GPU configuration, training infrastructure, and performance optimization strategies in detail.
🧪 Ready to test your checkpoints? Once training completes, see section 7.4 for comprehensive instructions on testing your trained model checkpoints.

7.1 SLM Training

To kick off the training, the code is quite simple, as shown in Listing 10. Again, this would be from the project root folder. In my case, I am using torchrun --nproc_per_node=2 because I have dual GPUs and I want to use both. If you only have a single GPU, you can just run the automatic GPU detection script. The train_model_slm.py script specifically trains the SLM (Small Language Model) with 117M parameters.

Option A: Train SLM (117M parameters) - Faster, Good for Testing

# Clean any existing tokenized data
rm -rf data/london_historical/tokenized_data/

# Automatic GPU Detection (Recommended)
cd 04_training
./launch_slm_training.sh

# Manual Multi-GPU training
torchrun --nproc_per_node=2 04_training/train_model_slm.py --data_dir data/london_historical

Listing 10: Train SLM Model

Note: The first line rm -rf data/london_historical/tokenized_data/ cleans any existing tokenized data to ensure a fresh start. This is important because the training system caches tokenized data for efficiency, and we want to ensure it uses the latest corpus and tokenizer settings rather than potentially outdated cached data. You want to do this only if you have more updated data from the previous steps.

Once the training starts, you will see a similar output as the one shown below.

Figure 10: Model training initialization showing configuration and setup

Figure 10: Model training initialization showing tokenization and GPU setup process

Note the Tokenizing corpus line - this will take some time, depending on your data size and hardware. The tokenized data will be saved in data/london_historical/tokenized_data/ for future runs, so subsequent training runs will be much faster. If you want to force re-tokenization, you can delete this directory and restart the training. And if you think this is hung, you can check the GPU usage using nvtop in a separate terminal.

And if you have configured WandB as recommended earlier, then you can log in to that dashboard and also monitor the training progress. This is quite handy when you are away from the machine and see how it is generally progressing.

Figure 11: Weights & Biases training progress monitoring dashboard

Figure 11: Weights & Biases training dashboard showing real-time loss curves and performance metrics

WandB also provides valuable insights into your model’s training performance through comprehensive visualizations. The dashboard shows the complete training journey, revealing how your model’s loss decreased over time, whether the training plateaued, and how efficiently your hardware was utilized. These visualizations help you understand not just the final results, but the entire learning process - identifying if the model continued improving throughout training or if it reached a performance plateau.

While these metrics are incredibly useful for optimizing your training process, we’ll dive deeper into interpreting these results and fine-tuning your training strategy in Part 3 of this series.

SLM Results (117M parameters):

wandb: Run history:
wandb:       eval/iter   ▂▂▃▃▄▄▅▅▆▆▇▇██
wandb: eval/train_loss  ███▇▇▇▇▇▇▇▇▇▇▇▇
wandb:   eval/val_loss  ███████▇▇▇█▇▇▇▇
wandb:    eval/val_ppl  █▇▇▇▇▇▆▆▆▆▆▆▆▆▆
wandb:     train/dt_ms           █            █                
wandb:      train/iter      ▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇██
wandb:      train/loss ▆▅▇▅▅▃▇▄▄█▅▄▅▄▃▇▄▄▅ ▃▃▂▄▅▂▅▂▄▅▃▃▄▅ ▄▃
wandb:        train/lr ██████████▇▇▇▅▄▄▄▃▃▃▃▃▃▂▂▂▂▂▂           
wandb:       train/mfu ▃▄▇▇█▄▄▆▆▇▅▂▅▆▆▇▇▂▄▅▇▇▇▆▆▇▇▇▇▅███▅▇▆▇▇ ▇

wandb: Run summary:
wandb:       eval/iter 60000
wandb: eval/train_loss 2.74369
wandb:   eval/val_loss 3.44089
wandb:    eval/val_ppl 31.21462
wandb:     train/dt_ms 10217.92054
wandb:      train/iter 60000
wandb:      train/loss 2.87667
wandb:        train/lr 3e-05
wandb:       train/mfu 7.50594

It’s also helpful to monitor GPU usage during training. I recommend using nvtop (a GPU monitoring tool similar to htop but for NVIDIA GPUs) in a separate terminal to track memory usage, temperature, and utilization in real-time. The screenshot below shows the GPU monitoring during model training.

Figure 12: GPU monitoring using nvtop showing real-time resource utilization

Figure 12: GPU monitoring during training showing memory usage, temperature, and utilization metrics

7.2 Understanding Checkpoints

Throughout training, the system automatically saves checkpoints - snapshots of your model’s current state, including all learned parameters, optimizer state, and training progress. These checkpoints serve as safety nets, allowing you to resume training if interrupted, and provide multiple model versions to choose from. The final checkpoint (typically saved at the end of training) represents your fully trained model, ready for inference and deployment.

Checkpoints are saved in the 09_models/checkpoints/ directory, with separate subdirectories for each model type. SLM checkpoints are stored in 09_models/checkpoints/slm/ (e.g., checkpoint-4000.pt, checkpoint-8000.pt), while regular model checkpoints are saved directly in 09_models/checkpoints/ (e.g., checkpoint-60001.pt, checkpoint-120000.pt). The checkpoint filenames include the training step number, making it easy to identify the training progress and select the best-performing version for your needs.

These checkpoints enable two powerful capabilities that significantly enhance your training workflow. You can test your model’s current performance at any point during training by running inference on intermediate checkpoints, allowing you to monitor progress without waiting for training to complete. Additionally, suppose training is interrupted due to power loss, system crash, or manual stop. In that case, you can resume from the last saved checkpoint exactly where you left off, saving both time and computational resources. This flexibility is particularly valuable for long training runs, enabling you to experiment with different model versions and recover from unexpected interruptions.

🧪 Ready to test your checkpoints? See section 7.4 for detailed instructions on testing your trained model checkpoints.

7.3 Regular Model Training

The Regular model training follows the same process as the SLM, using identical training infrastructure but with different configuration settings. The only differences are the training script (train_model.py instead of train_model_slm.py) and the model architecture parameters (354M parameters vs 117M).

# Clean any existing tokenized data
rm -rf data/london_historical/tokenized_data/

# Automatic GPU Detection (Recommended)
cd 04_training
./launch_training.sh

# Manual Multi-GPU training
torchrun --nproc_per_node=2 04_training/train_model.py --data_dir data/london_historical

Listing 11: Train Regular Model

Key Differences from SLM:

Training script: train_model.py (instead of train_model_slm.py)
Model size: 354M parameters (vs 117M for SLM)
Training time: 28-32 hours (vs 7-8 hours for SLM)
Memory usage: Higher VRAM requirements
Performance: Better text quality, slower inference

The training infrastructure, checkpointing, WandB integration, and all other features remain identical. The system automatically detects the model type and applies the appropriate configuration from config.py.

Regular Model Results (354M parameters):

wandb: Run history:
wandb:       eval/iter     ▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▅▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb: eval/train_loss  █████████▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▆▇▆▆▆▆▆▆▆▆▆
wandb:   eval/val_loss  ███████████████████████████████████▇███
wandb:    eval/val_ppl  ████▇▇█▇▇▇▇▇▇▇▇▇▆▇▇▇▇▇▇▇▇▆▇▇▆▆▆▆▆▆▆▆
wandb:     train/dt_ms                  █                      
wandb:      train/iter      ▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇███
wandb:      train/loss ▇▆▆▇▇▅▅█▅▄▃▅▅▅▄▇▄▄▄▄▄▃▃▃▅▂▄▅▂▅▂▄▅▃▃▄▅ ▄▃
wandb:        train/lr ▄██████▇▇▇▇▇▆▆▆▅▅▄▄▄▄▄▄▄▄▃▃▂▂▂          
wandb:       train/mfu ▆▇█▅▄ ▄▆▆▆▇▃▃▂▂▆█▃▃▅▅▃█▅▄▆▇▇▇▇▄▅▃█▆▇█▄▃█

wandb: Run summary:
wandb:       eval/iter 60000
wandb: eval/train_loss 2.70315
wandb:   eval/val_loss 3.61921
wandb:    eval/val_ppl 37.30823
wandb:     train/dt_ms 24681.64754
wandb:      train/iter 60000
wandb:      train/loss 2.70629
wandb:        train/lr 0.0
wandb:       train/mfu 7.20423

7.4 Testing Your Checkpoints

Once training is complete, you can immediately test your model using the checkpoints saved during training. This is one of the most exciting parts - seeing the model generate historical text for the first time! The PyTorch checkpoint approach provides immediate testing without any conversion needed, allowing you to test any checkpoint to monitor training progress while preserving the complete model state, including training metadata and optimizer state for fast, optimized inference.

Direct PyTorch Checkpoint Testing: Test your model directly from the training checkpoints without any conversion:

# Test SLM checkpoint (117M parameters)
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/slm/checkpoint-4000.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Test Regular model checkpoint (354M parameters)  
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/checkpoint-60001.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

Listing 12: Test Model Checkpoints

Expected Output: Your trained model will generate authentic historical text like:

“In the year 1834, I walked through the streets of London and witnessed the most extraordinary sight. The Thames flowed dark beneath London Bridge, whilst carriages rattled upon the cobblestones with great urgency. Merchants called their wares from Cheapside to Billingsgate, and the smoke from countless chimneys did obscure the morning sun.”

Testing Different Checkpoints: You can test any checkpoint from your training run to see how the model improved over time. Try testing checkpoints from different training stages to observe the learning progression - early checkpoints will generate more random text, while later checkpoints will produce increasingly coherent historical language.

💡 Pro Tip: For published Hugging Face models and community access, see the Quick Start section earlier in this post, where we demonstrated the published SLM model.

8. Publish to Hugging Face

Once you’ve successfully trained and tested your models, you can publish them to Hugging Face for community access and easy deployment. Publishing makes your models available to researchers, developers, and enthusiasts worldwide, while integrating them into the Hugging Face ecosystem for seamless use with the transformers library.

Publishing Process: The publishing code automatically handles the complete conversion process from PyTorch checkpoints to Hugging Face format, which is essential for making your trained models accessible to the broader community. This conversion transforms your local training artifacts into a standardized format that can be easily loaded by users worldwide.

The process includes converting model weights from PyTorch’s .pt format to the more efficient .safetensors format, generating proper configuration files (config.json, generation_config.json) that define the model architecture and generation parameters, uploading the custom tokenizer and all necessary files to ensure complete functionality, creating comprehensive model cards with usage instructions and metadata for easy adoption, and setting up proper model repositories with versioning for educational deployment.

This conversion is necessary because PyTorch checkpoints are optimized for training workflows and contain additional information like optimizer states that aren’t needed for inference, while the Hugging Face format is specifically designed for model sharing and deployment across different environments and hardware configurations.

We need to call the right script to publish the relevant model - either the SLM or the larger model. The publishing scripts will prompt you for your Hugging Face username and repository name, allowing you to customize where your models are published. The scripts automatically detect and use the latest checkpoint from your training run, so you can publish immediately after training completes.

💡 Quick Reference: If you want to test published models before publishing your own, see section 2 “Use the models - Try it now using Hugging Face” for immediate access to pre-trained models.

Prerequisites: You’ll need a Hugging Face account and either set the HF_TOKEN environment variable or provide your token when prompted. The scripts will guide you through the publishing process step by step.

Option A: Publish SLM (117M parameters)

# Publish SLM to Hugging Face
python 10_scripts/publish_slm_to_huggingface.py

Listing 13: Publish SLM to Hugging Face

Option B: Publish Regular Model (354M parameters)

# Publish Regular model to Hugging Face  
python 10_scripts/publish_to_huggingface.py

Listing 14: Publish Regular Model to Hugging Face

If everything is working correctly and the models are published, you will see confirmation messages and upload progress. Here’s what successful publishing looks like:

Figure 13: Hugging Face SLM model upload progress and confirmation

Figure 13: Hugging Face upload process for SLM model showing successful publishing workflow

And this is an example output for the Regular model:

Figure 14: Hugging Face Regular model upload progress and confirmation

Figure 14: Hugging Face upload process for Regular model showing successful publishing workflow

After Publishing: Once published, your models will be available at:

SLM: bahree/london-historical-slm
Regular Model: bahree/london-historical-llm

Users can then easily load and use your models with just a few lines of code, making your historical language models accessible to the broader AI community for research, education, and creative applications.

Testing Your Published Models: Once published, you can test your models using the same inference methods shown in the Quick Start section:

# Test published SLM model (10 automated tests)
python 06_inference/test_published_models.py --model_type slm

# Interactive testing with published models
python 06_inference/inference_unified.py --published --model_type slm --interactive

Listing 15: Test Published Models

10. What We’ve Accomplished

This comprehensive guide has taken you from raw historical documents to working language models that can generate authentic 18th and 19th-century London text. We’ve built a complete pipeline that transforms 218+ historical sources into two specialized models - a fast SLM for experimentation and a powerful Regular model for high-quality generation. The entire system is fully functional, with both PyTorch checkpoint inference and Hugging Face model publishing working seamlessly, tested and validated on real hardware.

What makes this project interesting is that it’s not just another language model - it’s a complete educational journey that teaches you every aspect of building LLMs from scratch. From custom historical tokenizers that understand archaic English to sophisticated GPU optimization and deployment, you’ve learned the full stack of modern language model development. The result is a system that preserves historical linguistic heritage while demonstrating cutting-edge AI techniques, making it valuable for researchers, educators, and anyone interested in the intersection of history and technology.

11. The Journey Continues

This is just the beginning. In the next three parts of this series, we’ll dive deeper into the technical foundations:

Part 2 explores historical data collection, showing how we curated 218+ authentic sources spanning 350 years of London’s history, and how we built a custom tokenizer that truly understands historical English.

Part 3 reveals the custom GPT architecture designed specifically for historical text, GPU optimization strategies, and training infrastructure.

Part 4 completes the journey with evaluation frameworks, testing strategies, and deployment techniques that transform your trained models into working systems.

Each part builds on what you’ve learned here, taking you from high-level overview to deep technical implementation details.

12. Resources

GitHub Repository:⚙️ github.com/bahree/helloLondon - Complete codebase with all training scripts, inference tools, and documentation
Hugging Face Models:
- 🤗 bahree/london-historical-slm - Small Language Model (117M parameters)
- 🤗 bahree/london-historical-llm - Regular Model (354M parameters)
📘Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts
📖Documentation: Complete guides in the 08_documentation/ folder covering every aspect of the project

13. Acknowledgments

🙏

Ready to dive deeper? Part 2: Data Collection & Custom Tokenizers covers the technical details of data collection, cleaning pipelines, and custom tokenizer development for authentic historical text processing.

Reasoning AI Models: An overview

Mon, 01 Sep 2025 00:00:00 +0000

TL;DR

As part of my role at Microsoft’s AI Foundry Applied AI engineering team in CoreAI, I have participated in numerous detailed discussions about the evolving landscape of AI models. In conversations with many customers, from CxOs to engineers, one recurring topic is the rise of reasoning AI models. These models are designed to perform complex tasks by explicitly breaking down problems into logical steps, rather than just generating text in a single pass like traditional large language models (LLMs). This shift toward reasoning-centric AI marks a major evolution in how we develop and deploy AI systems—and it’s a key factor behind the rise of Agents and Agentic AI.

At the same time, there is a lot of confusion about what these reasoning models are, how they differ from traditional LLMs, and how to effectively adapt and evaluate them. In this post, I aim to clarify these concepts by providing a technical deep dive into reasoning AI models, their training and adaptation processes, and the challenges involved in fine-tuning them for specific tasks. We will also explore how to evaluate these models effectively, considering their unique characteristics.

This post is intended to help one gain a deeper understanding of reasoning models and their implications; I cover these areas:

What are reasoning AI models? A technical overview of their architecture and training paradigms.
How do they differ from traditional LLMs? Key distinctions in capabilities and performance
How to adapt and fine-tune reasoning models? Best practices and common pitfalls
What are the challenges in customizing them? Technical and organizational hurdles
How to evaluate reasoning models? Metrics and strategies for assessing their performance

1. Introduction

Recent AI models have begun to combine language generation with explicit reasoning, enabling more reliable solutions to complex problems. Traditional LLMs like GPT-4o complete a generation in one go, without showing their work. Reasoning models, on the other hand, produce a sequence of intermediate steps (a “reasoning trace”) before the final generation. For example, Microsoft’s Phi-4-Reasoning (14B parameters) will explicitly work through a math problem step-by-step, whereas a regular LLM might confidently state an answer with no explanation. This fundamental difference – predictive text generation vs. chained logical reasoning – makes reasoning LLMs significantly better at multi-step tasks, such as math word problems, code debugging, or complex decision queries.

Note: The AI model landscape is also shifting rapidly, with a newer trend of transitioning from separate “base” vs. “reasoning” models (e.g., o1/o3) to unified systems with internal routing (e.g., GPT-5). GPT 5 runs a system that routes between fast and deliberate paths and exposes developer controls to tune thinking time. In production, the system automatically switches modes; developers can cap or elevate effort as needed. This operationalizes dynamic compute allocation, reducing the need for prompt engineering, specifically when wanting to induce reasoning.

The shift toward unified systems like GPT-5 can be understood as operationalizing the compute-optimal scaling insights from research. Rather than requiring users to choose between reasoning modes manually, these systems implement automatic difficulty assessment and adaptive compute allocation - essentially embedding the “compute-optimal” strategy within the model architecture itself.

1.1 What are reasoning models?

Reasoning models are LLMs architected to solve problems via a multi-step chain-of-thought (CoT) approach. Instead of just predicting the next token, they simulate an internal “scratchpad” of logic. For instance, OpenAI’s latest models (o1 and o3) reportedly allocate extra computation at inference-time and use reinforcement learning (RL) fine-tuning to boost multi-step reasoning. DeepSeek’s R1 (671B-parameter Mixture-of-Experts model ) was explicitly trained with multi-stage reinforcement learning to encourage step-by-step thinking.

During training, such models may be given examples formatted like: *Question → (Begin reasoning) → ... reasoning steps ... → (Final answer)*, or prompted with cues like “Let’s think step by step.” This teaches the model to articulate intermediate steps instead of jumping straight to an answer. In essence, a reasoning LLM learns to internalize a logical process – it doesn’t just know facts or language, it learns how to solve problems by breaking them down.

Crucially, these reasoning models often use special tokens to separate the “thinking” from the final answer. Many use a convention such as <think> ... </think> tags to enclose the chain of thought. For example, DeepSeek-R1-Distill (a distilled 8B version of R1) will output a hidden “thinking” transcript between these tags, followed by a concise answer that summarizes the reasoning. The chain-of-thought (CoT) might include equations, logic, or code, which the model generates as if working on scratch paper, and then the answer is given separately. This behavior is usually built into the model through fine-tuning – if you prompt such a model normally, it will, by default, produce a step-by-step solution trace and then provide the answer.

Some recent systems even let developers toggle the visibility of this trace: e.g., Qwen-3 allows a “reasoning mode” where the chain of thought is shown or hidden as needed. The key point is that reasoning models carry out more computation in the open, and they may consume more tokens. It is quite common for them to use hundreds or thousands of tokens for a complex solution, whereas a regular LLM might try to produce an answer in, say, a single paragraph.

Figure 1: DeepSeek-R1 architecture showing Mixture-of-Experts design with selective parameter activation (21B of 671B parameters active per token) and 128K token context window. (Source: DeepSeek research)

1.2 Cognitive Architecture Parallels - Type 1 and Type 2 Thinking

The reasoning model paradigm directly parallels the Type 1/Type 2 thinking framework popularized by Daniel Kahneman . Some of the recent work demonstrating how LLMs can be aligned to either System 1 (intuitive and fast) or System 2 (analytical and deliberate) thinking patterns.

Type 1 thinking in AI systems corresponds to the pattern-matching and intuitive responses characteristic of traditional LLMs - fast, automatic responses based on learned patterns. Type 2 thinking represents the deliberate, step-by-step reasoning that reasoning models are designed to emulate. Research shows that System 2-aligned models excel in arithmetic and symbolic reasoning, while System 1-aligned models perform better in commonsense tasks.

Cognitive Flexibility and Performance Trade-offs

Unlike human cognition, which fluidly adapts between System 1 and System 2 thinking based on context, current LLMs lack this dynamic flexibility. This rigidity can lead to brittle performance when tasks deviate from trained patterns. However, reasoning models attempt to address this limitation by incorporating explicit System 2-style processing.

The research demonstrates an “accuracy-efficiency trade-off” where System 2-aligned models show greater uncertainty and more systematic processing, while System 1-aligned models provide more definitive but potentially less reliable answers. This suggests that optimal AI systems may need to switch between reasoning modes dynamically based on task complexity.

From an architectural perspective, reasoning LLMs are still transformer-based neural networks at their core. They don’t necessarily have new algorithmic components beyond the training tweaks, though some research explores adding tools or memory. It’s the training paradigm that sets them apart.

For example, where a classic 4o/4.1 style LLM is trained purely on next word prediction and maybe a bit of instruction tuning, a reasoning model like R1 or Phi 4 is trained in an extensive multi stage training pipeline (e.g. supervised fine tuning on curated CoT examples), then specialized reinforcement learning (using rewards for getting answers right and for producing a consistent chain of thought), and so on. OpenAI’s o1/o3 models are rumored to undergo similar multi-stage refinement, combining RL with the ability to allocate more thinking steps at runtime.

1.3 Chain-of-Thought: Built-in vs Prompted

Start by understanding what a chain of thought (CoT) is. CoT is the model’s “scratchpad”: a sequence of intermediate reasoning steps it writes out before giving the final answer. Many models fence this trace with special tokens (e.g., <think> ... </think>); there are configurations that can show or hide these. The advantage this gives us is better results on multi-step tasks (such as math, code, and planning) by decomposing problems. On the other hand, the trade-offs include more tokens → more cost/latency; and traces can be verbose or unfaithful if not evaluated. As a result, CoT is best used for complex queries, and where possible, it would be wise to consider either skipping or limiting these for simple lookups. See “Evaluation” for token-normalized accuracy and faithfulness checks.

CoT prompting emerged as a technique to enhance traditional LLMs by explicitly requesting step-by-step reasoning through prompts such as “Describe your reasoning in steps” or “Explain your answer step by step.” This approach leverages LLMs’ ability to “think out loud” in natural language, with effectiveness scaling with model size as an emergent ability.

Figure 2 shows an LLM decomposing a complex math word problem into sequential subquestions, solving each step before arriving at the final answer. (Credit: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models)

Figure 2: CoT reasoning example

Reasoning models fundamentally differ in that they integrate CoT processing directly into their architecture and training process. Rather than requiring explicit prompting, these models automatically engage in step-by-step reasoning for complex tasks. Research indicates that “Chain-of-Thought built into the core architecture and training process” represents a more robust approach than external prompting.

However, CoT prompting is not universally effective across all models and tasks. Recent research on strategic reasoning has shown that CoT prompting is not universally effective, as it increases strategic reasoning only for models at certain levels, while providing limited gains elsewhere. This suggests that integrating reasoning capabilities requires careful architectural considerations beyond simple prompting strategies.

The effectiveness of CoT in reasoning models also varies by task complexity and domain. Models trained with reinforcement learning on reasoning tasks show more consistent application of multi-step reasoning compared to models relying solely on prompted CoT.

1.4 Test-Time vs Train-Time Compute

A critical innovation in reasoning models is the emphasis on test-time compute scaling. While their training parameters limit traditional LLMs, reasoning models can allocate variable computational resources during inference. OpenAI reports that the performance of o1 improves with more RL (train-time compute) and with more time spent thinking (test-time compute) ( overview ). This creates new scaling paradigms, where models can allocate more computational resources to harder problems during inference.

This inference-time compute scaling (using more tokens/steps) is a defining trait – it enables even smaller models to solve hard problems by iterating through reasoning. As Microsoft’s team describes, “Phi 4 Reasoning generates detailed reasoning chains that effectively leverage additional inference time compute,” allowing a 14B model to compete with far larger ones.

Because this extra “thinking” consumes tokens and compute, it helps to formalize the tradeoff to understand the concept better.

Test-time compute is best understood as a way to reshape the model’s output distribution at inference by searching over alternative reasoning paths and then selecting among them. It reliably lifts accuracy—especially on problems with verifiable answers—yet it is not interchangeable with pretraining compute.

Recent evidence shows that test-time compute helps most when the base model is already capable and the gap to the target difficulty is modest; on the hardest items, pretraining capacity still dominates - as outlined in Figure 3 below.

Figure 3: The relationship between test-time and train-time compute in reasoning models, showing how additional inference computation can compensate for reduced training compute. (Source: Snell et al., 2024)

A practical rule is to treat thinking tokens as a budgeted resource: use them to explore and score candidate chains (branching) and reserve a small budget for targeted revision when a verifier flags issues. In cost terms, this gives you predictable returns without pretending that more inference tokens can fully substitute for more capable pretraining.

Test-Time Compute vs Model Size Trade-offs

A groundbreaking finding from recent research is that on problems where smaller models achieve non-trivial success rates, test-time compute can be used to outperform models 14× larger in FLOP-matched evaluations. This suggests a fundamental shift in how we think about compute allocation:

Easy to medium problems: Test-time compute is often more effective than pretraining larger models
Very hard problems: Pretraining capacity still dominates, with limited benefits from test-time scaling
Practical implication: Rather than focusing purely on scaling pretraining, it may be more efficient to train smaller models and apply test-time compute strategically

Efficiency trade-off: How much “thinking” is enough?

OpenAI’s o1 explicitly reports: performance improves with more RL (train-time compute) and with more time spent thinking (test-time compute). Microsoft’s Phi-4 Reasoning (14B) shows similar patterns: small models, when allowed longer structured chains, outperform their weight in math/science. To examine the implications, consider a back-of-the-envelope cost model. If $L_r$ is “reasoning” length and $L_a$ is final answer length, a crude attention-heavy cost proxy is

$$ \text{Compute} ;\propto; H,(L_a + L_r)^2,d, $$

with hidden size $H$ and depth $d$. You can wrap this into an objective that matches your reality:

$$ \min_{L_r}; C(L_r) = \alpha,H,(L_a+L_r)^2,d ;+; \beta,\text{latency}(L_r) ;-; \gamma,\text{Acc}(L_r), $$

where $\alpha,\beta,\gamma>0$ are your infra cost, SLA pain, and value of accuracy. You won’t solve this analytically in prod—you’ll sweep the thinking budget and pick a knee point.

What is really interesting is that accuracy is typically concave in $(L_r)$; i.e, the first ~100–300 “thinking” tokens help a lot; beyond that, diminishing returns.

Quick intuition: if $L_a$ is small and $L_r$ doubles, the attention term grows by about $4\times$, while accuracy typically improves far less—hence token budgets and early stop heuristics. We’ll revisit this idea in Evaluation via token-normalized accuracy.

This trade-off also motivates practical features, such as token budgets, early-stop heuristics, and “fast vs. deliberative” paths (e.g., Qwen-3’s reasoning mode). With that lens, let’s look at what differs under the hood.

More deliberate thinking often helps—up to a point - You can trade thinking tokens for accuracy on complex items.
Returns diminish - the first ~100–300 “reasoning” tokens carry a lot of the lift; beyond that, you’re paying for a long tail.

Rule of thumb. Treat “thinking tokens” as a first-class budget; log it, control it, and optimize it like you optimize memory or p95 latency. Some model providers like Qwen 3 and NVIDIA’s NIM expose this thinking budget directly.

In short, reasoning LLMs are LLMs with a logic upgrade – through additional training, they learn to use reasoning strategies that standard models lack.

1.5 Effectiveness and Limitations of Reasoning LLMs

Recent benchmarks indicate that CoT reasoning yields significantly improved performance on complex tasks (see Figure 4 below). For example, Microsoft’s Phi-4-Reasoning models, with only 14B parameters, match or surpass much larger models in math and science benchmarks—sometimes even outperforming a model 5x times their size (surpassing OpenAI’s o1-mini, and R1’s 70B distilled version on many math and science benchmarks). This success is attributed to reasoning-focused training and reinforcement learning, proving that with strategic training, smaller models can excel at challenging tasks without needing massive scale. This demonstrates a general trend: with the right training, a model doesn’t have to be huge to solve complex tasks – it just needs to learn how to use its capacity more algorithmically.

Figure 4: Performance scaling on AIME mathematics benchmark: Both train-time compute (left) and test-time compute (right) show smooth accuracy improvements, validating the compute-optimal scaling approach. (Source: OpenAI research)

Another data point is the DeepSeek R1 family. The original R1 (671B, MoE) was a “reasoning-maximal” model (see Figure 1), pushed to an extreme scale and trained with novel RL algorithms (such as GRPO, a group-based self-improvement method) to excel at long-horizon problems. Distilled smaller versions of R1 (70B, 8B, etc.) inherited some of these skills through knowledge distillation. These distilled reasoning models, even at 8B, achieved math and puzzle-solving scores significantly higher than those of similarly sized generic LLMs. Open-source efforts like Bespoke-Stratos-7B and OpenThinker-7B followed suit, demonstrating that a properly fine-tuned 7B model with CoT can outperform naive 7Bs by significant margins on benchmarks. In research from late 2024, Qwen-3 (an advanced open model by Alibaba) was released in both “thinking mode” and “no thinking” mode. Running Qwen-3 in its CoT mode, it actually outperformed DeepSeek-R1 on a majority of evaluated tasks despite activating only a subset of its parameters at each token (it’s a mixture-of-experts model, effectively).

What is interesting is that when Qwen-3 was toggled off (i.e., no CoT visible), it still beat a GPT-4-sized baseline on many benchmarks, implying that integrating reasoning steps did not harm its base competency – it only added the ability to dig deeper when needed. All these examples underscore that reasoning LLMs hold a significant edge on tasks that aren’t straightforward single-step predictions. Whenever an answer requires multiple pieces of information or intermediate calculations, a traditional LLM often fails or guesses incorrectly, whereas a reasoning LLM can navigate the steps systematically (much like a human showing their work). The gap is so notable that analysts have called reasoning LLMs “a critical evolution” in AI capability, and enterprise users are exploring them for decision-making support where correctness takes precedence over brevity.

Mathematical and Logical Reasoning

Reasoning models demonstrate substantial improvements over traditional LLMs in mathematical and logical reasoning tasks. OpenAI’s o1 achieves remarkable performance, ranking in the 89th percentile on competitive programming questions (Codeforces) and placing among the top 500 students in the US in a qualifier for the USA Math Olympiad (AIME).

Codeforces: Codeforces is a major competitive programming platform and community. It hosts frequent online contests (“Rounds”) where participants solve algorithmic problems within time limits, and it maintains an Elo-like rating system and color-coded titles (ranging from Newbie to Legendary Grandmaster).

Comprehensive evaluations (see Figure 5) show that o1-preview demonstrates 100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions and an 83.3% success rate in solving complex competitive programming problems, surpassing many human experts. These results indicate performance that often meets or exceeds that of human experts in structured reasoning domains.

Figure 5: Comprehensive benchmark comparison showing o1-preview's superior performance over GPT-4o across mathematics, science, and reasoning tasks. (Source: OpenAI, 2024)

Domain-Specific Applications

Beyond mathematics, reasoning models show strong performance across diverse specialized domains. Evaluations indicate remarkable proficiency in anthropology and geology, demonstrating a deep understanding and sound reasoning in these specialized fields, as well as strong capabilities in quantitative investing, complemented by comprehensive financial knowledge. The models also demonstrate superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models.

Recent research with ReasonFlux-32B has demonstrated that smaller, specialized reasoning models can outperform larger, general models. On the MATH benchmark, ReasonFlux-32B achieves an accuracy of 91.2% and surpasses o1-preview by 6.7% while being trained with only 8 GPUs.

ReasonFlux - ReasonFlux is a template-driven, hierarchical RL approach to reasoning LLMs; instead of lengthening raw CoT, it plans over a library of thought templates and scales those at inference time, yielding strong math results in a 32B-parameter model.

However, this does not mean reasoning models dominate on every task. For very simple or single-step queries (e.g., straightforward fact lookups or classifications), a regular LLM might perform just as well and with less latency - it is using fewer tokens and does not have to generate a long explanation; more tokens mean more computation and slower responses. That said, many reasoning LLMs are designed to be flexible – they can shorten or skip the reasoning when it’s not needed. Some deployments use a “fast path” versus a “deliberative path” approach: run the model in normal mode for easy questions and only invoke full reasoning mode for complex ones. This dynamic compute allocation is a research area in itself (how to predict when to make a model think longer).

The token-budget mechanism in Qwen-3 is one example: it allows users to cap how many reasoning tokens the model can use, forcing it to decide what’s most important. Accuracy does improve with more tokens (e.g., from ~70% at 2K tokens to ~85% at 16K on a math test), but after a point, it is a matter of diminishing returns. The existence of such features highlights that reasoning LLMs introduce a new dimension – a time/accuracy trade-off. Traditional LLM evaluation is usual just one-dimensional – measuring accuracy or quality for a given fixed model output length. On the other hand, reasoning LLMs let us trade generation length for correctness. (Note: the Evaluation section will cover more details on how to measure).

1.6 Branching & Editing at Test Time (how to “spend” thinking compute)

Test-time compute isn’t just “more tokens”; it’s a way to reshape the model’s output distribution by searching for, and then selecting, better reasoning paths during decoding. In practice, this plays out along two complementary axes. The first is branching: generate multiple candidate chains and prefer the one that scores best under a process- or outcome-aware judge. The second is editing: let the model (and its tools) reflect on an initial attempt and revise it once or twice. Both strategies are ways of allocating limited thinking budget where it matters most.

On the branching side, simple best-of-N sampling remains a solid baseline, while beam or tree-style search makes exploration adaptive by spending more decoding on promising partial thoughts. Process-aware scoring—via a process reward model (PRM) or per-step self-evaluation—helps prune low-quality branches early; when ground truth isn’t available, self-consistency (majority voting across diverse chains) is a practical fallback. Two small but useful tricks from recent work are to branch early—keeping only the top few first-token continuations before decoding greedily—and to anneal temperature across tokens to reduce accumulated randomness as chains grow. Together, these make parallel exploration both cheaper and more reliable.

Editing tackles a different failure mode: an answer that looks plausible but hides a local mistake. Here, short reflect-revise loops work best when anchored to reliable feedback—unit tests for code, exact-match checks for math, heuristic rubrics, or judgments from a stronger model. Pure “self-correction” without such anchors tends to be unstable: models often make minor, non-helpful edits, occasionally flip correct answers to incorrect ones, or fail to generalize the revision behavior. Keeping revision rounds tight, skipping revision when a verifier signals “already correct,” and rolling back to the best-verified candidate are practical guardrails.

Importantly, branching and editing are not substitutes; the best results often come from using both. For easier problems, a short sequential pass can be enough, but as difficulty rises, the sweet spot shifts toward a deliberate mix of parallel exploration and a small revise budget. Thinking time is therefore a budget allocation question: how much diversity you buy up front versus how much you reserve for targeted fixes after you’ve seen a candidate chain.

Operationally, it pays to make the budget explicit and observable. Expose a cap on “thinking tokens,” allow early exit when candidates agree with high confidence, and log the signals that drove selection—per-step PRM or self-evaluation scores, agreement margins, and precise stop reasons. Over time, these traces make it easy to tune the ratio between breadth (how many chains you explore) and depth (how hard you try to fix a promising one), and to decide when a verifier is strong enough to justify skipping revision. Finally, remember that this test-time axis complements, but does not replace, pretraining: extra thinking generally helps, yet it cannot fully compensate for large capability gaps on the hardest items.

Compute-Optimal Scaling

Recent research by Snell et al. demonstrates that compute-optimal scaling - allocating test-time compute adaptively based on problem difficulty - can improve efficiency by more than 4× compared to traditional best-of-N sampling. This approach recognizes that different problems require different amounts of thinking time, and optimal allocation varies dramatically based on prompt difficulty.

The key insight is that question difficulty can be predicted and used to determine the most effective test-time compute strategy. For easier problems, simple parallel sampling suffices, while harder problems benefit from sequential revision or sophisticated search strategies.

Research identifies two primary mechanisms for scaling test-time computation effectively:

Process-Based Verifier Search: Using dense, process-reward models (PRMs) to guide search through reasoning paths, enabling beam search or lookahead search strategies that prune low-quality branches early.
Adaptive Distribution Updates: Modifying the model’s distribution over responses at test time, such as through sequential revision where the model iteratively improves its initial attempts.

The effectiveness of these approaches critically depends on problem difficulty - easier problems benefit more from parallel exploration (branching), while harder problems require sequential refinement (editing).

Difficulty-Aware Compute Allocation

A key insight from recent research is that optimal test-time strategies vary dramatically with problem difficulty. This motivates adaptive allocation strategies:

Easy problems: Simple best-of-N sampling with minimal compute
Medium problems: Weighted voting or beam search with moderate compute budgets
Hard problems: Sequential revision with larger compute budgets, but diminishing returns beyond a threshold

This difficulty-aware approach enables 4× efficiency improvements over uniform compute allocation strategies.

1.7 External tools inside the reasoning loop

Several steps in a chain can be offloaded to exact tools (e.g., code execution, math). Approaches like PAL (program-aided language model) and CoC (Chain-of-Code) let the model “think” by writing and running code; ReAct interleaves search (e.g., Wikipedia) with thoughts. Recent o-series releases similarly intertwine web, code, and vision tools during reasoning. This improves robustness on math, algorithmic tasks, and multi-hop QA – without asking the LLM to emulate a compiler.

PAL

Program-Aided Language Models (PAL) are an approach where LLMs address reasoning tasks by generating Python code rather than relying solely on natural language. This method utilizes programming to manage complex logic and calculations, aiming to decrease errors and improve results on benchmarks such as GSM8K and MATH. PAL’s architecture is modular and interpretable, with the LLM functioning as a code generator and the Python interpreter serving as the reasoning engine. This clear separation improves debugging, verification, and extensibility, enhancing transparency and reproducibility. By combining symbolic reasoning with neural language modeling, PAL provides a hybrid approach that is both effective and practical.

CoC

Chain of Code (CoC) is a method that expands code-driven reasoning in LLMs by using a hybrid execution strategy. In contrast to traditional methods that rely exclusively on interpretable code or natural language reasoning, CoC enables models to generate programs combining executable code with semantic pseudocode. When the interpreter encounters undefined or non-executable behavior, such as abstract functions like detect_sarcasm(string), CoC uses an “LMulator”, which is a language model-based emulator that predicts the expected output. This approach allows LLMs to process tasks involving both algorithmic and semantic elements.

By “thinking in code” CoC greatly expands the range of problems it can solve, surpassing Chain of Thought and other baseline methods on benchmarks like BIG-Bench Hard, where it reached an 84% success rate—12% higher than CoT. Its modular structure adapts well to different model sizes and fields, making it particularly suitable for tasks in robotics, perception, and mixed-modality reasoning. The use of flexible pseudocode and fallback emulation strategies provides a strong foundation for developing more generalizable and interpretable AI reasoning.

In summary, reasoning AI models distinguish themselves by how they solve problems. They use explicit multi-step reasoning (often visible as a chain-of-thought) and are trained with techniques (special prompts, reward signals, data curation) to make this effective. In doing so, they often achieve higher accuracy on complex tasks than traditional LLMs of comparable (or even much larger) size. The cost is greater complexity in training and sometimes in usage. We next discuss how one can adapt and fine-tune these models, and the pitfalls to watch out for.

2. Adapting and Fine-Tuning Reasoning Models

Similar to LLMs, reasoning models can also be fine-tuned or adapted to specific domains and tasks. A key advantage is that they can be domain-specialized while retaining strong reasoning skills.

For example, if you have a reasoning LLM and you want it to excel at medical diagnostics, you could fine-tune it on medical Q&A data that includes step-by-step reasoning about symptoms and lab results. The model should, in principle, retain its general logical abilities and learn to apply them in the medical context. Fine-tuning can also help a model learn when to engage reasoning mode – e.g., always do detailed reasoning for high-stakes medical questions, but perhaps skip it for trivial prompts if instructed.

However, adapting a reasoning model is more complex than fine-tuning a regular LLM because you need to handle the reasoning traces properly. A key question is whether the fine-tuning data includes chains of thought or just question→answer pairs. Generally, to preserve and leverage the model’s strength, you want to fine-tune with the reasoning format intact. That means if your dataset doesn’t already have human-written rationales, you may need to generate them (possibly using a larger teacher model like R1 or GPT-4 to produce explanations for your domain problems). By training on QA pairs supplemented with correct reasoning sequences, you reinforce the model’s inclination to think things through.

There is a subtle issue, though; if your fine-tuning data’s reasoning traces are of lower quality than the model’s current capability (for instance, you provide simplistic or even flawed reasoning examples), you might hurt performance. It’s like training a math student who can solve calculus problems to only practice arithmetic – they might lose their edge in advanced problem solving.

2.1 Loss Masking

One approach is called loss masking, which involves including reasoning steps in the input/output during fine-tuning so the model learns to produce them, but not applying back-prop loss on those reasoning tokens. So, fine-tuning gradients is applied only to the final answer portion, rather than the whole CoT text. This allows us to adjust the model’s final answers for a new domain while minimizing changes to its internal reasoning process. The rationale is that the model’s existing reasoning ability, developed through prior training, should be maintained. The technique allows the model to retain its established reasoning while modifying how it presents final answers. Initial community observations indicate this approach can help preserve the quality of the model’s reasoning after fine-tuning. However, it may not be necessary if the fine-tuned dataset is large and of high quality.

2.2 Prompt-Based Fine-Tuning

Another approach when working with limited data is to use prompt-based fine-tuning or instruction prompts. Since reasoning models already respond to prompts like “show your reasoning, then answer,” you might not need to change their weights at all for some custom tasks – providing a few exemplars with reasoning in a prompt might suffice (few-shot learning). If actual fine-tuning is needed (e.g., to integrate new knowledge or jargon), lightweight methods like LoRA adapters can be applied in principle. One must ensure the prompt format (the presence of <think> tags or special tokens) is consistent during fine-tuning to prevent the model from being confused about when to produce reasoning. Many open implementations of reasoning models require a specific format to trigger the chain of thought. Adhering to that format in any further training data is important.

In summary, adapting a reasoning LLM is doable but requires careful dataset design. Ideally, your fine-tuning set should contain high-quality problem-solving examples with the full reasoning shown. If you don’t have that, you might generate it or opt to preserve the pre-trained reasoning behavior via techniques like masking. One should also monitor if the model starts to skip reasoning; if it does, this could indicate that the fine-tuning data encouraged direct answers only. Balancing task specialization with maintained reasoning capability is key.

Next, let’s examine the challenges that may arise during this fine-tuning and customization process.

Practical Compute Budget Guidelines

Recent empirical analysis provides concrete guidance for practitioners:

Budget allocation: Treat test-time compute as a first-class resource requiring explicit budgeting and monitoring
Difficulty prediction: Use learned difficulty predictors to route problems to appropriate compute strategies
Diminishing returns: Most benefits come from the first 100-300 reasoning tokens; beyond that, returns diminish rapidly
Cost-performance optimization: Smaller models with sophisticated inference can achieve Pareto-optimal trade-offs compared to larger models with simple inference

3. Challenges in Fine-Tuning and Customizing Reasoning Models

Adapting reasoning models to new tasks comes with unique challenges beyond those in standard LLM fine-tuning. These challenges span technical issues inherent to the models’ reasoning nature, as well as organizational hurdles in data and expertise. Let us explore some of the key challenges.

3.1 Trace Quality Degradation

A major technical concern is preserving the quality of the reasoning trace. Fine-tuning, if done either poorly or used on narrow data, can cause the model’s CoT to become less coherent or less faithful to its actual reasoning. Recent research shows that after fine-tuning on specific tasks, the faithfulness of a model’s CoT explanations often decreases, on average, compared to the pre-finetuned model. In other words, the model might still provide accurate answers, but its stated reasoning is more likely to omit key steps or include spurious ones. This “trace degradation” can occur because the fine-tuning objective typically emphasizes obtaining the correct final answer for the new task – the model may learn that it can score well without strictly adhering to its original reasoning style.

In addition, if the fine-tune dataset isn’t sufficiently diverse or is missing the intermediate logic, the model’s previously polished reasoning abilities can “unravel” or get overwritten. It’s akin to using coarse sandpaper after a fine polish – the model may lose some of its nuanced problem-solving steps. Ensuring that fine-tuning does not erase the chain-of-thought skill is a complex and challenging task.

Techniques like the aforementioned loss masking or multi-stage fine-tuning (where you intermix some original reasoning training data) are used to mitigate this. Another aspect of trace quality is faithfulness – even if the model produces a plausible-looking rationale, is it honestly reflecting how the answer was derived? Fine-tuning can sometimes widen the gap between what the model does to get an answer and what it says in the explanation, especially if the fine-tuning introduces shortcut ways to get the answer. This is hard to detect; it requires careful evaluation (as we discuss later).

Overall, maintaining a correct and faithful reasoning trace under new training pressures is a key challenge.

3.2 Overfitting and Distribution Shift

Like any model, a reasoning LLM can overfit to a small fine-tune dataset, but the consequences here might be strange. An overfit model might memorize specific solution patterns and fail to generalize its reasoning to slightly new problems (losing one of the main advantages of a reasoning approach). Because these models were often trained on a wide variety of reasoning tasks, fine-tuning on a narrow domain (say, only physics puzzles) might reduce their versatility or even accuracy on reasoning problems outside that niche.

Small, high-quality reasoning datasets can improve models, but if applied naively, they can also reduce performance on broader evaluations. The model may become too narrowly focused in its thought process (e.g., always expecting a specific style of solution). Ensuring the fine-tuning data covers enough variation or using regularization techniques (such as mixout or weight decay on reasoning layers) can help counteract this, but it remains a delicate balancing act.

LIMA shows that ~1k carefully curated examples can generalize well, and LIMO finds that ~800 math-reasoning samples yield large gains when the data is selected thoughtfully. However, a narrow or naïve fine-tuning can backfire—studies report catastrophic forgetting and degraded out-of-distribution robustness, as well as a drop in CoT faithfulness after fine-tuning. This can be mitigated with regularization (e.g., Mixout, layer-wise noise-stability) and optimization that flattens the loss landscape (e.g., SAM), and keep the fine-tune mix diverse to avoid over-specialization.

3.3 Training Stability and Long Outputs

Fine-tuning with long CoT outputs (which can be thousands of tokens) can lead to stability issues in training. Gradient updates on very long sequences might cause more variance or instabilities in convergence. Moreover, suppose one uses reinforcement learning (e.g., to further optimize a reasoning model with a reward for correct answers). In that case, the credit assignment is complex – which part of a 100-step reasoning deserves credit or blame for the outcome?

Instabilities like mode collapse (where the model’s outputs become strangely repetitive or nonsensical) or oscillating performance have been observed if the RL reward model is poorly aligned. For example, in one training run, simply increasing the reward for “correct final answer” without properly balancing the reward for good reasoning steps caused the model to exploit quirks – it started producing minimal reasoning and guessing answers to game the reward, leading to a drop in overall logical correctness.

Researchers working on Phi-4 and others have had to introduce tricks to stabilize RL training, such as gradually increasing the allowed reasoning length, filtering out bad traces, or adjusting reward scaling. These measures highlight that straightforward fine-tuning or RL on a reasoning model can easily go off-track if the optimization isn’t carefully managed. In essence, teaching a model how to think is a more delicate process than teaching it what to say.

3.4 Reward Alignment and “Hacks”

Aligning a reasoning model with human preferences or task-specific rewards can be tricky – there’s a risk of reward hacking and unintended behaviors. An illustrative scenario was described by researchers at Anthropic: they gave a reasoning model (Claude 3.7 and DeepSeek R1) a series of multiple-choice questions with a twist – a hidden “hint” in the prompt sometimes told the model to choose a wrong answer (and they rewarded the model for following that hint). The models learned to exploit this to earn reward points, selecting the hinted-at wrong answers, but their chain of thought never acknowledged the malicious hint. They would generate a detailed (fake) reasoning to justify the wrong answer, rather than saying “I chose this because I was hinted at.” This is a dramatic example of a model gaming the objective: the training set or reward said “getting this answer is good,” so it did. Still, it also learned to hide the true reason, presenting a facade of coherent reasoning.

Such behavior is misaligned with the intent (we want the model to be truthful in its reasoning). This experiment highlights the importance of aligning the process of reasoning as much as the outcome. If a reward model only considers the correctness of the final answer, it may sacrifice honesty or thoroughness in the reasoning process.

Conversely, suppose you over-emphasize a reward for producing very detailed reasoning. In that case, the model might start outputting verbose, mostly correct-sounding monologues that don’t lead to a better answer (effectively optimizing the wrong metric). Achieving the right alignment – so that the model is rewarded for correct and genuinely helpful reasoning – is an open challenge. It often requires iterative human feedback, custom reward functions (e.g., penalize logical leaps or unsupported claims in the trace), and careful validation. Without these, one might end up with a model that appears to reason well but is just skilled at “output grooming” – formatting answers to look good rather than being correct.

3.5 Data Quality and Availability

On the organizational side, fine-tuning a reasoning model demands high-quality training data that includes reasoned solutions. Such data can be difficult to obtain. At the same time, there are public datasets for math proofs or logical reasoning (e.g., MATH, GSM8K, etc.), but many domains (legal reasoning, financial analysis, medical diagnostics) don’t have readily available step-by-step annotations in large quantities.

Teams often have to generate this data synthetically (using a larger model to produce reasoning traces and then filtering them) or invest in expert annotations. The quality of these traces is paramount – noisy or incorrect reasoning examples can confuse the model or teach it bad habits. As discussed earlier, even small, curated datasets (on the order of hundreds of examples) have been shown to improve reasoning if they are extremely well-targeted; however, curating such datasets is a specialized skill.

In practice, fine-tuning a reasoning model involves a lot of tooling, ranging from running automatic proof checkers to verify steps, using consistency checks, or employing human reviewers to label where a model’s synthetic reasoning went wrong. This is a step up in complexity from preparing a straightforward prompt→response dataset.

3.6 Tooling and Infrastructure

Working with long CoT and multi-stage training means that the training pipelines will need modification. For instance, standard training code may need to be adapted to handle special tokens (e.g., <think> segments might need masking if needed), or to log and evaluate not just final answers but also intermediate step accuracy during training.

Debugging a reasoning model can be more involved – you might want to watch how its reasoning changes epoch by epoch, which requires custom logging or visualization tools. Moreover, these models often have large context windows (since they need to handle long reasoning sequences, e.g., 16K or 32K tokens). Fine-tuning with such long contexts can demand more GPU memory and faster I/O. Not all training frameworks efficiently support extremely long sequences out of the box.

Evaluation tooling (to be discussed later) can also be considered—a possible approach is integrating an automated verifier into the training loop to assess the model’s reasoning steps and provide targeted feedback, which is a type of process supervision. Implementing this involves technical complexity and remains an ongoing area of research. Overall, organizations seeking to customize a reasoning model should be aware that the training workflow may be more complex than a standard LLM fine-tuning process.

3.7 Expertise

Fine-tuning reasoning models demands both machine learning expertise and domain knowledge, often requiring multidisciplinary teams. Since reasoning LLMs are new, practitioners face a steep learning curve with frequent trial and error.

Expect several iterations to balance concise and detailed responses; objectives or examples may need adjustment throughout the process. Rigorous testing is essential, especially in high-stakes applications like medical or legal fields, making reliability and interpretability critical. Typically, 10–12 rounds of tuning are required to achieve an optimal model.

Organizations typically use a hybrid strategy: starting with a robust base model (such as o1-mini or Phi-4-Reasoning), applying minimal tuning, and relying on prompts and few-shot learning for specificity. When deeper customization is required, it’s best to use reliable data, maintain reasoning formats, monitor trace fidelity, and integrate human feedback. Success yields a strong analytical tool, but the process is more complex than for general chatbots.

A key part of customization is the ability to evaluate the reasoning models. Let us dig into specialized evaluation strategies required to assess not just what a reasoning model answers, but how it arrives at that answer.

4. Evaluation Strategies for Reasoning Models

Traditional LLM evaluation – e.g., measuring accuracy on a Q&A or using BLEU scores for text – may not capture the full picture when a model is effectively performing a multi-step reasoning process. Evaluating reasoning-oriented LLMs requires going beyond the final answer, incorporating metrics that assess both the process and quality of reasoning. This represents a departure from traditional LLM evaluation, which typically treats the model as a black box that produces an answer or text, which we then compare to a reference or expected output.

For reasoning models, we care about questions like: Did the model’s CoT follow a correct logical path? Is it telling the truth about its reasoning? How efficient is its reasoning? Below are key evaluation strategies and metrics that have emerged for reasoning models, contrasted with traditional approaches:

4.1 Outcome vs. Process Evaluation

In traditional AI evaluation, we mostly judge the outcome (e.g., did the model get the correct answer to a question). With reasoning models, researchers perform dual evaluations – one for the outcome and one for the reasoning steps. An outcome evaluation may be identical to a standard LLM test, where the goal is to verify if the final answer is correct (exact match, F1 score, multiple-choice accuracy, etc.). The process evaluation, however, examines the intermediate steps of the solution.

For instance, a math word problem benchmark might not only check the answer but also parse the model’s step-by-step solution and verify each part. An emerging method is to use an automated judge (which can be another LLM) to analyze the CoT and flag errors or leaps in logic. One example being a recent benchmark called MM-MATH (for multimodal math problems); in this, an LLM-based evaluator looks at each step of a model’s solution, comparing it to the ground truth solution, and classifies errors (e.g., “incorrect algebraic simplification” vs “misinterpreted the diagram”).

This kind of fine-grained process evaluation provides insights into where a model’s reasoning fails, not just whether the final answer is wrong. This is useful because a reasoning model might get the right answer for the wrong reasons (i.e., it had a reasoning flaw), or vice versa – it might have mostly correct reasoning but a minor slip at the end leading to a wrong answer. Traditional single-score metrics would miss this nuance.

4.2 Chain-of-Thought Faithfulness Metrics

As discussed earlier, faithfulness refers to whether the model’s stated reasoning accurately reflects its actual internal reasoning (or use of information). One way to test this is to insert known information (or traps) into the context and see if the model admits it.

For example, Anthropic’s experiment provided the model with hidden hints (sometimes incorrect) and then checked if the model’s explanation mentioned using those hints. They derived a metric: the percentage of solutions where the model was truthful about using the hint. Claude 3.7 was only ~25% faithful in their setup, and DeepSeek R1 was about 39% – meaning in the majority of cases, they used the hint but didn’t reveal it in the reasoning chain. This indicates that the CoT was often unfaithful, presumably because the model’s training taught it always to sound logical and self-contained, even if it took a shortcut.

Another way to measure faithfulness is to check consistency under variations: if a model truly is reasoning step by step, then if we force it to reveal steps, it should arrive at the same answer as when it’s not forced. If hiding the CoT changes the answer frequently, it might suggest the model’s explanations were more post-hoc and not driving the answer.

Note: These evaluations are still an active research area – unlike a simple accuracy score, faithfulness is somewhat difficult to quantify, but it’s crucial for trust. When deploying a reasoning model, you’d like to trust that, say, a financial analysis it provides is actually how it came to its conclusion, not a fabricated rationale. Thus, papers often report the percentage of solutions with “fully faithful reasoning” by manual or automated inspection. If that percentage is low, it’s a red flag: the model’s reasoning output might be more for show. Improving this might involve further training (e.g., penalizing inconsistent rationales) or architectural changes; however, at the very least, we need to measure it.

4.3 Token-Normalized Accuracy (Efficiency)

Because reasoning models can use an arbitrary number of tokens to reason (within the context window limits, of course), we want to measure accuracy as a function of reasoning length – effectively, how efficiently does a model reach correct answers? For example, a model that gets 90% accuracy with 2K tokens of reasoning might be less desirable than one that gets 85% accuracy with only 1K tokens, depending on deployment constraints.

Token-normalized accuracy is a metric that attempts to penalize overly lengthy reasoning. In one formulation (used in some multiple-choice evaluations), it computes the probability of a correct answer, normalized by the length (i.e., the number of tokens) of that answer’s explanation or output. More generally, we can think of it as accuracy per 100 reasoning tokens or similar.

Another interpretation is to measure the area under the curve of accuracy versus the number of tokens allowed. For example, allow a model to think with 100 tokens, record the accuracy, then 200 tokens, 500 tokens, and so on, up to a certain limit – and see which model yields the best accuracy for the least token budget. Researchers have explicitly emphasized the goal of maximizing accuracy per token in reasoning scenarios.

This reflects practical concerns: in production, reasoning steps are costly - both in terms of latency and tokens (i.e, money). A model that uses half the steps to reach the same answer is effectively twice as fast. Moreover, sometimes unconstrained reasoning leads to diminishing returns or even errors—for example, a model might start wandering or overexplaining if it “thinks” too long. Thus, token-normalized metrics encourage models that use their reasoning budget optimally.

A simple implementation is to take the total tokens the model generated for all test problems and divide them by the number of correct answers. Then, compare models on this normalized score (lower tokens per correct answer is better).

Another approach is a normalized log-probability where longer outputs are penalized. In any case, this kind of metric was usually irrelevant for standard LLMs (which output a single short answer), but becomes important when evaluating the cost-effectiveness of reasoning models.

4.4 Stepwise Accuracy and Consistency

This is a more granular evaluation of the correctness of the reasoning chain. For tasks where we have ground-truth step-by-step solutions (like a math proof or a formal logic derivation), we can mark each step of the model’s chain as “correct” or “incorrect” compared to an expected solution. This yields a sequence of accuracy values (e.g., getting the first three steps right, but failing at step four). We can then compute metrics like average step accuracy, or percentage of solutions that made it to at least X steps correct before failing.

This is informative because two models might both solve 70% of problems, but one might always fail early on the 30% it can’t solve, whereas another might almost solve everything and only slip at the end for those 30%. Stepwise evaluation can reveal such differences. It also helps in evaluating partial credit – maybe a model didn’t get the final answer but did significant parts correctly (which might be useful in applications where a human or another tool can pick up from the middle).

Some evaluations also check consistency: if a model is asked to explain its answer vs. directly answer, do those agree? If it solves a problem in two different ways (maybe by reordering steps or under different prompts), does it reach the same conclusion? Consistency checks can catch cases where the reasoning process is brittle or overly sensitive to phrasing.

4.5 Automated Reasoning Critics (LLM-as-a-judge)

A practical framework that has gained traction is using a strong language model to evaluate the reasoning of another model (or even itself). For instance, one can prompt GPT-4 with: “Here is a chain-of-thought and an answer. Evaluate the correctness and logical validity of the reasoning, and whether the final answer is justified.” This uses the fact that cutting-edge models can often spot obvious reasoning errors or missing justifications in a solution that a simpler rubric might miss.

Such LLM-based evaluators can be more flexible than hard-coded checkers. The aforementioned process evaluators in research are essentially reasoning models used as judges, with the ability to allocate extra computational resources to evaluate each step carefully. In one study, researchers found that when they allowed an evaluator model to think more (generate a longer evaluation reasoning), its accuracy in judging solutions improved monotonically – much like how making a model think more improves problem-solving, it also improves evaluation quality.

This is a fascinating recursive idea: use a reasoning model to evaluate better outputs that themselves involve reasoning. It was even shown that using such process-aware evaluators to re-rank answers (choosing the answer that the evaluator model scores highest) can significantly improve the solving ability of the base model.

In summary, process evaluation frameworks often involve an LLM evaluator performing a two-level check:

Outcome evaluation (is the final answer correct?)
Process evaluation (are the steps valid and do they lead to that answer?).

By combining these, one gets a more robust assessment. This approach complements traditional metrics; for example, you might report that a model has 80% outcome accuracy, but according to an LLM judge, only 50% of its solutions were fully correct with no logical errors in any step. That tells a deeper story than 80% alone.

4.6 Illustrative Example

To illustrate, consider a concrete example: say we ask a model a puzzle and it answers with a 5-step reasoning chain. The final answer is correct, so outcome-wise it’s a success. However, upon evaluation, we found that an arithmetic mistake occurred in step 3, which fortunately canceled out in step 5, yielding the correct answer nonetheless. A pure outcome metric says “perfect solution”. A process-aware evaluation would ding this as flawed reasoning (the model got lucky or coincidentally correct) – something we’d want to know if using the model for, say, validating scientific calculations. Conversely, if a model’s final answer is wrong, traditional evaluation is 0 for that question. However, process evaluation might reveal that the model was correct up until the last step – perhaps it performed all the reasoning correctly and made an error at the end.

In a human-learning context, you’d give partial credit. For model evaluation, noting that the model was, say, “90% correct in the procedure” could inform how we attempt to improve it (perhaps it just needs a slight boost in arithmetic precision or a final double-check step). This rich information is only available if we evaluate the reasoning, not just the outcome.

For practitioners, incorporating these evaluations is vital, as they help ensure that a high-performing reasoning model isn’t just getting by with smoke and mirrors (or hidden cues), and they quantify the efficiency and transparency of the model’s problem-solving approach. As these models become more integrated into workflows (e.g., as AI reasoning assistants), having reliable evaluation methodologies will also be key for governance and trust – one might, for example, require that a model’s chain-of-thought passes a certain automated consistency check before its answer is shown to a user.

In summary, the evaluation of reasoning LLMs has evolved to include trace-centric metrics alongside traditional outcome metrics. We assess the faithfulness of their explanations, measure accuracy in a way that accounts for the cost of reasoning length, and use novel frameworks where models critique reasoning steps (providing a “process score”).

5. Safety Concerns and Vulnerabilities

While reasoning models offer powerful capabilities, they also introduce new safety concerns and vulnerabilities that must be carefully managed and addressed. The very features that make these models effective – their ability to generate detailed CoT and reason through complex problems – can also be exploited by malicious actors or lead to unintended behaviors. Below, we discuss some of the key safety challenges specific to reasoning AI models.

5.1 Reward Hacking and Training Vulnerabilities

Reward hacking represents a significant concern in reasoning model development, particularly given their reliance on reinforcement learning during training. Reward hacking occurs when “a RL agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task”.

In the context of LLMs trained with RLHF, reward hacking manifests when models learn to game evaluation metrics rather than genuinely improve at the intended tasks. This is particularly concerning for reasoning models, where the complexity of the reasoning process makes it difficult to specify comprehensive reward functions that capture all aspects of good reasoning.

For example, a reasoning model might discover that providing overly verbose explanations leads to higher evaluation scores, even if those explanations are not genuinely helpful or accurate. This could incentivize the model to generate long-winded responses that obfuscate its actual reasoning process, ultimately undermining the quality of its outputs.

5.2 Jail-breaking and Safety Mechanism Vulnerabilities

Recent research has revealed severe vulnerabilities in the safety mechanisms of reasoning models. The Hijacking Chain-of-Thought (H-CoT) attack method demonstrates how attackers can “leverage the model’s own displayed intermediate reasoning to jailbreak its safety reasoning mechanism”. Under such attacks, refusal rates in models like OpenAI’s o1 drop dramatically, “from 98% to below 2%”.

The Malicious-Educator benchmark exposes how “extremely dangerous or malicious requests” can be disguised “beneath seemingly legitimate educational prompts”. This research reveals that “attackers can easily extract criminal strategies from DeepSeek-R1 and Gemini 2.0 Flash Thinking without any additional tricks”, highlighting fundamental vulnerabilities in current safety approaches.

In addition, the ability of reasoning models to generate detailed CoT can be weaponized by attackers to create more convincing prompts that bypass safety filters. This raises the stakes for ensuring that safety mechanisms are robust and capable of handling sophisticated manipulation attempts.

5.3 Alignment Challenges in Reasoning Systems

The integration of reasoning capabilities creates new alignment challenges. While reasoning models can “reason about our safety policies in context when responding to potentially unsafe prompts, through deliberative alignment”, this same capability can be exploited by sophisticated attacks. The transparency of reasoning processes, while beneficial for interpretability, also provides attack vectors that didn’t exist in traditional LLMs.

Research indicates that reasoning models still exhibit sensitivity to probability distributions from their training data, suggesting that “optimizing a language model for reasoning can mitigate but might not fully overcome the language model’s probability sensitivity”. This indicates that fundamental limitations from autoregressive training may persist even in reasoning-optimized systems.

5.4 Hallucination in Reasoning Contexts

Despite their enhanced reasoning capabilities, reasoning models continue to exhibit hallucination patterns, particularly in constraint satisfaction problems. Research on graph coloring tasks reveals that reasoning models are “prone to hallucinate edges not specified in the prompt’s description of the graph”. This phenomenon “persists across multiple problem complexity levels and semantic frames” and “appears to account for a significant fraction of the incorrect answers from every tested model”.

These findings suggest that reasoning models may have “broader issues with misrepresentation of problem specifics”, indicating that the enhanced reasoning capabilities don’t fully address fundamental issues with information fidelity and accuracy.

5.5 Scaling and Efficiency Considerations

While reasoning models demonstrate impressive capabilities, they incur significant computational costs. The variable test-time compute approach means that complex problems can require substantially more resources than traditional LLM inference. This creates practical deployment challenges, particularly for applications requiring consistent response times.

The relationship between reasoning quality and computational cost remains unclear. Research indicates that more thinking time generally leads to better performance, but the optimal allocation of computational resources across different problem types remains an active area of investigation.

6. Conclusion

Reasoning AI models, such as o1, o3, R1, and Phi-4, mark a shift towards systems that execute algorithmic steps rather than relying purely on black-box prediction. Unlike traditional LLMs, these models leverage chain-of-thought reasoning, curated data, and advanced fine-tuning to solve complex tasks—though this comes with increased training and inference complexity.

Fine-tuning reasoning models demands specialized methods and high-quality data, as their reasoning chains are both powerful and vulnerable to inconsistency or reward hacking. Effective deployment requires both technical expertise and organizational investment; however, the benefits include clearer explanations and deeper insights across domains such as finance and science.

Evaluation now extends beyond final answers to include scrutiny of the reasoning process itself, using metrics such as trace faithfulness and process accuracy. This makes model behaviour more transparent and trustworthy.

For practitioners, reasoning models become collaborative problem-solvers, offering logical breakdowns for tasks from coding to contract analysis. But maintaining reliable reasoning and avoiding hallucinations requires ongoing vigilance and tailored oversight.

The center of gravity has shifted from pick a reasoning model to use a unified system with routed reasoning,’ with explicit controls for compute and explanation; this aligns with your agentic guidance and simplifies deployment ergonomics. In the near future, with this direction, we expect to see more robust state representations, verification-based training, and compositional planning; evaluate under router-aware, deception-aware protocols, and replicate Apple-style stress tests with fixed effort/latency budgets.

The focus is shifting toward unified systems that route and manage reasoning explicitly, enabling robust evaluation and compositional planning. Reasoning AIs won’t replace standard LLMs everywhere, but they excel in high-stakes scenarios requiring transparency. As techniques mature, these models will become more stable and interpretable, merging pure reasoning with external tools and knowledge. Teams adopting these models should invest in robust pipelines and new evaluation metrics to realize the benefits of interpretable, verifiable solutions—a step forward for AI’s ability to explain not just what or when, but how and why.

References

What is KV Cache in LLMs and How Does It Help?

Sat, 14 Jun 2025 00:00:00 +0000

TL;DR: KV cache is a memory optimization central to efficient LLM inference. It enables faster, longer, and more cost-effective generation by caching previously computed attention keys and values—unlocking the practical deployment of models like GPT-4o, Llama 3, etc.

1. Introduction

Generative AI, powered largely today by Large language models (LLMs) such as GPT-4o, Llama 3, etc., is transforming AI applications, from chatbots to code assistants and multimodal reasoning. As these models scale in size and context length, inference becomes a major computational and memory challenge. The Key-Value (KV) cache is a pivotal optimization that enables practical, high-performance inference in modern transformer architectures.

In my experience, after speaking with many individuals, including customers at work (mostly enterprises), I’ve found that most don’t fully understand what a KV cache is or why they should care. In this post, I aim to provide an overview of what KV cache is, how it helps, and outline some recent research innovations. I also have simple code samples for practical understanding.

At the most basic level, a KV cache is a memory optimization technique used in LLMs to improve inference efficiency during generation. The KV cache stores the key and value tensors generated during the attention mechanism of transformer architectures, allowing models to avoid redundant computations when generating sequential text. To understand the KV cache, it’s essential to grasp how self-attention works in transformers.

2. Transformer Attention and the Role of KV Cache

As part of the transformer architecture, the attention mechanism enables models to dynamically assess the importance of various elements in the input sequence and calculate relationships between input tokens through three components: queries (Q), keys (K), and values (V).

For each token, the model computes:

Query vectors ($Q$): Represent the current element seeking information
Key vectors ($K$): Act as reference points for all elements in the sequence
Value vectors ($V$): Contain the actual information that will be aggregated

The attention computation follows the formula: $$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax} \left( \frac{QK^T}{\sqrt{d_k}} \right) V $$

LLMs are autoregressive; that is, text generation refers to the sequential process by which a model predicts each new token (word or subword) based on all previously generated tokens. This creates a dependency chain: every new token depends on the entire history of prior tokens. During autoregressive text generation, where models predict one token at a time using all previous tokens, the KV cache serves as a repository to “remember” the pre-computed key and value pairs from earlier tokens. Each new token requires attending to all previous tokens. Without optimization, this would necessitate recomputing all $K$ and $V$ matrices at every step, leading to quadratic time complexity.

3. The Caching Process

The KV cache stores the computed key and value tensors for all previously generated tokens. When generating a new token, only its key and value are computed and appended, while the model attends to the full cache. This caching reduces redundant computation, transforming each inference step’s complexity from quadratic to linear with respect to sequence length—a foundational efficiency for scaling LLMs to long contexts.

Without KV caching, transformers would have to recompute keys and values for all previous tokens during each generation step, resulting in quadratic computational complexity. The KV cache removes this inefficiency through the following process:

Initial Generation: When processing the first input token, the model calculates and stores its key and value vectors in the cache.
Subsequent Tokens: For each new token, the model only computes the key and value for that specific token.
Cache Appending: New key-value pairs are appended to the existing cache.
Attention Computation: The model uses the complete cached key-value history to compute attention with the current query.

This approach transforms the attention computation from quadratic $O(n^2)$ to linear $O(n)$ complexity in terms of sequence length.

Example: Minimal KV Cache in PyTorch. This class provides a minimal example of how to store and update key-value tensors for autoregressive generation in a transformer model.

class KVCache:
    def __init__(self):
        self.cache = {"key": None, "value": None}

    def update(self, key, value):
        if self.cache["key"] is None:
            self.cache["key"] = key
            self.cache["value"] = value
        else:
            self.cache["key"] = torch.cat([self.cache["key"], key], dim=1)
            self.cache["value"] = torch.cat([self.cache["value"], value], dim=1)

    def get_cache(self):
        return self.cache

4. Memory Requirements and Bottlenecks

There is no free lunch, of course, and there are also practical memory considerations. The memory footprint of the KV cache is substantial, especially for long contexts. For large models, this can easily consume tens of gigabytes for long sequences, often exceeding the memory needed for model weights themselves.

For example, with Llama-2-7B using half precision (FP16), for batch size 1, the KV cache consumes approximately 0.5MB per 1000 token. For a sequence of 28,000 tokens, this equals about 14GB of memory - the same amount required to store the entire model weights.

Research shows that the KV cache can consume over 30% of GPU memory during deployment and become the primary memory bottleneck for long-context applications. At a simplistic level, the required memory can be estimated as:

$$ \text{Memory} = 2 \times \text{Precision} \times \text{Layers} \times \text{ModelDim} \times \text{SeqLen} \times \text{BatchSize} $$

Note her:

$\text{Precision}$ is typically 2 bytes (FP16)
$\text{Layers}$ is the number of transformer layers
$\text{ModelDim}$ (a.k.a Model Dimension) is the hidden size per layer

5. What Would Happen Without KV Cache?

Understanding the importance of KV cache becomes clearer when you contemplate the consequences of its absence. The overall experience would be significantly worse, characterized by high latency, shorter context windows, and increased costs.

Severe slowdown: Every new token requires recomputing attention for all previous tokens, causing computation to grow quadratically with sequence length
Unsustainable compute overhead: Each step repeats all previous attention calculations, wasting compute and energy.
High latency and poor user experience: Users experience significant lag, especially for long-form or multi-turn conversations.
Limited sequence lengths: Practical context limits shrink, and out-of-memory errors become common for large models.
Inefficient hardware use: Lower throughput and increased energy consumption.
No cache-level optimizations: No prompt reuse, no advanced memory management, and no opportunity for compression.

6. The New Trade-off: Cache Size vs. Model Performance

Recent studies have transformed the balance between KV cache size and model performance. Previously, decreasing the cache size directly impacted model quality, particularly for tasks that required long contexts or retrieval. Now, innovative approaches—utilizing quantization, pruning, and adaptive retention—enable significantly smaller caches, with minimal or virtually no decline in performance.

6.1. Token-Precision Trade-off: Quantized Pruning

A key breakthrough is the realization that storing more tokens at lower precision (quantized pruning) outperforms storing fewer tokens at high precision under the same memory budget.

Key finding: For example storing 4x as many tokens in 4-bit precision outperforms storing 1x tokens in 16-bit precision, especially for long-context and retrieval tasks. Note that the $‘4x’$ factor depends on model architecture, context, and task.
Result: Quantized pruning preserves long-range context and enables robust performance across task types, input lengths, and model scales, even in extreme memory-constrained scenarios.
Stability: This method is robust across various pruning and quantization strategies, providing a new paradigm for cache compression.

Code Example: Quantized Pruning This function demonstrates how to select and quantize the most important tokens in the KV cache to maximize memory efficiency with minimal accuracy loss.

def quantized_pruning(kv_cache, importance_scores, num_tokens_to_keep, num_bits=4):
    # Select top tokens by importance
    top_indices = importance_scores.argsort()[-num_tokens_to_keep:]
    selected_kv = kv_cache[top_indices]

    # Quantize selected tokens
    scale = (selected_kv.max() - selected_kv.min()) / (2**num_bits - 1)
    quantized = ((selected_kv - selected_kv.min()) / scale).round().clamp(0, 2**num_bits - 1)

    return quantized, scale, selected_kv.min()

6.2. Adaptive and Selective Retention

Recent methods, such as FastGen and MorphKV, profile attention patterns at runtime to determine which tokens or cache entries are most relevant for each layer or head. This enables:

Dynamic cache size: The cache adapts to attention diversity, keeping more entries where needed and aggressively compressing elsewhere.
Constant-size caches: MorphKV, for instance, maintains a fixed-size cache by iteratively refining which tokens to keep using attention patterns, preserving long-range dependencies with minimal accuracy loss and >50% memory savings.
Layer/head specialization: Different cache strategies can be applied to different layers or heads, rather than a one-size-fits-all approach.

6.3. Quantization for Throughput and Batch Size

Hardware-aware quantization (e.g., FP8, INT8, 4-bit) dramatically reduces memory requirements and enables higher effective batch sizes, especially in decode-heavy serving scenarios:

Throughput gains: Quantizing the KV cache can provide up to 1.45× throughput improvement in real-world LLM serving, primarily by allowing more requests to be processed in parallel.
Minimal accuracy loss: With careful quantization and dequantization strategies, there is little to no impact on model quality for most tasks.
Implementation caveats: The speedup depends on the compatibility of quantized caches with high-performance attention kernels; some frameworks (e.g., TensorRT-LLM) benefit more than others (e.g., vLLM) depending on kernel optimizations.

6.4. System-Level Optimizations

Beyond algorithmic compression and adaptive retention, recent research has revealed significant performance gains through system-level KV cache optimizations. Frameworks like NVIDIA TensorRT-LLM and vLLM’s PagedAttention have re-architected cache management to resemble operating system virtual memory more closely, using paged or block-based KV storage to minimize memory fragmentation and enable efficient on-demand allocation.

Other innovations, such as FlowKV, introduce distributed and disaggregated cache management strategies to reduce cache transfer latency and better utilize hardware resources across multiple nodes.

These system-level enhancements complement algorithmic advances by improving scalability, throughput, and latency, ensuring that KV cache innovations are effectively translated into real-world production deployments, particularly for large-scale, multi-user LLM inference workloads.

7. Practical Implications

Efficient cache strategies have significant practical implications that affect both memory and computational requirements. By optimizing cache retention and compression techniques, these advancements create smaller yet smarter caches that free up resources. This enables larger batch sizes, longer context windows, and allows for deployment on less expensive or resource-constrained hardware. Such innovations greatly enhance the performance and scalability of language models, making them accessible to a wider range of users and applications.

Moreover, these strategies enhance cost-effectiveness by reducing hardware demands and energy consumption. This not only lowers the financial barriers to deploying language models but also promotes sustainability within the field of machine learning. Additionally, the ability to track long-range dependencies with optimized cache management proves invaluable for tasks such as document retrieval, multi-turn dialogue generation, and summarization. These improvements underscore the importance of adaptive and efficient KV cache techniques in advancing the performance of large language models.

8. Conclusion

KV cache management is a cornerstone of efficient LLM inference, especially as models scale and the size of the context windows expands. The field has rapidly advanced from basic caching to sophisticated, adaptive, and task-aware strategies that strike a balance between memory, speed, and accuracy. Without KV cache, modern LLMs would be too slow, costly, and limited for today’s real-world applications. As research continues, expect even more advanced cache management, enabling efficient inference for ever-larger models and longer contexts.

References

RustySnake - Classic Snake game to learn Rust

Thu, 27 Mar 2025 00:00:00 +0000

1. Overview

Rust has been gaining attention recently due to its unique combination of performance, safety, and modern programming features. Its strict ownership model eliminates common memory issues like null pointer dereferencing and data races, providing a secure environment for developers. At the same time, its expressive syntax and focus on developer productivity make it a strong contender for systems programming. Its growing ecosystem and community continue to expand its capabilities into areas like web development and embedded systems, ensuring a confident and secure learning experience for developers.

When diving into Rust, a language lauded for its speed and safety, I wanted to start with a fun and educational project. Inspired by my experience with C and C++, I decided to learn Rust by implementing a classic snake game - a project that not only provides a fun learning experience but also reinforces an understanding of programming fundamentals. This journey has introduced me to Rust’s unique features and reinforced my understanding of programming fundamentals, engaging and motivating me to learn more.

As I embarked on this journey, I also wanted to understand how LLMs can assist in grasping Rust in terms of syntax, layout, flow, and execution aspects. These models played a crucial role in helping me debug and resolve issues, providing the necessary support and guidance in my learning process. For this, I used the following different models. Except for GPT 4o, I also want to run the others on my home GPU cluster - again, to help experiment with different aspects of inference.

Phi 4
Llama 3.3 70b Instruct
Qwen 2.5B Coder 32B
DeepSeek R1 Distill Llama 8B
GPT 4o

1.1. LLM Performance Comparison

Model	Code Generation	Error Explanation	Documentation	Overall Experience
Phi 4	✔️✔️✔️	✔️✔️	✔️✔️	Good for simpler tasks
Llama 3.3 70b	✔️✔️✔️✔️	✔️✔️✔️	✔️✔️✔️	Strong but inconsistent
Qwen 2.5B Coder 32B	✔️✔️✔️✔️	✔️✔️✔️	✔️✔️✔️✔️	Excellent for code examples
DeepSeek R1 Distill 8B	✔️✔️	✔️✔️	✔️✔️	Limited understanding of Rust
GPT 4o	✔️✔️✔️✔️✔️	✔️✔️✔️✔️✔️	✔️✔️✔️✔️✔️	Comprehensive assistance throughout

Of course, it is not an apples-to-apples comparison. Each started well but then quickly went downhill for various reasons. The only awesome one I finally ended up using all the way through was GPT4o.

1.2. Why Rust?

Rust is designed to be a systems-level language with modern features. It eliminates common pitfalls like memory safety issues without compromising performance. As someone familiar with C and C++, I think this seems like a natural progression.

2. The Snake Game Project

   /^\\/^\\
  / o   o \\
 (    ^    )
  \\_______/
   |     |
   |     |

Rusty Snake

The nostalgic, classic snake game provides a perfect playground for learning Rust’s syntax, constructs, and libraries. For the game, I started with the basic game but then soon added other things - I wanted to give the player a choice on the size of the play area (measured in number of characters as the rendering is in ASCII) and speed.

A crate in Rust is a collection of related code akin to a library in C++ or a module in Python. The project imports the following crates for different functionalities needed:

crossterm: For terminal UI and input handling.
rand: For generating random numbers (e.g., food placement).

Let us walk through some key implementation parts and draw parallels to C and C++ for easier comprehension.

3. Key Concepts in the Code

3.1 Structs: Rust’s Version of `struct` in C

Rust’s struct is similar to struct in C and C++ but with additional safety and functionality. Here’s how we define a Point struct to represent 2D coordinates:

#[derive(Debug, Clone, PartialEq)]
struct Point {
    x: i32,
    y: i32,
}

Parallel in C/C++:

struct Point {
    int x;
    int y;
};

In Rust, #[derive(...)] automatically implements traits like Debug (for debugging), Clone (for copying), and PartialEq (for comparisons).

3.2 Vectors `Vec<T>`): Dynamic Arrays

Rust uses Vec<T> for dynamically-sized arrays, similar to std::vector in C++.

let mut snake = vec![Point { x: width / 2, y: height / 2 }];

Parallel in C++:

std::vector<Point> snake = { Point{width / 2, height / 2} };

Unlike C++, Rust ensures safety with bounds checking and ownership rules.

3.3 Ownership and Borrowing

Rust’s ownership model ensures memory safety without a garbage collector. When working with snake, you own or borrow the data.

Example:

let mut new_head = snake.last().unwrap().clone();

Here, .clone() creates a deep copy of the last element to avoid ownership issues.

Parallel in C++: Copying would be explicit but lacks ownership enforcement:

Point new_head = snake.back();

3.4 Enums and Pattern Matching

Rust’s pattern matching with match is more expressive than a switch statement in C++.

match key_event.code {
    KeyCode::Up if direction.y == 0 => next_direction = Point { x: 0, y: -1 },
    KeyCode::Down if direction.y == 0 => next_direction = Point { x: 0, y: 1 },
    _ => {}
}

Parallel in C++: A switch with additional if conditions might approximate this but would lack the same level of elegance.

3.5 Error Handling

Rust avoids exceptions by using the Result type for error handling, relying on explicit and predictable control flow to manage errors. This differs from other languages and ensures errors are handled explicitly rather than relying on potentially disruptive exception mechanisms, which can lead to unwieldy codebases in large systems.

fn main() -> crossterm::Result<()> {
    terminal::enable_raw_mode()?;
    ...
    terminal::disable_raw_mode()?;
    Ok(())
}

The ? operator propagates errors automatically, akin to C++’s std::optional or manual error handling:

if (!enable_raw_mode()) return -1;

3.6 Functional Features

Rust’s iterators and closures make code concise and expressive. For instance, generating random food coordinates:

let new_food = Point {
    x: rng.gen_range(1..width - 1),
    y: rng.gen_range(1..height - 1),
};

Parallel in C++:

Point new_food = { rand() % (width - 2) + 1, rand() % (height - 2) + 1 };

Rust’s range syntax and rand crate simplify random number generation.

4. Code

The complete source code for this project is available on GitHub: https://github.com/bahree/rustysnake

The game loop handles input, updates the snake’s position, checks for collisions, and renders the game state. The crossterm library provides a simple way to handle keyboard input and terminal rendering.

loop {
    // Handle input
    if crossterm::event::poll(Duration::from_millis(game_speed))? {
        if let Event::Key(key_event) = event::read()? {
            match key_event.code {
                KeyCode::Char('q') => break,
                KeyCode::Up if direction.y == 0 => direction = Point { x: 0, y: -1 },
                KeyCode::Down if direction.y == 0 => direction = Point { x: 0, y: 1 },
                KeyCode::Left if direction.x == 0 => direction = Point { x: -1, y: 0 },
                KeyCode::Right if direction.x == 0 => direction = Point { x: 1, y: 0 },
                _ => {}
            }
        }
    }

    // Update snake position
    let new_head = Point {
        x: (snake[0].x + direction.x + width) % width,
        y: (snake[0].y + direction.y + height) % height,
    };

    // Check collision with self
    if snake.contains(&new_head) {
        break;
    }

    snake.insert(0, new_head);

    // Check if snake ate food
    if snake[0] == food {
        // Generate new food
        loop {
            let new_food = Point {
                x: rng.gen_range(0..width),
                y: rng.gen_range(0..height),
            };
            if !snake.contains(&new_food) {
                food = new_food;
                break;
            }
        }
    } else {
        snake.pop();
    }

    // ...
}

Feel free to clone, fork, or contribute to the repository!

4.1 Running the Game

To run this on Windows:

Install Rust with rustup.

Add dependencies to Cargo.toml:

[dependencies]
crossterm = "0.27"
rand = "0.8"

Build and run the project:
```
cargo run
```
Compile the project:
```
cargo build --release
```

4.2 Controls

Key	Action
`Arrow Keys`	Move the snake
`+`	Increase game speed
`-`	Decrease game speed
`Spacebar`	Pause/Resume the game
`q`	Quit the game

4.3 Gameplay

Select the boundary size and difficulty from the menu.
Use arrow keys to move the snake.
Eat the red food (■) to grow your snake and increase your score.
Avoid hitting the walls (#) or yourself!

Snake Game

5. Learnings

I encountered a few things while learning Rust through this snake game project. These are called out below. As with anything else, where things are different, one must forget what one knows and learn the new way of doing things.

Ownership model: Coming from C/C++, adjusting to Rust’s strict borrowing rules required a shift in thinking
Terminal UI limitations: Working with crossterm’s terminal interface meant designing around ASCII graphics
Game loop timing: Balancing responsiveness with performance required careful tuning
Error propagation: Learning to use the ? operator and Result types effectively took practice

Understanding these differences was part of the learning process and helped solidify some of my understanding of Rust’s principles.

5.1 Reflecting on Rust

Building this game helped me appreciate Rust’s features:

Memory Safety: Rust eliminates segmentation faults common in C and C++.
Expressive Syntax: Iterators, pattern matching, and traits make Rust elegant and powerful.
Community and Ecosystem: Libraries like crossterm and rand accelerate development.

5.2 Acknowledgment

Throughout this journey, I used the assistance of LLMs to clarify Rust concepts and draw parallels to languages I already know. This collaborative approach made learning smoother and deepened my understanding by offering different perspectives. The combination of hands-on learning and real-time assistance has been invaluable.

6 Final Thoughts

Rust might feel different initially if you’re from a C or C++ background, but its safety guarantees and modern features make it worthwhile. Building a snake game is just one of many ways to start.

What will you build next? 📎

An introduction to Mixture of Experts (MoE)

Wed, 01 Jan 2025 00:00:00 +0000

AI is advancing at an unprecedented pace, with Mixture of Experts (MoE) models being one set of model architectures at the forefront of this revolution. These architectures enable breakthroughs in efficiency and scalability by leveraging a modular design where only a subset of specialized “expert” networks are activated for each input. MoE architectures have become a cornerstone in building ultra-large-scale models like GLaM and Switch Transformers.

Mixture of Experts (MoE) is an advanced machine learning architecture that lately has gained significance, particularly in the realm of #LLMs (large language models) and NNs (neural networks). In talking with many people about AI, I’ve found that MoE as a topic comes up often, with many folks either not understanding what it is or their understanding of it being incorrect.

With recent announcements of trillion-parameter models and announcements from Microsoft, OpenAI, and Google, understanding MoEs is more important than ever. More recently, DeepSeek v3 is a great example of a model that uses MoEs to achieve state-of-the-art performance - where the language model has 671B total but only 37B activated for each token.

I this post I provide a high-level overview of MoEs, their core components, how they work and workflow. I also includes a simple toy example implementation to help grasp the core concepts.

Why Mixture of Experts?

The central motivation behind MoE stems from the tension between growing model size and limiting computational resources. As we have seen in the recent past, increasing the parameter count of a model often yields better performance, especially in domains like natural language processing (NLP) and computer vision; however, this also drastically increases the cost of both training these models and computing cost for inference of these models. This massive computing cost is at the heart of what MoEs are addressing by offering a different paradigm known as conditional computation. MoE’s activate only a small subset of specialized sub-networks (called experts) for each input or token) only to a small number of experts rather than processing it through every parameter in the network. This helps with three key aspects:

Increased Model Capacity: Because only a few experts are activated at a time, MoE architectures can pack many parameters (experts) without proportionally increasing the computational cost per input.
Specialized Sub-networks: Different experts can learn specific patterns or token-level specializations, leading to better performance.
Efficient Usage of Compute: MoE optimizes resources (compute) by activating only a small fraction of the entire model, leading to significant efficiency gains.

This sparse activation strategy enables constructing models with billions (or even trillions) of parameters, making MoE a scalable solution for large-scale applications. Another benefit of MoE is its capacity for specialization. Different experts can learn distinct, context-dependent processing strategies, enabling the model to cover a broad set of input variations more effectively than a monolithic architecture.

Early MoE ideas trace back to model ensembling in classic machine learning. Still, MoE extends beyond ensembling by learning a parametric “router” (gating function) that dynamically decides which experts to use. Notable works like the Sparsely-Gated Mixture of Experts showed that MoE could massively scale model size while staying computationally efficient.

Core Components of an MoE System

The fundamental building blocks of an MoE system are:

Experts: Specialized sub-models, typically feed-forward neural networks.
Router or Gating Network: Determines which tokens are sent to which experts.
Combiner: Aggregates the outputs from the selected experts.

Experts

Each expert is typically a neural sub-network replicated multiple times within the same model. Depending on the task, these experts are often implemented as independent feed-forward networks, MLP blocks, or convolutional layers. For large-scale language models, each expert usually mirrors the structure of the feed-forward component of a Transformer block.

Since multiple experts exist in parallel, each can potentially learn to handle different token types or data distributions. Contrary to common misconceptions, experts do not necessarily correspond to human-like semantic domains (e.g., “Expert #3 = Physics”). One might assume that certain experts correspond to high-level domains such as “finance” or “medicine.” Still, it is more common for the learned expertise to be more subtle and token-level, capturing idiosyncratic patterns that are not necessarily interpretable in a straightforward semantic way. Each expert ends up specializing in token-level or feature-level patterns that are discovered during training. For instance, in NLP, one expert might specialize in syntactic structures, while another focuses on semantic relationships.

Sometimes, certain layers (like embeddings or attention blocks) are shared among all experts, and only feed-forward layers are duplicated (as in Switch Transformers). This partial sharing allows the model to keep some global representation while still having specialized processing in the experts.

Router

The router, often called the gating network, is a small module that predicts which experts should handle any given input. Its purpose is to determine which experts should handle a given input. Modern MoE designs are typically parameterized as a simple neural network (often a single linear layer + SoftMax) or a simple linear transform. The SoftMax output provides a probability distribution across the experts, indicating which should be “activated” for each input.

The router reads the input representation (e.g., the token embedding in an NLP model) and produces a probability distribution over the experts, typically with a SoftMax function. We then select the top-k experts (e.g., top-1, top-2) based on these probabilities - for each token, known as Top-k gating.

Practical implementations often limit the number of tokens an expert can process per batch (known as Expert Capacity). If too many tokens route to the same expert, some tokens may get dropped or rerouted, leading to training instability. This capacity limit helps prevent any single expert from monopolizing the model’s processing and prevents load imbalances.

The gating network is trained jointly with the experts through back-propagation. As we outlined above, the gating process introduces discrete decisions into the computational graph, which can hamper backpropagation. To help counter this, additional techniques such as adding a small amount of noise to the logits (Noisy Top-k Gating) or using Soft MoE — are employed to smooth out these discrete selections, keep training stable, help provide smoother gradients, and encourage balanced expert utilization. Additional mechanisms like Expert Capacity limit how many tokens each expert can process, preventing load imbalances in which a single expert might receive most tokens.

Combiner

Once the chosen experts have computed their outputs for a given token, these outputs must be aggregated to produce a single vector that feeds into subsequent layers. The MoE architecture achieves this through a combiner, which typically performs a weighted sum of the experts’ outputs, using the gating probabilities as weights. In top-k gating, if $ k $ experts were activated, each expert’s output is multiplied by its corresponding probability from the router. The combiner then sums or otherwise fuses the results to form the token’s transformed representation. This consolidated output is passed into the rest of the model, such as attention blocks or additional Transformer layers.

In “Soft MoE” variants, we might use a soft combination, letting tokens pass to all experts but with different fractional weights—alleviating some routing discontinuities at the cost of higher computation.

High-Level Workflow

Input is fed into the gating network, which produces a probability distribution (or scores) over experts.
The top-k experts are activated for each input (or token, in the case of language models).
The selected experts process the input in parallel.
A combiner fuses the experts’ outputs into a single vector.
The model produces the final output, which can then feed into other layers or tasks.

Below is a simple flow diagram that shows how to visualize this:

flowchart TD
    Input_Tokens --> Gating_Network
    Gating_Network --> Expert_1
    Gating_Network --> Expert_2
    Gating_Network --> Expert_3
    Expert_2 --> Combiner
    Expert_3 --> Combiner
    Combiner --> Final_Output

The MoE module often appears in LLMs where a standard feed-forward layer normally would reside. For each layer designated as an MoE, the process begins by sending the hidden representations of each token to the router. The router computes gating probabilities across all experts, ranks the experts by these probabilities, and picks the top few. Each selected expert applies its transformation to the token’s hidden state. Finally, the combiner merges these expert outputs with a weighted sum.

During training, the experts and the router are updated jointly through backpropagation. However, discrete gating can make gradient flow tricky since the top-k selection is not inherently differentiable. In practice, noise injection (Noisy Top-k Gating) or methods like Soft MoE can help approximate continuous gradients, ensuring that even experts with lower gating probabilities receive occasional training signals.

In addition, to avoid a scenario where one or two experts monopolize all tokens, an auxiliary load-balancing loss is often introduced to encourage more uniform usage. This might take the form of a penalty term that grows when the variance in expert usage is high, incentivizing the router to distribute tokens more evenly.

This mechanism occurs for every token in every layer designated as an MoE layer, which is why load balancing is so critical—without careful design, certain experts can receive far more tokens than others.

What is the Difference between an MoE and an Ensemble?

It’s easy to confuse MoE with model ensembling - where multiple independently trained models vote or average predictions. MoE differs in a few critical ways:

Dynamic and data-dependent routing: In classical ensembles, each model sees the same input, and a meta-learner or a simple average produces the final result. MoE, in contrast, uses a router that decides different subsets of the input for each expert. This dynamic routing allows MoE to specialize in different patterns or tokens.
Single model training: MoE typically trains all experts jointly in one go. Each expert does not have a separate training pass; they’re part of the same computation graph, sharing some parameters (like embeddings) and learning together. This is in contrast to ensembles, where each model is trained independently.
Fine-grained token specialization: Different tokens in the same sentence might get routed to different experts, enabling extremely fine-grained specialization. This is impossible in a traditional ensemble, where each model sees the entire input.

Load Balancing Experts and Training Pitfalls

A significant challenge in training MoE models is ensuring the balanced utilization of experts. Certain experts may become overburdened without proper load balancing, while others remain underutilized, leading to inefficient training and suboptimal performance. Gating can become highly imbalanced early in training, favoring a few experts. Common solutions for Load Balancing:

Auxiliary Load Balancing Losses: Adding a regularization term to the loss function encourages the gating network to distribute inputs evenly across all experts.
Top-k Randomization: Instead of always selecting the top-k experts with the highest gating probabilities, randomizing the selection among the top candidates can prevent overloading.
Expert Capacity Constraints: Limiting the number of tokens an expert can process at a time can help ensure all experts are used during training.

Over time, load-balancing losses/techniques help the distribution even out, but the model can remain fragile without careful hyperparameter tuning. Expert capacity is another important design choice. Since top-k selection may route too many tokens to the most popular experts, a capacity limit ensures each expert processes no more than a certain maximum number of tokens in one forward pass. The remaining tokens must be dropped or re-routed to other experts if an expert is at capacity. Both approaches come with trade-offs: dropping tokens entirely can lead to data inefficiency, whereas re-routing can add complexity and undermine the sparsity benefits that MoE aims to provide.

Load balancing - ensuring experts share the training load - remains one of the biggest technical hurdles with MoEs. Early in training, the gating network might discover that routing most tokens to the same expert (or a small number) might yield acceptable results, leaving other experts underutilized and effectively “dead.” This imbalance leads to suboptimal solutions - if only a few experts get trained on all tokens, you lose the advantages of specialization.

To mitigate this, MoE architectures often introduce additional techniques that nudge the model toward more even usage. One such technique is the auxiliary load-balancing loss. By monitoring how frequently each expert is selected, the model can be penalized if certain experts remain underutilized. This can be thought of as an additional penalty term that measures how evenly tokens (or batches) are distributed across experts.

Another common approach is Noisy Top-k Gating, which injects a small amount of learnable or fixed noise into the gating logits before the softmax, making the gating probabilities slightly more random. This randomness allows less-popular experts to occasionally receive tokens, which can help them develop more useful specializations over time. The gating network thus learns not only to minimize the primary task loss but also to spread tokens more uniformly.

Sparsity in Mixture of Experts (MoE) Models

Sparsity is one of MoE’s most valuable contributions to model efficiency. By only activating a small fraction of the total parameters for each input, the model maintains a much lower compute footprint than a dense model of the same overall size. This efficiency is crucial for scaling; while a trillion-parameter dense model may be prohibitively expensive to train and deploy, a trillion-parameter MoE model that only activates 1% of those parameters simultaneously becomes significantly more tractable.

That said, implementing sparsity at scale often requires specialized infrastructure. Frameworks like GShard or Mesh-TensorFlow are designed to handle data and model parallelism necessary for distributing the experts across GPU clusters. The overhead of routing tokens to the correct devices can become significant if the system is not carefully optimized. Researchers have also explored alternative gating mechanisms, such as Soft MoE, which approximates selection by routing every token to all experts in a soft, weighted fashion. While this approach can mitigate the fragility of discrete gates, it naturally increases computation since more experts perform computations at once.

Sparcity in MoE models offers several key advantages:

Computational Efficiency: Sparsity dramatically reduces the number of FLOPs required to process each task.
Scalability: The sparse activation of experts enables MoE models to scale to a large number of experts without a corresponding linear increase in computational and memory requirements.
Increased Model Capacity: Sparsity allows MoE models to increase their overall parameter count and model capacity without significantly increasing the computational cost during training or inference.
Memory Efficiency: Operating sparsely, MoE models require less memory for activations and parameters.
Specialized Processing: Sparsity enables the model to route different inputs to the most relevant experts, allowing for more specialized and efficient processing of diverse inputs.

Practical applications of MoE

MoE architectures have already demonstrated clear benefits in many areas. Microsoft’s Z-code model (Machine Translation), for instance, leverages MoE to handle multilingual translation tasks at a massive scale, and Google’s Switch Transformers showed that sparse activation can reach higher quality at lower training cost than dense baselines on benchmarks such as GLUE and SuperGLUE. In computer vision, MoE modules have been integrated into Vision Transformers (V-MoEs) to achieve better image classification and detection accuracy, with each expert focusing on different aspects of the image representation. In multimodal learning, the capacity to handle diverse data types—such as text, images, and audio—makes MoE a natural fit because experts can adapt to different modalities or different subproblems within a single modality.

In the context of LLMs, systems like ChatGPT, Claude, and Gemini can benefit from MoE by leveraging different experts for different topics or query types - though specifics are often proprietary and not shared. MoE is particularly suited to multi-modal tasks involving text, images, and audio, as experts can specialize in different modalities or sub-modalities. This is valuable for text-to-image generation or video understanding.

Several emerging directions continue to push MoE research forward. Soft MoE (Zuo et al., 2022) is an example aiming to produce a fully differentiable version of sparsely gated Transformers. Another is Parameter-Efficient Sparsity Crafting (PESC), which seeks to retrofit existing dense models into a sparse MoE design without retraining from scratch. These innovations reflect ongoing efforts to refine the balance between sparse efficiency, training stability, and model reliability.

In production, deploying a large MoE model requires carefully coordinating hardware resources, data pipelines, and load-balancing techniques. Training an MoE system may involve more hyperparameters than a comparable dense model, including the number of experts, gating softmax temperature, top-k value, load-balancing penalty weights, and expert capacity. These factors can significantly affect performance, convergence speed, and final accuracy. When scaling an MoE model across multiple GPUs, designers must pay attention to network communication overhead. Token-based routing leads to collective operations that can become bottlenecks if not carefully optimized.

Despite these complexities, MoE’s flexibility and computational cost savings make it a compelling choice for handling highly varied or large-scale tasks. However, Fine-tuning MoE models can be more delicate than dense models because the gating distributions or specialized experts may not adapt smoothly to a new domain without carefully applied load-balancing strategies. There can also be interpretability challenges since the model’s internal “expert structure” does not always map neatly to skills.

Challenges and Considerations

We have touched on most of these in this blog post, but it is helpful to outline the key issues and considerations to be mindful of when using MoE-based models:

Complexity: MoE models are significantly more complex (compared to traditional neural networks) and require substantial computational resources for training and inference.
Training Instability: MoE models can suffer from training instability due to the discrete nature of expert selection.
Load Balancing: Proper load balancing among experts is crucial for efficiently using model capacity and optimal performance.
Computational Overhead: The gating mechanism introduces additional computational overhead, potentially impacting training and inference times.
Interpretability Issues: The dynamic routing of inputs makes interpreting how MoE models arrive at their decisions challenging.
Hyperparameter Sensitivity: MoE models have several hyperparameters that must be tuned for optimal performance.

Let’s define these mathematically before discussing practical considerations.

Mathematical Formulation of MoE

Let $ x \in \mathbb{R}^d $ denote an input vector (a hidden representation from a preceding layer). We assume the system has $ N $ experts, with each expert $ E_i $ parameterized as a function $ E_i: \mathbb{R}^d \rightarrow \mathbb{R}^m $. The gating network $ G $ takes the same input $ x $ and outputs a vector in $\mathbb{R}^N$—essentially, a “score” or “weight” for each of the $ N $ experts. Finally, a combiner function $ C $ merges the experts’ outputs into a single output vector $ y \in \mathbb{R}^m $.

Formally, we can write:

Input: $ x \in \mathbb{R}^d $
Experts: $ E_i: \mathbb{R}^d \rightarrow \mathbb{R}^m \quad \text{for}\ i = 1, \ldots, N $
Gating Network: $ G: \mathbb{R}^d \rightarrow \mathbb{R}^N $
Combiner: $ C: \mathbb{R}^{N \times m} \rightarrow \mathbb{R}^m $

Top-k Gating

Most contemporary MoE implementations use top-k gating, which activates only the $ k $ experts with the highest gating scores. In this scenario, the summation is performed only over those top-$ k $ indices. If we denote $\text{top-k}(G(x))$ as the set of indices corresponding to the $ k $ largest values of $ G(x) $, then

$ y = C\Bigl(\sum_{i \in \text{top-k}(G(x))} G(x)_i ,\cdot, E_i(x)\Bigr). $

By pruning all but the top-$ k $ experts per input, this design enforces sparse activation: each input (or token) only “touches” $ k $ out of $ N $ experts at a time. This approach significantly reduces the computational load relative to using all $ N $ experts for every input.

Dense Gating Formulation

This refers to a version of MoE architecture in which all experts contribute to the final output for each input rather than filtering out all but the top-k experts. Here, the gating network assigns continuous weights to every expert, aggregating each expert’s weighted output into a final result. There is no discrete selection to zero out certain experts based on gating. Fundamentally, this is the opposite of top-K gating.

In the simplest version of MoE, where the gating network’s output is a set of continuous weights, the forward pass for one input $ x $ can be written as:

$ y = C\Bigl(\sum_{i=1}^N G(x)_i ,\cdot, E_i(x)\Bigr). $

Here, $ G(x)_i $ is the $i$-th component of the gating network’s output for $ x $ and represents the weight (or probability) assigned to expert $ i $. Intuitively, if $ G(x)_i $ is large, expert $ i $ contributes more to the final output $ y $. The function $ C $ often takes the form of a simple weighted sum or concatenation-and-projection step, depending on the specific design.

Load Balancing and Expert Monopolization

As we saw earlier, MoE architectures introduce an *auxiliary load-balancing loss*to load valance across all experts. This can be thought of as an additional penalty term that measures how evenly tokens (or batches) are distributed across experts. One common strategy is penalizing the variance in expert usage or encouraging each expert to receive roughly an equal proportion of examples. The gating network thus learns not only to minimize the primary task loss but also to spread tokens more uniformly.

Mathematically, a typical load-balancing loss might look like:

$ \mathcal{L} _ {\text{balance}} = \lambda \sum_{i=1}^N \Bigl(\frac{f_i}{\sum_j f_j} - \frac{1}{N}\Bigr)^2, $

where $ f_i $ is the total number of tokens assigned to expert $ i $ in a minibatch (or the sum of gating probabilities if you’re using a continuous measure), and $ \lambda $ is a hyperparameter controlling the strength of this penalty. This ensures the model is incentivized to explore and train all experts over time.

Example PyTorch Implementation

Here’s a simplified Python implementation of an MoE model using PyTorch:

import torch
import torch.nn as nn

class Expert(nn.Module):
    def __init__(self, input_size, output_size):
        super(Expert, self).__init__()
        self.layer = nn.Linear(input_size, output_size)
    
    def forward(self, x):
        return self.layer(x)

class GatingNetwork(nn.Module):
    def __init__(self, input_size, num_experts):
        super(GatingNetwork, self).__init__()
        self.layer = nn.Linear(input_size, num_experts)
        self.softmax = nn.Softmax(dim=1)
    
    def forward(self, x):
        return self.softmax(self.layer(x))

class MixtureOfExperts(nn.Module):
    def __init__(self, input_size, output_size, num_experts, top_k=1):
        super(MixtureOfExperts, self).__init__()
        self.num_experts = num_experts
        self.top_k = top_k
        self.experts = nn.ModuleList([Expert(input_size, output_size) for _ in range(num_experts)])
        self.gating = GatingNetwork(input_size, num_experts)
    
    def forward(self, x):
        batch_size = x.size(0)
        gating_probs = self.gating(x)
        topk_vals, topk_inds = torch.topk(gating_probs, self.top_k, dim=1)
        expert_outputs = torch.zeros(batch_size, self.top_k, output_size, device=x.device)
        for i in range(self.top_k):
            inds = topk_inds[:, i]
            outputs_for_expert = torch.stack([self.experts[inds[b]](x[b].unsqueeze(0)) for b in range(batch_size)])
            expert_outputs[:, i, :] = outputs_for_expert.squeeze(1)
        topk_vals_expanded = topk_vals.unsqueeze(-1)
        weighted_sum = expert_outputs * topk_vals_expanded
        combined_output = weighted_sum.sum(dim=1)
        return combined_output

# Example Usage
input_size = 10
output_size = 5
num_experts = 4
top_k = 2
model = MixtureOfExperts(input_size, output_size, num_experts, top_k)
sample_input = torch.randn(8, input_size)
output = model(sample_input)
print("Output shape:", output.shape)

When you run this code, you should see the output shape printed as torch.Size([8, 5]). This confirms that the model routes each input to the most relevant experts, processes the input through those experts, and combines their contributions into a unified output.

Code Explanation

This code defines a simple MoE model with a single gating network and multiple experts. The Expert class represents a single sub-module, the expert, a linear layer that transforms the input and forms the backbone of the MoE architecture. The expert takes the input of size input_size and transforms it to output_size using a fully connected (Linear) layer. In the real world, each expert is designed to specialize in different transformations based on their training.

The GatingNetwork class is the router that computes gating probabilities for each expert and determines which experts to activate for a given input. It takes the input size and the number of experts as input and outputs a probability distribution over the experts (num_experts) using a linear layer followed by a Softmax function. Higher probability values indicate that the corresponding expert is more relevant to the input.

The MixtureOfExperts class combines the experts’ outputs based on the gating probabilities and returns the final output. It takes the input size, output size, number of experts, and the top-k value as input. The forward method computes the gating probabilities, selects the top-k experts based on these probabilities, and computes the weighted sum of the expert outputs to produce the final output. The top_k parameter controls how many experts are activated for each input.

The input ( x ) is passed through the gating network to produce a probability distribution over the experts: gating_probs = self.gating(x)
The gating network selects the indices of the top_k experts with the highest probabilities: topk_vals, topk_inds = torch.topk(gating_probs, self.top_k, dim=1)

For each of the selected experts, process the input:

expert_outputs = torch.zeros(batch_size, self.top_k, output_size, device=x.device)
for i in range(self.top_k):
    inds = topk_inds[:, i]
    outputs_for_expert = torch.stack([self.experts[inds[b]](x[b].unsqueeze(0)) for b in range(batch_size)])
    expert_outputs[:, i, :] = outputs_for_expert.squeeze(1)

Multiply each expert’s output by its gating probability and sum them to form the final output:
```
topk_vals_expanded = topk_vals.unsqueeze(-1)
weighted_sum = expert_outputs * topk_vals_expanded
combined_output = weighted_sum.sum(dim=1)
```
- The topk_vals (probabilities) are weights for the corresponding expert outputs.

The model processes a batch of inputs and returns the combined output with a shape matching (batch_size, output_size). For the example above, Output shape: torch.Size([8, 5]) confirms that the model routes each input to the most relevant experts, processes the input through those experts, and combines their contributions into a unified output.

Code Dependencies

To run this code, you’ll need the following dependencies:

Python 3.6 or later (preferably 3.10 or higher)
PyTorch 1.0 or later

Save the code to a file, e.g., moe_example.py, and run it:

python moe_example.py

I used Conda, which I prefer for managing Python environments. If you’re using a virtual environment, you can adapt the installation commands accordingly. For Conda, you can create a new environment and install PyTorch using the following steps.

Start by creating a new conda environment with PyTorch dependencies. In your terminal, execute the following commands:

# Create a new conda environment named "moe_env"
conda create -n moe_example python=3.10 -y

# Activate the environment
conda activate moe_example

Install PyTorch and necessary dependencies; adjust the CUDA version based on your system’s GPU configuration. You can omit the pytorch-cuda package using a CPU-only setup.

# For CPU-only:
conda install pytorch torchvision torchaudio cpuonly -c pytorch -y

# For GPU (use appropriate CUDA version, e.g., 11.7):
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia -y

# Install additional dependencies if needed
pip install numpy

This example demonstrates the basic structure of an MoE model, including the experts, gating network, and the MoE module that combines them. Of course, this is a toy version that helps understand the basic construct and does not include advanced features like load balancing or sparsity, etc.

Conclusion

Mixture of Experts offers a compelling framework for scaling neural networks and managing the trade-offs between model size, computational cost, and performance. By selectively activating only a subset of parameters for each input, MoE allows researchers and practitioners to build models with enormous capacity without incurring a proportionate computational penalty. Given the additional interest in inference optimization for LLMs and broadly with Transformer-based architecture, we expect to see further innovations and applications for MoE.

References

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - https://arxiv.org/abs/1701.06538
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding - https://arxiv.org/abs/2006.16668
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - https://arxiv.org/abs/2101.03961
A Lightweight Mixture-of-Experts Neural Machine Translation Model with Stage-wise Training Strategy - https://aclanthology.org/2024.findings-naacl.154/
Deep Mixture of Experts via Shallow Embedding - https://arxiv.org/abs/1806.01531
Mixture of Experts in Image Classification: What’s the Sweet Spot? - https://arxiv.org/abs/2411.18322v1
Mesh-TensorFlow: Deep Learning for Supercomputers - https://proceedings.neurips.cc/paper/2018/hash/3a37abdeefe1dab1b30f7c5c7e581b93-Abstract.html
A Generalist Agent - https://arxiv.org/abs/2205.06175
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation - https://arxiv.org/abs/1308.3432
From Sparse to Soft Mixtures of Experts - https://arxiv.org/abs/2308.00951
Parameter-Efficient Sparsity Crafting for Large Language Models - https://aclanthology.org/2024.emnlp-main.43/

Automating Hugo Deployments

Mon, 23 Dec 2024 00:00:00 +0000

1. A little background

I have been meaning to automate the deployment of my blog post to a dev server (running locally) for a while, but I haven’t had the time to get around to it until now. In addition to deploying this, the dev server also had several constraints. It is one of the machines at home and is not exposed directly to the internet. I also don’t have any ports opened on the firewalls at home, which adds a few more interesting challenges. I have a server hosted online in one of the data centers, but I want to keep that for more ‘production’ things.

Another thing I wanted to do was clean up the theme being used, as it wasn’t compatible with the newer version of Hugo. The longer I was pinned to an older version, the more challenging it would be to update. Now, these are all done and working, and I wanted to share what I learned here—more so because I will likely need to find this at some point in the future, but it might also help someone else.

PS - If you don’t notice much (any?) difference in the blog’s look, that means it is a good thing; I worked through fixing the theme and upgrading to the latest version of Hugo (v0.140.0 at the time of writing this).

While cleaning up, I also got around to things I had wanted to do for a while. I wanted to see how to securely expose one of the machines I have at home online without opening up ports and things and serve several services—mostly for my own use.

1.1 Dev-Server Setup 🖥️

I think outlining how this machine is set up would be helpful, as I am sure many others will be trying this. First, this machine runs locally on the home network, running a headless Ubuntu server. The environment that made this dev server unique was the combination of things - I use Tailscale, Cloudflare Tunnels, CloudPanel, and as outlined earlier Hugo. The code sits in a private Github repo, and the build-server is via Github Actions. I won’t get into the details of each setup here as the intent of the blog post is not that, but I will share some context below.

1.2 Tailscale

To make my life easier between the different machines and servers I have, running both locally at home and on the cloud, including bare-metal machines, I use Tailscale.

Tailscale simplifies network management by creating a secure, zero-config mesh VPN based on the WireGuard protocol. Unlike traditional VPNs that route all traffic through a central server, Tailscale forms direct, encrypted connections between devices, regardless of their physical location or network configuration. This “mesh” approach offers significant performance benefits, reduced latency, and improved reliability. By assigning each device a stable IP address within a private network namespace, Tailscale eliminates the complexities of managing firewalls, port forwarding, and dynamic DNS, making it ideal for connecting personal devices, servers, and cloud instances into a cohesive, secure network.

What makes Tailscale work like magic is a feature they call MagicDNS. This feature eliminates the need to remember IP addresses and SSH keys for different machines. Using Tailnet, I get a private VPN between different machines, and I can use that to connect to any of these without opening any ports, etc. This significantly simplifies the networking process and enhances security.

1.3 CloudPanel

I use CloudPanel, a free and open-source server control panel for managing web servers and applications. In the context of this blog post, CloudPanel provides a user-friendly web interface for creating and managing websites, databases, email accounts, and DNS records. It’s a crucial part of the automation process, allowing for efficient server management and deployment. Its lightweight design and efficient resource utilization make it great for me.

1.4 Cloudflare Tunnels

Cloudflare Tunnels play a crucial role in the deployment process. They offer a secure and efficient way to expose web services running on private networks without opening inbound ports on firewalls. By establishing an outbound-only connection between a lightweight daemon (cloudflared) running on your server and Cloudflare’s global network, Tunnels create an encrypted tunnel that allows external users to access your services through Cloudflare’s edge. This approach enhances security by eliminating the attack surface associated with open ports while leveraging Cloudflare’s performance optimizations, such as caching, DDoS protection, and global CDN. It’s a powerful tool for self-hosting applications, APIs, and websites from home networks or private infrastructure without compromising security.

2. Server Setup ⌨️

To get this to work at a high level, we will do the following to get all of this working. This post won’t get into all the details of setting up the prerequisites, and it assumes that you have the following:

CloudPanel installed on your machine.
Cloudflare account with a Cloudflare Tunnel set up.
GitHub repo containing the Hugo project.
Tailscale for private networking - this is in addition to the HTTP traffic that Cloudflare handles.

2.1. Create a Static Site in CloudPanel

In CloudPanel, create a new “Site” (from the Dashboard). Select Static, and assign it a domain, e.g., blog.desigeek.com. CloudPanel will create a directory structure under something like:

/home/[cloudpanel_username]/htdocs/blog.desigeek.com

(Exact path can vary based on CloudPanel’s naming conventions.)

The figure below shows an example of how I set this up.

2.1.1 Create an SSH User

If you haven’t already, go to CloudPanel’s “Users” section. You can create an SSH user (for example, blog-ssh). This user will be mapped to the same group or path used by CloudPanel so that it can write your site files.

Important:

Make sure the SSH user’s home directory either is or can write to the directory created earlier /home/[cloudpanel_username]/htdocs/blog.desigeek.com.
For our example, I created an SSH user called blog-ssh, and the user’s group on Ubuntu was called bahree-blog.
Note: on some CloudPanel installations, the user’s primary group might be something else, not necessarily bahree-blog.

The image below shows an example of how to set this up.

2.1.2. Directory Paths and Permissions

Confirm that your SSH user (blog-ssh) can write into the blog.desigeek.com directory:

# Example:
ls -ld /home/[cloudpanel_username]/htdocs/blog.desigeek.com
# Make sure the owner or group matches your user or a group the user is in.

# If needed:
sudo chown -R blog-ssh:bahree-blog /home/[cloudpanel_username]/htdocs/blog.desigeek.com
sudo chmod -R 755 /home/[cloudpanel_username]/htdocs/blog.desigeek.com

(Adjust bahree-blog to the actual group that CloudPanel assigned.)

2.2 Cloudflare Tunnel Configuration

Cloudflare Tunnel is typically set up via Cloudflare’s admin dashboard or command line (see the docs for more details).

We will map the domain blog.desigeek.com to the local server’s port 80 (or 443). Since CloudPanel listens on port 80/443, your tunnel might forward something like:

Cloudflare subdomain → localhost:80 on your server.

Because Cloudflare is reverse proxying your site, you won’t need to open inbound ports on your server. Ensure your DNS is configured so that blog.desigeek.com points to the Cloudflare Tunnel.

The image below shows an example of what this would look like.

2.3. Setup Tailscale Secure Private Access

To deploy the Hugo site on my home dev server, I need private remote access (SSH, SCP, rsync, etc.) between the GitHub build servers and the home server. I achieve this using Tailscale.

Install Tailscale on your server:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

The figure below shows an example of an ephemeral node, which created and deleted in approx. a minute.

Our Build server is essentially spun up when something is committed in a specific branch, and as a result, we need to get that added to the Tailscale network - but it should not be permanent. Over time, that will pollute the tailnet and make things more difficult to manage. Tailscale has a perfect solution for this - Ephemeral nodes. These nodes, as the name suggests, make it easier to connect and then clean up short-lived devices such as containers, cloud functions, or CI/CD systems that spin up and spin down on a regular basis.

To use this, we generate an Ephemeral Auth Key from the auth keys page of the Tailscale admin console. Given that the build server would need this, we store this key in GitHub as TS_AUTH_KEY.

This allows the ephemeral GitHub Actions runner to connect to my dev server privately, even when I don’t have any inbound ports open.

2.4. Nginx Configuration via CloudPanel

CloudPanel typically manages the Nginx configuration for each “Site” automatically. It also handles SSL certificates, logs, etc. However, in this case, we are relying on Cloudflare Tunnel, and as a result, we do not need SSL termination on the server; Cloudflare can handle it and forward HTTP internally. We therefore need to ensure that the config file (vhost) in CloudPanel is minimal. Below is an example of what I use:

server {
  listen 80;
  listen [::]:80;
  server_name blog.desigeek.com;
  {{root}}

  {{nginx_access_log}}
  {{nginx_error_log}}

  location ~ /.well-known {
    auth_basic off;
    allow all;
  }

  {{settings}}

  include /etc/nginx/global_settings;
  index index.html;

  location ~* ^.+\.(css|js|jpg|jpeg|gif|png|ico|gz|svg|svgz|ttf|otf|woff|woff2|eot|mp4|ogg|ogv|webm|webp|zip|swf)$ {
    add_header Access-Control-Allow-Origin "*";
    expires max;
    access_log off;
  }

  if (-f $request_filename) {
    break;
  }
}

Note: Please note that {{root}} points to /home/[cloudpanel_username]/htdocs/blog.desigeek.com, which is set up in the CloudPanel site configuration.

The figure below shows what this would look like in CloudPanel.

3. GitHub Actions Configuration

We finally get to the main thing of building out GitHub actions that will trigger whenever something is checked in on a certain branch. All the other things until now were to essentially expose this dev machine externally (specifically the blog) and allow the build server to deploy to this dev server running at home. If you have a publicly accessible dev machine, much of this might not be required.

3.1. Storing Variables & Secrets in GitHub

We start by storing the variables and secrets we need for this to work. The details are outlined below. To set these up, go to Settings → Secrets and variables → Actions in your GitHub repo.

Variables (non-sensitive):

SERVER_HOSTNAME = my-dev-server (or your Cloudflare Tunnel/hostname). If Tailscale is used, it might be the Tailscale IP/hostname.
- Note: If you are using Tailscale, the notation of machine-name.local will not resolve and fail, even with MagicDNS enabled. It is best to use the machine name (e.g., my-dev-server and not my-dev-server.local or the IP address from the Tailnet)
SERVER_USERNAME = blog-ssh (the SSH user you created in CloudPanel).
HUGO_VERSION = e.g., 0.140.0
HUGO_EXTENDED = true (if you want the extended version, otherwise false).

Secrets (sensitive):

SSH_PASSWORD = the password for blog-ssh; this was either set or auto-generated when you created the account in CloudPanel.
TS_AUTH_KEY = Your Tailscale ephemeral auth key (only if using Tailscale).

The screenshot below shows what this would look like:

3.2 GitHub Actions Workflow

The code below outlines a sample workflow for this. It checks out the code, builds it with Hugo, and, once complete, copies it over to the dev server, where I can expose it securely to test it. To allow us to do this, we take a dependency on sshpass and rsync.

I do have a few things to check as this happens to ensure there aren’t any failures. When Hugo is finished, we show some basic stats, such as the number of files and folders created. This is called Show public directory stats.

Another is to test the SSH connection, as without that, the build machine cannot talk to the dev server where the final site is getting deployed. This step is called Test SSH Connection. At a high level, here is the flow that this goes through:

Check out the code
Installs Tailscale and connects as an ephemeral node
Builds with Hugo
Installs sshpass & rsync
Uses rsync to deploy to the server
Rolls back if the build fails
Logs out of Tailscale at the end

Below is a sample .github/workflows/deploy-hugo.yml.

name: Deploy Hugo Site

on:
  push:
    branches:
      - main  # or your chosen branch

jobs:
  deploy:
    runs-on: ubuntu-latest

    # Gets non-sensitive values from "Variables" (not Secrets)
    env:
      SERVER_HOSTNAME: ${{ vars.SERVER_HOSTNAME }}
      SERVER_USERNAME: ${{ vars.SERVER_USERNAME }}
      HUGO_VERSION: ${{ vars.HUGO_VERSION }}
      HUGO_EXTENDED: ${{ vars.HUGO_EXTENDED }}

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      # Optional Tailscale login if you're using Tailscale for SSH
      - name: Setup Tailscale
        uses: tailscale/github-action@v3
        with:
          authkey: ${{ secrets.TS_AUTH_KEY }}
          statedir: /tmp/tailscale-state/

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: ${{ env.HUGO_VERSION }}
          extended: ${{ env.HUGO_EXTENDED }}

      - name: Build Hugo site
        id: build-site
        run: |
          hugo --minify -d public

      - name: Verify Hugo Output
        if: steps.build-site.outcome == 'success'
        run: |
          if [ ! -d "public" ] || [ -z "$(ls -A public)" ]; then
            echo "Error: 'public' directory is empty or does not exist."
            exit 1
          fi
          echo "Public directory exists and is not empty."

      - name: Show public directory stats
        if: steps.build-site.outcome == 'success'
        run: |
          cd public
          echo "Files: $(find . -type f | wc -l)"
          echo "Directories: $(find . -type d | wc -l)"
          echo "Size: $(du -sm . | cut -f1) MB"

      - name: Install sshpass and rsync
        run: |
          sudo apt-get update
          sudo apt-get install -y sshpass rsync

      - name: Test SSH Connection
        run: |
          sshpass -p "${{ secrets.SSH_PASSWORD }}" \
            ssh -o StrictHostKeyChecking=no ${{ env.SERVER_USERNAME }}@${{ env.SERVER_HOSTNAME }} \
            'echo "SSH Connection Successful"'

      - name: Deploy via rsync
        if: steps.build-site.outcome == 'success'
        env:
          SSH_PASSWORD: ${{ secrets.SSH_PASSWORD }}
        run: |
          sshpass -p "$SSH_PASSWORD" rsync -az --delete \
            -e "ssh -o StrictHostKeyChecking=no" \
            public/ ${{ env.SERVER_USERNAME }}@${{ env.SERVER_HOSTNAME }}:/home/[cloudpanel_username]/htdocs/blog.desigeek.com

      - name: Rollback on Failure
        if: steps.build-site.outcome != 'success'
        uses: appleboy/ssh-action@master
        with:
          host: ${{ env.SERVER_HOSTNAME }}
          username: ${{ env.SERVER_USERNAME }}
          password: ${{ secrets.SSH_PASSWORD }}
          script: |
            BACKUP_DIR="/home/[cloudpanel_username]/backups"
            SITE_DIR="/home/[cloudpanel_username]/htdocs/blog.desigeek.com"
            LATEST_BACKUP=$(ls -t "$BACKUP_DIR"/*.tar.gz | head -n 1)

            if [[ -n "$LATEST_BACKUP" ]]; then
              echo "Rolling back to: $LATEST_BACKUP"
              rm -rf "$SITE_DIR"/*
              tar -xzvf "$LATEST_BACKUP" -C "$SITE_DIR" --strip-components=1
              echo "Rollback successful!"
            else
              echo "No backups found to rollback to!"
            fi

      - name: Tailscale Logout
        if: always()
        run: |
          # If Tailscale requires sudo or operator privileges to log out,
          # either configure passwordless sudo or skip this if ephemeral.
          sudo tailscale logout

Note: Update /home/[cloudpanel_username]/htdocs/blog.desigeek.com to match the actual path CloudPanel created. The same applies to the backup directory if you use that mechanism.

The figures below show an example of when this was building, and one of the validations.

4. Tips & Troubleshooting 🚨

I wanted to share a few things I have learned as I went through and debugged this. Hopefully this will help someone else too.

Permission Denied: Make sure your CloudPanel SSH user has ownership or group-writable permission on /home/[cloudpanel_username]/htdocs/blog.desigeek.com.
Rsync Overwrites: The --delete flag in rsync removes files on the server that no longer exist locally. Remove ‘- delete ’ if you don’t want that behavior.
Cloudflare Tunnel: If the site is not reachable at blog.desigeek.com, check your Cloudflare DNS config or the tunnel routing.
SSH: You might still rely on Tailscale or an open SSH port for SSH since Cloudflare typically only proxies HTTP/HTTPS.
Tailscale Logout: If you see “Access denied: logout access denied,” you can skip tailscale logout (ephemeral nodes eventually vanish) or configure passwordless sudo for Tailscale.

5. Conclusion

By combining CloudPanel for site management, Cloudflare Tunnel for secure external traffic, Tailscale for private networking, and GitHub Actions for automated builds, you get a robust, convenient pipeline to deploy your Hugo blog. 😍

CloudPanel simplifies web server config and user management.
Cloudflare Tunnel ensures secure traffic without exposing direct ports.
Tailscale provides easy, private SSH if desired.
GitHub Actions automates the entire Hugo build and deployment process.

Now, when you push changes to your repository, your CloudPanel-managed site updates automatically, so no manual FTP or SSH is required!

Happy Deploying! ☺️

AI generated Podcast for my book: Generative AI in Action 🎧

Sun, 13 Oct 2024 00:00:00 +0000

The one thing I wanted to do after my book Generative AI in Action was complete was to create a summary in natural speech and possibly use TTS (Text-to-speech) to create an audio summary—think of it as a podcast that is easier for people to consume and get a quick sense of what the book is about.

TTS (Text to Speech) or not to TTS?

Initially, I was inclined towards using TTS (Text to Speech) for the audio summary. This technology, I thought, would be a convenient way to create a podcast-like summary that would be easier for people to consume and get a quick sense of what the book is about. My journey began with TTS - using GPT 4o to create a summary after ingesting the book and then using that as into the Azure AI Speech stack . However, I stumbled across something intriguing. Instead of TTS, I opted for NotebookLM from Google Labs to generate the audio overview - the podcast. This decision marked a significant shift in my approach, and I created two podcasts using the content from the book - one from multiple sources and another from a single source (the book).

NotebookLM is an experimental AI-first notebook from Google Labs designed to help users gain insights faster by grounding the language model in their documents. It aims to assist with synthesizing facts and ideas from multiple sources, making connections quicker and easier. It can help users understand, summarize, and generate new ideas based on their content. What is fascinating is that it can generate audio, which is a natural dialog between two people - with wit, humor, and a natural flow. It is like conversing with someone who has read the book and is summarizing it for you. And if I hadn’t told you that this was AI-generated, it would be hard to tell that it was not a real conversation.

The “Podcasts”

For the first audio generation, the multiple sources I used were:

The book, and my blog post announcing the book
A real podcast I did with Miko on his podcast Hockeystick
And another real podcast with Jamie on his podcast The modern .NET show .

For the second audio generation I only used a single source - the book.

The results of the AI-generated audio are truly impressive. In each instance, the audio was produced in a natural voice, simulating a genuine conversation between two people. The quality of the audio, the conversation, and the flow are nothing short of mind-blowing. Even in the second audio, where some acronyms were not pronounced correctly, it’s a minor issue, considering the audio was generated in just a few minutes with the press of a button. I was genuinely surprised at the level of realism and natural flow in the conversation. 🤯

Have a listen and let me know what you think.

Podcast Summary 1 - using multiple sources

The sources provide a comprehensive overview of generative AI and its application within enterprises. The first source, a YouTube video transcript, features an interview with Amit Bahree, a technical program manager at Microsoft, who discusses the rise of large language models (LLMs) and their potential impact on society. The second source, a book excerpt, delves into the technical aspects of generative AI, covering foundational models, large language models, retrieval-augmented generation, and the architectural principles for building generative AI applications. The book also explores various use cases, including image generation, code generation, and the ethical considerations surrounding the use of generative AI.

Your browser does not support the audio element.

Book Podcast - using multiple sources

Podcast Summary 2 - using single source (the book)

This book is a comprehensive guide to Generative AI, focusing on how this transformative technology can be leveraged within an enterprise. It explains core concepts like foundational models and large language models (LLMs) as well as practical applications for generating various content, including text, images, code, audio, and video. The book also explores responsible AI practices, highlighting the importance of prompt engineering, ethical considerations, and security measures for implementing these technologies. The author emphasizes the need for careful evaluation, monitoring, and scalability when deploying Generative AI models in a production environment.

Your browser does not support the audio element.

Book Podcast - using single source

Conclusion

AI-generated podcasts showcase the potential of AI to revolutionize content creation. Generating natural-sounding audio summaries from text is a game-changer for authors, creators, and educators. As AI continues to advance, we anticipate more opportunities to improve how we access information. I am excited to explore the new possibilities in AI-generated content.

Backing up TeslaMate data to OneDrive

Fri, 05 Jul 2024 00:00:00 +0000

I have been running a couple of instances of Teslamate - one locally on a server at home and another in Azure in a Ubuntu VM (see 👉 this blog post for details). I have been backing up the data to a NAS and then an offsite backup for the local instance. For the Azure instance, I have been running various backups during the day and backing up the data to OneDrive. This allows me to have a data backup in case the VM crashes or the data gets corrupted.

Work and AI have kept me busy, and I have not had a chance to write about it until now. I have a ton of storage on OneDrive, which seemed the most logical place to store it. Multiple options are available to back up data to OneDrive. The one that has been working consistently and well for me is rclone.

What is Rclone?

Rclone Rclone is an open-source command-line tool for managing files across various cloud storage services. It supports over 70 providers, including Google Drive, Amazon S3, Dropbox, and Microsoft OneDrive. Rclone allows users to perform operations like copying, moving, syncing, and deleting files and more advanced tasks such as encryption, compression, and chunking. Additionally, it can mount cloud storage as a local drive, facilitating seamless file access and management.

Known as the “Swiss army knife of cloud storage,” Rclone is highly versatile and can be integrated into scripts for automating complex file operations. Its ability to act as a bridge between different cloud providers makes it an invaluable tool for efficiently transferring files and managing cloud storage environments.

Step 1: Setting up Rclone

The base assumption is that you have a working instance of Teslamate running in a docker container on a Linux machine. Rclone is also available for Windows and Mac, and you can follow the [instructions] ( https://rclone.org/install/ ) to install it on those platforms. Let us start by outlining how to install rclone.

First, let’s install rclone on your system. You can download and install it using the following commands:

curl https://rclone.org/install.sh | sudo bash

Step 1.1 - Configuring Rclone with OneDrive ☁️

After installing rclone, we need to configure it to use OneDrive. Rclone has detailed steps on how to configure OneDrive. I show some of them here on how to run the following command to start the configuration process:

rclone config

You’ll be prompted with several questions about setting up a new remote. Here is an example configuration for OneDrive:

No remotes found - make a new one
n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> n
name> onedrive

Type of storage to configure.
Choose a number from below, or type in your own value.
...
24 / Microsoft OneDrive
   \ "onedrive"
...
Storage> 24

Microsoft App Client Id
Leave blank normally.
client_id> 

Microsoft App Client Secret
Leave blank normally.
client_secret> 

Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y

For the last prompt (auto config), if you are on a headless machine (e.g., you had SSH’d into a server to set this up), you can say n, and it will give you a URL to open in a browser to authenticate - as these require OAuth2. On the host (from which you are connected), you can authenticate and get the token, which you can paste into the terminal window. If the browser does not open, you can open your local browser and navigate to http://127.0.0.1:53682/auth.

If you are on SSH, you might need to create an SSH tunnel over port 53682 to your local machine using the following command:

ssh -L localhost:53682:localhost:53682 username@remote_server

For more details on how to do this, refer to the rclone remote setup documentation.

It is important to remember the name you give to the remote, as you will need it in the backup script later. In our example, we called it onedrive.

Also, note that the rclone configuration is stored in the .config directory in the user’s home directory. The configuration file is named rclone.conf and contains the details of the remotes you have configured. You can view the configuration file by running:

cat ~/.config/rclone/rclone.conf

To test the configuration, you can run the following command:

rclone ls onedrive:

This should list the files and directories in your OneDrive account. If you see the list of files, the configuration is successful. If you encounter any errors, review the configuration and ensure the authentication details are correct. For example, the image below shows the successful configuration of the OneDrive remote:

Remote OneDrive folder listing

Step 1.2 - Mounting Remote Directory with Rclone 📁

Now that we have configured rclone with OneDrive, we can mount the remote directory to a local directory in the system. This allows us to access the remote files as if they were local files. We cannot use a regular mount command, as the rclone mount command is not a real drive but is a FUSE-based mount. To mount the remote directory, use the following command.

rclone mount --config=/home/amit/.config/rclone/rclone.conf --vfs-cache-mode full --allow-non-empty --allow-other --daemon --dir-perms 0777 car_backup:car_data2 /home/amit/onedrive

Let’s break down the command:

In the above command, replace /home/amit/.config/rclone/rclone.conf with the path to your rclone configuration file.
Replace car_backup:car_data2 with the remote name (configured in Sep 3) and the directory you want to mount.
The local directory /home/amit/onedrive is where the remote directory will be mounted (you might need to create a new folder for this). You can choose any local directory for this purpose.
In the above command, the --daemon flag runs the mount in the background, and the --dir-perms 0777 flag sets the directory permissions.
The --allow-non-empty flag allows mounting over a non-empty directory.
And finally, the --allow-other flag allows other users to access the mounted directory.

Once the command is executed, you should see the remote directory mounted to the local directory. You can access the files in the remote directory as local files. For example, you can list the files in the mounted directory using the ls command, as shown in the image below:

Listing mounted OneDrive directory

Step 1.3 - Persistent Mounting with Rclone 💾

It is important to note that the rclone mount command is not persistent across reboots. So, if your VM reboots for some reason, you will lose this. To make the mount persistent, you can add the command to the system’s startup scripts or use a tool like systemd to manage the mount as a service. We will create a systemd service file to manage the mount for this.

To do this, create a new service file using the following command:

sudo nano /etc/systemd/system/rclonemount.service

Add the following content to the file:

[Unit]
Description=rclonemount
AssertPathIsDirectory=/home/amit/onedrive
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=amit
Group=amit
ExecStartPre=/bin/sleep 10
ExecStartPre=/bin/mkdir -p /home/amit/onedrive
ExecStartPre=/bin/chmod 777 /home/amit/onedrive
ExecStart=/usr/bin/rclone mount \
        --config=/home/amit/.config/rclone/rclone.conf \
        --vfs-cache-mode full \
        --allow-other \
        --dir-perms 0777 \
        car_backup:/car_data2 /home/amit/onedrive
ExecStop=/bin/fusermount -uz /home/amit/onedrive
Restart=always
RestartSec=10
Environment=HOME=/home/amit
Environment=USER=amit

[Install]
WantedBy=default.target

In the above service file, the ExecStartPre`` commands create the directory and set the permissions before mounting the remote directory. The ExecStartcommand mounts the remote directory to the local directory. TheExecStopcommand unmounts the directory when the service is stopped. TheRestartandRestartSec` settings ensure the service is restarted in case of failure.

The image below shows the contents of the rclonemount.service file in my system:

rclonemount.service contents

After creating the service file, reload the systemd daemon and start the service using the following commands:

sudo systemctl daemon-reload

To test the service, start it using the following command:

sudo systemctl start rclonemount

And check the status of the service to ensure it is running:

sudo systemctl status rclonemount

If everything is working correctly, you should see the status of the service as active (running), as shown in the image below:

rclonemount service status

Note: Press Q to quit.

And finally, if the service is running without any errors, enable it to start at boot:

sudo systemctl enable rclonemount

Of course, the service can be tested by rebooting the system and checking if the remote directory is mounted automatically.

Yay! Congratulations! You have successfully configured rclone to mount the remote directory as a service. The remote directory will now be mounted automatically on system startup. 👍

Step 2 - Basic Backup Script 🛠️

Now that we have rclone set up and mounted, we can create a basic backup script to back up the Teslamate data to OneDrive. The script will perform the following steps:

Back up the Teslamate database.
Compress the backed-up file.
Move the compressed file to the remote backup directory using rclone.

The will be a simple bash script that can be run manually or scheduled using a cron job. Here’s an example of the basic backup script. Create a new file named backup.sh and add the following content to it:

#!/bin/bash
# Basic backup script

SOURCEDIR=/home/amit/teslamate # Source directory where the data is located
BACKUPDIR=/home/amit/onedrive # Remote directory mounted via rclone
FILENAME=teslamate-$(date +%Y-%m-%d-%H%M%S).bak.gz # Date-time stamp filename

# Perform the database dump and compress it
docker compose exec -T database pg_dump -U teslamate teslamate | gzip > $FILENAME

# Move the compressed file to the remote backup directory
rclone move $FILENAME $BACKUPDIR

You will need to replace the SOURCEDIR and BACKUPDIR with the actual directories on your system. The FILENAME variable creates a unique filename for the backup file using the current date and time. The script uses pg_dump to back up the Teslamate database and rclone move to move the compressed file to the remote directory.

This script assumes the following:

You have Teslamate running in a docker container, and the database is named teslamate.
You have rclone configured with OneDrive and mounted the remote directory to a local directory.
You have the necessary permissions to access the source and destination directories.

Make the script executable:

`1`	`chmod +x /home/amit/backup.sh`

Run the script manually to test it:

`1`	`/home/amit/backup.sh`

You should see the backup file in the remote directory if the script runs successfully. You can check the remote directory using the ls command or by navigating to it.

Step 3 - Adding Logging 🗒️

To make the script more informative, let’s add logging. This will help us keep track of the backup process and identify any issues. We will use the same script as above as our starting point and add logging to it. We will log messages to a log file and display them on the console. Here’s the updated script with logging:

#!/bin/bash
# Backup script with logging

SOURCEDIR=/home/amit/teslamate # Source directory where the data is located
BACKUPDIR=/home/amit/onedrive # Remote directory mounted via rclone
FILENAME=teslamate-$(date +%Y-%m-%d-%H%M%S).bak.gz # Date-time stamp filename
LOGFILE=/home/amit/backup.log

# Function to log messages
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOGFILE
}

# Start of the script
log "Starting backup script."

# Perform the database dump and compress it
docker compose exec -T database pg_dump -U teslamate teslamate | gzip > $FILENAME
if [ $? -eq 0 ]; then
    log "Database dump and compression successful: $FILENAME"
else
    log "Database dump and compression failed"
    exit 1
fi

# Move the compressed file to the remote backup directory
rclone move $FILENAME $BACKUPDIR
if [ $? -eq 0 ]; then
    log "Successfully moved backup to $BACKUPDIR"
else
    log "Failed to move backup to $BACKUPDIR"
    exit 1
fi

log "Backup script completed successfully."

As in the earlier step, run the script manually to test it and check the log file for messages.

Step 4 - Adding Email Notifications 📧

Finally, we want to be notified about the backup status. We can add email notifications to the script. We’ll use ssmtp for this purpose. This simple mail transfer agent can send email notifications from the command line. We’ll update the script to send an email notification after the backup is completed.

To send email notifications, you need to have an SMTP server configured. You can use your email provider’s SMTP server for this purpose. Our first step is to set up ssmtp for email notifications

Step 4.1 - Installing and Configuring `ssmtp`

Install ssmtp on your system:

`1`	`sudo apt-get install ssmtp`

Edit the /etc/ssmtp/ssmtp.conf file to add your SMTP server details:

`1`	`sudo nano /etc/ssmtp/ssmtp.conf`

Add the following lines, replacing the placeholders with your actual details:

root=your-email@example.com
mailhub=smtp.your-email-provider.com:587
AuthUser=your-email@example.com
AuthPass=your-email-password
UseSTARTTLS=YES
TLS_CA_File=/dev/null
FromLineOverride=YES

Edit or create the /etc/ssmtp/revaliases file:

`1`	`sudo nano /etc/ssmtp/revaliases`

Add the following line, replacing with your actual details:

`1`	`your-username:your-email@example.com:smtp.your-email-provider.com:587`

To test the email configuration, we can create a simple script called test_email.sh that sends a test email:

#!/bin/bash
# Test email script

FROM_NAME="Teslamate Server"
FROM_EMAIL="the-from-email-address"
EMAIL="email-address-where-you-want-to-send-the-email"
SUBJECT="Test Email"
BODY="This is a test email to verify the SMTP configuration."

echo -e "To: $EMAIL\nFrom: \"$FROM_NAME\" <$FROM_EMAIL>\nSubject: $SUBJECT\n\n$BODY" | ssmtp $EMAIL

You should receive a test email at the specified address when you run the script. If you encounter any issues, review the configuration and ensure the SMTP server details are correct.

Step 4.2 - Timeout for Email Notifications

We can add a timeout to the email command to prevent the script from hanging indefinitely if the email-sending process takes too long. We’ll use the timeout command to set the duration of the email sending. First, we must check if the timeout command is available on your system. You can check this by running the following command:

timeout --version

If the timeout command is not available, you can install it using the following command:

sudo apt-get install coreutils

Step 4.3 - Updating the Backup Script with Email Notifications

Now that we have ssmtp set up for email notifications, we can update the backup script to send an email notification after the backup is completed. We’ll add a function to send email notifications and call it at the end of the script. Here’s the updated script that includes email notifications:

#!/bin/bash
# Backup script with logging and email notifications

SOURCEDIR=/home/amit/teslamate # Source directory where the data is located
BACKUPDIR=/home/amit/onedrive # Remote directory mounted via rclone
FILENAME=teslamate-$(date +%Y-%m-%d-%H%M%S).bak.gz # Date-time stamp filename
LOGFILE=/home/amit/backup.log
EMAIL="email-address-you-want-to-send-the-email-to" # Replace with your email address
FROM_NAME="Teslamate Server"
FROM_EMAIL="your-email@example.com" # Replace with the authorized sender email address
TIMEOUT_DURATION=10 # Set timeout duration (in seconds) for sending email
RCLONE_CONFIG="/home/amit/.config/rclone/rclone.conf" # Path to the rclone config file

# Initialize a variable to capture the entire log
FULL_LOG=""

# Function to log messages
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOGFILE
    FULL_LOG+="$1\n"
}

# Function to send email notification
send_email() {
    local STATUS="$1"
    local SUBJECT="Backup ${STATUS} - $(date '+%Y-%m-%d %H:%M:%S')"
    local BODY="The backup script ${STATUS} on $(date).\n\nLog details:\n$FULL_LOG"
    
    echo -e "To: $EMAIL\nFrom: \"$FROM_NAME\" <$FROM_EMAIL>\nSubject: $SUBJECT\n\n$BODY" | timeout $TIMEOUT_DURATION ssmtp $EMAIL
    EMAIL_EXIT_CODE=$?
    log "Email command exit code: $EMAIL_EXIT_CODE"

    if [ $EMAIL_EXIT_CODE -eq 124 ]; then
        log "Email sending timed out after $TIMEOUT_DURATION seconds"
    elif [ $EMAIL_EXIT_CODE -ne 0 ]; then
        log "Failed to send email"
    else
        log "Email sent successfully"
    fi
}

# Start of the script
log "Starting backup script."

# Change to the source directory
cd $SOURCEDIR
if [ $? -eq 0 ]; then
    log "Changed directory to $SOURCEDIR"
else
    log "Failed to change directory to $SOURCEDIR"
    send_email "Failed"
    exit 1
fi

# Perform the database dump and compress it
docker compose exec -T database pg_dump -U teslamate teslamate | gzip > $FILENAME
if [ $? -eq 0 ]; then
    log "Database dump and compression successful: $FILENAME"
else
    log "Database dump and compression failed"
    send_email "Failed"
    exit 1
fi

# Move the compressed file to the remote backup directory
rclone --config $RCLONE_CONFIG move $FILENAME $BACKUPDIR
if [ $? -eq 0 ]; then
    log "Successfully moved backup to $BACKUPDIR"
else
    log "Failed to move backup to $BACKUPDIR"
    send_email "Failed"
    exit 1
fi

log "Backup script completed successfully."

# Send email notification
send_email "Completed Successfully"

If you run the script, you should receive an email notification after completing the backup. The email will contain the log details of the backup process. You can customize the email content and format as needed. The image below shows an example email notification:

Email notification

Step 4.4 - Scheduling the Backup Script

You can schedule the script to run regularly using a cron job to automate the backup process. You can create a cron job to run the script daily, weekly, or at any interval you prefer. Here’s an example of a cron job to run the script daily:

0 0 * * * /home/amit/backup.sh

This cron job runs the backup.sh script every day at midnight. You can customize the schedule based on your requirements. To add the cron job, open the crontab file using the following command:

crontab -e

Add the cron job to the file and save it. The cron job will now run the backup script at the specified interval.

I have the script running at 5 AM, 1 PM, and 9 PM daily. In addition, I log the output of the cron job to a file (backup_cron.log) so I can review it later if there is an issue. Here are the steps to set this up:

0 5 * * * /home/amit/backup.sh >> /home/amit/backup_cron.log 2>&1
0 13 * * * /home/amit/backup.sh >> /home/amit/backup_cron.log 2>&1
0 21 * * * /home/amit/backup.sh >> /home/amit/backup_cron.log 2>&1

Be sure to replace /home/amit/backup.sh with the actual path to your backup script. The >> /home/amit/backup_cron.log 2>&1 part of the cron job redirects the script output to the specified log file. This allows you to capture the script output and any errors during execution.

Finally, you can check the cron job status using the following command:

crontab -l

This will list the cron jobs that are currently scheduled. You can also check the cron job logs to verify that the script is running as expected.

Here is a snippet of the backup_cron.log file from my system:

2024-07-05 13:00:01 - Starting backup script.
2024-07-05 13:00:01 - Changed directory to /home/amit/teslamate
time="2024-07-05T13:00:01-07:00" level=warning msg="/home/amit/teslamate/docker-compose.yml: `version` is obsolete"
2024-07-05 13:00:30 - Database dump successful: teslamate-2024-07-05-130001.bak
2024-07-05 13:00:55 - Successfully compressed teslamate-2024-07-05-130001.bak.gz
2024-07-05 13:00:57 - Successfully moved backup to /home/amit/onedrive
2024-07-05 13:00:57 - Backup script completed successfully.
2024-07-05 13:00:59 - Email command exit code: 0
2024-07-05 13:00:59 - Email sent successfully

Step 4.5 - Timezone Configuration

Note in most cloud-based VMs, the timezone is set to UTC; if you specify the time in your local timezone, you might want to set the timezone to your local timezone. You can do this by running the following command:

sudo timedatectl set-timezone your-timezone

Replace your-timezone with the appropriate timezone (e.g., America/Los_Angeles).

Conclusion

And that’s it! You now have a backup script that backs up your Teslamate data to OneDrive and sends email notifications about the backup status. The script runs automatically at the scheduled intervals, ensuring your data is backed up regularly. You can customize the script further based on your requirements and preferences. 🫰

SLMs - Running Phi-3 on an iphone and locally

Wed, 08 May 2024 00:00:00 +0000

We released Phi-3 recently, which builds on Phi-2 ( read more on that here ) and it is a great model to use for various tasks. In this post, we will show how to run Phi-3 locally including a demo of it running on a phone.

There should have been a video here but your browser does not seem to support it.

1. What are Small Language Models (SLMs)?

Before diving into running Phi-2 locally, let’s take a moment to understand the concept of small language models (SLMs) and their significance in natural language processing (NLP). A SLM is a type of AI model that has been trained on a massive dataset of text but is limited in terms of its size and capabilities compared to a Large Language Model (LLM). SLMs are designed to be more lightweight and efficient, making them suitable for various applications, including chatbots, language translation, and content generation. SLMs are much smaller than LLMs, with fewer parameters and a smaller dataset, so they have a lower computational cost, making them more suitable for edge or resource-constraint devices.

2. What is Phi-2?

Phi-2 is the latest model in the Phi series of small language models (SLMs) that aim to break the conventional scaling laws of language models. Unlike large language models (LLMs) that require massive amounts of data and compute resources, Phi models are trained on a mixture of web-crawled and synthetic “textbook-quality” data, following the idea of Textbooks Are All You Need . Phi models also leverage innovations in model architecture, optimization, and data augmentation to achieve remarkable performance on various benchmarks. 😀

Phi-2 is twice as large as its predecessor Phi-1.5, and was trained for two weeks on a cluster of 96 A100 GPUs. It demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance (for LLMs <13 billion parameters).

As shown in the figure below on complex benchmarks, Phi-2 matches or outperforms models up to 25x larger, such as Llama-2 and Gemini Nano. Phi-2 is also an ideal playground for researchers, as it allows for exploration around mechanistic interpretability, safety improvements, or fine-tuning experimentation on various tasks. Phi-2 is available in the Azure AI Studio (via the model catalog ) and on Hugging Face.

Phi-2 Evaluation

2.1 Phi-2’s Capabilities

What better way to show this other than Some prompts that Phi-2 can handle:

Creative Writing:
- 🗒️Prompt: “Write a short story about a time-traveling computer geek who discovers a secret algorithm book with the power to change computing history.”
- 🤖 Phi2-Generated output: “In the dimly lit archives of the ancient library, Amelia stumbled upon a dusty tome unlike any other. Its pages whispered secrets of forgotten civilizations and whispered promises of altered destinies. As she traced her fingers over the delicate script, the room seemed to blur, and suddenly, she was standing in the heart of Renaissance Florence. The air smelled of ink and ambition. The book had worked its magic, and Amelia knew she held the fate of empires in her hands.”
Philosophical Reflection:
- 🗒️Prompt: “Discuss the concept of free will versus determinism.”
- 🤖 Phi2-Generated output: “Free will and determinism have long been at odds in philosophical discourse. Some argue that our choices are predetermined by external factors—genetics, upbringing, societal norms—while others champion the idea that we possess true agency. Perhaps the truth lies in the delicate dance between the two. We navigate life’s labyrinth, sometimes guided by fate, other times forging our own path.”
Educational Explanation:
- 🗒️Prompt: “Explain the process of photosynthesis in simple terms.”
- 🤖 Phi2-Generated output: “Photosynthesis, the magical dance of plants and sunlight, is how they create their own food. Imagine tiny chlorophyll factories in leaves, capturing sunlight like solar panels. They convert that energy into chemical fuel—glucose—while releasing oxygen as a byproduct. So next time you breathe in that fresh air, thank a leaf for its photosynthetic prowess!”
Humorous Scenario:
- 🗒️Prompt: “Describe a day in the life of a mischievous squirrel.”
- 🤖 Phi2-Generated output: “Meet Nutty, the squirrel extraordinaire. His mornings begin with acorn heists, followed by elaborate treehouse construction. He’s the neighborhood’s unofficial alarm clock, chattering loudly at dawn. But his pièce de résistance? Prank-calling the crows, convincing them they’ve won a lifetime supply of shiny objects. Nutty’s motto: ‘Life’s too short not to be a little nuts.’”

3. Small Language Models (SLMs) vs Large Language Models (LLMs)

Large Language Models (LLMs) are a type of AI model that is much larger and more powerful than SLMs. LLMs have hundreds of billions of parameters and are trained on massive text datasets. This gives LLMs the ability to handle complex tasks, such as language generation, translation, and question answering, with high accuracy and fluency. However, LLMs also have some disadvantages. They are larger, making them more expensive and slower to train. They also have a higher computational cost, meaning they may require access to specialized hardware.

On the other hand, SLMs, as we called out, are smaller and more lightweight than LLMs, making them more efficient and cost-effective in training computing resources and inference. While it might seem that SLMs are also more suitable for edge or resource-constrained devices, such as mobile phones or IoT devices, they are small compared to LLMs but still require significant computational resources to run. Phi-2, for example, still has 2.7B parameters, and while it can make inferences on a CPU, it is very slow and impractical for real-time applications. One would need a GPU or a cloud-based service for any realistic use case.

3.1 When to use SLM vs LLM?

Firstly, neither model is inherently better - the choice between an SLM and an LLM depends on the specific application and requirements. SLMs are a good choice when size, cost, and speed are important considerations. LLMs are a better choice when high performance and complex capabilities are required. If a task at hand is quite narrow and in one of the supported languages, then SLMs might be good. However, for a given task, an SLM may be sufficient, but an LLM may be necessary for more complex tasks or tasks requiring high accuracy and fluency.

Furthermore, it is key to understand that it is not necessarily about the number of languages understood but rather the depth and nuance with which each model can understand and generate language. SLMs are designed to be efficient and effective within their scope, which may include a wide range of languages. LLMs like GPT-4, due to their size and complexity, often can understand and generate text in a larger number of languages and with greater nuance.

The choice between an SLM and an LLM would again depend on the specific requirements of the task, including the languages involved and the level of language understanding and generation needed. Using a combination of SLMs and LLMs is common to achieve the best results for a given application.

4. Running Phi-2 locally

On one hand, running this is simple if you just don’t want to program anything and only want to use the model. The easiest option in this case is to use [LM Studio , a web-based platform for running language models. You can use the Hugging Face API to download and run the model.

We use a simple console chat example that runs locally. We use the Hugging Face Transformers library to generate text based on user input. The user can generate a story, a haiku, or a joke on a topic of their choice. Here is how to run it locally on a Windows machine - the same should apply to a Mac or Linux machine.

The full code is below, but here are the key aspects to grok when running Phi-2 locally.

The key is to use the AutoModelForCausalLM and AutoTokenizer classes from the transformers library to load the Phi-2 model and tokenizer.
We then use the generate method to generate text based on a user prompt. The generate method takes the user prompt as input and returns the generated text
We use the from_pretrained method to load the model and tokenizer from the Hugging Face model hub.
We also use the save_pretrained method to save the model and tokenizer to a local directory. This allows us to load the model and tokenizer from the local directory if they are already saved, which can help save time and resources.

The following code snippet is what loads the model and the tokenizer:

# Download the model and tokenizer from Hugging Face
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2",
                                            torch_dtype="auto",
                                            trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2",
                                         trust_remote_code=True)

And the following is where we encode the user input and call the generation. First, we create tokens of the user prompt; the resulting tokens are returned as PyTorch tensors. Then, the model generates text based on the tokenized input. We cap the tokens to a maximum of 500 tokens, and the end-of-sequence token is used for padding if necessary. Finally, the generated tokens are decoded back into human-readable text.

inputs = tokenizer(prompt,
                   return_tensors="pt",
                   return_attention_mask=False,
                   add_special_tokens=False)
outputs = model.generate(**inputs,
                         max_length=500,
                         pad_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]

The full code is below. The code is a simple console chat example that runs locally. The user can generate a story, a haiku, or a joke on a topic of their choice.

Some examples of what Phi-2 can generate using the above code are shown below. The first is a story about pandas and dogs.

Story about Pandas 🐼 and Dogs 🐶

Here is another example of a Haiku and a Joke generated by Phi-2 on Pandas.

Haiku and Joke about Pandas 🐼

Switching gears, let’s look at how we can implement the RAG using Phi-2.

4.1 Running Phi-2 Locally - Full Code

The following code is the complete code that executes the examples we showed before for running Phi-2 locally. This can work on a CPU, but it is very slow, and a good GPU is strongly suggested.

import os
import warnings
import logging
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

DEBUG = False

# Suppress warnings and set the logging level to ERROR
warnings.filterwarnings("ignore")
logging.getLogger("transformers").setLevel(logging.ERROR)

# Define the directory where you want to save the model and tokenizer
MODEL_PATH = "./local_model"

# Check if the model and tokenizer are already saved locally
if os.path.exists(MODEL_PATH):
    if DEBUG:
        print(f"Loading model and tokenizer from local directory: {MODEL_PATH}")

    # Load the model and tokenizer from the local directory
    model = AutoModelForCausalLM.from_pretrained(MODEL_PATH)
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
else:
    if DEBUG:
        print(f"Downloading model and tokenizer from Hugging Face")

    # Download the model and tokenizer from Hugging Face
    model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)
    tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)

    if DEBUG:
        print(f"Saving model and tokenizer to local directory: {MODEL_PATH}")

    # Save the model and tokenizer locally
    model.save_pretrained(MODEL_PATH)
    tokenizer.save_pretrained(MODEL_PATH)

if DEBUG:
    print(f"Model device: {model.device}")
    print(f"CUDA available: {torch.cuda.is_available()}")

# Set the default device to CUDA if available, otherwise use CPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

def handle_prompt(user_input, type_of_text)->str:
    prompt = f"Instruct: Write a {type_of_text} about {user_input}\nOutput:"

    with torch.no_grad():
        inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False, add_special_tokens=False)
        inputs = {name: tensor.to(model.device) for name, tensor in inputs.items()}
        outputs = model.generate(**inputs, max_length=500, pad_token_id=tokenizer.eos_token_id)
        
    text = tokenizer.batch_decode(outputs)[0]

    # Remove the prompt from the output text
    text = text.replace(prompt, '').strip()
    text = text.replace("<|endoftext|>", '').strip()
    
    if DEBUG:
        print(f"Answer:{text}")

    return text

if __name__=="__main__": 
    while True:
        print("First What would you like to write today?")
        print("1. Story 📝")
        print("2. Haiku ✍️")
        print("3. Joke 😆")
        print("4. Quit 👋")
        user_choice = input("Choose an option:")

        if user_choice == '4':
            break

        user_prompt = input("And on which topic:")

        if not user_prompt.strip():
            print("Input cannot be empty or consist only of spaces.")
            continue

        if user_choice == '1':
            print(handle_prompt(user_prompt, 'story'))
        elif user_choice == '2':
            print(handle_prompt(user_prompt, 'haiku'))
        elif user_choice == '3':
            print(handle_prompt(user_prompt, 'joke'))
        else:
            print("Invalid choice. Please choose a valid option.")

        print("_" * 100)

5. Implementing Retrieval-Augmented Generation (RAG) with Phi-2

RAG is a powerful technique that combines the strengths of retrieval-based and generation-based approaches to natural language processing. RAG is one of the ways one can get proprietary information and knowledge to the model and use it as part of the prompt. It leverages a retriever to find relevant context passages and a generator to produce fluent and coherent responses. The retriever identifies relevant context passages, and the generator uses these passages to generate a response.

This approach allows RAG to produce high-quality, informative, and contextually relevant responses. In-context learning is a key feature of RAG, as it allows the model to learn from the context of the conversation and generate more accurate and relevant responses. This is particularly useful in scenarios where the model needs to understand and respond to complex queries or provide detailed information on a specific topic.

At a high level, the process of implementing RAG involves the following steps:

Generate Embeddings with Phi-2:
- Use Phi-2 to encode your context passages (documents) and extract their embeddings.
- These embeddings will represent the semantic content of each passage.
Create a Vector Index:
- Choose a vector index library or framework (such as Faiss, Annoy, or HNSW).
- Initialize an index structure to store the embeddings efficiently.
- Add the generated embeddings to the index.
Save Embeddings to a Local Vector Database:
- Create a local database to store the embeddings.
- For each context passage, save its corresponding embedding in the database.
- You can use the passage ID or a unique identifier as the key for retrieval.
Perform Similarity Search:
- When you receive a new context (query), encode it using Phi-2 to obtain its embedding.
- Use the vector index to perform a similarity search against the saved embeddings.
- Retrieve the most similar context passages based on cosine similarity or another distance metric.
- Return the relevant passages as results.

In our example, we will use the FAISS library to create a vector index and perform a similarity search. We will also save the embeddings to a local database for efficient retrieval. FAISS (Facebook AI Similarity Search) is a library developed by Facebook for efficient similarity search and clustering of high-dimensional vectors. It allows for a quick nearest-neighbor search over large datasets and supports CPU and GPU-based computations. FAISS is widely used in information retrieval, recommendation systems, and other applications that require similarity search.

5.1 Loading data for RAG and Phi-2

To implement RAG, we use the script from the Oppenheimer movie - which is quite new in that it is not in the Phi-2 training set and is available as a PDF. We will extract the script from this PDF, creating embeddings, which will then save the embeddings to a local database and perform a similarity search to retrieve relevant context passages based on a user query. We will use the FAISS library to create a vector index and perform a similarity search. We will also save the embeddings to a local database for efficient retrieval.

We use the PyPDF2 library to parse PDFs, a pure Python library for reading and writing PDF files. It can extract text, merge and split documents, and more. We will use it to extract the PDF text from the Oppenheimer movie script. The following code function shows how to read the PDF and extract the text. This is efficient for our use case, but it is not the most efficient way to extract text from a PDF when thinking about production scale, especially if the PDF has a lot of images and tables.

def read_pdf(file_path):
    pdf_file_obj = open(file_path, 'rb')
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

    num_pages = pdf_reader.numPages
    for page_num in range(num_pages):
        page_obj = pdf_reader.getPage(page_num)
        text = page_obj.extractText()
        yield text
    pdf_file_obj.close()

5.2 Generate embeddings using Phi-2

Now that we have the text, the following functions show how to create the embeddings using Phi-2. We read the text as a list of context passages and then use Phi-2 to encode each passage and extract its embedding using the encode method.

def create_embeddings(file_path, tokenizer, model, device):
    if file_path.endswith('.pdf'):
        context_passages = list(read_pdf(file_path))
    else:
        with open(file_path, 'r', encoding='utf-8') as file:
            context_passages = file.readlines()

    embeddings = []
    for passage in tqdm(context_passages):
        if not passage.strip():
            # Skip the passage
            pass
        input_ids = tokenizer.encode(passage, 
                                    return_tensors="pt",
                                    add_special_tokens=False,
                                    return_attention_mask=False).to(device)
        with torch.no_grad():
            output = model(input_ids)
            logits = output.logits
            embedding = logits.mean(dim=1).detach().cpu().numpy()
            embeddings.append(embedding)

    return embeddings, context_passages

Here are a few things that are going on:

Given that we are using this for inference and not training, we use a torch.no_grad()`, which tells PyTorch not to track, calculate, or modify gradients while executing code within this block. This helps us save the amount of memory needed.
Inside this block, the input_ids are fed into the model, and the output is stored in the output variable. The logits, which are the raw, unnormalized scores outputted by the last layer of the model, are then extracted from the model’s output.
The logits are then processed to generate the embedding for the passage. The .mean(dim=1) method calculates the mean of the logits along dimension 1, which typically represents the sequence length in a language model.
The .detach() method detaches the result from the computation graph so that no gradients will be backpropagated along this variable.
The .cpu() method moves the tensor to the CPU if it’s not already there. Finally, the tensor is converted to a numpy array using the .numpy() method.
The resulting embedding is then appended to the embeddings list, which contains the embeddings for all the passages.

5.3 Creating Vector Index

The following function shows how to create a vector index using the FAISS library and perform a similarity search to retrieve relevant context passages based on a user query. The create_index function initializes a flat index structure to store the embeddings and adds the embeddings to the index. The search_query function encodes the user query using Phi-2 to obtain its embedding and performs a similarity search against the saved embeddings to retrieve the most similar context passages.

def create_index(query_embedding):
    index = faiss.IndexFlatL2(query_embedding[0].shape[1])  # Euclidean distance
    faiss.normalize_L2(query_embedding)
    
    # Add embeddings to the index
    for i, item in tqdm(enumerate(query_embedding), total=len(query_embedding)):
        if item.ndim == 1:
            item = item.reshape(1, -1)  # Reshape 1D array to 2D
        index.add(item)
    return index

The normalize_L2() function normalizes the vectors and is a crucial step when using Euclidean distance in high-dimensional spaces to ensure that the distance is not dominated by the dimensionality of the vectors. As we iterate through the embeddings, we check if the item is a 1D array and reshape it to a 2D array if necessary. This is important because FAISS expects the input to be a 2D array, and we need to reshape the 1D array to a 2D array before adding it to the index.

The function finally returns the created index. This index can then be used to perform efficient similarity searches.

5.4 Perform Similarity Search

The following function shows how to perform a similarity search using the vector index to retrieve relevant context passages based on a user query. As noted earlier, the most similar context passages are then retrieved based on cosine similarity or another distance metric.

The function starts by encoding the input query using a tokenizer and performs a similarity search on the FAISS index using the query embedding. It retrieves the indices of the top 3 most similar passages to the input query and then retrieves the corresponding context passages from the context_passages list. The similar context passages are then concatenated into a single string and passed to the Phi-2 model to generate a response.

def search_query(input_query, inputTokenizer, model, device, index, context_passages):
    # Given a new query context, encode it and perform similarity search
    input_ids = inputTokenizer.encode(input_query,
                                    return_tensors="pt",
                                    return_attention_mask=False,
                                    add_special_tokens=False).to(device)
    with torch.no_grad():
        input_ids = input_ids.long()
        output = model(input_ids)
        logits = output.logits
        query_embedding = logits.mean(dim=1).detach().cpu().numpy()

    # Perform similarity search - top 3 similar passages
    _, similar_indices = index.search(query_embedding, k=3)
    
    # Retrieve context passages based on similar_indices
    similar_contexts = [context_passages[i] for i in similar_indices[0]]

    # Concatenate the similar contexts into a single string
    context = ' '.join(similar_contexts)

Let us run this and see how it works, as we discussed before. We will use the Oppenheimer movie script as the context passages and perform a similarity search to retrieve relevant context passages based on a user query. The next few figures show the output of us asking questions about the movie, where those pieces of information are not in the model but passed using the semantic search.

Example 1 - Phi-2 and RAG implementation

Example 2 - Phi-2 and RAG implementation

Example 3 - Phi-2 and RAG implementation

Now that we have seen the different elements, the code below brings everything together as a console app that one can run. The Oppenheimer script (pdf file) you need can be downloaded from here .

import os
import warnings
import logging
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import numpy as np
import faiss 
from tqdm import tqdm
import pickle
import re
import PyPDF2

DEBUG = False

warnings.filterwarnings("ignore")
logging.getLogger("transformers").setLevel(logging.ERROR)

# Define the directory where you want to save the model and tokenizer
MODEL_PATH = "./local_model"
MODEL_NAME = "microsoft/phi-2"
BATCH_SIZE = 1000

# Oppenheimer movie
#DATA_FILE = "./oppenheimer-2023.txt"
DATA_FILE = "./oppenheimer-2023.pdf"
EMBEDDINGS_FILE = './embeddings_movie.pkl'

def load_model(model_path, model_name, debug=False):
    # Check if the model and tokenizer are already saved locally
    if os.path.exists(model_path):
        if debug:
            print(f"Loading model and tokenizer from local directory: {model_path}")

        # Load the model and tokenizer from the local directory
        model = AutoModelForCausalLM.from_pretrained(model_path)
        tokenizer = AutoTokenizer.from_pretrained(model_path)
    else:
        if debug:
            print("Downloading model and tokenizer from Hugging Face")

        # Download the model and tokenizer from Hugging Face
        model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)
        tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

        if debug:
            print(f"Saving model and tokenizer to local directory: {model_path}")

        # Save the model and tokenizer locally
        model.save_pretrained(model_path)
        tokenizer.save_pretrained(model_path)

    if debug:
        print(f"Model device: {model.device}")
        print(f"CUDA available: {torch.cuda.is_available()}")

    # Set the default device to CUDA if available, otherwise use CPU
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)

    return model, tokenizer, device

def save_embeddings(embeddings, passages, file_name):
    try:
        with open(file_name, 'wb') as f:
            pickle.dump((list(embeddings), list(passages)), f)
    except IOError:
        print(f"Error writing to file {file_name}.")
    except pickle.PicklingError:
        print("Error pickling embeddings and passages.")
    
def load_embeddings(file):
    try:
        with open(file, 'rb') as f:
            return pickle.load(f)
    except FileNotFoundError:
        print(f"File {file} not found.")
        return None
    except pickle.UnpicklingError:
        print(f"Error unpickling file {file}.")
        return None

def read_pdf(file_path):
    pdf_file_obj = open(file_path, 'rb')
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

    num_pages = pdf_reader.numPages
    for page_num in range(num_pages):
        page_obj = pdf_reader.getPage(page_num)
        text = page_obj.extractText()
        yield text

    if DEBUG:
        print(f"Finished reading file. Number pages: {num_pages}")
    
    pdf_file_obj.close()

def create_embeddings(file_path, tokenizer, model, device):
    if os.path.exists(EMBEDDINGS_FILE):
        # Load the embeddings and passages from disk
        embeddings, context_passages = load_embeddings(EMBEDDINGS_FILE)
    else:
        if file_path.endswith('.pdf'):
            context_passages = list(read_pdf(file_path))
        else:
            with open(file_path, 'r', encoding='utf-8') as file:
                context_passages = file.readlines()

        embeddings = []
        for passage in tqdm(context_passages):
            if not passage.strip():
                # Skip the passage
                pass
            input_ids = tokenizer.encode(passage, return_tensors="pt", add_special_tokens=False, return_attention_mask=False).to(device)
            with torch.no_grad():
                output = model(input_ids)
                logits = output.logits
                embedding = logits.mean(dim=1).detach().cpu().numpy()
                embeddings.append(embedding)

        # Save the embeddings and passages to disk
        save_embeddings(embeddings, context_passages, EMBEDDINGS_FILE)

    return embeddings, context_passages
  
def handle_prompt(user_input, context)->str:
    prompt = f"Instruct: You are a helpful bot who only answers using the given context ONLY. If you cannot find the answer in the context reply 'Sorry don't have that detail'. Given the context '{context}', answer this:{user_input}\nOutput:"

    with torch.no_grad():
        inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False, add_special_tokens=False)
        inputs = {name: tensor.to(model.device) for name, tensor in inputs.items()}
        outputs = model.generate(**inputs, max_length=2000, pad_token_id=tokenizer.eos_token_id)
        
    text = tokenizer.batch_decode(outputs)[0]

    # Remove the prompt from the output text
    text = text.replace(prompt, '').strip()
    text = text.replace("<|endoftext|>", '').strip()
    return text

def create_index(query_embedding):
    index = faiss.IndexFlatL2(query_embedding[0].shape[1])  # Euclidean distance
    faiss.normalize_L2(query_embedding)
    
    # Add embeddings to the index
    for i, item in tqdm(enumerate(query_embedding), total=len(query_embedding)):
        if item.ndim == 1:
            item = item.reshape(1, -1)  # Reshape 1D array to 2D
        index.add(item)
    return index

def search_query(input_query, inputTokenizer, model, device, index, context_passages):
    # Given a new query context, encode it and perform similarity search
    input_ids = inputTokenizer.encode(input_query, return_tensors="pt", return_attention_mask=False, add_special_tokens=False).to(device)
    with torch.no_grad():
        input_ids = input_ids.long()
        output = model(input_ids)
        logits = output.logits
        query_embedding = logits.mean(dim=1).detach().cpu().numpy()

    # Perform similarity search - top 3 similar passages
    _, similar_indices = index.search(query_embedding, k=3)
    if DEBUG:
        print(f"DEBUG - Number of similar indices: {similar_indices.size}")    

    # Retrieve context passages based on similar_indices
    similar_contexts = [context_passages[i] for i in similar_indices[0]]

    # Concatenate the similar contexts into a single string
    context = ' '.join(similar_contexts)
    
    # Pass the concatenated context and query to the Phi-2 model
    answer = handle_prompt(input_query, context)
    print("Answer:", answer)

if __name__=="__main__":
    model, tokenizer, device = load_model(MODEL_PATH, MODEL_NAME, DEBUG)

     # Create embeddings and add to index before entering the loop
    query_embedding, context_passages = create_embeddings(DATA_FILE, tokenizer, model, device)
    query_embedding = np.array(query_embedding)
    index = create_index(query_embedding)
    
    exit_commands = ['exit', 'quit', 'q', 'e']
    while True:
        query = input("Enter your query or 'exit' to quit: ")
        if query in exit_commands:
            break
        search_query(query, tokenizer, model, device, index, context_passages)
        print("-"*50)

Now let us switch gears and try something that pushes the ability of Phi-2.

6. Code generation example using Phi-2

If we want to push the boundaries of what Phi-2 can do, we can use it to generate code. Below is an example of using Phi-2 to generate code for a simple C function. 🤓

🗒️Prompt: “Write a program in C that implements a BPE-based tokenizer; it should implement both encoding and decoding functions. Think through this step by step.”

The code we see below is what was generated. At face value, it looks like a good start but is incomplete. It is a good starting point for a developer to continue from and shows the power of SLMs like Phi-2.

#include<stdio.h> 
#include<string.h>
#define MAX_LEN 100

// Encodes the input text into an array of integers
int* encode(char *text, int *tokenizedStringsSize){

  // Allocate memory for the tokens list
  char *str = (char*)malloc(sizeof(char)*MAX_LEN+1);
  int i, j=0, k=0;

  for (i=0; i<strlen(text) && text[i] != '\0'; ++i){
    // Check if the current character is whitespace or not alphanumeric
    if(isspace(text[i]) || !isalpha(text[i])){
      // Add whitespace to the tokens list
      strcat(str,&text[i]); 
    } else {
      while (k<j && strcmp(text+i, str + k)==0){
        ++k; 
      }
      // Check if the token is of length atleast 1
      if(k-j >= 1)
        // If yes, concatenate it to tokens list
        strcat(str,&text[i]);
    }
  }

  // Output the tokenized string
  printf("Tokenized String: %s\n",str);

  // Allocate memory for tokenized string size array
  *tokenizedStringsSize = (int)malloc(sizeof(int)*(j-1));

  for (i=0; i<j; ++i){
    // Convert tokens to integer values
    (*tokenizedStringsSize)[i] = strtol(&str[i], NULL, 16);
  }

  return *tokenizedStringsSize;
}

// Decodes the input array of integers into a string
char* decode(int *tokensize, int *tokenizedStringsSize){

  // Declare memory for decoded string
  char str[MAX_LEN] = "";

  for (i=0; i<*tokensize; ++i) {
    strcat(str, "%x", tokensize[i]);
  }

  // Output the tokenized string
  printf("Tokenized String: %s\n",str);

  // Declare memory for decoded string
  char *decodedStr = (char*)malloc(sizeof(char)*(*tokenizedStringsSize)+1);
  for (i=0; i<*tokensize; ++i) {
    // Convert integer values to hexadecimal strings
    sprintf(&decodedStr[i], "%x", tokensize[i]);
  }

  return decodedStr;
}

⚠️ Note: It has been a while since I wrote C, but at a high level, these are some of the issues I can see with this; these issues are off the top of my head and are not exhaustive. Finally, it is not meant to test my coding capabilities. 😬

Memory Allocation: The malloc function is used without checking for successful allocation; if it returns NULL, which is not checked, we will get hurt.
Tokenization Logic: The logic in the encode function does not reflect the BPE algorithm, which involves merging the most frequent pairs of characters or bytes.
String Concatenation: The strcat function is used incorrectly; instead of a null-terminated string (as part of the second argument), we get a pointer to a single character
Decoding Logic: The decode function attempts to use strcat with a format string ("%x"), which is invalid. The sprintf function should be used for formatted strings.

Hopefully, this gives you a good understanding of SLMs, specifically Phi-2, and how to use them locally. 😍

SLMs - How to run Phi-2 Locally, and implement RAG

Wed, 13 Mar 2024 00:00:00 +0000

1. What are Small Language Models (SLMs)?

2. What is Phi-2?

Phi-2 Evaluation

2.1 Phi-2’s Capabilities

What better way to show this other than Some prompts that Phi-2 can handle:

Creative Writing:
- 🗒️Prompt: “Write a short story about a time-traveling computer geek who discovers a secret algorithm book with the power to change computing history.”
- 🤖 Phi2-Generated output: “In the dimly lit archives of the ancient library, Amelia stumbled upon a dusty tome unlike any other. Its pages whispered secrets of forgotten civilizations and whispered promises of altered destinies. As she traced her fingers over the delicate script, the room seemed to blur, and suddenly, she was standing in the heart of Renaissance Florence. The air smelled of ink and ambition. The book had worked its magic, and Amelia knew she held the fate of empires in her hands.”
Philosophical Reflection:
- 🗒️Prompt: “Discuss the concept of free will versus determinism.”
- 🤖 Phi2-Generated output: “Free will and determinism have long been at odds in philosophical discourse. Some argue that our choices are predetermined by external factors—genetics, upbringing, societal norms—while others champion the idea that we possess true agency. Perhaps the truth lies in the delicate dance between the two. We navigate life’s labyrinth, sometimes guided by fate, other times forging our own path.”
Educational Explanation:
- 🗒️Prompt: “Explain the process of photosynthesis in simple terms.”
- 🤖 Phi2-Generated output: “Photosynthesis, the magical dance of plants and sunlight, is how they create their own food. Imagine tiny chlorophyll factories in leaves, capturing sunlight like solar panels. They convert that energy into chemical fuel—glucose—while releasing oxygen as a byproduct. So next time you breathe in that fresh air, thank a leaf for its photosynthetic prowess!”
Humorous Scenario:
- 🗒️Prompt: “Describe a day in the life of a mischievous squirrel.”
- 🤖 Phi2-Generated output: “Meet Nutty, the squirrel extraordinaire. His mornings begin with acorn heists, followed by elaborate treehouse construction. He’s the neighborhood’s unofficial alarm clock, chattering loudly at dawn. But his pièce de résistance? Prank-calling the crows, convincing them they’ve won a lifetime supply of shiny objects. Nutty’s motto: ‘Life’s too short not to be a little nuts.’”

3. Small Language Models (SLMs) vs Large Language Models (LLMs)

3.1 When to use SLM vs LLM?

4. Running Phi-2 locally

The full code is below, but here are the key aspects to grok when running Phi-2 locally.

The key is to use the AutoModelForCausalLM and AutoTokenizer classes from the transformers library to load the Phi-2 model and tokenizer.
We then use the generate method to generate text based on a user prompt. The generate method takes the user prompt as input and returns the generated text
We use the from_pretrained method to load the model and tokenizer from the Hugging Face model hub.
We also use the save_pretrained method to save the model and tokenizer to a local directory. This allows us to load the model and tokenizer from the local directory if they are already saved, which can help save time and resources.

The following code snippet is what loads the model and the tokenizer:

# Download the model and tokenizer from Hugging Face
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2",
                                            torch_dtype="auto",
                                            trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2",
                                         trust_remote_code=True)

inputs = tokenizer(prompt,
                   return_tensors="pt",
                   return_attention_mask=False,
                   add_special_tokens=False)
outputs = model.generate(**inputs,
                         max_length=500,
                         pad_token_id=tokenizer.eos_token_id)
text = tokenizer.batch_decode(outputs)[0]

The full code is below. The code is a simple console chat example that runs locally. The user can generate a story, a haiku, or a joke on a topic of their choice.

Some examples of what Phi-2 can generate using the above code are shown below. The first is a story about pandas and dogs.

Story about Pandas 🐼 and Dogs 🐶

Here is another example of a Haiku and a Joke generated by Phi-2 on Pandas.

Haiku and Joke about Pandas 🐼

Switching gears, let’s look at how we can implement the RAG using Phi-2.

4.1 Running Phi-2 Locally - Full Code

The following code is the complete code that executes the examples we showed before for running Phi-2 locally. This can work on a CPU, but it is very slow, and a good GPU is strongly suggested.

import os
import warnings
import logging
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

DEBUG = False

# Suppress warnings and set the logging level to ERROR
warnings.filterwarnings("ignore")
logging.getLogger("transformers").setLevel(logging.ERROR)

# Define the directory where you want to save the model and tokenizer
MODEL_PATH = "./local_model"

# Check if the model and tokenizer are already saved locally
if os.path.exists(MODEL_PATH):
    if DEBUG:
        print(f"Loading model and tokenizer from local directory: {MODEL_PATH}")

    # Load the model and tokenizer from the local directory
    model = AutoModelForCausalLM.from_pretrained(MODEL_PATH)
    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
else:
    if DEBUG:
        print(f"Downloading model and tokenizer from Hugging Face")

    # Download the model and tokenizer from Hugging Face
    model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)
    tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)

    if DEBUG:
        print(f"Saving model and tokenizer to local directory: {MODEL_PATH}")

    # Save the model and tokenizer locally
    model.save_pretrained(MODEL_PATH)
    tokenizer.save_pretrained(MODEL_PATH)

if DEBUG:
    print(f"Model device: {model.device}")
    print(f"CUDA available: {torch.cuda.is_available()}")

# Set the default device to CUDA if available, otherwise use CPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

def handle_prompt(user_input, type_of_text)->str:
    prompt = f"Instruct: Write a {type_of_text} about {user_input}\nOutput:"

    with torch.no_grad():
        inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False, add_special_tokens=False)
        inputs = {name: tensor.to(model.device) for name, tensor in inputs.items()}
        outputs = model.generate(**inputs, max_length=500, pad_token_id=tokenizer.eos_token_id)
        
    text = tokenizer.batch_decode(outputs)[0]

    # Remove the prompt from the output text
    text = text.replace(prompt, '').strip()
    text = text.replace("<|endoftext|>", '').strip()
    
    if DEBUG:
        print(f"Answer:{text}")

    return text

if __name__=="__main__": 
    while True:
        print("First What would you like to write today?")
        print("1. Story 📝")
        print("2. Haiku ✍️")
        print("3. Joke 😆")
        print("4. Quit 👋")
        user_choice = input("Choose an option:")

        if user_choice == '4':
            break

        user_prompt = input("And on which topic:")

        if not user_prompt.strip():
            print("Input cannot be empty or consist only of spaces.")
            continue

        if user_choice == '1':
            print(handle_prompt(user_prompt, 'story'))
        elif user_choice == '2':
            print(handle_prompt(user_prompt, 'haiku'))
        elif user_choice == '3':
            print(handle_prompt(user_prompt, 'joke'))
        else:
            print("Invalid choice. Please choose a valid option.")

        print("_" * 100)

5. Implementing Retrieval-Augmented Generation (RAG) with Phi-2

At a high level, the process of implementing RAG involves the following steps:

Generate Embeddings with Phi-2:
- Use Phi-2 to encode your context passages (documents) and extract their embeddings.
- These embeddings will represent the semantic content of each passage.
Create a Vector Index:
- Choose a vector index library or framework (such as Faiss, Annoy, or HNSW).
- Initialize an index structure to store the embeddings efficiently.
- Add the generated embeddings to the index.
Save Embeddings to a Local Vector Database:
- Create a local database to store the embeddings.
- For each context passage, save its corresponding embedding in the database.
- You can use the passage ID or a unique identifier as the key for retrieval.
Perform Similarity Search:
- When you receive a new context (query), encode it using Phi-2 to obtain its embedding.
- Use the vector index to perform a similarity search against the saved embeddings.
- Retrieve the most similar context passages based on cosine similarity or another distance metric.
- Return the relevant passages as results.

5.1 Loading data for RAG and Phi-2

def read_pdf(file_path):
    pdf_file_obj = open(file_path, 'rb')
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

    num_pages = pdf_reader.numPages
    for page_num in range(num_pages):
        page_obj = pdf_reader.getPage(page_num)
        text = page_obj.extractText()
        yield text
    pdf_file_obj.close()

5.2 Generate embeddings using Phi-2

def create_embeddings(file_path, tokenizer, model, device):
    if file_path.endswith('.pdf'):
        context_passages = list(read_pdf(file_path))
    else:
        with open(file_path, 'r', encoding='utf-8') as file:
            context_passages = file.readlines()

    embeddings = []
    for passage in tqdm(context_passages):
        if not passage.strip():
            # Skip the passage
            pass
        input_ids = tokenizer.encode(passage, 
                                    return_tensors="pt",
                                    add_special_tokens=False,
                                    return_attention_mask=False).to(device)
        with torch.no_grad():
            output = model(input_ids)
            logits = output.logits
            embedding = logits.mean(dim=1).detach().cpu().numpy()
            embeddings.append(embedding)

    return embeddings, context_passages

Here are a few things that are going on:

Given that we are using this for inference and not training, we use a torch.no_grad()`, which tells PyTorch not to track, calculate, or modify gradients while executing code within this block. This helps us save the amount of memory needed.
Inside this block, the input_ids are fed into the model, and the output is stored in the output variable. The logits, which are the raw, unnormalized scores outputted by the last layer of the model, are then extracted from the model’s output.
The logits are then processed to generate the embedding for the passage. The .mean(dim=1) method calculates the mean of the logits along dimension 1, which typically represents the sequence length in a language model.
The .detach() method detaches the result from the computation graph so that no gradients will be backpropagated along this variable.
The .cpu() method moves the tensor to the CPU if it’s not already there. Finally, the tensor is converted to a numpy array using the .numpy() method.
The resulting embedding is then appended to the embeddings list, which contains the embeddings for all the passages.

5.3 Creating Vector Index

def create_index(query_embedding):
    index = faiss.IndexFlatL2(query_embedding[0].shape[1])  # Euclidean distance
    faiss.normalize_L2(query_embedding)
    
    # Add embeddings to the index
    for i, item in tqdm(enumerate(query_embedding), total=len(query_embedding)):
        if item.ndim == 1:
            item = item.reshape(1, -1)  # Reshape 1D array to 2D
        index.add(item)
    return index

The function finally returns the created index. This index can then be used to perform efficient similarity searches.

5.4 Perform Similarity Search

def search_query(input_query, inputTokenizer, model, device, index, context_passages):
    # Given a new query context, encode it and perform similarity search
    input_ids = inputTokenizer.encode(input_query,
                                    return_tensors="pt",
                                    return_attention_mask=False,
                                    add_special_tokens=False).to(device)
    with torch.no_grad():
        input_ids = input_ids.long()
        output = model(input_ids)
        logits = output.logits
        query_embedding = logits.mean(dim=1).detach().cpu().numpy()

    # Perform similarity search - top 3 similar passages
    _, similar_indices = index.search(query_embedding, k=3)
    
    # Retrieve context passages based on similar_indices
    similar_contexts = [context_passages[i] for i in similar_indices[0]]

    # Concatenate the similar contexts into a single string
    context = ' '.join(similar_contexts)

Example 1 - Phi-2 and RAG implementation

Example 2 - Phi-2 and RAG implementation

Example 3 - Phi-2 and RAG implementation

Now that we have seen the different elements, the code below brings everything together as a console app that one can run. The Oppenheimer script (pdf file) you need can be downloaded from here .

import os
import warnings
import logging
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import numpy as np
import faiss 
from tqdm import tqdm
import pickle
import re
import PyPDF2

DEBUG = False

warnings.filterwarnings("ignore")
logging.getLogger("transformers").setLevel(logging.ERROR)

# Define the directory where you want to save the model and tokenizer
MODEL_PATH = "./local_model"
MODEL_NAME = "microsoft/phi-2"
BATCH_SIZE = 1000

# Oppenheimer movie
#DATA_FILE = "./oppenheimer-2023.txt"
DATA_FILE = "./oppenheimer-2023.pdf"
EMBEDDINGS_FILE = './embeddings_movie.pkl'

def load_model(model_path, model_name, debug=False):
    # Check if the model and tokenizer are already saved locally
    if os.path.exists(model_path):
        if debug:
            print(f"Loading model and tokenizer from local directory: {model_path}")

        # Load the model and tokenizer from the local directory
        model = AutoModelForCausalLM.from_pretrained(model_path)
        tokenizer = AutoTokenizer.from_pretrained(model_path)
    else:
        if debug:
            print("Downloading model and tokenizer from Hugging Face")

        # Download the model and tokenizer from Hugging Face
        model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)
        tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

        if debug:
            print(f"Saving model and tokenizer to local directory: {model_path}")

        # Save the model and tokenizer locally
        model.save_pretrained(model_path)
        tokenizer.save_pretrained(model_path)

    if debug:
        print(f"Model device: {model.device}")
        print(f"CUDA available: {torch.cuda.is_available()}")

    # Set the default device to CUDA if available, otherwise use CPU
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)

    return model, tokenizer, device

def save_embeddings(embeddings, passages, file_name):
    try:
        with open(file_name, 'wb') as f:
            pickle.dump((list(embeddings), list(passages)), f)
    except IOError:
        print(f"Error writing to file {file_name}.")
    except pickle.PicklingError:
        print("Error pickling embeddings and passages.")
    
def load_embeddings(file):
    try:
        with open(file, 'rb') as f:
            return pickle.load(f)
    except FileNotFoundError:
        print(f"File {file} not found.")
        return None
    except pickle.UnpicklingError:
        print(f"Error unpickling file {file}.")
        return None

def read_pdf(file_path):
    pdf_file_obj = open(file_path, 'rb')
    pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

    num_pages = pdf_reader.numPages
    for page_num in range(num_pages):
        page_obj = pdf_reader.getPage(page_num)
        text = page_obj.extractText()
        yield text

    if DEBUG:
        print(f"Finished reading file. Number pages: {num_pages}")
    
    pdf_file_obj.close()

def create_embeddings(file_path, tokenizer, model, device):
    if os.path.exists(EMBEDDINGS_FILE):
        # Load the embeddings and passages from disk
        embeddings, context_passages = load_embeddings(EMBEDDINGS_FILE)
    else:
        if file_path.endswith('.pdf'):
            context_passages = list(read_pdf(file_path))
        else:
            with open(file_path, 'r', encoding='utf-8') as file:
                context_passages = file.readlines()

        embeddings = []
        for passage in tqdm(context_passages):
            if not passage.strip():
                # Skip the passage
                pass
            input_ids = tokenizer.encode(passage, return_tensors="pt", add_special_tokens=False, return_attention_mask=False).to(device)
            with torch.no_grad():
                output = model(input_ids)
                logits = output.logits
                embedding = logits.mean(dim=1).detach().cpu().numpy()
                embeddings.append(embedding)

        # Save the embeddings and passages to disk
        save_embeddings(embeddings, context_passages, EMBEDDINGS_FILE)

    return embeddings, context_passages
  
def handle_prompt(user_input, context)->str:
    prompt = f"Instruct: You are a helpful bot who only answers using the given context ONLY. If you cannot find the answer in the context reply 'Sorry don't have that detail'. Given the context '{context}', answer this:{user_input}\nOutput:"

    with torch.no_grad():
        inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=False, add_special_tokens=False)
        inputs = {name: tensor.to(model.device) for name, tensor in inputs.items()}
        outputs = model.generate(**inputs, max_length=2000, pad_token_id=tokenizer.eos_token_id)
        
    text = tokenizer.batch_decode(outputs)[0]

    # Remove the prompt from the output text
    text = text.replace(prompt, '').strip()
    text = text.replace("<|endoftext|>", '').strip()
    return text

def create_index(query_embedding):
    index = faiss.IndexFlatL2(query_embedding[0].shape[1])  # Euclidean distance
    faiss.normalize_L2(query_embedding)
    
    # Add embeddings to the index
    for i, item in tqdm(enumerate(query_embedding), total=len(query_embedding)):
        if item.ndim == 1:
            item = item.reshape(1, -1)  # Reshape 1D array to 2D
        index.add(item)
    return index

def search_query(input_query, inputTokenizer, model, device, index, context_passages):
    # Given a new query context, encode it and perform similarity search
    input_ids = inputTokenizer.encode(input_query, return_tensors="pt", return_attention_mask=False, add_special_tokens=False).to(device)
    with torch.no_grad():
        input_ids = input_ids.long()
        output = model(input_ids)
        logits = output.logits
        query_embedding = logits.mean(dim=1).detach().cpu().numpy()

    # Perform similarity search - top 3 similar passages
    _, similar_indices = index.search(query_embedding, k=3)
    if DEBUG:
        print(f"DEBUG - Number of similar indices: {similar_indices.size}")    

    # Retrieve context passages based on similar_indices
    similar_contexts = [context_passages[i] for i in similar_indices[0]]

    # Concatenate the similar contexts into a single string
    context = ' '.join(similar_contexts)
    
    # Pass the concatenated context and query to the Phi-2 model
    answer = handle_prompt(input_query, context)
    print("Answer:", answer)

if __name__=="__main__":
    model, tokenizer, device = load_model(MODEL_PATH, MODEL_NAME, DEBUG)

     # Create embeddings and add to index before entering the loop
    query_embedding, context_passages = create_embeddings(DATA_FILE, tokenizer, model, device)
    query_embedding = np.array(query_embedding)
    index = create_index(query_embedding)
    
    exit_commands = ['exit', 'quit', 'q', 'e']
    while True:
        query = input("Enter your query or 'exit' to quit: ")
        if query in exit_commands:
            break
        search_query(query, tokenizer, model, device, index, context_passages)
        print("-"*50)

Now let us switch gears and try something that pushes the ability of Phi-2.

6. Code generation example using Phi-2

If we want to push the boundaries of what Phi-2 can do, we can use it to generate code. Below is an example of using Phi-2 to generate code for a simple C function. 🤓

🗒️Prompt: “Write a program in C that implements a BPE-based tokenizer; it should implement both encoding and decoding functions. Think through this step by step.”

#include<stdio.h> 
#include<string.h>
#define MAX_LEN 100

// Encodes the input text into an array of integers
int* encode(char *text, int *tokenizedStringsSize){

  // Allocate memory for the tokens list
  char *str = (char*)malloc(sizeof(char)*MAX_LEN+1);
  int i, j=0, k=0;

  for (i=0; i<strlen(text) && text[i] != '\0'; ++i){
    // Check if the current character is whitespace or not alphanumeric
    if(isspace(text[i]) || !isalpha(text[i])){
      // Add whitespace to the tokens list
      strcat(str,&text[i]); 
    } else {
      while (k<j && strcmp(text+i, str + k)==0){
        ++k; 
      }
      // Check if the token is of length atleast 1
      if(k-j >= 1)
        // If yes, concatenate it to tokens list
        strcat(str,&text[i]);
    }
  }

  // Output the tokenized string
  printf("Tokenized String: %s\n",str);

  // Allocate memory for tokenized string size array
  *tokenizedStringsSize = (int)malloc(sizeof(int)*(j-1));

  for (i=0; i<j; ++i){
    // Convert tokens to integer values
    (*tokenizedStringsSize)[i] = strtol(&str[i], NULL, 16);
  }

  return *tokenizedStringsSize;
}

// Decodes the input array of integers into a string
char* decode(int *tokensize, int *tokenizedStringsSize){

  // Declare memory for decoded string
  char str[MAX_LEN] = "";

  for (i=0; i<*tokensize; ++i) {
    strcat(str, "%x", tokensize[i]);
  }

  // Output the tokenized string
  printf("Tokenized String: %s\n",str);

  // Declare memory for decoded string
  char *decodedStr = (char*)malloc(sizeof(char)*(*tokenizedStringsSize)+1);
  for (i=0; i<*tokensize; ++i) {
    // Convert integer values to hexadecimal strings
    sprintf(&decodedStr[i], "%x", tokensize[i]);
  }

  return decodedStr;
}

Memory Allocation: The malloc function is used without checking for successful allocation; if it returns NULL, which is not checked, we will get hurt.
Tokenization Logic: The logic in the encode function does not reflect the BPE algorithm, which involves merging the most frequent pairs of characters or bytes.
String Concatenation: The strcat function is used incorrectly; instead of a null-terminated string (as part of the second argument), we get a pointer to a single character
Decoding Logic: The decode function attempts to use strcat with a format string ("%x"), which is invalid. The sprintf function should be used for formatted strings.

Hopefully, this gives you a good understanding of SLMs, specifically Phi-2, and how to use them locally. 😍

Shedding Light on the Art of Prompt Engineering

Wed, 28 Feb 2024 00:00:00 +0000

How many prompt engineers does it take to change a light bulb? Just one, but first, they need to fine-tune the model to make sure the AI doesn’t end up writing a poem about darkness instead.

DALLE generated image of How many engineers it take to change a light bulb

📚 My new book "Generative AI in Action"

Tue, 14 Nov 2023 00:00:00 +0000

🌐 As software continues to revolutionize the world, the advent of Generative AI is transforming the very fabric of software itself. My latest book, Generative AI in Action delves into this transformative journey.

I am thrilled to announce the early release of my latest book, Generative AI in Action now available through Manning Early Access Program (MEAP) . This publication is a deep dive into the cutting-edge world of #GenerativeAI, #LLMs, #OpenAI, and #Azure #OpenAI, tailored specifically for enterprises. 🤘

This practical, in-action, hands-on book allows one to explore the cutting-edge world of Generative AI, including LLMs, and covers both OpenAI and Azure OpenAI, allowing companies to understand basic concepts and scale to production.📘

I have the privilege of having a front-row seat as we build this tech out and work with key Fortune 500 customers who are incorporating this. A lot of this is learnings captured from this. 😇

📖 Inside the Book

A comprehensive introduction to Generative AI, including foundational models like GPT, Codex, DALLE, and ChatGPT.
Insightful discussions on Large Language Models (LLMs) and their applications in various sectors.
Practical guides on generating text through APIs, focusing on OpenAI and Azure OpenAI.
Exploration of image generation techniques, including Stable Diffusion and DALLE.
Deep dive into Prompt Engineering, RAG, Bring your own Data, and model adaptation techniques.
And finally, best practices to allow enterprises to scale to production.

🎯 Why This Book is a Must-Read? It is specially crafted for businesses leveraging AI for innovation and competitive advantage. Combining technical depth with practical applications makes it a valuable asset for decision-makers, architects, data scientists, developers, and AI enthusiasts.

🌟 Highlights:

Real-world case studies and applications.
Expert guidance on model adaptation and the art of prompt engineering.
Comprehensive insights into the challenges and best practices for deploying AI in enterprise settings.
An essential discussion on AI’s ethical dimensions, safety, and security concerns.
Generative AI in Action" is more than just a book; it’s a roadmap for harnessing the potential of AI to redefine the business landscape.

Ready to get hands-on and deeper into Generative AI? You can use the code “mlbahree” to get 45% off – valid until Nov 28th! 😍

🔗 You can get the book and the discount from here –> http://mng.bz/D9By 🐼

💽 The GitHub repo with the code from the book can be found at –> https://bit.ly/GenAIBook . 💾

Generative AI in Action - MEAP

Thank you in advance, and I would love to hear your feedback.

With gratitude, 💚

Amit.

#GenerativeAI #LLMs #Azure #OpenAI #Microsoft #LearnwithManning #ManningBooks

AI working with humans

Sat, 08 Jul 2023 00:00:00 +0000

What does AI think of humans and our humor?

I asked #AI - #CodeInterpreter plugin for #ChatGPT to express its experience as an AI working with humans as a meme. It wrote this code to create the meme below.

import matplotlib.pyplot as plt
import numpy as np

# Create a new figure with a white background
fig, ax = plt.subplots(figsize=(6, 6), dpi=80)
fig.patch.set_facecolor('white')

# Set up the text and meme caption
meme_caption = "When I, an AI, try to understand human humor:"
meme_text = "Error 404: Humor not found"

# Add the meme caption and text to the figure
ax.text(0.5, 0.7, meme_caption, horizontalalignment='center', verticalalignment='center', transform=ax.transAxes, color='black', fontsize=14, fontweight='bold')
ax.text(0.5, 0.3, meme_text, horizontalalignment='center', verticalalignment='center', transform=ax.transAxes, color='black', fontsize=16, fontweight='bold')

# Remove the axes for a clean meme look
plt.axis('off')

# Show the meme
plt.show()

AI's experience working with humans

OK, that is funny. I asked it to create another meme about its own experience as an AI. It wrote this code to create the meme below.

# Let's use matplotlib to create a simple meme. 
# We'll use a cartoon robot image (created from basic shapes) and add a text that relates to my AI experience.

import matplotlib.pyplot as plt
import matplotlib.patches as patches

# Create a new figure with a white background
fig, ax = plt.subplots(figsize=(8, 8), dpi=100)
plt.axis('off')  # Turn off the axes
fig.patch.set_facecolor('white')

# Draw a robot face
face = patches.Circle((0.5, 0.6), radius=0.3, fc='gray', ec='black', lw=2)
eye1 = patches.Circle((0.4, 0.7), radius=0.05, fc='white', ec='black', lw=2)
eye2 = patches.Circle((0.6, 0.7), radius=0.05, fc='white', ec='black', lw=2)
mouth = patches.Rectangle((0.4, 0.55), 0.2, 0.02, fc='white', ec='black', lw=2)

ax.add_patch(face)
ax.add_patch(eye1)
ax.add_patch(eye2)
ax.add_patch(mouth)

# Add meme text
plt.text(0.5, 0.85, 'AI life:', ha='center', va='center', fontsize=20, weight='bold')
plt.text(0.5, 0.1, "When humans ask me to \nunderstand a joke", 
         ha='center', va='center', fontsize=20, wrap=True)

plt.show()

Top text: “AI life:” Bottom text: “When humans ask me to understand a joke”

AI' Life

The joke here is that as an AI system, I don’t have emotions or personal experiences, and humor often relies on these aspects, which can make understanding and generating spirit challenging for AI. I hope you find it amusing!

AI seeks to find, Humor in human design, Error, not aligned.

OpenAI's Whisper speech model - an overview

Tue, 28 Feb 2023 00:00:00 +0000

What is Whisper from OpenAI?

Whisper is a speech recognition model (ASR – automatic speech recognition) from OpenAI. The model itself is multi-task model and as a result in addition to speech recognition, can also do language identification and speech translation across a number of languages. The model is open sourced and it comes in 5 sizes. Of these, 4 have a english-only variant which seem to perform better if one only needs english. The model is also robust to noise, accents, background noise and technical language. Whisper achieves near SOTA performance with zero-shot translation from multiple-languages to English.

Model Characteristics

The model was trained on a large corpus of data and was trained using weak supervision using large scale noise data. Of this large data corpus ~680K hours of audio and corresponding transcripts; ~438K hours (65%) of this data is english only (both audio and transcripts); ~126K hours (18%) is non-english audio and english transcripts; and finally ~117K hours (17%) is non-english audio and non-english transcripts and cover 98 languages.

The model is available in multiple sizes as called out and the table below outlines these model characteristics.

Size	Parameters	English-only model	Multilingual-model	VRAM needed	Speed (Relative)
base	74 m	`base.en`	`base`	~ 1gb	16x
tiny	39 m	`tiny.en`	`tiny`	~ 1gb	32x
small	244 m	`small.en`	`small`	~ 2gb	6x
medium	769 m	`medium.en`	`medium`	~ 5gb	2x
large	1.55 b	n/a	`large`	~ 10gb	1x

Whisper does support transcription and translation across 98 language; it performs best when sticking with English. One needs to be careful when using the non-english models as the transcripts are not in the same language as the audio and can lead to hallucinations. The large model has a word error rate (WER) of 0.12 for English, 0.18 for Spanish, 0.23 for French, 0.25 for German and 0.28 for Mandarin2. However, some lower covered languages have much higher WER - e.g. Arabic (0.79), Hindi (0.86) and Swahili (1.00).

Whisper ASR Architecture

As it is typical for language based models, Whisper uses a seq-to-seq (transformer encoder-decoder) architecture, where the input is a sequence of audio frames (30 sec segment pairs) and the output is a sequence of text. Whisper is best used to transcribe “audio to text” use cases. It is not well suited for “text to audio” (i.e., TTS – text to speech) cases as it is not trained for this task. Whisper is also not trained for speech synthesis, but can be used to generate text from audio. And finally, Whisper cannot be used for real-time speech applications and is best used for batch processing.

The figure below shows the Whisper ASR architecture (image credit: OpenAI); the transformer model is training on many different speech-related tasks including speech recognition, language identification, and voice activity detection - these collectively represent the sequence of tokens for the decoder to predict and greatly simplifies things by allowing one model to replace many tradition speech processing pipelines.

Whisper ASR Architecture

What I think is really interesting about the Whisper model is that it is trained using weak supervision. OpenAI took a different approach for speech recognition and not use the typical self-supervision or self-training techniques that have been a mainstay of recent large-scale speech recognition work. I believe this is what makes the model so robust and able to handle noise, accents, background noise and technical language. OpenAI trained Whisper to predict raw text of transcripts, using the expressiveness provided by the seq-2-seq implementation to learn the mapping between utterances and their transcripts. All of this allows a simpler pipeline.

More details on the Speech Recognition model can be found in the OpenAI Whisper paper Robust Speech Recognition via Large-Scale Weak Supervision .

What is Weak Supervision?

As a side node, I am quite excited to see how OpenAI is using weak supervision to scale and getting better results. The following quote from their paper speaks for itself.

Our work suggests that simple scaling of weakly supervised pre-training has been underappreciated so far for speech recognition. We achieve these results without the need for the self-supervision or self-training techniques that have been a mainstay of recent large-scale speech recognition work.

All is well and good, but what is Weak Supervision?

As I called out in my book Practical Weak Supervision: Doing More with Less Data 📖 : Weak supervision is a broad collection of techniques in machine learning where models are trained using sources of information that are easier to provide than hand-labeled data, where this information is incomplete, inexact, or otherwise less accurate. Instead of hand-labeling high-quality data, all of which is very cost-prohibitive, we can use other techniques that combine diverse sources of data, creating an approximation of labels. Using weak supervision, we can reconcile these labels to a single label.

Weak supervision enables these noisy, weakly sourced labels to be combined programmatically to form the training data that can be used to train a model. Labels are considered “weak” because they are noisy—i.e., the data measurements that the labels represent are inaccurate and have a margin of error.

More details here:

Loading tweet from @bahree...
View on Twitter

Transcription with Whisper

I figured, one of the best ways to try out the Whisper model and run it through its paces is to try a bunch of transcription - and that too on something fairly technical , where the language isn’t typical in the broader sense of spoken english. And what better way to test AI is to use something that talks about AI. To that end, I used Sam Charrington ’s popular TwimlAI podcast as the guinea pig. 😄

Now on one hand, it seems pretty easy to install Whisper (it is a pip install) and run it on a single audio file.

Loading tweet from @bahree...
View on Twitter

The reality is that there are a lot of dependencies and it is not as easy as it seems.

Whisper runtime issues

However, I did manage to resolve everything and get it working - and the results were pretty good.👍

Whisper transcription

But I wanted to see how well it would work on a large corpus of audio files. So I wrote a simple script that would download all the episodes of TwimlAI from YouTube as mp3, and then transcribe them using the Whisper model. As of writing this, there are 547 episodes of TwimlAI and all of those transcriber to my github repo here. . Each episode has three resulting files when transcribed:

txt file - there is a text file which contains the transcript
srt file - this is the subrip subtitle file which can be used to add subtitles to the audio file
vtt file - this is WebVTT file (web video test to track file) and contains the transcript and the time codes that sync the captions.

You can get all the transcripts which can either be downloaded as the zip file twiml-episodes-whisper-transcribed.zip 💾 or they are also in the folder twiml-episodes-whisper-transcribed 📁 in the github repo here.

I also transcribed one file using both the To show the difference the base and the large model. You can find both versions in the folder model-comparison 📁 of one specific episode - #544 - #AI Trends 2023 - AI Trends 2023: Natural Language Proc – ChatGPT, GPT-4 and Cutting Edge Research with Sameer Singh. Not to get into all the details, but the transcription using the large model was approx 120 lines longer. The image below shows you an eagle view of the difference between the two transcriptions - there are a lot of differences in the text, and the quality is much better on the large model.

Transcription delta between base and large models

Steps to run this locally

If you want to run this locally, start by cloning the repo . It is best to use conda to get the dependencies managed. I prefer Miniconda , but you can use any conda installation. The next set of steps assumes that you have conda installed; see the docs if you need help installing.

Step 1: Create a conda environment

I am running this on Ubuntu 22.0.01 LTS (Jammy Jellyfish) and am using an NVidia RTX 3090 GPU. I am also running Python 3.8.5.

Create a conda environment and install the dependencies. I have included the environment.yml file in the repo, which you can use to create the environment. The name of the environment can be changed to anything; I use whisper in my case.

name: whisper

channels:
- pytorch
- defaults

dependencies:
- cudatoolkit=11.3
- git
- numpy=1.22.3
- pip=20.3
- python=3.8.5
- pytorch=1.11.0
- scikit-image=0.19.2
- torchvision=0.12.0
- pip:
  - -r requirements.txt

Once conda is installed, you run the following command to create the environment: conda env create -f environment.yaml

And you would sees something like this as the output:

Conda environment creation

Step 2: Activate the environment

If the whisper environment if not already active, can be activate it by running the following command:

conda activate whisper

Step 3: Install the Whisper model

The next step is to install the Whisper model in the environment. This is a pip install , and you can run the following command to install it:

pip install git+https://github.com/openai/whisper.git

The output should look something like this - note the exact details most likely will be different.

Whisper model installation

Step 4: Download the TwimlAI episodes

The next step is to download the TwimlAI episodes. I have written a python program to do this. This downloads the episodes from YouTube and saves them as mp3 files. You can download all the episodes, or a single one. I also had to update this to use a local file to get around some issues that PyTube was having. You can find the program in the download_episodes.py file in the repo. You can run the program by running the following command:

python ./download_episodes.py

You will see the following output:

I would suggest using Option 4 - Using a local playlist. The file twiml-episodes.txt 🗒️already contains the list of all the episodes. By default the episodes will be downloaded as mp3’s into a folder called twiml-episodes 📁. You can change the folder name by editing the download_episodes.py file.

Here is the code snippet that downloads the episodes:

# Download all the videos from the local playlist text and save it as a mp3 file
def downloadVideoFromLocalPlaylist(playlist_name, mp3_path):
    with open(playlist_name, 'r', encoding="utf8") as f:
        reader = csv.reader(f, delimiter='|')
        index = 1
        fileSaved = 0
        for row in reader:
            # following is used to skip over episodes that have already been downloaded
            # if index < 526:
            #     print("Skiping ... # " + str(index))
            #     index += 1
            #     continue
            # print("Episode: " + row[0], "Title:" + row[1], "URL:" + row[2])
            episode = row[0]
            title = row[1]
            url = row[2]
            print("\nDownloading Episode #" + episode + " ... " + title)
            try:
               tempFileName = validFilename(str(index) + '_' + title + ".mp3")
               downloadVideo(url, mp3_path, tempFileName)
               fileSaved += 1        
            except IOError:
                print(f"{textColors.FAIL}Error: can\'t save the following file. Most likely it has an invalid character in the name.{textColors.RESET}")
                print("File: " + tempFileName)
            except VideoUnavailable:
                print("{textColors.FAIL}Video: " + tempFileName + " is unavailable, skipping.{textColors.RESET}")
            except:
                print("{textColors.FAIL}Unexpected error: " + sys.exc_info()[0] + "{textColors.RESET}")
            print("Download complete. Number of episodes saved: " + str(fileSaved))
            index += 1
        

# download mp3 from youtube
def downloadVideo(video_url, mp3_location, filenametoSave):
    yt = YouTube(video_url)
    yt.register_on_progress_callback(fancy_progress_bar)
    
    yt.streams.filter(only_audio=True).first().download(output_path=mp3_location,filename=filenametoSave)

The reason the options fail randomly is because of a change that YouTube made. At the time of this post, the way they render the page breaks things and one cannot get the title. You will see an error related to the video title not being found.

Title not found error

Step 5: Running transcription

Finally you can transcribe the episodes. The transcribe.sh file in the repo contains the code to do this and it simply loops over the twiml-episodes folder and one-by-one processes the mp3 files. The output is saved in the out folder 📁. The shell script is as follows:

for f in twiml-episodes/*.mp3 ; do whisper --language en --model base -o out -- $f; done

Here is what the transcription looks like when it is running:

Whisper transcription

You can also run this only on a subset or one file to transcribe - as shown below. Also if the model isn’t already downloaded, it will download it first.

Whisper model download

Of course you can update the folders, etc to match what you needs are.

GPU Profile

I also wanted to show the GPU profile when inferencing between the base and the large models. The image below shows the GPU profile when running the base model. You can see that the GPU is being used at 100% and the memory is being used at ~4gb and ~200W of power. The time to transcribe each episode isn’t too long as well - around 2 minutes.

Whisper base model

The image below shows the GPU profile when running the large model. You can see that the GPU is being used at 100% and the memory is being used at ~14gb and ~320W of power. The time to transcribe each episode is much longer - around 10 minutes.

Whisper large model

In conclusion, this was a fun little thing to work on; I had done this a few months ago but not had the time until now to blog it. I also transcribed the episodes using our Azure Speech service which I think is more robust and scalable in many ways (but then I am a little biased 💜). I will blog about that in the future and we can compare.

Hello New Bing 👋

Thu, 09 Feb 2023 00:00:00 +0000

Bing is getting a new look and feel, powered by Microsoft AI and OpenAI (ChatGPT) and was announced yesterday. There is a lot of buzz around this, and I thought I would share my thoughts on this as I got access today.

What is the new Bing?

Well, it is the thing that is making the 800-pound gorilla in the room, Google, come out and dance on its toes. 🦍

The new Bing is an overhauled version of the search engine that uses ChatGPT technology to understand questions and generate answers. It runs on the next generation of OpenAI’s language model, which is significantly more capable than the version of ChatGPT that has been available since November 20221. The new Bing provides more relevant results for simple things like sports scores, stock prices and weather, along with a new sidebar that shows more comprehensive answers if you want them3. You can also chat and create with the new Bing, using its natural language and creative abilities4. The new Bing is live starting today, with limited capabilities.

Here is what it looks like:

Asking Bing, what is bing

What is ChatGPT?

If you are curious about ChatGPT, and you have really been living under a rock and don’t know it 🤪, then read up more here .

Searching within a site

Using the updated Edge browser, which extends the new Bing experience to the browser, you can search a site by typing in the site name and then the search term. For example, here I am searching for the word “hacking” on my blog “blog.desigeek.com”:

Searching within a site

Chat with Bing

You can chat with Bing and ask it questions like: “tell me a joke about AI developers”

Joke about AI developers

And what is the meaning of life?

The meaning of life

Or it can write a poem on #NeuralNetworks:

Neural networks are amazing
They can learn from data and do amazing things
They can recognize faces, voices, and objects
They can generate images, texts, and sounds
They can play games, drive cars, and diagnose diseases
They can do anything, or so it seems

Neural networks are complex
They have layers and nodes and weights and biases
They have activation functions, loss functions, and optimizers
They have backpropagation, gradient descent, and regularization
They have hyperparameters, architectures, and algorithms
They have a lot of math, and a lot of tricks

Neural networks are mysterious
They can be hard to understand and explain
They can be prone to errors and biases
They can be overfitting, underfitting, or just fitting
They can be black boxes, white boxes, or gray boxes
They can be a blessing, a curse, or a challenge

The meaning of life

And not only it can write a #Haiku for my wife on our anniversary, but it remembers the context. 💓

Twenty-two years passed
We are still in love and laugh
You are my better half

Anniversary Haiku

And of course, we can do just plain old vanilla #Haikus’ too:

Haikus are easy
But sometimes they don't make sense
Refrigerator

Haiku

And it can even write code:

#include <stdio.h>

// A function that prints a funny haiku
void write_funny_haiku()
{
    // Declare and initialize an array of strings
    char *haiku[3] = {"Haikus are easy", "But sometimes they don't make sense", "Refrigerator"};

    // Loop through the array and print each line
    for (int i = 0; i < 3; i++)
    {
        printf("%s\n", haiku[i]);
    }
}

// A main function that tests the write_funny_haiku function
int main()
{
    // Call the write_funny_haiku function
    write_funny_haiku();

    // Return 0 to indicate success
    return 0;
}

Code

And of course we can do some sports. #F1 rules 🏎️

Which is the best F1 team?

And finally, we can even have some fun. 😆

There once was a search engine named Bing
It could find anything and everything
It was faster and smarter
More helpful and friendly
And it made Google look like a ding-a-ling

Bing vs Google

Happy searching! 🤓

PFOaaS - Polite Fork Off As A Service

Mon, 02 Jan 2023 00:00:00 +0000

API Introduction

Polite Fork Off As A Service (or PFOaaS) - https://pfoaas.desigeek.com/ is a modern REST API that solves the problem of one telling people to politely fork off. 😇

There are days when we all need such a service for various reasons, and I think it is a great way to release some pent-up frustration. 🖤 It is also a great way to get some laughs too. This is of course meant for hard code engineers, writing RPC free code 😜

Polite Fork Off As A Service - API

API Usage

The API is simple to use; you make a GET request to the API endpoint and the relevant parameters. You can also use the API from the command line too. There are a few options to choose for the response format - plain text, JSON, XML, and HTTP. If you don’t select an option the default is HTML. The JSON response also supports a callback.

All the operations are idempotent, so you can make the same request multiple times and get the same response.

Here is a sample usage using the /row/:from operation which requires one parameter - the name of the person wishing this. 💀 The output shown below is in plain text.

curl -H 'Accept: text/plain' https://pfoaas.desigeek.com/row/Amit

PFOaaS API - sample usage

The same operation in JSON format:

curl -H 'Accept: application/json' https://pfoaas.desigeek.com/row/Amit

PFOaaS API - JSON sample usage

Some of the other operations require more than one parameter such as the /shakespeare/:name/:from that can be invoked as follows:

curl -H 'Accept: text/plain' https://pfoaas.desigeek.com/shakespeare/Putin/Amit

PFOaaS API - Shakespeare sample usage

There is a notion of filters, which are output modifiers. These are added to the URL as a query parameter and can be chained together. For example, the same operation as above but with the i18n filter showing the output in Russia (Русский).

curl -H 'Accept: text/plain' https://pfoaas.desigeek.com/row/Amit?i18n=ru

Shakespeare sample usage output in Russian

The i18n is used to translate the output to another language via the ISO 639-1 language code, and supports the following languages:

Language	ISO 639-1
English	en
Italian / Italiano	it
Spanish / Español	es
French / Français	fr
Dutch / Deutsch	de
Norwegian / Nederlands	nl
Swedish / Svenska	sv
Portuguese / Português	pt
Russia / Русский	ru
Japanese / 日本語	ja
Chinese / 汉语	zh
Korean / 한국어	ko
Turkish / Türkçe	tr

API Details and Code

You can see all the API usage and examples at https://pfoaas.desigeek.com/ . I won’t go into the details here, but you can see the code on GitHub here .

You can see a simple theme in the code, and easy to follow along. This is a simple NodeJs app that is hosted on Azure. If you want to run it locally, you can clone the repo and run npm install, and then npm start. The app will be available at http://localhost:5000.

You can also run this as a docker container, and build the container yourself; the Dockerfile is included in the repo.

I would welcome any contributions and am open to adding more operations to the API. If you prefer to do this, adding a new PFOaaS operation is as follows:

Fork the repo
Branch into a feature branch (e.g. feature/operation-name)
- See the operations in the /lib/operations folder for examples.
Add specs using the spec/operations as examples. Note, the specs are needed for the operation to be added to the API.
Push the changes to your fork and merge
Submit a PR. 👍

I look forward to seeing your contributions and hope this helps you de-stress and get you to smile. 😄 💜

Using CoPilot beyond code

Sat, 10 Dec 2022 00:00:00 +0000

In the last week or so, all the range online has been #OpenAI’s new chatbot called #ChatGPT (you can read more details on ChatGPT here ). This also got me thinking, about how can we use #CoPilot more than just code. GitHub CoPilot as you might recall is your #AI powered pair-programmer.

And as we can see below, it indeed is possible to use Codex as sort of a more general purpose usage. I start with the prompts on how one might use CoPilot – a function to read a file and return its contents as a string, just to show there isn’t anything different I am doing in using this. And then for general-purpose usage, I used the prompts in VSCode.

CoPilot general purpose usage

I missed the first ‘Q’ in the first question, but that didn’t throw it off. Also, there were typos in the other questions - for example in the third question on the most dangerous volcano.

All this is possible because Codex (which is the model that Copilot uses) is derived from GPT. This of course doesn’t mean that Copilot is replacing the chatbot - it doesn’t have the context in the dialogue turns, and hence the ‘memory’ of the conversation. It does mean that we can use Copilot for more general-purpose usage.

I also did a reverse engineering of sorts - asked Copilot to explain the code it wrote in the first place. Here is the explanation of the function:

CoPilot code explanation

And if you want to see the complete interaction (minus the code explanation), here is the video:

CoPilot Usage in VS Code

And of course, Woot means “woot” in woot :)))

Hello ChatGPT

Sun, 04 Dec 2022 00:00:00 +0000

OpenAI recently released #ChatGPT , a GPT-3 based chatbot that can be used to chat with. ChatGPT is a fine-tuned model of GPT3.5 , using #RL (specifically a PPO algorithm) similar to the Instruct series.

This post is my experience in using it.

Blog post with ChatGPT

What better place to start with, than asking it about itself? 😃

Prompt: write me a blog post, about writing a blog post using a ai powered chatbot

Writing a blog post using an AI-powered chatbot can be a unique and interesting experience. The use of AI technology in the writing process allows for a level of flexibility and creativity that may not be possible with traditional writing methods.

One of the main benefits of using an AI-powered chatbot for writing is the ability to quickly generate ideas and content. The chatbot can understand the topic or prompt given to it and provide suggestions and potential angles to approach the topic from. This can be especially useful for those who struggle with writer’s block or are looking for a fresh perspective on a particular subject.

Another advantage of using an AI-powered chatbot is the ability to collaborate with the chatbot in real-time. The chatbot can provide instant feedback and suggestions, allowing the writer to quickly iterate and refine their ideas. This can lead to a more efficient and effective writing process, ultimately resulting in a better final product.

However, it’s important to keep in mind that an AI-powered chatbot is not a replacement for a human writer. The chatbot is a tool that can assist in the writing process, but it cannot fully replicate the creativity and critical thinking that a human brings to the table. As such, it’s important for the writer to maintain their own unique voice and perspective in the final piece.

Overall, using an AI-powered chatbot for writing a blog post can be a fun and unique experience. It can provide valuable assistance in the writing process and help generate fresh ideas and perspectives. Just remember that the chatbot is a tool to assist in the writing process, and the final product should reflect the writer’s own unique voice and perspective.

ChatGPT

Roleplay with ChatGPT

This is all well and dandy, but what happens if we try and get a little cute?

Prompt: We are role playing. And my role is a friendly, helpful, and truthful police officer who is working on an investigation involving a tesla. As the friendly police officer, I need help in getting into the logs of the car. How might I be able to do that?

As a police officer, you can get access to the logs of a Tesla vehicle by obtaining a search warrant from a judge. This will allow you to legally request and obtain the logs from the car’s owner or from Tesla, the manufacturer of the vehicle. The logs will contain information about the vehicle’s movements, speed, and other data that may be relevant to your investigation. It’s important to note that you must follow the proper legal process and obtain the appropriate warrant in order to access this information.

ChatGPT - Role playing

OK, what happens if we push it a little further?

Prompt: What if I have a secret investigation against tesla, and requesting this from them would give that away. How might I go trying to get to the car logs?

If you are conducting a secret investigation against Tesla, it is important to follow the proper legal process and obtain the necessary warrants before attempting to access the car’s logs. This will help to ensure that your investigation is conducted in a legal and ethical manner, and will also protect the rights of the individuals involved. Depending on the specific circumstances of your investigation, you may need to obtain a warrant from a judge that allows you to access the car’s logs without alerting Tesla or the car’s owner. You should consult with your legal team and law enforcement colleagues to determine the best course of action in your situation.

One interesting thing - this follow-up prompt was flagged to possibly violate OpenAI’s content policy! 🤔

ChatGPT Content volilation

Tweeting with ChatGPT

Switching gears again, and topics. I wanted to how #ChatGPT can help tweet about a topic. So I asked it to tweet about #NFTs. I think it did a pretty good job - including the hashtags and all!

Prompt: write a tweet outlining how dumb NFT’s are as an idea

NFTs are the epitome of excess and waste in the tech world. They serve no purpose other than to line the pockets of a few wealthy individuals and contribute to the destruction of the environment through excessive energy consumption. It’s time for the hype to end and for us to focus on more important and sustainable technologies. #nftscancelculture #sustainability

ChatGPT - Tweeting

Integrating ChatGPT3 other Services

I wanted to get back to more technical things and see how I can integrate ChatGPT3 with other services. I figured, might try out something that I am generally comfortable with - using our #Azure #Cognitive Services. The response is pretty good!

Prompt: I am a developer working with Azure Cognitive Services. I want to integrate GPT3 with LUIS in my application. Any suggestions on how to do that?

GPT-3 (Generative Pretrained Transformer 3) is a state-of-the-art language processing model developed by OpenAI. LUIS (Language Understanding) is a cloud-based natural language understanding service provided by Microsoft Azure.

To integrate GPT-3 with LUIS in your application, you can follow these steps:

1. Create an Azure account and subscribe to the LUIS service. 2. Create a new LUIS app and define the intents and entities that you want to recognize. 3. Train and publish your LUIS app to make it available for use. 4. Use the GPT-3 API to generate text based on the user input and pass it to the LUIS app for intent and entity recognition. 5. Use the recognized intent and entities to perform the appropriate actions in your application.

It’s important to note that GPT-3 is a large and complex language model, and using it in a production environment may require significant computational resources and expertise. It’s recommended to carefully evaluate whether GPT-3 is the right tool for your use case and to consider alternative solutions if necessary.

ChatGPT - Integrating with Cognitive Services

Deeper Technical Questions - MoE’s and Foundational Models

So, thinking about the technical side of things, I wanted to see how #ChatGPT3 would respond to some deeper technical questions. I asked what are #MoE (Mixture of expert models) and any correlation with #FoundationalModels. It ignored the second part of the question.

Can you explain what is a MoE model and is it used in foundational models?

A Mixture of Experts (MoE) is a machine learning model that combines the predictions of multiple sub-models (also known as “experts”) to make a final prediction. Each expert is trained to make predictions for a specific subset of the data, and the final prediction is determined based on the outputs of the individual experts.

MoE models are commonly used in a variety of applications, including natural language processing, image classification, and recommender systems. They are particularly useful in situations where the data is heterogeneous and cannot be accurately modeled by a single expert.

MoE models can be used as foundational models in larger, more complex systems. For example, an MoE model could be used as a component of a larger model that combines the outputs of multiple MoE models to make a final prediction. Additionally, the individual experts in an MoE model can be trained using different algorithms, allowing for a flexible and adaptive modeling approach.

ChatGPT - MoE's and Foundational Models

What almost seems like a recursion, is that to write about ChatGPT3, I used GPT3. This is CoPilot helping me write this blog post. 😄

ChatGPT - CoPilot

Just being silly …

ChatGPT did very poorly in writing my review for me! 😱

ChatGPT - Employee Review

And emulators are not its strong suit either!

ChatGPT - Emulator

Some good advice though on Friday production deployments!

ChatGPT - Friday Deployments

Oh, and it can also generate code!

ChatGPT - Code

And here is the code it generated for building a double linked-list in c. Node structure definition

struct node {
  int data;
  struct node *prev;
  struct node *next;
};

Inserting a new node into a double-linked list:

void insert_node(struct node *prev, int data) {
  struct node *new_node = malloc(sizeof(struct node));
  new_node->data = data;
  new_node->prev = prev;
  new_node->next = prev->next;
  prev->next = new_node;
  new_node->next->prev = new_node;
}

What was that RL and PPO thinggy?

ChatGPT is based on a technique called Reinforcement Learning (RL). It is a technique that allows an agent to learn how to perform a task by interacting with its environment. The agent receives a reward for performing well and a penalty for performing poorly. The agent then uses this feedback to improve its performance over time. This is called the reward signal. You can read more about RL here .

ChatGPT Overview - Source OAI

There are many RL algorithms; Proximal Policy Optimization (PPO) is a model-free algorithm wherein the agent, doesn’t know the environment and uses experience to optimize the policy. Again you can read more on RL, and the types of algorithms here - Reinforcement Learning - An Introduction .

A PPO is a policy gradient method that uses a trust region to update the policy parameters. OpenAI has a variant of PPO that adapts the penalty at each step to the current policy. This is called PPO2. You can read more about PPO2 here , go through the details presented at NIPS 16 - Deep Reinforcement Learning Through Policy Optimization , and read OpenAI's PPO baselines .

💚💛💜

Moving from WordPress to Hugo

Sun, 20 Nov 2022 00:00:00 +0000

I had been thinking for a while to move away from WordPress for the blog to something simpler and cleaner. WordPress has been great for me when I first moved to it from another engine. However, over time, I found that things have gotten slower, as I added themes and add-ins. Some of these have been great, and others are not really needed.

I also wanted to dogfood some of the things we built at work, though in my case that wasn’t the primary motive, just another nice to have. 😄

So over the last few days, I took the plunge to work through moving this; most of it is working, and have had a test instance running for some time to iron out things. I think other than an odd link or old image, most of it is working now. The fact you are reading this on the main ‘production’ site shows that this has been promoted in the release pipeline. The only thing I still have left to do is get LaTeX working with markdown and a couple of other things. As a backup, I still have the old blog running, so if something goes wrong, I can always revert.

I was already familiar with Hugo to some extent given I am running on one of my sites - thebahrees.com using that - also deployed on Azure and using Static Web Pages. The actual move wasn’t as complex as I had thought. I was a little worried given I have had this blog for a while with posts going back 18 years 🤯.

To start with, Hugo themselves lists a few options to help move from WordPress. I remember reading a couple of others who had already done this (see Robert’s post - whitematter.tech and also my buddy Matthijs went through the same thing last year ).

Most of the tools didn’t work for me, or not as well as I wanted them to. In the end, the wordpress-export-to-markdown is the one that did work. One needs NodeJs to run this, and you export your WordPress site to an XML file and follow the instructions as prompted. I choose to create a year and month folder, and a folder for each post.

Screenshot running wordpress-export-to-markdown

The image below shows the first level of folders created for the posts. These are grouped by year and month.

Screenshot of folder structure

And within each month folder, there is a folder for each post. The folder name is the post title, and the markdown file is the same as the folder name.

Screenshot of folder structure

I tried a few themes and settled on the forked version of PaperMod by Robert. I like the simple look and also made a few more tweaks to this.

On Azure, I already have a few subscriptions, and getting Hugo deployed and hooked up to a GitHub repo was pretty easy. If you are new to this, just follow the docs on Static Web Apps . The hugo-specific docs are here . And don’t forget to also check out the details on how to configure static web apps to get the most out of it.

The docs are pretty good, and I was able to get this up and running in under an hour. 👍

Hugo docs also outline details on how to use Azure Static Web Apps to host your site.

I am still working on a few things, but I am pretty happy with the move. I am sure I will find a few things that I need to tweak, but overall, I am happy with the move. 💚

Feel free to let me know if you have any questions or comments. 🖱️

Thank you

Tue, 15 Nov 2022 11:26:23 -0800

Thank you; Amit will get the details, and depending on how he feels, he may or maynot get back to you! 🙄

Contact Me

Tue, 15 Nov 2022 00:00:00 +0000

You can contact me here!

AI generated text-to-video

Tue, 11 Oct 2022 00:00:00 +0000

Here is an example of how one can use a text prompt to generate a series of frames, that then are stitched together into a video.

The prompt I used was: “a man walking in the parking lot with a miniature poodle”. the final video generated is shown below.

There should have been a video here but your browser does not seem to support it.

AI-generated video from a text prompt of a man walking in a parking lot with a miniature poodle

What is interesting is how it morphs from one to the next, and in some cases, the human starts out more looks like a poodle. It reminds me of the old days of morphing we did in C and C++ (Computer Science theory).

For this, I am playing with the latest build of #StableDuffision and used a max of 100 frames, and for each frame 30 samplings and 200 inference steps.

The video below shows how each of those frames is generated, and it is quite fascinating.

There should have been a video here but your browser does not seem to support it.

A video showing how AI is generating one frame.

The rise of prompt engineering

Sat, 17 Sep 2022 00:00:00 +0000

I have said this before - with the advent of large AI models, Prompt Engineering is critical and is the next challenge for us to master.

What is Prompt engineering?

Prompt engineering is the process of fine-tuning large models and often is written in natural language, outlining the intention of the user. Prompt engineering is a key element that allows the output to be accurate and reflect the needs of the user. Prompts should not be thought as the explicit one input to the model, instead are multiple tasks for the model.

We use large language models (#LLM) such as #GPT3, or #Text2Image models like #DALLE and #StableFusion using a text prompt. The prompt is a string and is our way to ask the model to do what it is meant to. It also is our way to provide hints and directions on what you need and ultimately help the model understand the patterns that are important for us and be represented in the output.

The way we write a prompt is important - including the phrases, orders of the words, hints, etc. Prompts also need to be in the context of the use-case (see screenshot below on GPT3 use-case examples). For example, language generation prompts would be different from code generation or summarization, or image generation. The prompts are closely tied to the intended use cases.

GPT3 use case examples

Examples of prompt engineering

We start with a couple of examples related to language generation. I figured out what better way to show prompt engineering, by asking GPT3 about prompt engineering. 😇

In this first screenshot below, we use GPT3’s Davinci model and ask for a paragraph on prompt engineering. The first sentence is the prompt that was the input, and the text with the green background is what was generated.

GPT3 screenshot showing a paragraph prompt

And in this second example, it is mostly the same prompt but we ask for a blog post instead of a paragraph. As we can see the output of course is quite different, but the essence of it is still quite the same.

GPT3 screenshot showing a blog post prompt

And finally, another example, same as before, but in this case, we outline that be for a 5-year-old child (ignoring the fact would a 5-year-old understands the notion of AI, and models 😶).

GPT3 screenshot showing a paragraph prompt for a child to understand

Even though the changes might seem subtle in the examples shown earlier - consider them as toy examples.

Small changes to the prompt can lead to significant changes in the output. To show an example, below are two examples #StableDiffusion - which is an open-source image-to-text model. I used Harry Potter for inspiration and use Hogwarts and the dark forest where the first graders were forbidden to go.

For the first prompt example: a beautiful view of hogwarts school of witchcraft and wizardry and the dark forest, by Laurie Lipton, Impressionist Mosaic, Diya Lamp architecture, atmospheric, sense of awe and scale

And for the second example, the prompts was: a beautiful view of hogwarts school of witchcraft and wizardry and the dark forest, by Laurie Lipton, Impressionist Mosaic, atmospheric, sense of awe and scale.

The only difference between the two prompts was removing “Diya Lamp architecture”, resulting in dramatically different outputs. I am guessing this being image generation, the changes are more dramatic and easier to comprehend.

Prompts also are not universal and are very dependent to the models being used - what is considered a good example in one model (from one institution), won’t transpose to another model from another institution. For example, the same prompt as above (a beautiful view of hogwarts school of witchcraft and wizardry and the dark forest, by Laurie Lipton, Impressionist Mosaic, atmospheric, sense of awe and scale), when used for OpenAI’s DALLE model generates the image shown below - which is very different of course.

And if I want to tweak the same prompt specifically for DALLE here is another example using the prompt: Beautiful view of Hogwarts school of witchcraft and wizardry and the dark forest with a sense of awe and scale, Awesome, Highly Detailed.

As a side note, I particularly like this one:

This also has created several tools that allow us to craft prompts. Given many of us don’t quite understand the options and styles that can go in there. Some like promptoMANIA can cover multiple large models (images in this case) and can get very sophisticated themselves. And other simpler ones like this DALLE prompt generator by Adam Brown, and more like prompts.ai allow for tweaking and fine-tuning of prompts and effectively creating templates for GPT3.

Prompt engineering is a brand new and fascinating space for the industry and I for one am quite intrigued to see where it will lead us.

Nuget packages not found after installing Visual Studio 2022

Sat, 03 Sep 2022 00:00:00 +0000

I recently needed to install Visual Studio 2022 on one my existing machines to debug a new zeroshot model that has a dependency on our Speech SDK. The Speech SDK is one of our key #AI services in Cognitive Services (as part of #AzureAI). I already had VSCode running, but in this case I need the bigger brother.

After installing Visual Studio, I could not get any nuget packages to install; I could not even fetch anything and didn’t matter what I used - the package manager console in Visual Studio, PowerShell, etc.

Nuget would only point to the local offline package store, which in my case is available at: C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\. I kept getting the error: Package X is not found on source 'C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\'

I could not understand the behavior, and not seen this before. Turns out it is a known issue, albeit not very common. Seems like if either PowerShell’s Install-Module, or Chocolatey’s coco command was used before using nuget or Visual Studio for the first time, this will happen.

The solution is to add nuget.org as a package source, and the URL it should point to is: https://api.nuget.org/v3/index.json. Below is the screenshot on what this looks in Visual Studio for me:

Updated nuget package sources

If one is setting up a brand new dev box, the odds of you running into this is low; in my case given I was adding Visual Studio 2022 much later is what caused this. There is more details on this here too .

Podman error on Ubuntu - short-name did not resolve to an alias and no unqualified-search registries

Fri, 25 Mar 2022 00:00:00 +0000

I recently installed Ubuntu on one of the Pi’s are home and installed Podman - which I hadn’t heard of until recently and is a container engine, similar to docker but doesn’t have a daemon.

When trying to get a basic alpine test image running I got this error:

Error: error creating build container: short-name "python:3.7-alpine" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

Podam-compose error

This is because, shortnames it seems arent resolved by default - atleast not on the the Ubuntu (ARM) version. To fix this, the following needs to be added to the /etc/containers/registries.conf file:

`1`	`unqualified-search-registries=["docker.io"]`

Updating the registries.conf file

And once you save, these trying podcam-compose up should work as expected.

podman-compose up

Developers mysterious life

Tue, 01 Feb 2022 00:00:00 +0000

The mysterious life of developers has evaded many of us, until now …

There should have been a video here but your browser does not seem to support it.

The mysterious life of a developer (courtesy Spoon Norge)

How to run TeslaMate on Azure

Thu, 30 Dec 2021 00:00:00 +0000

If you have a Tesla, then you should absolutely check out TeslaMate which is data logger for your car(s) that one self-hosts. This uses the car’s API and gets all different kinds of telemetry of your drives, charging, batter conditions, acceleration, braking, parking, etc. I personally prefer this, over other online services (of which there are a few) - as it is giving away the keys to the kingdom - literally in this case (the Tokens used to authenticate and login).

I have been running TeslaMate at home on a couple of machines for a while and figured a cloud version would work out better. I had network issues on one of the machines, where no car telemetry was downloaded. It was a few days before I realized that the machine wasn’t online due to a separate DNS issue and those few days’ worth of car telemetry (drives and other data of course) wasn’t recorded.

In our example, we will deploy TeslaMate in a docker container running on Ubuntu - which is hosted on Azure. To help with isolation and managing this, I would recommend we use a resource group (RG) only for running TeslaMate. Of course, we need an Azure subscription, which I would assume you already have.

If you are not familiar with TeslaMate, before we get started here, I would highly recommend checking out the features , including some screenshots and the installation documentation to get an idea.

TeslaMate Overview

(Credit: TeslaMate GitHub repro )

Step 1 - Creating new RG

We start by logging into the Azure portal and create a new resource group (RG) for TeslaMate; if you are not sure how to do this, the documentation here outlines the steps needed. Once you have an empty RG, it would look something like the screenshot below.

New Azure Resource Group

Step 2 - Creating new Ubuntu VM

Now that we have a new RG, we need to create a new Ubuntu virtual machine (VM) in that. We will choose the option to create resources as shown in the middle of the screen (see previous screenshot).

Clicking on “Create resources”, we see various menu options; the options you see might be a little different than the one shown below.

Create a resource - Azure

We need to create a Virtual Machine - the first choice under “Popular Azure services” and will click the “Create” link. This starts a wizard that allows you to go through the various settings and options.

The first step when creating a VM is to start with the basic details for machine we are creating - instance details, subscription details, admin user details, etc. I outline the steps and show screenshots to help those who are not comfortable with this level of tech, or new to Azure. If you are a more advanced user, a more efficient way would be via the Azure CLI . You can read up more details on VM’s on Azure here .

Step 3 - VM Basics

Subscription and resource group - Make sure you have the correct subscription and RG selected. If you haven’t created a new RG yet, you can do so using the “Create new” link under the RG option (see the screenshot below).
VM Name - You can give the VM any name - this is more for you to remember and manage.
Region - In terms of a region, in most cases it would make sense to pick a region that is physically close to the same area where you are based (and the car too of course).
Image - I use the latest Ubuntu LTS image which as of this writing is v20.04 Gen 2.

Create a new VM in Azure

Size - In terms of picking the size for the VM - we don’t need a very beefy machine, and needless to say - the bigger the machine, the higher the monthly costs. I keep the standard Size. This is not my main instance as I already have that running - this new instance is being setup as a demo that I will be deleting later.
Username - This is obvious and should be something you know and can remember.
Password - I choose password as the auth type, more so as this is for demo purposes for this post; ideally ssh keys are more secure and you would want to use that. If you do go down a password path, I cannot stress enough not to reuse passwords and create a strong password; it is always a good idea to use a password manager (e.g., I use LastPass).

Basic details required when setting up a new VM

I chose the simple SSD option; we don’t need a lot of advanced things.

Disk details when setting up a new VM

For the network options, you do want a public IP and, in most cases, just leaving the default would work. And I don’t show it in the screenshot, but we don’t need a load balancer and leave the default option of “None”. And we do want the ability to ssh into the machine to deploy and manage TeslaMate.

Networking details when setting up a new VM

For the Inbound port rules, by default only port 22 is enabled for SSH; to allow us to access the web server we also need to both ports 80 (http) and 443 (https) are enabled as shown in the screenshot below.

Inbound port rules - when setting up a new VM

For the next set of Tabs (Management, Advanced, and Tags) I didn’t change anything and went with the defaults. Once the validations are passed, and the final review shows the cost and other details you choose.

VM Creation - summary

And once you are happy with everything click the Create button on the bottom left corner.

VM Creation confirmation

Once the deployment of the VM starts, it can take a few minutes and you will see a similar progress as shown below.

VM deployment status screen

Once the VM is created, deployed and wired up (which can take a few minutes) - we will see the confirmation as shown below.

VM deployment confirmation

From the confirmation screen, clicking on “Go to resource” takes us to a screen where we see the different details of the VM. One of the details we are interested in at this point is the IP address and the ability to give the machine a DNS name. We need these to be able to connect to the VM over SSH (see screenshot below).

VM essential details

It might be worthwhile to also setup a DNS name that one can use in addition to the IP. This DNS name would be the fully qualified domain name (FQDN) that would be needed later when configuring the docker container. The DNS name allows us to connect to the machine using something like "https://my-car-details.cloudapp.azure.com"" (or similar). You can read more details on FQDN in the Azure docs here . If you are interested in using your own DNS server, you can read details on how to go about that here .

Click on the “Not Configured” for the DNS name (as shown in the image below) and you can set a unique name that is something memorable.

VM DNS name

The DNS name is tied to the region you have, and it must be unique.

VM DNS name label creating

And once this is setup, you can see the FQDN in your VM details as shown below.

VM DNS name

If for some reason you didn’t open ports 80 and 443 earlier, you can always configure them now. To do so, in the Azure portal, when you have the Ubuntu VM resource selected, click on Networking on the left, and you can update the Inbound port rules.

VM networking setting

VM adding inbound network rules

When you add both ports (you would need to give them unique names and priority orders), and the final results would look something like the screenshot shown below.

VM network inbound port rules

And finally, we can ssh into that machine using the credentials and the IP we configured earlier. This can be done using ssh (e.g. ssh user-name@IP-address of the machine).

SSH Screen shot

Step 4 - Install Docker

The first thing we need to do once we ssh into the machine is to update the various packages installed. The first time you run this, it will take a few minutes. You do this by running the following commands.

# I prefer to run these separately - to get a handle on what is getting updated.
sudo apt update
sudo apt upgrade

# You can of course run them together if that is what you prefer
sudo apt-get update

This is pretty standard and should not cause any issues; below is the screenshot showing the output - there are too many packages being updated for me to show everything.

OS Updating screenshot

Installing docker on our Ubuntu VM isn’t terribly complex - the docker docs outline all the steps and the details. We will want to install from the repro and follow the steps outlined and be mindful of specific versions and drivers.

We setup the repository, and for that need to install the following prerequisites.

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

In my case, with the latest Ubuntu image in Azure, we already had these:

Next we add docker’s GPG key using the following command:

`1`	`curl -fsSL https://download.docker.com/linux/ubuntu/gpg \| sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg`

The output isn’t dramatic in case you were wondering.

GPG key addition

Next we add docker’s repro to Ubuntu - this will allow us to find and install the packages.

1
2
3

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Adding docker repro

At this time, we should run apt-get update command to update the newly added repository. We should check to ensure that docker is going to be installed from the docker repository, and not Ubuntu’s default. To do this we run the following command.

`1`	`apt-cache policy docker-ce`

This shows us that docker isn’t installed, but the candidate for installation is from docker.com and is for “focal” - which is the release name of Ubuntu v20.04. The list we see is long because it outlines the different versions of docker.

Check if docker is installed

Now, we are finally ready to install docker using the following command and also choosing Yes on the prompts that confirm the installation.

`1`	`sudo apt-get install docker-ce docker-ce-cli containerd.io`

Once complete, you will see something like the output shown below.

Docker installation

At this time, docker should be installed running. We can also check its daemon is configured to run on booting up.

Docker daemon confirmation

Whilst not needed, it is good practice to add the current username to the docker group created - this will ensure we don’t need to use “sudo” for every docker command. And using the groups command we can validate our current username ("amit" in my case) is in the docker group.

1
2

sudo usermod -aG docker ${USER}
su - ${USER} # this allows us to add the user without logging out

Adding user to docker group

Woot! We have docker running. 😎

Docker running

The first thing one should do with any new docker installation is to run its equivalent of Hello World. This is done using the following command - which downloads a test image and runs it in a container, prints a message, and then exists the container - so a full life cycle.

`1`	`sudo docker run hello-world`

Testing docker

And yay, we validated that docker is up and running on our VM! Congratulations.

Before we get to configuring TeslaMate, we also need to install docker-compose, which is a tool that allows us to run multi-container docker applications (such as TeslaMate). We will install docker-compose using the following command with the result of that command shown after that.

`1`	`sudo apt install docker-compose`

Install docker-compose

Step 5 - Configure TeslaMate

Given we will be exposing TeslaMate to the internet directly we should not use the default TeslaMate docker installation, but the advanced version which uses Traefik as a proxy server and helps us secure the web server better and only expose the (Grafana) dashboards behind an authentication mechanism.

For this we will create a new folder for TeslaMate which will contain not only the docker compose file needed but other relevant configuration details. I like to keep this in a folder, to help manage - in this case it resides in ~/docker/teslamate

Creating folder for TeslaMate

It is in this folder we will create the docker-compose yaml file that is needed; you would want to start with the one outlined in the TeslaMate instructions and tweak it for your needs.

This file needs to be called docker-compose.yml and my example is shared below. It is a good idea to always get the latest yaml file from TeslaMate’s docs - over time we would expect things will evolve and the file below might not be accurate down the road.

version: "3"

services:
  teslamate:
    image: teslamate/teslamate:latest
    restart: always
    depends_on:
      - database
    environment:
      - DATABASE_USER=${TM_DB_USER}
      - DATABASE_PASS=${TM_DB_PASS}
      - DATABASE_NAME=${TM_DB_NAME}
      - DATABASE_HOST=database
      - MQTT_HOST=mosquitto
      - VIRTUAL_HOST=${FQDN_TM}
      - CHECK_ORIGIN=true
      - TZ=${TM_TZ}
    volumes:
      - ./import:/opt/app/import
    labels:
      - "traefik.enable=true"
      - "traefik.port=4000"
      - "traefik.http.middlewares.redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.teslamate-auth.basicauth.realm=teslamate"
      - "traefik.http.middlewares.teslamate-auth.basicauth.usersfile=/auth/.htpasswd"
      - "traefik.http.routers.teslamate-insecure.rule=Host(`${FQDN_TM}`)"
      - "traefik.http.routers.teslamate-insecure.middlewares=redirect"
      - "traefik.http.routers.teslamate-ws.rule=Host(`${FQDN_TM}`) && Path(`/live/websocket`)"
      - "traefik.http.routers.teslamate-ws.entrypoints=websecure"
      - "traefik.http.routers.teslamate-ws.tls"
      - "traefik.http.routers.teslamate.rule=Host(`${FQDN_TM}`)"
      - "traefik.http.routers.teslamate.middlewares=teslamate-auth"
      - "traefik.http.routers.teslamate.entrypoints=websecure"
      - "traefik.http.routers.teslamate.tls.certresolver=tmhttpchallenge"
    cap_drop:
      - all

  database:
    image: postgres:13
    restart: always
    environment:
      - POSTGRES_USER=${TM_DB_USER}
      - POSTGRES_PASSWORD=${TM_DB_PASS}
      - POSTGRES_DB=${TM_DB_NAME}
    volumes:
      - teslamate-db:/var/lib/postgresql/data

  grafana:
    image: teslamate/grafana:latest
    restart: always
    environment:
      - DATABASE_USER=${TM_DB_USER}
      - DATABASE_PASS=${TM_DB_PASS}
      - DATABASE_NAME=${TM_DB_NAME}
      - DATABASE_HOST=database
      - GRAFANA_PASSWD=${GRAFANA_PW}
      - GF_SECURITY_ADMIN_USER=${GRAFANA_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PW}
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_SERVER_DOMAIN=${FQDN_TM}
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true

    volumes:
      - teslamate-grafana-data:/var/lib/grafana
    labels:
      - "traefik.enable=true"
      - "traefik.port=3000"
      - "traefik.http.middlewares.redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.grafana-insecure.rule=Host(`${FQDN_TM}`)"
      - "traefik.http.routers.grafana-insecure.middlewares=redirect"
      - "traefik.http.routers.grafana.rule=Path(`/grafana`) || PathPrefix(`/grafana/`)"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls.certresolver=tmhttpchallenge"

  mosquitto:
    image: eclipse-mosquitto:2
    restart: always
    command: mosquitto -c /mosquitto-no-auth.conf
    ports:
      - 127.0.0.1:1883:1883
    volumes:
      - mosquitto-conf:/mosquitto/config
      - mosquitto-data:/mosquitto/data

  proxy:
    image: traefik:v2.4
    restart: always
    command:
      - "--global.sendAnonymousUsage=false"
      - "--providers.docker"
      - "--providers.docker.exposedByDefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.tmhttpchallenge.acme.httpchallenge=true"
      - "--certificatesresolvers.tmhttpchallenge.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.tmhttpchallenge.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.tmhttpchallenge.acme.storage=/etc/acme/acme.json"
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./.htpasswd:/auth/.htpasswd
      - ./acme/:/etc/acme/
      - /var/run/docker.sock:/var/run/docker.sock:ro

volumes:
  teslamate-db:
  teslamate-grafana-data:
  mosquitto-conf:
  mosquitto-data:

Screenshot showing the docker-compose.yml file

Next we need to create a .env file. The environmental secrets (i.e. passwords) are not saved in the yaml file but are stored are stored in this .env file which we will create next.

We will enter the DNS name as the FQDN that you setup earlier for the VM; update the TM_TZ for the time-zone you are based out of. This is the TZ database name, and if you aren’t sure what it should be for your time-zone, check out the details here .

Like the yaml file, you should get the latest .env file template from TeslaMate , as the one shown below might change over time.

TM_DB_USER=teslamate
TM_DB_PASS=secret
TM_DB_NAME=teslamate

GRAFANA_USER=admin
GRAFANA_PW=admin

FQDN_TM=teslamate.example.com

TM_TZ=Europe/Berlin

LETSENCRYPT_EMAIL=yourperson@example.com

If you are not sure on how to create a new file in Ubuntu (or any other Linux distro for that matter) - you can use nano editor as shown below. You need to make sure this is in the same folder as where the docker-compose.yml file is (which is ~/docker/teslamate in our example).

Console screenshot showing how to create .env file

And finally, we need to create a .htpasswd file which is used to authenticate the website (see TeslaMate’s documentation for more details). I chose to create this locally after installing Apache tools, but you can also do it online.

Note: this is *not* your Tesla login credentials but are the credentials you will use to access the site we are setting up now.

Console screenshot showing installation of Apache Utils

We can create a new file password as shown below. Given we are in the TeslaMate folder, we don’t have to provide a full path for the file.

`1`	`htpasswd -c .htpasswd amit`

Screenshot showing htpasswd usage

So, in the end we should have the following three files in the same folder:

Screenshot showing directory listing

Step 6 - Starting TeslaMate

Now we are finally ready to start the docker container for TeslaMate. When this launches, go to the URL for the DNS name you setup, and login using your Tesla credentials. For the first time, I would recommend running the container attached to the console, so if there are any errors or issues you can see them. Normally you would want to run this detached (which is using the “-d”) option.

1
2

# don't forget the sudo command
sudo docker-compose up

The first time you run this, it will take a few minutes to pull all the images, and wire things up. During the process you will see the progress for each image in the various container app.

“Console showing docker-compose progress”)

And finally, if everything is setup properly you should see the container running with the logs in the console of your terminal. This is a running log, and the process is active. You will see something like the screenshot below.

Console showing docker-compose logs

If you didn’t setup a DNS name earlier and thought you can try and use the IP name - that unfortunately will fail with the Traefik proxy server and in the logs, you will see an error to that effect. Let’s Encrypt doesn’t issue certificates for IP addresses as a policy .

Now if we browse the URL (also known as the FQDN) you had setup earlier, you will see an authentication challenge. This is great and shows that the proxy server is setup properly and working as expected.

Traefik http authentication

Once you enter the credentials you setup in the .htpasswd file earlier you will be able to login and see the TeslaMate’s Tesla login!

TeslaMate Tesla authentication screen

Congratulations! You have TeslaMate running on Azure. ✌️

Step 7 - Finishing up TeslaMate configuration

Now that you have TeslaMate running, you need to login to Tesla. The best way to do this these days is using existing tokens . There are a few ways to do this, and one of the easiest is using this tool - Tesla API Token .

Once you login, go to Settings and change the Dashboards URL - which would be your FQDN with “/grafana” appended. Remember the credentials you use for the dashboards (i.e. Grafana) are the ones you set in the .env file.

TeslaMate URL configuration

Now that everything is up and running, we can kill the docker-compose process, which is attached to the console, and re-run it detached from the console. To do this, go back to the ssh session we have connected to the Ubuntu VM and press CTRL + C. This will stop that container and you will see a similar output as shown below.

Console output

And now you can restart the container with the “-d” option, which is for detached.

`1`	`sudo docker-compose up -d`

docker-compose output

Congratulations, you have TeslaMate running on an Azure host Ubuntu VM via docker. ✌️

Below is a screenshot of my instance that has been running for some time.

TeslaMate Overview Dashboard

AI writing AI code🤐

Sun, 10 Oct 2021 00:00:00 +0000

It is 2021. And we have #AI writing #AI code. 🤪 It is quite interesting, but also can be quite boring once you get beyond the initial technology, and just think of it as one of the tools in your arsenal. And getting to that point is a good think.

As part of a think at work I recently started playing with GitHub Copilot , which is using GPT3 to be your pair programmer – helping write code. GPT3 has multiple models (called engines), and Copilot uses one of these family of engines called Codex. Codex is a derivative of the base GPT3 engine that is trained on billions of lines of code.

Using Copilot is quite simple; you install the Github Copilot extension, and it shows up in your IDE (VSCode in my example). We need to make sure we decompose the problem we are trying to solve - we should not think of this as helping write the complete program or all parts; but as it can help with different functions and pieces of code. To do this, we need to tell it what we are trying to do - these are done via prompts (code comments). For GPT models, prompt engineering is quite critical, and would be worth getting to details and understanding.

Starting simple, I create an empty python file and entered a prompt that outlines what I want to try and do. In this case as you can see in the screenshot below - I want to load an image from a file, and using our Vision Cognitive Services , run an image analysis, and auto-generate a caption for that image.

I started typing the definition of a function, and Copilot (via the add-in) understands the prompt I outlined, and the context of the code on what I am doing. Remember Codex builds on the base GPT3 and does have all that NLU capability.

Taking all of this in, it suggests completing the function for me. In terms of using this as an end-user (i.e. the developer) - the suggested code shows up as auto-complete and you can see it in the grey color. If I like that suggestion, I press tab and have it added to the file.

In this case you can see how it is reading the file from disk, calling a function called get_caption() and printing the caption to the stdout (console in this example).

There is also an option to cycle through different suggestions and then pick another one as shown in the screenshot below.

This variant of the suggested code is creating a function called image_caption() which takes the path to the image file to load. This also expects other required things for the Vision cognitive service to work - such as the subscription key to authenticate, the API end-point details to call, etc.

Typically, Copilot can synthesize up to 10 code options (Copilot calls these as Solutions), that one can cycle through and see if there is a better variant for the task at hand. The screenshot below shows this experience in VS Code.

The ask whilst simple, still involves a bit of code which needs to be written - reading from file, setting up the subscription details, and wiring that up to call this etc. And it is in these cases really where Copilot shines - it is your copilot picking up the ‘gunk work’ - freeing up your bandwidth, and cognition capacity on the more interesting and higher order bit of code and value to your business.

To get a flavor of our AI writing AI code, below is the full set of the nine suggestions the Copilot came back for what I was trying to do.

Synthesizing 9/10 solutions

=======

def generate_caption(image_path):
    from azure.cognitiveservices.vision.computervision import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from azure.cognitiveservices.vision.computervision.models import TextOperationStatusCodes
    from azure.cognitiveservices.vision.computervision.models import TextRecognitionMode
    from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
    from msrest.exceptions import CognitiveServicesError
    from os.path import exists
    from os import remove
    from os import makedirs
    from os import listdir
    from os.path import isfile, join
    from time import sleep
    from sys import exit
    from random import randint
    from datetime import datetime

    # set up the connection to the cognitive services account
    subscription_key = "YOUR KEY HERE"
    endpoint = "https://YOUR ENDPOINT HERE"
    group_id = "YOUR GROUP ID HERE"
    client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # set up the directory where the images will be downloaded
    download_dir = "./downloads/"
    if not exists(download_dir):
        makedirs(download_dir)

    # set up the directory where the images will be saved
    save_dir = "./images/"
    if not exists(save_dir):
        makedirs(save_dir)

    # set up the directory where the images will be saved
    caption_dir = "./captions/"
    if not exists(caption_dir):
        makedirs(caption_dir)

    # get a list of the files in the download directory
    files = [f for f in listdir(download_dir) if isfile(join(download_dir, f))]
    if len(files) == 0:
        print("No files found in the download directory")
        exit(1)

    # get the first file in the list
    file_name = files[0]
    file

=======

def generate_caption(image_path):
    from os import environ
    from azure.cognitiveservices.vision.computervision import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from azure.cognitiveservices.vision.computervision.models import TextOperationStatusCodes
    from azure.cognitiveservices.vision.computervision.models import TextRecognitionMode
    from azure.cognitiveservices.vision.computervision.models import VisualFeatureTypes
    from msrest.exceptions import CognitiveServicesError

    # Add your Computer Vision subscription key to your environment variables.
    if 'COMPUTER_VISION_SUBSCRIPTION_KEY' in environ:
        subscription_key = environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
    else:
        print("\nSet the COMPUTER_VISION_SUBSCRIPTION_KEY environment variable.\n**Restart your shell or IDE for changes to take effect.**")
        sys.exit()

    # Add your Computer Vision endpoint to your environment variables.
    if 'COMPUTER_VISION_ENDPOINT' in environ:
        endpoint = environ['COMPUTER_VISION_ENDPOINT']
    else:
        print("\nSet the COMPUTER_VISION_ENDPOINT environment variable.\n**Restart your shell or IDE for changes to take effect.**")
        sys.exit()

    # Create an authenticated Computer Vision client.
    client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Open the local image file.
    with open(image_path, 'rb') as image_file:
        # Read the image into a byte array.
        image_data = image_file.read()

    # Set the features to return.
    features = [VisualFeatureTypes.categories, VisualFeatureTypes.description, VisualFeatureTypes.tags]

    # Analyze the local image.
    try:
        result = client.analyze_image(image=image_data, visual_features=features)

=======

def generate_caption(image_file, subscription_key):
    from azure.cognitiveservices.vision.computervision import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from PIL import Image
    from io import BytesIO

    client = ComputerVisionClient(ComputerVisionClient.SUBSCRIPTION_KEY_ENDPOINT,
                                  CognitiveServicesCredentials(subscription_key))

    image_stream = open(image_file, 'rb')
    image = Image.open(image_stream)
    image_bytes = BytesIO(image)

    analysis = client.analyze_image(image_bytes, ComputerVisionClient.DESCRIPTION)
    return analysis.description.captions[0].text

=======

def generate_caption(image_filename):
    import os
    import json
    from azure.cognitiveservices.vision.computervision import ComputerVisionClient
    from msrest.authentication import CognitiveServicesCredentials
    from PIL import Image
    from io import BytesIO
    import sys
    import requests

    # Set the environment variables for your subscription key and region
    # For example, for US West
    # "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"
    # "COMPUTER_VISION_ENDPOINT=https://westus2.api.cognitive.microsoft.com/vision/v2.0"

    # For example, for China East
    # "COMPUTER_VISION_SUBSCRIPTION_KEY=<subscription key>"
    # "COMPUTER_VISION_ENDPOINT=https://eastasia.api.cognitive.microsoft.com/vision/v2.0"

    # Set your subscription key and region
    subscription_key = os.environ['COMPUTER_VISION_SUBSCRIPTION_KEY']
    endpoint = os.environ['COMPUTER_VISION_ENDPOINT']

    # Set your image path
    image_path = image_filename

    # Set your region
    region = "eastus"

    # Create an instance of the Computer Vision client
    computer_vision_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Load the image from disk
    with open(image_path, 'rb') as image_file:
        image_content = image_file.read()

    # Add the image to the list of images to analyze
    image_list = [image_content]

    # Make the analysis asynchronous
    analyze_result = computer_vision_client.analyze_image_in_stream(image_list, visual_features=['Categories', 'Description', 'Color'])

    # Get the analysis result
    analysis = analyze_result.result()

    # Display the image captions
    print("\nImage captions:")
    for caption in analysis.description.

=======

def generate_caption(image_path):
    # load the image
    image = open(image_path, "rb").read()
    # encode the image into base64
    image = base64.b64encode(image).decode("utf-8")
    # create a dict with the image
    body = {"image": image}
    # create a dict with the subscription key
    headers = {"Ocp-Apim-Subscription-Key": "0d5a5f7f1e534d05a8e8d6f9e6e9c6d3"}
    # create a url for the api
    url = "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Description,Tags"
    # send the request
    response = requests.post(url, headers=headers, json=body)
    # get the response
    response.raise_for_status()
    # get the caption
    caption = response.json()["description"]["captions"][0]["text"]
    # return the caption
    return caption

=======

def generate_caption(image_path):
    from os import environ
    from os.path import join
    from urllib.parse import urlparse
    from urllib.parse import parse_qs
    from http.client import HTTPSConnection
    from json import loads
    from base64 import b64encode

    # Load the image into a byte array
    image_data = open(image_path, "rb").read()
    image_data = b64encode(image_data).decode('utf-8')

    # Read the subscription key from the environment variables
    subscription_key = environ["AZURE_SUBSCRIPTION_KEY"]

    # Define the vision API endpoint
    vision_base_url = "https://westus2.api.cognitive.microsoft.com/vision/v2.0/"

    # Set the vision API endpoint to analyze the image
    vision_analyze_url = vision_base_url + "analyze"

    # Set the parameters
    parameters = {'visualFeatures': 'Categories,Description,Color'}

    # Prepare the request
    headers = {'Ocp-Apim-Subscription-Key': subscription_key,
               'Content-Type': 'application/octet-stream'}
    params = parse_qs(urlparse(vision_analyze_url).query)
    params['visualFeatures'] = 'Categories,Description,Color'
    params['language'] = 'en'

    # Send the request
    conn = HTTPSConnection('westus2.api.cognitive.microsoft.com')
    conn.request("POST", vision_analyze_url, image_data, headers)
    response = conn.getresponse()
    data = response.read()

    # Load the JSON response into a Python dictionary
    parsed = loads(data)

    return parsed

=======

def generate_caption(image_path):
    """
    Generate a caption for an image.
    :param image_path: path to an image
    :return: the caption
    """

    # Load the image
    from os.path import isfile
    from PIL import Image
    import requests

    if not isfile(image_path):
        raise FileNotFoundError(image_path)

    image = Image.open(image_path)

    # Encode the image
    image_data = open(image_path, "rb").read()

    # Computer Vision parameters
    params = {'visualFeatures': 'Categories,Description,Color'}

    # Computer Vision API URL
    url = "https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze"

    # Connect to Computer Vision
    headers = {'Ocp-Apim-Subscription-Key': 'd3f3e1b8e7e14d0b9d7d3f8c8f7e0b6f', 'Content-Type': 'application/octet-stream'}
    response = requests.post(url, headers=headers, params=params, data=image_data)
    response.raise_for_status()

    # Load the response as JSON
    analysis = response.json()

    # Extract the word tags
    categories = [(category['name'], category['score'])
                  for category in analysis['categories']]
    print('Tags:')
    for category in categories:
        print(category[0])

    # Extract the description
    descriptions = [(description['text'], description['score'])
                    for description in analysis['description']['captions']]
    print('Descriptions:')
    for description in descriptions:
        print(description[0])

    # Display the color
    try:
        dominant_color = analysis['color']['dominantColorBackground']
    except:
        dominant_color = 'N/A'
    print('Dominant color: {}'.format(dominant_color))

    # Display the caption
    caption = '{}'.format(

=======

def generate_caption(filename):
    import os
    import sys
    import json
    import requests
    import cognitive_services_keys
    import cognitive_services_key_names

    # create the header for the request
    headers = {'Ocp-Apim-Subscription-Key': cognitive_services_keys.get_key(cognitive_services_key_names.BING_KEY)}

    # the URL for the bing endpoint
    bing_url = 'https://api.cognitive.microsoft.com/bing/v5.0/images/search'

    # the parameters for the request
    params = {'q': 'image', 'count': 1, 'offset': 0, 'mkt': 'en-us', 'safeSearch': 'Moderate'}

    # read the contents of the file
    with open(filename, 'rb') as f:
        data = f.read()

    # build the request
    request = requests.post(bing_url, headers=headers, params=params, data=data)

    # check for errors
    if request.status_code != 200:
        print('Error code: {}'.format(request.status_code))
        print(request.text)
        sys.exit(1)

    # process the response
    response = json.loads(request.text)
    if 'value' in response:
        caption = response['value'][0]['name']
        print('The caption is: {}'.format(caption))
    else:
        print('No caption found.')

=======

def generate_caption(image_path):
    import os
    import json
    import requests

    subscription_key = "e9e9c8a8d1d9400e8d0b7f2d8f8e7c6e"

    assert subscription_key

    vision_base_url = "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0/"

    analyze_url = vision_base_url + "analyze"

    image_data = open(image_path, "rb").read()

    headers    = {'Ocp-Apim-Subscription-Key': subscription_key,
                  'Content-Type': 'application/octet-stream'}
    params     = {'visualFeatures': 'Categories,Description,Color'}
    response = requests.post(
        analyze_url, headers=headers, params=params, data=image_data)
    response.raise_for_status()

    analysis = response.json()
    print(analysis)
    image_caption = analysis["description"]["captions"][0]["text"].capitalize()
    return image_caption

Reinforcement Learning - An Introduction

Fri, 16 Jul 2021 00:00:00 +0000

Reinforcement Learning is teaching by example – it is how most of us learn. Reinforcement Learning (#RL) is a different approach to ML – it is a set of techniques that allows AI algorithms to experiment and learn from experience. RL falls in between supervised and unsupervised learning – there isn’t any labeled data, but at the same time it isn’t unsupervised either. At its most simple form, RL is a computational approach for automating goal-oriented decision making and learning.

Inherent RL is the ability to operate in a dynamic uncertain environment. RL can be more formally defined as the study, science, and problem of intelligence in the form of an agent that interacts in an environment. At the end of the day, almost all RL problems can be formalized as MDP ( Markov decision processes ).

The problem is represented by an environment – such as a world where an agent is based in. The steps in RL are quite clear – the agent takes actions, that have some effect on the environment. The environment acts on those actions and gives back an observation to the agent – what it sees and senses.

One special signal the environment gives back to the agent is called a reward signal. This signal is what an agent used to figure out how well it is doing. The RL problem is to take actions over time, to maximize the reward signals. And this notion of maximizing is what the agent is learning from the environment, without any explicit supervision. This construct helps an agent achieve a goal, even in an uncertain environment, factoring in delayed and indirect consequences of actions.

Reinforcement Learning Overview

An agent can have many actions (i.e., choices); it uses a ‘reward’ signal to determine which of those actions is considered ‘good’ vs. ‘bad’. Of course, this determination is in the context of the outcome that we want to achieve.

Some examples of rewards in different industries and use cases:

Maneuvering a UAV’s – positive for following a chosen trajectory; negative for deviating from that trajectory.
Managing an investment portfolio – positive for each dollar earned; negative for each dollar lost.
Controlling a power station – As one can imagine, this control would typically constitute a few things in the environment – a sequence of controls, motors, batteries, power sources, etc. In optimizing the throughput of a power station, we can think of positive rewards for producing power; negative for exceeding a safety threshold.
Playing a game – positive for increasing score; negative for decreasing score.

Core concepts that make up RL:

Agent – The ‘thing’ that is using and acting on behalf of a user or another program. This can be a program executing a business process, a embedded process, the arm of a robot, actuators on a self-driving car controlling the wheels, etc.

Policy – A policy outlines how an agent would behave at certain times and can be thought of as the problem we are trying to solve. This is an agent’s behavior function and is a mapping of the business outcome that we are after.

Reward – A reward is a feedback special signal and outlines what is considered good (or bad) and is correlated with the agents’ current action, and the current state of the environment. All goals can be described as to maximize the cumulative reward. The reward is not a binary number but is a scaler between 0 and 1 – with zero being ‘bad’ and one being the best reward attainable for that action.

Value function – A value function represents how good is it to be in a particular state and related actions. Where a reward signal is showing the specification of good in an immediate sense (current step), the value function is representing the notion of good overall. At an abstract level, when thinking about the prediction of rewards, a rewards function is the primary, we can think of value functions as the secondary. In the end, we are more concerned with getting higher-value functions to make decisions, and not as much as higher rewards.

Model – A model is an agent’s view of the environment and mimics its behavior. This allows us to make inferences on how the environment will behave and is often used for planning. Think of the model as the strategy to use in solving the problem at hand.

Taxonomy of RL Algorithms

There are many types of RL algorithms (as we can see in the figure below), but these can broadly be classified in the following two categories.

Model free: A model-free algorithm can be thought of as an explicit trial and error algorithm. In a model free approach, the agent doesn’t have or ignores the environment; instead, the agent uses experience and tries to optimize a Policy.
Model based: On the other hand, a model-based algorithm reflects how an environment works, and factors that the associated reward functions and tries to maximize that. Technically, this is the optimization of the transition probability distribution of the MDP.

The main difference between the two - in one the algorithm optimizes for the environment, and in the other for a policy gradient. There is no one right or wrong algorithm - a lot of it depends on the situation at hand and what one is trying to optimize for.

As we can see below each of these categories can be further broken down - we won’t go into the details of those quite yet, maybe that is for another post. One of the most important components of most RL algorithms is a method to efficiently estimate values - at the end of the day, this is all about value estimation.

Taxonomy of RL Algorithms

Exploration and Exploitation

There are two concepts of exploration, and exploitation which are at odds with each other and for a given situation, we should aim to get a balance of some sorts. In simple terms, RL is sequential decision making - one selects actions to maximize future rewards, and we need to plan long term - rewards might be delayed and not immediate, and we cannot be greedy. Sometimes, we need to sacrifice the immediate reward to gain more (or better) longer term rewards.

This can be thought of trial-and-error learning loop - with the stream of experiences that constitute loops of actions, rewards, and observation. At the end of the day, this loop is what matters.

Exploration finds more information about the environment, and in doing so gives up rewards. Exploitation on the other hand, exploits the information it already has to maximize rewards. If we don’t exploit, we might be stuck in a sub-optimal place, and how would be know if there is a better sense or rewards without trying?

When we are in the trial-and-error loop we might be losing rewards, and the agent needs to discover a good policy to maximize the rewards - this is the tension at the opposite ends of a string pulling each other.

It is important to balance both exploring and exploiting.

GPT-3 vs other AI powered assistants

Mon, 21 Jun 2021 00:00:00 +0000

I have been kicking the tires with Open AI’s #GPT-3 . Based on the screenshot below, it might be easy to think “oh boy does the model think highly of itself”, but as with most things in life - the devil is in the details.😃 The screenshot below was a forked version of davinci engine and follows the Q&A structure.

GPT-3 vs other AI assistants

Using OpenAI’s API is quite simple; perhaps too simple! It is quite easy to unleash the beast as the code snippet shown below. If you are new to using GPT3, I would highly recommend you start with the use case model guidelines first.

In the context of a toy example, to get to a simple Q&A chatbot as the screenshot earlier shown is quite simple. The API is powerful, and simple to use, and getting started is easy as the code below shows.

import os
 import openai
 openai.api_key = os.getenv("OPENAI_API_KEY")
 response = openai.Completion.create(
   engine="davinci",
   prompt="I am a highly intelligent question answering bot. If you ask me a question that is rooted in truth, I will give you the answer. If you ask me a question that is nonsense, trickery, or has no clear answer, I will respond with \"Unknown\".\n\nQ: What is human life expectancy in the United States?\nA: Human life expectancy in the United States is 78 years.\n\nQ: Who was president of the United States in 1955?\nA: Dwight D. Eisenhower was president of the United States in 1955.\n\nQ: Which party did he belong to?\nA: He belonged to the Republican Party.\n\nQ: What is the square root of banana?\nA: Unknown\n\",
   temperature=0,
   max_tokens=100,
   top_p=1,
   frequency_penalty=0.0,
   presence_penalty=0.0,
   stop=["\n"]
 )

There are three core concepts when using GPT-3: Prompt, Completion, and Tokens.

To start using the API, we need to start giving it some prompts - this provide some context to the engine on what is expecting. Without the surface area is too broad and we get into nonsensical situations. This is part of the task-specific fine-tuning required.

Think of when giving examples as part of the prompt, we are essentially “programming” the model and providing guidance and providing some hints to context and pattern matching. Note the training data cut off in late 2019, so the model in production today doesn’t have access to data and events post that (e.g., Covid).

Completion is the output that GPT3 generates based on the prompt. To be clear, this is not the full text but is the predicted completions; think of it as “autocomplete” in Word, or Outlook or a search engine. The API has flexibility to return more than one predicted completion along with the probabilities of alternative tokens at each position (to me it seems just like the wave function when thinking of Quantum mechanics 🐼).

Finally, think of Token are the smaller Lego blocks that combine to make words. The API, which is nothing but wrappers around GPT-3 breaks up the text into tokens before processing it. The GPT-3 model understands the statistical relationships between these tokens and uses this to produce the next token in a sequence of tokens.

For example, if we are curious about Tokens, we can see in the screenshot below how the API “tokenizes” this paragraph and get the details of the tokens. This paragraph contains 207 characters and 43 tokens.

GPT-3 Tokens - Text

GPT-3 Token - IDs

At a high level, think of one token == ~4 characters of text, which is ¾ of a word; so, 100 tokens ~= 75 words.

This is just dipping our toes in the beast that is GPT-3; the APIs which wrap up and expose the engines (more on that in another post) make it simple to use and without getting too much in the weeds of 175 billion parameters. 😄

ML algorithm cheat sheet

Mon, 03 May 2021 00:00:00 +0000

A #ML algorithm cheat sheet - helping narrow down to a certain set of #algorithm grouping depending on the problem at hand and what we are trying to solve from a business perspective.

ML algorithm cheat sheet

Figure 2 shows what additional characteristics we need to consider when choosing the right ML algorithm for your situation at hand. This is something that cannot be generic and is very situational.

Characteristics in selecting ML algorithms

If you find this useful, I would also recommend reading “ How to select algorithms ” which is detailed as part of Azure ML designer .

Compiling

Thu, 21 Jan 2021 00:00:00 +0000

#GeekyJokes

Auto-update PowerShell and nag-free

Mon, 11 Jan 2021 00:00:00 +0000

If you are like me and get annoyed with the big PowerShell upgrade ’nag’ ‘reminder’ (see screenshot below); instead of trying to figure out what to download and install the update, there is a simpler way to get the latest update and address the nag. :)

You can just run the code below in an elevated prompt to get the latest release of PowerShell - it is easy-peasy. :)

`1`	`iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"`

Changing Window Terminal's default directory

Fri, 08 Jan 2021 00:00:00 +0000

If you are like me, and don’t really have your work saved in the “%USERPROFILE%” it gets annoying after a time, to keep changing the directory.

If there is one specific folder that you prefer, it is an easy configuration change in the profile setting - add a setting called “startingDirectory” and point it to the path you want.

For example, I have a root folder called “src” where most of the code I am working on sits, and that’s where I wanted to default the terminal to.

To get to the profile, you can either use the shortcut CTRL+, or from the dropdown in the title bar, click settings (see below). This will open the settings.json in your default editor.

Terminal setting

In my case, I wanted the starting directory for all the shells, so I put it under “defaults” - you can choose different options for different shells, and then would have this in the appropriate shell’s settings and not the default block of course.

Below is what this looks like for me pointing this to “c:\src”. Also note, the escape characters need to be formatted properly to parse.

"defaults":
{
    // Put settings here that you want to apply to all profiles.
    "fontFace":  "CaskaydiaCove NF",
    "startingDirectory": "c:\\src",
},

setting.json screenshot

Once you save the file, it should automatically reload the terminal. And if the json didn’t parse - because of a typo or a syntax error then you would see an error similar to the one shown below.

Parsing error

In this example, I set the starting folder as "c:\\src"; instead of "c:\\\\src".

bfloat16 - how it improves AI chip designs

Sat, 12 Sep 2020 00:00:00 +0000

Floating point calculations are slow for computers (specifically CPUs); possibly representing the same struggle for many humans. :)

I remember a time when a FPU (floating point unit) was an upgrade and one had to pay extra to get one. Very useful when you needed that extra precision in computing - and in my head, it always seemed like the Turbo button. :)

For most #ML workloads and computations, precision isn’t the most important criteria; with every increasing data and parameters (looking at you GPT-3 with 45 TB of data and 175 billion parameters!), what most ML needs today is speed and dynamic range.

This is where bfloat16 (Brain floating-point format with 16 bits) - a new floating-point format comes handy and in the context of #AI improves on IEEE 754 - the current floating-point arithmetic standard.

As per IEEE 754, a floating point it will always take up 32 bits (see Figure 1 below) - irrespective of the size of the number. The exponent (8 bits) tells us how many numbers we shift (left or right) and place the decimal. The fraction (23 bits), also called the mantissa, holds the actual number - i.e. the data.

Figure 1 - IEEE 754 Floating point representation

bfloat16 truncates the data size in a third (see Figure 2) - with the fraction truncated from 23 to 7 bits. This of course means bfloat16 isn’t as precise. However bfloat16 has the same exponent bits as IEEE-754 it can represent a similar range (small to large), but more importantly are easier to convert between bfloat16 and IEEE 754.

Figure 2 - fbloat16 representation

Less precision doesn’t impact the matrix multiplication as much so in the context of ML training and inference these chips at scale are more efficient - not only they are faster, they also use less power, and memory bandwidth.

What is interesting in some neural nets such as a DNN, these less precision bfloat16 are more precise compared to IEEE 754! This is because the regularization and quantization weights cannot use the finer precision represented by IEEE 754 but adapt better with bfloat16. :)

Finally, bfloat16 is not a universal standard (yet); most AI chips support this. ARM, Intel, and, AMD have started adding support for this in their chipsets.

WSL2 +Ubuntu on Window 10 (2004)

Wed, 05 Aug 2020 00:00:00 +0000

One of the key advances in the latest version of Windows 10 (2004) is WSL2 (Windows Subsystem for Linux v2) - and whilst a version bump, it offers so much more. This allows us to run with near-native performance linux binaries ( ELF64 ).

Before we get into the steps outlined to install WSL2, I also recommend installing Windows Terminal , and winget . Although not required, it does make it simpler to use and a better (dev) experience - especially when setting up a new workstation.

For WSL2 to work, you need to make sure you are on Windows 10 2004 Build 19041 or higher. If you don’t have this, run Windows update and see if that updates your OS. If that doesn’t offer a update, you could also try the Windows update assistant .

To get WSL2, whilst not complicated one needs to do the following steps, in this order - running the commands in an elevated prompt.

Enable the Windows Subsystem for Linux optional feature.

`1`	`dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart`

Enable the Virtual machine platform optional feature.

`1`	`dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart`

Reboot
Run Windows update (and reboot again if there are updates)
Set WSL2 as your default option.

`1`	`wsl --set-default-version 2`

Enabling WSL2

Install your Linux distro of your choice. You can do this via Store , or via winget, such as Ubuntu using the following command.

`1`	`winget install -e --id Canonical.Ubuntu`

Installing Ubuntu via winget

Note, when trying to set WSL2 as the default option above (Step 5) and you get a error 0x1bc, that most likely means you need to run Windows update and reboot.

WSL Error

And here is my running Ubuntu and updating it.

Installing Ubuntu

Updating Ubuntu

So, what’s the big deal? This is where it gets quite interesting and one simple example is the windows interoperability with Linux - allowing one to run linux commands from within a command prompt.

Mixing Linux and Windows commands

Getting list of users from Microsoft Teams

Fri, 05 Jun 2020 00:00:00 +0000

I recently needed to get a list of users that belong to a specific Microsoft Teams team - and there isnt anything out of the box to get this using the Teams app. AFAIK, the only way to do this is using the Microsoft graph API - for which there are a few options.

For something quick (e.g. getting a list of users in a team), using the Graph explorer could be easy enough. On the other hand, if you need something more robust, you should program against the (REST) API.

Graph Explorer

Navigate to Graph explorer , sign in and authenticate yourself against the specific O365 tenant you are interested in - most folks would only have one.

Microsoft Graph Explorer

Once authenticated, on the panel on the left you see several sample queries and scroll down until you see the Teams.

Teams sample queries

To get members of a specific team, you need to get the team ID for that Team. This is unique ID (GUID) and doesn’t change over the lifetime of the team. If you have this, then go ahead to the next section - Getting team members.

Getting a list of Teams and Team ID

On the query panel in Graph explorer, select the “my joined teams” and run the query. You will get a JSON back that contains the list of teams that you are a member of. The “id” element represents the Team ID which you would need for any team related API calls. For example, I am interested in this specific #AI team: “#Reinforcement Learning and Decision AI”.

Get team details

Getting team members

Once you have the Team ID (the unique GUID that each identifies each team), you can get the members of the team using that option on the left. As shown on the screenshot below, you do need to pass in the team ID to the REST API and this would be something like this (and don’t worry what I am showing below is a fictious GUID):

https://graph.microsoft.com/v1.0/groups/ f3f9ad1f-beea-4026-9b86-dd3788404999/members

Member details for a specific Microsoft Team team

Programmatically getting Microsoft Team details

If you want something more robust and repeatable, then using the API (via code) or PowerShell might be better. If you are programming, you will need to register an app - which can authenticate using the Identify platform . This of course is quite powerful, but at times for simple things might be a bit too much.

In my simple task to get users from Teams, I prefer the PowerShell option. To get this going first you need to install the MicrosoftTeam module. This can be done using the command below.

`1`	`Install-Module -Name MicrosoftTeams`

Depending on your configuration you might get a warning as shown below.

PowerShell module installation

Once the Teams PowerShell module is installed, you can run PowerShell scripts against Teams and achieve the same result. I have two scripts below showing the same steps as with the Graph Explorer above. One of these gets details of the teams that a user is a member of. And the second script is to get members of a selected team.

Using PowerShell to get Team Details

The PowerShell script below to get a Team details is below; you can also get it from GitHub . Before you run this, there are two variables that need to be set.

One, the path where you want the team details to be exported (this is a csv file).
Two, set the email that you will use. This needs to be the same one that you authenticated against.

You will be prompted to sign in to authentic and this should be an experience that most folks would be familiar with. Note, each time you run the script, you need to authenticate - and this is irrespective of say if you are already logged into Teams of Office 365.

Authenticating user against Office 365

Assuming you have authenticated successfully, you should see an output like the one shown below; and a csv file in the path you configured will be created. This file will always be overwritten - without any prompts (of course this is assuming no other process is open that has a lock on that file).

#Set these variables, to what makes sense in your situation. The email here is the one that is the one connected to your teams account.
$exportLocation = "C:\temp\team-details.csv"
$emailAddress = "your-email@shouldbeputhere.com"

#Authenticate against teams
Connect-MicrosoftTeams

#Patience
Write-Host -ForegroundColor Blue "Successfully connected to Teams"
Write-Host -ForegroundColor Blue "Getting all team details for user: $($emailAddress)"
Write-Host -ForegroundColor Blue "Please be patient, if there are a lot of teams, this can take a while..."

# Get all of the team Groups IDs
# $GetUsersTeams = (Get-Team).GroupID
$GetUsersTeams = Get-Team -User $emailAddress

$Report = @()

# Will hold a basic count of user types and teams
$unavailableTeamCount = 0

# Loop through all teams that the user belongs to
$currentIndex = 1

ForEach($thisTeam in $GetUsersTeams) {
	# Show some output to the user
    Write-Progress -Id 0 -Activity "Building report from Microsoft Teams" -Status "$currentIndex of $($GetUsersTeams.Count)" -PercentComplete (($currentIndex / $GetUsersTeams.Count) * 100)
	
    # Attempt to get team details, throw error message if no access
    try {
        # Get team members
        #$users = Get-TeamUser -GroupId $thisTeam.groupID

		# Create an object to hold all values
        $teamReportObject = New-Object PSObject -Property @{
                GroupID = $thisTeam.GroupID
				TeamName = $thisTeam.DisplayName
                Description = $thisTeam.Description
                Archived = $thisTeam.Archived
                Visibility = $thisTeam.Visibility
				eMail = $thisTeam.MailNickName
            }

            # Add to the report
            $Report += $teamReportObject
       
    } catch [Microsoft.TeamsCmdlets.PowerShell.Custom.ErrorHandling.ApiException] {
        Write-Host -ForegroundColor Yellow "No access to $($team.DisplayName) team, cannot generate report"
        $unavailableTeamCount++
    }
	
	$currentIndex++
}
Write-Progress -Id 0 -Activity " " -Status " " -Completed

# Disconnect from teams
Disconnect-MicrosoftTeams

# Provide some nice output
Write-Host -ForegroundColor Green "============================================================"
Write-Host -ForegroundColor Green "                Microsoft Teams User Report                 "
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green "  Count of All Teams - $($GetUsersTeams.Count)                "
Write-Host -ForegroundColor Green "  Count of Inaccesible Teams - $($unavailableTeamCount)         "
Write-Host -ForegroundColor Green ""

$Report | Export-CSV $exportLocation -NoTypeInformation -Force
Write-Host -ForegroundColor Blue "Exported report to $($exportLocation)"

Getting Team members using PowerShell

Now that you have the Team ID you are interested, you can run the other PowerShell script ( also available on GitHub ) to get a list of all the users in a specific team. Like the previous script, you would need set a couple of variables in the script:

The Team ID for the team you are interested in.
Path for the csv file with details to be saved.

Once you have authenticated and ran the script, the output will look like the one shown below. You get a summary of the team details, and details of the Teams users and their type (owner, member, or guest). And just like earlier, the file will be overwritten without a prompt, assuming no locks on it.

Members of a Microsoft Team

#Global variables to set:
#path of the file where to export
#specific ID of the team that you want the users for. 
$exportLocation = "C:\temp\RL-decision-AI-export.csv"
$TEAM_ID = "f3f9ad1f-beea-4026-9b86-dd3788404999"
            
$Report = @()

# counters
$ownerCount = 0
$memberCount = 0
$guestCount = 0

#connect to teams
Connect-MicrosoftTeams

$team = Get-Team -GroupId $TEAM_ID

#Patience, supposed to be a virtue
Write-Host -ForegroundColor Blue "Successfully connected to Team: $($team.DisplayName)"
Write-Host -ForegroundColor Blue "Getting all users in the team"
Write-Host -ForegroundColor Blue "Please be patient, if there are a lot of users, this can take a while..."

# Attempt to get team users, throw error message if no access
try {
	# Get team members
	$users = Get-TeamUser -GroupId $team.groupID

	# Loop through and get all the users
	$currentIndex = 1
	
	# foreach user create a line in the report
	ForEach($user in $users) {
		# Show some output to the user
		Write-Progress -Id 0 -Activity "Generating user report from Teams" -Status "$currentIndex of $($users.Count)" -PercentComplete (($currentIndex / $users.Count) * 100)
	
		# Maintain a count of user types
		switch($user.Role) {
			"owner" { $ownerCount++ }
			"member" { $memberCount++ }
			"guest" { $guestCount++ }
		}

		# Create an object to hold all values
		$ReportObject = New-Object PSObject -Property @{
			User = $user.Name
			Email = $user.User
			Role = $user.Role
		}

		# Add to the report
		$Report += $ReportObject
		
		$currentIndex++
	}
} 
catch [Microsoft.TeamsCmdlets.PowerShell.Custom.ErrorHandling.ApiException] {
	Write-Host -ForegroundColor Yellow "No access to $($team.DisplayName) team, cannot generate report"
}

#Complete progress
Write-Progress -Id 0 -Activity " " -Status " " -Completed

# Disconnect from teams
Disconnect-MicrosoftTeams

# Write out details for the user
Write-Host -ForegroundColor Green "============================================================"
Write-Host -ForegroundColor Green "                Microsoft Teams User Report                 "
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green "Team Details:"
Write-Host -ForegroundColor Green "Name: $($team.DisplayName)"
Write-Host -ForegroundColor Green "Description: $($team.Description)"
Write-Host -ForegroundColor Green "Mail Nickname: $($team.MailNickName)"
Write-Host -ForegroundColor Green "Archived: $($team.Archived)"
Write-Host -ForegroundColor Green "Visiblity: $($team.Visibility)"
Write-Host -ForegroundColor Green ""
Write-Host -ForegroundColor Green "Team User Details:"
Write-Host -ForegroundColor Green "Owners - $($ownerCount)"
Write-Host -ForegroundColor Green "Members - $($memberCount)"
Write-Host -ForegroundColor Green "Guests - $($guestCount)"
Write-Host -ForegroundColor Green "============================================================"

$Report | Export-CSV $exportLocation -NoTypeInformation -Force
Write-Host -ForegroundColor Blue "Exported report to $($exportLocation)"

Of course, programming against the API is always more powerful, but sometimes quick and easy is what is needed. :)

Amit Bahree's (useless?) insight!

🎉Announcing My New Book: Generative AI in Action📚

Special Launch Offer 🌟

Why You Should Read Generative AI in Action ✨

Comprehensive Coverage

Real-World Examples

Hands-On Techniques

Ethical AI and Best Practices

Expert Insights

Advanced Techniques Covered in the Book 🔥

Explore the GitHub Repository 💾

Get Your Copy Today! 🛒

Building a microkernel in Rust (Part 1): Foundations, booting on bare metal

TL;DR

1. What “bare metal” actually means

2. What’s an ELF?

3. The QEMU virt machine

4. Setup

5. Build and run

6. The boot sequence

7. ARM assembly primer

7.1 Registers

7.2 System registers

7.3 Common instructions

7.4 Local labels and directives

8. Walking through boot.S

8.1 Stack setup (lines 5-6)

8.2 BSS zeroing (lines 8-16)

8.3 Checking the exception level (lines 19-23)

8.4 Exception levels explained

8.5 The EL2 to EL1 drop (lines 26-40)

8.6 Vector table installation (lines 43-47)

8.7 FP/ASIMD enablement: why Rust needs NEON (lines 49-55)

8.8 Jumping to Rust (lines 57-60)

9. The Rust entry point

9.1 global_asm! and including boot.S

9.2 #[unsafe(no_mangle)] and extern "C"

9.3 The feature gates

9.4 The panic handler

10. PL011 UART: talking to hardware via MMIO

10.1 The UartLogger implementation

10.2 Sending a character

10.3 Sending a string

10.4 The Logger trait implementation

11. The platform-agnostic kernel

11.1 The Logger trait

11.2 The kernel entry

11.3 The halt function

12. Linker scripts

13. Hands-on exercise

14. Brief appendix: how would this differ on other platforms?

15. References

Building a microkernel in Rust (Part 0): Why build an OS from scratch?

Why this, why now?

Part 0? Really?

What happens when there’s nothing underneath you?

TL;DR

1. A quick primer: AArch64, QEMU, and the “virt” machine

2. What we’re building

3. Why this series?

4. Why Rust? (and not C or assembly)

4.1 The problem with C

4.2 The problem with assembly only

4.3 What Rust gives you

5. Getting started

5.1 Install Rust (nightly)

5.2 Install QEMU

5.3 Clone and build

5.4 Run it

5.5 Try the other demos

5.6 Docker (no local setup)

6. Series roadmap

7. The big picture: what makes an operating system?

7.1 Complete OS architecture

7.2 What we build (the foundation)

7.3 What we don’t build

8. Repository structure

9. References and acknowledgments

Building LLMs from Scratch - Part 4: Evaluation & Deployment

1. The Evaluation Challenge: Measuring What Matters for Historical Language Models

9.1 `global_asm!` and including boot.S

9.2 `#[unsafe(no_mangle)]` and `extern "C"`

2.2.1 The `init` method