Special Launch Offer 🌟

As a thank-you to my early supporters, I’m offering an exclusive discount. Use the code pbbahree at checkout to receive 45% off your purchase of Generative AI in Action in all formats (valid through Sept. 30, 2024)!

Get your discounted copy 👉 » here « .

Why You Should Read Generative AI in Action ✨

Comprehensive Coverage

This book offers an in-depth introduction to Generative AI, covering foundation models, large language models (#LLMs), small language models (#SLMs), and practical applications. From the basics to advanced topics like prompt engineering, Generative AI in Action provides everything you need to start building and scaling AI solutions. Whether you’re a beginner or a seasoned professional, you’ll find valuable insights to accelerate your AI journey.

Real-World Examples

Discover how enterprises across industries are leveraging Generative AI to innovate and solve complex problems. Whether it’s improving customer engagement or optimizing operations, the practical examples provided can be directly applied to your projects for immediate impact.

Hands-On Techniques

Dive into step-by-step guides and hands-on examples for integrating AI models into your workflows. Learn techniques such as:

• Prompt Engineering: Craft effective prompts to unlock the full potential of AI models like GPT-4.
• Retrieval-Augmented Generation (RAG): Enhance your AI models with real-time data for improved accuracy.
• Model Adaptation: Fine-tune AI models to meet your organization’s specific needs.

Ethical AI and Best Practices

As AI becomes more critical in decision-making, understanding its ethical implications is crucial. Generative AI in Action covers topics like privacy, security, and bias mitigation—ensuring your AI deployments are fair, transparent, and aligned with your organizational values.

Expert Insights

Drawing from my experience helping build the Azure AI platform, I share insider knowledge on leveraging the latest advancements in AI for your projects. This book provides you with the tools to make the most of cutting-edge technologies like large language models (LLMs) and small language models (SLMs).

Advanced Techniques Covered in the Book 🔥

Beyond the basics, Generative AI in Action delves into advanced techniques essential for mastering Generative AI in modern enterprise environments:

• Prompt Engineering: Strategies like zero-shot, few-shot, and many-shot learning, along with chain-of-thought reasoning, to optimize AI outputs.
• Retrieval-Augmented Generation (RAG): Combine retrieval-based methods with generative models for real-time, relevant data integration.
• Model Adaptation and Fine-Tuning: Customize generative models to specific tasks using techniques such as low-rank adaptation and reinforcement learning from human feedback (RLHF).
• Chatting with Your Data: Build AI-powered chat systems that interact with enterprise data using vector databases and retrieval techniques.
• Scaling and Production Deployment: Strategies for scaling AI solutions while ensuring performance, reliability, and compliance with enterprise standards.
• Evaluations and Benchmarks: Learn to evaluate and benchmark AI models using traditional metrics and cutting-edge frameworks.

Explore the GitHub Repository 💾

For those eager to dive into the code, the book has a companion GitHub repository filled with examples and projects to get you started. Check it out at bit.ly/GenAIBook . Explore the code, experiment, and start building your AI-powered solutions today.

Get Your Copy Today! 🛒

Don’t miss this opportunity to lead the AI revolution within your organization. Order your copy of Generative AI in Action and use the code pbbahree to receive 45% off (valid through Sept. 30, 2024). Transform your organization’s AI capabilities today!

With gratitude 💚

Amit Bahree.

In the world of AI, there’s a thrill,
With “Generative AI in Action,” you’ll skill.
From prompts to fine-tuning,
Your projects are blooming,
Grab your copy, and master the drill!

This post is a technical deep dive into that question. Instead of presenting a generic leaderboard, I focus on the details that usually matter in real deployments: throughput under fixed traffic shapes, latency behavior, scaling across 8x and 16x H200 shapes, and the caveats that only show up when you try to run these models end to end.

It also grew out of a public r/LocalLLaMA thread asking what people most wanted to see tested on this hardware. The model set in this post is therefore not just a random grab bag. It reflects the cluster bring-up requests that came up most often, plus the operational work required to turn those requests into comparable benchmark results.

TL;DR

• Llama 4 Scout and MiniMax M2.1 were the strongest overall performers in this benchmark set.
• In several cases, 8x H200 was a better serving shape than 16x H200 for the same workload mix.
• DeepSeek V4 Flash was healthy and interesting, especially on long-context runs, while DeepSeek V4 Pro only produced fallback-shape numbers.
• A fixed benchmark matrix plus Langfuse validation mattered almost as much as the raw throughput numbers.

1. Hardware and Profiles

The benchmark environment is a 16 x H200 cluster across two nodes, but the machine shape is important because this is not a generic collection of GPUs. The cluster is built on Dell PowerEdge XE9680L servers, each with 8 x NVIDIA H200, dual Intel Xeon Platinum 8570 CPUs, and 2.0 TiB of system RAM. Each node also carries 8 x 3.84 TB Dell U.2 NVMe drives plus a Dell BOSS-N1 boot device.

On the data-plane side, each node exposes 8 active 400G ConnectX-7 InfiniBand links, which is 3.2 Tb/s of raw InfiniBand link rate per node, along with 2 active 200G BlueField-3 / ConnectX-7 Ethernet links. So at the cluster level, this is roughly 2.30 TB of aggregate HBM, 4.0 TiB of host RAM, and a genuinely fast multi-rail fabric rather than a generic dual-server setup.

For this study, the primary serving runtime is vLLM, and the benchmark profiles are intentionally fixed so the comparisons do not drift from one model to the next. Every model was measured against the same three workload profiles:

• 1024 in / 256 out / concurrency 1
• 1024 in / 256 out / concurrency 16
• 8192 in / 256 out / concurrency 4

For anyone new to this notation:

• 1024 in means each request starts with a prompt of about 1024 input tokens.
• 256 out means each request is allowed to generate up to about 256 output tokens.
• concurrency N means how many requests are in flight at the same time.

What each target represents in practice:

Throughout the post, results are labeled with shorthand profile names that encode the same information:

Where a model was tested on multiple hardware shapes, the suffix -16x (or -8x) is appended to the profile name to indicate which shape that row covers, e.g. latency-1024x256-c1-16x means the latency profile run on the 16x H200 shape.

Later tables also use runtime topology labels such as TP, PP, DP, and EP. These describe how the model was distributed across GPUs during serving:

• TP = tensor parallelism, meaning tensor operations are split across multiple GPUs
• PP = pipeline parallelism, meaning different layers or blocks are split into sequential pipeline stages
• DP = data parallelism, meaning multiple replicas process different requests in parallel
• EP = expert parallelism, meaning MoE experts are distributed across GPUs

So TP=8, PP=2 means the model was served with 8-way tensor parallelism and 2 pipeline stages, which typically implies a 16-GPU deployment shape for that run.

The results tables in each model section report three metrics alongside the profile label:

• Output tok/s — aggregate output throughput across all concurrent requests. This is the headline number: how many tokens per second the serving stack is generating in total under that workload shape.
• TTFT (ms) — time to first token. How long from when the request was sent until the first output token arrived back. Lower is better. TTFT reflects prefill time, KV cache allocation, and scheduling overhead combined, and it is the number that determines how responsive the model feels to a user waiting for a reply.
• TPOT (ms) — time per output token. The average time between consecutive output tokens after the first one. Lower is better. TPOT reflects decode speed and is what determines whether a streaming response feels smooth or choppy once it starts.

The reason all three matter together: a model can post high aggregate tok/s at c16 but still have a painful user experience if TTFT is high, because every user waits that long before seeing any output. Conversely, a model with modest tok/s but low TPOT can feel snappier than the numbers suggest.

The model list below is presented in no particular order. These are the large open models that were both available and runnable during the benchmark window, spanning a mix of architectures, training approaches, and quantization formats. It is not an exhaustive survey of the open-model ecosystem, but it is a practical cross-section of the models people are actively evaluating and that I could run with enough stability to generate comparable results.

• Qwen/Qwen3-235B-A22B-Instruct-2507
• moonshotai/Kimi-K2.6
• deepseek-ai/DeepSeek-V4-Flash
• deepseek-ai/DeepSeek-V4-Pro
• meta-llama/Llama-4-Scout-17B-16E-Instruct
• zai-org/GLM-5.1-FP8
• MiniMaxAI/MiniMax-M2.1
• mistralai/Mistral-Large-3-675B-Instruct-2512

The goal was straightforward: one fixed benchmark suite, one cluster, several large open models, and enough implementation detail that the results are useful beyond this specific environment.

2. Method

All of the vLLM runs use the same serving profiles and the same result format. Each saved result includes the exact model name, prompt length, output length, concurrency, request rate, hardware label, engine, and timestamp. That consistency matters because a standalone “tokens per second” number stops being useful the moment people start changing prompt length, output length, or concurrency and still try to compare the outcome directly.

Not every model ran on the same vLLM build, and that was deliberate rather than inconsistent. The default lane was the stable v0.19.1 release. The exceptions were models where the generic stable image was either not the supported path, not feature-complete enough for that model’s runtime requirements, or not what the model vendor’s integration expected:

• GLM-5.1-FP8 used v0.19.1.dev1 — the FP8 and multi-node bring-up path for GLM required a newer-than-stable build to work cleanly.
• MiniMax M2.1 used v0.19.1rc1.dev203 via the vllm/vllm-openai:minimax27 image — MiniMax ships its own official runtime lane with MiniMax-specific parser and expert-parallel behavior; the stock image is not the right lane for this model.
• DeepSeek V4 Flash and V4 Pro both used vllm/vllm-openai:deepseekv4-cu130 — DeepSeek V4 requires the dedicated CUDA image; the generic image is not the supported serving path.

The short version: deviations happened only when the model or runtime integration required a special lane. Where the engine lane affected the result, it is noted in the model-specific metadata block.

2.1 Langfuse

Langfuse is the observability layer I used to verify that a run was genuinely healthy from end to end. In other words, the bar was not merely “the server came up” but “the model served requests correctly, produced output, and emitted traces that matched the saved benchmark artifacts.”

Each run emitted traces tagged with model identity, profile (c1, c16, c4), and hardware shape. Those traces served as a cross-check against the saved result files. Only runs with both complete result files and successful traces were counted as comparable in the summary tables.

3. Quick Comparison (Throughput-First)

If you want the numbers before the per-model context, this is the place to start. The tables and charts below cover all comparable runs side by side.

3.1 Top 3 by Output Throughput (tok/s) per Profile

The table below answers a simple question: which model led each workload profile? Each profile represents a different traffic shape, so the rankings should be interpreted within a profile rather than across profiles. Higher output tok/s means more generated tokens per second across all in-flight requests.

3.2 Completed Runs Side-by-Side (Output Throughput)

Where the Top 3 table highlights the leaders, this table shows the full completed field so every model can be compared across the same three benchmark profiles at once. The values here are output throughput (tok/s), and the Notes column captures important caveats such as fallback topologies.

The three charts below show output throughput (tok/s) for all models, split by workload profile so each chart can use an appropriate scale.

What each chart presents:

• c1 (latency-1024x256-c1): single-user responsiveness at concurrency 1
• c16 (serve-1024x256-c16): loaded serving throughput at concurrency 16
• c4 (longctx-8192x256-c4): long-context behavior with 8192-token prompts at concurrency 4

Single-user latency (c1) - output tok/s, all models

xychart-beta
  title "latency-1024x256-c1: output tok/s (higher = better)"
  x-axis ["MiniMax M2.1", "Llama 4 Scout", "Mistral L3", "GLM-5.1-FP8", "DS Flash", "Kimi K2.6", "Qwen 235B", "DS Pro*"]
  y-axis "output tok/s" 0 --> 160
  bar [145.94, 126.70, 93.07, 88.66, 69.96, 64.38, 56.46, 6.43]

Loaded serving throughput (c16) - output tok/s, all models

xychart-beta
  title "serve-1024x256-c16: output tok/s (higher = better)"
  x-axis ["Llama 4 Scout", "MiniMax M2.1", "Qwen 235B", "Mistral L3", "DS Flash", "GLM-5.1-FP8", "Kimi K2.6", "DS Pro*"]
  y-axis "output tok/s" 0 --> 1500
  bar [1378.30, 1358.19, 643.56, 554.50, 543.13, 509.93, 470.52, 90.10]

Long-context (c4) - output tok/s, all models

xychart-beta
  title "longctx-8192x256-c4: output tok/s (higher = better)"
  x-axis ["Llama 4 Scout", "MiniMax M2.1", "DS Flash", "Mistral L3", "Kimi K2.6", "Qwen 235B", "GLM-5.1-FP8", "DS Pro*"]
  y-axis "output tok/s" 0 --> 450
  bar [404.41, 379.29, 220.59, 199.59, 179.45, 170.47, 163.37, 23.27]

* DS Pro ran on a fallback topology (TP=8, --enforce-eager) - see the DeepSeek V4 Pro section for context.

4. Model Status Snapshot

Most models completed cleanly, but a few required fallback topologies or other caveats that materially change how their results should be interpreted. DeepSeek V4 Pro is the main exception: its published numbers come from a fallback deployment shape rather than the intended lane.

5. Llama 4 Official Scout

Llama 4 Scout ended up being one of the clearest scaling tests in the whole post because it ran cleanly on both shapes and produced directly comparable results. The question here is simple: once the benchmark matrix is held constant, does moving from a single-node 8x H200 lane to a two-node 16x H200 deployment actually improve the serving outcome?

Metadata

• model: meta-llama/Llama-4-Scout-17B-16E-Instruct
• engine: vLLM v0.19.1
• status: completed (8x and 16x lanes)

As a reminder, TP=8, PP=2 means the model was served with 8-way tensor parallelism and 2 pipeline stages, which typically implies a 16-GPU deployment shape for that run. That makes this a clean apples-to-apples scaling comparison: the 16x H200 rows show the intended two-node path for this model, the 8x H200 rows show the single-node alternative, and both lanes produced complete benchmark artifacts with matching Langfuse traces.

The result is clear — scaling from 8 to 16 GPUs was not beneficial for Llama 4 Scout on this workload mix:

• the single-request c1 profile got worse
• the c16 throughput profile also got worse
• the long-context profile only improved slightly on TTFT, while overall throughput and per-token decode still got worse

In practical terms, official Scout is a strong and fully benchmarkable 8x H200 result, but not a good candidate for this 16x H200 serving shape. The separate unsloth Scout lane never produced stable comparable runs, so it is excluded from the cross-model comparison.

6. Kimi K2.6

Kimi K2.6 completed on 16x H200 using the standard vLLM path. Unlike some of the other models in this post, there is no alternate shape comparison here, so the table below should be read as the canonical Kimi reference point.

Metadata

• model: moonshotai/Kimi-K2.6
• engine: vLLM v0.19.1
• shape: 16x H200
• status: completed

The main takeaway is that the final Kimi run was fully benchmarkable end to end — not just running without crashing, but producing clean, complete artifacts at every profile. These are the numbers used in the cross-model comparison.

7. Qwen 235B

Qwen 235B completed on 16x H200 using the standard vLLM path. The values below are the Qwen reference point used throughout the comparison tables and charts.

Metadata

• model: Qwen/Qwen3-235B-A22B-Instruct-2507
• engine: vLLM v0.19.1
• shape: 16x H200
• status: completed

Qwen produced stable, usable benchmark numbers across all three profiles, but did not challenge the top performers in this fixed workload mix.

8. DeepSeek V4 Flash

DeepSeek V4 Flash is interesting here not just for the final numbers, but for how the 4x and 8x shapes compare. Flash required the dedicated vllm/vllm-openai:deepseekv4-cu130 image plus one configuration fix: removing --attention_config.use_fp4_indexer_cache=True, which is Blackwell-only and fails on H200 after weight load.

I benchmarked Flash on both 4x H200 and 8x H200 to test local scaling behavior. All runs produced Langfuse traces confirming end-to-end health, and the cross-model comparison table uses the 8x row as the canonical Flash entry because that is the shape that best represents the larger benchmark set.

Metadata

• model: deepseek-ai/DeepSeek-V4-Flash
• engine: vLLM deepseekv4-cu130
• status: completed (4x and 8x lanes)

The scaling result is worth noting: 8x H200 did not uniformly help.

• The c1 single-request profile got slightly worse - the overhead of spreading across more GPUs outweighed any memory bandwidth benefit at that concurrency level.
• The c16 throughput profile was roughly flat.
• The c4 long-context profile improved meaningfully - the larger working set genuinely benefits from the extra capacity.

The practical conclusion is that 4x H200 is the better efficiency shape for ordinary Flash serving, while 8x starts to make sense primarily when prompt lengths get heavier.

9. DeepSeek V4 Pro

DeepSeek V4 Pro comes with an important caveat: the intended DP+EP MoE lane did not stabilize, so the published numbers come from a fallback shape. The intended deployment (DP + EP on 16x H200 using deepseekv4-cu130) repeatedly failed in the fused MoE router with expected scalar type Long but found Int, reproduced on both 16x multi-node and 8x single-node attempts, including runs with --enforce-eager. I filed the upstream issue at vllm-project/vllm#40862 .

The lane that did work was single-node TP=8 with --enforce-eager. The table below should therefore be read as a valid fallback benchmark, not as representative of the intended DP+EP deployment target.

Metadata

• model: deepseek-ai/DeepSeek-V4-Pro
• engine: vLLM deepseekv4-cu130
• shape: 8x H200 (fallback: TP=8, --enforce-eager)
• status: completed on fallback shape - intended DP+EP lane blocked

The numbers reflect the fallback shape constraints directly: per-token decode is slow across all profiles, TTFT is extremely high under load, and aggregate throughput is far below what the intended DP+EP lane should have produced. These are still real and reproducible results, but they answer a different question than the one the original benchmark plan set out to answer.

10. GLM 5.1 FP8

GLM-5.1-FP8 was tested on both 8x and 16x H200 under the same benchmark profiles. Because GLM-5.1 does not support pipeline parallelism, the two-node lane used TP=8 + DP=2. After a long initial model weight loading stage, both shapes ran cleanly and produced comparable results.

Metadata

• model: zai-org/GLM-5.1-FP8
• engine: vLLM v0.19.1.dev1
• status: completed (8x and 16x lanes)

The result is unambiguous: scaling to 16x H200 made every profile worse. Throughput dropped on all three profiles, TTFT got substantially worse, and TPOT degraded as well. For this benchmark mix, 8x H200 is the right serving shape for this model.

11. MiniMax M2.1

MiniMax M2.1 was tested on both 8x and 16x H200. As with the other dual-shape models, the key question is whether scaling improved the operational result enough to justify the additional complexity. Bring-up was straightforward on the official minimax27 runtime (TP=8 with expert parallelism), and the 8x lane was strong enough to justify a full 16x comparison.

Metadata

• model: MiniMaxAI/MiniMax-M2.1
• engine: vLLM v0.19.1rc1.dev203 (MiniMax minimax27 runtime image)
• status: completed (8x and 16x lanes)

The 16x H200 pass was worse across all three profiles. Like Llama 4 Scout and GLM-5.1-FP8, MiniMax M2.1 does not benefit from the two-node shape on this workload mix. The important nuance is that this is not a weak model result; it is a strong model result that simply performs better on the smaller shape.

12. Mistral Large 3

Mistral Large 3 was benchmarked on the single-node 8x H200 shape. It should be read as a stable single-shape reference lane rather than a scaling study: by this point in the benchmark cycle, enough models had already shown weak 16x returns for this workload mix that a two-node pass was intentionally skipped. The 8x run came up cleanly and produced complete Langfuse traces.

Metadata

• model: mistralai/Mistral-Large-3-675B-Instruct-2512
• engine: vLLM v0.19.1
• shape: 8x H200
• status: completed

This is a solid and reproducible result, but it is not a standout against the top performers in this set. c16 TTFT of 1193 ms and long-context TTFT of 1226 ms are notably higher than MiniMax M2.1 and Llama 4 Scout, and throughput stays in the middle of the field across all three profiles.

13. Benchmark Pipeline

The benchmark pipeline underneath these results is an important part of why the comparison holds together. In this setup, models are served in Docker and benchmarked through the vLLM API, with each run saving both performance metrics and metadata so results stay comparable across models, versions, and hardware shapes.

At a high level, the flow is:

1. Choose model and served model name
2. Run the standard 3-profile matrix
3. Monitor live engine throughput while runs execute
4. Save JSON results with run metadata
5. Cross-check with Langfuse traces
6. Aggregate into comparison tables and charts

13.1 Core Pipeline Components

Rather than running ad-hoc vllm bench serve commands by hand each time, the pipeline is split into four distinct roles. Each one is lightweight on its own, but together they are what makes the results repeatable and comparable across model runs, hardware shapes, and benchmark sessions.

• Benchmark runner - the vllm bench serve command pointed at a live vLLM endpoint. It handles prompt dispatch, concurrency, output token collection, and raw result saving. Everything else depends on this working cleanly.
• Matrix runner - a thin shell wrapper that calls the benchmark runner three times in sequence with fixed parameters for each profile (c1, c16, c4). The only thing that changes between model runs is the model identifier and served model name; input length, output length, prompt count, and concurrency are locked.
• Metadata enricher - a short Python script that reads the saved JSON from each run and adds structured run metadata before archiving. Without this, a result file from three weeks ago is just a pile of numbers with no clear link back to the hardware shape, engine version, or traffic profile that produced it.
• Live monitor - a docker logs tail filtered to the vLLM engine stats lines. It does not produce any saved artifact, but it is useful for spotting problems while a run is still in flight: flat generation throughput, GPU KV cache filling up earlier than expected, or a long tail of slow requests that suggests the serving shape is under-provisioned for the workload.

13.2 Single-Profile Benchmark Snippet

Before running the full three-profile matrix against a new model, it is worth running a single profile first to confirm the serving lane is actually healthy. A new model or container image can load without error and still fail silently on the first real request, produce garbled output, or hit a memory issue that only shows up under inference load. A single c1 run is cheap and fast, and it catches the obvious problems before committing to a longer matrix run that might need to be discarded anyway.

vllm bench serve \
  --host <api-host> \
  --port 8000 \
  --endpoint /v1/completions \
  --model <model-id> \
  --served-model-name <served-model-id> \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --num-prompts 20 \
  --max-concurrency 1 \
  --request-rate inf \
  --temperature 0 \
  --save-result \
  --result-filename ./benchmarks/latency-1024x256-c1.json

13.3 Standard Matrix Snippet

Once the single-profile sanity check passes, the matrix runner runs all three profiles back to back. The key discipline here is that nothing changes between profiles except the input length and concurrency level - the model, endpoint, serving stack, and result format all stay identical. That consistency is what makes the results comparable later, both within a single model and across all models in the post. Without it, you end up with numbers that look like a comparison but are actually measuring different things.

#!/usr/bin/env bash
set -euo pipefail

MODEL="<model-id>"
SERVED="<served-model-id>"

run_profile () {
  local tag="$1" in_len="$2" out_len="$3" prompts="$4" conc="$5"
  vllm bench serve \
    --host <api-host> \
    --port 8000 \
    --endpoint /v1/completions \
    --model "$MODEL" \
    --served-model-name "$SERVED" \
    --dataset-name random \
    --random-input-len "$in_len" \
    --random-output-len "$out_len" \
    --num-prompts "$prompts" \
    --max-concurrency "$conc" \
    --request-rate inf \
    --temperature 0 \
    --save-result \
    --result-filename "./benchmarks/${tag}.json"
}

run_profile latency-1024x256-c1   1024 256 20  1
run_profile serve-1024x256-c16    1024 256 200 16
run_profile longctx-8192x256-c4   8192 256 50  4

13.4 Metadata Enrichment Snippet

The raw JSON that vllm bench serve saves contains throughput, latency, and token count data, but nothing about the context that produced it. No hardware shape, no engine version, no timestamp, no indication of which traffic profile was running. That is fine for a one-off experiment, but it becomes a serious problem the moment you want to compare results across models or revisit a run a few weeks later.

The enrichment step adds a structured run_metadata block to each result file immediately after the run completes. With this in place, every result file is self-describing: it contains both the benchmark output and the exact context needed to understand and reproduce it. It also makes aggregating results into summary tables and charts much more reliable, since the grouping and filtering logic can operate on explicit metadata fields rather than trying to infer context from filenames.

import json
from datetime import datetime, timezone

path = "./benchmarks/serve-1024x256-c16.json"

with open(path, "r", encoding="utf-8") as f:
    data = json.load(f)

if not isinstance(data, dict):
    data = {"benchmark_result": data}

data["run_metadata"] = {
    "model": "<model-id>",
    "served_model_name": "<served-model-id>",
    "benchmark_type": "serve",
    "dataset_name": "random",
    "input_len": 1024,
    "output_len": 256,
    "num_prompts": 200,
    "max_concurrency": 16,
    "request_rate": "inf",
    "endpoint": "/v1/completions",
    "hardware": "16x H200",
    "engine": "vLLM",
    "cluster_label": "<cluster-label>",
    "result_tag": "serve-1024x256-c16",
    "saved_at_utc": datetime.now(timezone.utc).isoformat(),
}

with open(path, "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, sort_keys=True)
    f.write("\n")

13.5 Live Throughput Monitor Snippet

The benchmark runner saves results at the end of a run, which means you only find out something went wrong after it is already over. For longer matrix runs - especially on large models with slower TTFT - that is a lot of time to spend waiting on a result that turns out to be invalid. The live monitor is a simple docker logs tail filtered to the vLLM engine stat lines, so you can watch generation throughput, KV cache utilization, and request latency in real time while the run is executing. It does not produce any saved artifact, but it lets you catch a stuck or unhealthy run early and abort rather than waiting for it to finish.

docker logs -f <vllm-container> 2>&1 | \
  grep --line-buffered \
  'Avg prompt throughput\|Avg generation throughput\|GPU KV cache usage\|Prefix cache hit rate\|POST /v1/'

13.6 Validation and Extension Rules

The three-profile matrix only stays useful as a comparison baseline if it is actually treated as a baseline. The temptation when benchmarking multiple models is to make small adjustments for each one - a different prompt count here, a slightly different concurrency there, a custom profile added for one model but not others. Each individual change seems reasonable in isolation, but the cumulative effect is a result set where the numbers cannot be compared cleanly because they were not produced under the same conditions.

The rules below are the discipline that keeps that from happening. The standard matrix is the contract. Custom profiles, extra concurrency levels, or hardware variants are extensions that go alongside it, not replacements for it.

1. Always run the standard matrix (c1, c16, c4) first
2. Only swap model identifiers when comparing model families
3. Keep hardware/engine labels explicit in metadata
4. Treat runs as comparable only when both JSON artifacts and Langfuse traces are complete
5. Add custom profiles as extra runs, not replacements

14. Summary

The short version is that benchmarkability mattered almost as much as raw speed. Several models loaded and ran cleanly enough to produce useful, repeatable numbers, but the most important result was how often the simpler 8x H200 shape outperformed the larger 16x H200 deployment on this workload mix.

For practical serving, Llama 4 Scout and MiniMax M2.1 were the strongest overall performers in this set, especially once latency, loaded throughput, and long-context behavior were considered together. Mistral Large 3, GLM-5.1-FP8, Qwen 235B, and Kimi K2.6 all produced usable reference numbers, but they did not challenge the top tier in this fixed profile mix.

The other major takeaway is that topology caveats matter. DeepSeek V4 Flash was healthy and interesting, especially on long-context workloads, while DeepSeek V4 Pro only produced fallback-shape results because the intended DP+EP path did not stabilize. That is exactly why a fixed methodology, saved artifacts, and trace validation are worth the extra effort: they let you separate actual model behavior from deployment-path noise.

If there is one conclusion from the full run, it is this: large-model benchmarking is only useful when the numbers stay attached to the exact serving shape, traffic profile, and validation path that produced them. Once that discipline is in place, even a messy benchmark diary becomes a practical decision document.

The original community request thread also surfaced a separate follow-up idea that deserves its own methodology rather than being mixed into this write-up: controlled comparison work around quantization, KV-cache variants, and task-level coding benchmarks where the model, runtime, and evaluation target are all held fixed.

GitHub Repository: bahree/nanoclaw - full source code

NanoClaw is a headless AI assistant running on my personal server. It processes messages from WhatsApp, Telegram, and Slack, runs scheduled tasks, and manages conversations with Claude agents in isolated containers. It’s been incredibly useful, but it had one major pain point: no visibility into what it was doing or why. If something went wrong - a message didn’t get a reply, a task didn’t run - the only way to debug was to SSH into the server, tail logs, and piece together what happened. Not ideal when you’re on the go and your assistant just… stops responding.

I use NanoClaw as my personal AI assistant - it handles everything from answering questions, to tracking flights, to giving me a daily morning briefing with F1 standings and weather. It is my agent and allows for me to check on the kind of thing I want to check on from my phone, not have to SSH into a server for.

The problem with a headless service that runs 24/7 is that you can’t see what it’s doing. When a message doesn’t get a reply, or a scheduled task doesn’t fire, the debugging workflow is: SSH into the server, tail the pino logs, grep for timestamps, piece together what happened. Not great when you’re out and about and your assistant just… stops responding.

I wanted three things:

• A way to ask the system “what are you doing right now?” from the same WhatsApp chat I use to talk to it.
• A way to manage scheduled tasks without SSH.
• And a way to ask “why did you do that?” after the fact, with full traceability from triggering event to outbound action.

This post covers all three and how we built them in NanoClaw - nine features (~1100 lines of new TypeScript, two new modules, and three new SQLite tables).

TL;DR

Given the existing architecture of NanoClaw, we added a suite of observability features that are all accessible from the main WhatsApp group. No SSH, no separate dashboards, no external logging services - just commands you can type in the chat to see what’s going on and manage the system.

I added three groups of features to NanoClaw:

1. Real-time visibility - /status shows uptime, memory, active containers, channels, groups, and task summaries. /status tasks shows the full task list with schedules, next run times, and IDs.
2. Operational control - /task pause|resume|delete <id> manages scheduled tasks directly from the chat. No SSH, no restarts.
3. Event tracing and debugging - three SQLite tables that trace every action back to its triggering event. Pipeline is instrumented at message ingress, agent output, scheduled tasks, and IPC. Query with /debug last 10, /debug why, /debug event <id>, and /debug report. Auto-prunes after 3 days.

All operated entirely from the messaging channel.

1. The architecture (quick context)

Before diving in, here’s how NanoClaw works at a high level. Understanding this makes the instrumentation decisions clearer. NanoClaw is a OSS fork of OpenClaw, so it shares the same core architecture:

flowchart LR
    WA[WhatsApp] --> ORC[Orchestrator]
    TG[Telegram] --> ORC
    SL[Slack] --> ORC
    ORC --> DB[(SQLite)]
    ORC --> Q[Group Queue]
    Q --> C1[Container 1]
    Q --> C2[Container 2]
    C1 --> IPC[IPC Watcher]
    IPC --> ORC
    SCH[Task Scheduler] --> Q

Messages arrive from channels, get stored in SQLite, and the orchestrator polls for new messages every 2 seconds. When a registered group has unprocessed messages, the group queue spawns a container running Claude’s Agent SDK. The agent’s output streams back through the orchestrator to the originating channel. Scheduled tasks follow the same path but are triggered by a cron/interval scheduler rather than by incoming messages.

The key insight: everything flows through the orchestrator. That’s where we intercept commands, instrument actions, and expose state. The group queue manages the container lifecycle. The database is already there for message storage. All the pieces are in place - we just need to wire them up.

2. What was built

Here’s the full inventory. Two new modules, six modified files, nine distinct features:

All commands are restricted to the main group only. Non-main groups are silently ignored, preventing random group members from querying system status or managing tasks.

3. /status - real-time system dashboard

The /status command assembles information from several subsystems into a single message. It queries the group queue for container states, the database for registered groups and tasks, and formats it all as a WhatsApp-friendly message.

The output is designed to give a quick overview of the system’s health and activity at a glance:

What it shows:

• Uptime and memory - how long the process has been running, RSS in MB
• Timezone - the configured timezone (important for cron schedules)
• Containers - active/max concurrent, with per-group detail (idle, processing, running task, queued)
• Channels - which messaging channels are connected (WhatsApp, Telegram, etc.)
• Groups - all registered groups with the main group indicator
• Tasks - count of active/paused tasks, next upcoming task with time-until
• Remote control - whether a remote Claude Code session is active

The implementation pulls from existing subsystems - no new state tracking was needed:

export function buildStatus(queue: GroupQueue, channels: Channel[]): string {
  const uptime = formatDuration(Date.now() - startTime);
  const mem = Math.round(process.memoryUsage.rss() / 1024 / 1024);

  const qs = queue.getStatus();       // new method on GroupQueue
  const channelNames = channels.map((ch) => ch.name).join(', ');
  const groups = getAllRegisteredGroups();
  const tasks = getAllTasks();
  const activeTasks = tasks.filter((t) => t.status === 'active');
  const rc = getActiveSession();       // remote control state

  const lines: string[] = [
    `*${ASSISTANT_NAME} Status*`,
    `----------------`,
    `Uptime: ${uptime}`,
    `Memory: ${mem} MB`,
    `Timezone: ${TIMEZONE}`,
    ``,
    `*Containers:* ${qs.activeCount}/${qs.maxConcurrent} active`,
    `*Channels:* ${channelNames || 'none'}`,
  ];
  // ... active container details, groups, tasks, remote control
  return lines.join('\n');
}

3.1 Exposing queue internals

The GroupQueue class already tracked everything we needed internally: active containers, pending messages, pending tasks, and idle state. It just wasn’t exposed. A new getStatus() method surfaces this without leaking internal implementation:

getStatus(): {
  activeCount: number;
  maxConcurrent: number;
  waitingCount: number;
  groups: Array<{
    jid: string;
    active: boolean;
    idleWaiting: boolean;
    isTaskContainer: boolean;
    pendingMessages: boolean;
    pendingTaskCount: number;
  }>;
} {
  const groups = [];
  for (const [jid, state] of this.groups) {
    if (state.active || state.pendingMessages || state.pendingTasks.length > 0) {
      groups.push({
        jid,
        active: state.active,
        idleWaiting: state.idleWaiting,
        isTaskContainer: state.isTaskContainer,
        pendingMessages: state.pendingMessages,
        pendingTaskCount: state.pendingTasks.length,
      });
    }
  }
  return {
    activeCount: this.activeCount,
    maxConcurrent: MAX_CONCURRENT_CONTAINERS,
    waitingCount: this.waitingGroups.length,
    groups,
  };
}

Only groups with activity (active, pending messages, or pending tasks) are included - no noise from idle groups. The status output translates these states into human-readable labels: “idle”, “processing”, “running task”, or “queued”.

4. /status tasks - task detail view

While /status includes a task summary (count + next upcoming), /status tasks gives the full picture. Each task shows its prompt (truncated to 50 chars), schedule type, next run time, last run time, and the task ID you need for management commands.

function formatTaskLine(task: ScheduledTask, index: number): string {
  const status = task.status === 'active' ? ''
    : task.status === 'paused' ? ' [paused]' : ' [done]';
  const schedule = task.schedule_type === 'cron'
    ? `cron: ${task.schedule_value}`
    : task.schedule_type === 'interval'
      ? `every ${task.schedule_value}`
      : `once`;
  const prompt = task.prompt.length > 50
    ? task.prompt.slice(0, 50) + '...' : task.prompt;
  const next = task.next_run ? formatTimeUntil(task.next_run) : 'n/a';
  const lastRun = task.last_run ? formatTimeAgo(task.last_run) : 'never';

  return [
    `*${index + 1}.* ${prompt}${status}`,
    `   Schedule: ${schedule}`,
    `   Next: ${next} | Last: ${lastRun}`,
    `   ID: ${task.id}`,
  ].join('\n');
}

Tasks are grouped by status - active first, then paused, then completed. The relative time formatting (in 2h 15m, 3m 42s ago) makes it easy to see at a glance what’s coming up and what ran recently.

5. /task - operational control

The /task command turns the chat into a control plane. No more SSH-ing in to pause a runaway task or clean up a completed one.

export function handleTaskCommand(
  args: string,
): { ok: true; message: string } | { ok: false; error: string } {
  const parts = args.trim().split(/\s+/);
  const action = parts[0]?.toLowerCase();
  const taskId = parts.slice(1).join(' ');

  const task = getTaskById(taskId);
  if (!task) return { ok: false, error: `Task not found: ${taskId}` };

  switch (action) {
    case 'pause':
      if (task.status !== 'active')
        return { ok: false, error: `Task is already ${task.status}` };
      updateTask(taskId, { status: 'paused' });
      return { ok: true, message: `Paused: "${task.prompt.slice(0, 50)}"` };
    case 'resume':
      if (task.status !== 'paused')
        return { ok: false, error: `Task is ${task.status}, not paused` };
      updateTask(taskId, { status: 'active' });
      return { ok: true, message: `Resumed: "${task.prompt.slice(0, 50)}"` };
    case 'delete':
      deleteTask(taskId);
      return { ok: true, message: `Deleted: "${task.prompt.slice(0, 50)}"` };
    default:
      return { ok: false, error: `Unknown action: ${action}` };
  }
}

The result type ({ ok: true; message } | { ok: false; error }) is a pattern used throughout NanoClaw for commands; the caller doesn’t need to know the implementation details, just whether it succeeded and what to tell the user. State validation is done upfront (can’t pause an already-paused task, can’t resume an active one).

6. Command interception

All built-in commands (/status, /status tasks, /task, /debug) share the same interception pattern: they’re caught at the top of the onMessage callback, before the message is stored or processed.

const channelOpts = {
  onMessage: (chatJid: string, msg: NewMessage) => {
    const trimmed = msg.content.trim();

    // Built-in commands - intercept before storage
    if (trimmed === '/status' || trimmed === '/status tasks') {
      handleStatus(trimmed, chatJid, msg).catch(/* ... */);
      return;  // don't store, don't process
    }
    if (trimmed.startsWith('/task ')) {
      handleTaskCmd(chatJid, msg).catch(/* ... */);
      return;
    }
    if (trimmed.startsWith('/debug')) {
      handleDebugCmd(chatJid, msg).catch(/* ... */);
      return;
    }

    // ... sender allowlist filtering, event logging, message storage
    storeMessage(msg);
  },
};

The return before storeMessage() is the key design decision. These commands are ephemeral – they shouldn’t appear in the conversation history, shouldn’t trigger the agent, and shouldn’t affect message cursors. They’re handled entirely by the host process and respond instantly (no container spawn needed).

Each handler checks group?.isMain before proceeding. Non-main-group commands are silently dropped, with a warning in the server logs.

7. Event/action/tool logging

The commands above tell you what’s happening now. But when something went wrong an hour ago, you need a trail. That’s where event logging comes in.

7.1 The three-table schema

NanoClaw’s existing SQLite schema was focused on message storage and container state. We needed a new schema to capture the full traceability from inbound triggers to outbound actions to tool calls. The design is a simple three-table structure. Every inbound trigger is an event, every outbound action is an action linked to its triggering event, and every tool invocation is a tool call linked to its parent action.

CREATE TABLE IF NOT EXISTS event_log (
  id           TEXT PRIMARY KEY,    -- UUID
  timestamp    DATETIME DEFAULT CURRENT_TIMESTAMP,
  source       TEXT NOT NULL,       -- 'whatsapp', 'telegram', 'scheduled_task', 'ipc'
  source_id    TEXT,                -- message ID, task ID, etc.
  raw_content  TEXT,                -- full payload (JSON, truncated to 10KB)
  summary      TEXT                 -- human-readable one-liner
);

CREATE TABLE IF NOT EXISTS action_log (
  id            TEXT PRIMARY KEY,
  timestamp     DATETIME DEFAULT CURRENT_TIMESTAMP,
  triggered_by  TEXT REFERENCES event_log(id),
  action_type   TEXT NOT NULL,      -- 'message_sent', 'task_scheduled', 'tool_call'
  target        TEXT,               -- group JID, email address, task ID
  content       TEXT,               -- what was sent or done
  tool_calls    TEXT                -- JSON array of tool names
);

CREATE TABLE IF NOT EXISTS tool_call_log (
  id           TEXT PRIMARY KEY,
  action_id    TEXT REFERENCES action_log(id),
  timestamp    DATETIME DEFAULT CURRENT_TIMESTAMP,
  tool_name    TEXT NOT NULL,
  input        TEXT,                -- JSON (truncated to 10KB)
  output       TEXT,                -- JSON (truncated to 10KB)
  duration_ms  INTEGER,
  success      INTEGER DEFAULT 1
);

CREATE INDEX IF NOT EXISTS idx_event_log_timestamp ON event_log(timestamp);
CREATE INDEX IF NOT EXISTS idx_action_log_triggered_by ON action_log(triggered_by);
CREATE INDEX IF NOT EXISTS idx_tool_call_log_action_id ON tool_call_log(action_id);

The foreign key chain is event_log <- action_log <- tool_call_log. Given any action, you can trace back to why it happened. Given any event, you can see everything it caused. The indexes support the /debug query patterns: filtering by timestamp (pruning), joining by triggered_by (event lookups), and grouping by action_id (tool call chains).

flowchart TD
    E[Event: WhatsApp message received] --> A1[Action: message_sent to group]
    E --> A2[Action: task_scheduled]
    A1 --> T1[Tool Call: runContainerAgent]
    A1 --> T2[Tool Call: channel.sendMessage]

These tables live in the same messages.db file as everything else. No additional file handles, no additional backup concerns, no additional connection management. They’re created in the existing createSchema() function using CREATE TABLE IF NOT EXISTS, so they’re added transparently on first startup after the upgrade.

7.2 The logger module

Three functions, matching the three tables. All IDs are UUIDs via crypto.randomUUID(). All writes are fire-and-forget - wrapped in try/catch, errors logged at debug level, never blocking the pipeline.

export function logEvent(
  source: string,
  sourceId: string | null,
  rawContent: unknown,
  summary: string,
): string {
  const id = crypto.randomUUID();
  try {
    insertEventStmt().run(
      id, new Date().toISOString(), source, sourceId,
      truncate(rawContent), summary,
    );
  } catch (err) {
    logger.debug({ err, source, sourceId }, 'Failed to log event');
  }
  return id;  // always returns an ID, even if the write failed
}

export function logAction(
  triggeredBy: string | null,
  actionType: string,
  target: string | null,
  content: unknown,
  toolCalls?: string[],
): string {
  const id = crypto.randomUUID();
  try {
    insertActionStmt().run(
      id, new Date().toISOString(), triggeredBy, actionType,
      target, truncate(content),
      toolCalls ? JSON.stringify(toolCalls) : null,
    );
  } catch (err) {
    logger.debug({ err, actionType, target }, 'Failed to log action');
  }
  return id;
}

Every call returns an ID, so callers can chain events -> actions -> tool calls, even if the write fails silently. This is intentional - the pipeline code doesn’t check whether logging succeeded, it just carries the ID forward.

The truncate() helper caps content at 10KB to prevent DB bloat:

const MAX_CONTENT_SIZE = 10 * 1024;

function truncate(value: unknown): string | null {
  if (value === undefined || value === null) return null;
  const str = typeof value === 'string' ? value : JSON.stringify(value);
  if (str.length > MAX_CONTENT_SIZE) return str.slice(0, MAX_CONTENT_SIZE);
  return str;
}

7.3 The tool call wrapper

logToolCall is different from the other two - it wraps an async operation and automatically records input, output, duration, and success/failure:

export async function logToolCall<T>(
  actionId: string,
  toolName: string,
  input: unknown,
  fn: () => Promise<T>,
): Promise<T> {
  const id = crypto.randomUUID();
  const start = Date.now();
  let success = true;
  let output: unknown = null;

  try {
    const result = await fn();
    output = result;
    return result;
  } catch (err) {
    success = false;
    output = err instanceof Error ? err.message : String(err);
    throw err;  // re-throw - logging doesn't swallow errors
  } finally {
    const durationMs = Date.now() - start;
    try {
      insertToolCallStmt().run(
        id, actionId, new Date().toISOString(), toolName,
        truncate(input), truncate(output), durationMs, success ? 1 : 0,
      );
    } catch (logErr) {
      logger.debug({ err: logErr, toolName }, 'Failed to log tool call');
    }
  }
}

The finally block ensures the log entry is written regardless of whether the wrapped operation succeeded or failed. The error is always re-thrown - logToolCall is transparent to the caller. It’s a decorator pattern: wrap any async operation and get free instrumentation.

Insert statements are lazily prepared and reused across calls, avoiding the overhead of re-preparing the same SQL on every log write.

7.4 Instrumenting the pipeline

With the logger module in place, instrumentation is surgical - a few lines at each key point in the message flow:

Inbound messages (in the onMessage callback):

// Detect channel from JID format
const evtChannel = chatJid.includes('@g.us') || chatJid.includes('@s.whatsapp.net')
  ? 'whatsapp'
  : chatJid.startsWith('tg:') ? 'telegram'
  : chatJid.startsWith('dc:') ? 'discord'
  : chatJid.startsWith('sl:') ? 'slack'
  : 'channel';

logEvent(
  evtChannel, msg.id,
  { sender: msg.sender_name, content: msg.content?.slice(0, 200) },
  `Message from ${msg.sender_name}: ${(msg.content || '').slice(0, 80)}`,
);

Agent processing (when the orchestrator starts handling a group’s messages):

// Log the processing event - this ID links to all downstream actions
const eventId = logEvent(
  'message_batch', chatJid,
  { messageCount: missedMessages.length, group: group.name },
  `Processing ${missedMessages.length} message(s) for ${group.name}`,
);

Outbound messages (in the streaming output callback):

if (text) {
  await channel.sendMessage(chatJid, text);
  logAction(eventId, 'message_sent', chatJid, text.slice(0, 500));
}

The eventId from the processing event links the outbound action back to the batch that triggered it. This is the chain that /debug why follows.

Scheduled tasks - logged at the point the scheduler picks up a due task, and again when the result is sent to the user:

const eventId = logEvent(
  'scheduled_task', task.id,
  { prompt: task.prompt.slice(0, 200), schedule: task.schedule_type },
  `Scheduled task: ${task.prompt.slice(0, 80)}`,
);

// ... later, when the container produces output:
await deps.sendMessage(task.chat_jid, streamedOutput.result);
logAction(eventId, 'message_sent', task.chat_jid, result?.slice(0, 500) ?? null);

IPC - logged when the IPC watcher processes messages and task operations from containers:

const ipcEventId = logEvent(
  'ipc', null,
  { chatJid: data.chatJid, sourceGroup, text: data.text?.slice(0, 200) },
  `IPC message from ${sourceGroup}`,
);
await deps.sendMessage(data.chatJid, data.text);
logAction(ipcEventId, 'message_sent', data.chatJid, data.text?.slice(0, 500) ?? null);

8. The /debug commands

Four subcommands for querying the event log, all main-group only:

8.1 /debug last - quick scan

The “what happened recently?” view. Joins action_log with event_log to show each action alongside what caused it:

export function getLastActions(n: number): Array<{
  action: ActionLogRow;
  event: EventLogRow | null;
}> {
  const rows = getDb()
    .prepare(
      `SELECT a.*, e.id as e_id, e.timestamp as e_timestamp,
              e.source as e_source, e.source_id as e_source_id,
              e.summary as e_summary
       FROM action_log a
       LEFT JOIN event_log e ON a.triggered_by = e.id
       ORDER BY a.timestamp DESC
       LIMIT ?`,
    )
    .all(n);
  // ... map to structured result
}

The LEFT JOIN is important - some actions might not have a triggering event (e.g., system-initiated actions), and we still want to see them.

8.2 /debug why - full trace

Answers “why did the last thing happen?” by pulling the most recent action, its triggering event, and all associated tool calls:

export function getLastActionWithToolCalls(): {
  action: ActionLogRow;
  event: EventLogRow | null;
  toolCalls: ToolCallLogRow[];
} | null {
  const results = getLastActions(1);
  if (results.length === 0) return null;

  const { action, event } = results[0];
  const toolCalls = getDb()
    .prepare(
      `SELECT * FROM tool_call_log WHERE action_id = ? ORDER BY timestamp`,
    )
    .all(action.id) as ToolCallLogRow[];

  return { action, event, toolCalls };
}

The output shows the full chain: triggering event (source, summary, timestamp, ID), the action taken (type, target, content), and each tool call with its duration and success/failure status. Copy the event ID from here and use /debug event <id> to see everything else that event triggered.

8.3 /debug report - summary dashboard

The “is everything healthy?” view. Aggregates across all three tables into a single report:

What it includes:

• Retention period - configured days and current time window
• Table sizes - row counts for events, actions, and tool calls
• Events by source - breakdown by channel (whatsapp, telegram, scheduled_task, ipc)
• Actions by type - breakdown by what was done (message_sent, task_scheduled, etc.)
• Busiest hours - top 5 hours by event count, in local timezone
• Recent failed tool calls - last 10 with tool name, duration, and error output
• Recent errors - last 10 error-like actions with their triggering event

One thing worth noting: the busiest hours are computed in JavaScript using toLocaleString with the configured time zone, not in SQL. SQLite stores timestamps as UTC (as ISO strings), and performing timezone conversions in SQL would require loading an extension. Instead, we fetch the raw timestamps and bucket them in JS:

const allTimestamps = db.prepare(`SELECT timestamp FROM event_log`).all();
const hourCounts = new Map<string, number>();
for (const { timestamp } of allTimestamps) {
  const localHour = new Date(timestamp).toLocaleString('en-US', {
    timeZone: TIMEZONE,
    hour: 'numeric', hour12: true,
  });
  hourCounts.set(localHour, (hourCounts.get(localHour) || 0) + 1);
}

Learned this the hard way when the report initially showed UTC hours, and I couldn’t figure out why 5 PM was my busiest time. 😄

9. Auto-pruning and retention

An unbounded observability system is a liability. Logs older than 3 days are automatically deleted. The retention period is configurable via the EVENT_LOG_RETENTION_DAYS environment variable (set to 0 to disable pruning).

// config.ts
export const EVENT_LOG_RETENTION_DAYS = Math.max(
  0,
  parseInt(process.env.EVENT_LOG_RETENTION_DAYS || '3', 10) || 3,
);
export const EVENT_LOG_PRUNE_INTERVAL = 60 * 60 * 1000; // hourly

Pruning runs at startup (clean up anything that expired while the service was down) and then every 60 minutes:

export function pruneOldLogs() {
  if (EVENT_LOG_RETENTION_DAYS === 0) return;

  const cutoff = new Date(
    Date.now() - EVENT_LOG_RETENTION_DAYS * 24 * 60 * 60 * 1000,
  ).toISOString();

  const db = getDb();
  // Delete in FK-safe order: children first
  db.prepare(`DELETE FROM tool_call_log WHERE action_id IN (
    SELECT id FROM action_log WHERE timestamp < ?
  )`).run(cutoff);
  db.prepare(`DELETE FROM action_log WHERE timestamp < ?`).run(cutoff);
  db.prepare(`DELETE FROM event_log WHERE timestamp < ?`).run(cutoff);
}

export function startLogPruning(): void {
  pruneOldLogs();
  const timer = setInterval(pruneOldLogs, EVENT_LOG_PRUNE_INTERVAL);
  timer.unref();   // don't keep the process alive for pruning
}

The deletion order matters: tool_call_log rows reference action_log, which in turn references event_log. Deleting parents first would violate foreign key constraints. The timer.unref() call ensures the pruning interval doesn’t prevent graceful shutdown.

10. Design decisions

A few choices that are worth calling out:

Fire-and-forget, not await. Every logging call is synchronous (better-sqlite3) and wrapped in try/catch. If the write fails - disk full, DB locked, schema mismatch - the error is logged at debug level and the pipeline continues. The logging system is never on the critical path. An observability system that can take down the thing it’s observing is worse than useless.

Same database, no new dependencies. The logging tables live in messages.db alongside messages, tasks, sessions, and router state. No new files to back up, no new connections to manage, no new packages to install. CREATE TABLE IF NOT EXISTS means existing installations pick up the schema on restart.

Commands intercepted before storage. /status, /task, and /debug messages never reach the agent container. They don’t appear in conversation history, don’t trigger container spawns, and don’t affect message cursors. This is important - a /status check shouldn’t cost you a container slot or show up as context in the agent’s next conversation.

Prepared statements, lazily created. The insert statements are created on first use and reused across calls. For a system logging every message and action, re-preparing SQL on every call would add up.

UUIDs for everything. crypto.randomUUID() for all IDs. No auto-increment, no collision risk across restarts, and IDs are meaningful in isolation (you can paste one into /debug event <id> without context).

11. Try it yourself

If you’re running NanoClaw (or OpenClaw), these features are available out of the box. Here’s how to get started:

If you already have NanoClaw running:

1. Pull the latest code and rebuild:

   1 2	git pull npm run build

2. Restart the service:

   1 2 3 4 5

   # Linux (systemd) systemctl --user restart nanoclaw # macOS (launchd) launchctl kickstart -k gui/$(id -u)/com.nanoclaw

3. The new tables are created automatically on startup. No migration step needed.

4. Send /status in your main group to verify it’s working.

If you’re starting fresh:

1. Fork or clone bahree/nanoclaw (or the upstream OpenClaw )
2. Follow the setup instructions in the README
3. Once connected to a channel, all commands are available immediately

Command reference:

Configuration:

All commands are main-group only. They respond instantly (no container needed) and don’t appear in the conversation history.

12. Summary

Three problems, one philosophy: make the system controllable and observable from the same interface you use to interact with it.

/status gives you real-time visibility - what’s running, what’s queued, what’s scheduled, which channels are connected. /task gives you operational control: pause a runaway task, resume one you paused, and clean up completed ones. Event logging gives you after-the-fact traceability - every action links back to its triggering event, every tool call links back to its parent action. /debug commands let you query the trail. Auto-pruning keeps it from growing unbounded.

About ~1100 lines of new TypeScript across 8 files. Two new modules (status.ts and event-log.ts), three new SQLite tables, a handful of indexes, and one new config variable. No new dependencies, no separate services. It just works on the next restart.

The source code for NanoClaw is available at bahree/nanoclaw .

In this final part of our 4-part series on building language models from scratch, we explore the evaluation, testing, and deployment pipeline that transforms our trained historical language models into working systems. Part 1 showed you how to use the published models, Part 2 covered data collection and custom tokenization, and Part 3 detailed the model architecture and training infrastructure. Here, we complete the journey with evaluation frameworks, testing infrastructure, and deployment to Hugging Face Hub.

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need much larger datasets, more sophisticated infrastructure, and additional considerations not covered here.

As outlined in Part 1 , both the SLM (117M parameters) and the Regular Model (354M parameters) use the same training code and infrastructure with different configurations defined in config.py. The evaluation and deployment infrastructure is also identical - only the model architecture parameters differ.

Both PyTorch checkpoint inference and Hugging Face model inference are fully working and available. Both the SLM and the Regular model are published on Hugging Face Hub . Local PyTorch checkpoints can be used directly for inference with the script inference_pytorch.py.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete evaluation and deployment infrastructure (05_evaluation/, 06_inference/, 10_scripts/) plus guides (08_documentation/EVALUATION_GUIDE.md, 08_documentation/HUGGINGFACE_PUBLISHING.md, 08_documentation/DEPLOYMENT_GUIDE.md)

🟥 Series Posts: Part 1 - Using the Published Historical Models | Part 2 - Data Collection & Custom Tokenizer | Part 3 - Training Architecture & GPU Optimization | Part 4 (this post)

🟧 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📗 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts

1. The Evaluation Challenge: Measuring What Matters for Historical Language Models

Now that we have trained models from Part 3 , we face a critical question: How do we know if our models actually work? This isn’t just about checking if the code runs - it’s about validating that the models can generate historically accurate, linguistically appropriate text that captures the essence of 1500-1850 London English.

The challenge with evaluating historical language models goes far beyond standard LLM metrics. Standard evaluation approaches like Perplexity and BLEU scores (we explain these and other metrics in Section 2.1 ) tell us whether the model generates fluent text. Still, they don’t answer the questions that matter for historical applications: Does the model avoid anachronisms? Can it distinguish between Tudor and Victorian language patterns? Does it understand London geography and historical context?

Consider a simple example: if we prompt the model with “In the year 1600, I traveled to London by railway”, a standard language model might generate this without flagging the obvious problem - railways didn’t exist in 1600. The evaluation framework needs to catch these temporal inconsistencies, period-inappropriate language, and historical inaccuracies that standard metrics miss.

This evaluation challenge requires building a specialized assessment pipeline that understands historical context, temporal boundaries, and period-specific linguistic patterns. We need metrics that can distinguish between a model that generates fluent modern English and one that produces authentic historical text - two very different capabilities.

1.1 High-Level Evaluation Strategy

Our evaluation framework provides two complementary approaches that work with both PyTorch checkpoints and Hugging Face models, as illustrated in Figure 1 below.

graph TD
    A[🤖 Trained Models<br/>SLM 117M / Regular 354M] --> B{Evaluation Type}
    
    B -->|Quick| C[⚡ Quick Evaluation<br/>Historical accuracy, language quality, coherence]
    B -->|Comprehensive| D[🔬 Comprehensive Evaluation<br/>Benchmarks, G-Eval, groundedness]
    
    C --> E[📊 Evaluation Results<br/>Historical accuracy scores, metrics]
    D --> E
    
    E --> F{Quality OK?}
    F -->|Yes| G[🚀 Deployment Options]
    F -->|No| H[🔄 Retrain/Adjust]
    H --> A
    
    G --> I[📦 PyTorch Checkpoints<br/>Direct inference]
    G --> J[🤗 Hugging Face Hub<br/>Published models]
    G --> K[💻 Local Deployment<br/>API, CLI, notebooks]
    
    I --> L[✅ Working Models<br/>Ready for use]
    J --> L
    K --> L
    
    style A fill:#e1f5fe
    style E fill:#f3e5f5
    style L fill:#e8f5e8
    style H fill:#fff3e0

Quick Evaluation (quick_eval.py): Rapid validation testing historical accuracy on key events (e.g., 1665 plague, 1666 fire, etc.), language quality metrics (vocabulary diversity, historical pattern detection, readability), and coherence (ROUGE scores). Runs in minutes without external APIs.

Comprehensive Evaluation (comprehensive_evaluator.py): Extends quick evaluation with benchmark datasets (small MMLU and HellaSWAG subsets), groundedness/fluency metrics, and optional LLM-as-a-judge scoring via G-Eval (using an external GPT model). Produces detailed reports with generation samples.

Both evaluators test across historical periods (such as Tudor, Stuart, and Georgian), language patterns (archaic pronouns and verb forms), and London-specific knowledge (geography and landmarks). The framework goes beyond standard LM metrics to assess period-appropriate language, temporal consistency, and historical accuracy.

2. Model Evaluation Framework

Now that we’ve outlined the evaluation challenge, let’s dive into the implementation. Our evaluation framework provides two complementary approaches that work with both PyTorch checkpoints and Hugging Face models. The framework is designed to be practical for a learning project while still providing meaningful insights into model performance.

2.1 Historical, Linguistic, and Category-Specific Evaluation

To make the evaluation concrete, we look at the model from three complementary aspects that together capture how well it understands the period, writes fluent text, and handles the different slices of the corpus. This multi-dimensional approach ensures we catch various types of failures - a model might generate grammatically perfect text but fail historically, or vice versa.

• Historical assessments: Quick evaluation uses targeted prompts around key events (e.g., 1665 plague, 1666 fire, Old Bailey trials) and checks for expected keywords and phrases. Comprehensive evaluation adds temporal consistency checks (forbidden/required terms per period), date-range sanity checks, and historical benchmarks (custom historical questions and the MMLU subset).
• Linguistic assessments: We measure surface quality (chars/words/sentences per sample, words per sentence), vocabulary diversity (unique/total tokens), readability (Flesch-style scores), and presence of historical patterns (archaic verb forms like hath, doth, pronouns like thou, thee, conjunctions and interjections). This shows whether the model writes in a historically flavored yet readable style.
• Category-specific benchmarks: Evaluations are grouped by period (Tudor, Stuart, Georgian), by linguistic phenomena (archaic forms, dialogue patterns), and by London knowledge (Thames, Westminster, Old Bailey, etc.). The comprehensive evaluator further probes general reasoning using HellaSWAG and MMLU subsets to assess the model’s performance across broader benchmarks.

Industry-Standard Evaluation Metrics and Benchmarks

Our evaluation framework uses several standard metrics and benchmarks from LLM research. Here’s what each one measures and why we include it:

• Perplexity: How surprised the model is by the reference text; lower is better because it means the model assigns higher probability to what actually happened in the corpus.
• BLEU / ROUGE: N-gram overlap between generated and reference text, giving a rough sense of literal similarity and how closely the model “sticks” to the reference phrasing. We use ROUGE-L (longest common subsequence) to evaluate coherence and narrative flow.
• MMLU (Massive Multitask Language Understanding): A large multiple-choice exam covering many academic subjects. Here, we use a tiny subset as a sanity check for general knowledge and reasoning, not as a primary goal.
• HellaSWAG: A commonsense inference benchmark where the model must pick a plausible continuation for a short story-like context. We use it to see whether the model’s basic reasoning looks sensible.
• G-Eval: An LLM-as-a-judge pattern where a stronger reference model (for example, GPT) scores generated text along dimensions like coherence or groundedness. In this project, it is optional and requires an external API key.
• Groundedness: Asks: does the model stick to the provided context / known facts, or hallucinate? Our implementation approximates this by comparing generations against reference answers and historical constraints.

For a deeper treatment of evaluation benchmarks (including MMLU, HellaSWAG, and LLM-as-a-judge methods like G-Eval), see Chapter 12 - Evaluating and Monitoring Generative Systems in the book 📘 Generative AI in Action .

2.2 Automated Evaluation Pipeline

The run_comprehensive_evaluation function in 05_evaluation/comprehensive_evaluator.py orchestrates the entire evaluation process. Listing 1 shows how it works: We iterate over test sets, generate text with the model, compute all the metrics defined above, and aggregate the results into a results dictionary for analysis.

def run_comprehensive_evaluation(model, tokenizer, test_data, device='cuda'):
    """Run comprehensive evaluation on historical language model"""
    
    # Initialize evaluation metrics
    metrics = {
        'perplexity': [],
        'bleu_scores': [],
        'rouge_scores': [],
        'historical_accuracy': [],
        'linguistic_quality': [],
        'coherence_scores': [],
        'temporal_consistency': []
    }
    
    # Evaluate on different text types
    for text_type, samples in test_data.items():
        logger.info(f"Evaluating on {text_type} samples...")
        
        for sample in samples:
            # Generate text
            generated = generate_text(model, tokenizer, sample['prompt'], device)
            
            # Calculate metrics
            perplexity = calculate_perplexity(model, tokenizer, sample['text'], device)
            bleu = calculate_bleu(generated, sample['reference'])
            rouge = calculate_rouge(generated, sample['reference'])
            hist_acc = assess_historical_accuracy(generated, sample['context'])
            ling_qual = assess_linguistic_quality(generated)
            coherence = assess_coherence(generated)
            temp_cons = assess_temporal_consistency(generated, sample['time_period'])
            
            # Store metrics
            metrics['perplexity'].append(perplexity)
            metrics['bleu_scores'].append(bleu)
            metrics['rouge_scores'].append(rouge)
            metrics['historical_accuracy'].append(hist_acc)
            metrics['linguistic_quality'].append(ling_qual)
            metrics['coherence_scores'].append(coherence)
            metrics['temporal_consistency'].append(temp_cons)
    
    # Calculate aggregate metrics
    results = {}
    for metric_name, values in metrics.items():
        results[metric_name] = {
            'mean': np.mean(values),
            'std': np.std(values),
            'min': np.min(values),
            'max': np.max(values),
            'median': np.median(values)
        }
    
    return results

The pipeline computes all the metrics we outlined above (standard LM metrics such as perplexity and BLEU/ROUGE, plus our historically specific assessments of accuracy, linguistic quality, and coherence). Each metric provides a different lens through which to view model performance: perplexity measures how well the model predicts the training distribution, BLEU/ROUGE measures literal similarity to the reference text, and the custom metrics assess historical authenticity and linguistic appropriateness.

Why This Multi-Metric Approach Matters?

Standard language model evaluation often focuses on perplexity and n-gram overlap metrics, which measure general language quality but miss domain-specific requirements. For historical language models, we need to know not just whether the text is fluent, but whether it’s historically accurate, temporally consistent, and linguistically appropriate for the target period. This multi-metric approach ensures we catch different types of failures - a model might generate grammatically perfect text but fail historically, or produce historically accurate content with poor linguistic quality.

The aggregation step (computing mean, std, min, max, median) provides a comprehensive view of model performance across different test cases. This statistical summary helps identify whether the model performs consistently or has high variance, whether certain types of prompts cause failures, and how the model compares across different historical periods and linguistic phenomena.

2.3 Historical Accuracy Assessment

Standard LLM evaluation metrics (perplexity, BLEU, ROUGE) measure general language quality, but they don’t tell us whether the model generates historically accurate text for London between 1500-1850. To address this, we built customized evaluation tools that check period-appropriate language, temporal consistency, London-specific geography and landmarks, and historical fact accuracy. These tools are implemented in 05_evaluation/comprehensive_evaluator.py as shown in Listing 2:

def assess_historical_accuracy(generated_text, historical_context):
    """Assess the historical accuracy of generated text"""
    
    accuracy_score = 0.0
    total_checks = 0
    
    # Check temporal consistency
    temporal_score = check_temporal_consistency(generated_text, historical_context['time_period'])
    accuracy_score += temporal_score
    total_checks += 1
    
    # Check historical facts
    fact_score = check_historical_facts(generated_text, historical_context['facts'])
    accuracy_score += fact_score
    total_checks += 1
    
    # Check period-appropriate language
    language_score = check_period_language(generated_text, historical_context['time_period'])
    accuracy_score += language_score
    total_checks += 1
    
    # Check geographical accuracy
    geo_score = check_geographical_accuracy(generated_text, historical_context['location'])
    accuracy_score += geo_score
    total_checks += 1
    
    # Check social context accuracy
    social_score = check_social_context(generated_text, historical_context['social_class'])
    accuracy_score += social_score
    total_checks += 1
    
    return accuracy_score / total_checks

def check_temporal_consistency(text, time_period):
    """Check if text maintains temporal consistency with the specified period"""
    
    # Define period-specific constraints
    period_constraints = {
        '1500-1600': {
            'forbidden_terms': ['electricity', 'steam engine', 'railway'],
            'required_terms': ['ye', 'hath', 'doth'],
            'date_range': (1500, 1600)
        },
        '1600-1700': {
            'forbidden_terms': ['railway', 'telegraph'],
            'required_terms': ['hath', 'doth', 'verily'],
            'date_range': (1600, 1700)
        },
        '1700-1800': {
            'forbidden_terms': ['telegraph', 'telephone'],
            'required_terms': ['hath', 'doth', 'indeed'],
            'date_range': (1700, 1800)
        },
        '1800-1850': {
            'forbidden_terms': ['telephone', 'automobile'],
            'required_terms': ['indeed', 'verily', 'pray'],
            'date_range': (1800, 1850)
        }
    }
    
    if time_period not in period_constraints:
        return 0.5  # Neutral score for unknown periods
    
    constraints = period_constraints[time_period]
    score = 1.0
    
    # Check for forbidden terms (anachronisms)
    for term in constraints['forbidden_terms']:
        if term.lower() in text.lower():
            score -= 0.2
    
    # Check for required period-appropriate terms
    period_terms_found = 0
    for term in constraints['required_terms']:
        if term.lower() in text.lower():
            period_terms_found += 1
    
    if constraints['required_terms']:
        score += 0.3 * (period_terms_found / len(constraints['required_terms']))
    
    # Check date references
    date_score = check_date_references(text, constraints['date_range'])
    score += 0.2 * date_score
    
    return max(0.0, min(1.0, score))

The forbidden terms (like “electricity” for 1500-1600, “railway” for 1600-1700) are anachronisms - technologies or concepts that didn’t exist in those periods. We selected them based on historical timelines: electricity wasn’t harnessed until the late 1700s, railways didn’t appear until the early 1800s, and telegraphs came later. Similarly, the required terms (such as “hath”, “doth”, and “verily”) are archaic language patterns we observed frequently in the training corpus for each period.

We analyzed the corpus to identify which linguistic markers were most characteristic of each era, then selected a small set that would catch obvious anachronisms without being overly restrictive. This is a practical heuristic rather than an exhaustive historical grammar - we focus on high-impact anachronisms and common period markers that are easy to detect automatically.

How the scoring works

The check_temporal_consistency() function starts with a score of 1.0 and applies penalties and bonuses: each forbidden term found subtracts 0.2 (so finding “railway” in 1600-1700 text drops the score), while finding required period-appropriate terms adds up to 0.3 based on how many are present. Date references within the period add up to 0.2. The final score ranges from 0.0 to 1.0.

The overall assess_historical_accuracy() function then averages the five component scores (temporal consistency, historical facts, period-appropriate language, geographical accuracy, and social context) to produce a single score between 0 and 1, with higher values indicating better historical accuracy. In practice (and yes, we are generalizing), scores above 0.7 indicate good historical consistency, while scores below 0.5 suggest significant anachronisms or factual errors.

2.4 Linguistic Quality Evaluation

While historical accuracy checks whether the model gets facts and period-appropriate terms right, linguistic quality measures how well the model writes - grammar, coherence, vocabulary diversity, sentence structure, and the presence of historical language patterns.

Standard metrics like BLEU and ROUGE don’t capture whether the text reads naturally or uses appropriate archaic forms. We built customized tools that assess these dimensions, implemented in 05_evaluation/comprehensive_evaluator.py as shown in Listing 3:

To make this easier to read, it helps to view the code as a scoring scaffold rather than a complete NLP system. Each check_* function is expected to return a normalized score in the range [0, 1] (higher is better), and assess_linguistic_quality() simply averages those components so you can track one headline number over time.

This mirrors patterns from earlier in the series: in Part 2 we used lightweight, automatable checks to validate data quality, and in Part 3 we relied on simple, repeatable metrics to judge training health. Here, we do the same for generation quality: start with cheap checks that run everywhere, then iterate toward richer evaluators as needed.

Also note that the exact weights (0.3/0.2, etc.) are tunable. The main benefit is splitting “linguistic quality” into components you can inspect individually, so when output is bad, you can tell why (grammar-ish structure vs coherence vs vocabulary vs historically flavored patterns).

def assess_linguistic_quality(generated_text):
    """Assess the linguistic quality of generated historical text"""
    
    quality_score = 0.0
    total_checks = 0
    
    # Check grammatical correctness
    grammar_score = check_grammatical_correctness(generated_text)
    quality_score += grammar_score
    total_checks += 1
    
    # Check coherence and flow
    coherence_score = check_text_coherence(generated_text)
    quality_score += coherence_score
    total_checks += 1
    
    # Check vocabulary appropriateness
    vocab_score = check_vocabulary_appropriateness(generated_text)
    quality_score += vocab_score
    total_checks += 1
    
    # Check sentence structure variety
    structure_score = check_sentence_structure_variety(generated_text)
    quality_score += structure_score
    total_checks += 1
    
    # Check historical language patterns
    pattern_score = check_historical_language_patterns(generated_text)
    quality_score += pattern_score
    total_checks += 1
    
    return quality_score / total_checks

def check_grammatical_correctness(text):
    """Check grammatical correctness of generated text"""
    
    # Parse text into sentences
    sentences = nltk.sent_tokenize(text)
    
    if not sentences:
        return 0.0
    
    correct_sentences = 0
    
    for sentence in sentences:
        # Check for basic grammatical patterns
        if check_sentence_grammar(sentence):
            correct_sentences += 1
    
    return correct_sentences / len(sentences)

def check_historical_language_patterns(text):
    """Check if text follows appropriate historical language patterns"""
    
    score = 0.0
    total_patterns = 0
    
    # Check for appropriate use of historical verb forms
    historical_verbs = ['hath', 'doth', 'dost', 'art', 'wilt', 'shalt']
    verb_score = 0
    for verb in historical_verbs:
        if verb in text.lower():
            verb_score += 1
    if historical_verbs:
        score += 0.3 * (verb_score / len(historical_verbs))
    total_patterns += 1
    
    # Check for appropriate use of historical pronouns
    historical_pronouns = ['thou', 'thee', 'thy', 'thine', 'ye']
    pronoun_score = 0
    for pronoun in historical_pronouns:
        if pronoun in text.lower():
            pronoun_score += 1
    if historical_pronouns:
        score += 0.3 * (pronoun_score / len(historical_pronouns))
    total_patterns += 1
    
    # Check for appropriate use of historical conjunctions
    historical_conjunctions = ['whilst', 'betwixt', 'amongst', 'ere', 'anon']
    conj_score = 0
    for conj in historical_conjunctions:
        if conj in text.lower():
            conj_score += 1
    if historical_conjunctions:
        score += 0.2 * (conj_score / len(historical_conjunctions))
    total_patterns += 1
    
    # Check for appropriate use of historical interjections
    historical_interjections = ['verily', 'indeed', 'forsooth', 'prithee', 'marry']
    interj_score = 0
    for interj in historical_interjections:
        if interj in text.lower():
            interj_score += 1
    if historical_interjections:
        score += 0.2 * (interj_score / len(historical_interjections))
    total_patterns += 1
    
    return score / total_patterns if total_patterns > 0 else 0.0

About NLTK: We use NLTK (Natural Language Toolkit), a standard Python library for natural language processing, to handle text tokenization. If you followed Part 1 ’s setup instructions, NLTK was already installed as part of the data processing dependencies. In check_grammatical_correctness(), we use nltk.sent_tokenize() to split text into sentences so we can evaluate grammar sentence-by-sentence. NLTK also provides word tokenization (word_tokenize) and BLEU score calculation (sentence_bleu), which are used elsewhere in the evaluation pipeline.

We chose NLTK because it’s well-established, handles edge cases (like abbreviations and historical punctuation), and provides reliable sentence boundaries even with archaic English patterns. The same qualities made it useful during data collection and cleaning (covered in Part 2 ).

The historical language patterns we check (verbs like hath, doth, pronouns like thou, thee, conjunctions like whilst, betwixt, and interjections like verily, forsooth) are the same archaic forms we identified during corpus analysis for temporal consistency. The difference here is that we’re measuring their presence as a positive signal of historical authenticity, rather than using them as required/forbidden constraints. Each pattern category (verbs, pronouns, conjunctions, interjections) contributes proportionally to the score based on how many patterns from that category appear in the text.

How the scoring works

The assess_linguistic_quality() function averages five component scores (grammar, coherence, vocabulary appropriateness, sentence structure variety, and historical language patterns) to produce a single score between 0 and 1. Each component is evaluated independently and returns a score in the range [0, 1].

For example, check_grammatical_correctness() counts the proportion of grammatically correct sentences, while check_historical_language_patterns() weights the presence of archaic verb forms (30%), pronouns (30%), conjunctions (20%), and interjections (20%) to produce a pattern score. The final linguistic quality score is the simple average of all five components. In practice, scores above 0.75 indicate strong linguistic quality with good grammar and historical flavor, while scores below 0.6 suggest the model struggles with either basic grammar or historical language patterns.

2.5 Running Evaluations

You can run the evaluators directly from the command line. The framework defaults to CPU for safety (so you can evaluate during training without GPU conflicts), but you can use --device gpu when the GPU is free for faster evaluation.

Quick example:

# Quick evaluation (runs in minutes, no external APIs)
python 05_evaluation/run_evaluation.py --mode quick --device cpu

# Comprehensive evaluation (includes benchmarks, optional G-Eval)
python 05_evaluation/run_evaluation.py --mode comprehensive --device cpu

The unified launcher (run_evaluation.py) supports multiple modes: setup (install dependencies), quick (fast validation), comprehensive (full suite with benchmarks), dataset (generate test cases), and all (complete evaluation). You can also call quick_eval.py or comprehensive_evaluator.py directly if you need more control.

Practical Evaluation Workflow:

Our typical evaluation workflow follows this pattern:

1. After Training: Run a quick evaluation to get immediate feedback on model performance
2. Before Publishing: Run a comprehensive evaluation to ensure the model meets quality standards
3. During Development: Use interactive testing to explore model behavior on specific prompts
4. For Research: Generate custom test datasets and run targeted evaluations

The framework defaults to CPU for safety (so you can evaluate during training without GPU conflicts), but you can use --device gpu when the GPU is free for faster evaluation. This design allows continuous assessment throughout the training process without interfering with GPU resources needed for training.

For complete usage examples, command-line options, and troubleshooting, see the Evaluation Guide in the repository.

3. Comprehensive Testing Pipeline

3.1 Automated Testing Framework

The 06_testing package contains a parallel set of tests that double-check the full system. Listing 4 captures the idea behind run_comprehensive_tests.

We group tests into basic functionality, historical accuracy, linguistic quality, performance, edge cases, and integration, then run them as a batch and emit a structured report. This mirrors how you would build a real CI test suite, but at a scale appropriate for this learning project.

def run_comprehensive_tests(model, tokenizer, device='cuda'):
    """Run comprehensive tests on historical language model"""
    
    test_results = {
        'basic_functionality': test_basic_functionality(model, tokenizer, device),
        'historical_accuracy': test_historical_accuracy(model, tokenizer, device),
        'linguistic_quality': test_linguistic_quality(model, tokenizer, device),
        'performance_metrics': test_performance_metrics(model, tokenizer, device),
        'edge_cases': test_edge_cases(model, tokenizer, device),
        'integration_tests': test_integration(model, tokenizer, device)
    }
    
    # Generate test report
    generate_test_report(test_results)
    
    return test_results

def test_basic_functionality(model, tokenizer, device):
    """Test basic model functionality"""
    
    tests = {
        'text_generation': test_text_generation(model, tokenizer, device),
        'tokenization': test_tokenization(tokenizer),
        'model_loading': test_model_loading(model, device),
        'memory_usage': test_memory_usage(model, device),
        'inference_speed': test_inference_speed(model, tokenizer, device)
    }
    
    return tests

def test_historical_accuracy(model, tokenizer, device):
    """Test historical accuracy of generated text"""
    
    tests = {
        'temporal_consistency': test_temporal_consistency(model, tokenizer, device),
        'factual_accuracy': test_factual_accuracy(model, tokenizer, device),
        'period_appropriate_language': test_period_language(model, tokenizer, device),
        'geographical_accuracy': test_geographical_accuracy(model, tokenizer, device),
        'social_context_accuracy': test_social_context(model, tokenizer, device)
    }
    
    return tests

Automated tests cover basics, historical accuracy, linguistic quality, performance, edge cases, and integration.

3.2 Interactive Testing and Validation

For manual exploration, the interactive testing interface (conceptually similar to the CLI flows in 06_inference/inference_unified.py) lets you type prompts, trigger specific test groups, and immediately inspect analysis for each generation. Listing 5 shows a simple REPL loop that dispatches to the same evaluation helpers used in the automated tests.

def interactive_testing(model, tokenizer, device='cuda'):
    """Interactive testing interface for historical language model"""
    
    print("Interactive Testing Mode")
    print("=" * 50)
    print("Enter prompts to test the model. Type 'quit' to exit.")
    print("Available commands:")
    print("  - Enter any text prompt to generate continuation")
    print("  - 'test_historical' - Run historical accuracy tests")
    print("  - 'test_linguistic' - Run linguistic quality tests")
    print("  - 'test_performance' - Run performance tests")
    print("  - 'quit' - Exit testing mode")
    print()
    
    while True:
        try:
            prompt = input("Enter prompt: ").strip()
            
            if prompt.lower() == 'quit':
                break
            elif prompt.lower() == 'test_historical':
                run_historical_tests(model, tokenizer, device)
            elif prompt.lower() == 'test_linguistic':
                run_linguistic_tests(model, tokenizer, device)
            elif prompt.lower() == 'test_performance':
                run_performance_tests(model, tokenizer, device)
            elif prompt:
                # Generate text
                generated = generate_text(model, tokenizer, prompt, device)
                print(f"Generated: {generated}")
                print()
                
                # Analyze generated text
                analysis = analyze_generated_text(generated, prompt)
                print(f"Analysis: {analysis}")
                print()
            else:
                print("Please enter a valid prompt or command.")
                
        except KeyboardInterrupt:
            print("\nExiting interactive testing mode...")
            break
        except Exception as e:
            print(f"Error: {e}")
            print("Please try again.")

def analyze_generated_text(text, prompt):
    """Analyze generated text for quality and accuracy"""
    
    analysis = {
        'length': len(text),
        'sentences': len(nltk.sent_tokenize(text)),
        'historical_accuracy': assess_historical_accuracy(text, {}),
        'linguistic_quality': assess_linguistic_quality(text),
        'coherence': assess_coherence(text),
        'relevance': assess_relevance(text, prompt)
    }
    
    return analysis

Interactive mode lets you try prompts, run quick tests, and see immediate analysis.

3.3 Performance Benchmarking

Performance benchmarking follows the same pattern: generate controlled workloads and measure speed and resource usage. Listing 6 illustrates how we vary sequence length, measure average latency, and compute tokens-per-second, alongside separate helpers for memory, batch throughput, long-sequence handling, and basic concurrency.

def benchmark_model_performance(model, tokenizer, device='cuda'):
    """Benchmark model performance across different scenarios"""
    
    benchmarks = {
        'inference_speed': benchmark_inference_speed(model, tokenizer, device),
        'memory_usage': benchmark_memory_usage(model, device),
        'batch_processing': benchmark_batch_processing(model, tokenizer, device),
        'long_sequence_handling': benchmark_long_sequences(model, tokenizer, device),
        'concurrent_requests': benchmark_concurrent_requests(model, tokenizer, device)
    }
    
    return benchmarks

def benchmark_inference_speed(model, tokenizer, device):
    """Benchmark inference speed for different sequence lengths"""
    
    sequence_lengths = [50, 100, 200, 500, 1000]
    results = {}
    
    for length in sequence_lengths:
        # Generate test prompts of different lengths
        prompts = generate_test_prompts(length, num_prompts=100)
        
        # Measure inference time
        start_time = time.time()
        for prompt in prompts:
            generate_text(model, tokenizer, prompt, device)
        end_time = time.time()
        
        total_time = end_time - start_time
        avg_time_per_prompt = total_time / len(prompts)
        tokens_per_second = length / avg_time_per_prompt
        
        results[length] = {
            'avg_time_per_prompt': avg_time_per_prompt,
            'tokens_per_second': tokens_per_second,
            'total_time': total_time
        }
    
    return results

Benchmarks capture inference speed, memory, batch throughput, long-sequence handling, and simple concurrency.

4. Model Deployment and Publishing

With evaluation and testing complete, we’re ready to make our models available for use. This section covers the two deployment paths we support: direct inference from PyTorch checkpoints (useful during development and for maximum control) and publishing to Hugging Face Hub (for easy sharing and community access).

As called out in Part 1 , both the SLM (117M parameters) and the Regular Model (354M parameters) are fully trained and available. The SLM has already been published on Hugging Face Hub , while the Regular Model is ready for publication. Both can also be run directly from local PyTorch checkpoints.

4.1 Two Paths to Inference

We provide two complementary ways to run inference, each suited to different use cases.

PyTorch Checkpoint Inference gives you direct access to the trained model weights without any conversion overhead. This is ideal during development, when you want to test a freshly trained checkpoint, or when you need maximum control over the inference process. The checkpoints live in 09_models/checkpoints/ - the SLM at slm/checkpoint-4000.pt (117M parameters) and the Regular Model at checkpoint-60001.pt (354M parameters). The inference_pytorch.py script handles loading these directly: Listing 7

# SLM inference from checkpoint
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/slm/checkpoint-4000.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Regular model inference from checkpoint
python 06_inference/inference_pytorch.py \
  --checkpoint 09_models/checkpoints/checkpoint-60001.pt \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

Hugging Face Model Inference uses the published models on Hugging Face Hub, which means anyone can load and use them with just a few lines of code - no need to download checkpoints or set up the full training environment. The inference_unified.py script provides a consistent interface for both published models and local checkpoints: Listing 8

# Published model inference (downloads from Hugging Face Hub)
python 06_inference/inference_unified.py \
  --published \
  --model_name bahree/london-historical-slm \
  --prompt "In the year 1834, I walked through the streets of London and witnessed"

# Interactive mode for exploration
python 06_inference/inference_unified.py --published --model_type slm --interactive

# Demo mode with curated historical prompts
python 06_inference/inference_unified.py --published --model_type slm --demo

We’ve tested both paths extensively. The published SLM loads in about 9 seconds on a GPU, generates text in under 6 seconds, and passes all 10 automated validation tests. The unified inference script provides clean logging, proper model detection, and accurate parameter counts - small details that make a big difference when debugging or demonstrating the models.

4.2 Publishing to Hugging Face Hub

Publishing to Hugging Face Hub makes our models accessible to the broader community without requiring anyone to clone our repository or set up a training environment. The process involves converting our PyTorch checkpoints to the Hugging Face format, creating a model card with documentation, and uploading everything to the Hub.

The publishing workflow is handled by scripts in 10_scripts/ - specifically publish_slm_to_huggingface.py for the SLM and publish_to_huggingface.py for the Regular Model. Listing 9 shows the core publishing flow: authenticate with the Hub, create (or reuse) a repository, save the model and tokenizer locally in Hugging Face format, upload the folder, and generate a model card.

def publish_to_huggingface(model, tokenizer, model_name, description, tags):
    """Publish model to Hugging Face Hub"""
    from huggingface_hub import HfApi
    
    api = HfApi()
    
    # Create model repository
    repo_id = f"bahree/{model_name}"
    api.create_repo(repo_id=repo_id, exist_ok=True)
    
    # Save model and tokenizer locally
    model.save_pretrained(f"./models/{model_name}")
    tokenizer.save_pretrained(f"./models/{model_name}")
    
    # Upload to Hub
    api.upload_folder(
        folder_path=f"./models/{model_name}",
        repo_id=repo_id,
        commit_message="Initial model upload"
    )
    
    # Generate and upload model card (README.md)
    model_card = generate_model_card(model_name, description, tags)
    api.upload_file(
        path_or_fileobj=model_card,
        path_in_repo="README.md",
        repo_id=repo_id
    )
    
    return repo_id

The generate_model_card() function creates the README.md that appears on the Hugging Face model page. This includes model description, architecture details, training data sources, usage examples, and limitations. You can see the live model cards at bahree/london-historical-slm and bahree/london-historical-llm .

Listing 10 shows how to load and use the published models:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load the published model
model_name = "bahree/london-historical-slm"  # or "bahree/london-historical-llm"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Generate historical text
prompt = "In the year 1834, I walked through the streets of London and witnessed"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=50,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.2,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

4.3 Publishing Workflow

If you want to publish your own trained model to Hugging Face Hub, here’s the workflow we followed:

1. Set up authentication: Install huggingface_hub and authenticate with a token that has Write permissions. You can generate tokens at huggingface.co/settings/tokens .

2. Convert the checkpoint: PyTorch training checkpoints include optimizer states and training metadata that aren’t needed for inference. The conversion scripts extract just the model weights and translate them to Hugging Face’s naming conventions (covered in detail in Section 5).

3. Prepare the tokenizer: Save the tokenizer files alongside the model. Our custom tokenizer with 30,000 tokens and 150+ historical special tokens needs to be converted to the transformers library format.

4. Generate a model card: The README.md on your Hugging Face model page serves as documentation. Include model architecture details, training data sources, usage examples, evaluation results, and limitations. The scripts generate this automatically, but you should review and customize it.

5. Upload and validate: Push everything to the Hub, then immediately test with from_pretrained() to ensure the published model loads and generates correctly.

📝 Full documentation: See 08_documentation/HUGGINGFACE_PUBLISHING.md and 08_documentation/DEPLOYMENT_GUIDE.md in the repository for the complete step-by-step workflow with troubleshooting guidance.

5. PyTorch to Hugging Face Format Conversion

5.1 Why Format Conversion is Necessary

During training, our models are saved in PyTorch’s native .pt format. These checkpoints include everything needed to resume training: model weights, optimizer states, learning rate schedules, and training metadata. However, for deployment and sharing, we need a leaner, inference-optimized format compatible with the broader machine learning ecosystem.

Think of it like the difference between a development environment and a production deployment: training checkpoints are like a developer’s workspace with all the tools and intermediate files, while Hugging Face format is like a clean, standardized package that anyone can use without understanding the internal training details.

The Hugging Face Hub expects models to follow specific file structures, naming conventions, and metadata requirements. The conversion process extracts just the model weights (discarding optimizer states and training metadata), translates weight names to match Hugging Face conventions, creates proper configuration files, and ensures the tokenizer is compatible with the transformers library.

5.2 The Conversion Process

The conversion handles several transformations to bridge PyTorch and Hugging Face formats:

• Weight name mapping: PyTorch layer names like transformer.h.0.attn.c_attn.weight become Hugging Face names like transformer.h.0.attn.c_attn.weight (mostly the same for GPT-2, but with careful handling of edge cases)
• Automatic torch.compile handling: If you used torch.compile() during training, weights get prefixed with _orig_mod. - the conversion strips these prefixes
• Configuration translation: Model hyperparameters (n_layer, n_head, n_embd, etc.) are mapped to Hugging Face’s config.json format
• Tokenizer conversion: Our custom 30,000-token vocabulary with 150+ historical special tokens is converted to transformers library format
• Validation: After conversion, we verify that the model loads correctly and produces expected outputs

💻 Full Implementation: See 10_scripts/publish_slm_to_huggingface.py for the complete conversion pipeline with error handling, validation, and model card generation.

5.3 Dependencies for Hugging Face Integration

The Hugging Face integration requires specific dependencies and follows established patterns for model publishing and usage: Listing 11

# Required dependencies for Hugging Face integration
huggingface_dependencies = {
    "transformers": ">=4.21.0",
    "torch": ">=1.12.0",
    "tokenizers": ">=0.12.0",
    "safetensors": ">=0.3.0",
    "accelerate": ">=0.20.0",
    "huggingface_hub": ">=0.10.0"
}

# Model loading and usage example
def load_published_model(model_name="bahree/london-historical-slm"):
    """Load published model from Hugging Face Hub"""
    
    # Suppress warnings for cleaner output
    import os
    import warnings
    import logging
    
    os.environ['TRANSFORMERS_VERBOSITY'] = 'error'
    warnings.filterwarnings('ignore')
    logging.getLogger("transformers").setLevel(logging.ERROR)
    os.environ['TOKENIZERS_PARALLELISM'] = 'false'
    
    # Load model and tokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name)
    
    # Set pad token if not set
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token
        tokenizer.pad_token_id = tokenizer.eos_token_id
    
    return model, tokenizer

def generate_historical_text(model, tokenizer, prompt, max_length=50, temperature=0.3):
    """Generate historical text using the published model"""
    
    # Tokenize input
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    
    # Generate text
    with torch.no_grad():
        outputs = model.generate(
            inputs,
            max_new_tokens=max_length,
            do_sample=True,
            temperature=temperature,
            top_p=0.9,
            top_k=20,
            repetition_penalty=1.2,
            no_repeat_ngram_size=3,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
            early_stopping=True
        )
    
    # Decode output
    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated_text

# Example usage
if __name__ == "__main__":
    # Load the published model
    model, tokenizer = load_published_model()
    
    # Test prompts
    test_prompts = [
        "In the year 1834, I walked through the streets of London and witnessed",
        "The gentleman from the country said, 'we have never seen such a sight",
        "The Thames flowed dark and mysterious through the heart",
        "Parliament sat in Westminster Hall",
        "The Great Fire of 1666 had destroyed"
    ]
    
    # Generate text for each prompt
    for prompt in test_prompts:
        generated = generate_historical_text(model, tokenizer, prompt)
        print(f"Prompt: {prompt}")
        print(f"Generated: {generated}")
        print("-" * 80)

Hugging Face integration provides standard from_pretrained() loading and generation with minimal setup, making the models easy to share and reuse.

5.4 Comprehensive Testing and Validation Framework

Once a model is on the Hub, 06_inference/test_published_models.py provides a concrete implementation of the testing pattern in Listing 12. It loads the model via from_pretrained, runs functional, historical, linguistic, and performance checks, and prints a human-readable summary so you can verify the published artefact behaves like your local checkpoints.

def test_published_model(model_name="bahree/london-historical-slm"):
    """Comprehensive testing of published model"""
    
    print(f"Testing published model: {model_name}")
    
    # Load model
    model, tokenizer = load_published_model(model_name)
    
    # Test basic functionality
    basic_tests = test_basic_functionality(model, tokenizer)
    
    # Test historical accuracy
    historical_tests = test_historical_accuracy(model, tokenizer)
    
    # Test linguistic quality
    linguistic_tests = test_linguistic_quality(model, tokenizer)
    
    # Test performance metrics
    performance_tests = test_performance_metrics(model, tokenizer)
    
    # Compile results
    results = {
        "basic_functionality": basic_tests,
        "historical_accuracy": historical_tests,
        "linguistic_quality": linguistic_tests,
        "performance_metrics": performance_tests
    }
    
    # Print summary
    print("\nTest Results Summary:")
    print("=" * 50)
    for category, tests in results.items():
        print(f"\n{category.replace('_', ' ').title()}:")
        for test_name, result in tests.items():
            status = "PASS" if result else "FAIL"
            print(f"  {test_name}: {status}")
    
    return results

def test_basic_functionality(model, tokenizer):
    """Test basic model functionality"""
    
    tests = {}
    
    # Test model loading
    tests["model_loading"] = model is not None and tokenizer is not None
    
    # Test tokenizer functionality
    test_text = "In the year 1834, London was"
    tokens = tokenizer.encode(test_text)
    decoded = tokenizer.decode(tokens)
    tests["tokenizer_encode_decode"] = test_text in decoded
    
    # Test model generation
    try:
        inputs = tokenizer.encode(test_text, return_tensors="pt")
        with torch.no_grad():
            outputs = model.generate(inputs, max_new_tokens=10, do_sample=False)
        generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
        tests["model_generation"] = len(generated) > len(test_text)
    except Exception as e:
        tests["model_generation"] = False
    
    # Test special tokens
    special_tokens = ["<|london|>", "<|thou|>", "<|hath|>", "<|doth|>"]
    special_token_tests = []
    for token in special_tokens:
        if token in tokenizer.get_vocab():
            special_token_tests.append(True)
        else:
            special_token_tests.append(False)
    tests["special_tokens"] = any(special_token_tests)
    
    return tests

def test_historical_accuracy(model, tokenizer):
    """Test historical accuracy of generated text"""
    
    tests = {}
    
    # Test prompts for different historical periods
    period_prompts = {
        "1500-1600": "In the year 1550, the gentleman said",
        "1600-1700": "In the year 1650, the gentleman said",
        "1700-1800": "In the year 1750, the gentleman said",
        "1800-1850": "In the year 1834, the gentleman said"
    }
    
    for period, prompt in period_prompts.items():
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=30)
            
            # Check for period-appropriate language
            period_terms = {
                "1500-1600": ["ye", "hath", "doth", "thou", "thee"],
                "1600-1700": ["hath", "doth", "thou", "thee", "verily"],
                "1700-1800": ["hath", "doth", "thou", "thee", "indeed"],
                "1800-1850": ["indeed", "verily", "whilst", "pray"]
            }
            
            found_terms = sum(1 for term in period_terms[period] if term in generated.lower())
            tests[f"period_{period}"] = found_terms > 0
            
        except Exception as e:
            tests[f"period_{period}"] = False
    
    # Test London-specific knowledge
    london_prompts = [
        "The Thames flowed through",
        "Westminster Hall was",
        "The Tower of London",
        "Cheapside was filled with"
    ]
    
    london_tests = []
    for prompt in london_prompts:
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=20)
            london_tests.append(len(generated) > len(prompt))
        except Exception as e:
            london_tests.append(False)
    
    tests["london_knowledge"] = any(london_tests)
    
    return tests

def test_linguistic_quality(model, tokenizer):
    """Test linguistic quality of generated text"""
    
    tests = {}
    
    # Test prompts for linguistic quality
    quality_prompts = [
        "The gentleman walked through the garden",
        "In the morning, the sun rose",
        "The old man sat by the fire",
        "The young woman read her book"
    ]
    
    quality_tests = []
    for prompt in quality_prompts:
        try:
            generated = generate_historical_text(model, tokenizer, prompt, max_length=30)
            
            # Check for basic linguistic quality
            sentences = generated.split('.')
            quality_tests.append(len(sentences) > 1)
            
        except Exception as e:
            quality_tests.append(False)
    
    tests["linguistic_quality"] = any(quality_tests)
    
    # Test coherence
    coherence_prompt = "The gentleman walked through the garden and"
    try:
        generated = generate_historical_text(model, tokenizer, coherence_prompt, max_length=50)
        tests["coherence"] = len(generated) > len(coherence_prompt)
    except Exception as e:
        tests["coherence"] = False
    
    return tests

def test_performance_metrics(model, tokenizer):
    """Test performance metrics of the model"""
    
    tests = {}
    
    # Test inference speed
    test_prompt = "In the year 1834, London was"
    try:
        start_time = time.time()
        generated = generate_historical_text(model, tokenizer, test_prompt, max_length=50)
        end_time = time.time()
        
        inference_time = end_time - start_time
        tests["inference_speed"] = inference_time < 5.0  # Should complete within 5 seconds
        
    except Exception as e:
        tests["inference_speed"] = False
    
    # Test memory usage
    try:
        import psutil
        process = psutil.Process()
        memory_usage = process.memory_info().rss / 1024 / 1024  # MB
        tests["memory_usage"] = memory_usage < 8000  # Should use less than 8GB
        
    except Exception as e:
        tests["memory_usage"] = True  # Skip if psutil not available
    
    return tests

Published model tests validate loading, generation, historical accuracy, and basic performance before and after publication.

5.5 Model Card Generation

The model card serves as the primary documentation on Hugging Face Hub, making it the first thing users see when they discover your model. A well-crafted model card helps users understand what the model does, how to use it, and its limitations. The generate_comprehensive_model_card() function in 10_scripts/publish_slm_to_huggingface.py creates this documentation automatically.

What Makes an Effective Model Card:

The model card for our historical language models includes several key sections that provide users with everything they need to get started. At a minimum, include:

1. Model Description & Key Features: A clear explanation that the model was trained from scratch (not fine-tuned), emphasizing the 117M parameter SLM variant and 354M parameter Regular Model, with details about the custom 30,000-token vocabulary and 150+ historical special tokens.

2. Setup Instructions: Platform-specific guidance for creating virtual environments (Linux/macOS/Windows), installing dependencies (transformers, torch, accelerate), and handling different accelerators (CPU, NVIDIA CUDA, AMD ROCm).

3. Quick Start Code: Auto-device detection that works across CPU, CUDA, and ROCm with sensible generation parameters (temperature=0.8, top_p=0.95, repetition_penalty=1.2).

4. Training Details: Architecture specifics (GPT-2 Small/Medium), training infrastructure (2x GPU with Distributed Data Parallel), performance metrics (training loss, MFU utilization), and data sources (218+ historical sources spanning 1500-1850).

5. Example Prompts: Period-specific prompts demonstrating different historical eras (Tudor, Stuart, Georgian, Victorian) and London-specific contexts (Thames, Westminster, Parliament).

6. Testing & Validation: Instructions for running the automated test suite (test_published_models.py) and interactive testing with custom prompts.

7. Troubleshooting: Common issues and solutions for PyTorch installation, GPU detection, and memory constraints.

8. Citation & License: BibTeX citation format and MIT license information.

Key Implementation Details:

The model card generation follows Hugging Face conventions with YAML frontmatter specifying license, library, pipeline type, language, and tags. The script emphasizes that models were trained from scratch (not fine-tuned) and provides device-agnostic code examples that run on CPU, CUDA, and ROCm.

The card also includes detailed model selection guidance comparing the SLM (faster, lower memory) versus the Regular Model (higher quality, more parameters), helping users choose the right model for their use case - whether that’s quick experimentation, educational purposes, or production deployment.

💻 Complete Implementation: See 10_scripts/publish_slm_to_huggingface.py and 10_scripts/publish_to_huggingface.py for the full model card generation implementation.

👀 Live Model Cards: View the published cards at bahree/london-historical-slm and bahree/london-historical-llm .

📝 Documentation: See HUGGINGFACE_PUBLISHING.md and DEPLOYMENT_GUIDE.md for complete publishing and deployment workflows.

5.6 Local Deployment Options

Finally, Listing 13 sketches how you might wrap a trained model into a simple REST API or CLI. These patterns are intentionally minimal, meant to help you connect the dots between the inference utilities in 06_inference/ and real applications (dashboards, notebooks, small services).

def setup_local_deployment(model, tokenizer, deployment_type='api'):
    """Set up local deployment for historical language model"""
    
    if deployment_type == 'api':
        return setup_api_deployment(model, tokenizer)
    elif deployment_type == 'cli':
        return setup_cli_deployment(model, tokenizer)
    elif deployment_type == 'notebook':
        return setup_notebook_deployment(model, tokenizer)
    else:
        raise ValueError(f"Unknown deployment type: {deployment_type}")

def setup_api_deployment(model, tokenizer):
    """Set up REST API deployment"""
    
    from flask import Flask, request, jsonify
    import torch
    
    app = Flask(__name__)
    
    @app.route('/generate', methods=['POST'])
    def generate_text():
        data = request.get_json()
        prompt = data.get('prompt', '')
        max_length = data.get('max_length', 100)
        temperature = data.get('temperature', 0.3)
        
        # Generate text
        inputs = tokenizer.encode(prompt, return_tensors="pt")
        with torch.no_grad():
            outputs = model.generate(
                inputs,
                max_length=max_length,
                temperature=temperature,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )
        
        generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
        
        return jsonify({
            'generated_text': generated_text,
            'prompt': prompt,
            'parameters': {
                'max_length': max_length,
                'temperature': temperature
            }
        })
    
    @app.route('/health', methods=['GET'])
    def health_check():
        return jsonify({'status': 'healthy', 'model_loaded': True})
    
    return app

def setup_cli_deployment(model, tokenizer):
    """Set up command-line interface deployment"""
    
    import argparse
    
    def main():
        parser = argparse.ArgumentParser(description='Historical Language Model CLI')
        parser.add_argument('--prompt', type=str, required=True, help='Text prompt')
        parser.add_argument('--max_length', type=int, default=100, help='Maximum length')
        parser.add_argument('--temperature', type=float, default=0.3, help='Temperature')
        parser.add_argument('--interactive', action='store_true', help='Interactive mode')
        
        args = parser.parse_args()
        
        if args.interactive:
            run_interactive_mode(model, tokenizer)
        else:
            generate_and_print(model, tokenizer, args.prompt, args.max_length, args.temperature)
    
    return main

Local deployment options: REST API, CLI, or notebook integration for different workflows.

6. Quality Assurance and Validation

Before wrapping up, let’s look at the quality assurance systems that ensure the models behave reliably across different scenarios.

6.1 Automated Quality Checks

The system includes automated quality checks that validate model performance and reliability: Listing 14

def run_quality_checks(model, tokenizer, device='cuda'):
    """Run quality checks on historical language model"""
    
    quality_checks = {
        'model_integrity': check_model_integrity(model),
        'tokenizer_consistency': check_tokenizer_consistency(tokenizer),
        'generation_quality': check_generation_quality(model, tokenizer, device),
        'historical_accuracy': check_historical_accuracy(model, tokenizer, device),
        'performance_metrics': check_performance_metrics(model, tokenizer, device),
        'memory_usage': check_memory_usage(model, device),
        'error_handling': check_error_handling(model, tokenizer, device)
    }
    
    # Generate quality report
    quality_report = generate_quality_report(quality_checks)
    
    return quality_checks, quality_report

def check_model_integrity(model):
    """Check model integrity and consistency"""
    
    checks = {
        'parameter_count': check_parameter_count(model),
        'weight_distribution': check_weight_distribution(model),
        'gradient_flow': check_gradient_flow(model),
        'activation_patterns': check_activation_patterns(model)
    }
    
    return checks

def check_generation_quality(model, tokenizer, device):
    """Check quality of generated text"""
    
    test_prompts = [
        "In the year of our Lord 1750, London was",
        "The Thames flowed through the heart of",
        "Merchants and tradesmen plied their wares",
        "The Great Fire of 1666 had changed",
        "Parliament sat in Westminster, making laws"
    ]
    
    quality_scores = []
    
    for prompt in test_prompts:
        # Generate text
        generated = generate_text(model, tokenizer, prompt, device)
        
        # Check quality metrics
        quality_score = {
            'coherence': assess_coherence(generated),
            'grammatical_correctness': assess_grammatical_correctness(generated),
            'historical_accuracy': assess_historical_accuracy(generated, {}),
            'linguistic_quality': assess_linguistic_quality(generated),
            'relevance': assess_relevance(generated, prompt)
        }
        
        quality_scores.append(quality_score)
    
    return quality_scores

Quality checks cover model integrity, generation quality, historical accuracy, performance, and error handling, ensuring the models behave reliably across different scenarios.

6.2 Continuous Integration and Testing

If you want to wire this into a lightweight CI gate, keep it simple and CPU-friendly. The goal is not to re-run full benchmarks in CI - it’s to catch obvious regressions (can the model load, can it generate, do the evaluators still run).

Minimal CI smoke checks (suggested):

# 1) Run a fast, local evaluation pass (no external APIs)
python 05_evaluation/run_evaluation.py --mode quick --device cpu

# 2) Run a local inference smoke test from a checkpoint (replace with your path)
python 06_inference/inference_pytorch.py --checkpoint <path-to-checkpoint.pt> --prompt "In the year 1834, London was"

# 3) Optional: test the published model (requires downloading from Hugging Face)
python 06_inference/test_published_models.py --model_name bahree/london-historical-slm

7. Summary

We’ve now completed the full cycle of building language models from scratch. This final part has shown how to transform trained models into working systems that can be evaluated, tested, and deployed for real-world use. The journey that began in Part 1 with using published models, continued through Part 2 ’s data collection and tokenization, and Part 3 ’s training architecture, now concludes with evaluation and deployment - the critical final steps that make models usable.

What we’ve built:

The evaluation, testing, and deployment pipeline provides a practical approach for bringing historical language models from research to deployment. We’ve created specialized assessment metrics that go beyond standard LLM evaluation to catch historical inaccuracies, temporal inconsistencies, and period-inappropriate language. The testing infrastructure ensures reliability across different scenarios, while multiple deployment options make the models accessible to researchers, educators, and developers worldwide.

Current Deployment Status:

• PyTorch Checkpoint Inference: Fully working with both SLM and Regular models
• Hugging Face Model Inference: SLM published and available, Regular model ready
• Local Testing: Both inference methods tested and validated on a remote Ubuntu machine
• Documentation: Complete guides and examples for all inference methods
• Performance: Clean logging, proper model detection, accurate parameter counts

The Complete Pipeline:

This four-part series has demonstrated the complete LLM development lifecycle:

1. Data Collection ( Part 2 ): We gathered 218+ historical sources spanning 1500-1850, processed them through a sophisticated cleaning pipeline, and created a 500M+ character corpus of authentic historical English.

2. Custom Tokenization ( Part 2 ): We built a specialized BPE tokenizer with 30,000 vocabulary tokens and 150+ special tokens that understand historical language patterns, London geography, and period-specific terminology.

3. Model Training ( Part 3 ): We implemented custom GPT architectures, optimized for multi-GPU training, and successfully trained two models - an SLM (117M parameters) and a Regular model (354M parameters) - both capable of generating authentic historical text.

4. Evaluation & Deployment (This Part): We built comprehensive evaluation frameworks that assess historical accuracy, linguistic quality, and temporal consistency. We created a testing infrastructure for reliability and deployed models to the Hugging Face Hub for community access.

The Learning Journey:

What started as a learning project has become a complete, working system that demonstrates every aspect of LLM development - from raw data collection through model deployment. The principles and techniques we’ve covered scale from the 500M-character corpus to production-scale systems, and the evaluation frameworks we’ve built can be adapted to any domain-specific language modeling task.

Whether you’re a researcher exploring historical linguistics, an educator teaching AI concepts, or a developer building specialized language models, this series provides the complete toolkit for understanding and implementing LLM development from scratch. The models are published, the code is available, and the journey from data to deployment is complete.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete training infrastructure (04_training/), model architecture (config.py), and evaluation/deployment (05_evaluation/, 06_inference/, 10_scripts/)

🟥 Series Posts: Part 1 - Using the Published Historical Models | Part 2 - Data Collection & Custom Tokenizer | Part 3 - Training Architecture & GPU Optimization | Part 4 (this post)

🟧 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📗 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts

8. Resources

If you want to reproduce the full pipeline (or adapt it to your own domain), these are the most useful starting points:

• Public GitHub repo: github.com/bahree/helloLondon
• Evaluation guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/EVALUATION_GUIDE.md
• Hugging Face publishing guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/HUGGINGFACE_PUBLISHING.md
• Deployment guide: https://github.com/bahree/helloLondon/blob/main/08_documentation/DEPLOYMENT_GUIDE.md
• Published models: bahree/london-historical-slm | bahree/london-historical-llm

References

1. Vaswani et al. (2017) - Attention Is All You Need: https://arxiv.org/abs/1706.03762
2. Radford et al. (2019) - Language Models are Unsupervised Multitask Learners: https://www.semanticscholar.org/paper/Language-Models-are-Unsupervised-Multitask-Learners-Radford-Wu/9405cc0d6169988371b2755e573cc28650d14dfe
3. Lin (2004) - ROUGE: A Package for Automatic Evaluation of Summaries: https://aclanthology.org/W04-1013/
4. Papineni et al. (2002) - BLEU: A Method for Automatic Evaluation of Machine Translation: https://aclanthology.org/P02-1040/
5. Hendrycks et al. (2021) - Measuring Massive Multitask Language Understanding (MMLU): https://arxiv.org/abs/2009.03300
6. Zellers et al. (2019) - HellaSwag: Can a Machine Really Finish Your Sentence?: https://arxiv.org/abs/1905.07830
7. Liu et al. (2023) - G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment: https://arxiv.org/abs/2303.16634

Acknowledgments

This project builds upon the excellent work of the open-source community. Special thanks to haykgrigo3’s TimeCapsuleLLM for the initial inspiration and framework for historical language model training, and to Andrej Karpathy’s nanoGPT for the foundational GPT architecture and training methodology. The project extends these foundations with specialized adaptations for historical text, including custom tokenizers, advanced data filtering, evaluation frameworks, and educational deployment infrastructure.

In this third part of our 4-part series on building language models from scratch, I explore the complete training infrastructure that transforms our clean historical data and custom tokenizer into working language models.

• Part 1 How to build a Large Language Model from Scratch - covered using the published model
• Part 2 Building LLMs from Scratch - Part 2: Data Collection & Custom Tokenizers - detailed data collection and custom tokenizer development.

Here, we build the complete training pipeline from a custom GPT architecture through deployment-ready checkpoints.

This post demonstrates how to design custom model architectures, optimize GPU utilization, and implement comprehensive training pipelines that transform our 500M+ character historical corpus into two working language models.

⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need significantly larger datasets, more sophisticated infrastructure, and additional considerations that are not covered in this post.

As outlined in Part 1 , both the SLM (117M parameters) and the regular Model (354M parameters) use the same training code and pipeline (04_training/train_model_slm.py and 04_training/train_model.py) with different configurations defined in config.py. The training infrastructure, GPU optimization, checkpointing, and WandB integration are identical - only the model architecture parameters differ.

Both PyTorch checkpoint inference and Hugging Face model inference are fully working and available. Both the SLM and the Regular model are published on Hugging Face Hub . Local PyTorch checkpoints can be used directly for inference with the inference_pytorch.py script.

🔗 GitHub Repository: github.com/bahree/helloLondon - Complete training infrastructure (04_training/), model architecture (config.py), and GPU configuration (08_documentation/GPU_TUNING.md)

🧱 Series Posts: Part 1 – Using the Published Historical Models | Part 2 – Data Collection & Custom Tokenizer | Part 3 (this post) | Part 4 – Evaluation & Deployment

🤗 Published Models: SLM Model | Regular Model - Ready-to-use historical language models on Hugging Face

📚 Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts.

1. The Training Challenge: From Data to Working Models

Now that we have our clean historical corpus and custom tokenizer from Part 2 , we need to transform this data into working language models. This isn’t just about running training scripts – it’s about designing an architecture that can learn from historical text, optimizing for the unique patterns of 1500-1850 English, and building infrastructure to handle the computational demands of language model training.

The challenge with historical language modeling isn’t just having enough data - it’s having the right architecture and training process that can learn from the complex linguistic patterns in historical texts. Unlike modern text, historical English contains archaic vocabulary, period-specific terminology, and cultural references that require specialized attention mechanisms and training strategies.

1.1 High-Level Training Process Overview

The model training pipeline transforms our clean historical data and custom tokenizer into working language models through several key stages:

1. Model Architecture Design - involves a custom GPT implementation optimized for historical text patterns
2. GPU Configuration covers - multi-GPU training with precision optimization and memory management
3. Training Infrastructure - includes distributed training, checkpointing, and experiment tracking
4. Performance Optimization - encompasses mixed precision, compilation, and hardware-specific tuning
5. Model Validation - covers testing and evaluation of trained models.

Figure 1 below illustrates this complete training pipeline:

graph TD
    A[📚 Clean Historical Corpus<br/>500M+ characters] --> B[🔤 Custom Tokenizer<br/>30K vocab + 150+ special tokens]
    B --> C[🏗️ Model Architecture<br/>Custom GPT for Historical Text]
    
    C --> D[⚙️ GPU Configuration<br/>Multi-GPU + Precision Optimization]
    D --> D1[Mixed Precision<br/>bf16/fp16]
    D --> D2[Torch Compile<br/>JIT optimization]
    D --> D3[Memory Management<br/>Gradient checkpointing]
    
    D1 --> E[🏋️ Training Process<br/>60K iterations, checkpointing]
    D2 --> E
    D3 --> E
    
    E --> E1[SLM: 117M params<br/>7-8 hours training]
    E --> E2[Regular: 354M params<br/>28-32 hours training]
    
    E --> E3[WandB Integration<br/>Experiment tracking]
    E --> E4[Checkpointing<br/>Resume capability]
    E --> E5[Multi-GPU Support<br/>Distributed training]
    
    E1 --> F[📊 Model Evaluation<br/>Historical accuracy testing]
    E2 --> F
    E3 --> F
    E4 --> F
    E5 --> F
    
    F --> G{Quality OK?}
    G -->|Yes| H[🚀 Deployment<br/>Hugging Face + Local Inference]
    G -->|No| I[🔄 Retrain/Adjust]
    I --> E
    H --> J[💬 Text Generation<br/>Historical language output]
    
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style H fill:#e8f5e8
    style J fill:#fff3e0

We will explore each of these components in detail, starting with the model architecture design, but first, let’s discuss why I chose PyTorch as the framework for this project.

1.2 Using PyTorch

I chose PyTorch for this project based on three key factors: educational accessibility, integration with the research ecosystem, and practical convenience. PyTorch provides many components out of the box - transformer blocks, attention layers, feed-forward networks, training loops, and CUDA support - which makes it much easier for learners building their first language model.

From a technical perspective, PyTorch’s memory management and GPU optimization features-including automatic mixed precision, gradient checkpointing, and efficient attention implementations well-suited for the resource-intensive task of training language models on historical text.

PyTorch’s recent developments, such as torch.compile, FlashAttention kernels, and SDPA operator (scaled dot-product attention), provide significant performance improvements, making training more efficient. These improvements enhance both speed and memory efficiency, which are critical for scaling LLMs. Of course, in our case, we’re building a working toy example rather than scaling to production levels, and these optimizations help keep training times reasonable on available hardware.

What about other Frameworks?

I also considered TensorFlow and JAX, but neither seemed right for helloLondon; TensorFlow’s API felt too complex, specifically from a beginner’s perspective. JAX has excellent performance and a clean, functional approach, but it’s more research-focused and has a smaller ecosystem, which would make it harder to follow along and experiment with.

2. Model Architecture Overview

2.1 Understanding the GPT Architecture

Our custom GPT (Generative Pre-trained Transformer) is a decoder-only transformer model designed for autoregressive language modeling on historical text. The architecture consists of four core components, each serving a distinct purpose in the sequence-to-sequence prediction pipeline. These are: token embeddings, position embeddings, causal self-attention mechanisms, and the language modeling head. Let us double-click into each component to understand its role and implementation.

2.1.1 Token Embeddings

Token embeddings convert discrete token IDs from our 30,000-token historical vocabulary into dense, continuous vector representations. Each token (whether it’s a word, subword unit, or special token) is mapped to a point in a high-dimensional space (768 dimensions for SLM, 1024 for the regular model).

This is implemented as a simple lookup table - wte = torch.nn.Embedding(config.vocab_size, config.n_embd). When processing the token sequence, we look up the corresponding vector for each token ID. These embeddings are learned during training - the model learns which tokens should be close together in this vector space based on their co-occurrence patterns in historical text.

For historical language models, this is particularly valuable because rare historical terms (like “yeoman” or “guildhall”) get their own representations that can capture contextual relationships with related terms from that era.

2.1.2 Position Embeddings

Position embeddings encode each token’s absolute position within the sequence. This is crucial because, unlike recurrent models, transformer architectures have no inherent notion of temporal order or sequence position - they process all tokens in parallel. Let us double-click into the problem why.

Think of it like reading words without any sense of order. The words “The cat chased the mouse” would be indistinguishable from “Mouse the chased cat the” - you’d see the same words but lose all meaning because you don’t know which word came first, second, or third. Transformers face exactly this problem because they process all words simultaneously rather than sequentially, unlike older RNN models.

To help the model understand word order, we add position embeddings to the token embeddings. This way, each token’s representation includes information about both “what” the token is and “where” it appears in the sequence. We use learned position embeddings (as opposed to fixed sinusoidal patterns): wpe = torch.nn.Embedding(config.block_size, config.n_embd). For the SLM with a 512-token context window, we learn 512 different position vectors (one for each possible position). Similarly, the regular model with a 1024-token context learns 1024 position vectors.

Position embeddings work like giving each word a “timestamp” or “address” that tells the model where it sits in the sequence:

1. Token embedding says: “This is the word ‘cat’” → converts to a vector like [0.2, -0.5, 0.8, ...]
2. Position embedding says: “This word is at position 3” → adds another vector like [0.1, 0.3, -0.2, ...]
3. Combined: The model sees both “what” the word is AND “where” it appears

The embedding vectors are combined element-wise: x = token_emb + position_emb. This allows the model to understand both what each token is (via the token embedding) and where it appears in the sequence (via the position embedding).

Our model uses learned position embeddings, meaning during training the model discovers that:

• Position 1 tends to be capitalized (start of sentence)
• Position 512 might be mid-sentence (needs different handling)
• Certain positions in historical documents have patterns (formal openings, closings, etc.)

This is different from fixed sinusoidal embeddings (used in the original Transformer paper), which use a mathematical formula to encode positions. Learned embeddings are generally better because they adapt to specific patterns in the training data.

In historical texts, word order is crucial for understanding meaning. Consider “The King granted the land” versus “The land granted the King” - same words, completely different meanings. Historical legal documents and Victorian-era writings often have precise word order that changes legal or semantic meaning. Position embeddings ensure the model can distinguish between these critical variations.

2.1.3 Causal Self-Attention

Causal self-attention is the mechanism that allows each position in the sequence to selectively attend to previous positions. The “causal” constraint ensures the model can only look at past tokens (not future ones), which is essential for autoregressive generation.

When you read a sentence, you naturally use context from earlier words to understand later ones. If you see “The King granted the land to his loyal…”, you can predict that “servant,” “knight,” or “subject” might come next because you remember what came before. The model needs to do the same thing - use previous words to predict the next word.

However, there’s a crucial constraint: during training, when predicting word 7, the model must only see words 1-6, never word 8 or beyond. This “causal” (cause-and-effect) constraint ensures the model learns realistic patterns - in the real world, you can’t use future information to predict the present.

How Attention Works

Think of attention as a sophisticated “relevance detector”. When the model is processing the word “loyal” in our example above, it needs to look back and ask: “Which previous words are most relevant for understanding this context?” The attention mechanism computes a weighted sum of all previous token representations, where the weights are determined by how relevant each previous token is to the current one.

This is done through three learned linear projections that create different “views” of each word:

• Query (Q): “What am I looking for?” - The current word asks a question
• Key (K): “What do I have to offer?” - Previous words advertise their content
• Value (V): “What information should I contribute?” - The actual information to pass forward

Let us see a practical example to help us grok the concept. Consider the historical phrase: “The alderman of Cheapside, having served the city faithfully, was granted…”

When processing “granted” the attention mechanism:

1. Creates a Query from “granted” asking “what context do I need?”
2. Compares this Query against Keys from all previous words
3. Finds high relevance with “alderman” (who is being granted something) and “faithfully” (why the grant is happening)
4. Uses these attention weights to pull relevant Values from those words
5. Combines this information to understand better “granted” in context

The attention score between token i and token j is computed as:

$$\text{Attention}(Q_i, K_j) = \text{softmax}\left(\frac{Q_i K_j^T}{\sqrt{d_k}}\right)V_j$$

Breaking this down:

• $Q_i K_j^T$ computes how well the query from token i matches the key from token j (higher = more relevant)
• $1/\sqrt{d_k}$ is a scaling factor that prevents scores from getting too large (which would make softmax too “sharp”)
• $\text{softmax}$ converts scores into probabilities that sum to 1 (so each word gets a weighted “vote”)
• Finally, we use these weights to combine the Values from all previous tokens

The $1/\sqrt{d_k}$ scaling factor (where $d_k = 64$ for our SLM, so $\sqrt{64} = 8$) prevents the dot products from growing too large with high-dimensional embeddings, ensuring stable gradients during training. The softmax ensures the weights sum to 1, creating a proper probability distribution over the previous tokens.

Why This Matters for Historical Text?

Historical documents present unique challenges that make attention particularly valuable. Consider a legal document from 1750: “John Smith, yeoman of the parish of St. Giles, being of sound mind and body, doth hereby bequeath…”

The attention mechanism enables the model to:

• Connect “doth bequeath” back to “John Smith” across multiple clauses
• Understand that “yeoman” modifies “John Smith” even though they’re separated
• Learn that “doth” (archaic) and “does” (modern) serve similar grammatical functions
• Recognize that formal legal phrasing follows specific patterns

For our historical language models, this attention mechanism learns which historical terms and phrases co-occur and relate to one another contextually - crucial for understanding historical documents where terminology and phrasing differ from modern English. The model learns to attend to relevant historical context, enabling it to generate coherent text that maintains period-appropriate language patterns and references.

2.1.4 Language Modeling Head

The language modeling head (also called the “output projection” or lm_head) is the final translator that turns the rich internal representation (after all the attention + MLP refinements) back into a decision: “Given everything I’ve seen so far, what is the most likely next token?” It does this by mapping each hidden vector at every position into a vector of length equal to the vocabulary size (30,000 in our historical tokenizer). Each element of that output vector is a logit - an unnormalized score indicating how likely the model thinks the token is to be the next one.

Implementation is intentionally simple: lm_head = torch.nn.Linear(n_embd, vocab_size). We don’t put an activation function after it because we want raw, unconstrained scores. Those scores then flow into:

• Inference: Apply softmax -> probabilities -> sample or greedy pick
• Training: Feed logits + target token IDs into cross-entropy loss -> gradients flow backward

You can think of logits as evidence totals. The softmax transforms those evidence values into a normalized probability distribution that the model can sample from. High logit = more supporting evidence; low logit = less.

Step-by-step (Inference vs Training):

1. Hidden state at last position (e.g., index 511) enters lm_head.
2. Linear projection produces a 30,000-dimensional logit vector.
3. In inference: probs = softmax(logits / temperature); optionally apply top-k/top-p filtering.
4. Sample (or argmax) a token → append to sequence → repeat.
5. In training: Cross-entropy compares logits to the true next token; loss scalar backpropagates through the head into all prior layers.

Because our vocabulary mixes common function words (“the”, “and”, “of”) with rare era-specific tokens (“yeoman”, “guildhall”, “paternoster”, “quoth”), the head must reliably distinguish both frequent and infrequent patterns. Rare historical tokens need consistent representations from embedding -> transformer -> head so they are not forgotten. If their logits remained perpetually low, the model would never learn to generate them in authentic contexts.

Logits (not probabilities) inside the model - We retain logits (raw, unnormalized scores) instead of immediately converting to probabilities because they yield numerically stable loss computation - PyTorch efficiently fuses log_softmax with negative log-likelihood - allow cleaner gradient flow before any normalization (we only invoke softmax when we actually need a distribution), and enable flexible post-processing (temperature scaling, top-k or top-p filtering, repetition penalties) directly in score space without forcing an extra probability recomputation step.

We reuse the input embedding matrix for the output projection to keep input and output semantics aligned and reduce parameter and memory traffic. This concept is called Weight Typing, which we will cover in detail in Section 2.2.2 - Weight Tying .

We share the embedding and output projection weights (self.transformer.wte.weight = self.lm_head.weight) so input token interpretation and next-token scoring occur in the same semantic space.

Using the shared embedding matrix $E$ (shape $(V,d)$), the logits are computed with $\text{logits} = h \cdot E^T$, reusing the same rows used for token lookup. This saves parameters (~23.0M SLM, ~30.7M Regular), keeps gradients for rare historical tokens coupled, and reduces memory traffic (Press & Wolf, 2017; Inan et al., 2016). See Section 2.2.2 for detailed mechanics, benefits, and the historian/scribe analogy.

In short, the lm_head converts rich contextual understanding into next-token scores; with weight tying (details in Section 2.2.2) it stays efficient and semantically consistent.

2.1.5 The Complete Flow

The complete forward pass through our GPT model works as follows:

1. Input: A sequence of token IDs (batch × sequence_length, e.g., 512 tokens)
2. Token Embedding: Convert each token ID to a dense vector (768 or 1024 dimensions)
3. Position Embedding: Add position information to each token
4. Transformer Blocks: Pass through n_layer transformer blocks (12 for SLM, 24 for regular model), each containing:
   • Layer normalization
   • Causal self-attention (with multiple heads)
   • Residual connection
   • Layer normalization
   • Feed-forward MLP
   • Residual connection
5. Final Layer Norm: Normalize the final hidden states
6. Language Head: Project to vocabulary logits (30,000 dimensions)
7. Output: Probability distribution over next token

This architecture design is conventional and follows the GPT-style pattern established by OpenAI’s GPT models. The traditional design is intentional - it allows for clear, educational learning from the implementation while being configured to work seamlessly with our historical tokenizer from Part 2.

Figure 2 below illustrates the complete architecture for the SLM:

graph TD
    A[Input Tokens<br/>512 tokens] --> B[Token Embedding<br/>30K vocab → 768 dim]
    A --> C[Position Embedding<br/>512 pos → 768 dim]
    B --> D[Add Embeddings]
    C --> D
    D --> E[Layer Norm]
    E --> F[Transformer Block 1<br/>12 heads, 768 dim]
    F --> G[Transformer Block 2<br/>12 heads, 768 dim]
    G --> H[...]
    H --> I[Transformer Block 12<br/>12 heads, 768 dim]
    I --> J[Final Layer Norm]
    J --> K[Language Head<br/>768 → 30K vocab]
    K --> L[Output Logits<br/>30K probabilities]
    
    subgraph "Key Specifications"
        M[Layers: 12<br/>Heads: 12<br/>Embedding: 768<br/>Context: 512<br/>Parameters: 117M<br/>Training: 7-8 hours<br/>MFU: 8-9%]
    end
    
    style A fill:#e1f5fe
    style L fill:#e8f5e8
    style M fill:#fff3e0

The Regular model, as shown in Figure 3 below, follows the same architectural pattern as the SLM but with increased capacity: 24 transformer layers instead of 12, 16 attention heads instead of 12, and 1024-dimensional embeddings instead of 768.

This represents a ~3x increase in parameters (354M vs 117M), ~2x more attention heads, and ~33% larger embedding dimensions, providing significantly more computational power for learning complex historical language patterns.

graph TD
    A[Input Tokens<br/>1024 tokens] --> B[Token Embedding<br/>30K vocab → 1024 dim]
    A --> C[Position Embedding<br/>1024 pos → 1024 dim]
    B --> D[Add Embeddings]
    C --> D
    D --> E[Layer Norm]
    E --> F[Transformer Block 1<br/>16 heads, 1024 dim]
    F --> G[Transformer Block 2<br/>16 heads, 1024 dim]
    G --> H[...]
    H --> I[Transformer Block 24<br/>16 heads, 1024 dim]
    I --> J[Final Layer Norm]
    J --> K[Language Head<br/>1024 → 30K vocab]
    K --> L[Output Logits<br/>30K probabilities]
    
    subgraph "Key Specifications"
        M[Layers: 24<br/>Heads: 16<br/>Embedding: 1024<br/>Context: 1024<br/>Parameters: 354M<br/>Training: 28-32 hours<br/>MFU: 15-20%]
    end
    
    style A fill:#e1f5fe
    style L fill:#e8f5e8
    style M fill:#fff3e0

2.2 SimpleGPT Class

Now that we’ve covered the theory behind the GPT architecture, let’s examine the actual implementation. The SimpleGPT class is at the heart of our implementation - it’s the core class that brings together all the components discussed in section 2.1 into a working language model. The class inherits from PyTorch’s **torch.nn.Module**, which is the base class for all neural network components in PyTorch. This gives us access to automatic differentiation, GPU support, and other PyTorch features.

2.2.1 The __init__ method

The __init__ method is the constructor that assembles our entire language model from individual components. First, it stores all hyperparameters (such as vocabulary size, embedding dimensions, and the number of layers) in a configuration object that the rest of the model can reference. Next, it creates the embedding layers - one that converts our 30,000 historical tokens into dense vectors, and another that encodes position information. Hence, the model knows where each word appears in the sequence.

Next, it builds the transformer blocks - the core processing units that do the heavy lifting. Each block contains self-attention mechanisms and feed-forward networks that learn to understand relationships between words. The method also initializes the language modeling head, the final layer that converts all internal processing back into probabilities for which word should come next.

Finally, it sets up proper weight initialization to ensure the model starts with good random weights (not too big, not too small), and implements weight tying between the input embeddings and the output layer. This clever technique reduces the number of parameters while improving training efficiency by sharing weights between the first and last layers.

This is important because if the weights are too large, the model’s gradients can explode during training, leading to unstable learning. If they start too small, gradients can vanish, rendering the model unable to learn. Our initialization ensures the model begins in the “Goldilocks zone” - just right for effective learning. Without this, even a perfectly designed architecture might fail to train properly.

Now, let me show you the actual implementation. The code in Listing 1 demonstrates how we implement the core GPT architecture:

class SimpleGPT(torch.nn.Module):
    """Simple GPT model based on nanoGPT
    
    This class implements a decoder-only transformer model optimized for 
    historical text generation. It inherits from PyTorch's Module class
    to get automatic differentiation and GPU support.
    """
    
    def __init__(self, config):
        super().__init__()  # Initialize the parent PyTorch Module class
        self.config = config  # Store all model hyperparameters
        
        # Create the main transformer components using ModuleDict
        # ModuleDict allows us to organize related layers together
        self.transformer = torch.nn.ModuleDict(dict(
            # Token Embedding Layer (wte = "word token embedding")
            # Converts each token ID to a high-dimensional vector
            # Input: token IDs (integers 0 to vocab_size-1)
            # Output: dense vectors of size $n_{embd}$ (e.g., 768 dimensions)
            wte = torch.nn.Embedding(config.vocab_size, config.n_embd),
            
            # Position Embedding Layer (wpe = "word position embedding") 
            # Encodes where each token appears in the sequence
            # Input: position indices (0 to block_size-1)
            # Output: dense vectors of size $n_{embd}$
            wpe = torch.nn.Embedding(config.block_size, config.n_embd),
            
            # Dropout Layer for regularization
            # Randomly sets some inputs to zero during training to prevent overfitting
            drop = torch.nn.Dropout(config.dropout),
            
            # Stack of Transformer Blocks (h = "hidden layers")
            # Each SimpleBlock contains self-attention and feed-forward layers
            # We create n_layer blocks (e.g., 12 for SLM, 24 for regular model)
            h = torch.nn.ModuleList([SimpleBlock(config) for _ in range(config.n_layer)]),
            
            # Final Layer Normalization
            # Normalizes the output before the language modeling head
            ln_f = torch.nn.LayerNorm(config.n_embd, bias=config.bias),
        ))
        
        # Language Modeling Head
        # Converts the final hidden states back to vocabulary space
        # Input: hidden states of size $n_{embd}$
        # Output: logits for each token in vocabulary ($vocab_{size}$ logits)
        self.lm_head = torch.nn.Linear(config.n_embd, config.vocab_size, bias=False)
        
        # Initialize all weights using our custom initialization method
        # This ensures the model starts with good random weights
        self.apply(self._init_weights)
        
        # Weight Tying: Share weights between input embeddings and output layer
        # This technique improves training efficiency and model performance
        # by ensuring the same representation space is used for input and output
        self.transformer.wte.weight = self.lm_head.weight