TL;DR
In this second part of our 4-part series on building language models from scratch, I explore the two foundational areas of LLM development: data collection and custom tokenizer creation. Part 1 - Building LLM from Scratch covered using the published model; here, we build the complete pipeline from raw historical documents to a custom tokenizer that understands archaic English, London geography, and period-specific terminology.
The challenge with historical LLMs isn’t just having enough data—it’s having the right data processed to preserve linguistic nuances across different historical periods. This post demonstrates how to transform over 218 historical sources into a corpus of more than 500 million characters using a specialized tokenizer for authentic historical text generation.
⚠️ Educational Purpose: This is a learning project designed to teach LLM development concepts. For production-scale LLMs, you’ll need significantly larger datasets, more sophisticated infrastructure, and additional considerations that are not covered in this post.
1. The Historical Language Modeling Challenge
Building a language model for historical text presents unique challenges. Historical English from 1500 to 1850 contains linguistic patterns, vocabulary, and cultural references that modern tokenizers have never encountered. Standard tokenizers like TikToken fragment archaic words like “quoth” and “hast” into multiple subword tokens, destroying semantic meaning crucial for historical text generation.
A simple phrase like Quoth the alderman, 'Tis a fair day at Newgate becomes dozens of meaningless fragments, losing both historical context and linguistic coherence. This fragmentation is why we built a custom tokenizer trained specifically on historical English patterns, ensuring the model can generate coherent, historically accurate text.
As a reminder, both the SLM (117M parameters) and Regular Model (354M parameters) utilize the same training code and infrastructure, including GPU optimization, checkpointing, and WandB integration. The only difference lies in the model architecture parameters, which are specified in config.py.
🔗 GitHub Repository: github.com/bahree/helloLondon - Complete source code for data collection (
02_data_collection/) and tokenizer training (03_tokenizer/). We will see the relevant code snippets in this post show key concepts—see the full implementation in the repository.
What will you learn?
This project provides hands-on experience with real-world LLM development challenges, including data collection from over 218 historical sources, cleaning OCR errors and encoding issues, and developing custom tokenizers for historical text. Unlike theoretical tutorials, you receive complete, runnable code that demonstrates actual trade-offs and decisions—such as choosing BPE over WordPiece or handling different file formats—that you’d encounter in any serious LLM project.
While operating at a learning scale, the principles taught here directly apply to larger systems. Data collection patterns, cleaning strategies, and tokenizer design principles scale from our 500M character corpus to the 500B+ character datasets used in production models.
1.1 High-Level Process Overview
The complete pipeline transforms raw historical documents into a working language model through five key stages:
- Data Collection: 218+ historical sources (1500-1850), including literature, newspapers, court records, and personal diaries
- Cleaning Pipeline: Handles multiple file formats (PDF, HTML, XML, TXT) while removing OCR artifacts and preserving authentic historical language
- Quality Validation: Removes duplicates, filters non-English content, and ensures only meaningful historical text reaches the final corpus
- Custom Tokenizer Training: BPE-based tokenizer with ~150 special tokens capturing archaic pronouns, historical landmarks, and period-specific terminology
- Model Training: Two language models (SLM 117M and Regular 354M parameters) trained on the same historical corpus
The result is a system capable of generating authentic historical text that captures the linguistic patterns and cultural context of 1500-1850 English. Figure 1 illustrates this complete pipeline:
graph TD
A[📚 218+ Historical Sources<br/>1500-1850] --> B[🔍 Data Collection<br/>Download and Filter]
B --> C[🧹 5-Phase Cleaning Pipeline<br/>Format-Specific Processing]
C --> D[📊 Quality Validation<br/>Duplicate and Language Detection]
D --> E[📝 500M+ Character Corpus<br/>Clean Historical Text]
E --> F[🔤 Custom Tokenizer Training<br/>BPE with 150+ Special Tokens]
F --> G[🤖 Language Model Training<br/>SLM 117M + Regular 354M]
style A fill:#e1f5fe
style E fill:#f3e5f5
style F fill:#fff3e02. Data Collection: The Foundation of Historical Language Modeling
Let us dig deeper into steps 1-4: data collection, cleaning, validation, and corpus creation. The data collection system processes over 218 sources spanning the years 1500-1850 to create a corpus of over 500 million characters of authentic historical English text. But collecting historical data isn’t just about downloading files - it’s about handling the sheer variety of formats and quality levels that historical documents present.
Historical documents come in all shapes and sizes - scanned books with OCR errors, HTML pages with messy markup, XML archives with rich metadata, and plain text files with inconsistent encoding. This is especially true for the earlier periods, when the quality of the documents can vary significantly, and most modern techniques for processing them struggle to cope. This data diversity requires a cleaning pipeline that transforms raw historical documents into training data while preserving the authentic language patterns of 1500-1850 English.
2.1 System Architecture: Processing 218+ Historical Sources
The data collection system employs a modular architecture, with historical_data_collector.py serving as the primary orchestration engine, coordinating with a data_sources.json configuration file that contains metadata for over 218 historical sources. This enables easy management and updates without code changes.
Supporting scripts include add_data_source.py for interactive source addition with built-in validation, and generate_report.py for comprehensive reporting and analysis across multiple output formats.
The data_sources.json file contains metadata for each source, including time periods, formats, licensing, and processing priorities. Each entry includes:
time_period(e.g., [1690, 1800] for London Lives)format(XML, HTML, PDF)priority(high/medium/low)search_termsfor collection guidance
Our data sources span multiple categories, each contributing unique perspectives to the historical corpus.
Project Gutenberg: This provides foundational literature with 8+ carefully selected texts, using relaxed quality criteria that accept texts with as low as 40% meaningful words to capture the full spectrum of historical writing styles.
Historical Archives: Historical Archives like London Lives (240,000 pages of personal records) and Old Bailey (197,000+ trial transcripts) offer rich historical content and were initially enabled in our data collection.
- Note: I was using the aggressive cleaning earlier (enabled using the
aggressive_cleaningflag designed to remove structured legal data and semantic markup), and discovered that it was too aggressive and caused generation quality issues. After initial training runs revealed repetitive and incoherent text patterns, I turned off these sources. Enabling this back might be an exercise for you to try.
- Note: I was using the aggressive cleaning earlier (enabled using the
Archive.org: Archive.org has an API access that can be used for file filtering, and this makes it relatively straightforward.
The National Archives (TNA): TNA records contribute government correspondence and official documents that provide the institutional context for historical events.
British History Online: Finally, these supplements our collection with historical surveys and period documents that offer scholarly perspectives on the time periods we’re modeling.
However, each source type presents unique technical challenges that require specialized processing approaches. One example is Project Gutenberg, which contains files with standardized headers and footers that must be removed. (As a side note, I really appreciate the effort that has gone into this to make this formatting consistent, which makes the process of this relatively straightforward.)
On the other hand, PDF files often suffer from OCR errors, especially for older documents that contain corrupted historical language, requiring sophisticated text correction algorithms to restore proper spelling and grammar from scanned documents. The figure below shows one example of how older documents look. This example is “The abridgment of the charter of the city of London” from 1680.
As you can see, the text is faded, has ink blots, and the font style is very different from modern text. OCR software often misinterprets characters in such documents, resulting in numerous errors, as illustrated in the image below. These OCR artifacts can severely degrade the quality of our training data if not properly addressed.
HTML files from sources like Archive.org contain navigation elements, advertisements, and modern web markup that contaminate the historical corpus, demanding careful content extraction that preserves only the meaningful historical text.
XML archives like London Lives and Old Bailey require specialized parsing to extract meaningful text while preserving semantic markup that provides context about speakers, dates, and document structure - a delicate balance between removing technical artifacts and maintaining historical authenticity.
Government records from TNA often contain bureaucratic formatting, form fields, and institutional language that need careful filtering to extract the human stories and historical narratives.
British History Online documents present challenges with academic formatting, footnotes, and scholarly apparatus that must be processed to maintain readability while preserving the scholarly context that makes them valuable for historical language modeling.
2.2 Cleaning Pipeline
I implement a 5-stage cleaning pipeline that helps transform the raw historical documents into training-ready text. Each phase addresses specific challenges that would otherwise contaminate our language model training.
2.2.1 Stage 1: File Discovery & Initial Filtering
Historical archives often contain files in various formats, which may be missing proper file extensions or have non-standard naming conventions. Many files contain non-English content that would contaminate our English historical corpus. Additionally, many sources employ their own templates and standards for this purpose. To resolve this, we first implement a simple file detection and naming cleanup, as shown in Listing 1. The code itself is simple and self-explanatory.
def detect_file_type(file_path: str) -> str:
"""Detect file type based on extension and content analysis"""
# Extension-based detection
if file_path.endswith(('.txt', '.txt.utf-8', '_txt.utf-8')):
return 'text'
elif file_path.endswith(('.pdf',)):
return 'pdf'
elif file_path.endswith(('.html', '.htm')):
return 'html'
elif file_path.endswith(('.xml',)):
return 'xml'
# Content-based detection for files without extensions
with open(file_path, 'rb') as f:
content = f.read(1024) # Read first 1KB
if b'<html' in content.lower() or b'<!doctype' in content.lower():
return 'html'
elif b'<?xml' in content.lower():
return 'xml'
elif content.isascii() and b'\x00' not in content:
return 'text'
else:
return 'binary'When we run this locally, we will see the flow as outlined below, which illustrates how the detection works. This, of course, can be made more robust for non-English characters, but for now, we reject these.
File Type Detection Flow:
📁 Raw Files (218+ sources)
↓
🔍 File Type Detection
├── .txt, .txt.utf-8, _txt.utf-8 → Text Processing
├── .pdf → PDF Processing
├── .html, .htm → HTML Processing
├── .xml → XML Processing (Old Bailey, London Lives)
└── No Extension → Content Detection
├── HTML-like content → HTML Processing
├── Text-like content → Text Processing
└── Binary/Unknown → REJECTED
↓
🚫 Filename Language Check
├── Non-English characters → REJECTED (logged)
└── English/Latin → ContinueHistorical archives often lack standardized file extensions and contain content in languages other than English. Our two-stage detection ensures we capture valuable historical documents while filtering out irrelevant files, preventing both data loss and processing waste.
2.2.2 Stage 2: Format-Specific Content Extraction
Each file format requires specialized processing due to its unique contamination sources, including Project Gutenberg headers, PDF OCR errors, HTML navigation elements, and XML structural markup. Our format-specific extraction functions clean these artifacts while preserving authentic historical content.
Text Files (.txt, .txt.utf-8)
Project Gutenberg texts contain standardized headers and footers that would confuse our language model. The cleaning process removes these while preserving the actual historical content. The code snippet in Listing 2 demonstrates this approach and is quite straightforward. Of course, this can be made more robust, but this works well for our selected texts.
def clean_gutenberg_text(text: str) -> str:
"""Clean Project Gutenberg text by removing headers/footers and metadata"""
lines = text.split('\n')
cleaned_lines = []
in_content = False
for line in lines:
# Skip Gutenberg headers (before "*** START OF")
if "*** START OF" in line:
in_content = True
continue
# Skip Gutenberg footers (after "*** END OF")
if "*** END OF" in line:
break
# Skip metadata lines
if line.startswith(('Title:', 'Author:', 'Release Date:', 'Language:')):
continue
# Skip empty lines at start
if not in_content and not line.strip():
continue
if in_content:
cleaned_lines.append(line)
return '\n'.join(cleaned_lines).strip()Real Example - Before Cleaning:
Title: A Journal of the Plague Year
Author: Daniel Defoe
Release Date: March 2003
Language: English
*** START OF THE PROJECT GUTENBERG EBOOK A JOURNAL OF THE PLAGUE YEAR ***
It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland...
*** END OF THE PROJECT GUTENBERG EBOOK A JOURNAL OF THE PLAGUE YEAR ***After Cleaning:
It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland...Without this cleaning, the model would learn to generate Gutenberg headers and metadata instead of authentic historical text, contaminating the training data with modern digital artifacts.
PDF Files
PDF files from historical archives often contain OCR errors and digital artifacts that require correction. The cleaning process in Listing 3 addresses these issues while preserving historical content, removing page numbers and all-caps headers. While not perfect, it significantly improves text quality.
The OCR correction rules are based on common patterns in historical documents and can be refined for specific datasets. Libraries like PyMuPDF or pdfplumber extract text, while regex-based cleaning corrects common OCR errors and removes digital stamps. More advanced techniques, such as layout analysis or AI-based OCR correction, can further enhance this process.
def clean_pdf_text(text: str) -> str:
"""Clean PDF text by removing OCR artifacts and digital stamps"""
# Remove page numbers: [Page 123], standalone numbers
text = re.sub(r'\[Page \d+\]', '', text)
text = re.sub(r'^\d+$', '', text, flags=re.MULTILINE)
# Remove library stamps: Internet Archive, Google, etc.
stamps = [
'Internet Archive', 'Google Books', 'HathiTrust',
'Digitized by Google', 'Scanned by Google'
]
for stamp in stamps:
text = text.replace(stamp, '')
# Fix common OCR artifacts
ocr_fixes = {
r'\b0\b': 'O', # 0 → O
r'\b1\b': 'I', # 1 → I
r'\b5\b': 'S', # 5 → S
r'\b8\b': 'B', # 8 → B
r'\brn\b': 'm', # rn → m
r'\bcl\b': 'd' # cl → d
}
for pattern, replacement in ocr_fixes.items():
text = re.sub(pattern, replacement, text)
# Remove all-caps lines (usually headers)
lines = text.split('\n')
cleaned_lines = [line for line in lines if not line.isupper() or len(line) < 10]
return '\n'.join(cleaned_lines)Real Example - Before Cleaning:
[Page 1]
INTERNET ARCHIVE
A JOURNAL OF THE PLAGUE YEAR
BY DANIEL DEFOE
It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland. For it was indeed a very terrible time, and the people began to be very much alarmed at it.After Cleaning:
A JOURNAL OF THE PLAGUE YEAR
BY DANIEL DEFOE
It was about the beginning of September 1664, that I, among the rest of my neighbours, heard in ordinary discourse that the plague was returned again in Holland. For it was indeed a very terrible time, and the people began to be very much alarmed at it.OCR errors can significantly impact the quality of model training. For example, if London appears as L0nd0n due to OCR errors, the model won’t learn the correct spelling and will generate nonsensical text when asked about historical London. The correction process ensures our model learns authentic historical language patterns rather than digital artifacts, which is crucial for generating coherent and historically accurate text.
HTML Files
HTML files from historical websites and digital archives contain markup that needs to be stripped while preserving the actual text content. We use the BeautifulSoup library in Listing 4 to clean the HTML structure and extract only the meaningful text.
def clean_html_text(html_content: str) -> str:
"""Clean HTML content by removing markup and extracting text"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html_content, 'html.parser')
# Remove unwanted elements
for element in soup(['script', 'style', 'nav', 'header', 'footer',
'aside', 'menu', 'form', 'input', 'button']):
element.decompose()
# Remove wiki-specific elements
for element in soup.find_all(['div', 'span'], class_=['navbox', 'infobox', 'sidebar']):
element.decompose()
# Remove navigation elements
for element in soup.find_all(['div', 'ul'], class_=['breadcrumb', 'navigation', 'menu']):
element.decompose()
# Extract text content
text = soup.get_text(separator=' ', strip=True)
# Clean up excessive whitespace
text = re.sub(r'\s+', ' ', text)
return text.strip()Real Example - Before Cleaning:
<!DOCTYPE html>
<html>
<head><title>London History</title></head>
<body>
<nav>Home | About | Contact</nav>
<header>London Historical Society</header>
<div class="content">
<h1>The Great Fire of London</h1>
<p>In the year 1666, a great fire consumed much of London...</p>
</div>
<footer>© 2024 London Historical Society</footer>
</body>
</html>After Cleaning:
The Great Fire of London in the year 1666, a great fire consumed much of London...HTML tags and navigation elements would contaminate training, causing the model to generate markup instead of historical text. Our cleaning process extracts meaningful content while preserving natural flow and structure.
XML Files (Historical Archives):
XML files from historical archives, such as the Old Bailey and London Lives, use specific schemas that require specialized parsing. Old Bailey employs TEI (Text Encoding Initiative) with TEI.2 elements, while London Lives uses semantic markup (name, geo, occupation). These structured formats contain authentic historical language with rich metadata, as shown in Listing 5.
def extract_old_bailey_text(soup) -> str:
"""Extract text from Old Bailey XML using TEI schema structure"""
extracted_text = []
# Check for TEI.2 elements (Old Bailey schema)
tei_elements = soup.find_all('TEI.2')
if tei_elements:
# Extract trial accounts (main narrative content)
trial_accounts = soup.find_all('div1', {'type': 'trialAccount'})
for trial in trial_accounts:
trial_text = extract_trial_narrative(trial)
if trial_text:
extracted_text.append(trial_text)
# Extract front matter (session information)
front_matter = soup.find_all('div1', {'type': 'frontMatter'})
for front in front_matter:
front_text = extract_front_matter_narrative(front)
if front_text:
extracted_text.append(front_text)
return '\n\n'.join(extracted_text)
def extract_london_lives_text(soup) -> str:
"""Extract text from London Lives XML using semantic markup schema"""
extracted_text = []
# Check for London Lives specific elements (name, geo, occupation, date)
name_elements = soup.find_all('name')
geo_elements = soup.find_all('geo')
occupation_elements = soup.find_all('occupation')
if name_elements and geo_elements and occupation_elements:
# Extract paragraphs with semantic markup
paragraphs = soup.find_all('p')
for para in paragraphs:
p_text = extract_paragraph_with_semantic_markup(para)
if p_text.strip():
extracted_text.append(p_text)
return '\n\n'.join(extracted_text)Real Example - Old Bailey XML (Before Processing):
<trial>
<frontmatter>
<session>Session 1</session>
<date>1674-04-15</date>
<location>Old Bailey</location>
</frontmatter>
<proceedings>
The prisoner being brought to the bar, and the indictment being read, he pleaded Not Guilty. The witnesses being sworn, the first witness deposed that on the 15th day of April last, he saw the prisoner in the company of several suspicious persons...
</proceedings>
</trial>After Processing:
Session 1 1674-04-15 Old Bailey The prisoner being brought to the bar, and the indictment being read, he pleaded Not Guilty. The witnesses being sworn, the first witness deposed, that on the 15th day of April last, he saw the prisoner in the company of several suspicious persons...These XML files contain the most authentic historical language in our entire dataset. The Old Bailey trials show how people actually spoke in court during the 17th-19th centuries, while London Lives reveals the everyday language used in personal records and official documents. This authentic historical language is very useful for training a model that can generate historically accurate text, as it provides the model with genuine examples of how people wrote and spoke during different historical periods.
2.2.3 Stage 3: Text Normalization
After extraction, text normalization ensures consistency and compatibility with the training data. Historical documents contain encoding issues, inconsistent formatting, and special characters that confuse the model. Our normalization process fixes these issues and breaks long lines to fit within the model’s context window. This is critical because lines exceeding the context window appear as incomplete sentences to the transformer, severely degrading generation quality due to the attention mechanism’s inability to process fragmented text.
Inconsistent encoding and formatting can severely confuse the language model during training. For example, if some files use smart quotes (") and others use straight quotes ("), the model might not learn that they represent the same concept, leading to inconsistent and potentially incorrect text generation. Normalization ensures that the model observes consistent patterns across all training data, which is crucial for learning coherent language patterns and generating high-quality historical text.
The code snippet in Listing 6 demonstrates how we implement this normalization, which is quite straightforward.
def normalize_text(text: str) -> str:
"""Normalize text for consistent training data"""
import unicodedata
# Fix common encoding issues
encoding_fixes = {
'’': "'", # Smart apostrophe
'“': '"', # Smart quote left
'â€': '"', # Smart quote right
'â€"': '—', # Em dash
'•': '•', # Bullet point
'…': '…', # Ellipsis
}
for old, new in encoding_fixes.items():
text = text.replace(old, new)
# Normalize Unicode (NFC)
text = unicodedata.normalize('NFC', text)
# Break long lines for training compatibility (max 2000 chars)
lines = text.split('\n')
normalized_lines = []
for line in lines:
if len(line) > 2000:
# Split at sentence boundaries
sentences = re.split(r'(?<=[.!?])\s+', line)
current_line = ""
for sentence in sentences:
if len(current_line + sentence) > 2000:
if current_line:
normalized_lines.append(current_line.strip())
current_line = sentence
else:
current_line += " " + sentence if current_line else sentence
if current_line:
normalized_lines.append(current_line.strip())
else:
normalized_lines.append(line)
# Normalize line endings and whitespace
text = '\n'.join(normalized_lines)
text = re.sub(r'[ \t]+', ' ', text) # Multiple spaces/tabs to single space
text = re.sub(r'\n\s*\n', '\n\n', text) # Multiple newlines to double newline
return text.strip()Real Example - Before Normalization:
The year was 1666, and the plague had come to London. “It was indeed a very terrible time,†wrote one observer. The streets were filled with the sounds of horse-drawn carriages and the cries of the afflicted.After Normalization:
The year was 1666, and the plague had come to London. "It was indeed a very terrible time," wrote one observer. The streets were filled with the sounds of horse-drawn carriages and the cries of the afflicted.2.2.4 Stage 4: Quality Validation
Not all extracted text is suitable for training. Some files contain duplicates, non-English content, or poor-quality text that would degrade model performance. We need a comprehensive validation system that ensures only high-quality, relevant text is included in our training corpus.
The key challenge is striking a balance between quality standards and historical value. A strict approach might reject valuable historical documents that have some OCR issues, while a lenient approach might include too much low-quality content, which can degrade model training. To address this, I implemented a tiered quality threshold system that applies different standards based on content type:
- General Content: 200+ chars, 50+ words, 50% meaningful words
- Project Gutenberg: 200+ chars, 50+ words, 40% meaningful words (relaxed for historical value)
- Historical Documents: 1000+ chars, 100+ words, 30% meaningful words (very relaxed for historical value)
This tiered approach ensures that we capture valuable historical content while maintaining quality standards, filtering out duplicates, non-English content, and low-quality text, thereby preserving the integrity of useful historical documents. Again, these implementations are quite simple, in the context of a toy project, but can be made more robust. The code itself is quite straightforward, as shown in Listing 7.
def analyze_text_quality(text: str, source_type: str = 'general') -> dict:
"""Analyze text quality and determine if it should be included in training corpus"""
import hashlib
import re
# Length validation
char_count = len(text)
word_count = len(text.split())
# OCR artifact detection using regex patterns
ocr_patterns = {
'long_capitals': r'[A-Z]{5,}\s+[A-Z]{5,}',
'spaced_letters': r'\b[A-Za-z]\s+[A-Za-z]\s+[A-Za-z]\s+[A-Za-z]\b',
'special_chars': r'[!@#$%^&*()]{3,}',
'mixed_alphanumeric': r'\b\d+[A-Za-z]+\d+\b',
'long_non_word': r'[^\w\s]{10,}'
}
ocr_issues = []
for pattern_name, pattern in ocr_patterns.items():
if re.search(pattern, text):
ocr_issues.append(pattern_name)
# Advertisement detection
ad_patterns = [
'this day is published', 'just ready', 'elegantly bound',
'now ready', 'new novels', 'advertisements',
'price \d+s', 'paternoster row', 'corner of', 'publishers'
]
ad_count = sum(1 for pattern in ad_patterns if re.search(pattern, text, re.IGNORECASE))
ad_density = ad_count / max(word_count, 1)
# Meaningful word ratio calculation
words = text.split()
meaningful_words = [w for w in words if w.isalpha() and len(w) > 2]
meaningful_ratio = len(meaningful_words) / max(len(words), 1)
# Quality thresholds based on source type
thresholds = {
'general': {'min_chars': 200, 'min_words': 50, 'min_meaningful_ratio': 0.50},
'gutenberg': {'min_chars': 200, 'min_words': 50, 'min_meaningful_ratio': 0.40},
'historical': {'min_chars': 1000, 'min_words': 100, 'min_meaningful_ratio': 0.30}
}
threshold = thresholds.get(source_type, thresholds['general'])
# Quality scoring
score = 100
score -= len(ocr_issues) * 3 # OCR issues
score -= ad_density * 50 # Advertisement density
score -= (1 - meaningful_ratio) * 20 # Meaningful word ratio
# Check if text meets quality thresholds
meets_thresholds = (
char_count >= threshold['min_chars'] and
word_count >= threshold['min_words'] and
meaningful_ratio >= threshold['min_meaningful_ratio'] and
ad_density < 0.1 # Less than 10% advertisement content
)
return {
'char_count': char_count,
'word_count': word_count,
'meaningful_ratio': meaningful_ratio,
'ocr_issues': ocr_issues,
'ad_density': ad_density,
'score': score,
'meets_thresholds': meets_thresholds,
'content_hash': hashlib.md5(text.encode()).hexdigest()
}Content Quality Validation
Our validation system employs multiple detection mechanisms to ensure training corpus quality:
- OCR Artifact Detection: Regex patterns identify common digitization errors, including misread headers, character separation failures, scanning artifacts, alphanumeric misinterpretations, and corrupted text regions
- Advertisement Filtering: Pattern matching detects commercial content using phrases like “this day is published”, “just ready”, “elegantly bound”, and price references
- Quality Scoring: A 100-point system deducts points for OCR artifacts (-3 each), advertisement density (-50), and low meaningful word ratios (-20)
This multi-layered approach balances quality standards with preservation of valuable historical content, ensuring the model trains on authentic historical language while filtering out contamination sources.
2.2.5 Stage 5: Final Processing and Corpus Creation
After cleaning and validation, we create a final training corpus optimized for language model training. This requires intelligent segmentation that breaks long texts into manageable chunks while preserving the historical narrative flow, which is essential given the context window limits (e.g., 2048 tokens). The code snippet in Listing 8 demonstrates this final processing stage.
def create_comprehensive_corpus(cleaned_files: list) -> str:
"""Create final training corpus with intelligent segmentation"""
corpus_parts = []
for file_path in cleaned_files:
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# Split into training segments
segments = split_into_training_segments(content)
corpus_parts.extend(segments)
# Create final corpus
final_corpus = '\n\n'.join(corpus_parts)
# Save to file
with open('london_historical_corpus_comprehensive.txt', 'w', encoding='utf-8') as f:
f.write(final_corpus)
return final_corpus
def split_into_training_segments(text: str, max_length: int = 2000) -> list:
"""Split text into training segments while preserving narrative flow"""
# First split on double newlines (paragraphs)
paragraphs = text.split('\n\n')
segments = []
current_segment = ""
for paragraph in paragraphs:
if len(current_segment + paragraph) <= max_length:
current_segment += paragraph + '\n\n'
else:
if current_segment:
segments.append(current_segment.strip())
current_segment = paragraph + '\n\n'
if current_segment:
segments.append(current_segment.strip())
# Further split long segments at sentence boundaries
final_segments = []
for segment in segments:
if len(segment) > max_length:
sentences = re.split(r'(?<=[.!?])\s+', segment)
current_segment = ""
for sentence in sentences:
if len(current_segment + sentence) <= max_length:
current_segment += sentence + " "
else:
if current_segment:
final_segments.append(current_segment.strip())
current_segment = sentence + " "
if current_segment:
final_segments.append(current_segment.strip())
else:
final_segments.append(segment)
# Filter out segments that are too short
return [seg for seg in final_segments if len(seg) >= 50]During my local runs, this final processing stage generated a comprehensive corpus of over 500 million characters across ~250,000 segments, with an average segment length of around 2,000 characters. The success rate of files making it into the final corpus ranged from 70% to 90%, depending on the quality and availability of the source.
Final Corpus Statistics:
- Total Sources Processed: 218+ historical sources
- Final Corpus Size: 500M+ characters
- Training Segments: ~250,000 segments
- Average Segment Length: ~2,000 characters
- Success Rate: 70-90% (depending on source availability)
2.3 Detailed Data Processing Flow
Building on the high-level flow and having reviewed each of the areas, the detailed flow below illustrates the complete data cleaning process, including rejection paths, error handling, and statistics tracking. This is intended to provide a bird’s-eye view of the entire process.
graph TD
A[📁 Raw Files] --> B{File Type Detection}
B -->|.txt, .txt.utf-8| C[📄 Text File]
B -->|.pdf| D[📄 PDF File]
B -->|.html, .htm| E[📄 HTML File]
B -->|.xml| F[📄 XML File]
B -->|No Extension| G{Content Detection}
G -->|HTML-like| E
G -->|Text-like| C
G -->|Binary/Unknown| REJECT1[❌ REJECTED]
C --> H[🧹 clean_gutenberg_text]
D --> I[🔧 extract_text_from_pdf]
E --> J[🧹 clean_html_text]
F --> K{XML Type Detection}
I --> L[🧹 clean_pdf_text]
K -->|Old Bailey| M[🔧 extract_old_bailey_text]
K -->|London Lives| N[🔧 extract_london_lives_text]
M --> O[🧹 clean_old_bailey_text]
N --> P[🧹 clean_london_lives_text]
H --> Q[🔧 normalize_text]
L --> Q
J --> Q
O --> Q
P --> Q
Q --> R[🔍 Duplicate Detection]
R -->|Duplicate| REJECT2[❌ REJECTED - Duplicate]
R -->|Unique| S[🌍 Language Detection]
S -->|Non-English| REJECT3[❌ REJECTED - Non-English]
S -->|English| T[📊 Quality Analysis]
T --> U{Quality Check}
U -->|Poor Quality| REJECT4[❌ REJECTED - Poor Quality]
U -->|Good Quality| V[💾 Save to Processed Directory]
V --> W[📊 Update Statistics]
W --> X[✅ Successfully Processed]
REJECT1 --> Y[📝 Log Rejection Reason]
REJECT2 --> Y
REJECT3 --> Y
REJECT4 --> Y
Y --> Z[📊 Update Rejection Stats]
style A fill:#e1f5fe
style X fill:#c8e6c9
style REJECT1 fill:#ffcdd2
style REJECT2 fill:#ffcdd2
style REJECT3 fill:#ffcdd2
style REJECT4 fill:#ffcdd2
style Y fill:#fff3e0
style Z fill:#fff3e02.5 Corpus Creation Process
After cleaning, the system creates the final training corpus through intelligent segmentation that preserves historical narrative flow:
📁 Cleaned Files
↓
🔧 create_comprehensive_corpus()
├── Read all cleaned_*.txt files
├── Split into training segments (split_into_training_segments)
│ ├── Split on double newlines (paragraphs)
│ ├── Max length: 2000 characters
│ ├── Min length: 100 characters
│ └── Further split long segments at sentence boundaries
├── Filter segments (min 50 characters)
└── Write to london_historical_corpus_comprehensive.txtThe corpus creation process reads all cleaned text files and intelligently segments them into training-ready chunks. It first splits on double newlines to preserve paragraph boundaries, which are natural break points in historical text. Segments are constrained to a maximum of 2000 characters to fit within the model’s context window, with a minimum of 100 characters to ensure substantial content. Long segments are further split at sentence boundaries to maintain readability. Finally, segments shorter than 50 characters are filtered out as they’re unlikely to contain meaningful historical content.
Proper segmentation is crucial for training language models. The model needs to learn from coherent text segments that maintain historical narrative flow while fitting within its context window. Splitting on paragraph boundaries preserves the natural structure of historical documents, while sentence-level splitting ensures that very long paragraphs don’t exceed the model’s processing capabilities. This approach maximizes the model’s ability to learn from authentic historical language patterns while maintaining training efficiency.
2.6 Outcome: Training-Ready Corpus
The result is a clean, historically faithful corpus containing over 500 million characters of authentic historical English spanning 350 years of London history from 1500-1850. The corpus comprises high-quality text with minimal OCR artifacts, preserving historical language patterns and a rich cultural context that reflects the social, political, and economic realities of various historical periods. The text has been intelligently segmented for optimal language model training, with careful attention to maintaining the natural flow of historical narratives while ensuring compatibility with modern training techniques.
This corpus serves as the essential foundation for training our specialized historical tokenizer and language model, ensuring the model learns authentic historical English rather than modern text patterns. By providing the model with genuine examples of how people wrote and spoke during different historical periods, we enable it to generate text that captures the linguistic nuances, cultural references, and historical context that make historical language modeling both challenging and rewarding.
💻 Try It Yourself: The complete implementation, including all the data collection scripts, cleaning algorithms, and quality validation systems described in this section, is available in the helloLondon GitHub repository . The repository includes detailed documentation, example usage, and step-by-step guides for setting up your own historical language model training pipeline.
Now that we have examined the data collection and cleaning process, we can proceed to the next steps: creating a custom historical tokenizer and preparing for model training.
3. Custom Historical Tokenizer: The Key to Authentic Historical Text Generation
Creating a custom tokenizer is crucial for generating effective historical text. This section examines the necessity of a custom tokenizer, the challenges presented by historical language, and our chosen architecture. The tokenizer preserves the semantic meaning of historical words and phrases, enabling coherent and contextually accurate historical narratives.
Standard tokenizers like GPT-2’s fragment archaic words like “quoth” and “hast” into multiple subword tokens, destroying semantic meaning crucial for historical text generation.
Real Example - Standard Tokenizer vs. Our Custom Tokenizer:
Standard GPT-2 Tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"
→ ['Qu', 'oth', ' the', ' ald', 'erman', ',', ' ', ''', 'T', 'is', ' a', ' fair', ' day', ' at', ' New', 'gate']Our Custom Historical Tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"
→ ['<|quoth|>', ' the', ' alderman', ',', ' ', ''', '<|tis|>', ' a', ' fair', ' day', ' at', ' <|newgate|>']The standard tokenizer breaks historical language into 18 meaningless fragments, losing semantic meaning and historical context. Our custom tokenizer reduces this to 12 meaningful tokens, preserving authentic historical language patterns essential for coherent text generation.
A tokenizer that fragments historical language destroys the model’s ability to learn authentic patterns. The model needs to perceive “quoth” as a single concept, rather than fragmented subwords, to capture the linguistic nuances of different historical periods.
3.1 What Happens with Off-the-Shelf Tokenizers
What would happen if we used standard tokenizers like tiktoken or GPT-2’s tokenizer?
Standard tokenizers would force the model to waste capacity reconstructing fragmented historical words from subwords rather than learning historical language patterns. The model might learn to generate “Qu” + “oth” but struggle to use “quoth” in new contexts. Historical phrases like “methinks” would split into meaningless fragments, losing semantic coherence. London geography becomes particularly problematic, as place names like “Newgate” fragment, making spatial relationships harder to understand.
Generation Quality Issues:
# What you'd get with standard tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"
→ Generates: "Qu oth the ald erman, 'T is a fair day at New gate"
→ Result: Broken, unreadable historical text
# What you get with our custom tokenizer:
"Quoth the alderman, 'Tis a fair day at Newgate"
→ Generates: "Quoth the alderman, 'Tis a fair day at Newgate"
→ Result: Authentic, coherent historical textA vocabulary that’s too small (10K tokens) would fragment even more historical words, making the problem worse, while a vocabulary that’s too large (100K+ tokens) would overfit to rare historical terms, wasting capacity on words that appear only once. Our choice of 30K tokens provides a balanced approach that captures common historical patterns without overfitting, ensuring the model learns the most important historical language patterns efficiently.
Real-World Example: With a standard tokenizer, our model might generate:
"The ald erman walk ed to New gate where he saw the pris oner"
With our custom tokenizer, it generates:
"The alderman walked to Newgate where he saw the prisoner"
The difference in historical text authenticity is significant between the two approaches.
3.2 Tokenizer Architecture
I had started with the easier WordPiece tokenizer (more of an accident rather than by design). Still, I realized later that it was unsuitable for historical text due to the ## subword prefix artifacts. We need a tokenizer that can handle historical English efficiently while preserving semantic meaning, unlike standard tokenizers like GPT-2’s WordPiece approach, which fragments historical language and, as a result, destroys the linguistic patterns we want to preserve. After some experimentation, I settled on a custom Byte Pair Encoding (BPE) tokenizer trained specifically on historical English.
BPE is a subword tokenization algorithm that learns to break text into meaningful subword units by iteratively finding the most frequent character pairs in the training corpus and merging them into single tokens. The process begins with individual characters and gradually evolves into common words and phrases.
For example, if "th" appears frequently in our historical corpus, BPE will learn to treat it as a single token rather than separate "t" and "h" tokens. This is particularly valuable for historical English, where words like "thou", "thee", and "thine" share common prefixes and suffixes.
3.2.1 Tokenizer Training Process
The BPE training algorithm analyzes our entire historical corpus to identify the most frequent character combinations, building a vocabulary that’s optimized for historical language patterns. We start with a base alphabet (comprising all letters) and special tokens, then iteratively merge the most frequent pairs until we reach our target vocabulary size of 30,000 tokens. This ensures that common historical words, such as "quoth", "hast", and "methinks", are treated as single tokens, while still allowing for the handling of rare or unknown words by breaking them into learned subword units.
The training process is computationally efficient and produces a tokenizer that’s specifically tuned to the linguistic patterns found in our historical corpus.
In this case, we don’t have to reinvent the wheel and use the Hugging Face tokenizers library, which provides a modular approach to building custom tokenizers. The library is organized into several key components: models define the core tokenization algorithm (BPE, WordPiece, Unigram), pre_tokenizers handle initial text splitting, normalizers clean and standardize text, trainers configure the learning process, and processors handle special token insertion. This modular design enables us to mix and match components to create a tokenizer tailored to our specific use case.
The models module offers several tokenization algorithms: BPE() for Byte Pair Encoding (what we use), WordPiece() for Google’s WordPiece algorithm, Unigram() for Google’s Unigram language model, and WordLevel() for simple word-level tokenization.
Each has different strengths - BPE is efficient and handles unknown words well, WordPiece is used by BERT but creates ## artifacts, Unigram is more flexible but computationally expensive, and WordLevel is simple but creates very large vocabularies.
Let us look at the code in Listing 9 for training our custom historical BPE tokenizer:
def train_tokenizer(self):
"""Train a custom tokenizer for historical English"""
# Import the tokenizers library components
from tokenizers import Tokenizer, models, pre_tokenizers, processors, trainers
from tokenizers.normalizers import Sequence, NFD, StripAccents
logger.info("Training custom historical tokenizer...")
logger.info(f"Corpus: {self.corpus_path}")
logger.info(f"Target vocabulary: {self.vocab_size:,} tokens")
logger.info(f"Output directory: {self.output_dir}")
# Initialize BPE tokenizer (not WordPiece)
# models.BPE() creates a Byte Pair Encoding model that will learn subword patterns
tokenizer = Tokenizer(models.BPE())
# Normalizers for historical text - preserve case for better text reconstruction
# Normalizers clean and standardize text before tokenization
tokenizer.normalizer = Sequence([
NFD(), # Unicode normalization - converts characters to canonical form
StripAccents() # Remove accents - converts "café" to "cafe"
])
# Pre-tokenizer for historical English - use simple whitespace splitting
# Pre-tokenizers split text into initial segments before the main tokenization
tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
pre_tokenizers.WhitespaceSplit(), # Split on whitespace
pre_tokenizers.Punctuation() # Split punctuation from words
])
# Special tokens for historical English
special_tokens = [
"<|startoftext|>", "<|endoftext|>", "<|pad|>", "<|unk|>", "<|mask|>",
# Historical language tokens
"<|thou|>", "<|thee|>", "<|thy|>", "<|thine|>", "<|hast|>", "<|hath|>",
"<|doth|>", "<|dost|>", "<|quoth|>", "<|tis|>", "<|twas|>", "<|twill|>",
# London geography tokens
"<|london|>", "<|thames|>", "<|westminster|>", "<|tower|>", "<|newgate|>",
"<|southwark|>", "<|cheapside|>", "<|fleet|>", "<|ludgate|>", "<|aldgate|>",
# Historical period tokens
"<|tudor|>", "<|stuart|>", "<|georgian|>", "<|regency|>", "<|victorian|>",
# Social class tokens
"<|noble|>", "<|gentleman|>", "<|commoner|>", "<|apprentice|>", "<|yeoman|>",
# Professional tokens
"<|apothecary|>", "<|coachman|>", "<|chimneysweep|>", "<|baker|>", "<|butcher|>"
]
# BPE trainer configuration
# The trainer defines how the BPE algorithm learns from our corpus
trainer = trainers.BpeTrainer(
vocab_size=self.vocab_size, # Target vocabulary size (30,000 tokens) - balanced between coverage and efficiency
special_tokens=special_tokens, # Pre-defined tokens that are always included
min_frequency=2, # Minimum frequency prevents vocabulary pollution from OCR errors
show_progress=True, # Display training progress
# Removed continuing_subword_prefix="##" to eliminate WordPiece-style artifacts
# This ensures pure BPE tokenization without ## symbols in generated text
initial_alphabet=["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l", "m",
"n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z",
"A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M",
"N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"]
)
# Train the tokenizer on our historical corpus
# This is where the BPE algorithm learns the optimal subword patterns
tokenizer.train([str(self.corpus_path)], trainer)
return tokenizer3.2.2 Tokenization Architecture Decisions
Our custom historical tokenizer necessitated several critical design decisions to handle historical English effectively. We evaluated multiple tokenization approaches including Byte Pair Encoding (BPE) (
Sennrich et al., 2016
), WordPiece (
Schuster & Nakajima, 2012
), Unigram Language Model (
Kudo, 2018
), SentencePiece (
Kudo & Richardson, 2018
), and traditional character-level and word-level tokenization. Each approach has distinct trade-offs: BPE produces clean subwords without special markers (used by GPT models), WordPiece adds ## prefixes that contaminate generated text (used by BERT), Unigram uses probabilistic modeling but is computationally expensive, SentencePiece treats text as raw bytes and excels at multilingual scenarios, while character-level and word-level tokenization either produce impractically long sequences or massive vocabularies.
For historical text generation, BPE provides the optimal balance of clean output, efficient training, and effective vocabulary coverage, as demonstrated by Radford et al., 2019 and Mielke et al., 2021 . We also preserve case throughout tokenization, since historical text often uses capitalization for semantic meaning (e.g., “Thou” vs. “thou”), and include over 150 carefully designed special tokens that capture historical language patterns, London geography, and social context. This combination ensures our tokenizer can effectively learn and generate authentic historical language while maintaining computational efficiency.
3.3 Special Token Design: Capturing Historical Language Patterns
Historical English contains linguistic patterns, vocabulary, and cultural references that are no longer present in modern English. Standard tokenizers fragment these patterns, destroying the semantic meaning crucial for historical text generation. The solution here was to design 150 special tokens that capture the essence of historical English, organized into strategic categories that reflect the linguistic and cultural structure of 1500-1850 English.
def create_special_tokens() -> list:
"""Create special tokens for historical English"""
special_tokens = [
# Basic control tokens
"<|startoftext|>", "<|endoftext|>", "<|pad|>", "<|unk|>", "<|mask|>",
# Historical language tokens
"<|thou|>", "<|thee|>", "<|thy|>", "<|thine|>", # Second person pronouns
"<|hast|>", "<|hath|>", "<|doth|>", "<|dost|>", # Archaic verb forms
"<|quoth|>", "<|tis|>", "<|twas|>", "<|twill|>", # Common contractions
# London geography tokens
"<|london|>", "<|thames|>", "<|westminster|>", "<|tower|>", "<|newgate|>",
"<|southwark|>", "<|cheapside|>", "<|fleet|>", "<|ludgate|>", "<|aldgate|>",
# Historical period tokens
"<|tudor|>", "<|stuart|>", "<|georgian|>", "<|regency|>", "<|victorian|>",
# Social and professional tokens
"<|noble|>", "<|gentleman|>", "<|commoner|>", "<|apothecary|>", "<|coachman|>",
"<|merchant|>", "<|court|>", "<|jury|>", "<|verdict|>", "<|church|>", "<|parish|>"
]
return special_tokens3.3.1 Token Category Analysis
Our special token vocabulary spans ten carefully curated categories, each designed to capture essential aspects of historical London life. The largest categories focus on Historical Language (25 tokens) and London Geography (20 tokens), providing the linguistic and spatial foundation for authentic historical text generation. These tokens capture archaic pronouns like "thou" and "thee," along with specific London locations like "Thames" and "Newgate" that were central to historical narratives.
The remaining categories address the social, professional, and cultural dimensions of historical society. Social Class and Professional tokens (35 tokens combined) reflect the highly stratified nature of historical London, enabling accurate dialogue between nobles, commoners, and various tradespeople. Legal and Judicial tokens support court proceedings from the Old Bailey, while Religious tokens capture the central role of faith in historical society. Temporal, Currency, and Transportation tokens (35 tokens combined) provide the temporal, economic, and logistical context that makes historical narratives authentic and believable.
3.3.2 Special Token Categories Visualization
Let us visualize the special token categories and their relationships as shown below. These special tokens enable the model to understand and generate authentic historical language. Without them, the model would fragment historical concepts into meaningless subwords, losing the cultural and linguistic context that makes historical text generation both challenging and rewarding.
graph LR
A[🔤 Special Tokens<br/>150+ Total] --> B[📜 Historical Language<br/>25 tokens]
A --> C[🏛️ London Geography<br/>20 tokens]
A --> D[⏰ Historical Periods<br/>10 tokens]
A --> E[👥 Social Classes<br/>15 tokens]
A --> F[💼 Professions<br/>20 tokens]
A --> G[⚖️ Legal & Judicial<br/>10 tokens]
A --> H[⛪ Religious<br/>10 tokens]
A --> I[🕐 Temporal<br/>15 tokens]
A --> J[💰 Currency & Measurement<br/>10 tokens]
A --> K[🚗 Transportation<br/>10 tokens]
B --> B1["<|thou|>, <|thee|>, <|hast|>, <|doth|>, <|quoth|>"]
C --> C1["<|london|>, <|thames|>, <|newgate|>, <|westminster|>"]
D --> D1["<|tudor|>, <|stuart|>, <|georgian|>, <|regency|>"]
E --> E1["<|noble|>, <|gentleman|>, <|commoner|>, <|yeoman|>"]
F --> F1["<|apothecary|>, <|coachman|>, <|chimneysweep|>, <|baker|>"]
G --> G1["<|court|>, <|jury|>, <|verdict|>, <|prisoner|>"]
H --> H1["<|church|>, <|parish|>, <|prayer|>, <|blessed|>"]
I --> I1["<|morn|>, <|eve|>, <|season|>, <|year|>"]
J --> J1["<|shilling|>, <|pound|>, <|yard|>, <|furlong|>"]
K --> K1["<|coach|>, <|carriage|>, <|horse|>, <|vessel|>"]
%% class definitions (custom palette matching your original)
classDef cls_root fill:#e1f5fe,stroke:#81d4fa,color:#000;
classDef cls_hist fill:#f3e5f5,stroke:#ce93d8,color:#000;
classDef cls_geo fill:#e8f5e8,stroke:#a5d6a7,color:#000;
classDef cls_period fill:#fff3e0,stroke:#ffe0b2,color:#000;
classDef cls_social fill:#fce4ec,stroke:#f8bbd0,color:#000;
classDef cls_prof fill:#f1f8e9,stroke:#c5e1a5,color:#000;
classDef cls_legal fill:#e0f2f1,stroke:#80cbc4,color:#000;
classDef cls_relig fill:#f9fbe7,stroke:#e6ee9c,color:#000;
classDef cls_temp fill:#e3f2fd,stroke:#90caf9,color:#000;
classDef cls_curr fill:#fef7e0,stroke:#ffe082,color:#000;
classDef cls_trans fill:#f3e5f5,stroke:#e1bee7,color:#000;
%% assign classes
class A cls_root;
class B cls_hist;
class C cls_geo;
class D cls_period;
class E cls_social;
class F cls_prof;
class G cls_legal;
class H cls_relig;
class I cls_temp;
class J cls_curr;
class K cls_trans;3.5 Post-Processing and Hugging Face Integration
After training our custom tokenizer, we need to make it compatible with the broader machine learning ecosystem and ensure it works properly with language model training. Raw tokenizers can only convert text to tokens and back. Still, language models require additional functionality, such as special token handling, sequence padding, and integration with popular frameworks like Hugging Face Transformers.
The challenge, though, is that language model training requires specific formatting that raw tokenizers don’t provide. For example, training sequences need to be wrapped with special start/end tokens (<|startoftext|> and <|endoftext|>), padded to consistent lengths for batch processing, and integrated with the rest of the ecosystem. In our case, we also want to utilize Hugging Face and its ecosystem, allowing us to leverage standard training scripts and model architectures. Without proper post-processing, our custom tokenizer would be incompatible with existing training infrastructure.
We add post-processing capabilities that wrap text sequences with control tokens and create Hugging Face-compatible tokenizer files, ensuring seamless integration with the broader machine learning ecosystem while preserving our historical text optimizations.
There are three key areas that we need to consider:
Understanding Post-Processing: The first step is adding a post-processor that automatically wraps every text sequence with special start and end tokens. This is crucial because language models must be able to identify where sequences begin and end during training. For example, when we tokenize
"Hello world", the post-processor automatically converts it to<|startoftext|> Hello world <|endoftext|>. This template processing ensures consistent formatting across all our training data.Hugging Face Integration: Next, we create a Hugging Face-compatible wrapper around our custom tokenizer. This wrapper maps our special tokens to the standard token types that Hugging Face expects: beginning-of-sequence (bos), end-of-sequence (eos), padding, unknown, and masking tokens. This mapping allows our custom tokenizer to work seamlessly with standard training scripts and model architectures.
Special Token Functions: Each special token serves a specific purpose in language model training. The beginning-of-sequence token indicates when a new text starts, the end-of-sequence token marks the end of the text, padding tokens ensure all sequences in a batch have the same length, unknown tokens handle words not in our vocabulary, and masking tokens are used during training for masked language modeling tasks.
The code in Listing 11 demonstrates how we implement these post-processing steps and create a Hugging Face-compatible tokenizer:
def create_huggingface_tokenizer(tokenizer: Tokenizer, max_length: int = 1024) -> PreTrainedTokenizerFast:
"""Create Hugging Face compatible tokenizer"""
from transformers import PreTrainedTokenizerFast
# Add post-processor for sequence formatting
tokenizer.post_processor = processors.TemplateProcessing(
single="<|startoftext|> $A <|endoftext|>",
special_tokens=[
("<|startoftext|>", 1),
("<|endoftext|>", 0),
]
)
# Create Hugging Face tokenizer wrapper
hf_tokenizer = PreTrainedTokenizerFast(
tokenizer_object=tokenizer,
bos_token="<|startoftext|>",
eos_token="<|endoftext|>",
pad_token="<|pad|>",
unk_token="<|unk|>",
mask_token="<|mask|>",
model_max_length=max_length
)
return hf_tokenizerWithout this integration, our custom tokenizer would be incompatible with standard language model training. The post-processor ensures proper sequence formatting, while the Hugging Face wrapper enables seamless integration with existing training infrastructure and model architectures. This makes our tokenizer compatible with standard training frameworks, allowing for easy sharing and deployment.
3.6 Testing and Validation
We need to ensure the tokenizer works correctly with historical text before using it for model training. This requires testing on diverse historical samples and validating both encoding and decoding accuracy. A simple way to do this is to encode a set of historical text samples, decode them back, and check if the original text is perfectly reconstructed. We also want to verify that special tokens are used correctly in the tokenized output.
def test_historical_tokenizer(tokenizer: Tokenizer) -> dict:
"""Test the trained tokenizer on historical text samples"""
test_texts = [
"In the year of our Lord 1834, the streets of London were filled with the sounds of horse-drawn carriages.",
"The gentleman from the country said, 'I have never seen such a sight in all my days.'",
"The Thames flowed dark and mysterious through the heart of the city.",
"It was the best of times, it was the worst of times."
]
results = {'perfect_reconstruction': 0, 'special_token_usage': 0, 'failed_tests': []}
for i, text in enumerate(test_texts):
# Encode and decode text
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded.ids)
# Check reconstruction accuracy
if decoded.strip() == text.strip():
results['perfect_reconstruction'] += 1
else:
results['failed_tests'].append({'index': i, 'original': text, 'decoded': decoded})
# Check special token usage
special_tokens = [token for token in encoded.tokens if token.startswith('<|') and token.endswith('|>')]
if special_tokens:
results['special_token_usage'] += 1
return resultsTest Results:
- Perfect Reconstruction: 99%+ accuracy on test cases
- Special Token Usage: 80%+ of test cases use special tokens
- Average Compression Ratio: ~0.3 tokens per word (highly efficient)
- Success Rate: 99%+ for historical text samples
It is essential to conduct comprehensive testing to ensure the tokenizer operates reliably. In our case, the test cases cover different historical periods, writing styles, and linguistic patterns, giving us confidence that the tokenizer can handle the full range of historical text in our corpus. For a real-world LLM, this is, of course, more complex and would need to cover a broader set of areas.
Tokenizer Performance Validation
Not surprisingly, our custom tokenizer significantly outperforms standard approaches on historical text, as demonstrated by comprehensive metrics that compare it to GPT-2’s tokenizer, as shown in the table below. These metrics indicate that our custom tokenizer significantly outperforms standard approaches for historical text. The improved compression ratio and reconstruction accuracy ensure that the model learns from authentic historical language rather than tokenization artifacts, which is crucial for generating coherent and historically accurate text.
| Metric | Standard GPT-2 | Our Custom Tokenizer | Improvement |
|---|---|---|---|
| Vocabulary Size | 50,257 tokens | 30,000 tokens | 40% smaller |
| Special Tokens | 4 tokens | 150+ tokens | 37x more |
| Compression Ratio | ~0.4 tokens/word | ~0.3 tokens/word | 25% better |
| Reconstruction Accuracy | 95% | 99%+ | 4% better |
| Historical Language Support | Poor | Good | N/A |
These metrics validate that our 30K token vocabulary provides optimal coverage for historical text while remaining manageable for small language models. The 150+ special tokens capture linguistic patterns of 1500-1850 English, and the 25% better compression ratio means historical text is represented more efficiently, allowing the model to process longer sequences. The 99%+ reconstruction accuracy ensures no information is lost during tokenization, while excellent performance on archaic vocabulary, period-specific terminology, and London geography demonstrates the tokenizer’s effectiveness for historical language modeling.
3.8 Implementation and Usage
The complete tokenizer implementation, including training scripts, testing utilities, and validation tools, is available in the helloLondon GitHub repository . The repository provides:
- Training Code: Complete BPE tokenizer training with configurable vocabulary sizes and special token definitions.
- Testing Utilities: Comprehensive validation tools for testing tokenizer performance on historical text
- Integration Examples: Ready-to-use code for incorporating the tokenizer into your own projects
- Documentation: Detailed usage guides and API references
This implementation demonstrates how to build production-ready tokenizers for specialized domains, with particular focus on historical language processing and integration with modern ML frameworks.
4. Current Limitations
This project is designed as a learning exercise for those new to AI and LLM development. While we’ve built a functional system that demonstrates core concepts, this is not production-ready code and has several limitations that would need to be addressed for real-world deployment:
Data Scale & Quality:
- Corpus size: Our 500M character corpus is tiny compared to production LLMs, which typically use 100x-1000x more data (50B-500B+ characters). This limits the model’s ability to learn diverse patterns and reduces the quality of generated output.
- Source diversity: With only 218 sources, we lack comprehensive historical coverage across the 1500-1850 span, potentially missing important linguistic evolution patterns and regional variations.
- Geographic bias: Heavy focus on London may not accurately represent broader historical English patterns from other regions, limiting the model’s generalizability.
- Bias detection: We lack systematic approaches to identify or mitigate historical biases in the data, which could lead to the model perpetuating outdated or problematic language patterns.
- Quality assessment: Our cleaning pipeline, while effective for common issues, overlooks many edge cases and artifacts that would require more sophisticated ML-based quality assessment in production.
Tokenizer & Model Architecture:
- Vocabulary size: Our 30K token vocabulary is small compared to modern models (which often use 50K-100K+ tokens), limiting the model’s ability to represent diverse vocabulary efficiently.
- Special tokens: The 150+ special tokens are manually curated rather than learned from data, which may miss important patterns that data-driven approaches would discover.
- Context length: The 1024 token context window is very short compared to modern models (which often use 4K-32K+ tokens), limiting the model’s ability to maintain coherence in longer texts.
- Language support: No support for other languages or historical variants beyond English, significantly limiting the model’s applicability.
- Tokenization approach: While our BPE approach is clean and avoids WordPiece artifacts, it may not be optimal for all historical text patterns and could benefit from more sophisticated techniques.
Technical Infrastructure:
- Error handling: Basic error handling with limited logging and monitoring makes it difficult to debug issues and track system health in production.
- Testing: Minimal test coverage that excludes edge cases means many potential failure modes remain undetected until they occur in production.
- Performance: No optimization for speed, memory, or distributed processing, making the system unsuitable for production-scale deployment.
- Data management: Lacks data versioning and reproducibility guarantees, making it difficult to track changes and reproduce results across different environments.
- Security: No security considerations for data handling and model deployment, creating potential vulnerabilities for sensitive historical data.
- Compliance: Missing compliance considerations for GDPR, data privacy, and regulatory requirements, which are essential for production deployment.
- Monitoring: No production monitoring, alerting, or observability features, making it impossible to detect and respond to issues in real-time.
These limitations are intentional trade-offs made to keep the project manageable and focused on core learning objectives, but they represent significant gaps for production deployment.
4.3 What You’d Need for Production
Data Engineering and Legal Framework
Production systems require 100x-1000x more data from diverse sources, with ML-based quality assessment, bias detection, and filtering that goes far beyond our simple heuristics. You’d need robust ETL pipelines with proper error handling and monitoring, as well as a comprehensive legal framework for copyright clearance, data licensing, and compliance management, which we haven’t addressed.
Model Architecture and Training
Meaningful historical language understanding would require models with over 1 billion parameters, utilizing sophisticated training techniques, regularization, and optimization. You’d need a comprehensive evaluation on diverse historical text tasks and domain-specific fine-tuning capabilities that our current system doesn’t support.
Infrastructure and Operations
Production deployment requires a multi-GPU, multi-node distributed training infrastructure, production-grade model serving with load balancing and scaling, comprehensive monitoring and alerting systems, and end-to-end security for both data and model protection—none of which our learning-focused system currently provides.
This progression from data → tokenizer → training → deployment provides a complete methodology for building specialized historical language models.
5. Resources and Further Reading
- GitHub Repository: github.com/bahree/helloLondon - Complete source code for data collection and tokenizer training
- Part 1: Building LLMs from Scratch - Part 1 - Quick start and overview
- Documentation: Complete guides in the
08_documentation/folder covering every aspect of the project - Book Reference: Generative AI in Action - For deeper understanding of core LLM concepts
6. Summary
This post represents Part 2 of our learning journey into the fundamentals of LLM development. While we’ve built a functional data collection and tokenization system demonstrating core concepts, the real value lies in understanding:
- Data flow from raw sources to training-ready corpora
- Tokenization impact on model performance across different approaches
- Challenges in processing historical and domain-specific text
- Trade-offs between quality, scale, and complexity
- Debugging and improvement strategies for encountered problems
The limitations we’ve identified are great learning opportunities. Every production LLM started as a learning project, and every limitation teaches you something new about how these systems work. This foundation prepares us for the next phase of our journey.
Ready for Part 3? Part 3 will cover the custom GPT architecture, GPU optimization strategies, and training infrastructure that transforms our clean data and custom tokenizer into working language models—while maintaining the same educational focus on understanding the fundamentals.