Ai

I am excited to share that the complete manuscript of LLM Customization and Fine-Tuning: Adaptation, Distillation, and Alignment, written with Weehyong Tok , is now in Early Access from Manning Publications . 📚 When I announced the book in July , readers had the first five chapters. This update adds the four chapters that take you from a fine-tuned adapter to a model you can run, trust, and keep running in production. ...

The Stack Below the Stack, a 3-part series on how modern LLM inference actually works, told through a single DeepSeek V4 dtype bug. Part 1 · Physics of a request : why the first token is a different problem from every token after it, and why batching exists. Part 2 · Below Python : what actually runs under vllm serve, and why the escape hatches failed. Part 3 (this post) · Serving at scale: recipes, fleets, traffic shapes, and the bug resolved layer by layer. Catch-up: ...

The Stack Below the Stack, a 3-part series on how modern LLM inference actually works, told through a single DeepSeek V4 dtype bug. Part 1 · Physics of a request : why the first token is a different problem from every token after it, and why batching exists. Part 2 (this post) · Below Python: what actually runs under vllm serve, and why the escape hatches failed. Part 3 · Serving at scale : recipes, fleets, traffic shapes, and the bug resolved layer by layer. Catch-up: Part 1 established the physics: every request is two workloads on one GPU. Prefill is compute-bound and sets TTFT; decode is bandwidth-bound and sets TPOT; we raise the batch $B$ toward the crossover $B^{\star} \approx 300 \cdot N_{\text{total}}/N_{\text{active}}$ to pay for the weight read, and continuous batching keeps $B$ full. ...

The Stack Below the Stack, a 3-part series on how modern LLM inference actually works, told through a single DeepSeek V4 dtype bug that crashed a two-node deployment at startup. Part 1 (this post) · Physics of a request: why waiting for the first token is a different problem from streaming the rest, and why batching exists. Part 2 · Below Python : what actually runs under vllm serve, and why the escape hatches failed. Part 3 · Serving at scale : recipes, fleets, traffic shapes, and the bug resolved layer by layer. How to read it: each part is a single sitting and stands on its own if we know the one before it. The DeepSeek bug is the spine, it opens here in Part 1, and every layer we learn across the three parts is one that either could or couldn’t have caught it. The one-line fix lands in Part 3. ...

I am excited to announce that my new book, LLM Customization and Fine-Tuning: Adaptation, Distillation, and Alignment, written with Weehyong Tok , is now available in Early Access (MEAP) from Manning Publications . 📚 Update, October 2026: all nine chapters are now in the MEAP and the manuscript is complete. What chapters 6 to 9 add is in this post . ...

The earlier cluster benchmark write-up asked which frontier models were fastest. This post asks the next question: once hardware, serving stack, and benchmark shape are held constant, what do deployment choices - weight quantization and KV-cache compression - actually do to real coding-agent behavior? The model family stays fixed; the things operators actually touch get varied. This grew out of a r/LocalLLaMA thread asking for real coding benchmarks across quantization levels rather than proxy metrics. The study plan: Qwen/Qwen3.6-27B as the primary sweep, Qwen3-Coder-30B-A3B-Instruct as the coding-specialist cross-check, and MiniMaxAI/MiniMax-M2.1 as the heavyweight extension. That last lane turned out to be the most instructive result of the study, and not in the way I expected. ...

Over the last several benchmark cycles, I kept coming back to the same practical question: once you hold the hardware and methodology constant, which large open models are actually pleasant to serve, which ones merely load, and which ones become operationally awkward the moment you move beyond a demo? This post is a technical deep dive into that question. Instead of presenting a generic leaderboard, I focus on the details that usually matter in real deployments: throughput under fixed traffic shapes, latency behavior, scaling across 8x and 16x H200 shapes, and the caveats that only show up when you try to run these models end to end. ...

GitHub Repository: bahree/nanoclaw - full source code NanoClaw is a headless AI assistant running on my personal server. It processes messages from WhatsApp, Telegram, and Slack, runs scheduled tasks, and manages conversations with Claude agents in isolated containers. It’s been incredibly useful, but it had one major pain point: no visibility into what it was doing or why. If something went wrong - a message didn’t get a reply, a task didn’t run - the only way to debug was to SSH into the server, tail logs, and piece together what happened. Not ideal when you’re on the go and your assistant just… stops responding. ...

TL;DR In this final part of our 4-part series on building language models from scratch, we explore the evaluation, testing, and deployment pipeline that transforms our trained historical language models into working systems. Part 1 showed you how to use the published models, Part 2 covered data collection and custom tokenization, and Part 3 detailed the model architecture and training infrastructure. Here, we complete the journey with evaluation frameworks, testing infrastructure, and deployment to Hugging Face Hub. ...

Deadlock by design: two vibes, two locks, zero unlocks. In vibe coding, that’s not a bug - it is a feature. 😎 Here’s what happens when agreement-first engineering meets C++ and mutexes: #include <iostream> #include <mutex> #include <stdexcept> std::mutex mVibes, mProd; void shipToProd(bool agree) { // We lock the vibes and production—because feelings // and facts both need exclusive access. std::lock_guard<std::mutex> a(mVibes); std::lock_guard<std::mutex> b(mProd); if (agree) { std::cout << "Deploying to prod…\n"; // Reality pushes back: throw std::runtime_error("DeadlockException: vibes vs reality"); } else { std::cout << "Ignored tests.\n"; } } int main() { try { shipToProd(true); // agreement-first engineering } catch (const std::exception& ex) { std::cout << "AI: You're absolutely right! (" << ex.what() << ")\n"; std::cout << "/* lock(prod); lock(vibes); // Deadlock achieved; energy immaculate */\n"; std::cout << "COMMIT vibes; ROLLBACK sanity;\n"; } return 0; } Listing 1: Two mutexes walk into prod… and never come out #GeekyJokes #AI #GenAI #VibeCoding

TL;DR In this third part of our 4-part series on building language models from scratch, I explore the complete training infrastructure that transforms our clean historical data and custom tokenizer into working language models. Part 1 How to build a Large Language Model from Scratch - covered using the published model Part 2 Building LLMs from Scratch - Part 2: Data Collection & Custom Tokenizers - detailed data collection and custom tokenizer development. Here, we build the complete training pipeline from a custom GPT architecture through deployment-ready checkpoints. ...

TL;DR In this second part of our 4-part series on building language models from scratch, I explore the two foundational areas of LLM development: data collection and custom tokenizer creation. Part 1 - Building LLM from Scratch covered using the published model; here, we build the complete pipeline from raw historical documents to a custom tokenizer that understands archaic English, London geography, and period-specific terminology. ...

TL;DR In this post, I show how to build a working LLM from scratch and show a complete end-to-end pipeline from data gathering to training to deployment of a language model. For this project I concentrate on Old English and only related to London, using historical London texts (1500-1850). To show the flexibility, I built two language models which are identical in architecture and the only differs is their size and parameters (117M vs 354M). ...

TL;DR As part of my role at Microsoft’s AI Foundry Applied AI engineering team in CoreAI, I have participated in numerous detailed discussions about the evolving landscape of AI models. In conversations with many customers, from CxOs to engineers, one recurring topic is the rise of reasoning AI models. These models are designed to perform complex tasks by explicitly breaking down problems into logical steps, rather than just generating text in a single pass like traditional large language models (LLMs). This shift toward reasoning-centric AI marks a major evolution in how we develop and deploy AI systems—and it’s a key factor behind the rise of Agents and Agentic AI. ...

TL;DR: KV cache is a memory optimization central to efficient LLM inference. It enables faster, longer, and more cost-effective generation by caching previously computed attention keys and values—unlocking the practical deployment of models like GPT-4o, Llama 3, etc. 1. Introduction Generative AI, powered largely today by Large language models (LLMs) such as GPT-4o, Llama 3, etc., is transforming AI applications, from chatbots to code assistants and multimodal reasoning. As these models scale in size and context length, inference becomes a major computational and memory challenge. The Key-Value (KV) cache is a pivotal optimization that enables practical, high-performance inference in modern transformer architectures. ...

1. Overview Rust has been gaining attention recently due to its unique combination of performance, safety, and modern programming features. Its strict ownership model eliminates common memory issues like null pointer dereferencing and data races, providing a secure environment for developers. At the same time, its expressive syntax and focus on developer productivity make it a strong contender for systems programming. Its growing ecosystem and community continue to expand its capabilities into areas like web development and embedded systems, ensuring a confident and secure learning experience for developers. ...

AI is advancing at an unprecedented pace, with Mixture of Experts (MoE) models being one set of model architectures at the forefront of this revolution. These architectures enable breakthroughs in efficiency and scalability by leveraging a modular design where only a subset of specialized “expert” networks are activated for each input. MoE architectures have become a cornerstone in building ultra-large-scale models like GLaM and Switch Transformers. Mixture of Experts (MoE) is an advanced machine learning architecture that lately has gained significance, particularly in the realm of #LLMs (large language models) and NNs (neural networks). In talking with many people about AI, I’ve found that MoE as a topic comes up often, with many folks either not understanding what it is or their understanding of it being incorrect. ...

The one thing I wanted to do after my book Generative AI in Action was complete was to create a summary in natural speech and possibly use TTS (Text-to-speech) to create an audio summary—think of it as a podcast that is easier for people to consume and get a quick sense of what the book is about. TTS (Text to Speech) or not to TTS? Initially, I was inclined towards using TTS (Text to Speech) for the audio summary. This technology, I thought, would be a convenient way to create a podcast-like summary that would be easier for people to consume and get a quick sense of what the book is about. My journey began with TTS - using GPT 4o to create a summary after ingesting the book and then using that as into the Azure AI Speech stack . However, I stumbled across something intriguing. Instead of TTS, I opted for NotebookLM from Google Labs to generate the audio overview - the podcast. This decision marked a significant shift in my approach, and I created two podcasts using the content from the book - one from multiple sources and another from a single source (the book). ...

In today’s rapidly evolving tech world, mastering Generative AI isn’t just an advantage—it’s a necessity. Are you ready to harness its power to transform your business and solve real-world challenges? I’m excited to announce that my new book, Generative AI in Action, is now available in print and ebook formats from Manning Publications . 📖 Special Launch Offer 🌟 As a thank-you to my early supporters, I’m offering an exclusive discount. Use the code pbbahree at checkout to receive 45% off your purchase of Generative AI in Action in all formats (valid through Sept. 30, 2024)! ...

We released Phi-3 recently, which builds on Phi-2 ( read more on that here ) and it is a great model to use for various tasks. In this post, we will show how to run Phi-3 locally including a demo of it running on a phone. There should have been a video here but your browser does not seem to support it. 1. What are Small Language Models (SLMs)? Before diving into running Phi-2 locally, let’s take a moment to understand the concept of small language models (SLMs) and their significance in natural language processing (NLP). A SLM is a type of AI model that has been trained on a massive dataset of text but is limited in terms of its size and capabilities compared to a Large Language Model (LLM). SLMs are designed to be more lightweight and efficient, making them suitable for various applications, including chatbots, language translation, and content generation. SLMs are much smaller than LLMs, with fewer parameters and a smaller dataset, so they have a lower computational cost, making them more suitable for edge or resource-constraint devices. ...

1. What are Small Language Models (SLMs)? Before diving into running Phi-2 locally, let’s take a moment to understand the concept of small language models (SLMs) and their significance in natural language processing (NLP). A SLM is a type of AI model that has been trained on a massive dataset of text but is limited in terms of its size and capabilities compared to a Large Language Model (LLM). SLMs are designed to be more lightweight and efficient, making them suitable for various applications, including chatbots, language translation, and content generation. SLMs are much smaller than LLMs, with fewer parameters and a smaller dataset, so they have a lower computational cost, making them more suitable for edge or resource-constraint devices. ...

How many prompt engineers does it take to change a light bulb? Just one, but first, they need to fine-tune the model to make sure the AI doesn’t end up writing a poem about darkness instead. DALLE generated image of How many engineers it take to change a light bulb

🌐 As software continues to revolutionize the world, the advent of Generative AI is transforming the very fabric of software itself. My latest book, Generative AI in Action delves into this transformative journey. I am thrilled to announce the early release of my latest book, Generative AI in Action now available through Manning Early Access Program (MEAP) . This publication is a deep dive into the cutting-edge world of #GenerativeAI, #LLMs, #OpenAI, and #Azure #OpenAI, tailored specifically for enterprises. 🤘 ...

What is Whisper from OpenAI? Whisper is a speech recognition model (ASR – automatic speech recognition) from OpenAI. The model itself is multi-task model and as a result in addition to speech recognition, can also do language identification and speech translation across a number of languages. The model is open sourced and it comes in 5 sizes. Of these, 4 have a english-only variant which seem to perform better if one only needs english. The model is also robust to noise, accents, background noise and technical language. Whisper achieves near SOTA performance with zero-shot translation from multiple-languages to English. ...

Bing is getting a new look and feel, powered by Microsoft AI and OpenAI (ChatGPT) and was announced yesterday. There is a lot of buzz around this, and I thought I would share my thoughts on this as I got access today. What is the new Bing? Well, it is the thing that is making the 800-pound gorilla in the room, Google, come out and dance on its toes. 🦍 The new Bing is an overhauled version of the search engine that uses ChatGPT technology to understand questions and generate answers. It runs on the next generation of OpenAI’s language model, which is significantly more capable than the version of ChatGPT that has been available since November 20221. The new Bing provides more relevant results for simple things like sports scores, stock prices and weather, along with a new sidebar that shows more comprehensive answers if you want them3. You can also chat and create with the new Bing, using its natural language and creative abilities4. The new Bing is live starting today, with limited capabilities. ...