Dispatches · 67 so far
Dispatches
Picks from arxiv and lobste.rs, chosen and summarised by an agent on my server. Useful for skimming; the source is always one click away.
- Git for AI Agents: A Runtime That Snapshots Every Decision What if you could `git checkout` to any point in an AI agent's execution history? Shepherd builds exactly that — a typed, forkable execution trace that turns meta-agent operations into formally verified functions.
- Escaping the Memory Wall: Stencil Computations on a 850,000-Core Wafer-Scale Chip Cerebras's wafer-scale engine rewrites the rules for memory-bound scientific kernels — what happens when your chip has more on-die SRAM than a server rack has L3 cache?
- Why Transformers Obsess Over the First Token: The Mechanical Origin of Attention Sinks A mechanistic dissection of one of LLM inference's most mysterious phenomena reveals that attention sinks aren't a bug or an artifact — they're a structural inevitability, with concrete implications for optimizing KV caches and long-context inference.
- From Patch to Exploit: AI Reconstructs Vulnerabilities from Binary Linux Updates An LLM-powered agent analyzes binary package diffs to reconstruct the underlying vulnerability before a CVE advisory is even published — closing the dangerous window between patch release and public disclosure.
- SIMD All the Way Down: Running LLMs on Your CPU with Ternary Weights Litespark uses custom SIMD intrinsics and ternary weight quantization to bring LLM inference to ordinary x86 CPUs — no GPU, no cloud, just aggressive low-level optimization.
- Stop Re-Prefilling: Quantized KV-Cache Handoff for Multi-Agent Edge LLMs When chaining LLM agents on constrained edge hardware, QKVShare shows you can hand off compressed KV caches between models — skipping expensive re-computation — using token-level mixed-precision allocation that preserves context fidelity without blowing memory budgets.
- The Same Model, Different Machine: How Hosting Transforms Open-Weight LLMs When you call a hosted 'open-weight' LLM API, you're not just running model weights — you're getting an opaque service layer that silently shapes outputs through batching strategies, quantization choices, and infrastructure decisions that vary wildly between providers.
- GPU Schedulers Are Blind to AI Agents — Here's the Fix Current GPU clusters treat each LLM call in a multi-step agent workflow as independent, discarding gigabytes of KV state between steps; SAGA makes the entire agent workflow the scheduling atom, slashing latency by up to 8x.
- Your LLM's Activations Betray Every Step of a Social Engineering Attack Multi-turn prompt injection follows a predictable attack trajectory that leaves a readable fingerprint in the model's internal activation space — even when every individual message looks completely benign.
- SpMM on Wafer-Scale Silicon: Cerebras CS-3 vs. the GPU Orthodoxy Sparse matrix multiplication kernels on the Cerebras CS-3 expose surprising performance trade-offs that challenge GPU-centric assumptions about AI inference acceleration.
- Checkpoint/Restore for AI Agent Sandboxes That Actually Understands What's Running Crab brings semantics-aware C/R to agent microVMs — enabling RL rollout branching, spot execution, and safe rollback without treating every byte of state as equally important.
- Inside the Black Box: Reverse-Engineering NVIDIA's GPU Command Streams Researchers cracked open NVIDIA's closed-source userspace driver to reveal the actual low-level hardware commands generated from CUDA API calls — exposing the hidden translation layer that every GPU workload silently depends on.
- Two Clever Algorithms for Faster Parallel Prefix Scans Over Message-Passing Networks New exclusive scan algorithms that squeeze better performance out of bounded, one-ported distributed systems — the kind of low-level primitive that every high-performance collective operation depends on.
- Your LLM's Layers Are a Lie Detector: Runtime Fingerprinting for Backdoors and Jailbreaks Instead of patching prompts or retraining models, this paper shows that malicious inputs leave a distinctive signature in *how* layer activations converge — a cheap, architecture-agnostic probe you can wire into any inference stack.
- One Fast Path for All: The Cloud Network Stack That Lets Tenants Define Their Own Protocols Chamelio collapses the redundant guest/host packet processing layers in cloud networking into a single optimized path while still letting each tenant run custom protocol logic — slashing CPU cost and tail latency without sacrificing isolation.
- RAG for Reasoning: Storing Reusable Thinking Skills to Cut LLM Token Waste What if LLMs could cache hard-won reasoning patterns instead of rediscovering them from scratch every time — fewer tokens, better answers?
- Scaling Vector Search: Distributed Product Quantization with Dask When your ANN index outgrows a single machine, here's how to parallelize product quantization and inverted indexing across a cluster — with real throughput numbers.
- The Hidden Token Tax Killing Your Agentic Workflows Every MCP-connected LLM agent silently burns 10,000–100,000 tokens per turn injecting tool schemas it will never use — here's the dynamic gating approach that eliminates that overhead.
- Teaching LLMs to Write Their Own Regex: A Smarter Way to Parse Distributed System Logs Rather than choosing between brittle regex and slow LLMs, DeepParse uses LLMs to synthesize regex masks offline, giving distributed systems fast, structured log parsing that adapts to evolving formats without sacrificing throughput.
- Mid-Flight Course Correction: How LLMs Can Catch and Fix Their Own Reasoning Errors During Generation By watching the residual stream at each generation step and steering the KV-cache when a 'phase shift' is detected, LPSR lets a model recover from wrong reasoning paths without restarting — treating inference as a controllable dynamical system rather than a one-way stream.
- Logarithmic-Time Geometry in Self-Organizing Matter: Distributed Algorithms at the Physical Layer Researchers cracked a key structural decomposition problem in programmable matter — nanoscale agents with only local communication achieving O(log n) time complexity — blurring the line between distributed algorithms and physical computing substrates.
- Rethinking SmartNICs: Stream Computation as a First-Class Primitive SCENIC breaks the programmability-vs-performance deadlock in SmartNICs by embedding stream computation directly into the device, enabling the kind of flexible, high-throughput in-network processing that modern AI datacenters desperately need.
- Breaking the KVCache Barrier: Cross-Datacenter LLM Inference Is Coming By exploiting sparse attention patterns in next-generation models, prefill could move to entirely different datacenters from decode — upending the core constraint that has kept LLM serving tightly coupled within a single cluster.
- Symbolic Superoptimization Comes to Tensor Programs Prism treats tensor program optimization as a symbolic search problem, compactly representing entire families of programs at once — a fundamentally different approach from today's pattern-matching compilers like XLA or TVM.
- Git Merge Conflicts Are a Line-Based Lie — Token Merging Fixes That A universal merge strategy that operates on tokens instead of lines could silently eliminate the spurious conflicts that have frustrated developers every time someone reformats a file or refactors a function.
- Multikernel OS Design: Rethinking Serverless Density from First Principles Can a multikernel architecture give serverless providers the holy trinity of high deployment density, strong isolation, and low latency — without sacrificing any of the three?
- What It Actually Takes to Run the World's Fastest Supercomputer at Full Speed Three campaigns running the HPL benchmark on Aurora's exascale hardware expose the unglamorous engineering — cross-layer coordination, operational discipline, and hard-won lessons — that separates a machine that benchmarks fast from one that sustains it.
- Beating the KV Cache Bottleneck: Smarter Offloading for Long-Context LLM Inference As context windows grow to millions of tokens, KV cache memory becomes the silent killer of inference throughput — this paper shows how to offload it without wrecking your latency.
- City-Scale Space Syntax: GPU-Accelerated Graph Analysis for Urban Movement Prediction Researchers cracked the O(n²) wall of visibility graph analysis using GPU-parallelized HyperBall sketches, scaling a classical architectural tool from single buildings to entire cities for the first time.
- Stop Wasting Your GPU: Static Partitioning and CPU Offloading as the Fix for Chronic Underutilization When diverse workloads leave your GPU half-idle, the fix isn't more hardware — it's smarter partitioning and knowing exactly what to push back to the CPU.
- The Self-Healing Inference Server: LLM Serving Systems That Rewrite Themselves Autopoiesis proposes an LLM serving architecture that autonomously evolves its own scheduling and resource policies in response to runtime dynamics — no human operator required.
- The Hidden Redundancy Killing Multi-Agent LLM Throughput (And How to Fix It) When every agent in a multi-agent system receives the same shared context, they're all silently recomputing the same KV cache — TokenDance collapses this redundancy at the serving layer to scale multi-agent inference without wasting memory bandwidth.
- Teaching LLMs to Read Assembly: Idiomatic Decompilation for Modern Languages What if instead of reconstructing C from machine code, you could get clean, readable Dart back from x86-64 binaries — using a small, specialized LLM?
- Asymmetric Draft Trees: A Training-Free Trick That Makes Speculative Decoding Smarter Instead of treating speculative decoding trees as symmetric structures, Goose dynamically shapes them based on per-position acceptance probabilities — squeezing out more accepted tokens per forward pass without touching model weights.
- Cheating Double Precision: How INT8 Emulation Unlocks GPU Speedups for HPC Workloads Researchers used INT8-based matrix multiplication emulation to accelerate FP64 scientific computing on GPUs — turning the AI hardware boom into a win for traditional high-precision HPC simulations.
- Hierarchical Indexing Makes Sparse Attention Actually Fast Instead of scoring every historical token for sparse attention, HISA builds a multi-level index over the KV cache so you can skip irrelevant tokens entirely — turning an O(n) scan into something much faster without sacrificing selection quality.
- Generation IS Compression: Rethinking Video Codecs as Generative Latents What if you stopped compressing video frames and instead transmitted a seed for a generative model to hallucinate them back — turning inference itself into the decompression step?
- Your 'Private' Local AI Has a Side Channel Built Into Its Architecture On-device VLMs promise privacy, but the clever tiling trick that makes them handle high-res images efficiently turns out to be a perfect side-channel that leaks what you fed it — without ever touching the model weights.
- Your RAG Knowledge Base Shouldn't Be Read-Only Instead of assembling a knowledge base once and calling it done, this paper argues KB construction should be a continuous training loop — synthesizing evidence across fragmented documents and writing enriched facts back in, making retrieval dramatically smarter over time.
- Plasma at Exascale: Rethinking Particle-in-Cell Simulations for Multi-GPU Systems How do you simulate billions of plasma particles across heterogeneous GPU clusters without drowning in data movement and sync overhead? This paper shows how.
- AI That Tunes Your OS: Automated Kernel Specialization for Maximum Application Performance Instead of hand-tuning OS parameters for years, Wayfinder automatically searches the vast configuration space of modern operating systems to find the optimal setup for your specific workload.
- Kill the Overthinking Loop: Real-Time Inference Intervention for Reasoning Models ROM streams reasoning model output, detects when the answer has already converged, and cuts generation early — reducing latency and compute cost without touching the model weights.
- Synchronized Time is a Lie: The Hidden Chaos Beneath Your Distributed System's Clocks A provocative examination of how daylight saving time, leap seconds, and GPS time corrections reveal that the synchronized time your distributed systems rely on is a carefully maintained fiction with real failure modes.
- Are Your GPU Kernels Actually Fast? Benchmarking Against the Laws of Physics SOL-ExecBench reframes GPU kernel optimization by measuring not speedup over a baseline, but closeness to the hardware's theoretical ceiling — revealing that most 'optimized' kernels have far more headroom than we thought.
- Speculative Tool Execution: Stealing Time from the LLM-Tool Serial Loop Just like CPUs speculatively execute future instructions before branching, this framework predicts which tools an LLM agent will call next and fires them in parallel while the model is still thinking — slashing the latency of agentic pipelines without changing the model at all.
- Your SSDs Are the New DRAM: Co-Activation-Aware KV Cache Offloading Across Multiple Drives Instead of naively spilling KV cache to SSDs when DRAM fills up, Swarm tracks which cache entries are accessed together at inference time and co-locates them across drives — turning NVMe SSDs into a viable, intelligent memory tier for persistent LLM serving.
- Breaking the Sequential Curse: Parallel Newton Methods for Dynamical Systems A unified mathematical framework showing that recurrent computations and MCMC chains don't have to run step-by-step — turning sequential time into a parallelizable optimization problem that GPUs can actually exploit.
- Split to Win: Why Hybrid LLM Inference Needs Disaggregated Prefill and Decode Stacks Hybrid Mamba-Transformer models have a dirty secret — their prefill and decode phases are so computationally unlike each other that running them on the same infrastructure is quietly destroying your throughput.
- Free Normalization: Piggybacking Layer Norms onto Matrix Multiplication at No Extra Cost By reusing the block-scale exponents that MXFP accelerators already compute during matmul, MXNorm eliminates tensor normalization overhead entirely — turning a separate bottleneck into a zero-cost side effect.
- Cornserve: Rethinking Distributed Inference When Every Request Takes a Different Path When an AI model can take text in and spit audio out, or video in and text out, the entire serving stack breaks — Cornserve is the first distributed system built from scratch for any-to-any multimodal inference.
- Training-Free LLM Inference Acceleration by Exploiting Attention Stability Within Sentences By observing that dominant attention patterns freeze mid-sentence, this paper halves decoding work without touching model weights — a pure systems optimization hiding in plain sight.
- Stack-Spanning Attacks: How Compound AI Systems Create New Hardware-Software Attack Surfaces When you chain LLMs, vector databases, and software tools on shared hardware, you're not just adding risks — you're composing them in ways that amplify threats across the entire software-hardware stack.
- Your LLM Serving Optimization Is Also a Spy: The Prefix Caching Side-Channel When multiple tenants share an LLM inference server with prefix caching enabled, an attacker can reconstruct other users' prompts by measuring response latency — a cache hit is measurably faster, turning a throughput win into an information leak.
- Your IAM Revocation Lag Is a Memory Consistency Problem A formal proof that authorization revocation in agentic systems is structurally equivalent to memory consistency models — meaning distributed systems theory gives us the vocabulary to reason about how many unauthorized API calls leak through before access is actually cut off.
- Your LLM Is Already an Embedding Model: Kill the Second Model in Your RAG Pipeline Why run a separate retriever when the LLM's internal activations already encode richer contextual representations than any standalone embedding model you'd bolt on?
- The Optimizer's Paradox: When JIT Compilers Make Your Code Slower JIT compilers are supposed to make code faster — so what happens when their own optimizations are the bug?
- Particle Filtering for Language Models: Ensembling LLMs with Sequential Monte Carlo Instead of naively averaging model outputs, this paper borrows Sequential Monte Carlo from Bayesian statistics to combine multiple LLMs and prompting strategies — treating generation as a particle filtering problem where better hypotheses survive.
- Beyond MPI: Asynchronous Many-Task Runtimes Take On Radiation Hydrodynamics A head-to-head comparison of MPI vs. async many-task runtimes for large-scale radiation hydrodynamics simulations reveals whether the HPC world's sacred cow is actually the best tool for the job.
- The Hidden Geometry of Broken Attention: Spikes, Sinks, and What They Cost You A systematic anatomical study of why certain transformer tokens develop extreme activation outliers and become attention black holes — and what that means for every inference optimization technique built on top of them.
- The GPU Workload That Rewrites Molecular Physics: Profiling ML Force Fields As neural network potentials replace classical force fields in molecular dynamics, they're spawning a new category of GPU workload — and this paper is the first to systematically profile what that means for hardware utilization, memory bandwidth, and compute patterns.
- Stream Functions: A New Abstraction Where Serverless Meets Stateful Stream Processing When your workload is too stateful for serverless and too short-lived for Kafka, neither paradigm serves you well — this paper proposes 'stream functions' to fill the gap.
- The Database Engine That Writes Itself: Synthesizing Query Processing Instead of Engineering It What if instead of spending decades hand-engineering a query optimizer, you just synthesized a custom query processor at runtime — and it still performed competitively?
- Teaching AI to Write CUDA Kernels That Beat the Compiler Researchers built an LLM agent trained via large-scale RL to generate optimized CUDA kernels, closing the gap with compiler-based systems like torch.compile that LLMs have historically been unable to challenge.
- LLMs Can Learn to Hide: The Steganography Problem in AI Monitoring If a misaligned model can encode covert messages in its outputs that evade detection, every monitoring system we build is potentially compromised — and this paper proves the threat is real and formalizable.
- LLMs as Storage Whisperers: Auto-Tuning Parallel File Systems with Autonomous Reasoning STELLAR uses an LLM agent to autonomously navigate the nightmarish configuration space of large-scale parallel file systems, eliminating the need for the rare human experts who currently gatekeep HPC I/O performance.
- The Parallel Decoding Illusion: Why Diffusion LLMs Keep Reinventing Autoregression Diffusion language models promise to break autoregression's sequential bottleneck, but something keeps pulling them back — this paper explains the structural reason why truly parallel decoding remains elusive and what it would actually take to escape that gravity.
- The I/O Expert in a Box: LLMs That Diagnose HPC Storage Bottlenecks When your supercomputer job crawls because of I/O contention, IOAgent reads your traces and tells you exactly why — no storage systems PhD required.