Inference
Batching, KV cache reuse, quantization, and semantic caching, so the first token arrives sooner and every token after it costs less.
AI Systems Engineer at Mursion, specializing in LLM inference, reliability, and evals. I serve models with low tail latency, keep them up under load, and measure that their answers hold up.
p99 latency cut in production
Deploy image, down from 14 GB
SKUs served by hybrid retrieval
Click through rate lift
Batching, KV cache reuse, quantization, and semantic caching, so the first token arrives sooner and every token after it costs less.
Async load shedding, backpressure, and circuit breakers under heavy traffic, incident root cause work, and drift monitoring that catches a decaying model early.
I build the evals that decide which model ships: candidate models are tested against product criteria and performance, side by side, before one is chosen.
A model is only as good as its slowest response and its worst answer. I build for both.
AI Systems Engineer
Mursion
My work sits where async systems, concurrency, and ML serving meet. I am going deeper into LLM serving: throughput, KV cache efficiency, and tail latency.
Building evals for model decisions: candidate models are tested against product criteria and performance before one is chosen.
Cut p99 latency by moving blocking calls off the asyncio event loop, while keeping session order intact.
Kept throughput steady under heavy concurrent load with an async logging queue: watermark based load shedding and backpressure to reduce GIL contention.
Rebuilt multi stage Docker builds. The image went from 14 GB to 355 MB, and PR to production went from 30 minutes to under 7.
Found a silent misrouting between builds that caused a production traffic skew during a Jenkins to ArgoCD migration. Engineering leadership recognized the fix.
Built hybrid retrieval (two tower network, FAISS, and BM25) over 5M+ SKUs. Conversion rose 7.2% and click through rose 34%, driving $300K+ ARR.
A production grade CNN inference service with P90 at 250 ms, on FastAPI and NVIDIA Triton. It runs GPU (TensorRT) and CPU (OpenVINO) backends, with Prometheus and Grafana to catch data drift and model decay in real time.
A contextual RAG research assistant on AWS. Binary quantized embeddings and an in memory vector store keep retrieval fast.
Reproducible LLM serving with vLLM, packaged in Docker images for portable, high throughput inference.
A semantic cache, vector store, and RAG on FAISS. Repeat questions with the same meaning come from cache, which cuts latency and cost.
Model packaging and deployment with Truss: a trained model becomes a served endpoint with very little glue code.
A language model built from scratch in PyTorch, grown step by step toward current production architecture.
A rebuild of Karpathy's micrograd, a tiny autograd engine, to learn backprop from first principles.
How transformer fine tuning, LLMs, and recurrent baselines trade accuracy against scoring cost. The fine tuned approach beat prior benchmarks by 5.7%.
What breaks when training leaves a single GPU: memory blowups, collective communication failures, and the patterns that keep long runs alive.
Read on Substack Deep diveThe transformer from self attention up to the KV cache: the mechanics that inference optimization is built on.
Read on SubstackQuoted by name on the power efficiency of on device LLM inference.
On Device AI Builders Hackathon · Qualcomm, LM Studio, and Microsoft, at NYU Tandon
Tell me about the team and the problem you want to solve.
Message me on LinkedIn