Inference optimization
Cutting latency and cost out of model serving — batching, KV-cache reuse, quantization, and semantic caching to shave every millisecond off time-to-first-token.
inference engine — demo
Satyam
Chatrola
AI Systems Engineer · LLM inference & serving
response
> whoami
AI Systems Engineer. I build low-latency systems that serve large language models — from inference optimization and serving infrastructure to ML systems under real production load.
focus
What I do
Serving infrastructure
Packaging models as reliable, reproducible low-latency endpoints with vLLM, Truss, and Docker — shipped through CI/CD and built to stay up under load.
Systems from first principles
Understanding the stack end to end: autograd, transformer internals, and language models built from scratch — so optimization is grounded, not guessed.
Scale & reliability
Designing AI systems with software-engineering rigor and MLOps discipline — efficient, observable, and robust under real production load.
signal
I turn models into
fast, dependable endpoints.
−71%p99 latency reduction
5×telemetry serialization speedup
5M+SKUs searched
+34%click-through lift
I'm an AI Systems Engineer at Mursion, focused on the part of AI that decides whether it's actually usable in production: how fast it responds, under real load. My work sits where async systems, concurrency, and ML serving meet — and I'm deepening further into LLM inference-serving: throughput, KV-cache efficiency, and tail latency at scale.
- Cut p99 latency by 71% in production by moving blocking calls off the asyncio event loop, preserving session ordering.
- Sustained throughput under heavy concurrent load with an async logging queue — watermark-based load-shedding and eventual-consistency backpressure to minimize GIL contention.
- Shrank a deploy footprint from 14GB to 355MB and cut PR-to-prod from 30 minutes to under 7 by re-architecting multi-stage Docker builds.
- Root-caused a production traffic-skew incident — recognized by engineering leadership — by isolating silent misrouting between builds during a Jenkins-to-ArgoCD migration.
- Architected a hybrid retrieval system (two-tower NN + FAISS + BM25) scaling to 5M+ SKUs — lifting conversion 7.2% and click-through 34%, driving $300K+ ARR.
Quoted in Qualcomm's developer blog · On-Device AI Builders HackathonMS Computer Science · NYU
work
Selected projects
serving · Triton
care-companion
Production-grade CNN inference service (P90 250ms) on FastAPI & NVIDIA Triton — GPU (TensorRT) and CPU (OpenVINO) backends, plus a full Prometheus/Grafana stack detecting data drift and model degradation in real time.
RAG · latency
research-mate
Contextual-RAG research assistant on AWS. Compresses embeddings with binary quantization and keeps the vector store in memory to minimize retrieval latency at scale.
serving
vllm-demo
Reproducible LLM serving with vLLM, packaged in Docker images for consistent, portable high-throughput inference.
caching
semantic_cache_and_RAG
Semantic cache, vector store, and RAG built on FAISS — cutting repeat-query latency and cost by serving from cache when meaning matches.
deployment
truss-demo
Model packaging and deployment with Truss — turning a trained model into a served endpoint with minimal glue.
from scratch
my_LLM
A language model built from scratch in PyTorch, iteratively evolved from foundations toward current industry-standard architecture.
foundations
my_micrograd
Reimplementing Karpathy's micrograd — a tiny autograd engine — to understand backprop from first principles.
research
Published · peer-reviewed
Benchmarking Fine-Tuned Transformers, LLMs and LSTM Networks for Automated Essay Scoring
A comparative study of how transformer fine-tuning, large language models, and recurrent baselines trade off accuracy against the cost of scoring at scale — with the fine-tuned approach exceeding prior benchmarks by 5.7%.
writing
Deep dives
FeaturedQuoted by name on the power efficiency of on-device LLM inference. On-Device AI Builders Hackathon · Qualcomm × LM Studio × Microsoft, hosted at NYU TandonA Pragmatic Engineer's Guide to OOMs, NCCL, and Resilience in Large-Scale Training
What actually breaks when training falls off the single-GPU happy path — memory blowups, collective-communication failures, and the resilience patterns that keep large runs alive.
Read on DeepDivesThe Transformer Architecture: A First-Principles Analysis from Self-Attention to KV Caching
Deconstructing the transformer from self-attention up through the KV cache — the mechanics that decode-time inference optimization is actually built on.
Read on DeepDives