Skip to content
View poojithdevan4D's full-sized avatar
4 followers · 0 following

Achievements

Achievement: Pull Shark

Achievements

Achievement: Pull Shark

Block or report poojithdevan4D

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
poojithdevan4D/README.md

Applied inference & performance engineering. I work on making LLMs run fast and cheap on hardware most people can actually afford — serving, quantization, and benchmarking, mostly on a 4 GB laptop GPU and free cloud T4s rather than rented H100s.

Right now I'm an Edge AI Intern at OneBit (1.58-bit ternary inference) and a Research Intern at Trebuchet (Q1.15 fixed-point inference).

Projects

vllm-benchmark — Benchmarked vLLM on a T4. Continuous-batching throughput scaled to ~790 tok/s at batch 64, roughly 7× a GGUF backend's peak, and kept climbing where the GGUF backend tailed off.

llm-gateway — An OpenAI-compatible gateway with health-based failover across local and cloud backends, a semantic cache that answers similar questions without calling a model, per-key rate limiting, and graceful degradation when the cache is unavailable.

llm-inference-services — An OpenAI-compatible microservice that serves a local GPU model or a cloud backend from the same code, deployed live.

serving-benchmark — An async load test of the throughput-vs-latency tradeoff. Throughput peaks around 113 tok/s at concurrency 4; beyond that, tail latency climbs about 5× for no real gain.

quantization-tradeoff — Size, speed, and quality across Q4/Q8/FP16. Q4_K_M ran 3.3× faster at a third of the VRAM of FP16, with no measurable accuracy loss on my eval.

Research and earlier work

  • Cloe v1 (co-authored at OneBit) — a hybrid ternary/BF16 Qwen 0.8B reached 44% MMLU, beating full-precision 0.5B and 1B baselines.
  • Q1.15 fixed-point inference (Trebuchet) — a true-integer forward pass with zero float32 leakage; 77.71% CIFAR-10 on ResNet-20.
  • QwenQuant — a 4-way quantization benchmark where I caught a tokenizer measurement bug that had made Q4_K_M look 75% worse than it actually was.
  • Speculative decoding from scratch — Qwen2.5-1.5B with a 0.5B draft model on a single 4 GB GPU, verified byte-for-byte against greedy decoding.

Stack

  • Serving: FastAPI, Docker, Ollama, vLLM, OpenAI-compatible APIs
  • Quantization: GGUF / K-quants, 1.58-bit ternary, Q1.15 fixed-point, INT4 / INT8, KV-cache quant
  • Performance & infra: throughput/latency benchmarking, semantic caching, routing & failover, rate limiting
  • Core: Python, PyTorch, HuggingFace, httpx, matplotlib. Currently learning CUDA C.

poojithdevan@gmail.com · linkedin.com/in/poojith-devan

Pinned Loading

  1. vllm-benchmark vllm-benchmark Public

    vLLM continuous-batching throughput benchmark on a T4 — scales to ~790 tok/s at batch 64 (~7x a GGUF backend's peak). Documents the CUDA-12/13 install fix.

    Jupyter Notebook

  2. llm-gateway llm-gateway Public

    OpenAI-compatible LLM gateway — health-based failover across local + cloud backends, a semantic cache (local embeddings), and per-key rate limiting.

    Python

  3. llm-inference-services llm-inference-services Public

    OpenAI-compatible LLM inference microservice (FastAPI + Docker) — runs a local quantized model on a 4 GB GPU or a cloud backend; deployed live.

    Python

  4. serving-benchmark serving-benchmark Public

    Async serving benchmark for a local LLM on a 4 GB GPU — throughput, latency percentiles (p50/p95/p99), and the throughput-vs-latency tradeoff under concurrent load.

    Python

  5. quantization-tradeoff quantization-tradeoff Public

    Quantization tradeoff study (size / speed / quality) for Qwen2.5-1.5B at Q4_K_M / Q8_0 / FP16 on a 4 GB GPU.

    Python

  6. qwen-speculative-decoding qwen-speculative-decoding Public

    Speculative decoding from scratch — Qwen2.5-1.5B accelerated by a 0.5B draft on one 4 GB GPU, verified byte-for-byte against greedy decoding.

    Python