Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

Tailor your resume to this Machine Learning Engineer, Inference & Serving (Speech LLM) role.

Rezi rewrites your resume against IntelliPro Group Inc.'s job description. Free.

Resume score gauge reading 58 out of 100

Don't guess if your resume is good enough.

See how it scores against the Machine Learning Engineer, Inference & Serving (Speech LLM) posting at IntelliPro Group Inc. — free, in seconds.

About the Role

We are looking for a Machine Learning Engineer, Inference & Serving (Speech LLM) to build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models.

Responsibilities

Build and deploy high-throughput, ultra-low-latency inference systems for large language models and foundational speech models.
Understand intricate tradeoffs between latency, throughput, and Time-To-First-Token (or Time-To-First-Audio) in real-time streaming environments.
Implement practical experience with continuous batching, KV cache management (e.g., PagedAttention), and stateful connections necessary for real-time conversational AI.
Identify and eliminate hardware bottlenecks by deeply understanding GPU architectures (NVIDIA Ampere/Hopper) and memory hierarchy.
Communicate clearly and collaborate effectively at the intersection of core ML training and backend infrastructure teams.
Squeeze every last drop of performance out of a cluster of GPUs in fast-moving environments.
Build AI systems that natively understand and generate speech.
Gain deep, under-the-hood familiarity with modern LLM serving frameworks like vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server.
Handle continuous audio streams over WebSockets or WebRTC, deploy neural audio codecs, and manage chunked audio generation to minimize conversational latency.
Implement cutting-edge generation algorithms such as speculative decoding, lookahead decoding, or chunked prefill.
Perform hands-on experience with post-training quantization (PTQ), deploying models in FP8, INT8, AWQ, or GPTQ, without degrading audio naturalness or ASR accuracy.
Deploy multi-GPU (Tensor Parallelism) and multi-node inference pipelines, and manage autoscaling infrastructure using Kubernetes.

Requirements

Hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models.
Practical experience with continuous batching, KV cache management (e.g., PagedAttention), and stateful connections necessary for real-time conversational AI.
Deep understanding of GPU architectures (NVIDIA Ampere/Hopper) and the memory hierarchy.
Clear communication and effective collaboration skills.
Ability to thrive in fast-moving environments and enjoy systems-engineering challenges.
Obsession with building AI systems that natively understand and generate speech.
Experience with frontier serving frameworks (vLLM, TensorRT-LLM, SGLang, NVIDIA Triton Inference Server).
Experience with real-time audio streaming (WebSockets, WebRTC, neural audio codecs, chunked audio generation).
Experience with advanced inference techniques (speculative decoding, lookahead decoding, chunked prefill).
Hands-on experience with model compression and quantization (PTQ, FP8, INT8, AWQ, GPTQ).
Experience with large-scale distributed systems (multi-GPU, multi-node inference, Kubernetes autoscaling).

Skills

Inference & Serving
Speech LLM
Large Language Models
Speech Models
High-throughput systems
Ultra-low-latency inference
Latency
Throughput
Time-To-First-Token
Time-To-First-Audio
Real-time streaming
Continuous batching
KV cache management
PagedAttention
Stateful connections
Conversational AI
GPU architectures
NVIDIA Ampere
NVIDIA Hopper
Memory hierarchy
Hardware bottlenecks
Systems engineering
AI systems
Speech generation
vLLM
TensorRT-LLM
SGLang
NVIDIA Triton Inference Server
Open-source contributions
Real-Time Audio Streaming
WebSockets
WebRTC
Neural audio codecs
Chunked audio generation
Speculative decoding
Lookahead decoding
Chunked prefill
Model Compression
Quantization
Post-training quantization (PTQ)
FP8
INT8
AWQ
GPTQ
Large-Scale Distributed Systems
Tensor Parallelism
Multi-node inference
Autoscaling
Kubernetes

Location

San Francisco, CA

Work Type

Hybrid

Experience Level

Hands-on experience
Deep familiarity
Practical experience
Strong candidates may also have experience with

Salary/Compensations

$170K - $320K base salary + performance bonus + Equity

Benefits

Top-tier healthcare for employees and dependents, including dental and vision, and a generous employer subsidy.
401(k) plan for full-time employees with company matching.
Unlimited PTO, plus 13 paid holidays.
12 weeks of paid time off to spend time with your new family, regardless of gender.
Choice of top-of-the-line laptops/workstations, annual offsites, and a fully stocked office.

About the Company

IntelliPro Group Inc. is a global staffing and talent solutions company.
We connect exceptional professionals with leading organizations across technology, engineering, and professional services.
By partnering closely with both clients and candidates, we build long-term relationships that help businesses grow and careers thrive.

Equal Opportunity

We are an equal opportunity employer.
We do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristics.