Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
Tailor your resume to this (Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure role.
Rezi rewrites your resume against Together AI's job description. Free.
Tailor your resume to this (Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure role.
Rezi rewrites your resume against Together AI's job description. Free.
Don't guess if your resume is good enough.
See how it scores against the (Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure posting at Together AI — free, in seconds.
Don't guess if your resume is good enough.
See how it scores against the (Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure posting at Together AI — free, in seconds.
About the Role
We are seeking a software engineer who is reliability-minded but also a builder to manage the production environment for our Kubernetes-native control plane that provisions and runs our GPU inference fleet. You will be responsible for the manifest-driven API and controllers that enable the inference team to declare their needs, such as cluster setup or model deployment, and ensure these are reconciled. Your role involves troubleshooting issues like InfiniBand problems, diagnosing serving faults, and enhancing the automated self-healing system to permanently resolve failure modes. The core value is maintaining decoupling and fleet efficiency without introducing instability. You will own SLOs and error budgets for inference serving, lead incident response, and implement learnings from postmortems into the controllers and automation. You will also focus on the reliability aspect of capacity efficiency, as mistuned defragmentation or bin-packing can pose production risks. Ideal candidates possess hands-on production-reliability experience with large-scale distributed or GPU-serving systems, are adept at troubleshooting InfiniBand/RDMA and Kubernetes controller/CRD-based platforms, and view GPU infrastructure and its reliability tools as software to be engineered. A product mindset is essential, focusing on improving the on-call experience, alert clarity, and automated remediation.
Responsibilities
- Operate the provisioning state machine in production, owning the health of the host lifecycle pipeline, discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA, and triaging stuck or failed state transitions at fleet scale.
- Feed real failure patterns back into the design of states and transitions within the host lifecycle pipeline.
- Triage and extend automated self-healing by diagnosing live issues, driving fixes to production, and creating new detection/remediation plugins for recurring problems.
- Address new failure modes, InfiniBand issues in disaggregated serving, GPU Xid variants, and hero-model or LoRA-specific faults.
- Own end-to-end reliability of the pipeline, holding SLOs and error budgets for provisioning and fleet health.
- Lead incident response when idempotency, retry, rollback, or drift assumptions break down, driving systemic fixes.
- Partner with the inference/ML platform team from the production side, feeding field failure modes back as requirements into the platform's abstractions.
- Work alongside machine learning researchers, engineers, and product managers to bring new technologies into production.
- Introduce new techniques, tools, and architecture to improve the performance, latency, throughput, and efficiency of the model inference stack.
- Build tools for visibility into bottlenecks and instability sources, then design and implement solutions.
- Lead reliability cross-functionally by building coalitions, managing stakeholders through incidents and roadmap trade-offs, and representing the reliability point of view in planning.
Requirements
- Experience operating a provisioning or host-lifecycle pipeline in production.
- Comfortable troubleshooting stuck or failed state transitions (discovery, bring-up, GPU driver/CUDA validation, health checks, decommission/RMA) at fleet scale.
- Ability to translate real failure patterns back into improvements to the state machine’s design.
- Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
- Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi-node training or disaggregated-inference context.
- Familiarity with the operational realities of multi-tenant model serving (LoRA/adapter-based serving).
- Experience running differentiated SLOs across model pools of different priority.
- Demonstrated track record of eliminating operational toil through automation and tooling built by yourself.
- Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
- Comfort with event-driven architectures, backpressure/circuit-breaking design, and treating observability (metrics, logs, traces) as a built-in system component.
- A software-engineering mindset: ability to write and ship production code and view GPU infrastructure as software to be engineered.
Skills
- Kubernetes
- InfiniBand
- RDMA
- GPU infrastructure
- Reliability engineering
- Distributed systems
- ML serving
- Controller/CRD patterns
- Reconciliation loops
- Event-driven architectures
- Observability (metrics, logs, traces)
- Automation
- Tooling
Location
- US
Work Type
- Full-time
Experience Level
- Senior
Salary/Compensations
- $250,000 - $320,000
Benefits
- Startup equity
- Health insurance
- Other competitive benefits
About the Company
- Together AI, the AI Native Cloud, is purpose-built for AI engineers.
- AI application developers get high-performance inference that scales reliably, fine-tuning and reinforcement learning for creating frontier-level specialized models, and pre-training at massive scale for fully custom intelligence.
- The platform includes a marketplace of leading open models that teams can run, adapt, and own.
- Trusted by Cursor, Decagon, ElevenLabs, Salesforce, and Zoom, Together serves 400+ trillion tokens a month.
Equal Opportunity
- Together AI is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.
- Please see our privacy policy at https://www.together.ai/privacy.