Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
Tailor your resume to this Senior Site Reliability Engineer role.
Rezi rewrites your resume against Andromeda Cluster's job description. Free.
Tailor your resume to this Senior Site Reliability Engineer role.
Rezi rewrites your resume against Andromeda Cluster's job description. Free.
Don't guess if your resume is good enough.
See how it scores against the Senior Site Reliability Engineer posting at Andromeda Cluster — free, in seconds.
Don't guess if your resume is good enough.
See how it scores against the Senior Site Reliability Engineer posting at Andromeda Cluster — free, in seconds.
About the Role
Design, operate, and debug large-scale GPU infrastructure for distributed training and inference, working directly with customers pushing the limits of modern AI systems. This role requires engineers who have personally run GPU clusters in production, understand distributed training failure modes, and can reason about performance from network fabric to kernel to framework.
Responsibilities
- Design and evolve multi-provider, multi-region GPU compute clusters optimized for large-scale training.
- Make topology-aware scheduling, networking, and storage decisions impacting training throughput and cost efficiency.
- Serve as the primary technical point of contact for customers running large-scale training workloads, onboarding, troubleshooting, and optimizing in real time.
- Define SLOs and error budgets accounting for GPU infrastructure failure modes.
- Own capacity planning across heterogeneous GPU fleets optimized for training throughput.
- Ensure the health and performance of high-speed interconnects (InfiniBand, RoCE, NVLink) underpinning distributed training.
- Diagnose and resolve fabric-level issues degrading collective operations.
- Build deep visibility into GPU utilization, memory pressure, interconnect throughput, training job performance, and hardware health.
- Build production-grade automation for cluster provisioning, GPU health checks, job scheduling, self-healing, and firmware/driver lifecycle management.
- Lead incident response for complex, multi-layer failures spanning hardware, networking, orchestration, and ML frameworks.
- Drive blameless postmortems and systemic fixes.
Requirements
- Deep, hands-on experience operating large-scale GPU clusters (NVIDIA A100/H100/B200 or equivalent).
- Understanding of GPU memory hierarchies, ECC behavior, thermal throttling, and hardware failure modes from direct experience.
- Production experience with InfiniBand, RoCE, or NVLink fabrics in the context of distributed training.
- Ability to diagnose slow all-reduce operations, identify degraded links in a fat-tree topology, and reason about congestion control at scale.
- Working knowledge of how large training jobs run, including NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, FSDP, or similar.
- Expert-level Linux knowledge: kernel tuning, driver management (NVIDIA drivers, CUDA toolkit), cgroup/namespace internals, performance profiling at the syscall and hardware level.
- Strong experience running Kubernetes in production with GPU workloads, including device plugins, topology-aware scheduling, multi-cluster federation, and custom operators.
- Experience with Slurm or other HPC schedulers is valued.
- Strong engineering skills in Python, Go, or Bash.
- Build production-grade tools and services.
- Infrastructure-as-Code proficiency (Terraform, Helm, Ansible, or equivalent).
- Hands-on experience building monitoring and alerting for GPU infrastructure, integrating GPU-specific telemetry (DCGM, nvidia-smi, fabric manager metrics) into actionable dashboards.
- Proven track record leading incident response for complex distributed systems where the failure could be in hardware, firmware, networking, drivers, orchestration, or application code.
Skills
- GPU Systems Expertise
- High-Performance Networking
- Distributed Training & ML Frameworks
- Linux & Systems Internals
- Kubernetes & Orchestration
- Automation & Software Engineering
- Observability & Monitoring
- Incident Management
- Distributed Storage
- Training Optimization
- Cluster Buildout & Hardware
- Team Leadership
Location
- Global Remote
- San Francisco
Work Type
- Full-Time
Experience Level
- Senior
About the Company
- Andromeda is a market and infrastructure platform to buy, sell, and operate compute.
- We believe demand for compute will grow exponentially, leading to fragmentation in AI factories.
- Our job is to enable all of this fragmented compute to flow through one platform, delivering reliable capacity to model builders, research labs, and inference providers.
- We believe every spare electron should be made productive for AI and we're building the platform that makes that possible.
- We sit at the center of companies needing reliable compute, fragmented global GPU supply, and capital/risk/operational complexity.
- When we succeed, trillions of dollars of compute will flow through Andromeda, benefiting builders, providers, and capital.
- In five years, Andromeda will shape the AI infrastructure market.
Equal Opportunity
- Andromeda Cluster is an equal opportunity employer.
- We celebrate diversity and are committed to creating an inclusive environment for all employees.
- We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.