Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
Tailor your resume to this Research Scientist - Pre-training role.
Rezi rewrites your resume against Pluralis Research's job description. Free.
Tailor your resume to this Research Scientist - Pre-training role.
Rezi rewrites your resume against Pluralis Research's job description. Free.
Don't guess if your resume is good enough.
See how it scores against the Research Scientist - Pre-training posting at Pluralis Research — free, in seconds.
Don't guess if your resume is good enough.
See how it scores against the Research Scientist - Pre-training posting at Pluralis Research — free, in seconds.
About the Role
Pluralis Research is developing Protocol Learning, a method for training and serving large models in a decentralized manner on consumer-grade devices. This role involves building the training system to scale Protocol Learning from an 8B model run to frontier scale, focusing on large models on heterogeneous hardware connected via the internet.
Responsibilities
- Implement and optimize model-parallel training, including data, pipeline, and tensor parallelism for large models on heterogeneous GPUs under low-bandwidth, high-latency links.
- Implement techniques to reduce communication overhead while maintaining model convergence in challenging network environments.
- Develop solutions for runs to survive node churn, including robust checkpointing, state synchronization, and recovery as participants join and leave.
- Build monitoring systems to track throughput, bottlenecks, and model quality across hundreds of devices.
Requirements
- Trained models across multiple devices using PyTorch with FSDP, DeepSpeed, Megatron, or a custom implementation.
- Proficiency in data, tensor, and pipeline parallelism.
- Production-quality Python programming skills.
- Experience with concurrency, failure handling, and profiling for optimization.
- Demonstrated experience through shipped systems, research code, open-source contributions, or significant personal projects.
- Alignment with the mission of Protocol Learning as a viable path for collective, trustless, and sovereign AI.
- Comfort working across different time zones.
- Professional-level English proficiency (written and spoken).
Skills
- Distributed pretraining
- Model parallelism
- Data parallelism
- Pipeline parallelism
- Tensor parallelism
- Performance optimization
- Elasticity
- Fault tolerance
- Run instrumentation
- PyTorch
- FSDP
- DeepSpeed
- Megatron
- Python
- Concurrency
- Failure handling
- Profiling
- Large language models
- P2P networking
- NAT traversal
- Post-training
- RL
- Inference
- Serving systems
Location
- Remote
- Australia
- North America
- US
Work Type
- Remote-First
- Full-time
Experience Level
- Research Scientist
Benefits
- Significant ownership through equity
- High base salary
- Flexible work environment
- Optional full visa sponsorship and relocation support to Australia or the US
About the Company
- Pluralis Research is focused on Protocol Learning, training and serving large models in a decentralized way on consumer-grade devices.
- We have made significant advances, including Agora, which pretrained an 8B model from scratch on distributed consumer GPUs.
- Our published methods include Subspace Networks, Factored Gossip DiLoCo, AsyncMesh, and Sentinel.
- We are backed by Union Square Ventures and other tier-1 investors.
- We are a world-class, deeply technical team of ML researchers.
- Pluralis is unapologetically ideological, believing AI and the world benefit from implementing the protocol for intelligence.