About the Role
We are seeking an engineer to join our team and contribute to data infrastructure. You will architect and scale the core infrastructure for distributed training pipelines, multimodal data catalogs, and intelligent processing systems handling petabytes of data. This role is crucial for enabling research breakthroughs by working directly with researchers to accelerate experiments, develop new datasets, and improve infrastructure efficiency.
Responsibilities
- Design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities.
- Develop high-throughput systems for data ingestion, processing, and transformation — including training data catalogs, deduplication, quality checks, and search.
- Build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.
- Implement and maintain monitoring and alerting to support platform reliability and performance.
- Collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles.
Requirements
- Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
- Proficiency in at least one backend language (Python or Rust).
- Fluent in distributed compute frameworks such as Apache Spark or Ray.
- Deeply familiar with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
- Comfort operating across the stack and owning projects end-to-end.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.
Skills
- Python
- Rust
- Apache Spark
- Ray
- Cloud infrastructure
- Data lake architectures
- Batch and streaming pipelines
- Kafka
- dbt
- Terraform
- Airflow
- Web crawler development
- Deduplication
- Data mining
- Search scaling
- File formats
- Storage systems (Parquet, Delta Lake)
- Documentation
- Testing
- Tooling
Location
- San Francisco, California
Work Type
- Onsite
Experience Level
- Mid-level
- Senior
Education Level
- Bachelor's degree or equivalent experience
Salary/Compensations
- $350,000 - $475,000 USD
Benefits
- Generous health, dental, and vision benefits
- Unlimited PTO
- Paid parental leave
- Relocation support
About the Company
- The mission of Thinking Machines is to build AI that extends human will and judgment.
Equal Opportunity
- We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
