About the Role
We are building a Large Physics foundation Model (LPM) to achieve general causal intelligence, capable of predicting the future and identifying actions to alter it. We are seeking data engineers excited to tackle unsolved problems by building the data platform for continuous, large-scale physical observations.
Responsibilities
- Design and operate petabyte-scale storage, including lakehouse architecture, file formats, and data layout optimized for batch and real-time queries.
- Own the shared compute and orchestration platform (e.g., Spark, Ray, workflow scheduling) for ingestion and research pipelines.
- Optimize data strategy end-to-end, from storage to loading, owning high-throughput data loading into training up to the tensor boundary.
- Build systems for cataloging, deduplication, lineage, search, and reproducibility throughout the data lifecycle.
- Implement platform-level quality and monitoring tooling for data and research teams.
- Scale infrastructure to improve engineering velocity and ensure reliability, with matching monitoring and alerting.
- Work across the full data lifecycle, including building and operating ingestion pipelines for critical data sources.
Requirements
- Demonstrated experience building large-scale data pipelines and distributed compute systems (e.g., Spark, Ray, Beam).
- Knowledge of state-of-the-art methods and tools for data ingestion, storage, and loading, including file formats and storage systems (e.g., Parquet, Zarr, Delta Lake) and their impact on performance and scalability.
- Deep familiarity with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
- Understanding of how data loading throughput affects large-scale training, and experience optimizing it.
- Ability to own deliverables end-to-end, from collecting and translating requirements to autonomously driving execution.
Skills
- Spark
- Ray
- Beam
- Parquet
- Zarr
- Delta Lake
- Cloud infrastructure
- Data lake architectures
- Batch pipelines
- Streaming pipelines
About the Company
- Our mission is general causal intelligence; AI that is capable of (1) predicting the future and (2) identifying the actions to alter it.
- We are building a Large Physics foundation Model (LPM) because physical systems, unlike text or images, are governed by verifiable cause and effect.
- We believe that scaling on physics will enable an understanding of causality required to predict and control physical systems, starting with weather.
- Our founding team has built and deployed AI against the physical world in robotics, drug discovery, and particle physics at institutions like DeepMind, Waymo, Cruise, Insitro, Nabla Bio, and CERN.
