About the Role
We are seeking a Staff Data Platform Engineer to lead the Data Transfer Hub project, a critical infrastructure responsible for transferring petabytes of sensor data daily from partner vehicles globally. This hands-on technical leadership role involves designing and building a globally distributed ingestion system, directly collaborating with partners like Uber and Nissan, and enabling large-scale AI model training.
Responsibilities
- Set technical direction for the Data Transfer Hub, balancing reliability, throughput, cost, partner constraints, observability, and operational support.
- Lead the technical design and delivery of the Data Transfer Hub project, ingesting large volumes of video, LiDAR, and sensor data from global partners at up to PB/day scale.
- Design and build parallelized, distributed data transfer pipelines using Flyte for workflow orchestration, Kafka for event-driven patterns, and Azure for storage and transfer.
- Build and operate a globally distributed hub-and-spoke data transfer model for retrieving and sharing sensor data at scale with partners.
- Write infrastructure-as-code in Terraform and build pipeline logic primarily in Python.
- Drive cloud-to-cloud and cross-cloud networking solutions, with Azure as the primary platform.
- Adapt to a fast-changing environment with evolving requirements and high comfort with ambiguity.
Requirements
- Proven experience building and operating large-scale data transfer pipelines at multi-TB or PB scale.
- Experience orchestrating parallelized transfer jobs (Flyte preferred; other workflow orchestration tools considered).
- Experience building reliable, observable data systems, including retry strategies, backfills, data integrity checks, lifecycle/state tracking, and operational alerting.
- Strong cloud platform skills (Azure preferred); experience with multiple cloud providers (AWS/GCP) is a strong advantage.
- Proficiency in Python and Terraform.
- Cloud-to-cloud and networking experience (e.g., blob/object transfer and storage at scale, cross-cloud data movement).
- A product mindset, focusing on customer needs and problem-solving.
- Comfort working in a fast-changing environment with evolving requirements.
- High degree of autonomy and ownership.
Skills
- Flyte
- Kafka
- Azure
- Python
- Terraform
- Cloud-to-cloud networking
- Cross-cloud data movement
Location
- London
Work Type
- Full-time
- Hybrid
Experience Level
- Staff
About the Company
- Portal is the cloud platform connecting Wayve to the outside world, handling data exchange with partners, surfacing AI insights, and enabling commercial deployment of Wayve's autonomy stack to OEM partners globally.
- The team operates as a startup within a scale-up, with its own infrastructure, codebase, and direct relationships with commercial partners including Uber, Nissan, and other OEMs.
- We move fast, own our decisions, and build systems that matter.
