About the Role
Red Badger is looking for a DataBricks Engineer to join our Data & AI practice. This is a delivery-focused role where you will work directly with clients and Red Badger consultants to shape complex technical solutions and lead a team of data engineers to build them through to production. You will also partner on pre-sales and proposals to add technical credibility and deepen client partnerships.
Responsibilities
- Lead a team of data engineers to build solutions through to production.
- Work closely with the Director, Data & AI, and partner on pre-sales and proposals.
- Shape complex technical solutions as an output from discovery phase.
- Spend roughly half of your time hands on, pairing with data engineers on the team, and helping them pick the right design to solve the problem.
- Support other functions (delivery, product, software engineering) and clients with decisions impacted by the technical strategy of the project.
- Provide support and line management to data engineers on your team.
- Run and contribute to agile ceremonies, helping to shape and improve the agile process.
- Work with the project’s delivery manager and product owner to lead the team in delivering high quality digital products.
- Lead and support peers in technical conversations.
- Be an advocate for lean development, building for what is required now.
- Pair with other members of the team, including designers.
- Own the relationship with the client’s technical stakeholders.
- Proactively engage with clients to shape and move the project forward, including giving updates on delivery progress and highlighting potential technical blockers.
- Upskill and embed modern data practices in client teams where necessary.
- Manage and inspire teams, navigating enterprise-scale problems and turning them into opportunities for clients.
- Guide, mentor and grow your team, bringing the Red Badger culture and building effective, high-trust teams.
- Translate complex data concepts into business value, managing multi-vendor environments with a focus on long-term partnership.
- Design systems with a focus on delivering value, ensuring extensible and scalable solutions meet evolving business and user needs.
- Champion quality assurance and performance tuning so data solutions are robust, reliable and respected.
- Keep a pulse on industry standards and best practice.
- Build strong relationships with technology partners to deliver the best outcomes for clients.
- Negotiate SLAs and SLOs with the business and supporting teams.
- Help shape and resource technical teams.
- Articulate commercial objectives into a technical strategy.
- Own the tech leadership on complex data projects, setting the technical vision and managing technical stakeholders.
- Coach and line manage data engineers and data delivery individuals starting their line management journey.
- Be a part of the public face of the company in media and at meetups, conferences, etc.
Requirements
- At least 2 years of commercial Databricks experience (current).
- Strong AWS Skills.
- Ideally have worked on a large scale Databricks migration project previously.
- Strong understanding of Medallion Architecture.
- Hands on experience over theoretical.
- Deep, production-grade experience across AWS &/or Azure native services, with expert-level Databricks and/or Snowflake and an Agile delivery mindset.
- Fluent across core AWS services.
- Fluent across core Azure services.
- Build Data Products where every pipeline and dataset is discoverable, addressable and secure by design.
- Know the full data lifecycle, from discovery, storage, pipelines, ETL and streaming to data quality, governance, metadata management and cataloguing, through to advanced analytics and decision-making.
- Understand how AI is built and deployed in modern data platforms (Databricks, Snowflake) and cloud-native services like Amazon Bedrock, with enough depth to guide teams and advise clients, spanning LLMs and SLMs, agentic automation, Human in the Loop and MLOps.
- Know when to use RAG (Retrieval-Augmented Generation), Serverless, DWH models, Data Mesh, Data Fabric, Medallion and Data Products.
- Fluent in SQL, Python, PySpark, R or Scala.
- Familiar with open formats including Parquet, Iceberg and JSON.
- You care deeply about high-quality work that delivers real value.
- You're comfortable with ambiguity and solving complex problems collaboratively.
- You bring strong expertise in your craft, alongside a willingness to keep learning.
- You communicate clearly and build trust fast with clients, data engineering leaders and data and software engineers.
- You're pragmatic, adaptable, outcome-focused and low-ego in multidisciplinary teams.
- You lead with a people-first attitude, helping others grow and making sure people are heard, valued and set up to thrive.
Skills
- Databricks
- AWS
- Medallion Architecture
- Snowflake
- Agile delivery
- RDS
- DynamoDB
- S3
- SageMaker Lakehouse
- Glue
- EMR
- Athena
- CDC/DMS
- DataBrew
- Kinesis
- dbt
- Lake Formation
- Glue Data Catalog
- SageMaker Catalog / DataZone
- Bedrock (including AgentCore)
- SageMaker AI
- SQL
- PostgreSQL
- Cosmos
- Blob
- Data Lake
- OneLake
- Fabric Warehouse
- Fabric Data Factory
- Data Factory CDC
- Fabric/Synapse Spark
- ADF
- Synapse Pipelines
- Fabric
- Purview
- Foundry
- Agent Service
- ML
- Python
- PySpark
- R
- Scala
- Parquet
- Iceberg
- JSON
Location
- Central London
Work Type
- Onsite (2 days a week)
- Full-time
Experience Level
- 2 years commercial Databricks experience
- Senior
About the Company
- Red Badger is a tech and product consultancy, building enterprise software products for blue chip companies, multinationals and PLC’s.
- The team has been around for 15+ years and is over 120 strong.
- The company supports and learns from each other.
- They work hard but have fun doing it.
- It is a diverse group made up of 22 different nationalities, speaking 17 different languages.
- Headquartered in London, they also have offices in Leeds and Cape Town.
