Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

Tailor your resume to this AI Engineer (Harness) role.

Rezi rewrites your resume against Valarian Technologies Limited's job description. Free.

Resume score gauge reading 58 out of 100

Don't guess if your resume is good enough.

See how it scores against the AI Engineer (Harness) posting at Valarian Technologies Limited — free, in seconds.

About the Role

As an AI Harness Engineer at Valarian, you will own the experimental design, evaluation methodologies, and benchmark infrastructure for our AI models and autonomous agentic workloads. You will serve as the bridge between rigorous data science, statistical validation, and production agent scaffolding in an emerging domain.

Responsibilities

Architect scalable Python evaluation harnesses to measure task completion, trajectory quality, tool-use correctness, cost, latency, and variance across multi-step agentic systems.
Apply rigorous statistical methods to determine whether performance changes are true system improvements or stochastic noise.
Develop automated judging systems validated against human ground truth, measuring alignment and detecting judge failure modes.
Define sampling strategies, annotation guidelines, and scoring rubrics, managing label noise and measuring inter-annotator agreement.
Conduct hands-on failure analysis on agent runs, categorize root causes, and communicate findings to engineering teams and stakeholders.
Translate evaluation results into architectural improvements for agent scaffolding, prompt design, tool execution loops, context window compaction, retries, and safety guardrails.

Requirements

Demonstrated expertise in hypothesis testing, bootstrapping, power analysis, and statistical significance within probabilistic machine learning systems.
Proven experience building, auditing, and validating LLM judge setups against human annotations, including tracking agreement metrics and mitigating judge biases.
Strong track record designing evaluation datasets, crafting precise scoring rubrics, managing label noise, and measuring inter-annotator agreement.
Deep comfort evaluating complex agent trajectories, tool execution sequences, latency, cost, and variance across repeated runs.
Exceptional judgment in defining metrics tied to real-world outcomes, with a keen eye for detecting benchmark gaming, target drift, or shortcut learning.
Ability to dissect complex execution logs, categorize failure modes, and synthesize clear, data-driven recommendations.
Practical proficiency with LLM APIs, prompt engineering, structured outputs, function/tool calling, and retrieval systems in Python.
Clear understanding of planning loops, tool orchestration, memory management, and multi-agent coordination, including their trade-offs and common failure modes.
Ability to write clean, maintainable Python code to iterate on harness components such as guardrails, retry logic, state handling, and context management.

Skills

Python
Statistical Methods
Hypothesis Testing
Confidence Intervals
Bootstrapping
Power Analysis
LLM APIs
Prompt Engineering
Structured Outputs
Function/Tool Calling
Retrieval Systems
Agent Frameworks (LangChain, LangGraph, AutoGen, lm-evaluation-harness, Promptfoo, Ragas)
Kubernetes Environments

Location

London HQ

Work Type

Hybrid

Experience Level

Mid-level

Salary/Compensations

Competitive salary

Benefits

Equity
Competitive salary
Employer pension contributions
Private health insurance
Hybrid work setup
Rewarding company retreats and meetups

About the Company

Valarian Technologies is a dual-use technology company building critical tools to safeguard the future in an era of evolving global security challenges.
We're rethinking security beyond traditional military domains, addressing asymmetric threats that impact our technological advantage, economic strength, and democratic institutions.
We build Acra – the platform foundation for everything we do as a dual-use technology company.
The platform’s name, rooted in the Greek word for citadel (or, fortress), reflects the design and purpose of our infrastructure-agnostic secure enclaves: protecting critical data.
Some of the government and commercial workflows include: increased operational resiliency for mission-critical systems and functions; enabling organisations to more quickly and widely adopt emerging technologies while ensuring the integrity of their intellectual property; information flow during disaster response scenarios, and zero-trust / least-privilege environments for M&A, attorney-client privileged communications, etc.
At our core, we're driven by a shared mission and a belief in making a tangible impact on our world.
Whether you join our London HQ or the wider global organisation, you’ll be a part of collaborative, high-performing teams, creating cutting-edge software, platforms, and infrastructure.
Our culture is built on inclusivity, compassion and flexibility – we want everyone to be empowered to achieve their goals at Valarian.
The work we do is vital, but so are the connections that make it happen.
We thrive on the shared energy, spontaneous conversations, and mutual trust built when we spend time together.
We operate on a hybrid model, gathering in our London office 3 days a week to support one another and collaborate.

Equal Opportunity

Valarian Technologies Limited is an equal opportunity employer and welcomes applications from individuals regardless of race, colour, religion, sex, sexual orientation, gender, identity or expression, national origin, age, disability, genetic information, marital status, veteran, amnesty, or any other legally protected characteristic.
We are committed to ensuring a fair and inclusive recruitment process and providing employment opportunities to all applicants.
Decision recruitment, hiring, and employment are based solely on qualifications, skills, and experience relevant to the job requirements.