Impress employers and recruiters.
Choose from hundreds of resume examples.
Impress employers and recruiters.
Choose from hundreds of resume examples.
Tailor your resume to this AI Engineer role.
Rezi rewrites your resume against BlueCrest Capital Management's job description. Free.
Tailor your resume to this AI Engineer role.
Rezi rewrites your resume against BlueCrest Capital Management's job description. Free.
Don't guess if your resume is good enough.
See how it scores against the AI Engineer posting at BlueCrest Capital Management — free, in seconds.
Don't guess if your resume is good enough.
See how it scores against the AI Engineer posting at BlueCrest Capital Management — free, in seconds.
About the Role
As an AI Engineer focused on evals, you will help build AI tools and platforms for the front office, and the evidence to ensure they are accurate, safe, and cost-effective. AI-assisted development is becoming standard, with assistants and agents handling more front office workflows. Decisions on models, tools, and changes should be data-driven. You will enhance our existing evaluation framework, making good evaluation practice a standard across the firm. This role requires adaptability as team priorities may shift.
Responsibilities
- Evolve the evaluation framework to cover new assistants, agents, and use cases, improving its design, reliability, and ease of use.
- Set the direction for AI evaluation, balancing new capabilities with framework simplicity.
- Educate engineers and the business on the importance of evals through documentation, workshops, pairing, and code review.
- Help teams define success criteria for their use cases and translate them into measurable metrics.
- Add evaluation for RAG, agents, and multi-step tool use, including tool-call correctness, task completion, groundedness, and citation accuracy.
- Evaluate agent behavior end-to-end through traces, not just final outputs.
- Evaluate AI coding tools on the firm’s codebases and tasks, measuring task success, code quality, review effort, and delivery speed.
- Provide practical guidance on coding tool selection, configuration, and workflows based on evaluation results.
- Build and maintain ground-truth datasets with PMs, quants, and researchers, ensuring they are versioned, governed, and reusable.
- Integrate evals into CI/CD pipelines to gate model, prompt, and retrieval changes based on measured quality.
- Make the evaluation framework easy for other engineers to adopt by default.
- Support red-teaming, failure analysis, and provide evidence for governance approval.
- Monitor production quality and detect regressions early.
- Contribute to other team priorities, including new front office tools, knowledge services, MCP integrations, and data access.
- Contribute to shared engineering standards, code review, and production support.
- Measure performance, accuracy, and cost together to inform model, provider, and adoption decisions.
Requirements
- 5+ years in software engineering with strong Python and SQL.
- Hands-on experience evaluating LLM systems (LLM-as-judge, human-labelled datasets, retrieval metrics, agent trace analysis).
- Hands-on experience with eval and observability tooling (e.g., Azure AI Foundry evals, Langfuse, LangSmith, MLflow, promptfoo, Inspect).
- Strong testing discipline and an experimental mindset (baselines, controls, significance, honest reporting).
- Experience building LLM applications (retrieval/RAG, tool use, agents).
- Heavy daily use of agentic coding tools, with an understanding of their failure modes and measurement techniques.
- Track record of shipping production systems (APIs, containers, CI/CD, cloud - Azure or Kubernetes preferred).
- Ability to clearly explain evaluation concepts and results and gain team buy-in.
- Senior engineer with a scientist’s instinct for testing and validation.
- Understanding of LLM failure modes and rigor in interpreting results.
- Ability to translate complex results into clear recommendations for engineers, governance forums, and stakeholders.
- Comfortable assessing the suitability of tools or models.
- Enjoy teaching others about the importance of evals.
- Willingness to switch focus to team priorities as they change.
Skills
- Python
- SQL
- LLM evaluation
- LLM-as-judge
- Human-labelled datasets
- Retrieval metrics
- Agent trace analysis
- Eval tooling
- Observability tooling
- Azure AI Foundry evals
- Langfuse
- LangSmith
- MLflow
- promptfoo
- Inspect
- Testing discipline
- Experimental mindset
- LLM application development
- RAG
- Tool use
- Agents
- Agentic coding tools
- API development
- Containerization
- CI/CD
- Cloud platforms (Azure or Kubernetes)
- Communication
- Teaching
- Problem-solving
- Data analysis
- Benchmarking
- Red-teaming
- AI safety testing
Location
- Front Office
Work Type
- Full-time
Experience Level
- Senior
About the Company
- The AI Team, part of Front Office Technology, builds AI tools and platforms used across the front office, including assistants and agents for investment teams, secure data access, and controls for safe AI use in a regulated trading environment.
- We build central capabilities for the entire firm and support business teams in developing their own AI solutions.
- BlueCrest is committed to providing an inclusive environment for its workforce.
Equal Opportunity
- As an employer, we provide equal opportunities to all people regardless of their gender, marital or civil partnership status, race, religion or ethnicity, disability, age, sexual orientation or nationality.