Impress employers and recruiters.
Choose from hundreds of resume examples.

Impress employers and recruiters.
Choose from hundreds of resume examples.

Target Resume Now

About the Role

Town is building the most personalized and capable AI assistant, which requires proving its superiority through measurable improvements in every aspect of its performance. You will own the development of evaluation systems and quality metrics to ensure the assistant's effectiveness across all its functions and optimize model routing for a balance of cost, quality, and speed.

Responsibilities

• Build a generalized eval system to measure assistant quality across all surfaces and multi-step agent trajectories.
• Establish golden datasets and a labeling loop for continuous validation and regression prevention.
• Develop model routing and online evaluation tools for model selection and learning.
• Ensure all prompt and system changes are measurable to facilitate rapid, safe iteration.
• Collaborate with engineers to instrument quality and implement feedback loops for issue resolution.

Requirements

• Experience building or managing LLM eval systems, or offline/online quality measurement at scale.
• Strong analytical skills for measurement, with a focus on the instinct to measure 'better'.
• Hands-on experience with eval tooling and frameworks, with informed opinions on their application.
• Ability to reason about model routing and associated tradeoffs.
• Proven track record of shipping fixes, not just dashboards and metrics.
• Comfort working in a greenfield environment where systems are yet to be built.
• Senior or staff engineering level experience.

Skills

• LLM evaluation systems
• Offline/online quality measurement
• Measurement and analytics
• Eval tooling and frameworks
• Model routing
• System design
• Greenfield development

Location

• San Francisco, CA

Work Type

• Onsite
• Full-time

Experience Level

• Senior
• Staff

About the Company

• Town is an AI company building a persistent model of user identity, voice, judgment, and relationships to perform tasks across various tools like email, calendar, documents, and Slack.
• The AI acts proactively, learning and improving over time to become an extension of the user.
• Founded by Jean-Denis Greze (CTO of Plaid) and Tony Vincent (Director of Applied AI Product at Google).
• A small, talent-dense team backed by prominent investors including Andreessen Horowitz, Forerunner Ventures, First Round Capital, and Conviction.
• Has raised over $73M to date.