Location: San Francisco, CA, United States
Job Type: Full-time
Salary: $150K - $300K
Equity: 0.01% - 1.00%
Experience: 3+ years
Visa: US citizen/visa only
The role
Own the models behind Coval’s judges. Coval is the evaluation platform for conversational AI: we help enterprise teams simulate, monitor and improve voice and chat agents before those agents ever reach a customer. Because our product is the judge, this role sits at the center of the company.
You’ll own the model side end to end. You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models and non-tuning work like tool use and context engineering. Your improvements ship directly into the product enterprises rely on to launch dependable voice agents.
What you’ll do
Design and ship the LLM judges that score conversations: rubrics, judge prompting, and calibration against human raters
Build the reliability tracking that keeps judges honest from release to release
Run post-training where training beats prompting: LoRA adapters on open-weight models, trained on data distilled from our own scored conversations
Everywhere else, do the non-tuning capability work: tool use, structured outputs, and context engineering
Keep asking which lever actually moves the metric, and focus there
What we’re looking for
4+ years as an engineer, with strong systems instincts
A track record of shipping measurable quality gains by pairing rigorous measurement with targeted training, not vibes and prompt tweaks
You’ve built eval infrastructure you actually trust
You’ve run LoRA/SFT experiments end to end, including preparing your own data rather than relying on clean data from other teams, so you know which lever really wins
High agency: on a small Series A team, nobody hands you tickets. You figure out what matters next and drive it there
Ambitious and kind. You’ll be part of the group that founds our engineering org, sets its bar and grows the team
Summary
Own the LLM-as-a-judge models that score voice and chat conversations at scale. Design rubrics, prompting, and calibration, run LoRA fine-tunes on open-weight models, and build reliability tracking. Requires 4+ years as an engineer with trusted eval infrastructure and end-to-end training experience.
Responsibilities
Design and ship LLM judges that score conversations; Build reliability tracking; Run post-training with LoRA adapters on open-weight models; Tool use, structured outputs, and context engineering
Qualifications
4+ years as an engineer with strong systems instincts; track record shipping measurable quality gains; built eval infrastructure; run LoRA/SFT experiments end to end including data preparation; high agency
Experience requirements
3+ years experience; 4+ years as an engineer with strong systems instincts
Benefits
Equity 0.01% - 1.00%