Role Summary
You'll design and run AI data projects and build the automation behind them, combining deterministic tooling with LLM agents to deliver solutions in quality management, project management, and data analytics.
Key Responsibilities
Build hybrid automation: Pair deterministic components (scripts, validation rules, trackers, scheduled jobs) with stochastic ones (LLM agents that score, classify, check, report, etc.). Decide which steps must be rules and which can be models, and design the hand-offs between them.
Set up and run agents in Claude and in our native enterprise agentic environment: task specs, connections to data and tools, schedules, monitoring and failure handling.
Quality management: Design QA workflows, guidelines, rubrics and scorecards; calibrate automated checks against human reviewers; turn review findings into error categories you can measure and fix.
Project management: Run data collection and annotation projects end to end (scoping, vendors, workflows, tracking, delivery) and build the tooling that lets coordinators run the day.
Data analytics: Collect, clean, and analyse production and quality data (throughput, rework, error rates, contributor performance) and turn it into reporting that teams and clients act on.
Verify everything: Test automation before release and build continuous auditing parallel workflows to ensure quality.
Document so others can run it: short specs, tickets and runbooks that let coordinators and engineers operate and extend what you build.
Required Qualifications
Intermediate JavaScript and Python: enough to build and maintain automation in production, not just prototypes.
Hands-on agent building in Claude (projects, skills, connectors, scheduled tasks), and the ability to do the same in our native enterprise agentic environment (connecting different APIs with our native infrastructure).
LLM evaluation experience: prompt and rubric design, calibration against human judgement, and the skepticism to catch a confident wrong answer.
A data analytics background, with experience in the data collection or annotation industry.
Experience managing projects with external vendors or contributors and multiple stakeholders.
Comfort making calls on incomplete information, and knowing how to make and own your decisions.
Summary
Design and run AI data projects and build automation combining deterministic tooling with LLM agents across quality management, project management, and data analytics. Requires intermediate JavaScript and Python, hands-on Claude agent building, LLM evaluation experience, a data analytics background, and experience managing projects with external vendors and stakeholders.
Responsibilities
Design and run AI data projects; build hybrid automation combining deterministic components with LLM agents; set up and run agents in Claude and a native enterprise agentic environment; design QA workflows, rubrics, and scorecards; run data collection and annotation projects end to end; collect, clean, and analyze production and quality data; test automation before release; document specs, tickets, and runbooks.
Qualifications
Intermediate JavaScript and Python; hands-on agent building in Claude and a native enterprise agentic environment; LLM evaluation experience; data analytics background in data collection or annotation; experience managing projects with external vendors, contributors, and multiple stakeholders; comfort making calls on incomplete information.
Experience requirements
Experience managing projects with external vendors or contributors and multiple stakeholders; data analytics background with experience in the data collection or annotation industry.