Job Description
Job Title: AI Engineer / Generative AI Engineer
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Extension Possible)
Experience: 15+ Years
Job Summary
We are seeking a hands-on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production-grade AI agents.
Key Responsibilities
Develop AI agents using Python and Claude Agent SDK.
Deploy agents using AWS Bedrock AgentCore Runtime.
Design golden datasets and implement LLM-as-a-Judge evaluations.
Integrate automated evaluations into CI/CD pipelines as quality gates.
Implement Langfuse/OpenTelemetry for AI observability and tracing.
Evaluate agent accuracy, task completion, tool execution, and performance.
Support RAG pipelines and AWS data processing workflows.
Required Skills
Python and Agentic AI development.
AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
Golden-dataset design and curation.
LLM-as-a-Judge design and calibration against human labels.
Automated evaluation and CI gate integration.
Langfuse and/or OpenTelemetry.
Good to Have
RAGAS or equivalent evaluation frameworks.
Trajectory evaluation and agentic red-teaming.
Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.
Qualifications
6–8+ years of software engineering, cloud engineering, or AI/ML experience.
Hands-on experience with production AI agents and continuous evaluation.
Strong problem-solving and troubleshooting skills.
Additional Information
All your information will be kept confidential according to EEO guidelines.
Summary
Build, deploy, monitor, and improve production-grade AI agents using Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation. Requires 15+ years of experience, Python and Agentic AI development, AWS Bedrock AgentCore evaluations, golden-dataset design, LLM-as-a-Judge, and Langfuse or OpenTelemetry.
Responsibilities
Develop AI agents using Python and Claude Agent SDK; deploy agents using AWS Bedrock AgentCore Runtime; design golden datasets and implement LLM-as-a-Judge evaluations; integrate automated evaluations into CI/CD pipelines; implement Langfuse/OpenTelemetry for AI observability; evaluate agent accuracy, task completion, tool execution, and performance; support RAG pipelines and AWS data processing workflows.
Qualifications
15+ Years; 6–8+ years of software engineering, cloud engineering, or AI/ML experience; hands-on experience with production AI agents and continuous evaluation; strong problem-solving and troubleshooting skills.
Experience requirements
15+ Years; 6–8+ years of software engineering, cloud engineering, or AI/ML experience; hands-on experience with production AI agents and continuous evaluation.