AI Engineer (AI System Calibration & Optimization)
4 days ago
Highlight
OnsiteFull-timeSenior
Summary
Robots & Pencils is seeking an AI Engineer (AI System Calibration & Optimization) to partner with a strategic client in Seattle, WA. The role focuses on improving the accuracy, reliability, and transparency of AI systems through calibration and optimization.
About the Role
As an AI Engineer, you will embed with the client's AI and product engineering teams to enhance their Azure-hosted GPT model. Your responsibilities include building calibration pipelines, optimizing prompts, and establishing workflows for continuous improvement. You will lead initiatives to deliver production-ready calibration systems, measure performance metrics, and provide strategic guidance on AI architecture.
About You
Required:
- Bachelor's degree in computer science, Engineering, or equivalent experience.
- 7+ years of professional software development with significant ownership of architecture and delivery.
- 3+ years of Python in ML/AI systems with a strong focus on data processing and evaluation pipelines.
- 2+ years building with Generative AI including hands-on prompt engineering and optimization work.
- Experience with prompt optimization frameworks - DSPy strongly preferred.
- Deep understanding of RAG architectures and evaluation methods.
- Strong Azure cloud experience with focus on AI/ML services.
Preferred:
- Direct hands-on experience with DSPy framework and GEPA optimizer.
- Understanding systematic optimization principles.
- Familiarity with prompt optimization frameworks and methods.
- Experience with LLM-as-judge patterns and automated evaluation pipelines.
- Knowledge of advanced RAG patterns and retrieval evaluation methods.
Benefits
- Opportunity to work on impactful AI applications.
- Collaborative and innovative work environment.
- Professional development and mentorship opportunities.
Robots & Pencils
Robots & Pencils develops digital strategies and products that deliver exponential impact to our clients. We design and build solutions that unlock data and insights, infuse intelligent automation, and accelerate product innovation across the organization. Everything we do starts by blending the sciences with the humanities, the Robots with the Pencils. Our top-tier talent fuses creativity + technology to help brands transform their businesses, deliver delightful customer and employee experiences, and maintain a competitive edge amidst a constantly changing industry landscape.
Seattle, WA