US Jobs US Jobs     UK Jobs UK Jobs     EU Jobs EU Jobs


Staff AI Engineer - QA

Harris Computer is hiring a Staff AI Engineer to join their, expanding team! We’re seeking a seasoned expert to pioneer the quality and reliability of our cutting-edge AI and machine learning solutions for healthcare.

As a Staff AI Engineer focused on Quality Assurance, you will be the technical authority responsible for ensuring our AI systems are robust, fair, accurate, and safe.

This is a unique opportunity to define the future of AI quality in a domain where it matters most, mentoring a talented team and establishing the gold standard for testing intelligent systems that impact patient care.

This is a full-time, remote position open to candidates in the U.S.

or Canada.

What your impact will be:


* Build the core AI system end-to-end, from the service layer and data pipelines to the evaluation infrastructure, and own its architecture.


* Design and build the evaluation infrastructure, including dataset management, automated test harnesses, and replay of AI systems against historical workflows.


* Build and maintain evaluation datasets with clinician-reviewed correct answers, ensuring they evolve as our systems change.


* Create error metrics weighted by severity, so mistakes with clinical or financial impact are prioritized accordingly.


* Build production monitoring for accuracy, overrides, and safety issues, with actionable alerts for the engineering team.


* Write adversarial tests for AI systems, including hidden or malicious instructions in documents and invented content.


* Work with product and clinical leads to define accuracy thresholds and the criteria for expanding or pulling back an AI capability.


* The candidate must possess a curiosity and willingness to actively adopt and leverage emerging AI tools to improve workflows, solve problems, and drive efficiency along with being comfortable using a range of AI-enabled tools (such as copilots, chat-based AI, and automation solutions) as part of everyday work.

What we’re looking for:


* 7 or more years in software or machine learning engineering, with at least 3 years focused on evaluation, testing, or quality for AI systems.


* A track record of designing and building complete systems from scratch, taking them from prototype to production.


* A track record of building and running evaluation systems that teams use in production, not just for one-off analyses.


* At least 2 years of experience evaluating large language model (LLM) systems in a production environment.


* Strong statistical skills for measuring accuracy, including sampling, confidence intervals, and error analysis.


* 1 year + of experience designing evaluation sets for document extraction, classification, or structured output.


* Deep knowledge of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).


* Clear written communication, including design documents, and the ability to explain complex trade-offs to clinicians, e...




Share Job