Senior Automation Engineer (AI and LLMs) - Remote

Summary

This role will be joining a team transforming healthcare and improving the lives and vitality of millions of members we serve.

We leverage cutting-edge Artificial Intelligence (AI) and Machine Learning (ML) - including Large Language Models (LLMs) - to revolutionize healthcare applications, from automated document processing to intelligent chatbots.

We are seeking a Senior Automation Engineer with strong engineering and full-stack testing expertise to build scalable, continuous testing frameworks that ensure rapid delivery and unwavering product quality.

Your work will directly influence the healthcare journey of our members and providers, shaping the future of healthcare every day.

A relentless focus on speed to market, product excellence, and innovation is essential to succeed in this role as we strive to transform healthcare at scale.

Responsibilities


* Design & Implementation of Automated Test Frameworks
+ Develop, maintain, and optimize automated testing solutions to validate AI platform functionalities and LLM-powered applications
+ Utilize tools and frameworks (e.g., Selenium, PyTest, Jest, etc.) to ensure test coverage and reliability



* Testing AI Solutions & LLMs
+ Build automation scripts to test and validate AI-driven chatbots, conversational agents, and enterprise-level AI applications
+ Define testing strategies specific to AI/ML workflows, including data preprocessing, model validation, and deployment
+ Collaborate with platform and solution teams to identify potential edge cases and performance bottlenecks for AI models



* Integration Testing & Continuous Delivery
+ Work closely with DevOps teams to integrate automated tests into CI/CD pipelines (e.g., Jenkins, GitHub Actions, Azure DevOps)
+ Contribute to pipeline scripts and infrastructure-as-code to ensure smooth, automated deployments
+ Liaise with development, product, and design teams to understand new features and enhancements, translating them into scalable test cases
+ Provide feedback and recommendations to improve usability, performance, and overall product quality



* Monitoring & Reporting
+ Implement real-time test reporting and dashboards to track testing progress and issues

Qualifications


* 5+ years' experience with application automation


* Proven experience building automation scripts to test and automate AI-driven chatbots, conversational agents and enterprise-level AI applications


* Proven experience test strategies including data preprocessing, model validation and deployment


* Experience with DevOps tools such as Jenkins, GitHub Actions and Azure DevOps


* Experience using tools and frameworks such as Selenium, PyTest and Jest

If you will be working at home occasionally or permanently, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds ...