Computational Linguist, Generative AI - Sr. Associate

We're looking for a versatile Computational Linguist to join our team focused on evaluating and supporting Generative and Agentic AI systems.

This role combines linguistic expertise, data analysis, and hands-on experimentation with large language models.
This role is ideal for someone who can move between qualitative language analysis and quantitative evaluation.

You'll work cross-functionally with Machine Learning Engineers, Analytics team and annotators to design innovative, rigorous, and scalable evaluation processes for LLM-powered workflows.

Core Responsibilities
• Serve as a subject matter expert, engaging with various stakeholders throughout the product lifecycle, and maintain a strong understanding of the Chase Digital Assistant's model from both customer and technical perspectives.
• Manage, monitor, and evaluate and version the Chase Digital Assistant's intent and entity taxonomy and the model training; Enforce taxonomy versioning practices to ensure traceability and rollback capability
• Work closely on the adaptation to LLM-driven workflows, ensuring seamless integration of LLMs with existing conversational AI architectures and event tracking systems.
• Implement and evolve metrics and KPIs across Model Correctness, Customer Experience, AI Assurance, and Business Metrics and ensure evaluation is transparent, repeatable, and release-decision-ready
• Maintain established metrics and introduce new guardrail metrics for LLM and generative use cases
• Manage full artifact suite for LLM models including descriptions, prompts, evaluation rubrics, LLM-as-judge prompts, guidelines, calibration data, data statistics & reliability measures.
• Apply ontology design principles to improve semantic reasoning and data integration aligned to business standards.
• Design frameworks to incorporate knowledge graphs within classification and extraction model architectures.
• Align knowledge models with RAG pipelines and agent orchestration to enhance AI functionality.
• Work with data scientists, software engineers, and business stakeholders to translate requirements into robust solutions.
• Identifying optimization opportunities across teams, supporting continuous improvement across model performance, data quality, and feature coverage for improving customer experience.

Required Qualifications, Capabilities, and Skills
• Master's degree in Computational Linguistics, NLP, Linguistics, or related field
• 2+ experience in Computational Linguistics or NLP applied to chatbot or conversational AI development
• Hands-on experience with Generative AI and Agentic AI frameworks and evaluation (e.g., AutoGen, LangGraph, CrewAI, Sierra)
• Linguistic background in discourse & pragmatics
• Advanced knowledge of conversational AI product development lifecycle - training, design, conversation analysis
• Hands-on experience with LLM integration, prompt engineering, evaluation, and performance monitoring
• Proficient in Python, Git...