Senior Director of Data Engineering

Senior Director of Data Engineering 

Overview 

At Penn Foster Group, our purpose is to open doors to brighter futures by helping learners gain the skills they need to grow their careers.

We are seeking a highly experienced and dynamic Senior Director of Data Engineering to lead and manage our data engineering team.

The ideal candidate will have over 10 years of experience in the field, with a minimum of 5 years as a data architect.

This role requires a deep understanding of Databricks, Azure, AI/MLops, strong leadership skills, and proficiency in managing teams of data analysts and data engineers. This leader will also oversee data governance, data quality, observability, and cost optimization across the data platform to ensure reliability and scalability. 

Key Responsibilities 


* Lead, mentor, and develop a team of approximately 15 professionals, including 4 direct reports comprised of high-quality analysts, data engineering, and Tableau/Power BI developers.


* Oversee the design and implementation of data architecture strategies.


* Partner with engineering teams to incorporate AI/ML models into our core products.


* Ensure the effective operationalization of AI/ML models.


* Oversee data quality frameworks and enforce medallion architecture principles.


* Manage onshore and offshore staffing models to optimize productivity and efficiency.


* Implement and maintain event streaming architectures, preferably in Azure.


* Run and support agile teams, ensuring adherence to agile methodologies and best practices.


* Set and carry out engineering standards, including coding practices, data management, and security protocols, ensuring compliance with regulatory requirements.


* Introduce and scale data observability practices to monitor pipeline health, lineage, and performance.


* Confirmed proficiency in Python, PySpark, and SQL, with a deep understanding of data processing frameworks and large-scale distributed systems.


* Collaborate with senior leadership to align data strategies and team priorities with business objectives.


* Ensure strong data governance policies and compliance across the data ecosystem.


* Manage and optimize the data platform budget, including Databricks and Azure compute/storage costs. 

Qualifications 


* 10 plus years of experience in data engineering or related fields, with a minimum of 5 years as a data architect.


* 6-10 years of experience running agile teams.


* BA/BS in Computer Science, Math, Physics, or other technical fields


* Proven experience with Databricks, preferably on Azure managing its implementation and cost.


* Familiarity with SAP Business Objects is a plus


* Extensive experience managing teams of data engineers, data analysts and Tableau/Power BI engineers.


* Strong leadership skills with the ability to mentor and develop team members.


* Experience ...