Lead Data Engineer

Position Summary: 

The Lead Data Engineer is responsible for the technical direction, reliability, and scalability of Penn Foster Group’s data platform.

This role provides hands-on technical leadership across data ingestion, transformation, and analytics enablement, while setting engineering standards and guiding the work of other data engineers. 

The Lead Data Engineer partners closely with BI, product, platform, and MLOps teams to ensure data systems are trusted, well-governed, and aligned to business outcomes.

This role operates as a player-coach, combining deep hands-on technical execution with technical leadership, mentoring, and architectural ownership in a highly visible and strategic data organization. 

Essential Job Functions:  


* Lead and coordinate the day-to-day technical work of data engineers, providing direction, prioritization, and technical guidance.


* Set clear technical expectations and hold the team accountable to engineering standards and delivery commitments.


* Own the technical architecture and engineering standards for the data platform, with a primary focus on Databricks in Azure.


* Lead the design, development, and operation of scalable, reliable data pipelines using SQL, Python, Spark, and Databricks.


* Establish and enforce best practices for data modeling, data quality, testing, performance optimization, and observability.


* Provide hands-on technical leadership through code reviews, design reviews, and mentoring of data engineers.


* Guide and influence the work of other data engineers, setting clear technical direction and expectations.


* Partner with BI and analytics teams to enable trusted datasets, performant semantic models, and governed self-service analytics.


* Collaborate with platform, DevOps, and MLOps teams on infrastructure, security, deployment patterns, and operational readiness.


* Balance hands-on delivery with technical roadmap planning and architectural decision-making.


* Act as the escalation point for complex data issues, production incidents, and cross-team technical challenges.


* Contribute to the evolution of the data operating model, including governance, access control, and self-service enablement 

 

Knowledge, Skills, Abilities:  


* 7+ years of experience in data engineering or platform engineering.


* 3+ years of experience leading or mentoring data engineers in a technical lead, player-coach, or team lead capacity.


* Demonstrated ability to guide and influence engineers without formal people-management authority.


* Deep hands-on expertise with Databricks, including Spark (PySpark/Scala), Delta Lake, Unity Catalog, cluster policies, jobs, notebooks, and performance optimization.


* Strong expertise with Azure, including ADLS Gen2, Databricks on Azure, networking, identity and permissions (AAD/RBAC), and integration with other Azure data services.


* Advanced proficiency in SQ...