Senior Data Engineer

Are you an experienced, passionate pioneer in technology who wants to work in a collaborative environment? As an experienced Senior Data Engineer you will have the ability to share new ideas and collaborate on projects as a consultant without the extensive demands of travel.

If so, consider an opportunity with Deloitte under our Project Delivery Talent Model.

Project Delivery Model (PDM) is a talent model that is tailored specifically for long-term, onsite client service delivery.

Recruiting for this role ends on 12/31/2026.

Work you'll do/Responsibilities


* Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, including escalating any matters that require additional attention and consideration from engagement management



* Design, develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks



* Write and tune PySpark / Spark SQL notebooks for large-scale data transformation


* Architect end-to-end data solutions across dev → UAT → prod environments using Unity Catalog


* Lead and drive design discussions with client architects and other counterparts


* Collaborate with different teams on data contracts and schema agreements


* Lead design and optimization of high-volume data pipeline.


* Define and enforce data engineering standards - naming conventions, partitioning strategies, cluster configurations, Spark tuning


* Drive performance optimization - AQE tuning, liquid clustering, broadcast joins, shuffle partition management


* Design Databricks cluster policies, autoscaling configurations, and cost optimization strategies


* Conduct root cause analysis on production incidents and implement permanent fixes


* Mentor junior and mid-level engineers through code reviews and pair programming


* Evaluate new technologies and recommend adoption (e.g., DABs, DLT, Auto Loader, Serverless Compute, event hubs)

The Team

AI & Engineering leverages cutting-edge engineering capabilities to build, deploy, and operate integrated/verticalized sector solutions in software, data, AI, network, and hybrid cloud infrastructure.

These solutions are powered by engineering for business advantage, transforming mission-critical operations.

We enable clients to stay ahead with the latest advancements by transforming engineering teams and modernizing technology & data platforms.

Our delivery models are tailored to meet each client's unique requirements.

Required Qualifications


* Python, PySpark, Spark SQL, SQL Server



* Azure (ADF, ADLS Gen2, Key Vault, Azure Monitor)



* Databricks (Delta Lake, Unity Catalog, Workflows)


* Apache Airflow


* Git / Azure DevOps


* Deep Spark internals (DAG optimization, spill analysis, skew handling)


* Delta Lake advanced features (time travel, deletion vectors, predictive I/O)


* Unity Catalog governance (row/column secu...