Databricks Data Architect

Databricks Data ArchitectReference Code 1847

Country: United States (US)

US Locations: USA - Tampa; USA - Hermitage; USA - Nashville

Deloitte Global is the engine of the Deloitte network.

Our professionals reach across disciplines and borders to develop and lead global initiatives.

We deliver strategic programs and services that unite our organization.

Work you'll do

The Databricks Data Architect is a senior technical leader responsible for building and optimizing a robust data platform in a financial services environment.

In this full-time role, you will lead a team of 10+ data engineers and own the end-to-end architecture and implementation of the Databricks Lakehouse platform.

You will collaborate closely with application development and analytics teams to design scalable data solutions that drive business insights.

This position demands deep expertise in Databricks (Azure), hands-on experience with PySpark and Delta Lake, and strong leadership to ensure best practices in data engineering, performance tuning, and governance.

Key Responsibilities

Lead, mentor, and manage a team of 10+ data engineers, providing technical guidance, code reviews, and career development to foster a high-performing team.

Own the Databricks platform architecture and implementation, ensuring the environment is secure, scalable, and optimized for the organization's data processing needs.

Design and oversee the Lakehouse architecture leveraging Delta Lake and Apache Spark.

Implement and manage Databricks Unity Catalog for unified data governance.

Ensure fine-grained access controls and data lineage tracking are in place to secure sensitive financial data and comply with industry regulations.

Provision and administer Databricks clusters (in Azure), including configuring cluster sizes, auto-scaling, and auto-termination settings.

Set up and enforce cluster policies to standardize configurations, optimize resource usage, and control costs across different teams and projects.

Collaborate with analytics teams to develop and optimize Databricks SQL queries and dashboards.

Tune SQL workloads and caching strategies for faster performance and ensure efficient use of the query engine.

Lead performance tuning initiatives for Spark jobs and ETL pipelines.

Profile data processing code (PySpark/Scala) to identify bottlenecks and refactor for improved throughput and lower latency.

Implement best practices for incremental data processing with Delta Lake, and ensure compute cost efficiency (e.g., by optimizing cluster utilization and job scheduling).

Work closely with application developers, data analysts, and data scientists to understand requirements and translate them into robust data pipelines and solutions.

Ensure that data architectures support analytics, reporting, and machine learning use cases effectively.

Integrate Databricks workflows into the CI/CD pipeline using Azure DevOps and Git.

Develop automated deployment processes for notebooks, jobs, and clusters (...