Platform Engineer Consultant

Are you an experienced, passionate pioneer in technology who wants to work in a collaborative environment? As an experienced Platform Engineer Consultant you will have the ability to share new ideas and collaborate on projects as a consultant without the extensive demands of travel.

If so, consider an opportunity with Deloitte under our Project Delivery Talent Model.

Project Delivery Model (PDM) is a talent model that is tailored specifically for long-term, onsite client service delivery.

Work you'll do/Responsibilities

As a Platform Engineer Consultant, we are seeking a hands-on Senior Data & AI Engineer with deep expertise in the Databricks platform, Python/PySpark development, and emerging agentic AI frameworks.

This role combines traditional data engineering/platform operations with modern AI orchestration, requiring someone equally comfortable tuning Spark jobs as debugging an LLM agent's tool-calling loop.

Responsibilities;


* Troubleshoot cluster, job, and workflow failures across Databricks (compute, networking, permissions, library conflicts)


* Perform cost optimization: cluster right-sizing, job cluster vs.

all-purpose cluster strategy, auto-scaling policies, spot/on-demand mix, DBU usage analysis


* Conduct performance tuning: Spark query optimization, partitioning strategies, caching, Photon engine utilization, Delta Lake optimization (Z-ordering, compaction, vacuum)


* Administer and automate platform usage via the Databricks REST API and SDKs (workspace provisioning, job orchestration, cluster policies, Unity Catalog management)

Python & PySpark Development


* Build and maintain production-grade ETL/ELT pipelines


* Write performant, testable PySpark code for large-scale distributed data processing


* Apply software engineering best practices (version control, CI/CD, code review, unit/integration testing)

AI / Agentic AI Development


* Design, build, and deploy at least one AI agent end-to-end - from use case definition through production deployment


* Call and orchestrate LLMs (prompt design, context management, tool/function calling, multi-step reasoning chains)


* Implement agent workflows using frameworks such as LangChain and LangGraph (state machines, multi-agent orchestration, tool routing)


* Instrument and monitor agent behavior using LangFuse or similar observability tools (tracing, evaluation, prompt versioning, cost/latency monitoring)


* Collaborate with data scientists/ML engineers to productionize AI-driven features

Azure Cloud Infrastructure


* Design and manage Azure Storage Accounts (ADLS Gen2, blob storage, access tiers, lifecycle policies) integrated with Databricks


* Implement secure secrets management using Azure Key Vault (service principals, managed identities, Databricks secret scopes)


* Work across core Azure services supporting the data platform (e.g., Azure Data Factory, Azure Monitor/Log Analytics, Virtual Networks, Entra ID/RBAC, Az...