Data Platform Manager

Description

Data Platform Manager

What you will do

The Data Platform Manager will be responsible for managing, optimizing, and scaling our Databricks environment, enabling data engineering, machine learning, and analytics capabilities across the organization.

Key Responsibilities

• Own the end-to-end management of the Databricks platform, including setup, configuration, scaling, and performance optimization to meet diverse business needs.

• Build, mentor, and lead a high-performing team of platform engineers and architects, fostering a culture of collaboration, innovation, and continuous learning.

• Implement and enforce best practices for data governance, security, and compliance within Databricks, including managing Unity Catalog for streamlined data discovery and access control.

• Monitor platform usage and optimize for cost efficiency without compromising performance, leveraging features like cluster policies and auto-scaling.

• Partner with data scientists, analytics teams, and business stakeholders to ensure the platform meets evolving requirements and drives business value.

• Stay at the forefront of Databricks advancements, experiment with new features, and drive the adoption of best practices to continuously enhance the platform's capabilities.

• Serve as the primary point of contact with Databricks and other relevant vendors, managing contracts, service levels, and fostering a strong partnership.

What we are looking for

Required Qualifications

- Bachelor's degree in computer science, Data Science, Engineering, or a related field; Master's degree preferred.

- Minimum 8 years of experience in data platform management, with at least 3 years specifically managing Databricks environments at scale.

- Deep expertise in Databricks, Apache Spark, Delta Lake, and related big data technologies.

- Proven track record of designing and implementing complex data architectures and pipelines in a cloud environment (Azure, AWS, GCP).

- Strong understanding of Databricks-specific features like Delta Sharing, Unity Catalog, and MLflow.

- Experience with data security, compliance, and governance practices, including managing access controls and data lineage within Databricks.

- Excellent leadership skills with a passion for developing talent and building cohesive, empowered teams.

- Exceptional problem-solving abilities, strategic thinking, and a results-driven mindset.

- Outstanding communication skills, with the ability to articulate technical concepts to non-technical stakeholders.

Preferred Qualifications:

- Databricks certifications such as Databricks Certified Data Engineer Associate/Professional or Databricks Certified Machine Learning Professional.

- Experience with DevOps practices for Databricks, including CI/CD pipelines and automation tools.

- Familiarity with integrating Databricks with other data tools like Power BI, Tableau, Snowflake, and other ETL solutions.