Lead Software Engineer - Databricks/Spark/AWS

This is your chance to change the path of your career and guide multiple teams to success at one of the world's leading financial institutions.

As a Lead Software Engineer at JPMorgan Chase within Corporate Sector, Chief Technology Office, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way.

As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm's business objectives.

Job Responsibilities:


* Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).


* Establish lakehouse patterns with Delta Lake (ACID transactions, schema evolution, time travel, Z-ordering, compaction) and ensure performance at scale.


* Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.


* Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.


* Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, unit scripts, cluster policies, pools, and instance profiles.


* Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.


* Design secure data ingestion and transformation frameworks leveraging AWS services:
+ S3 for data lake storage and lifecycle management
+ Glue for catalog/metadata and ETL jobs
+ IAM and Secrets Manager for role-based access and credential management
+ CloudWatch for logging, metrics, and alerting
+ Lambda for serverless utilities
+ Kinesis and/or Kafka/MSK for streaming ingestion


* Enforce data quality, lineage, and governance using Unity Catalog and/or Glue Catalog; embed expectations and validation into pipelines.


* Drive Spark performance engineering: partitioning strategies, file sizing, AQE, broadcast joins, shuffle tuning, caching, spill/memory control, and job right-sizing to optimize cost.


* Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit, integration, and data validation testing.


* Implement CI/CD for data projects (Git-based workflows), Terraform Infrastructure deployments environment promotion, and automated deployments; champion engine...