AI/ML Career
10,038 roles at 2,899 companies

Lead Data Engineer - Data Engineering, Databricks, Model Serving, ML Ops, Data Architecture and modeling

Join us as we embark on a journey of collaboration and innovation, where your unique skills and talents will be valued and celebrated. Together we will create a brighter future and make a meaningful difference. 

As a Lead Data Engineer at JPMorganChase within the Consumer and Community Banking , you are an integral part of an agile team that works to enhance, build, and deliver data collection, storage, access, and analytics solutions in a secure, stable, and scalable way. As a core technical contributor, you are responsible for maintaining critical data pipelines and architectures across multiple technical areas within various business functions in support of the firm’s business objectives. 

Job responsibilities

  • Architects and maintains large-scale data pipelines using Apache Spark and related distributed computing frameworks, ensuring efficient processing of structured and unstructured data.

  • Builds and optimizes both batch and streaming data workloads to support analytical and operational use cases.

  • Develops, optimizes, and productionizes notebooks, jobs, and workflows within the Databricks environment to support analytical and machine learning use cases.

  • Designs and implements data architecture and data modeling strategies, including dimensional models, lakehouse structures, and schema design that balance performance, scalability, and governance requirements.

  • Writes and optimizes complex SQL queries for data transformation, validation, and performance tuning across large datasets.

  • Supports ML model serving infrastructure, ensuring trained models are deployed, monitored, and scaled reliably in production environments.

  • Collaborates closely with data scientists and ML engineers on MLOps practices, including model versioning, CI/CD pipelines for ML artifacts, monitoring, and automated retraining workflows.

  • Partners with cross-functional stakeholders — including analytics, risk, and platform engineering teams — to translate business requirements into technical data solutions.

  • Maintains strong data quality, lineage, and security standards consistent with data governance and compliance expectations.

  • Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity across complex deliverables (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards; contributes learnings and reusable patterns to improve broader team effectiveness.

  • Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation

Required qualifications, capabilities, and skills

  • Formal training or certification on data engineering concepts and 5+ years applied experience 

  • Demonstrates strong, hands-on proficiency with Apache Spark or comparable distributed compute frameworks, including experience optimizing jobs for performance and cost at scale. 

  • Demonstrates hands-on exposure to distributed computing, cloud data platforms, and production ML infrastructure.

  • Holds direct, production-level experience with Databricks, including notebooks, Delta Lake, job orchestration, and cluster management.

  • Demonstrates solid command of SQL, with the ability to write, debug, and optimize complex queries against large and varied datasets.

  • Brings practical experience supporting ML model serving infrastructure in production environments and shows familiarity with MLOps tooling and practices, such as model monitoring, versioning, and deployment automation.

  • Possesses a strong foundation in data architecture and data modeling, covering both operational and analytical data store design.

  • Has working knowledge of cloud data platforms such as AWS, Azure, or GCP.

  • Maintains proficiency with version control systems such as Git.

  • Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security.

  • Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices 

 Preferred qualifications, capabilities, and skills

  • Experience with Airflow or Databricks Workflows, Docker and Kubernetes, and CI/CD pipelines for data and machine-learning workflows.

  • Familiarity with feature stores and model registries such as MLflow, and experience with real-time streaming architectures such as Kafka and Spark Structured Streaming.

  • Prior experience in financial services or another highly regulated industry, with an understanding of data governance and compliance requirements.

Summary

Architects and maintains large-scale data pipelines and architectures using Apache Spark and Databricks for analytical and operational use cases. Requires 5+ years of applied data engineering experience with strong proficiency in SQL, data modeling, distributed computing, and production ML infrastructure. Supports ML model serving, MLOps practices, and cross-functional data solutions.

Responsibilities

Architect and maintain large-scale data pipelines; build batch and streaming workloads; develop Databricks notebooks, jobs, and workflows; design data architecture and models; optimize SQL; support ML model serving and MLOps; ensure data quality, lineage, and security

Qualifications

Formal training or certification on data engineering concepts and 5+ years applied experience; hands-on proficiency with Apache Spark, Databricks, SQL, cloud data platforms, Git, and production ML infrastructure; experience with AI-assisted development tools

Experience requirements

5+ years applied experience in data engineering