Lead Data Engineer

Added: 01/09/2020

REF: 4687

Contract: Permanent

Location: New York, United States

About the Opportunity

Our client is looking for a Lead Data Engineer to be part of a team of people accomplished in diverse scientific and engineering disciplines, focused on using the best of what lies at the forefront of technology to address complex, real-world problems that have a positive impact on potentially millions of peoples' lives. The Data Engineer will work very closely with our front-end developers, back-end developers, development operations engineers, and data scientists. The platform is fully cloud-based and is being built around modern tools and frameworks in an incredibly fast-moving agile environment.

Key Responsibilities
  • Help the people who report to you grow and accomplish more by setting clear goals, providing direct feedback, managing project loads and allocating resources accordingly, and ensuring deadlines are met
  • Lead the development of the data platform to drive data science powered applications
  • Develop, and implement data infrastructure and pipelines that ingest and transform data from various external sources, storing it in highly optimized database systems, and making it useful to our application and reporting layers
  • Create automation systems and tools to configure, monitor, and orchestrate data infrastructure and pipelines
  • Create data integration services to help onboard new customers as quickly as possible
  • Maintain ongoing reliability, performance, and support of the data infrastructure, providing solutions based on application needs and anticipated growth
  • Participate in creating and maintaining strict compliance, data privacy and security measures
  • Develop robust and production-level code to implement new product features in collaboration with other engineers and subject matter experts
  • Identify and resolve performance and scalability issues, troubleshoot problems, and improve product quality
  • Enable effective working relationships across Product Management, Front-End Development, Data Science, Development Operations and Data Engineering
  • Collaborate with the Front-End Development team to thread the right information through to forward-facing applications

Minimum Requirements
  • 3+ years relevant experience leading data engineering
  • 5+ years relevant experience with data engineering
  • Strong proficiency with Python (ideally PySpark) and SQL
  • Experience with AWS S3, EC2, EMR, or an equivalent cloud-hosted infrastructure
  • Experience with cloud-hosted database/data warehouse architecture (e.g. Redshift, Snowflake, etc.)
  • Experience writing and productionizing complex data transformations in SQL and related frameworks
  • Interest in building distributed computing and orchestration frameworks (e.g. Spark, Kubernetes, Airflow, etc.)
  • Experience working in an Agile software development environment
Preferred (Nice-to-have) Qualifications
  • Experience building and deploying large-scale data processing pipelines
  • Experience integrating data from disparate data sources
  • Experience with continuous integration and automation tools and processes (e.g. Jenkins, Semaphore, etc.)
  • Experience with healthcare data, ideally clinical/operational clinical trial data

Apply Now

Complete the form below to apply for the Lead Data Engineer role:

Add Your CV

Alternatively select from

View all jobs