Staff Data Software Engineer

New York, New York

Software Engineering

Permanent

Our client, an AI-driven organization within the Healthcare industry, is hiring a Staff Data Software Engineer to join their team in New York. The successful candidate will design, build and scale the data infrastructure that underpins agent improvement, clinical analytics and research collaboration. You will own streaming and batch pipelines to process agent conversations, clinical events and patient outcomes at scale.

Responsibilities

Build and operate streaming and batch data pipelines on Databricks using Spark and Delta Lake.
Design, implement and maintain CDC (Change Data Capture) pipelines that sync operational databases into Delta Lake.
Develop data mining pipelines for persona discovery, scenario extraction and edge-case detection.
Build and own the data backend for the Research Platform, including natural-language-to-SQL capabilities.
Implement robust data quality checks, staleness detection and automated alerting.
Develop pipelines for voice and SMS analytics, including call quality and engagement metrics.
Support multi-region data deployments and compliance requirements.
Collaborate closely with agent engineers and data scientists to surface insights that improve agent performance.

Skillset

At least 4 years of experience in production data engineering roles.
Deep, hands-on experience with Databricks, Spark and Delta Lake.
Strong proficiency in Python and SQL for building and maintaining data pipelines.
Experience designing and operating streaming pipelines and CDC (change data capture) systems.
A solid understanding of data modelling, medallion architectures (bronze/silver/gold) and query optimisation.
Experience implementing data quality frameworks, monitoring and alerting.
A proven track record of delivering reliable, production-grade data infrastructure.
Exposure to machine learning pipelines, including feature engineering and training infrastructure is desirable.
Experience building natural-language query interfaces or LLM-powered data tools is a bonus.
Experience working with healthcare data and familiarity with HIPAA compliance requirements is a plus.