Location: Dublin, Ireland
We are looking for a Principal Data Engineer. This role is in the centerpiece of the ATC tech stack, and collaborates with many teams (machine learning, infrastructure, operations) and has strong ties to the business (project delivery)
• Craft and deliver, scalable, low-cost solution flows, moving data from production streams to data lake for consumption by ML teams.
• Be passionate about working with huge & heterogenous data types and love bringing datasets together to answer business questions and drive next level automation of processes.
• Write code, automate and leverage tools, to ingest and transform data using clean coding principles, while incorporating business logic as defined in conjunction with stakeholders
• Define best practices and standards in the Data area for processing and analyzing data
• Migrate data from legacy systems to new solutions, such as from on-premises clusters to Cloud
• Design conceptional and logical data models, architecture diagrams and flowcharts
• Design and facilitate data-monitoring and observability models for pipelines deployed
- Proven experience as a Data Engineer or in a similar role.
- Experience with data modeling, data warehousing and building ETL pipelines.
- Strong experience in SQL, Python, PySpark or other related technologies.
- Experience using Airfow for orchestrating data pipelines.
- Knowledge working with Kubernetes/Docker.
- Experience building data pipelines on either AWS, Azure or GCP, following best practices in Cloud deployments
- Familiar with Databricks development environment, creating scheduled data analysis workloads on cluster environment.
- Ability to debug complex data issues while working on very large data sets with billions of records
- Understanding of DevOps tools, Git workflow and building CI/CD pipelines
- Ability to work a portion of monthly work hours from our Dublin or Letterkenny office is required
- Strong communication and documentation skills.
- Experience working with Apache Kafka, building appropriate producer/consumer apps
- Familiarity with production quality ML and/or AI model development and deployment.
- Knowledge of modern DevOps tools and practices, around code-based release and deployment
- Experience building ML pipelines that rely on quality-controlled data inputs.
- Experience in testing database systems.
Complete the form below to apply for the Principal Data Engineer role: