Senior Data Engineer

Added: 15/07/2022

REF: 21901

Contract: Permanent

Location: Dublin, Ireland

Senior Data Engineer

 

The Data Engineer will be involved in the architecture, design, development and management of large scale, highly distributed, multitenant data stores and pipelines. In addition to building and maintaining these, the Data Engineer will also work to ensure that the data is easily accessible and available to data scientists and business users across the enterprise.

 

Main Duties

 

• The candidate will work across the full SDLC following agile best practises

• Enable data science and business units by implementing infrastructure and tools required to access and query large data sets

• Design, build, scale and maintain big data pipelines and stores

• Produce reusable components and microservices for completing data operations on cloud and edge

• Produce detailed documentation such as user guides, technical reports, and presentations

• Work on green field and legacy projects to different levels of quality i.e. proof of concept, alpha products, production quality products.

• Collaborate broadly across multiple functions (data science, engineering, product management, IT, etc.) to readily make key data readily available and easily consumable

• Explore and recommend new tools and processes which can be leveraged across the data preparation pipeline for capabilities and efficiencies

• Mentor others in the use of tools and techniques

 

Qualifications Required

 

• 4+ years of progressive experience in delivering data engineering solutions in a production or R&D environment

• Experience with distributed computing using Spark and PySpark or Scala.

• Experience designing data models and creating scalable, optimised big data stores

• Experience building and scaling big data ETL pipelines using tools such as Kafka, Spark, Airflow and Hadoop based stores

• Experience creating infrastructure and systems on cloud, preferably Azure

• Exposure to Kubernetes, Docker or other similar container-based systems

• Experience building APIs to support data consumption

  • Excellent communication (verbal, presentation, documentation) skills, working with teams that are geographically dispersed, to produce solutions that satisfy functional and non-functional requirements

Qualifications Desired

 

• Experience with data system security e.g data encryption, system network design etc

• Experience in edge computing, creating and delivering data/components/microservices to edge devices

• DevOps experience, particularly building CI/CD pipelines

• Understanding of Machine Learning tools & processes

  • Experience with Data visualisation tools  

Apply Now

Complete the form below to apply for the Senior Data Engineer role:

Add Your CV

Alternatively select from

View all jobs