Data Engineer

Job Description

Mission:
In collaboration with the data architect, data experts, and the product manager/owner, you will participate in operations and the implementation of the technical roadmap of the orchestration platform based on the Airflow product. You will ensure the development and expert support of both the platform’s functionalities and the surrounding DevOps tooling, with priority on observability, performance management, and continuous improvement of the Orchestration service.

Activities:

Participate in the development cycle of the Airflow product

Implement/develop the necessary tooling and functionalities of the platform, in alignment with the Data Architect and the group’s Data Experts.
Contribute to the development of Directed Acyclic Graphs (DAGs) required for the proper operation of the datalake and datamart platforms (AWS and Snowflake).
Provide technical support to end-users during their use of the platform.
Handle deliveries and code deployments.
Ensure the platform’s run mode

Provide level 2 production support and participate in crisis situations.
Contribute to the product’s Knowledge Base.
Report to the Airflow platform manager on IT performance, real-time health status, and main operational risks of the platform.
Contribute to the Data Orchestration community

Act as a key user liaison with the teams using the platform: BI, business projects, Data Science, DataLake, etc.
Participate in regular meetings with users to share best practices in usage and development.
Knowledge transfer to the offshore support team

Produce the necessary documentation for knowledge transfer: tutorials, operational documents, etc.
Conduct and lead knowledge transfer sessions through product demos.
Support the offshore team in their initial developments.
Assist the support team in resolving incidents/problems.

Deliverables:

Coding and production deployment of DAGs in Airflow.
Unit test reports of implemented DAGs.
Contribution to documents proposing platform evolutions.
Delivery/roadmap implementation planning via JIRA.
Documentation of development best practices in Confluence.
Updates of incident and request tickets in JIRA.
Support knowledge base (in ServiceNow and Confluence).
Meeting materials and minutes.
Required Skills:

AWS Cloud/Data Engineer
Process orchestration: Airflow
ETL/ELT principles: dbt
Agile Scrum/Kanban methodology
Knowledge of the data management ecosystem (Data Engineer profile)
DevOps basics
Basics of IT system observability

Soft skills:

Proactive
Autonomous
Dynamic
Team player / Good communicator (a plus)
Knowledge of the following tools:

Airflow (mandatory)
AWS
Python
Snowflake
dbt
JIRA/Confluence
Jenkins
Gitlab
Terraform
Kubernetes/Containers
Elasticsearch/Kibana (a plus)