Backend Chapter Lead ​(Data engineering focused)

vacancy-thumb
Location

Remote

Experience required

Airflow - 4 years Experience in ETL Experience in Argo Experience in Airbyte Experience in DBT Kafka - 4 years Experience in Java Experience in Akka Experience in Shapeless Experience in JSON

Department

Software Development

Project description

We are looking for a Backend Chapter lead-Big data developer (streaming data software development)

Libraries used:
Cats framework, cats-effects, cats-free
http4s/fs2
Circe (JSON library)
monocle
spark

Kafka and Kafka connect (here we have some code in Java where we extend current connectors and create custom SMT (single message transformers)
Spark: batch and structured streaming
Shapeless (not used yet but it can be needed)
Reactive manifesto, Actor pattern Akka (not used yet but it may be needed)

Technical Experience:​
-Knowledge of functional programming E Key contributions to technical architecture and design​
-Experience with defining data pipelines including a complete end-to-end life cycle including data quality and governance, metadata management approach, and actual deployment methodologies for data pipeline in a fully automated manner​
-Hands experience with Orchestration technologies such as Airflow or Glue ​

Responsibilities

-12 plus years of experience around big data/streaming data software development – streaming is from a data point of view, not service ​
-Experience with Big Data tools like Spark batch and structured streaming Spark Streaming, Storm, KSql, Kafka Streams ​
-Experience creating frameworks around data jobs batch, micro-batch, and streaming / near real-time computation ​
-Technical Mentorship and People Leadership of backend engineers across squads​

Requirements

ETL/Orchastration tools: Airflow, Argo, Airbyte, DBT.

Data quality and profiling: Deequ , Great Expectations.

Big Data: understanding shuffle, data skew, partitioning/bucketing, etc

Data: CAP theorem, understanding different types of databases and where to use them

Streaming: Event sourcing, CDC, actor pattern, consumer/producer pattern, etc

Kafka and Kafka connect (here we have some code in Java where we extend current connectors and create custom SMT (single message transformers)

Apply via site