Build and manage the data asset using some of the most scalable and resilient open source big data technologies like Airflow, Spark, Kafka, etc
Build and manage a highly scalable, efficient Data and ML Infrastructure by adopting microservices driven design and architecture with proper DevOps principles and practices
Design and deliver the next-gen data lifecycle management suite of tools/frameworks, including ingestion and consumption on the top of the data lake to support real-time as well as batch use cases
Help the team in integrating various data sources across GFGs group vertical
Build and expose metadata catalog for the Data Lake for easy exploration, profiling as well as lineage requirements
JOB REQUIREMENT
At least 3+ years of relevant experience in developing scalable, secured, fault tolerant, resilient & mission-critical Big Data platform.
Able to maintain and monitor the ecosystem with high availability.
Must have sound understanding for all Big Data components & Administration Fundamentals. Hands-on in building a complete data platform using various open source technologies.
Must have good fundamental hands-on knowledge of Linux and building big data stack on top of AWS/GCP using Kubernetes.
Strong understanding of big data and related technologies like HDFS, Spark, Presto, Airflow, Kafka, Apache Atlas etc.
Good knowledge of Complex Event Processing systems like Spark Streaming, Kafka, Apache Flink, Beam etc.
Able to drive devops best practices like CI/CD, containerization, blue-green deployments, secrets management etc in the Data ecosystem.
Able to develop an agile platform with auto scale capability up & down as well vertically and horizontally.
Able to develop an observability and monitoring ecosystem for all the components in use in the data ecosystem.
Proficiency in at least one of the programming languages Java, Scala, Python or Go.
BENEFIT
Annual Leave: 15 days
Sick leave/Mental Health Days: 30 days.
Occasion leave: 1 day.
13th month salary.
Annual bonus: up to 3 months based salary (depends on company performance).
Home Workspace support: up to 7mil/person.
Learning budget up to EUR 500/person.
Medical benefits: Social insurance, medical insurance & AON insurance.
Macbook is provided.
Support for English class.
Support Gym membership.
Additionally, we offer a flat hierarchy, a fun and flexible work environment.