These platforms draw insights from all types of data be it IOT, Network, Device, Customer, Click Steam, etc using 100% open source technologies. This platform also enables Data engineers, Data Scientists and Subject matter experts to implement solutions in a fraction of the time they would have taken to build use cases.
- Well-versed in shell scripting for working on Unix/Linux-based systems.
- Sharp analytical and problem-solving skills.
- Hadoop based computing solutions including but not limited to – MapReduce, Spark, Hive, Tez, YARN, HBase, Kudu, Druid, Presto
- In-depth knowledge and hands-on experience of
- Distributed Memory Grid like Ignite, Redis, Hazelcast
- Distributed search systems like Solr, ElasticSearch
- Experience with handling data engineering services at scale experience with Airflow or similar workflow management platforms
- Excellent coding skills in Java/Scala/python especially in OOP constructs & concurrency systems; experienced in building highly optimal software systems. Good understanding of different Design Patterns and their usage.
- Building real-time streaming solutions including but not limited to – Kafka, Spark Streaming, Flink, Storm, NiFi
- Database and Dataware house-related experience
- Experience in designing, building, tuning & troubleshooting distributed, scalable data pipelines & data streaming solutions
Qualification & Experience:
- B.E/B.Tech in computer science, engineering, information systems, or equivalent, with a proven experience in data engineering
- Experience with common processing frameworks and technologies (eg Apache Spark, HDFS, Hbase)