Educational requirements: Bachelor
English requirements: Competent English
Requirements for skilled employment experience for years: 3-5 years
Required residence status: Temporary visa, Permanent resident, Citizen
Accept remote work: unacceptable
Implementation of a gateway that will improve consumer ability to compare and switch between financial Products and services thereby encouraging competition between service providers to enable better product pricing in customer services and help business to lead in Open Banking Domain in Australia.
Implementation of robust marketing solutions using the latest tech stack like Spark, Kafka, Scala, HBase and Azure to create a better reach to existing customer & find new areas of expanding customer base.
To provide a solution and design which can be easily scalable depends on the business needs and flexible and secure to operate on Azure cloud using existing Data Analytics Platform services & creating new services on Azure using the native services along with Spark Streaming, Kafka, Scala and HBase that meets the industry standards.
Must have strong banking domain knowledge and should be able to handle & process large volumes of customed financial products data and generate reports to help bank understand the customer issues, growth opportunities and improve customer satisfaction.
Should be highly proficient in latest Big Data technologies like Azure ADLS2, Azure event hub, Spark, Spark Streaming, Kafka, HBase, Hive, Scala, Python etc.
Should be able to handle complex business requirements including Data Integration from different sources, Data Transformations and Data Distributions across Azure.
Mandatory Skills:
At least 11 years of experience in Big Data Technology domain at various positions on development projects.
Experience in Banking Domain is must.
At least 5+ years of experience in Hadoop, Spark, HDFS, Spark, Azure, Python, and Hive.
Must have at least 5 years of experience with building real time stream-processing systems, using technologies such as Spark-Streaming, Kafka, Scala, HBase and Azure.
Proficient experience of Big Data querying tools Hive?and Impala.
Expertise in building & deploying batch solutions using Spark, Python, Scala, Hive and Azure.
Experience with various messaging systems, such as Flume, Kafka and Azure event hub.
Analysing industry wide data models, systems and Big Data & Data Analytics for solutions deployed in on-premises & cloud.
Must have experience with code management tools such as Bitbucket and GitHub.
Must have hands on experience in devops deployment tools – Jenkins, JIRA, Maven, sbt and Azure Devops.
Duties and Responsibilities:
Responsible to develop and enhance new and existing Big Data Streaming pipelines using Spark Streaming, Kafka and Azure.
Responsible to build robust framework for ETL and real-time data ingestion using Scala with Spark.
Performance Tuning of Spark processing.
Responsible to develop code to ingest data into Azure ADLS2.
Responsible to develop code to ingest data into Azure Event hub.
Develop pipelines to extract data from Azure Event hub and process the same using Spark Streaming and Scala.
Develop analytics applications using Spark, Spark Streaming, Spark SQL, Hive, Hadoop and Azure.
Design and build robust generic frameworks for both streaming and batch pipelines.
Develop ETL framework using Pyspark and unit tested the same to load data from external systems to Hadoop and Hive.
Provide roadmaps and impact analysis for the proposed solution.
Provide estimates for end -to-end solution that includes operational costs, resources etc.
Responsible to design the end-to-end data pipelines as per business requirements.
Responsible to manage the development sprints effectively and efficiently from planning to execution to review in an agile development environment.
Responsible to build and manage end-to-end big data pipeline as per business requirements.
Participating WSR’s and PWC meetings and stakeholder management.
Responsible to build Data Governance Platform.
Collaborating on requirement documentation.
