A Hadoop developer is responsible for the design, development and operations of systems that store and manage large amounts of data. Most Hadoop developers have a computer software background and have a degree in information systems, software engineering, computer science, or mathematics.
The Developer will be supporting the EDW and GWCS projects. The developer needs working experience to make changes to the Abinitio code along with building Spark code for GWCS. The developer will lead the Claims Diamond process which will be loading data in near real time into GWCS. The developer needs to have working knowledge to support different data formats like xmls, Jsons, Fixed length files. Supporting fellow developers doing code reviews and provide documents for best standards and practices. Datawarehousing concepts along with SQL querying skills a must.
Write complex Spark code to perform ETL and load into different target databases like DB2, Mongo, Postgres.
Knowledge in scheduling tools like Control M.
Experience in integrating with Kafka/MQ is required,
Knowledge in code migration and deployment using Jenkins and GITLab.
Experience to process bulk volume data in batch and real time for data warehousing is required.
Experience working with S3 datalake is preferred.
Provide guidance on projects and ensure the project is implemented in the specified timelines.
Build reusable code for complex processes.
Perform unit testing and debugging, be able to debug most program errors and provide solution to resolve the error.
Write detailed technical specifications , identify integration points, ensure sufficient quality and compliance of documentation to architectural standards.
Build Dataflow diagrams, solution documents, be involved in the architecture design for projects and provide inputs on the design.
Perform code reviews with lead and work as an individual and as a team.
Act as a liaison for the team and work with all the users consuming our data for analytics/reporting needs.
Experience in Spark using Scala, Spark Sql, Hive, Hbase, NIFI, Performance tuning, best practices to follow while writing Spark code, experience in Kafka and real time data processing experience.
DATA SCIENCE TECHNOLOGIES LLC is an equal opportunity employer inclusive of female, minority, disability and veterans, (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. DATA SCIENCE TECHNOLOGIES LLC will not make any posting or employment decision that does not comply with applicable laws relating to labor and employment, equal opportunity, employment eligibility requirements or related matters. Nor will DATA SCIENCE TECHNOLOGIES LLC require in a posting or otherwise U.S. citizenship or lawful permanent residency in the U.S. as a condition of employment except as necessary to comply with law, regulation, executive order, or federal, state, or local government contract