Big Data Engineer Lead 812 views

Manages and responsible for the successful delivery of large scale data structures and pipelines and efficient Extract/Load/Transform (ETL) workflows. Acts as the data engineering team lead for large and complex projects involving multiple resources and tasks, providing individual mentoring in support of company objectives.

Fundamental Components include but are not limited to:

Designs and develops complex and large scale data structures and pipelines to organize, collect, and standardize data to generate insights and addresses reporting needs. Writes complex ETL (Extract / Transform / Load) processes, design database systems, and develop tools for real-time and offline analytic processing. Develop frameworks, standards & reference material for architecture and associated products. Designs data marts and data models to support Data Science and other internal customers. Behaves as a mentor to junior team members to provide technical advice. Applies knowledge of Aetna systems and products to consult and advise on additional efforts across multiple domains spanning broader enterprise. Collaborates with the data science team to transform data and integrate algorithms and models into highly available, production systems. Uses in-depth knowledge of Hadoop architecture, HDFS commands, and experience designing & optimizing queries to build scalable, modular, and efficient data pipelines. Uses advanced programming skills in Python, Java, or any of the major languages to build robust data pipelines and dynamic systems. Integrates data from a variety of sources, assuring that they adhere to data quality and accessibility standards. Experiments with available tools and pieces of advice on new tools in order to determine the optimal solution given the requirements dictated by the model/use case.

Qualifications Requirements and Preferences:

Strong collaboration and communication skills within and across teams. Ability to communicate technical ideas and results to non-technical clients in written and verbal form.Proven ability to create innovative solutions to highly complex technical problems.Ability to leverage multiple tools and programming languages to analyze and manipulate large data sets from disparate data sources.Ability to understand and build complex systems and solve challenging analytical problems.

Advanced knowledge in Java, Hive, Cassandra, Pig, MySQL or NoSQL or similar.

Python and Hadoop experience is a must.

Advanced knowledge in Hadoop architecture, HDFS commands, and experience designing & optimizing queries against data in the HDFS environment.

Experience building and implementing data transformation and processing solutions.

Has in-depth knowledge of large scale search applications and building high volume data pipelines.

Experience with bash shell scripts, UNIX utilities & UNIX Commands.

7 or more years of progressively complex related experience.

Master’s degree or Ph.D. preferred. Bachelor’s degree or equivalent work experience in Computer Science, Engineering, Machine Learning, or related discipline.