Data Operations Engineer

Key roles and responsibilities:

• Undertake the processing of structured and unstructured data to utilize in data science activities.

• Support the data analysis of large information to discover trends and patterns and provide feedback to the business.

• Assist in designing and implementing scalable and robust processes for ingesting and transforming datasets.

• Design, implement, and support the creation and maintenance of data pipelines from a multitude of sources.

• Ingest large, complex data sets that meet functional and non-functional requirements.

• Request For Change (RFC) management in Amazon Managed Services (AMS) (logging, escalation, resolution)

• Provisioning infrastructure in AWS (via RFC process, CloudFormation, Console)

• AWS Security Management (authoring IAM security policies, logging security requests, user management/permissions)

• Infrastructure As Code (development of CloudFormation templates)

• Development, deployment, and management of CICD (Continuous Integration Continuous Deployment ) pipelines.

• Linux and Windows System Administration within AWS (SSH, RDP).

• Design and build bulk and delta data lift patterns for optimal extraction, transformation, and loading of data.

• Supports the organisations cloud strategy and aligns to the data architecture and governance including the implementation of these data governance practices.

• Work with various stakeholders across the organization to understand data requirements and apply technical knowledge of data management to solve key business problems.

• Provide support in the operational environment with all relevant support teams for data services.

Knowledge, skills and attributes

• Working with large volumes of structured and unstructured data and leveraging them to build artificial intelligence (AI)/machine learning (ML) and predictive modelling (PM) solutions through end-to-end automated data pipelines.

• Deep and Extensive AWS knowledge and skills: EMR, Lambda, EC2, VPC, Transit Gateway, S3,IAM, Infrastructure As Code (e.g. CloudFormation), Code Commit, Code Pipeline and Code Build.

• Advantageous AWS experience with DynamoDB, RDS (MS SQL, Oracle), Landing Zone, Control Tower and AMS

• The candidate having DBA ability and knowledge across at least 2 platforms (for example TSQL, SAS, PSQL, IBM VSAM DB2 etc.) will also be beneficial.

• Experience with the Python programming language.

• Experience with designing and implementing Cloud (AWS) solutions including the use of APIs available.

• Experience with Dev/OPS architecture, implementation, and operation would be advantageous.

• Some experience in applying SAFe/Scrum/Kanban methodologies would be advantageous.

• Knowledge of Engineering and Operational Excellence using standard methodologies. Best practices in software engineering, data management, data storage, data computing and distributed systems to solve business problems with data.

• Strong AWS cost management, billing resolution and cost optimization

Academic qualifications and certifications:

• Matric, with a degree in Computer Science, Business Informatics or Data Engineering

• 5+ years IT work experience and 2+ years of experience in a senior technical position in a relevant role e.g. System Administrator, Network Engineer, DevOps Engineer, Cloud Engineer.

• 5 + years of data engineering experience

• 3+ years of experience with any data warehouse technical architectures, ETL/ELT, and reporting/analytics tools including, but not limited to, any of the following combinations (1) SSIS and SSRS, (2) SAS ETL Framework, (3) SAP ETL Framework, (4) MongoDB ETL deployments, (5) Apache Spark and Apache Hive deployments will be beneficial.

Originally posted on Himalayas