Reference ID: D1TECH-61
Are you a critical thinker? Do you love algorithms and large data?
Our client is looking for a savvy Data Engineer to create solutions suitable for the position in a real-time, highly data-driven organization and make technical innovations happen. With a range of projects to develop data processing applications, you will have the opportunity to use cutting edge technology across all stages of the project lifecycle.
Summary of the Role:
As part of the Data & Analytics team, you will responsible for designing, developing, and supporting the infrastructure to collect, analyze, and transform large volumes of business-data into enterprise reporting and analytic solutions. You will also be responsible for optimizing data flow and collection for cross functional teams. The Data Engineer will support our software engineers, database architects and analysts on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. You must be self-directed and comfortable supporting the data needs of multiple teams, systems and products.
What you'll be doing:
•Develop core data relationship, data ingest, data transformation services and search capabilities.
•Maintain all facets of data infrastructure, including backups/disaster recovery, software deployment/configuration, security best practices, performance tuning, and troubleshooting.
•Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
•Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS 'big data' technologies
•Review project objectives and determine best technology for implementation. Implement best practice standards for development, build and deployment automation.
•Design, maintain and optimize highly distributed analytics data stores
•Write well-abstracted, reusable and efficient code
•Analyze large data sets and extract patterns that lead to greater customer insight, optimized performance, enhanced engagement, improved retention, increased revenue and decreased cost
•Work with business partners and management teams to the ensure collection and analysis of appropriate data and metrics to facilitate improvements in processes and profit
•Work with executives and leadership teams to formulate hypotheses and design research aimed at improving key performance indicators
•Utilize expertise in data modeling, ETL architecture, and report design for department initiatives, and produce detailed documentation including data flow diagrams, logical diagrams, and physical diagrams as needed
What we need from you:
•5+ years of relevant Data Engineering technical experience
•Bachelor's degree in Technology based discipline preferred
•Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
•Build processes supporting data transformation, data structures, metadata, dependency and workload management
•Experience with AWS cloud services: Lambda, S3, Glue, Redshift, and Athena, or their open source equivalent (Zeppelin, Presto, etc)
•Experience with data pipelining with AWS Datapipeline, Airflow or Luigi
•Experience with object-oriented/object function scripting languages: Python code development, python unit testing (tox), Pyspark
•Up-to-date on latest industry trends; able to articulate trends and potential clearly and confidently
•Understanding of best practices within the development process
•Experience in providing clients with reports, dashboards, and analytics
•Strong organizational skills and the ability to manage a diverse work load in a fast-paced environment.
If this describes you and your interest, please submit a cover letter highlighting why you would be great at this
role along with an updated resume in word format.
Due to the volume of resumes we receive, only those applicants whose skill set match our requirements will be contacted.
All applicants must have legal authorization to work in Canada for an unlimited period of time.
Posted: April 18, 2019
Closes: June 17, 2019