Data Engineer

Data Engineer (SQL / Databricks / Data Factory )
6-Month Contract
Location: Sydney, Hybrid
Day rate: $700-950 a day inc super DOE


Company Overview:
Talent Insights is currently partnered with an innovative fintech company transforming the financial services industry through advanced technology. They are currently undergoing a migration from on-premise systems to Azure cloud, and are looking for a Data Engineer to join the team for a 6-month contract to support this migration effort. The role is hybrid, based in Sydney, offering an exciting opportunity to work on cutting-edge cloud solutions.
  
Role Overview:
As a Data Engineer, you will play a key role in the migration of data infrastructure and pipelines from on-premise to Azure. Your expertise in SQL, Python, Azure Data Factory, and Azure Databricks will be essential in ensuring smooth and efficient migration of data workflows and processes to the cloud. You will collaborate with cross-functional teams to re-architect and optimize data solutions for the Azure environment, enabling better scalability, performance, and reliability.
  
Key Responsibilities for the Azure Data Engineer include :

• Assist in the migration of data infrastructure and workflows from on-premise systems to Azure Cloud.
• Design and develop Azure Data Factory pipelines to facilitate smooth data transfer and transformation from on-prem sources to cloud-based storage.
• Rebuild and optimize data pipelines using Azure Databricks, ensuring they are robust, scalable, and aligned with cloud best practices.
• Write and optimize SQL queries for migrating data from on-prem databases to Azure, ensuring data integrity and minimizing downtime.
• Work with data architects and business stakeholders to understand the current on-prem data architecture and map out the migration strategy to the cloud.
• Perform thorough testing, validation, and troubleshooting of the migrated data pipelines to ensure accuracy and performance.
• Collaborate with security and governance teams to ensure compliance with industry standards and regulations during the migration process.
• Provide ongoing support for the migration process, including monitoring and resolving any issues that arise during the transition.

Required Skills & Qualifications for the Azure Data Engineer include :

• Proven experience as a Data Engineer, with a strong background in migrating on-premise data infrastructure and systems to the cloud.
• Expertise in SQL for managing and migrating data between on-prem systems and Azure.
• Strong experience with Python for automating migration tasks and developing data transformation scripts.
• Hands-on experience with Azure Data Factory for data orchestration and Azure Databricks for data processing in cloud environments.
• Solid understanding of cloud-based data storage solutions in Azure and familiarity with Azure SQL Database, Azure Synapse Analytics, or similar services.
• In-depth knowledge of data migration strategies and best practices, especially from on-prem to cloud-based environments.
• Strong problem-solving skills, with the ability to troubleshoot and optimize cloud-based data workflows.
• Ability to work collaboratively in a hybrid setting, communicating effectively with both technical and non-technical teams.

  
Desirable Skills for the Azure Data Engineer in include :

• Familiarity with Azure Data Lake Storage and other Azure analytics services.
• Experience with data security and compliance best practices in cloud migration projects.
• Knowledge of version control systems such as Git.
• Previous experience in the fintech sector is an advantage but not required.

  
Keywords: SQL / Azure Data Factory / Databricks / Python / Data Engineering / CICD / Data Lake / Synapse