Azure or AWS Data Engineer

Job Description

We are looking for a highly skilled Lead Data Engineer with expertise in Azure or AWS and Databricks to join our team. The ideal candidate will lead the design, development, and implementation of data engineering solutions, ensuring scalability, security, and efficiency in our data infrastructure. This role requires strong technical skills, and experience in managing large-scale data processing pipelines.

Key Responsibilities

• Lead the design and development of scalable and reliable data pipelines using Azure Data Services or AWS Data Services and Databricks.
• Architect, implement, and optimize ETL/ELT processes to process large volumes of structured and unstructured data.
• Develop and maintain data models, data lakes, and data warehouses to support analytics and business intelligence needs.
• Collaborate with data scientists, analysts, and business stakeholders to ensure data availability and integrity.
• Implement and enforce data governance, security, and compliance best practices.
• Optimize and monitor performance of data processing frameworks (Spark, Databricks, etc.).
• Automate and orchestrate data workflows using tools such as Apache Airflow, Azure Data Factory, AWS Step Functions, or Glue.
• Guide and mentor junior data engineers in best practices and modern data engineering techniques.

Mandatory Qualifications

• 5+ years of experience in data engineering
• Strong expertise in Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory) or AWS Data Services (S3, Redshift, Glue, Lambda, Step Functions, EMR).
• Proficiency in Databricks and experience with Apache Spark for large-scale data processing.
• Strong programming skills in Python.
• Experience working with SQL and NoSQL databases such as PostgreSQL, MySQL, DynamoDB, or Cosmos DB.
• Solid understanding of data governance, security, and compliance (GDPR, HIPAA, etc.) is a plus.
• Experience with real-time streaming technologies such as Kafka, Kinesis, or Event Hubs is a plus.
• Excellent problem-solving skills and the ability to work in a fast-paced, agile environment.

Good to have Qualifications

• Experience with machine learning pipelines and MLOps.
• Familiarity with data visualization and BI tools like Power BI, Tableau, or Looker.
• Strong communication and leadership skills to drive best practices across the team.