Data Engineer - Tech Lead (Databricks, Pyspark)

Responsibilities

• Lead the architecture, design and build of large-scale data platforms using Azure Databricks and modern cloud technologies
• Implement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principles
• Enhance performance, manage cloud costs and ensure platform reliability for structured streaming workloads
• Define data governance, security and quality standards to maintain consistency across the platform
• Collaborate with stakeholders to translate complex business requirements into actionable technical solutions
• Develop integration approaches using Azure-native services such as Data Factory, Synapse and Blob Storage
• Mentor data engineers, promote modern engineering practices and perform technical reviews
• Drive adoption of CI/CD, Infrastructure as Code and automated testing in data engineering environments
• Implement observability and monitoring using tools like Databricks Workflows and related frameworks
• Contribute to AI-driven initiatives by leveraging Databricks ML/MosaicML to integrate Generative AI and LLM-based solutions

Languages