Senior Data Software Engineer with Azure Databricks

Responsibilities

• Lead the engineering team: break designs (HDD/TDD) into tasks, distribute work, track delivery and remove blockers
• Turn designs into working pipelines and find the right technical solutions where a design leaves gaps
• Evolve reusable ingestion patterns (API, file, database, streaming) on the metadata-driven EDL framework
• Lead curation across the Medallion layers: append-only Bronze, SCD2 history in Silver, and support for Gold models built with data modelers
• Own the physical table design (liquid clustering vs. partitioning), the Unity Catalog structure and RBAC
• Lead data quality and reconciliation: source-to-lake checks, DQ rules, audit logging and alerting
• Drive performance improvements: find bottlenecks, optimize Spark jobs, Delta tables, SQL queries and Power BI models and reduce compute cost
• Guide the creation of Power BI reports and Databricks dashboards for business reporting and pipeline monitoring
• Review code, enforce engineering standards, and own CI/CD promotion (DEV → TST → PRD)
• Work with the Solution Architect, platform/CloudOps teams and business stakeholders, and prepare runbooks and the handover to operations

Languages