Senior/Lead Data Quality Engineer

Responsibilities

• Build and architect a strong automated testing strategy and framework for data engineering pipelines, emphasizing integration with Databricks, PySpark, Scala, Spark SQL, and other pertinent technologies
• Deploy automated tests that guarantee data quality, integrity, and reliability across every data pipeline and platform
• Embed automated testing into Azure DevOps CI/CD pipelines to enable smooth testing, build, and deployment workflows
• Carry out performance testing to evaluate the scalability and efficiency of data processing systems
• Collaborate closely with data engineers and additional stakeholders to gather requirements and align testing strategies with business goals
• Set up monitoring for automated tests and deliver routine reports covering test coverage, defects, and data quality concerns

Languages