Senior/Lead Data Quality Engineer

Responsibilities

• Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
• Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
• Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
• Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
• Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
• Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues

Languages