All vacancies
Data Platform Engineer (102-08SENG-01)
OpsBrasil Serviços Cloud LTDA
Remote · worldwide, UTC+4 acceptedSalary not disclosedcontractVerified 4 days agoHimalayas
This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one: 2,094 ADF pipelines to migrate — built as reusable templates rather than one-by-one — 9,859 pipeline activities to translate (some map directly, others need rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring.
Responsibilities
- Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time.
- Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.
- Rewrite ADF Web Activities as Lambda functions or Step Functions tasks, and replace ADF-specific scaling with native Databricks mechanisms.
- Build and tune PySpark transformations for production data volumes.
- Replace Azure Synapse Serverless with Databricks SQL Warehouse.
- Reconcile migrated data against source systems as part of the definition of done.
- Production experience with Databricks: workspaces, jobs and workflows. The central skill for this role.
- Strong Spark and PySpark experience for real data volumes, including tuning.
- Production-grade Python.
- Experience building or migrating Azure Data Factory pipelines, with a solid understanding of the ADF activity model.
Languages
- Work format
- Remote
- Seniority
- Senior
- Posted
- 22 Aug 2026 (3w ago)
- Last verified
- 13 Sept 2026
- Apply by
- 21 Oct 2026