Consultant for the Development of the Campaign Information Management System (CI

Responsibilities

• Databricks Assessment and Pilot
• Assess Databricks against PALU's current and expected data warehouse needs and against the requirements of active projects, including comparison with the PostgreSQL environment MCD already runs on AWS.
• Set up and configure a small data model in Databricks, using a representative subset of PALU data, to estimate compute, storage and licensing costs under realistic use.
• Configure connectors to the priority external data sources agreed with PALU, to test how a data warehouse would be populated and kept current.
• Prepare a cost benefit analysis for MCD PALU covering direct costs, staff time, skills required, and the risks of adopting and of not adopting the platform.
• Draft plans for future projects, addressing complex data interoperability and data governance and localization requirements, including where data must be stored and who controls access.
• Document all configuration, test results and findings.
• GitLab CI/CD for PALU and the New CIMS
• Configure pipelines, pipeline schedules and related files to automate PALU processes, including automated reports, dbt model builds, data extraction from external sources, and other automations agreed with PALU.
• Configure a CIMS repository to handle CI/CD for the new CIMS codebase, including automated builds, version control conventions, release tagging and test stages.

Languages