Introduction
Navigating the Unknown and Overcoming Migration Hurdles
The problem
Setting up Collibra Edge and Designing an Onboarding Process
Setting up Collibra Edge and creating an efficient onboarding process for new systems posed a significant challenge for one of our clients in the retail sector. The client's existing environment relied on Jobserver, requiring manual upgrades and connections per schema to ingest the structural metadata. However, with the need to classify data for regulatory purposes like GDPR, migrating to Collibra Edge became imperative. Edge was the replacement for Jobserver, relying on modern technologies such as Kubernetes to function and ensure a more secure setup for ingesting metadata. Doing so, we could set a solid foundation for effective data governance, unlocking the true potential of data management, paving the way for better decision-making and regulatory compliance.
How we solved it
Navigating the Unknown and Overcoming Migration Hurdles
We embarked on the journey of setting up Collibra Edge. The task involved configuring an on-premise Edge server, connecting it to both on-premise and SaaS data sources. With determination, we tackled networking complexities and successfully established a healthy Edge Site.
During the migration process of already existing metadata, we faced challenges due to the lack of a streamlined workflow for migrating Jobserver ingested schemas to Collibra Edge. However, through meticulous analysis of existing connections and collaboration with stakeholders, we devised an alternative approach.
There were 4 steps in this alternative approach:
- Extracting all the existing information from a Jobserver connection.
- Analysing this information for any links not related to automated catalog ingestions.
- Where applicable, we needed to update the full name of those existing assets to comply with Edge asset naming convention.
- Linking the schema with its linked tables and columns to the new Edge database asset.
By following the above steps Edge would accept the existing assets as if they were ingested through the normal flow.
Of course, we also had a lot of new system that needed to be onboarded. About 60+ systems were onboarded by prioritizing the systems based on importance and difficulty. Doing so we could efficiently divide the onboarding process into manageable waves. To facilitate communication with system owners, we provided them with a comprehensive step-by-step guide containing all the necessary information. This approach significantly reduced troubleshooting time and ensured smooth onboarding of multiple systems.
Navigating the Unknown and Overcoming Migration Hurdles
We embarked on the journey of setting up Collibra Edge. The task involved configuring an on-premise Edge server, connecting it to both on-premise and SaaS data sources. With determination, we tackled networking complexities and successfully established a healthy Edge Site.
During the migration process of already existing metadata, we faced challenges due to the lack of a streamlined workflow for migrating Jobserver ingested schemas to Collibra Edge. However, through meticulous analysis of existing connections and collaboration with stakeholders, we devised an alternative approach.
There were 4 steps in this alternative approach:
- Extracting all the existing information from a Jobserver connection.
- Analysing this information for any links not related to automated catalog ingestions.
- Where applicable, we needed to update the full name of those existing assets to comply with Edge asset naming convention.
- Linking the schema with its linked tables and columns to the new Edge database asset.
By following the above steps Edge would accept the existing assets as if they were ingested through the normal flow.
Of course, we also had a lot of new system that needed to be onboarded. About 60+ systems were onboarded by prioritizing the systems based on importance and difficulty. Doing so we could efficiently divide the onboarding process into manageable waves. To facilitate communication with system owners, we provided them with a comprehensive step-by-step guide containing all the necessary information. This approach significantly reduced troubleshooting time and ensured smooth onboarding of multiple systems.
The results
Streamlining Onboarding and Unlocking Data Governance Potential
Our efforts led to streamlined onboarding of source systems onto the Collibra catalog, resulting in having the metadata to field level in our catalog. Meaning, we could now start with classifying data source fields to indicate their sensitivity level. For example, Customer E-mail would be classified as Confidential, Company Address as Public. This classification was crucial for other teams that required access to the data and for ensuring compliance with regulations. Additionally, we could link business information to physical data, enabling a holistic view of data governance and strengthening the overall framework.
In conclusion, our successful implementation of Collibra Edge and the streamlined onboarding process had a profound impact. Together with the client we overcame challenges by navigating the unknown, finding alternative migration solutions, and ensuring efficient communication. As a result, we established a solid foundation for effective data governance, enabling the client to classify and connect data seamlessly.
Key Learnings
More case studies
We deliver impact where it matters most.
Unlocking Viewer Insights with a Scalable Data Platform
Streamz is the Flemish video-on-demand platform offering local and international series and films through a subscription-based model. Streamz focuses on delivering high-quality content tailored to local audiences, supported by a strategic partnership with Paramount.
