The Ocean Cleanup designs and develops advanced technologies to rid the oceans of plastic. It does this by cleaning up pollution already present in the oceans, and intercepting plastic on its way to the ocean via rivers. Their ocean technology is moved by the ocean forces to passively catch and retain plastic. The Interceptor™, their solution for river debris, is a 100% solar-powered device which autonomously halts and extracts plastic from rivers before it reaches the ocean.
The Ocean Cleanup believes that to develop the most optimal cleanup technologies, it’s essential to truly understand the problem. That is why they have been conducting extensive research using data ranging from low-tech sources (visual counting from bridges) to high-tech sources (automated camera monitoring). The ingestion of these various sources located all over the world is no easy task for the data engineering team. Over time, this has led to some internal challenges: fragmentation in the IT landscape, unclear data lineage and scalability issues.
The Ocean Cleanup needed a data ingestion framework which could scale together with the ever increasing list of sources and which standardized the way data was handled by the rest of the organization.
Together with The Ocean Cleanup’s data engineering team, we discussed the current architecture and saw how the new ingestion framework would fit in. During these brainstorming sessions, we set out to design a system that would not only work for the current processes, but also for future use cases. This resulted in the following design goals:
With these design goals in mind, Xomnia's data engineers started working on the ingestion framework. Leveraging on our expertise and experience, we moved from an initial prototype to a productionized system in just three months. During this period, Xomnia closely collaborated with The Ocean Cleanup to ensure a seamless integration into the existing Azure Cloud Environment.
The framework provides an API to which data providers can push their payloads and metadata. These are ingested and stored in a raw form before being posted on a messaging bus. This all happens independent of the data source and does not require updating the framework when new sources are added. Once the data is on the messaging bus applications (e.g. image recognition algorithms, business logic, cleansing & parsing) apply further processing in a streaming manner.
The framework uses the following services: Azure Kubernetes Service (AKS), Azure Service Bus and Azure Blob Storage.
The framework is already live in the production environment and in use by some data providers. It enables The Ocean Cleanup to keep focusing on the things that matter the most, designing and developing advanced cleanup technology. The ingestion framework has gone fully operational already.
Xomnia has helped overcome the various challenges posed by the wide variety of data sources that The Ocean Cleanup uses. It has built a strong foundation which is flexible by nature, which is a framework that moves with the dynamic environment requiring little to no changes when new data sources are added. It can scale with the volume of data coming in and allows for processing data in a streaming manner.
