New geospatial technology organizes, interprets marine Big Data
By Ariana Hurtado, Editor and Director of Special Reports
HOUSTON — Geospatial technology has proven its worth in the offshore energy industry. For instance, geospatial software and subsurface services support offshore carbon storage licensing. Cloud-based geospatial software also can provide analytics via high-resolution spectral imagery from the planet’s surface using satellites, drones and planes. As another example, geospatial navigation and analysis services can define reservoir architecture by mapping multiple bed boundaries in real time.
In July 2023, TrueOcean released an update to its marine data platform by adding smart geospatial search for hydrographic, geotechnical and sensor data in the cloud. The new geospatial search function enables data on the platform to be located and accessed in seconds using a map-based interface.
In addition to supporting the whole data organization, the technology offers complex dataset visualizations, secure data storage, quality control and risk mitigation, interoperability and standardized interfaces, and smart indexing and spatial search tools.
Germany-based software companies north.io, TrueOcean and NatureConnect merged in September 2023 to form one cloud geodata specialist company, which took the name north.io. The company focuses on the management of geospatial data on land and at sea, offering software development, geodata management and cloud analytics for both onshore and offshore applications.
north.io CEO Jann Wendt said the need for cost efficiency, increasing time pressure and the availability of experts is what keeps offshore-focused business executives up at night.
"Much of this can be addressed through automation and digitization," he continued. "This is exactly why we developed our TrueOcean cloud technology, because data efficiency is key to addressing these challenges. To date, only a very small fraction of the digital opportunities has been exploited; it is simply time for the introduction of new technologies, the use of the cloud, Big Data and artificial intelligence in our field of industry."
In this Q&A, Offshore chatted in detail with Wendt about the company's TrueOcean cloud digital platform, which focuses on underwater geodata management, and its applications for the offshore energy industry.
Offshore: What is the offshore energy sector's need for this type of marine data platform?
WENDT: Offshore energy faces a complex array of data challenges, ranging from data fragmentation, complex and heterogenous data formats as well as data management issues due to the sheersize and complexity. The dispersed nature of offshore energy data across multiple sources, coupled with inconsistencies in data formats and protocols, hinders the ability to gain a unified understanding of the marine environment and assets. Integrating and analyzing the vast and heterogeneous data generated is a complex task.
Traditional data integration tools often lack the scalability and sophistication to handle the volume, complexity and velocity of this data.
Added to this, the offshore industry needs to focus on efficiency to achieve economies of scale and reduce costs, ensuring competitiveness. Standardizing processes across projects enhances consistency and operational reliability. Therefore, integrating new technologies demands a process-oriented approach for continuous improvement and industrywide best practices.
TrueOcean provides a single repository for consolidating and managing all relevant marine
data, ensuring data consistency and ease of access. This centralized approach eliminates the
problem of data fragmentation and facilitates comprehensive data management.
In particular, the ability to search petabytes of data across the world in different projects by simply drawing a search box on the map, even for complex spatial data formats such as multibeam, side scan, sub-bottom profiler, is something that no other software currently offers. Developing and integrating our own cloud-based data reader technology allows handling of complex binary data formats, which usually need to be handled by a multitude of specific desktop applications. The developed technologies are the foundation of the digital transformation of an entire domain.
Offshore: How did the company develop the geospatial technology?
WENDT: TrueOcean's Geospatial Data Hub technology has been developed over the last three years by more than predominantly 65 software and research engineers based on lessons learned from more than 13 years of cloud and geospatial expertise. The application is designed as a highly scalable microservice architecture with more than 70 individual scalable services powered by the powerful orchestration framework Kubernetes.
Two of the most important design decisions were neutrality and security; we designed the application to be completely cloud-agnostic, meaning that we are not tied to any cloud providers. This allows us to deploy our application in public, private and hybrid environments and, with full isolation, immediately address the key security concerns of the enterprises IT departments.
The technology was designed to handle a wide range of data formats and types, originating from different sensor types and survey purposes. This includes data from hydroacoustic measurements, satellites, computer-aided design workflows and geospatial sources, which are common in large offshore infrastructure projects. Therefore, a brand new and cloud-native binary and geospatial data reader technology was developed based on the most modern and programming languages Golang and Rust for highest efficiency and scalability.
In addition, we utilize Big Data ready formats, which are optimized for cloud applications and enable efficient data management and cooperation. The platform, for example, operates internally with Apache Parquet and Apache Arrow, data formats that are column-based and very advantageous for complex and large data handling and processing. Parquet's combination of efficiency, wide compatibility and continuous enhancements make it extremely suitable for cloud data storage and marine Big Data management. By transforming internally heterogeneous marine data formats into standardized and highly efficient columnar approaches, we can apply a wide range of advanced Big Data capabilities at scale.
Another significant technological development is the introduction of geospatial search functionality. The development of new algorithmic approaches to data reduction and spatial data intersection, combined with a high-performance architecture, enables users to quickly locate and access data in petabyte-scale environments with the simplicity of a map-based search.
Offshore: What are the tool's capabilities and offshore applications?
WENDT: Implementing intelligent data management, standardization and collaboration solutions is the building block for a future-ready data value chain in offshore applications. Especially in the offshore industry, where large-scale underwater projects are characterized by intricate engineering challenges, environmental impacts and multi-faceted stakeholder involvement, the role of data management and standardization is of highest importance. Given the sheer volume and variety of data generated, from hydrographic, geological, metocean and other monitoring technology, having a cohesive system for organizing and interpreting this information is paramount.
The TrueOcean technology facilitates the collaborative nature of offshore ventures, which typically includes a mix of contractors, regulatory bodies and local authorities, necessitates a unified data framework to facilitate clear communication and shared decision-making. The cloud native TrueOcean platform automatically geospatially indexes and standardizes every dataset uploaded and therefore allows a highly efficient spatial search of datasets never seen before. Having data organized, structured and transformed to cloud native data formats is the building block for the whole data value chain by using Big Data frameworks and AI analytics. Based on these building blocks, an organization can become part of a Blue Data Economy and create tremendous additional value out of its data.
Offshore: Can you share any additional work planned to put into development and/or additional energy industry applications the technology might have beyond those defined in its scope of work?
WENDT: The upcoming integration of the in-house developed Big Data technology framework Zeus into the TrueOcean marine data platform offers several exciting prospects for future developments and applications in the energy industry.
With Zeus, we will introduce a new paradigm for working with marine data: the point cloud way. While today’s approaches are converting (e.g., multibeam) data for quality control and analytical approaches immediately into gridded raster data, we are harnessing the abilities of cloud technology and algorithmic optimizations to perform processing on the highest resolution possible, the point cloud.
Behind the doors, Zeus's uses highly adapted Big Data technologies like Apache Spark/Sedona and Apache Parquet for data storing and processing. This allows us to not only work on the highest possible resolution and receive incredible detailed results but also to combine single file-based measurements and treat them as one large dataset of an entire area in the tera- or even petabyte range.
Data quality such as data point spacing, horizontal and vertical resolution, motion and many more can be assessed on an individual ping base in hours, rather than weeks or months. And the ability to monitor infrastructure by performing point cloud-based change detection over large areas is simply breathtaking.
The application of this kind of advanced analytics and AI will enable sophisticated data analysis that will massively speed up processes, reduce costs and provide the highest levels of confidence at unprecedented levels. And we are already working with our technology partner Nvidia to take it to the next level by integrating the massive parallelization power of GPUs [graphic processing units] into our processing technology.
We are confident that the scalability of Zeus and the paradigm shift from working on a single
ping or point cloud basis will revolutionize the way the entire industry works with large
amounts of underwater data and will have a huge impact on the energy industry and the way
data analysis is carried out in the field.
02.05.2024
