Terrabyte: Germany’s New High Performance Data Analytics Platform for Earth Observation

(evgeniy jamart/Shutterstock)

Satellites and the data they generate enable scientists to monitor changes in the Earth’s environment such as deforestation, desertification, melting ice caps, and other forms of environmental degradation. They can also be used to track natural disasters like wildfires, droughts, hurricanes, and floods.

A new high performance data analytics platform called terrabyte has been announced that will enable researchers in Bavaria and Germany to use satellite and earth observation data for classical simulations and evaluation using artificial intelligence methods.

The new system is jointly run by the German Aerospace Center (DLR), a national research center for aeronautics and space, and the Leibniz Supercomputing Center (LRZ), one of the top three supercomputing centers in Germany.

The terrabyte platform is a heterogeneous system that combines multiple CPUs, accelerators, and GPUs. Its core consists of Lenovo ThinkSystem SD650-N V2 servers and DSS-G memory. The platform contains 61 CPU nodes, each with two 40-core Intel Xeon Platinum processors, as well as 15 GPU nodes, each featuring four Nvidia A100s. The terrabyte platform also features close to 50 petabytes of additional online storage. Data processing speeds can reach 320GB/s with terrabyte’s InfiniBand HDR. Data is transferred from the DLR’s German Satellite Data Archive in Oberpfaffenhofen to the terrabyte platform at the LRZ in Garching via a direct network connection running at 100GB/s.

A release from the LRZ states the terrabyte platform’s equipment allows it to achieve a performance of up to 1.3 petaflops, or 1.3 quadrillion floating point operations per second. The supercomputing center expects this performance to double by the fall of this year as it works to expand the system.

The new terrabyte system, housed at the Leibniz Supercomputing Center near Munich. (Source: LRZ)

The DLR hosts a trove of historical data from 50 years of Earth observation along with current radar and multispectral images from satellites sent back to Earth each day. A release from the LRZ says this data is the basis for research projects like the DLR’s “World Settlement Footprint,” which has tracked the development of settlements globally since 1985. The project provides support for urban and transportation planners and has been especially active in megacities such as Cairo, Mumbai, and Rio de Janeiro. The datasets of all calculations since 2015 have become industry standard and are used by the United Nations, the World Bank, and many other institutions, the LRZ statement said.

Data from satellites are also being used to evaluate locations for offshore wind farms, the LRZ says, as well as to analyze their operations and assess their contributions to the national power supply.

The DLR has invested approximately €19 million in German federal funds from the Bavarian State Ministry of Science and the Arts and the Federal Ministry of Economics and Climate Action for building and expanding terrabyte. The LRZ maintains operations of the platform, handles ongoing operating costs, and provides staff.

“Terrabyte is a big step forward for geoinformation research. Big Data, the latest AI-based processing methods and high-performance computing are combined in a unique way. Terrabyte not only forms the foundation for our science but is also a significant economic driver for Bavaria and Germany. The basis for this is the cooperation with our partners and the associated knowledge transfer,” said Anke Kaysser-Pyzalla, chair of the DLR executive board.

This map shows data related to Hong Kong from the World Settlement Footprint, a project led by the German Aerospace Center. (Source: LRZ)

Terrabyte users can use their own algorithms to process geoinformation data or train machine learning applications on the platform, or they can use a range of available open-source software, tools, and algorithms developed by the Earth Observation Center (EOC) and other DLR institutes.

Previously, compute and scale restraints relegated certain geoscientific analysis tasks to commercial platforms, and these analyses are now possible with the terrabyte platform.

“With the Leibniz Supercomputing Center, we have found the ideal cooperation partner for this challenging project. With terrabyte, we have an autonomous and secure platform with comprehensive global datasets from Earth observation that we can access together with our partners. At the same time, terrabyte forms a technological bridge between our Oberpfaffenhofen and Garching sites,” said Stefan Dech, director of the German Remote Sensing Data Center at DLR.

Dieter Kranzlmüller, chairman of the board of directors at the LRZ, said in a statement that terrabyte meets the diverse requirements for efficient data storage, data science, AI and cloud services, and even HPC and supercomputing.

“With our many years of know-how in these areas, we combine all these technologies into a customized service for Earth observation. It is not only DLR researchers who benefit from the huge amount of historical and current satellite data via terrabyte, but also scientists in Bavaria and Germany. A very exciting, but also challenging project which was made possible thanks to the intensive and excellent cooperation between the colleagues from DLR and the LRZ,” he said.

Related Items:

Drones Put the AI into Aerial Intelligence

European Space Agency Uses AI, Satellites to Monitor Pandemic Air Traffic

NOAA Updates Its Massive Climate Dataset