The digital data volume annually generated by Big Data and the industrial Internet of Things (IIoT) will be over 163 ZB by 2025 - that is 447 EB daily, 310 PB per minute or 5,200 TB per second that have to be processed.
In view of this enormous volume of data, it makes only limited sense and it is often even impossible to transport all of the data to be processed to the cloud, where it can be analysed, processed and then returned back to the application.
It is much more effective to evaluate the required data as close as possible to their place of origin (edge) and to process them there (edge processing). Many IoT applications rely on real-time data analysis and therefore have to fal back on edge processing.
Powerful services for processing mass data on IoT edge devices are becoming essential. An intelligent IoT database system such as Transbase® Edge, which cleverly analyses and aggregates data, makes IoT applications faster, more convenient, cheaper and easier to use.
With Transbase® Edge we have created a database technology that can withstand large amounts of data and provides the results on the IoT edge device itself - where a quick decision is necessary. This reduces the communication bandwidth between the IoT sensors and a central computing instance. The analysis and generation of knowledge takes place directly at the source of the determined data. With Transbase® Edge, we combine decentralized measurement and central decision-making in one system of networked IoT edge devices.
The industrial Internet of Things (IIoT) places new demands on the retrieval and analysis of data. IoT sensors permanently generate very large amounts of data. In contrast to decisions in the classic IT environment, the loss of data within a certain measurement interval does not normally lead to significant changes in decision-making (cooling is switched on because a sensor exceeds the threshold value once or 50 times per interval).
IIoT data triggers actions for specific events in real time. The actions carried out are the result of certain sensor values or constellations of sensor values. It is important to be able to rely on the correctness of the input data in order not to impair or falsify the entire process.
To ensure this, we add a 'gatekeeper' in our IIoT reference architecture, consisting of a blockchain application combined with Transbase® Edge as an intelligent data retrieval machine, between the sensor world and the process world. Only correct, subsequently unchangeable sensor data are forwarded to the subsequent systems via this Layer of Trust.
The following IIoT reference architecture shows Transbase® Edge in interaction with analysis and process tools from sensors to cloud applications. In this way, IIoT processes can be designed safely and efficiently.
First Layer of Trust
The values of the device sensors are intercepted by the blockchain application directly at the source and forwarded to Transbase® Edge as a secure, traceable and unalterable 'blockchain'. It thus ensures the correctness of the sensor data.
Using smart contracts, specific transaction chains can already be created in this phase for certain result constellations or alerts as input for further processing and also stored in Transbase® Edge.
Edge / On-Premise Processing
Transbase® Edge has a pull / push mechanism (crowd) to control parallel queries and local analyses via any number of connected Edge Devices from a central instance.
Using the pull function, aggregated data can be called up and analysed locally (real-time analysis). Via the push function, this data can then be made available On-Premise or in the Cloud for more complex AI / ML analyses, statistics or storage.
Second Layer of Trust
A second Layer of Trust makes it possible to provide analysis results from the first stage (Edge Processing of sensor data) as a secure, traceable and unalterable 'blockchain' for the next stage (processing On-Premise or in the Cloud).
While the original data is still on the Edge Devices, the query results are collected, aggregated, analysed and further processed in the following stages. Transbase® Edge can be used to ensure the correctness and traceability of the data within each process stage.
On-Premise / Cloud Processing
Using the Transbase® Edge crowd function, data can be partitioned and distributed both horizontally and vertically over a pool of databases (with an identical database scheme). This means that pre-qualified data packets can be forwarded to AI / ML systems, DHW etc. for targeted processing or stored at a central storage space.
- the bandwidth is not big enough to transfer the data
- the server database is not fast enough to receive the data
- there are privacy restrictions that could prevent such a transfer
- the scalability of the overall system is not sufficient to support the growth in the number of devices
Edge Nodes and Parents can register or log out at any time, either on purpose or due to a failed database connection. The query results are therefore incomplete and can be repeated once the connection has been re-established.
Transbase® Edge enables the control of parallel database queries via any number of connected Transbase® Edge databases, starting from a central instance, via the new crowd function (from V. 8.3). While the original data is still on the edge devices, the query results can be collected, aggregated, analysed and further processed by the central instance. Databases can be operated as database clusters or grids. Query results can be aggregated and consolidated across groups of Edge Nodes or forwarded to appropriate analysis tools for more complex evaluations.
Transbase® databases can be easily integrated into a Transbase® crowd hierarchy - a loosely coupled hierarchy of databases with the same or similar database schema (see figure).
For example, if a company has 1 million edge devices that each produce 1 kB of data per second, then 1 GB per second is required to transfer and store this data on the server or in the cloud. In contrast, a Transbase® Edge Crowd query of all registered Transbase® Edge databases can be carried out in seconds and the results collected - independently of the data volume.
Every Transbase® Edge database as an 'Edge Node' has a 'higher-level database' ('Parent'), with which it can be registered. Every Transbase® Edge installation (regardless of whether as an Edge Node or Parent) has exactly the same feature set. A change within the hierarchy or constellation can therefore be carried out very easily and without programming effort.
Transbase® Edge takes care of the registered Edge Nodes as a 'Parent' (intermediate node) and can send queries to these registered Edge Nodes and collect the query results. This mechanism enables Transbase® Edge to process queries about all registered Edge Nodes.
Each Parent can be queried directly if only the following Edge Nodes are to be part of the query. This functionality is particularly interesting for IoT environments because it only uses data that has been generated and stored in the corresponding IoT devices. They can be easily stored, analysed and processed there.
Every hierarchy has a Transbase® Edge database as 'Root', which can be queried if all Edge Nodes and Parents in the hierarchy are to participate in the query. Those Edge Nodes and Parents that are not involved in the query (e.g. because they are offline) are simply not taken into account.
Different timeouts can be specified when designing the query to either get more accurate results sent from more participating Edge Nodes or to get faster results from less participating Edge Nodes.
It is also possible to specify a time interval within which an Edge Node or Parent can (re-)register. The query is then carried out with a time delay after this time interval has elapsed, e.g. include as many Edge Nodes as possible in a query result.
Flash Storage Support
IoT devices typically use nonvolatile storage (flash storage) based on electrically programmable high-speed memory (see Solid State Drive = SSD). Data can be written at high speed and I/O operations can be performed very quickly. Flash storage is very fast, but frequent writing can significantly reduce its life.
Transbase® Edge is optimized for flash storage. It detects potential memory errors for persisted data and can minimize the number of writes. Transbase® Edge controls this through the flexible configuration of its main memory cache. Depending on the application, the cache is configured as large as necessary, minimizing physical I/O. The application gets faster and writes less often. The Transbase® cache is loaded and, depending on the size of the cache, written back to the flash storage at the latest when the database is shut down.
To ensure the stability of database changes, the Transbase® Log is permanently written to disk. This can be flexibly influenced by the IoT application. The size of the log block can be configured to be larger than the size of the database block, which affects both performance and reliability.
The log is written asynchronously whenever a log block is completed. As a result, there is always only one log block in memory and not on hard disk, which limits the number of lost updates in the event of a power failure. The time and frequency of how often a log block is written can be configured.
The use of Transbase® Edge as embedded database offers numerous advantages in terms of efficiency, robustness, performance, reusability, modularization and security. The integrated transaction concept allows software products to be developed in a completely different quality level.
Despite full and complete functionality, the complete footprint of Transbase® Edge (code and system data) is less than 2 MB if configured accordingly. Because all Transbase® interfaces remain identical, classic Transbase®-based PC applications can easily be ported to embedded platforms without modification. On these platforms, applications and data can be used unchanged.