Digitization, Big Data Analysis & IoT with Transbase® Edge
Big Data and the Internet of Things (IoT) are placing new demands on database management systems (DBMS), we fulfill as the manufacturer of Transbase® Edge.
IoT data triggers actions on specific events in real time. If, for example, the temperature of a machine is too high, an action (switch on cooling) must be triggered in the shortest possible time after the occurrence of the event (threshold exceeded temperature).
IoT sensors permanently generate data in very large quantities. The loss of data within the measurement interval usually does not lead to significant decision changes (the cooling is switched on because of exceeding the threshold value, even if only 50 instead of 60 measured values are processed), in contrast to decisions in the classic IT environment (one share should be bought at 1.50 € and sold at € 1.51 again).
A DBMS in the IoT environment must therefore be highly scalable, very smart, slim and secure (4S Technology). Transbase® is completely optimized in its architecture and conception to fulfill these criteria.
Thanks to its new crowd functionality, Transbase® Edge can be distributed. Results can be aggregated and consolidated across groups of edge nodes. The data can also be forwarded to analysis tools for more complex analysis. Transbase® Edge databases can also be operated as a database cluster or grids.
IoT & Transbase® Crowd
Transbase® Edge uses the new crowd function (as of v. 8.3) to control parallel database queries via any number of connected edge databases from a central instance.
While the original data is still on the edge device, the query results can be collected and aggregated by the central instance. This makes Transbase® Edge ideal as an IoT database.
For example, if a company maintains 1 million devices, each producing 1 kB of data per second, then 109 bps or 1 Gbps is required to transfer and store that data on the server. In contrast, a Transbase® Crowd query of all registered Transbase® Edge databases can be performed in seconds and their results collected.
The Transbase® Crowd feature also allows data to be partitioned and distributed horizontally across a pool of databases with the same database schema. Vertical partitioning of data can be accomplished by integrating the primary key into a view.
Transbase® databases can be easily integrated into a transbase® crowd hierarchy - a loosely coupled hierarchy of databases with the same or similar database schema (see figure).
Each edge database has a "parent database" to which it can be registered. The parent database looks after registered edge databases and can send queries to registered edge databases and collect their query results. This mechanism allows the parent database to process queries across all registered edge databases.
Transbase® Crowd takes care of the registered Transbase® Edge databases and can send queries to these registered Transbase® Edge databases (edge nodes) and collect the query results. This mechanism allows Transbase® Crowd to process queries on all registered Transbase® Edge databases.
The hierarchy has a root database that can be queried if all databases in the hierarchy are to participate in the query.
Each intermediate node can be queried directly if only the following databases are to be part of the query. This functionality is particularly interesting for IoT environments because it only uses data that has been generated and stored in the corresponding IoT devices. They can be easily stored, processed and maintained there.
Important: Data transfer to a central server database is always a challenge. Questions need to be clarified like:
- Is the bandwidth large enough to transfer the data?
- Is the server database fast enough to receive the data?
- Are there any privacy restrictions that could prevent such a transfer?
- Reaches the scalability of the overall system to support the growth of devices?
Note when using Transbase® Crowd:
- Databases can register and deregister at any time.
- Database connections can fail at any time, resulting in incomplete or no results.
Transbase® Crowd can specify different timeouts to either get more accurate results from more participating databases, or to get faster results from less-participating databases.
IoT & Transbase® Flash Storage Support
IoT devices typically use nonvolatile storage (flash storage) based on electrically programmable high-speed memory (see Solid State Drive = SSD). Data can be written at high speed and I / O operations can be performed very quickly. Flash memory does not require power to maintain the integrity of stored data. The data is stored in memory cells, whereby the cells with previously written data must first be deleted before new data can be written.
Flash storage is very fast, but frequent writing can significantly reduce its life. Transbase® Edge is optimized for flash storage. It detects potential memory errors for persisted data and can minimize the number of writes.
Transbase® Edge controls this through the flexible configuration of its main memory cache. Depending on the application, the cache is configured as large as necessary, minimizing physical I / O. The application gets faster and writes less often. The Transbase® cache is loaded and, depending on the size of the cache, written back to the flash storage at the latest when the database is shut down.
IoT & Transbase® Log
To ensure the stability of database changes, the Transbase® Log is permanently written to disk. This can be flexibly influenced by the IoT application.
On the one hand, the size of the log block can be configured to be larger than the size of the database block, which affects both performance and reliability. In addition, the log is written asynchronously whenever a log block is completed. As a result, there is always only one log block in memory and not on hard disk, which limits the number of lost updates in the event of a power failure.
The time and frequency of how often a log block is written can be configured.
The use of Transbase® Edge in the embedded environment offers numerous advantages in terms of efficiency, robustness, performance, reusability, modularization, and security. The integrated transaction concept allows software products to be developed in a completely different quality level.
Despite full and complete functionality, the complete footprint (code and system data) is less than 2 MBytes if configured accordingly. Because all Transbase® interfaces remain identical, classic Transbase®-based PC applications can easily be ported to embedded platforms without modification.
Applications and data can be used unchanged on these platforms.
Do you have questions about Transbase® Edge? We are happy to show you our system live. Schedule an appointment.