The ELT way of transformation
Maximize performance and reduce costs
Understanding the traditional approach
The purpose of traditional data integration tools is to install and use a proprietary transformation engine.
This type of architecture relies on an old perception of information systems, dating from the last century ! At that time, when the source and target technologies were flat files, hierarchical files, or simply not open systems, the use of a proprietary engine was most necessary.
Today, the situation is quite different. The information system consists in very open solutions: relational databases with high reliability and a great capacity to transform and analyze the data, XML files, Standard Web Services (which themselves do transformations) and so on !
But the major fact is that proprietary engines can become a real pain in the neck when you are dealing with large amounts of data, or simply when you haven't got a lot of time to do the work.
It’s a very heavy approach that is not adapted to our actual way of life which is more decentralized and cloud computed !
A case example
Imagine that you need to integrate some data into a Business Intelligence system. You have a target table in which there already are 1 hundred million rows.
Imagine now that you just have ten thousand rows that have been changed in your source systems.
Now you will have to integrate these rows. But before you do that, you will need to know which are the new rows (that don't already exist in your target), which are the already existing rows, and then, which rows really need to be updated (because some may still be identical).
To do this type of incremental integration, you will have to compare data, which will be very inefficient with a proprietary engine. As you can see on the schema below, the engine will need to look up the entire target table to compare the data.
Towards a decentralized and efficient approach
The E-LT Stambia Approach consists in doing the transformations on the underlying systems without adding a proprietary engine.
The transformations are made by the databases or other technologies (OLAP, Operating Systems, etc.) according to their transformation capabilities. Stambia will use all the native components of the underlying technologies (such as the database loaders) to maximize the performance.
To summarize, here are the benefits of this architecture:
- A high level of performance
- Efficient : no intermediate technologies
- Fast : native communication with existing technologies
- Simplified architecture
- Non intrusive : no system to install
- Decentralized : the load is distributed over the systems
- Optimization on the use of existing technologies
- Make profits : you take advantage of the technologies you have already invested in
- Better experience : knowledge and control of the existent technologies are consolidated