Thursday, February 11, 2010

ELT - Extract, Load and Transform. When to use?

Heard so much about it around 2005-06, now the buzz word has faded on "ELT". So what exactly, is an ELT? Why can't I traditionally use an ETL? Well, ELT pretty much can do the same transformations as an ETL. It is for you to decide why, when, where and how to do the transformations.

Mostly, if I/O is a concern then ELT is your answer. ELT does not create any memory/process/swap overheads on the hosting server since all calculations are done on the database. Talend's offerings for ELT based processing is a must watch. Also, ELT is mostly relied for 3rd normal form to 3rd normal form data loading. If you have an enterprise datawarehouse that has batch/real time loading then ELT is not your choice, the answer is always a YES to use ETL for datawarehouse loading.

Talend offers open source ETL and ELT. Interesting features around Talend are you could use both ELT and ETL objects in the mapping you are designing. Also, Talend lets you choose between Java and PERL source code type, just at a click before creating a mapping.

A typical ELT mapping/job contains a source query object(E), then creation of a table, followed by loading (L), and finally transforming(T) such as aggregating, then loading into a target table.

The old school method of doing an ELT was, when a stored proc. created temp tables, then getting them loaded and then performing some calculations before loading into target tables. InfoPump, Microsoft DTS was another old school method of doing an ELT. 

With lowering hardware costs, ELT seems like an almost obsolete concept now.

Disclaimer: The above are my personal opinions based on my observation and do not represent my employer's view in any way.

No comments:

Post a Comment