Overall Engineering Philosophy

The goal is to build high performing easily maintained processes with six nines of reliability.

What does "high performance" mean? It is not just about speed of processing. The framework gives organizations the ability to rapidly build and maintain sophisticated OLAP solutions that just happen to process data quickly.

The enterprise data warehouse is not something you can cut corners on. It HAS to be right and it has to be highly available, otherwise it will not be used. People will go back to their department specific reports built on transactional systems which slows the transactional system down which in turn makes users unhappy.

The data warehouse is not a place where you can say stuff like, “we can’t let the perfect be the enemy of the good”. In this specific scenario, the “good” is the enemy of a properly functioning system. You build data warehouse ETL to tight tolerances to ensure that everybody maintains faith that the system is popping out reliable, accurate, actionable data in a timely fashion. You get data to the right people, at the right place, at the right time.

Business analytics demands perfectionism and do not let anybody tell you otherwise. You may never create the perfect system, but that is not the point. As long as you strive for perfection and never back down from people trying to water down your processes, you will be able to build a system arbitrarily close enough to perfection that it is indistinguishable from an actual perfect system.

The final element of the overall engineering philosophy is the ethos of the silent professional. Data warehouse ELT should hum along in the background like a utility that is fundamental to our daily lives but rarely noticed because of its ubiquity and reliability. If you do things right, people will not be certain you did anything at all.