ETL Antipattern: Ignoring the “Why?”

In this final post in my ETL Antipatterns series, I’ll talk about a common big-picture mistake when building ETL processes: ignoring the fundamental question, “why are we doing this?”

ETL Antipattern: ignoring the “why?”

When I first began my consulting career more than ten years ago, most of my work was transactional. I’d be assigned to a project and would be given a spec, and I’d build what was asked of me from that list of deliverables. As I created more and more of these solutions, I found that some of them were simply scratching an itch rather than addressing the bigger business concerns.

Later, after I launched my own consulting firm, I discovered that there’s a great deal of value to clients in asking a lot of “Why?” questions prior to jumping into building a solution. Often the technical details of a solution trickle down from a specific business unit, and I’ve found that it’s valuable to step back to get a better sense of how a new solution fits into the big picture of the business.

Taking time to ask these questions can add to the length of a project, but ultimately it may reduce the amount of new technical debt. In some cases, asking “Why?” can actually cut the project timeline significantly. I ran into this recently where a proposed ETL project was cut from months to days because we started first with a business needs analysis, only to learn that the data that was needed was already available as part of an earlier initiative.

While there is a natural bias to start building something immediately, keep in mind that every ETL project should be driven by business needs and not on the technology.

Ask “Why are we doing this?”

Don’t be afraid to ask the big question. While this may take some extra time to answer, the resulting solution will likely be of more value to the business, and will make future maintenance and modifications easier.