Question from someone in one my recent classes: “What tool do you use to send email from ETL processes?” My response: “I don’t.” The tl;dr version of this post is I let my extract-transform-load processes do just ETL, and leave notifications to the scheduling system where they belong. Get Your Email Out of my ETL Before you read any further,…
I’ve had a lot of luck in my career, especially as it pertains to my work environment. Although I can’t say every technical job I’ve had was all unicorns and bacon, I’ve been very fortunate to have been treated fairly and professionally through most of my employment history. Not only was I treated respectfully, but the last three employers I…
It seems like it was just yesterday that we all said goodbye in Seattle at last year’s PASS Summit, but the calendar on the wall says that the next event is a mere four months away. This means that it’s that time of year again: the schedule of sessions has been published. I am happy to announce that I’ll be presenting…
It happens far too often: Once an ETL process has been tested and executes successfully, there are no further checks to ensure that the operation actually did what it was supposed to do. Sometimes it takes a day, other times it takes a year, but eventually that call comes from a client, coworker, or boss: “What’s wrong with this data?”…
If you were to poll data professionals on which tasks they enjoy working on the most, ETL logging would probably not make the list. However, it is essential to the success of any ETL architecture to establish an appropriate logging strategy. I like to compare a good logging infrastructure to the plumbing of a house: it is not outwardly visible,…
Extract, Transform, and Load (ETL) processes are the centerpieces in every organization’s data management strategy. Each step the in the ETL process – getting data from various sources, reshaping it, applying business rules, loading to the appropriate destinations, and validating the results – is an essential cog in the machinery of keeping the right data flowing. Establishing a set of ETL…
Save money and train your entire team with a private class delivered onsite at your office. Data architect and trainer Tim Mitchell has a full catalog of Microsoft business intelligence courses available as private, customized classes.
Earlier this week I wrote about the basics of change tracking in SQL Server, and showed how to get started using this technology for change detection. In this post, I’ll continue what I started by demonstrating how change tracking fits into a larger design pattern for end-to-end incremental load ETL processes. Incremental Load Overview ETL processes fall into one of…
I’m a fixer. A tinkerer. As a kid, I would take my toys apart just to see how they worked, and to prove that I could put them back together again (and these experiments often had mixed results). Even today, I’m still kind of a shade tree handyman. In fact, earlier this week I had to fix a broken safety…
Through the course of my 8-someodd years of building and fixing ETL processes, I’ve had the opportunity to see a lot of ETL code. Some of that code was really good, well-thought-out and carefully executed. Other load processes were – well, let’s just say that they provide plenty of consulting opportunities (and I include much of my early code in…