Most traditional ETL processes perform their loads using three distinct and serial processes: extraction, followed by transformation, and finally a load to the destination. However, for some large or complex loads, using ETL staging tables can make for better performance and less complexity. As part of my continuing series on ETL Best Practices, in this post I will some advice…
Among the new features of SSIS 2016, one of my favorite is incremental package deployment. This new functionality allows the deployment of a single package to an SSIS catalog without having to deploy the entire project. Since the release of this version, I’ve been asked the following question a few times: “How does one go about deleting a package from…
This summer, I will be delivering my popular full-day training class Building Better SSIS Packages in three different cities across the southern United States: Thursday, June 1st: Pensacola, Florida Friday, June 16th: Pasadena, Texas (just outside of Houston) Friday, July 14th: Lawrenceville, Georgia (near Atlanta) This course is designed for those with some previous exposure to or experience with SQL…
“We have all of that information. It’s in a database in my office.” This phrase was music to my ears. I was working on one of my first-ever data reporting projects, and I had been searching in vain for a way to access historical point-in-time data from one of our apps in these new reports. Even though this data set…
Making the most of the SSIS catalog requires an understanding of how to access the information stored in the logging tables. Although there are built-in reports to show this information, there are limitations in their use. Fortunately, the logging tables in the SSIS catalog database are (mostly) straightforward and easy to understand once you’ve worked with them a bit. In…
When designing a data warehouse or reporting architecture, the smallest questions can pose some of the biggest challenges. Identifying what may appear to be an obvious term or metric can slow down or even derail a business intelligence initiative. Early in my career as a BI professional, I was faced with just such a scenario when trying to address a…
I’ve spent a lot of time browsing and participating in online technical forums. In my early years in the industry I asked a lot of questions, and as my skills grew I evolved to the point where I was answering more questions than I was asking. The questions and answers I encountered were quite varied, but there was one specific…
Those who follow my blog know that I write a lot about data quality. Measuring and improving the quality of data is an important part of any data initiative, especially in the data warehousing space. While data quality does get its share of attention, there is a concept that is equally important but is sadly overlooked during most data projects:…
In my introduction to the topic of using SSIS in the cloud, I noted that SSIS is not currently available in a platform as a service (PaaS). Unlike SQL server databases and more recently SSAS, which are both available as both on-prem and PaaS services, Integration Services must be run on a user-installed instance of SQL Server. This limitation means…
J.C. Penney is one of the oldest and most storied retailers still in existence in the United States. For years, J.C. Penney used a pricing strategy based on setting their retail prices high and then offering discounts and coupons which savvy shoppers could use to bring down the price of goods. Prior to 2012, a trip through any Penney’s store…