When moving data in an extraction, transformation, and loading (ETL) process, the most efficient design pattern is to touch only the data you must, copying just the data that was newly added or modified since the last load was run. This pattern of incremental loads usually presents the least amount of risk, takes less time to run, and preserves the…
I am a professional data geek, having worked in this space since 2003 (that was way back even before MySpace, for those who are old enough to remember). I have been a full-time consultant and trainer for more than 10 years, working with clients of all sizes and business sectors. I am available for hire for data strategy, tactics, or…
Did you know that SQL Server Management Studio can help you arrange your SQL scripts into logical groupings? SSMS includes the ability to associate multiple code files together into projects for better clarity and ease of maintenance. In this post, I’ll show you the basics of SQL Server Management Studio script projects. SQL Server Management Studio Script Projects For those…
The weather is turning cooler, there is holiday music everywhere, and the relatives are all excited to get together and talk about politics. That can only mean one thing: it’s almost time for Festivus! For the last three years, I have spent the 11 days leading up to Festivus writing a blog post each day, and I’m delighted to continue…
Today is the first full day of the PASS Summit, and this morning we kicked things off with a 2-hour opening ceremony and keynote. Networking and Growth PASS president Grant Fritchey starts things off appropriately by focusing on networking and career growth. We say it every year, but it bears repeating: the most important thing you’ll get out of being…
Change detection is a critical component of any system that moves data from one structure to another. Most data movement mechanisms are designed to move a subset of the data – only that which is new or changed since the last load. For that to work properly, there must be a reliable system for detecting delta data touched since the…
Call me pedantic, but I can be a stickler for standardized naming conventions. Whether it’s application code, database objects, documentation, or other digital assets, I find a tremendous amount of value in establishing a set pattern for naming and formatting. Especially for organizations with a large digital landscape, having a documented and agreed-upon standard for object names can make developers…
It’s that time again! My Eleven Days of Festivus blogging extravaganza is now in its third year, and I’m happy to share one blog post per day with you for the 11 days between now and Festivus Eve (December 22nd). As before, I’ll link all of the posts from this page. May your feats of strength be mighty, your airing…
I am honored to have been selected to be a presenter at this year’s PASS Summit coming up this November. I will be presenting a half-day talk entitled Build a Metadata-Driven ETL Repository with Biml and SSIS: Your boss tells you that a new field will be added to one of your vendor’s data files, and asks how long it will…
Encapsulating business logic into data movement and presentation is a critical part of a stable information management strategy. Too often, though, business logic is built and added late in the process, forcing it into whatever nooks and crannies are available. While this duct-tape approach sometimes works, it makes the resulting system difficult to maintain when the business logic is spread…