The cloud has evolved. Just a few years ago, cloud-based applications were the exception rather than the rule, and on-premises apps were the clear standard for database apps. However, with the numerous advantages of building in or migrating to managed services, the cloud isn’t just a niche anymore. Managed services and cloud VMs are quickly becoming the standard platform rather…
Who has the legal right to access your personal and private digital assets? The answer can be complex, and will depend on where you live, where you are traveling from and to, and whether or not you’ve been suspected of a crime. The rules governing personal digital access are evolving rapidly, and are often subject to creative, on-the-fly interpretations. Digital…
In the last post in my ongoing series about ETL best practices, I discussed the importance of error handling in ETL processes, reviewing best practices for application flow to prevent or gracefully recover from a systematic error or data anomaly. In this post, I’ll dig a bit further into that topic to explore the design patterns for managing bad data in ETL…
The FTP protocol is one of the oldest methods for sharing and moving files. Although frequently considered to be an “old-school” way to transfer data, FTP is still relied upon in most every data movement architecture. Sadly, the functionality around FTP is very limited in SQL Server Integration Services. A common project requirement is to retrieve a list of files from…
I’ll be teaching my popular day-long course, Building Better SSIS Packages, on Thursday, June 1 in Pensacola, Florida. This class is being delivered in conjunction with SQL Saturday Pensacola. Registration is open to the public, and early-bird pricing is in effect until early May. Let me know if you have any questions about this class. I look forward to seeing…
Data warehouse projects are among the most visible and expensive initiatives an organization can undertake. Sadly, they are also among the most likely to fail. At one time, Gartner reported that more than 50% of data warehouses would fail to make it to user acceptance. Because of the size of investment (both time and money) required, the success of such…
In my ongoing series on ETL Best Practices, I am illustrating a collection of extract-transform-load design patterns that have proven to be highly effective. In the interest of comprehensive coverage on the topic, I am adding to the list an introductory prequel to address the fundamental question: What is ETL? What Is ETL? ETL is shorthand for the extraction, transformation,…
When I first started working as a technologist some 17 years ago, I was doing fairly low-level stuff: moving and reimaging computers, network cabling, and basic desktop support were the extent of my responsibilities. I decided early on that I wanted to move into a more advanced role, but faced a common quandary: I couldn’t get the job I wanted…
In designing a proper ETL architecture, there are two key questions that must be answered. The first is, “What should this process do?” Defining the data start and end points, transformations, filtering, and other steps must be done before any other work can proceed. The second question that must be answered is “What should happen when the process fails?” Too…
Last week I posted a brief survey about working the holiday. To those who normally are not required to work the holidays, I asked if they would be willing to do so if the price was right, and if so, what that price would be (relative to what they normally make). There were 46 responses to this survey, and I’ve…