Let’s Talk About Your Development Environment

Let’s talk about your development environment.

Specifically, I’d like to chat with you about the virtual space where your data architecture team, software developers, and information curators do their development and testing work. A proper development environment is logically separated from the production environment, and is often further partitioned into different realms for initial development, data or functional validation, and user acceptance testing. For mature enterprise-ready environments, there is also usually a build and deployment process that automates the movement of code from one environment to the next, reducing the chance for human error when moving code through its paces and ultimately into the production environment.

Seeing that your company is doing some great things and has an eye toward growth, I’m confident you’ll want to have an effective development environment to keep the data and software initiatives on the right track.

Now that we’re on the same page as to what a dev environment is, tell me a little bit about yours.

“We don’t have a dev environment. We push code straight to production.”

I see. While this straight-to-production approach is risky, it’s not uncommon for relatively new data or software teams to do this. However, you’ll want to remedy this as soon as possible, because pushing your new and changed code straight to production exposes your organization to some potentially devastating consequences. Unexpected downtime, system anomalies, information security leaks, or permanent data loss can result from a lack of proper testing and validation.

Testing code in production is one of those behaviors that you can get away with for a while, but it’s just a matter of time before it goes wrong. And when it goes wrong, it can go very wrong.

“But we back up our code and data regularly.”

That’s great! Please continue to do that, but keep in mind that backups are no substitute for proper pre-production testing and validation. Restoring from backup is an option if things go completely off track during a code deployment, but it shouldn’t be your first or only recourse.

When you restore from a backup, there’s almost certainly going to be downtime associated with that restore operation, and you may not be able to get back to exactly the point at which the erroneous code was introduced.

Further, if you don’t discover the issue immediately, you may be in a situation where you have to restore from an older backup and then try to merge any changed data or settings that occurred after the deployment into the restored copy.

The bottom line here is that a restore from backup isn’t a magic eraser, nor should it be your only means of correcting the effects of untested code.

“We can’t afford a development environment.”

I can certainly empathize with the need to limit costs for these systems, but let me ask you this: What is the cost of downtime? What is the cost of a loss of trust in the systems and data? How many customers might be lost if untested code leads to data loss or inadvertent exposure?

I don’t like the phrase “It’s a cost of doing business”, but when it comes to testing code and data, skipping the code and data testing prior to production shouldn’t be an option. Enterprises can’t afford not to have a proper testing and validation environment.

“We have a very small team, so we don’t need a separate development environment.”

I understand that initially it may feel like overkill to set up a separate development environment for just a couple of staff members (or in some cases, a team of one). However, I suggest that it’s at least as important for small teams as it is for large ones. With a small team, you’re likely to have fewer procedural safeguard prior to code promotion, and the chances are good that a feature or change will go from concept to production use with only one set of eyes having reviewed it.

Having a small team should never imply that the source code or systems are less important. As such, proper environments for development and testing should not be limited to large enterprise teams.

“We’re not doing ‘real’ software development, so we don’t need a development environment.”

Even though you’re not creating commercially available software, your staff is still creating software. Powershell scripts, T-SQL code, SSIS packages, Azure data factory functionality, and even database metadata is all source code. These code assets manipulate what your staff members, executives, customers, and shareholders use to interact with and evaluate your business.

Even if you don’t create packaged software, your code and data artifacts are still critical to the success of the business. Make sure they’re ready for the big stage before pushing them out to production.

“We move fast here. Those extra steps would just slow us down.”

The goal of building out a development environment and a formal testing and deployment process isn’t to slow things down. After all, you’re still going to be testing any changes pushed directly into production – it’s just much riskier with a smaller margin of error, and you’re invariably pushing some of the burden the testing onto your end users, customers, and clients.

With a dedicated environment to perform pre-production testing and validation, you’ll have a safe place to perform those tasks you’re already going to be doing anyway. By doing these things in a non-production environment, you’ll have more time and freedom to correct any errors before the changes are pushed out to your front line users and paying customers.

Further, with a contained blast radius in a dev or test environment, your developers can more quickly test potential changes without the risk of impacting the production system. Your team will likely end up doing more tests, but they’ll be doing them each more quickly and without the usual agony and worry of deploying untested code straight into production.

“We do have a development environment, but it’s not like our production environment.”

In a perfect world, the development and testing environments would contain a full set of real production data, stale by no longer than one day, and running on identical hardware or resource level. Each of the endpoints (sources, destinations, APIs, and other supporting cast members) would have a separate implementation for development and testing to allow for as realistic and risk-free testing as possible. Each of the potential points of failure – hardware, network, application, and disk I/O – should have their own development and test implementations with a production-like resource workload to ensure realistic testing that is fully decoupled from production assets.

That said, those things all cost time and money. Fully redundant dev and test environments require setup and maintenance much like their production counterparts, and the systems on which those run (whether physical, virtual, or SaaS-based) add to the total cost of the solution. Further, there are some businesses that are governed by data regulations that may limit the use of real data for development and testing.

My advice to clients is to build non-production environments to resemble production as much as budget, support resources, and regulations will allow. A development environment that uses stale or artificially generated data, or one that has resources that are very dissimilar to production, is still far better than doing development and testing in a production environment.

“We use made-up data for development. Is that sufficient?”

Again, fake test data is superior to having no test data at all, but you’ll find that it’s almost impossible to duplicate the nuances and data quality challenges that your application or data solution will be exposed to in a production environment. There are some decent tools in the marketplace to generate fake testing data in bulk, but those utilities will likely never be able to generate the kind of creative outliers found in the real world.

“We’ve been deploying straight to production for years without a problem. Why change now?”

I’ve worked with clients who have expressed a similar sentiment, but each time I dig deeper into this, it turns out that their straight-to-production approach hasn’t been without problems. The real truth is that they’ve not had a major catastrophe related to the lack of a development environment yet.

It’s not a matter of if, but when, a well-meaning but untested change takes down the machinery. Moving code and data into production will inevitably result in a few surprises. Using a realistic development and testing environment can keep those surprises to a minimum.