Database DevOps stares down testing issues

Typically, developers are the first to uncover the technical difficulties of working with databases in automated test environments.

“They don’t have the right data,” said Brandon Cipes, managing director of DevOps at cPrime, an Agile consulting firm in Foster City, Calif. “How do they make the test environment illustrative of production, and how do they keep that environment updated and secure?”

Test data management tools are available from vendors such as DBmaestro and CA Technologies, but this is where people and process challenges begin to emerge as well, said Nirmal Mehta, senior lead technologist at Booz Allen Hamilton, a consulting firm based in McLean, Va.

“Siphoning production data off a queue and cloning production traffic for realistic inputs can incur infrastructure costs a lot of organizations don’t want to pay for,” Mehta said. “It comes down to not wanting to spend the money for automation — DevOps is trying to force the issue.”

Containers are all the rage in automated application tests because developers can quickly spin them up and then destroy them, but stateful database applications don’t yet lend themselves well to containerization.

“To some extent, there’s a fundamental incongruity between databases and containers,” said E.T. Cook, chief advocate at Dallas-based consulting firm, etc.io, which helps companies deploy containers.

There are tools for persistent storage in containerized applications, but the most tried-and-true approaches don’t solve the issue of true data portability, Cook said. Those require pinning a container to a host so it remains attached to a data volume. Products are available that offer truly distributed container persistent storage, but enterprises still must learn how to manage and integrate these systems.

New tools also tackle containerizing databases so they’ll spin up quickly in test scenarios. Some examples are Ocean9’s containerization offering for SAP HANA and WinDocks’ containerization product for Microsoft SQL Server. Microsoft also supports containerization of SQL Server in its most recent version. For the most part, containerized databases for automated test environments come closest to aligning databases and DevOps.

Production databases present further DevOps challenges

Even when database DevOps problems are solved in test, the gulf is still wide between the test environment and the production infrastructure where apps will actually run. Production databases can be cumbersome to update, require tribal knowledge of complex data schemas to access and operate and are slow to scale.

To tackle these issues, some organizations have moved production applications to NoSQL databases, which don’t require schemas. However, many have backtracked after NoSQL systems presented management problems in production, Cook said.

“A lot of teams realized that, when people said MongoDB was simpler, they meant for setup,” Cook said. “But for a production app, people end up having to build a schema and data sanitization system on top of it anyway, and if you’re not a DBA [database administrator], NoSQL turns into a mess really easily.”

At the same time, databases such as PostgreSQL have advanced in the last 18 months to incorporate nonschema objects, which allow unstructured data to be part of queries against indexes.

“All of a sudden, Postgres became the best of both worlds,” Cook said.

The result is that many organizations find themselves with a patchwork of different databases that range from in-memory databases, such as ElastiCache and Redis, to traditional central data stores in MySQL and Postgres systems and repositories of unstructured objects in NoSQL databases, such as Mongo and Amazon’s DynamoDB. This can burden DBAs with too many systems to manage in the production environment.

Many organizations turn to cloud database services to avoid the need to hire and train armies of DBAs. Cloud database services can also solve common on-premises database issues with rapid scaling and performance tuning in production as applications change. As always, cloud computing comes with caveats around operational costs and the security of systems outside the corporate firewall.

And all of that, of course, assumes an organization’s DBAs have agreed to come along for the database DevOps ride.