Voice of the DBA

Category: Editorial

  • The Challenges of Resetting Databases

    I was working on a demo recently where we had a database in version control and a development database. This was a team environment, with a few of us making changes and syncing them across our dev systems using git. We had some advanced technology with our dev environments in containers, Flywaydb, and GitHub. Once we had our scenarios working, someone wanted to reset our git repo and capture a new database image.

    However, when we reset the repo back, we had some issues with the database. In this case, there were changes in the database that didn’t exist in the repo, giving us a mismatch. Not a big problem, but cleaning things out to get the db to match the repo, without putting those changes into the repo, was a challenge.

    A developer I was working with got a little frustrated, because when working in C#, there is no state. If we reset the repo and sync our local copy, we have everything ready to go. However, a database repo isn’t the same because there is often a database that exists separately.

    This is the main challenge when working with databases, relational or otherwise, in a development environment. Experiments, bug fixes, even testing data changes persist over time. Resetting data to repeat tests, or even automating tests, can be hard.

    This is one reason I think containerization and subsetting of production datasets will become very important over time as we try to ensure we can react to business requirements and keep our teams coordinated. These technologies ensure we always have a known starting point for our databases. At least at any particular moment. We certainly need to update this foundation as we deploy changes to databases.

    Hopefully Microsoft, more vendors, and us as developers help advance these technologies, as well as help all developers build more skills. I’m grateful to Andrew Pruski, Anthony Nocentino, and others for the information they share about containers and databases. Hopefully we see more people engaging in these areas over time.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Microservices

    Recently I wrote about software trends from 2019 and how applicable they might be today. One of those was microservices, which I don’t see implemented that often with relational databases. I do see developers wanting to use them, but I don’t see them often with a RDBMS backend.

    I did see a piece on microservices, which mirrors that view. Lots of talk about the architecture, but not a lot of implementation. O’Reilly did a survey in Feb 2020, just before the world changed, and the found that only 10% reported complete success, but 54% felt mostly successful, and 92% some success. That’s better than I thought.

    The respondents are self-selecting, so it’s not surprising that there might be some success. It’s not surprising that a third of these people are moving over half of their legacy systems to this architecture. That doesn’t mean they’ll be successful in the move, or that they’ll continue.

    They also note that containers are important, and I think this might be more the nature that containers lend themselves to microservices. The companies I’ve know that heavily use containers tend to use microservices, almost as if the two technologies reinforce each other.

    I think culture plays an important part in any success, and if I were in an organization, I’d focus on this. Perhaps just like DevOps, with a small team proving some success and using that as evidence to start convincing others to move. However, I’d also need to have a good argument about why we should rewrite applications, or even bet heavily on a new architecture. I’ve yet to see a good reason why microservices work well with an RDBMS, but I’m open to someone convincing me.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Better Government Security Through DevOps

    Most teams building software seem to go a little too fast to ensure their code is both secure and of high quality. I don’t think it really  matters whether you are working in a waterfall process, agile, lean, or any other methodology. Whether fast or slow, humans will make mistakes, new code can introduce a vulnerability. Even if you follow great practices, it seems that hackers and criminals find new attack vectors all the time. I’m not sure we really can go slow enough and stay in business.

    Those of us working as data professionals know that protecting the data in our databases is important. We are reluctant to allow too much change too quickly, especially when there might be changes that affect security. However, is limiting change the best idea?

    I’d argue no. DevOps preaches the ability to update on demand, and often, as soon as code is complete. This doesn’t mean we don’t test or pen test or run security scans or anything else. It does try to limit the work in progress, which means that we aim to allow updates to our lives systems regularly.

    An article for CIOs notes this that DevOps helps us improve security, precisely because we can fix things quickly. This might be especially important in high security environments, like government systems. The ability to patch, correct faulty code immediately, and respond to threats is important. There could be breakage from fast moving code, but another part of DevOps is improving your knowledge and skills, working to improve not only the quality of existing code, but also the quality of all future, first written code.

    I would rather manage database systems that backed applications being updated on demand in a DevOps flow. I’d rather be able to patch and update libraries, platforms, and frameworks quickly. We’ve seen the problems in systems that aren’t updated with the Equifax breach. We should learn from this incident and ensure we can patch and update systems on demand, whenever we need to do so.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Data Analysis Without a Server

    Most of us that subscribe to this site are data professionals, and we work with large amounts of data for our organizations that is usually stored on a server class system, with TB sized, high performance storage systems. Whether on premises, in the cloud, or another data center, our employers have made an investment to provide high quality data services for our clients. This large investment is often a big decision, and setting up a new system to handle lots of data as an experiment with data analysis can often seem to take ages.

    The R language has been popular for data analysis for years, though the data sets examined were often limited in size, usually because of workstation limitations. One of the reasons Microsoft added R services to the data platform was to move analysis closer to big stores of data and increase the ability of organizations to “operationalize” or deploy their analysis and models to a wider audience.

    Deciding when to make that investment can be tricky, but the more that someone can prove some value from a smaller experiment, the more likely it is that an organization might decide to move forward. Recently, I ran across an interesting article, where the author had analyzed a billion row dataset on a commodity laptop. In this case, a Macbook Pro costing US$4000, but that’s a pittance compared to deciding to invest in HDFS storage, a Big Data Cluster, or even a large cloud experiment.

    What caught my eye here is that the analysis tool used, OmniSciDB, was engineered to run on CPUs, not GPUs, and performed very well in analyzing the data. I haven’t found the time or set up the disk space to try and load the billion rows into a SQL Server columnstore index, but I’d be curious how that might perform on the same data. The queries run are fairly simple aggregations, and my guess is SQL Server would perform extremely well once the index was built. If someone else wants to try it and take notes, I’d love to read the experiment as an article on SQLServerCentral.

    It has become more and more likely that before we embark on any large project in an enterprise that we perform some sort of prototyping and development on a small system. I think that’s true whether we’re building a web app or setting up a data science experiment that might drive our business forward. I always enjoy reading when someone has run tried a large scale analysis experiment on a workstation, not a server, and I hope we continue to see more people doing this and sharing their results in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.