Voice of the DBA

Category: Editorial

  • Microservices

    Recently I wrote about software trends from 2019 and how applicable they might be today. One of those was microservices, which I don’t see implemented that often with relational databases. I do see developers wanting to use them, but I don’t see them often with a RDBMS backend.

    I did see a piece on microservices, which mirrors that view. Lots of talk about the architecture, but not a lot of implementation. O’Reilly did a survey in Feb 2020, just before the world changed, and the found that only 10% reported complete success, but 54% felt mostly successful, and 92% some success. That’s better than I thought.

    The respondents are self-selecting, so it’s not surprising that there might be some success. It’s not surprising that a third of these people are moving over half of their legacy systems to this architecture. That doesn’t mean they’ll be successful in the move, or that they’ll continue.

    They also note that containers are important, and I think this might be more the nature that containers lend themselves to microservices. The companies I’ve know that heavily use containers tend to use microservices, almost as if the two technologies reinforce each other.

    I think culture plays an important part in any success, and if I were in an organization, I’d focus on this. Perhaps just like DevOps, with a small team proving some success and using that as evidence to start convincing others to move. However, I’d also need to have a good argument about why we should rewrite applications, or even bet heavily on a new architecture. I’ve yet to see a good reason why microservices work well with an RDBMS, but I’m open to someone convincing me.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Better Government Security Through DevOps

    Most teams building software seem to go a little too fast to ensure their code is both secure and of high quality. I don’t think it really  matters whether you are working in a waterfall process, agile, lean, or any other methodology. Whether fast or slow, humans will make mistakes, new code can introduce a vulnerability. Even if you follow great practices, it seems that hackers and criminals find new attack vectors all the time. I’m not sure we really can go slow enough and stay in business.

    Those of us working as data professionals know that protecting the data in our databases is important. We are reluctant to allow too much change too quickly, especially when there might be changes that affect security. However, is limiting change the best idea?

    I’d argue no. DevOps preaches the ability to update on demand, and often, as soon as code is complete. This doesn’t mean we don’t test or pen test or run security scans or anything else. It does try to limit the work in progress, which means that we aim to allow updates to our lives systems regularly.

    An article for CIOs notes this that DevOps helps us improve security, precisely because we can fix things quickly. This might be especially important in high security environments, like government systems. The ability to patch, correct faulty code immediately, and respond to threats is important. There could be breakage from fast moving code, but another part of DevOps is improving your knowledge and skills, working to improve not only the quality of existing code, but also the quality of all future, first written code.

    I would rather manage database systems that backed applications being updated on demand in a DevOps flow. I’d rather be able to patch and update libraries, platforms, and frameworks quickly. We’ve seen the problems in systems that aren’t updated with the Equifax breach. We should learn from this incident and ensure we can patch and update systems on demand, whenever we need to do so.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Data Analysis Without a Server

    Most of us that subscribe to this site are data professionals, and we work with large amounts of data for our organizations that is usually stored on a server class system, with TB sized, high performance storage systems. Whether on premises, in the cloud, or another data center, our employers have made an investment to provide high quality data services for our clients. This large investment is often a big decision, and setting up a new system to handle lots of data as an experiment with data analysis can often seem to take ages.

    The R language has been popular for data analysis for years, though the data sets examined were often limited in size, usually because of workstation limitations. One of the reasons Microsoft added R services to the data platform was to move analysis closer to big stores of data and increase the ability of organizations to “operationalize” or deploy their analysis and models to a wider audience.

    Deciding when to make that investment can be tricky, but the more that someone can prove some value from a smaller experiment, the more likely it is that an organization might decide to move forward. Recently, I ran across an interesting article, where the author had analyzed a billion row dataset on a commodity laptop. In this case, a Macbook Pro costing US$4000, but that’s a pittance compared to deciding to invest in HDFS storage, a Big Data Cluster, or even a large cloud experiment.

    What caught my eye here is that the analysis tool used, OmniSciDB, was engineered to run on CPUs, not GPUs, and performed very well in analyzing the data. I haven’t found the time or set up the disk space to try and load the billion rows into a SQL Server columnstore index, but I’d be curious how that might perform on the same data. The queries run are fairly simple aggregations, and my guess is SQL Server would perform extremely well once the index was built. If someone else wants to try it and take notes, I’d love to read the experiment as an article on SQLServerCentral.

    It has become more and more likely that before we embark on any large project in an enterprise that we perform some sort of prototyping and development on a small system. I think that’s true whether we’re building a web app or setting up a data science experiment that might drive our business forward. I always enjoy reading when someone has run tried a large scale analysis experiment on a workstation, not a server, and I hope we continue to see more people doing this and sharing their results in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Looking Back at Software Development Trends

    In some ways, the world of software development hasn’t changed much. The same sorts of skills and techniques I saw people using on COBOL programs hitting DB/2 and C++ over Oracle 6 are used today in React and C# against Azure SQL database. On the other hand, it does seem that we are more mature in how we work together and the flexibility with which we design systems.

    I saw the results from a survey from 2019 that Atlassian ran for software developers. This was a look at what modern trends might exist, though over a year later, perhaps the world is dramatically changed again. Let me look at each of the four trends they point out.

    I’ve been hearing about microservices for years, but have found relatively few customers using them. I don’t hear a lot about them in the RDBMS space, and I think this is because the idea of separating out each entity (or small set) in a database and having data access only through a front end component doesn’t make sense. There’s power in using an RDBMS to enforce data integrity rules and allow aggregations. I’ve also seen some companies looking for miniservices, not micro ones.

    Manual testing is still very prevalent for customers. While there are lots of unit tests, including some against a db, they don’t always extend through CI to more complex scenarios. Quite a few customers still have bottlenecks where humans look at the application in a larger sense. I think the high cost of tools that run more complex tests is a part of the problem. I’m not sure how we improve this, though I do hope to see more unit or functional tests for db code.

    Feature flags are extremely complex, though I do see them more as a defensive measure, where features are released, but then turned off if there are issues. This prevents a rollback from the app and db standpoint. I also don’t see a lot of use of dark deploys for database features, perhaps because until the app is working, we aren’t sure the data model is correct. Feature flag cleanup is certainly an issue for some clients.

    The last trend is one I rarely see implemented. Looking at customer outcomes, and not just immediate sales, is something that few people seem to do. Perhaps because developers are evaluated on the work they complete, not whether it’s in use. Managers are evaluated based on getting developers to do work, or on the sales that are produced, but it seems that few organizations try to measure the customer impact. We’ve started doing that at Redgate, and I’m interested to see how this evolves.

    Keeping developers motivated, excited about their jobs, and productive with creative solutions is tough. The trends listed from the article seem to me more aspirational for most of the organizations with which I deal. Most clients I know see developers as interchangeable parts, similar to factory workers. I think if they invested a little more in the well-being of developers, both from their mental focus and the growth of their skills, they might find a lot more benefits accruing from their software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.