Voice of the DBA

Tag: software development

  • Eyes Wide Open

    Not many of us work in startup environments, but many of us do work with new databases that are created for new applications. These might be carefully designed, thrown together, or your database might be constructed by an ORM. In any case, I find many people make decisions and write database code for today, solving the problems that they see in front of them. they often do this with little data and a single system. They might have their eyes on using some new technology, and they decide on a data strategy without really considering what they will need later.

    That seems to be what happened with Expensify, which started in the financial industry. Their system had requirements for low response times, multiple locations, and detailed logging. This required a robust database architecture, which turned out to be helpful when the company pivoted to a new business model. Their CTO talks about some of the problems he sees with startups making database decisions. I think many of these lessons are helpful for all organizations that are trying to ensure their database can grow and meet their needs.

    I think one of the most common problems I see is that developers and leaders get enamored with new technology. There is a lot of promise in some of the platforms and designs that are being put forth. Some are even proving themselves in high-profile situations, but not all. For most of us, however, we aren’t going to be solving the same problems in the same way. As the article notes, we’re not Google, but we’re also no Uber, Facebook, or Spotify. Choosing to mimic their choices because of their success doesn’t necessarily map to our business model. I find no shortage of companies that struggle to adopt some new platform because they built a proof-of-concept and assumed the way the system works with small amounts of data. This becomes an issue later with the moderate or large amounts that they have over time.

    I also see companies creating complexity, with the chance that they will need to deal with many petabytes or exabytes of data at some point. Face it, most of us will barely deal with terabytes of data in any particular system. We ought to plan for a high-performing system at that scale, not worry about a future that will not likely come. At the same time, we aren’t going to be dealing with megabytes of data, so if your developers only test on MBs, they are going to miss problems.

    I like the advice to go into your decisions with your eyes wide open. Don’t copy others, and realistically think about what you will need. I believe that engaging a data professional early is helpful. Developers do some amazing things when they build software, but so often the majority of them don’t really think about the challenges of a database system. They don’t consider low response times or ensuring there are HA and DR (two separate things) strategies. They also forget about the challenges of aggregation and reporting lots of data. Most humans work with a few rows of data at a time, which is what developers do on their systems. When you need to aggregate things, or all your customers are generating a workload, that’s when a data professional can help ensure you’ve properly indexed entities and planned for a demanding workload.

    I do like the common sense advice that most startups won’t outgrow a relatively modest single database server. Many applications might not as well, but that doesn’t mean you can put all your eggs in that one server basket. Make sure if it dies that you have multiple people that can recover it and ensure your system is quickly running in another place. There are different ways to handle this, but engage someone that knows your platform and have them ensure you have some staff, operations or developers, that understand how HA and DR work in your environment.

    Lastly, be secure. I really like the idea of always using stored procedures. I know this becomes a pain for developers, who now write code in two places, but this really helps you ensure better security, and maybe more importantly, ensures you can tune one part of your code regularly, the database side, without impacting the other side.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Digitizing Construction

    Perhaps you think of the construction industry as being more analog and less interested in building a robust digital infrastructure. This article talks about one large company that is modernizing, and has accelerated that move during the pandemic. They are generating lots of data with drones capturing images of projects and and with more collaboration between many different groups that are often not physically co-located together. While their business is construction, they are really a large enterprise that faces many of the same challenges any enterprise faces with moving information around, in a secure and convenient way for their staff.

    However, even smaller companies are depending more and more on a modern infrastructure. I had a contractor renovate my bathroom recently, and while there certainly is some in-person work, plenty of the planning, payment, supply chain, and more comes from the use of digital technologies. Even a 55 year old contractor is embracing new ways of doing business.

    This isn’t just in new construction. Many repair companies around the US are consolidating and joining into groups, with better guarantees for their work, but also more precise scheduling, bidding, smoother (and quicker) payments, and even the sharing of designs and problems through images, reducing the number of physical trips that staff makes, and giving a better experience to the customer. This does come at a charge, often with a higher cost for projects, but I find that I can count on many of these services more than I could with random handy people and contractors in the past.

    Construction will always be a hands-on job for some.  After all, plumbers need to go where the pipes are located. The workers and their companies are taking advantage of more technology and innovations to make their jobs easier. We use new materials, new techniques, and products that make the construction of physical spaces, and the deconstruction, quicker and easier than in the past. In many cases, we often get a superior product, though there is often more cost.

    There are also problems. Any time someone is building something, there is a chance they make an error, or a physical part fails, or the supply chain doesn’t keep up. It does seem that I tend to find fewer of these issues than in the past. When I do, often an organization will fix a problem and stand behind their work. If they don’t, the one part of the digital world that definitely affects construction is the sharing of information by customers. Bad ratings, complaints, and more are common, and can often affect business for contractors quite a bit. Positive ratings can result in more work than an individual can handle.

    Maybe that’s the part I like the best, that the digital connections we have online allow us to share feedback and images of workmanship. At least in my area, that’s led to some good outcomes.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Interconnected Temp Files

    The other day I went to cook dinner for the family. I had picked a new recipe (everyone loved it), and it was going to be a bit of prep. Before I started, I turned on the speaker in the kitchen, connected my phone, and started Spotify. I got 2 sec into the song, just enough for me to turn and reach for the cutting board when the music stopped. I turned back, started it and everything repeated.

    I tried a few times, but it kept happening. I opened Spotify on the iPad we have in the kitchen, where the recipe was displayed and tried there. I had the same experience. At this point, I was getting annoyed and a little stressed. I needed to get cooking, but I also wanted some music. Maybe a little bit of OCD coming out as I checked my desktop with the same result. I updated the credit card and had my daughter check her app.

    A little searching around had me try different things (rebooting, log out/in, etc.). Finally, I found one person that noted clearing the temp files on my desktop might help. I did that, deleting a few GB and cleaning out the UserData folder for Spotify. I restarted the app, and things worked. I walked back to the kitchen and the iPad, and music played there as well. Finally, I could get dinner started.

    I’ve been enamored with some of the Spotify-connected features, allowing a few of us to listen together. I like when I listen in the car (or desktop) and then move to the other location, I can pick up where I left off. However, I hadn’t expected something like corrupt or data problems on my desktop to affect me on another device. As we start to interconnect more apps, it’s possible that a problem on one device might affect others.

    We do interconnect some systems in the data world. We have clusters and Availability Groups, and we certainly sometimes have instances or databases that create dependencies  between two systems. I doubt that many of you have one instance cause a problem with another, but it’s worth keeping in mind. We want connected systems, but we don’t want failures in one place to cascade throughout all the nodes.

    I like connected things, but I want loose coupling. I want one system to run on its own if the other has an issue, but I do want them to share data or status to improve the operation of the software. The big thing is that I don’t want one device (my desktop) to affect the operation of another (my phone). At least not while I’m cooking.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Enterprise Software

    “I’ll go in the store; it’s built for that.”

    That is a quote from Jessica Kerr, who is a developer. She has a great blog on software development, called The Enterprise Eats Software. It starts with a poor experience for an online order for Lowe’s. Interestingly enough, as I read this, I was starting a home renovation project, but I didn’t have a bad experience, I had a great one. Read Jessica’s blog, and see what you think.

    For my short story, we had contracted for a bathroom project, but we forgot to get some of the fixtures. The contractor came a day early to talk about things, and we realized we needed a shower diverter valve and trim kit. We looked for the trim, which is what was important to my wife, and found a valve to fit it. While the contractor called two local supply houses, I looked at Amazon. The suppliers didn’t have the valve, but Amazon did, noting delivery in two days. I ordered it around 10:30 am that day. It arrived around 11:00 am the next day, a day early and before the existing shower had been demoed.

    That was fantastic. It made me think why would I want to use any service that wasn’t that amazing and quick. I got an email at my desk with a picture, telling me the package had been delivered and showed me where it was. I walked it upstairs thinking that calling multiple suppliers and then driving around town would have been a pain,  not to mention a time sink.

    I don’t often find a lot of large companies do a good job with their software integrating into real world operations. A few do, and apart from Amazon, I’ll say build.com was incredible for us to get a tub and Wayfair got us a vanity very quickly (too quickly, actually). Both have built great systems that not only took the order but updated us and handled the complexities of shipping large items. Not to mention they both had an incredible selection available to peruse easily and quickly. Just finding a place to look at something like a tub in person is incredibly difficult.

    Apart from the retail challenges, just the general experience from many enterprises leaves something to be desired. They just aren’t good at being agile, flexible, and maybe more importantly, constantly improving. They get caught up in some of the hassles, like bureaucracy, power struggles, too many rules, etc. They don’t know how to operate in a flexible, agile, constant change environment.

    I’m reading Project to Product, which in many ways sees a lot of the same problems. Software is disconnected from the goals of the business, and too often there are individuals and processes that get in the way of becoming more effective across teams and partners. BMW is one of the success stories here, and I still think they’re behind Tesla in their industry. They might catch up, and if they do, it’s because they are learning to be a better software company, not a better manufacturer.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.