Tag: software development

  • Patching the Patch

    I had to make a few changes to a SQL Saturday event recently. The repo is public, and some of the organizers submit PRs for their changes, and others send me an email/message/text/etc. for a change. In this case, an organizer just asked for a couple of image updates to their site. I opened VS Code, created a branch, added a URL for the images, and submitted my own PR. After the build, I deployed it.

    And it didn’t work.

    I had a broken image. I checked the URL in code and realized I had “events” copied before the URL, which wasn’t valid. Ok, edit the URL to be correct and repeat: new PR, build, merge, deploy.

    And it didn’t work.

    I was looking at the code live on the site, the code in the repo, and I was trying to reconcile paths and file names and keys and values and a few other things.

    I realized the world for a developer hadn’t changed a lot, and in fact, I was in the age-old loop: deploy, patch, patch the patch, fix the patch for the patch, and so on. I don’t even know that I could have gotten better here with testing, as these were one-off data changes that affected the site for users. If I enter the wrong data, it’s wrong. I can’t easily test for this.

    I have written code that was wrong, and a few simple tests would have caught my issues. I’ve also written code that isn’t easy to test. If I am adding or changing data, it’s hard to test that. Often, I might do some copy/pasting between the code and the test to generate the test. If I’ve typoed something, the typo continues through the test (in some cases). Even using a code generator or an AI to produce the INSERT or UPDATE code might not solve the problem. They might read my typos in a prompt.

    One of the best things to help code quality in the last few decades is continuous integration (CI), where we have automated systems that compile code, test it, and run it. It’s not perfect, but it does help reduce the silly mistakes many of us likely make every day when writing code. These can’t prevent typos and issues, but if we are testing intermediate systems, hopefully somewhere along the way, a human or AI agent tries to verify that the things we were typing exist and can catch a typo.

    In this case, I had to find where I’d mistyped the line and realized that I had the path wrong. The image was in a subfolder and I needed to add that to the img url.

    Working with data is hard, and it’s a constant source of simple mistakes. I don’t know we’ll ever get away from patching the patch when data manipulation is involved.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Shorten the Debate

    Many of us are faced with choices and decisions constantly in our jobs. How do we approach a problem? What should we do as a team to get the work done? How do we code or manage or test or do something else with a database?

    Maybe more importantly, how long do we spend deciding?

    I have seen teams spend way too long (in my opinion) debating options and examining possibilities. I’ve seen them take days or weeks arguing and considering edge cases and move slowly. It seems there is no shortage of reasons why something isn’t done. It can drive me a little crazy.

    I was listening to a podcast recently and heard about this technique, which I love.

    Get a whiteboard that everyone can see (physical or virtual). One person is designated to write down all the discussion items about the issue. Each person can make an argument for or against an idea for the solution, and nobody can stop that argument from being added. However, nobody can remove anybody else’s argument, and nobody can repeat an argument that is on the board.

    This can shorten discussions because people can’t repeat things. I’ve seen far too many debates (arguments) continue in a circle because people keep repeating things or circling back. When no one has anything new, we just take a vote and move on.

    I am a big fan of getting things done. Even if we don’t have the best, or optimum, or more efficient solution, we need to get moving. Perfect is the enemy of good enough, and far too often, I find technical people chasing perfection, or near perfection, at the expense of moving forward.

    Timebox decisions and get moving. It’s how you accomplish more, and it’s what your employer wants.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • The Pervasive Nature of Open Source

    This week there was a headline that said “Open Source Software Powers 96% of Modern Applications, New Study Finds” and if you stopped reading there, you might think, hey, it’s not in the apps I work on. Or you might think that because you use OSS software, most of the world also does. Microsoft, Oracle, etc. are headed for disaster.

    If you read a little further, there’s this gem: Open source components are present in 96% of codebases. That’s a far cry from OSS powering most modern applications. I think a better headline might be that OSS helps build most modern software. However, this isn’t an editorial on bad journalism.

    I do think OSS stuff is amazing. Many of us in the Microsoft Data Platform space use sp_whoisactive or the First Responder Kit or Diagnostic Queries or some other OSS in our work. Lots of commercial products are built with OSS libraries or components, or there are free versions. Flyway (from Redgate) has an OSS version. OSS helps us build better software, though commercial packages also help.

    One of the interesting things about OSS is that many very popular projects have just a few people maintaining them. If those 1 or 2 people disappear, then the project might stagnate. Or worse, if there are security issues, no one addresses them. One of the main attractions of OSS is that anyone can provide a fix or enhancement, but the reality is that most people don’t. Most people just use whatever is out there.

    And most people rarely upgrade their OSS. They get something that works and don’t want to change. I get that, as I feel the same way often, especially in the real world. I have shoes, gadgets, tires, etc. that work and I don’t want to change. In the software world, this creates vulnerabilities and security issues, as the report shows with many people still using Python 2. I both understand and don’t understand why this is the case, but I do worry about security.

    The other concern is that few people review changes to OSS packages, which has led to previous supply chain attacks with backdoors or vulnerabilities introduced in packages that many other software developers use. Again, OSS is supposed to be better than closed software at preventing this, but the reality is that most (the vast majority) of us are just too busy to look for issues. Even when vulnerabilities are published, far too few developers see the information. Automated scans in CI/CD systems are great, but again, too few people add these to CI/CD pipelines consistently.

    Software is hard. In some sense, I’m glad databases don’t have external compilers or use anything other than raw code, but plenty of people still write SQL Injection vulnerabilities in their functions and stored procedures, and many don’t have good visibility into the code that is submitted to their databases, often because the code is assembled at runtime. I wish more people just used stored procedures and included more testing and vulnerability scanning, but that’s a dream. For now, I suggest most of you developer patterns your staff can use and stick with them.

    And use version control. At least then we can find all the old, bad code and fix it with some search and replace.

    Steve Jones

  • Distributed Monoliths

    I was watching a video called Microservices are Technical Debt. In it, the person being interviewed said that a lot of people really have a distributed monolith. That caught my eye since I’ve worked with a number of customers who are trying to adopt microservice architectures for their applications. I think this is less a performance/scaling choice than a reworking of their software development teams, and I’m not sure they will end up with a better system.

    What is a distributed monolith? I am not an expert, but this appears to be a place where all the services still depend on each other. For example, I might have a service getting user profile info that an app calls, with another service getting previous orders, and a third service that returns inventory. In a monolith, if any of these are down, the others don’t work. In a distributed monolith, these might be built independently, but perhaps the core app/web page still requires all these to be working to show the user something.

    That’s how my gym app seems to work sometimes. It can’t get my profile and the list of classes and the things I’ve registered for, so it hangs. This is very annoying when all I really need is my ID QR code, which never changes, and is validated by another system.

    In a microservices architecture, if any of these were down, the others would still be working, and the app would continue functioning. If inventory is down, I can still update my profile. If previous orders are working, I can view them even if the current inventory doesn’t work.

    Things are loosely coupled for the end user and for the development teams. Not for just one of those groups.

    I’m not sold on microservices, especially when they include databases. There is a lot of power, and efficiency, from being able to pull together different data sets in a database efficiently rahter than doing so in an application, which might require different API calls, separate db connections, and the delays that come with returning more data than needed to let the client join it together. At the same time, I can see how separate services could potentially scale much higher over time.

    The thing that I tend to see is that most of us never need to really scale as high as the Google/Facebook/Spotify/Netflix/etc. organizations that espouse these microservice architectures. They have different problem domains and different needs than most of us. Even Uber has realized that one size doesn’t fit all and is moving some things from microservices to macro ones. I’m not against microservices, but they need to be used in appropriate situations, not all of them.

    What most of us need is a little more help from Brent or Erik or Paul or someone else in the form of a little consulting that tunes our code to run more efficiently. And if we use them, we should learn a few things from them so that we write better code the next time.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.