The Pervasive Nature of Open Source

Posted on December 7, 2024 by way0utwest

This week there was a headline that said “Open Source Software Powers 96% of Modern Applications, New Study Finds” and if you stopped reading there, you might think, hey, it’s not in the apps I work on. Or you might think that because you use OSS software, most of the world also does. Microsoft, Oracle, etc. are headed for disaster.

If you read a little further, there’s this gem: Open source components are present in 96% of codebases. That’s a far cry from OSS powering most modern applications. I think a better headline might be that OSS helps build most modern software. However, this isn’t an editorial on bad journalism.

I do think OSS stuff is amazing. Many of us in the Microsoft Data Platform space use sp_whoisactive or the First Responder Kit or Diagnostic Queries or some other OSS in our work. Lots of commercial products are built with OSS libraries or components, or there are free versions. Flyway (from Redgate) has an OSS version. OSS helps us build better software, though commercial packages also help.

One of the interesting things about OSS is that many very popular projects have just a few people maintaining them. If those 1 or 2 people disappear, then the project might stagnate. Or worse, if there are security issues, no one addresses them. One of the main attractions of OSS is that anyone can provide a fix or enhancement, but the reality is that most people don’t. Most people just use whatever is out there.

And most people rarely upgrade their OSS. They get something that works and don’t want to change. I get that, as I feel the same way often, especially in the real world. I have shoes, gadgets, tires, etc. that work and I don’t want to change. In the software world, this creates vulnerabilities and security issues, as the report shows with many people still using Python 2. I both understand and don’t understand why this is the case, but I do worry about security.

The other concern is that few people review changes to OSS packages, which has led to previous supply chain attacks with backdoors or vulnerabilities introduced in packages that many other software developers use. Again, OSS is supposed to be better than closed software at preventing this, but the reality is that most (the vast majority) of us are just too busy to look for issues. Even when vulnerabilities are published, far too few developers see the information. Automated scans in CI/CD systems are great, but again, too few people add these to CI/CD pipelines consistently.

Software is hard. In some sense, I’m glad databases don’t have external compilers or use anything other than raw code, but plenty of people still write SQL Injection vulnerabilities in their functions and stored procedures, and many don’t have good visibility into the code that is submitted to their databases, often because the code is assembled at runtime. I wish more people just used stored procedures and included more testing and vulnerability scanning, but that’s a dream. For now, I suggest most of you developer patterns your staff can use and stick with them.

And use version control. At least then we can find all the old, bad code and fix it with some search and replace.

Steve Jones

Posted in Editorial | Tagged , | Comments Off on The Pervasive Nature of Open Source

Big Queries, Big Money

Posted on December 6, 2024 by way0utwest

I had been meaning to post this, so as I finished a piece that referenced this, I decided to post the picture. This was from Small Data SF, where the opening keynote referenced the Google BigQuery demo of a 1PB database.

Here was the slide shown later in the talk.

2024-12_0187

The thing the demo didn’t explain was that query cost $5,580. The conclusion, big data is just too expensive to query often.

Posted in Blog | Tagged , | 4 Comments

Distributed Monoliths

Posted on December 6, 2024 by way0utwest

I was watching a video called Microservices are Technical Debt. In it, the person being interviewed said that a lot of people really have a distributed monolith. That caught my eye since I’ve worked with a number of customers who are trying to adopt microservice architectures for their applications. I think this is less a performance/scaling choice than a reworking of their software development teams, and I’m not sure they will end up with a better system.

What is a distributed monolith? I am not an expert, but this appears to be a place where all the services still depend on each other. For example, I might have a service getting user profile info that an app calls, with another service getting previous orders, and a third service that returns inventory. In a monolith, if any of these are down, the others don’t work. In a distributed monolith, these might be built independently, but perhaps the core app/web page still requires all these to be working to show the user something.

That’s how my gym app seems to work sometimes. It can’t get my profile and the list of classes and the things I’ve registered for, so it hangs. This is very annoying when all I really need is my ID QR code, which never changes, and is validated by another system.

In a microservices architecture, if any of these were down, the others would still be working, and the app would continue functioning. If inventory is down, I can still update my profile. If previous orders are working, I can view them even if the current inventory doesn’t work.

Things are loosely coupled for the end user and for the development teams. Not for just one of those groups.

I’m not sold on microservices, especially when they include databases. There is a lot of power, and efficiency, from being able to pull together different data sets in a database efficiently rahter than doing so in an application, which might require different API calls, separate db connections, and the delays that come with returning more data than needed to let the client join it together. At the same time, I can see how separate services could potentially scale much higher over time.

The thing that I tend to see is that most of us never need to really scale as high as the Google/Facebook/Spotify/Netflix/etc. organizations that espouse these microservice architectures. They have different problem domains and different needs than most of us. Even Uber has realized that one size doesn’t fit all and is moving some things from microservices to macro ones. I’m not against microservices, but they need to be used in appropriate situations, not all of them.

What most of us need is a little more help from Brent or Erik or Paul or someone else in the form of a little consulting that tunes our code to run more efficiently. And if we use them, we should learn a few things from them so that we write better code the next time.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Posted in Editorial | Tagged | Comments Off on Distributed Monoliths

A New Word: Fardle-dun

Posted on December 6, 2024 by way0utwest

fardle-din – n. a long-overdue argument that shakes up a relationship, burning wildly through your issues like a forest fire, which clears out your dry and hollow grievances and reminds you that your roots run deeper than you think.

I have had a few fardle-dins. Both in my personal life and at work. The private ones are private, but let me tell you about a work faux-pax on my part.

I had complained to an individual at a previous company that the way their team approached some software work was incomplete. In my mind, the team often simplified things too often and looked at the software for only a simple use-case, not examining common situations outside of the core use case. At the same time, the teams thought I wasn’t focused enough on different problems to ensure they were well designed.

After seeing this a few times, I publicly criticized their approach and offended the manager of that team. I received a private tongue lashing for this, with the manager pointing out that they do a lot of different work to consider different situations.

We ended up working through our differences, with me understanding their view, but the team also starting to understand why their approaches each tended to be over simplifications and created tunnel vision. I apologized publicly and we burned through our issues, creating a better relationship for the long term.

It’s never a pleasant process when you encounter a fardle-dun, but it can be cathartic and build something better.

From the Dictionary of Obscure Sorrows

Posted in Blog | Tagged , | Comments Off on A New Word: Fardle-dun