A Lack of Architecture and Planning

Posted on August 19, 2024 by way0utwest

A few weeks ago, I was sitting in the audience, waiting for my turn to speak at DevOps Days in Minneapolis. Just before me, Xe Iaso delivered a funny and thought-provoking talk on building a social network on a whiteboard. It was very well done and had me feeling nervous about following that session.

The talk is a bit of a satirical look at an interview Xe had for a company that tried to get them to derive an architecture for a large distributed system. It was interesting to hear Xe note that often we have architecture diagrams of what we’d like to have, but never an explanation of how we implement a large system, especially one that has to grow as our workload grows.

This talk was a nice analogy of how often we get into situations where many of us can’t believe our system was structured this way. We often wish we could completely redesign things from scratch, and we’d do it better. Why didn’t the previous engineers think things through?

Watch the talk. It shows how a lot of software is built. We build what we think we need, but when we get overwhelmed, or often when we get unexpected pressures from others, we make decisions that seem to be the best ones in the moment. However, a few years later, with a few of these decisions behind us, we realize that each of those choices was too short-sighted. We have a mess of software the seems cobbled together rather than well-engineered.

I don’t believe we can engineer everything well from the beginning. I also don’t believe in early optimization, mostly because I think we are asked to build a lot of things that are never (or lightly) used. Who uses the paintbrush in Word or the FACTDOUBLE() function in Excel? However, I do believe that we ought to write code that performs well the first time, following patterns we (should) know well. We certainly should assume that whatever code we write will see at least 10X more data in production than in dev/test, so prepare for processing more data.

I’m sure many of you know of a codebase and system that doesn’t run well. I hope most of you have another application that does run well and is fun to work on. Hopefully, the goal of your organization is to turn out more software like the latter than the former.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Posted in Editorial | Tagged | Comments Off on A Lack of Architecture and Planning

SQL Saturday Denver 2024 Resources

Posted on August 17, 2024 by way0utwest

Here are the resources from my talks today.

Best Practices for Seamless Database Deployments

Architecting Zero Downtime Deployments

Some good questions today, which I’ll write a few posts on:

  • What are feature flags
  • How do you handle adding a clustered index
  • How do you communicate with downstream groups on something like a rename

Probably a few more, but it’s been a long day and I’m tired.

Posted in Blog, Presentations | Tagged , , , | Comments Off on SQL Saturday Denver 2024 Resources

Analyzing Breached Data

Posted on August 17, 2024 by way0utwest

A few of you out there might be data scientists who profile data regularly. Probably a fair number of you do import/export work and learn to check data values, perhaps with counts, distincts, or other aggregates. I don’t know if the performance tuners out there look at the skew of data or the details of what is in a query that needs improvement. However, all of you are likely familiar with data and trying to query it for some type of meaning.

One of the largest data breaches occurred with National Public Data. Troy Hunt analyzed the breach as a part of his work with haveIbeenpwned. The piece is an interesting analysis of the data, trying to determine both it’s legitimacy as well as what is actually included in the breach. It’s a fascinating read and I encourage you to look at it not just from the data analysis side, but also to be aware of what data about you is being aggregated and sold by companies.

The read is interesting as it is a bit of a detective story, digging through data in a folder, which is something I’ve had to do. I’ve had people in previous jobs just dump a bunch of data on me and ask me to load it into a database. Or a table. Often without them knowing what type of data it is, what formats, do files relate to each other? Are there multiple tables worth of data in a file? All questions I’ve had to ask myself (and answer), and similar to what Troy did to analyze the breach.

Data is very important to many of us, in different ways, but I’m often amazed at how few people actually understand how to organize data and ensure others can track the metadata about their data (what their data represents). I’m guessing this is why every person that gets an extract of data to load into Excel formats it in different ways.

In many cases, people want the ability to query data, but they prefer to just focus on one table that contains a lot of information. They don’t want to know how to “join” data together. I think this might be the reason we see so many views in databases, and why we have views built on views. Each new client of the database needs their own view structure.

The world of data is a mess, even inside an organization. Once we start moving data between organizations, it’s truly a mess. We might bemoan all the inefficiencies and work we do to move, change, and re-load data as custom, human ETL machines, but there is one great thing about this tangled web. It provides for steady, secure jobs for many of us with no end of work in sight.

Steve Jones

Posted in Editorial | Tagged , , | Comments Off on Analyzing Breached Data

What Do You Drop?

Posted on August 16, 2024 by way0utwest

Many years ago I was training for a triathlon. I had competed in the Sandman Triathlon the previous year in Virginia Beach and wanted to do it again. I had a young child, work was busy, and I was struggling to find time to swim, bike, and run every week. One night, I was at a work event with a customer who was also a triathlete. He was much more competitive and successful than I was at competing in triathlons, and he told me I should just get up earlier and find time to train or ensure I spent time after work on training before I went home to ensure I was meeting my goals.

That sounded fine. Want to be better at something, then spend time on it. Certainly, that’s what I often advocate for your career. Spend time on your career.

However, if I get up earlier, then that means I’m more tired at the end of the day. I’ll go to sleep, or more likely fall asleep, earlier and miss time with my wife. She won’t like that. If I try to ensure we get the same amount of time, I’d likely shortcut time with my kid. There’s no magic way to find more time. If I take time to do one thing, I’m taking time away from something else.

The same thing happens at work. Our Chief Marketing Officer noted this at our global meetup recently, saying that too many people are adding new tasks or projects and letting other work fall away. They don’t mean to let other work drop, and sometimes that’s a problem, but the reality is that we can only get a certain amount of work done as a group, and if we add new work, old work gets lost. The same thing applies to coding software. We might get more work from a developer in the short term, but that falls apart long term, and it can be bad for retention.

Time is one of the most valuable resources you have in life. I see this more and more as I age, and you must recognize that it’s a limited resource. For a short time, you might be able to get more time by sleeping less (or working more), but those things mean you are dropping other things in your life. That often isn’t good for our health, relationships, or happiness over time.

If you want to do something more, or new, then you should consciously decide what to drop. You have to make decisions and choose what is a priority and what is not. The things that are not a priority might get dropped (or their time reduced). That’s a big part of growing and maturing, as well as one of the worst parts. Making choices is hard.

Decisions you make are rarely permanent. They are often choices you make for a period of time. You’ll make some great choices and those might be long-term or permanent. You might make bad choices, which hopefully are short-term, and then decide to make a new choice. Whether this is at work or in your personal life, make the choices that drive you forward, towards your goals, but with an eye on keeping a balance across all parts of your life.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Posted in Editorial | Tagged , | Comments Off on What Do You Drop?