Voice of the DBA

Category: Editorial

  • When to Use a Database

    One of the trends of the last ten years has been for many developers to try and avoid using a relational database where possible. Some look to NoSQL data stores, and others even consider flat file type stores of JSON or other formats that allow developers to work with speed and agility. Quite often it seems that applications grow to require some sort of relational store, often as an additional data store.

    I ran across an article from a data science and analysis developer that is often performing work in R or Python on datasets. At some point, the post notes that when your dataset(s) become larger than memory, you might want to consider using a local database of some sort.

    Actually, the first question the author asked was “when is your data too big?” Their answer: when operations take a long time, which was 20 seconds for the author. I tend to agree as I am looking for Notepad-like startup performance for apps, and query results in low 10s of seconds.

    Most people that perform some sort of data analysis understand tables. Whether this is in R, Python, or even Excel, the table structure for data is familiar and easy to work with. While some analysts might not be overly concerned about normalization, that isn’t always a problem for situations where data is loaded into systems and rarely (if ever) updated. In these cases, just having a database of some sort, could speed up your work.

    I think you ought to use a database early, if for no other reason than this is good practice with loading and storing data in a form that is persistent, scalable, and often can perform better across time with disparate queries and data manipulation. While quick experimentation is rapid with in-memory tools, I think a database is better suited to queries across time.

    I know I’m biased, but if you find data scientists and other analysts struggling with data sets, offer them a database. They can easily share data, you can protect it with backups, and you might find that you both learn a few things from working together.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Taking Stock of Your Career

    Yesterday was Thanksgiving in the US, a day where families typically celebrate together and many people give thanks for what is going well in their lives. It’s also one of the days when more businesses and services are closed than any other day of the year.

    This is the time of year when most businesses look forward and plan the future. It’s also a time when lots of individuals may do the same thing, getting ready to make resolutions or attempt to change their lives in the new year.

    Today, I want to do something a little different. Rather than taking stock of your career and looking forward, let’s look back and find gratitude.

    Is there a situation, a person, a job that you were grateful for that helped your career for the better? Who would you thank? What helped you become the person you are today? It doesn’t matter if you are an older professional like me or someone starting your career. What are you grateful for in your past to put you where you are today?

    For me, there are lots of things to choose from. I’ve been very blessed and lucky. I’ve worked hard to take advantage of opportunities, but things have worked out well. There are a few things that stand out in my mind.

    First, I got a job interning at a nuclear power plant while in graduate school. That job changed my life. Someone recognized my value and gave me a chance to program, to learn how to run a network, and helped me learn SQL Server (on OS/2). I also met my wife there, so perhaps that’s the situation I’m most grateful for in my life.

    The other situation was the job that moved me to Denver. It allowed my family to change locations and my kids to grow up in a wonderful place. The job wasn’t great, long hours, and I slept in my office far too many nights, but I learned how not to do a lot of things. From poor management, poor software processes, and bad operations practices. It was a tough period of growth that I still value today.

    What are you thankful for in your career? Let us know in the discussion today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • A Strange AI Achievement

    We are seeing AI and ML becoming used in more and more industries, but one that seems to be a place where it is embraced with some success. From speech recognition and transcription to analyzing imaging, computers have helped medical professionals improve the care they give to patients.

    Just as CAD has helped manufacturers, AI systems are being used in medical research, trying to model and screen medications to try and determine which ones are potentially useful in treating various diseases. We also have used them to better tailor treatments for certain diseases, like some cancers. It seems that applying computing against vast troves of data is proving itself beneficial.

    What if an AI were able to develop something new and it were awarded the Nobel Prize? That’s the premise in a scenario published in the Economist. The article opens with the controversy of how the guidelines for the prize are interpreted, paving the way to award the prize to the AI. Perhaps even more interesting in the scenario is that the effort is the result of a poor software upgrade that allowed the AI to read more medical papers that it was previous given access to examine.

    It’s an interesting idea. Who gets the credit? Certainly, the humans that help train the model and put it to work deserve some credit, but they are really the assistants. If they were to use the prize for more research, do we think they could replicate the innovation? Maybe it doesn’t matter. Not many people win the Nobel prize twice. Perhaps using the prize to continue allowing the AI to conduct research would make the most sense, though I don’t know if the humans around the system would accept that.

    I don’t think AI systems are more intelligent than humans, but they can consider and try more possibilities than humans, given enough data. They might notice something that we’d miss, and they can remain more focused on a problem than we can. After all, we need to rest and care for our bodies.

    I don’t know if this will happen in the 2030s, as the scenario imagines, but I do think this is a possibility as we start to use computing to search for new innovations in research. I just hope that as these discoveries take place, they are used to better the entire world, and not just enrich a few humans.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Complexity of Modern Systems

    I am a big advocate of version control and testing for your software. The main reason is that modern systems are incredibly complex, and capturing what has been done (and what is happening), along with looking for potential issues, is important. It might be critical for ensuring you have success in building software.

    Part of the reason is this post on software complexity. Now I don’t think the complexity is killing software developers. I do think it adds to developers’ stress, and in poorly managed environments, the complexity makes for a poor job experience for many.  While I do think most people are smart, and they find ways to overcome this, they just don’t enjoy the process.

    This is part of the reason, IMHO, that we are starting to see lots of people move to a new role, often in a new company. With remote work, there are more options, and for people with talent, they can easily leave. My recent job searches seem to show more remote job options and more demand for mid and senior level people. I think some of this is looking to get better talent in your company, and some are because experienced people are tired of poor work environments and poor code.

    Modern software is big, and I don’t think any system of consequence is built by individuals anymore. There are still good projects and applications being built by a person, but once they start to grow, a team of developers is needed. This means that we have to learn how to better work together. We need to adhere to standards. We need to communicate and collaborate both in code as developers and together as humans.

    While someone may understand how the entire system works, it’s unlikely that many of our staff does. If the system is large, it’s unlikely anyone completely understands, and then our ability to function as a team is incredibly important to be sure that we are building something that works well together.

    To me, the complexity of modern systems emphasizes the need for teamwork and collaboration, the thing that many companies adopting DevOps struggle to emphasize.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.