Voice of the DBA

Tag: software development

  • Should You Learn R or Python?

    I’ve been working on various skills over the last few years, trying to solve some simple problems in Python and PowerShell, in addition to T-SQL, to continue to improve my skills. It’s an interesting challenge at times, especially when I need to use new features or functions to which I haven’t had exposure in the past.

    I also work with R lightly, as I need to build some Questions of the Day for SQLServerCentral and I try to alternate Python and R questions every week. This has caused me to dig in and try to learn more about the language and how to manipulate data.

    Recently I was reading an essay from a consultant that works with clients using both R and Python. The piece talks about the differences and how these work to solve business problems. If you don’t want to read the entire thing, the comparison starts with the simple “you need both”, though there is more to the story.

    The most interesting part of this for me was that the author notes that while these are good languages in different ways for data analysis, they aren’t great for data preparation and SQL is still required. Either a database like SQL Server or a platform like Apache Spark. Part of the reason is that R and Python aren’t very efficient, and as we work with more data and larger workloads, efficiency matters.

    The other part of the piece I liked was the note that we need to collaborate and our work needs to be reproducible for others. I love having git for moving code around and keeping configuration files in a repository of some sort. It has certainly helped me take advantage of bits that others have written and easily reproduce their work on my system.

    While some of us work with just SQL, I expect that we will get involved with other parts of projects and may need to help troubleshoot or improve code. I find both of these languages interesting and a nice complement to each other. I’ve also learned there are places where I much prefer one over the other, especially with some of the Advent of Code problems. Some are simple in SQL, but others are much more suited to Python. I haven’t tried them in R, but I bet some of them would be well suited to that environment.

    If you have tried either, or have a preference, let us know. What are the advantages or disadvantages of each when you are working in a business?

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Open APIs

    The US Supreme Court ruling in Oracle v Google was released recently, with Google winning and removing, or reducing, copyright protection for APIs. That means if you use the same method names, parameter names, etc. as some other software, it is unlikely that the vendor for that software can sue you or force you to change. I think that makes sense, as in many cases, someone might accidentally copy an API in something like PowerShell. Already I’ve seen companies and individuals build “Get-ClusterName” calls.

    I don’t know all the ins and outs of this case. It’s complex, and I’ve seen some notes that Google had permission from Sun Microsystems to use Java. After Oracle bought Sun, apparently things changed, but I’m not sure how this agreement was structured or the legalities of how things evolved.

    In any case, I’m glad Google won. I think the specification of an API isn’t something that should be protected and limited. If a company implements a list of method names and parameters, I don’t feel those deserve protection. The actual code that runs should be protected, but the names in the interface shouldn’t be. This might mean that if I interfaced with, say a piece of retail software that had “GetOrder”, “NewOrder”, etc., and I wanted to switch from vendor A to vendor B, I could without modifying my app.

    This prevents lock-in, which isn’t something I want to see enforced in software. In fact, for things like medical software, I’d want the same API for all of them. This is something that I think will encourage interaction of software, and likely reduce costs for many types of COTS systems that allow connections from PowerShell, bash, and other scripting systems. I think this also helps developers that might work across platforms or systems and could use the same code structures to switch from one to the other.

    This also means that vendors implementing APIs need to do a good job with their software. They can’t hold customers hostage with their system over the cost of reworking code the connects with their software. I think that pressure to be better, whether in features, security, or maybe in all ways, is important for software to evolve and improve.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Disclosing the How

    Many of us have written a computer program of some sort. Maybe just a script to complete a task, maybe complex software the clients interact with. For all of us, I would guess that we’ve been asked to do something, or we’ve asked ourselves to build a program, and we use that request as a specification for our code.

    When we do this, how many of us get this right the first time? I’m sure some do, and the simpler or narrower the task, the more likely we get it right. However, we also sometimes make mistakes and realize that our specification wasn’t correct or that we interpreted the specification incorrectly. Even when I write code for myself, sometimes I realize I haven’t described when I need clearly enough, even in my own head.

    That happens when I control the entire process. When someone else is my client, there are plenty of ways in which we can make mistakes, especially in decoding the specification. As we use more and more software to make decisions for us and automate our world, I wonder how often we’ve assumed the software works in a way different than it actually does. With machine learning and artificial intelligence programs, I think there is an even greater chance of a misunderstanding.

    Recently I saw that in New York City there is a proposal to disclose how software is used in hiring decisions. Humans are barred from discriminating by law, but what about software? What if there are bias that can come out in the software based on how someone enters data?

    I think there are certainly potential issues in having software evaluate individuals, though I do think software will do a better job than many people, once we’ve worked out the bugs and we know how we want it to work, and then we can compare that to how the algorithm produces results.

    These are tough issues, and difficult projects. I also think it’s important that we analyze the data that is used in testing software, to be sure that it doesn’t contain its own biases. Ultimately I think having public data sets that are evaluated by many individuals is important. I could see a reference set of data and the expected results, or some range of results being required to ensure software meets the goals we want as a society. This, along with the disclosure of how algorithms work will make our use of software better over time.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • UX Matters

    When I was early in my career, VB v4 was popular, and easy. So many basic apps were built by not only software developers, but people in other departments with an interest in programming. Through VB v6, I found so many apps being built, often with complex and un-intuitive interfaces.

    I was reminded of these apps when I saw a story about a mistake made by Citibank. Apparently an employee was supposed to send interest payments of US$7.8mm to creditors, however, the employee paid $500mm to Revlon, the entire amount of the loan. A mistake, and one for which Citibank is likely liable.

    Apparently there was a UI (user interface) issue. If you read the article, you’ll see a bad user interface, and one that is confusing. If you use this application regularly, it not be a big deal, but staff changes, and having useful UI matters. As I get older, and use more and more software applications, I really see the value in UI being well designed.

    Most of us data professionals tend to work with our tools, some of which are well designed, some not, but often we’re in a code editor, so we might not care as much. Writing code is code, though many of us do appreciate add-ins or extras that make it easy to organize our code.

    I do think for administrators, or anyone that is looking at the status for their systems, good UI helps. Even if you deal with just data and text, having things displayed in an easy to understand format is important. This is also certainly true for most end users of software.

    I’m not a UI/UX (user experience) person, but I know people that I can all for questions or hire for their experience. Over time, I’ve learned that paying attention to the UI/UX can make the difference between customers really enjoying what you’ve done and them feeling that you haven’t really finished your product. I’m sure that Citibank, and many other large orgs, might be re-thinking the emphasis, or lack thereof, they place on this part of software development after hearing about this story.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.