Voice of the DBA

Category: Editorial

  • Can Data Save the World?

    Years ago I read one of the Freakonomics books. It was an interesting look at how we might examine our world in unexpected ways, using lots of data. While I didn’t always agree with the conclusions of the authors in different areas, I did find the idea of using data to probe and examine for patterns in our world to be fascinating.

    Recently I caught a podcast with the authors, and the opening question caught my eye. It was “can data save the world?”, and I was hooked, needing to stop and listen to the episode. That’s because I think data is the most important asset in the world today, and data is what really powers all the software we have. We need data, and we need to ensure our data has a high level of quality and integrity in order to glean information from all those bits and bytes to get value from software. Certainly software matters, but data, to me, is more important.

    The authors’ position is interesting in that they see data as an important part of analyzing data. The type of data, industry, organization, etc. doesn’t matter, but the data also isn’t necessarily enough to change someone’s mind or convince them of an argument. There needs to be a story as well, which requires a different skill from that needed for analyzing data. I do think that often the way in which we present an analysis can be just as important as the data. Perhaps even more so.

    What I do find interesting in the podcast is that younger companies (and people) are willing to embrace more data driven approaches. I see that often. There have been no shortage of clients in my career that were sure they knew the answer to some question about their business job without referring to any data analysis. They trusted their experience. Even if there was data that might show their conclusions were slightly erroneous, they often didn’t want to change their decision or conclusion.

    Humans are creatures of habit. Even those of us that embrace some change will find that we like change in some parts of our lives, but not others. If someone has had success in their career without using data to support or alter their opinions, it can be hard to get them to change. I wish I had a good method for convincing people to get started using more data, but really I think that the best idea is to find a different person to convince. If you can do so, and produce real results, then you may be able to go back to the first person and show them some evidence.

    The one part of the podcast that I wish turned out different is at the end, where the authors note there isn’t a good way to teach someone how to analyze data and tell a story. They thought about creating their own curriculum, but gave up. It’s too much work, and apparently they weren’t sure they could do a good job.

    They imply this is harder than teaching someone to program, which is disappointing as we have lots of programmers that need help as well. If we can’t teach programmers at any scale, what does that mean for data scientists? The one part I would agree with them on is that data science is a great area to move towards your career if you have the talent.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.3MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A Christmas Bug

    It’s not really a big deal for me, despite the fact that I travel to Europe in a few days for two weeks. At least, I’m fairly confident this won’t be an issue for me. A few of my fellow #sqlfamily might not be so lucky, as they use a different airline.

    Apparently American Airlines has a bug in their scheduling system. It’s not a safety bug, don’t worry about that, but it’s potentially a huge bug for the company in terms of profit and cost. The bug is a part of the system that lets pilots request holiday time off. The system allowed too many pilots to take vacation, and as a result, American Airlines (AA) doesn’t have enough pilots to fly their planes.

    In case you wonder about the scope of this, it’s 15,000+ flights that are affected. AA flies about 6,700 flights a day and are the world’s largest airline by some measure. They are offering pilots 150% of their normal rate, which is apparently the max allowed by contract. I’m sure some pilots will take the offer, but I’m also sure there may be some flight cancellations. After all, pilots have likely made their own plans, which they may not want to change. Apparently some have as the news as of this writing is a few hundred flights.

    The complexity of airline scheduling software has to be quite high, and while this won’t really endanger lives, it will affect the profitability of AA. This is definitely a development mistake, and one that should have been caught in testing. I don’t know how often this software is updated, nor what their process is. Someone suggested this is outsourced software, but that doesn’t matter. We clearly have a software development failure here.

    I certainly think developers are responsible for testing their code and meeting requirements. This includes not assuming the happy path is followed. However, this is also a place where we need (as an industry) to learn to better test the bounds and limits of our software. We need to be sure that we account for not only security, but pushing the limits of use by legitimate users. Software mistakes cost money, and as we continue to use computing more and more, we need to become better at testing our code. Our systems are very complex these days, more complex than a single person can handle. We need better testing everywhere, including the database.

    Please, make an effort to test better next year, in both your application and database. Use larger data sets at some point (both numbers and dates), add automated testing as a part of your development routine, and assume that users will do stupid, silly, and malicious things with inputs.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Not Useless Features

    Some time ago I noticed Adam Machanic had written a post titled The SQL Hall of Shame. In it, Adam notes there are features that Microsoft will introduce in SQL Server and then let die. These are the useless features that aren’t widely used, or even useful for many projects. You can read his list, and agree or disagree.

    I think there certainly are some features that were silly experiments and I wish they had been introduced as experimental, or beta, features early, allowing users to give feedback on whether they’re useful and where they need refinement. I certainly think too many resources were initially wasted on items like MDW and Query Notifications, and far too little investment later. These days I think we’ll see more early releases of items in Azure where users can experiment and provide feedback to help Microsoft decide if more investment is needed. At least, that’s my hope.

    Today I’m wondering if you think there are features in SQL Server that do need more attention. Those features you use, but appear forgotten and you’d hope they are improved in the next version.

    For me, I certainly wish replication would get more attention, tooling, and resilience. I’m not sure if this needs an overhaul, like SQL Server 2005 was, or there just needs to be some reworking of the code, but in today’s distributed world, we need a more reliable way to hook up portions of databases and move data around. Not ETL, not Availabilty Groups, a better replication architecture and implementation.

    There might be other features you use or want, but let me know today. What’s a not useless feature that you want improved.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Certifying Schools

    We’ve debated the value of schools in technology careers plenty of times on this site. I have been torn on the value of schools, and I tend to go back and forth as to whether I’d recommend them for many people. Right now I tend to shy away from schools, mostly because I think the ROI is hard for a college. Working in technology can be a great job, one that pays well, but the cost of a degree seems high. Especially when there are plenty of ways that you can learn the same skills.

    I’m not saying that university makes you a better or worse developer or IT professional. I think that really depends on the individual. Plenty of smart people go through a university curriculum and go on to build great systems and applications. The same is true for self-taught individuals that learn on the job. Ultimately, you need to take responsibility for your own career, your knowledge, and your education, both in getting started and continuing as long as you work in this field.

    However, I do think that your responsibility also extends to researching and learning about the way in which you choose to learn. If that’s a school, be wary of their claims. I ran across this piece about a school that was fined for their marketing. They didn’t quite lie about the results from the curriculum, but they were deceiving. I think everyone should understand that finding a job can be hard in your area, especially when you are getting started. Whether you come out of a school of some sort, or you’ve been building your own projects. Ultimately, you need to convince someone you can do the work they need done in a competent manner at some reasonable price.

    Education has become a business, one where the profit and loss seem to be more important than the results experienced by their graduates. Some of the lack of success by schools is because students don’t make enough of an effort to become good at their craft. I see plenty of evidence of that, even in professional places, where workers aren’t learning to become better at their craft over time. That’s something we can all do, and it’s something I ask you to do on a regular basis throughout your career. However, some of the issues are inherent in modern curriculums.

    As we look for more secure coding practices, ask for more testing, and still see frameworks and platforms continue to advance, I don’t know if schools can keep up? I’d like to think that there are enough professional teachers that could update and alter curriculums on a regular basis, perhaps even every semester, to adapt to the changing workplace. I’d like to see more practical projects, still taught using basic, well known principles in computer science, but using modern platforms. That might change the nature of the professor in colleges, but perhaps groups of colleges should jointly develop curriculums and keep them current over time. There’s still a place for teachers to learn and bring their skills to teaching, but using information that is evolving too rapidly for one person to manage.

    I don’t know if we’ll get there, but regardless, each of you needs to look to improve your skills over time. Don’t expect that what you knew last year will be good enough for next year. Continue to learn, grow, evolve, and question the ways in which you build software and systems.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.