Voice of the DBA

Tag: Database Weekly

  • We Don’t Have Perfect Information

    I was discussing the PASS Summit with someone and they were wondering about building their schedule. Actually, they wanted to pick sessions, but see the choices in a calendar format, but the schedule wasn’t out. My suggestion was to just build the schedule and then sort out conflicts later.
    A few people have mentioned over the years that they want to build a schedule and be ready for the event to maximize their experience and be efficient. I think that’s a common, normal, technical person thing to do. We’re Type-A, we like knowing and having a set schedule.
    The problem is that we don’t have perfect information. Even if the descriptions and abstracts included perfect information about the agendas, what is covered, and to what depth, including demos, we’d still not necessarily assimilate and recognize all that data. We’d think a session on database design covered fourth normal form, even when the text said third normal, or we’d expect that an SSIS data load talk included something on CSVs when the presenter described the talk as being with flag text files.
    We’re human, and that means we have flaws in how we deal with the world. This includes the ways in which we model and analyze data. While we can make mistakes in our analysis, we often may simplify our view of a problem to the point where our analysis is inherently flawed.
    I try to remember this when I write reports from systems that others will use. I won’t have every piece of information that might affect a system, but I try to ensure I have the most important, or significant, data. At least, the data I (and the users) feel is significant. The important thing to remember is that out data is always incomplete, and it’s entirely possible that we have missed a valuable piece of data.
    When that happens, we have to adapt and adjust our systems, just like our conference schedule. We’ll learn more across time and we can use that information to change our system. I know that my view of a conference like the PASS Summit today, or even a week before the event, will be different than what I know, and how I feel, at the event. I should have a plan, but be willing to flex as circumstances change.
    Steve Jones

  • Destroying Data

    For many of the data professionals out there, our job is to ensure that data is protected. We work to protect the information in our databases in a variety of ways that allow our organization to continue to work in adverse situations. We often schedule backup schemes to ensure we can recover data from losses. We perform maintenance to ensure efficient access to information, and we move data from one system to another so that a wide variety of applications and clients can use it. We spend most of our workday trying to ensure that data is where it needs to be.

    It’s worth considering that there may be times when we are asked to destroy data as well. If hardware is being upgraded, then it might be a part of our job description to ensure that no traces of data exist in old systems. Even if it’s not our job, we might want to ensure that hardware is destroyed or permanently wiped. It’s unlikely to be an issue, but there are numerous stories of someone buying used equipment with old data on it. This is such a low bar of something to check that there isn’t an excuse to avoid removing old data.

    If you aren’t sure what to do, don’t have time, or want independent proof, maybe you want to use a service. There are numerous ones, which makes sense with the growth in legislation requiring stronger data protections as well as the embarrassment from management that wants to outsource their risk to another entity. This might be especially handy for larger organizations that could have multiple types of devices, some of which aren’t as accessible as a hard drive in a server chassis. There are services to ensure wiping of phones, tablets, and more.

    It’s a little funny to me to think of destroying data when most of my career has revolved around protecting data. What I mostly think about here is that too often many organizations have thought about data as an asset of the company to be used as needed, but not necessarily protected throughout its life cycle, and certainly not classified according to it’s sensitivity.

    I hope that with the growing value of data, and regular misuse of information, and the security issues that appear in the news that most companies would start to actively manage their data in the future, from capture to destruction.

    Steve Jones

  • AI Regulators

    With the GDPR being enforced in the European Union, there are plenty of companies that are getting concerned about the potential fines from regulatory authorities if they aren’t complying with the law, or at least, making an attempt. There certainly is leeway for regulators to adjust fines or give warnings if a company is making efforts to comply.

    Other companies might not worry, since there are relatively few regulatory employees and lots of companies. There are lots of complaints coming in, which could easily overwhelms the relatively small staff in each EU country. The problem will likely get worse as more consumers complain about data processing practices.

    There is one way to help amplify the capabilities of the relatively small staffs reviewing complaints. There are researchers in the EU Institute in Florence that are are working with consumer organizations to create AI programs that can help. The initial thrust is to evaluate privacy policies of companies. If there are issues, the software doesn’t assess a fine, but it does alert a human to perform additional checks.

    In one sense, this is exactly what computers can do well. They amplify the capabilities of humans by doing a piece of the work. We can build systems, whether traditional programmed ones or AI based applications, that handle a piece of the work that requires lots of human labor. Once initial evaluations are made, a human can review the work and make more refined judgments.

    The danger, to me, is that humans will be lazy. They’ll start to trust the AI systems as authorities and use less of their own judgment, mostly because it’s just easier. I could see these systems evolve over time to actually train humans involuntarily. New employees would initially trust the AI results, learning from the AI rather than teaching it and constantly evaluating its effectiveness.

    I think AI can really help improve the way that we accomplish work in many ways, but it should be audited and regularly approached with some skepticism by some sort of supervisory group. We should be sure that the goals and results from any AI system continue to be focused on what we want to achieve, and that we transparently define those. Otherwise we might end up having AIs evolve in ways that are counter to the original purpose.

    Steve Jones

     

  • Do you use DAX?

    It’s not that often that I see many posts about DAX. Despite the Data Analysis Expressions being used in Power BI, it seems that relatively few people call this out in their work. Chris Webb writes about it often, and this week is no exception. There’s also a short piece on time series from Philip Seamark and another one on optimizing expression from Marco Russo.

    DAX came out of the Analysis Services world and was the way that Microsoft expected people to analyze data in PowerPivot. That never quite became as popular as Microsoft hoped, but the expressions were folded into Excel and still heavily used in SSAS, but it’s really become the way that you can assemble amazing reports in Power BI. If you’re looking to get started, you might check out the Guy in a Cube, Adam Saxton, and his videos on DAX.

    DAX isn’t a native part of Excel, despite the fact that many of the functions are similar or even have the same name. You can use these expressions in the PowerPivot tab in Excel, which you can easily enable. Once you’ve done this, then the power of complex analysis using DAX is available for you.

    Should you learn DAX? That depends. If you need to build reports, or your users are asking for help with more complex analysis, then this might be a skill that helps your career. Your users will appreciate the way in which they can view data with more complex time series and statistical functions available. This seems especially valuable when looking at data across time, like comparing previous periods to current ones.

    As I look around, there are seemingly endless ways in which you can express your creativity with DAX to build more interesting reports. Some of these techniques are useful, some just fun (or perhaps silly), but a change of pace or a new view can help users become more engaged in your reports. They can be fun to build as well.

    Just remember that ultimately you need to use DAX to solve problems and generate insights. Whether you use Power BI, SSAS, or just PowerPivot, don’t build complex reports just because you can. Make sure you understand how the functions enable you to analyze data and the meanings behind their results.

    If you’re looking to get started, we have a great Stairway Series on DAX that can help you, in addition to watching the Database Weekly newsletter for new resources every week.

    Steve Jones