Voice of the DBA

Tag: Database Weekly

  • Destroying Data

    For many of the data professionals out there, our job is to ensure that data is protected. We work to protect the information in our databases in a variety of ways that allow our organization to continue to work in adverse situations. We often schedule backup schemes to ensure we can recover data from losses. We perform maintenance to ensure efficient access to information, and we move data from one system to another so that a wide variety of applications and clients can use it. We spend most of our workday trying to ensure that data is where it needs to be.

    It’s worth considering that there may be times when we are asked to destroy data as well. If hardware is being upgraded, then it might be a part of our job description to ensure that no traces of data exist in old systems. Even if it’s not our job, we might want to ensure that hardware is destroyed or permanently wiped. It’s unlikely to be an issue, but there are numerous stories of someone buying used equipment with old data on it. This is such a low bar of something to check that there isn’t an excuse to avoid removing old data.

    If you aren’t sure what to do, don’t have time, or want independent proof, maybe you want to use a service. There are numerous ones, which makes sense with the growth in legislation requiring stronger data protections as well as the embarrassment from management that wants to outsource their risk to another entity. This might be especially handy for larger organizations that could have multiple types of devices, some of which aren’t as accessible as a hard drive in a server chassis. There are services to ensure wiping of phones, tablets, and more.

    It’s a little funny to me to think of destroying data when most of my career has revolved around protecting data. What I mostly think about here is that too often many organizations have thought about data as an asset of the company to be used as needed, but not necessarily protected throughout its life cycle, and certainly not classified according to it’s sensitivity.

    I hope that with the growing value of data, and regular misuse of information, and the security issues that appear in the news that most companies would start to actively manage their data in the future, from capture to destruction.

    Steve Jones

  • AI Regulators

    With the GDPR being enforced in the European Union, there are plenty of companies that are getting concerned about the potential fines from regulatory authorities if they aren’t complying with the law, or at least, making an attempt. There certainly is leeway for regulators to adjust fines or give warnings if a company is making efforts to comply.

    Other companies might not worry, since there are relatively few regulatory employees and lots of companies. There are lots of complaints coming in, which could easily overwhelms the relatively small staff in each EU country. The problem will likely get worse as more consumers complain about data processing practices.

    There is one way to help amplify the capabilities of the relatively small staffs reviewing complaints. There are researchers in the EU Institute in Florence that are are working with consumer organizations to create AI programs that can help. The initial thrust is to evaluate privacy policies of companies. If there are issues, the software doesn’t assess a fine, but it does alert a human to perform additional checks.

    In one sense, this is exactly what computers can do well. They amplify the capabilities of humans by doing a piece of the work. We can build systems, whether traditional programmed ones or AI based applications, that handle a piece of the work that requires lots of human labor. Once initial evaluations are made, a human can review the work and make more refined judgments.

    The danger, to me, is that humans will be lazy. They’ll start to trust the AI systems as authorities and use less of their own judgment, mostly because it’s just easier. I could see these systems evolve over time to actually train humans involuntarily. New employees would initially trust the AI results, learning from the AI rather than teaching it and constantly evaluating its effectiveness.

    I think AI can really help improve the way that we accomplish work in many ways, but it should be audited and regularly approached with some skepticism by some sort of supervisory group. We should be sure that the goals and results from any AI system continue to be focused on what we want to achieve, and that we transparently define those. Otherwise we might end up having AIs evolve in ways that are counter to the original purpose.

    Steve Jones

     

  • Do you use DAX?

    It’s not that often that I see many posts about DAX. Despite the Data Analysis Expressions being used in Power BI, it seems that relatively few people call this out in their work. Chris Webb writes about it often, and this week is no exception. There’s also a short piece on time series from Philip Seamark and another one on optimizing expression from Marco Russo.

    DAX came out of the Analysis Services world and was the way that Microsoft expected people to analyze data in PowerPivot. That never quite became as popular as Microsoft hoped, but the expressions were folded into Excel and still heavily used in SSAS, but it’s really become the way that you can assemble amazing reports in Power BI. If you’re looking to get started, you might check out the Guy in a Cube, Adam Saxton, and his videos on DAX.

    DAX isn’t a native part of Excel, despite the fact that many of the functions are similar or even have the same name. You can use these expressions in the PowerPivot tab in Excel, which you can easily enable. Once you’ve done this, then the power of complex analysis using DAX is available for you.

    Should you learn DAX? That depends. If you need to build reports, or your users are asking for help with more complex analysis, then this might be a skill that helps your career. Your users will appreciate the way in which they can view data with more complex time series and statistical functions available. This seems especially valuable when looking at data across time, like comparing previous periods to current ones.

    As I look around, there are seemingly endless ways in which you can express your creativity with DAX to build more interesting reports. Some of these techniques are useful, some just fun (or perhaps silly), but a change of pace or a new view can help users become more engaged in your reports. They can be fun to build as well.

    Just remember that ultimately you need to use DAX to solve problems and generate insights. Whether you use Power BI, SSAS, or just PowerPivot, don’t build complex reports just because you can. Make sure you understand how the functions enable you to analyze data and the meanings behind their results.

    If you’re looking to get started, we have a great Stairway Series on DAX that can help you, in addition to watching the Database Weekly newsletter for new resources every week.

    Steve Jones

  • Are you GDPR Ready?

    The GDPR is being enforced as of yesterday. It’s been a law for two years, but fines are now going to be assessed for violations. If you’re like me, you’ve been getting a lot of different types of privacy policy updates, new opt-in requests in email, some notices in email with the burden on you to opt-out, and more. A few people joked about how many they’ve been getting, and certainly I’ve seen no shortage of updates. I’ve even seem some updates to services that don’t allow access to content, such as YouTube, without clicking some accept button.

    It’s interesting to see the various approaches being taken. Last week in London, Redgate held a SQL Privacy Summit and I was honored to host a panel discussion from various industry experts. They had different takes on the GDPR, though most of these people were pro-GDPR, happy that some proper data handling was being enforced. That’s the attitude that many DBAs in know in the EU, as they now have some legal reasoning why we should implement better data handling and security practices.

    However, I’ve also seen that there are different interpretations of how to deal with data. Do you need to ask all customers to opt-in? Can you continue to use data in development and test environments? Can you process data as you already have if you disclose what you’re doing? Is the burden on the company or the data subject? I’m sure we’ll see various decisions and rulings from regulatory authorities across the next year as data subjects complain and companies try to do the minimum level of work.

    The idea of data being somewhat co-owned by a business and an individual is fascinating  I see both sides, and I certainly would like to have some rights over data about me. I definitely think my address, my date of birth, and more should be secured and companies that use my data should have some liability if it’s disclosed. I’m not sure about rights over how it’s used, but that’s certainly a discussion that’s coming.

    I’ve already seen one organization file suit over access to data, because they’re being forced to consent to handling that they disagree with. That is going to be something I watch carefully. Can a company change their terms arbitrarily, ask me to consent in a take-it-or-leave-it fashion, and withhold access to data? Do I own my messages and data stored in services? Is it co-owned?

    Like it or not, the GDPR is forcing us to have some discussions and debates about digital information, which is good.

    We’re mostly ready at Redgate, and certainly continuing to do work. Ultimately, our reading of the GDPR (with some backing from auditors), is that we don’t have to be perfect today, but we need to be making an effort and be able to prove that we are doing so. So on a day after the GDPR went into enforcement, what are you doing? Do you think you’re ready?

    Steve Jones