Voice of the DBA

Category: Editorial

  • Big Companies are Improving with DevOps

    One of the comments I’ve often heard from people that work in IT and haven’t adopted DevOps is that the principles and changes required won’t work at their organization. Quite a few people think that only small, new companies, like Flickr and Spotify can use the ideas. Plenty of others look at only high-tech, progressive companies like Amazon and Facebook able to change.

    That’s not true.

    In fact, there are four Fortune 500 companies using DevOps, including a large (though young) bank, Capital One, and another, older one, WestPac. I’m not sure anyone would consider American Airlines or Hertz to be small, agile companies, though certainly they are in highly competitive industries and need every advantage over their competitors that they can get. I suspect that is the driving reason for many of these companies to adopt fast, quick software development. They can’t afford to have an idea take months to implement.

    Those companies aren’t like yours? What about Maersk or Nationwide? Ticketmaster? Maybe Norstrom (and a few more)? I actually had the chance to speak with a number of Nordstrom employees that had taken a POC concept for the mobile group and proved that DevOps has value. From there, almost the entire IT department, hundreds of employees in groups from internal IT to mainframe to web, all have adopted various types of DevOps processes, starting with value stream analysis. Over a few yeasr, they have dramatically transformed their delivery of software. When someone in the business proposes an idea or need, it used to take over 6 months for something to get deployed. That’s down to a couple of weeks, and it’s released in a true, get-something-useful-to-the-customer fashion. This isn’t alpha or beta software, but a basic item that can be used and is then grown and changed according to customer feedback on a daily or weekly basis.

    The transition to DevOps really requires some belief and understanding of the ways in which you can deliver better software, faster. This requires some slow growth, which seems crazy, but the the cultural changes take time, and even the technology tools you choose, require some patience, trust, and experimentation from your technology staff. While it might take months or even a year to get a DevOps process working well and one you’re comfortable with, the gains grow and grow over time.

    Even if you don’t believe in DevOps now, why wouldn’t you try to get someone in management to set up a proof-of-concept and build something. It’s a small investment, that could have huge payback with limited risk.  You’ll learn a lot and can then decide if it helps you delivery value to your customers in a better way. And if you do adopt DevOps, don’t forget to include the database in your process.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Hacked

    Hacked

    By Steve Jones, 2013/01/02

    This editorial was originally published on Jan 2, 2013. It is being rerun as Steve is out of the office.

    I’ve been hacked before. My personal web site has been hacked with a variety of injection and XSS attacks over the years. None too serious, and I’ve had backups that allowed me to fix things fairly easily, especially once I had a copy of Data Compare, which saved me a lot of time. At SQLServerCentral, we’ve been hacked as well, though not in a long time. I think we’ve closed most of the security holes, and I haven’t had any issues to deal with in quite some time.

    However as I was reading a note from Richard Douglas about being hacked, it brought back memories of working at JD Edwards. Richard was hacked at work, on his personal system. At JD Edwards, we were required to lock our workstations at all times when we were not physically in front of them. We also had two accounts: a normal user and a domain admin “privileged” user. As you might expect, there were numerous lapses of people walking to the kitchen or bathroom and forgetting to lock their workstations. It was considered fair game to change settings, send email to our group, even place semi-SFW pictures on someone’s desktop. It was quite embarrassing to be caught, and was much more a an effective security reminder than a reprimand from our boss.

    However there is a serious security problem here. Many of us would use our privileged account all too often, since it was a hassle to log out and back in. The “run as” option didn’t work well for some applications, and we were less secure than we probably should have been. If someone walking by, whether an employee, guest, consultant, or someone else noticed SSMS running, how long would it take them to type:

      sp_addlogin 'joeuser', 'joeuser'
  sp_addrole 'joeuser', sysadmin

    I type quickly and that took me less than 30 seconds. I’m sure even a slow typist could get that entered, and erased, inside of a minute. That might result in a serious security breech, if the system to which you were connected contained HIPAA, PCI, or any identity information. Perhaps even worse these days is the chance someone might attach a USB key logger to your keyboard.

    You might be safe in your environment, but you can never be sure. A little care in ensuring you are not unnecessarily exposing security holes, and making sure that outsiders are always escorted can prevent embarrassing incidents from occurring.

  • Change Approvals

    Part of a developer’s or DBA’s career is getting their work released to some system where others can use the system. Some of us may do this more than others, but we all usually have some role, whether in packaging changes up or actually pushing the button that runs a script or copies files to some live server. This can be an exciting and stressful time, depending on how you feel about the quality of the work.

    Releasing software often isn’t something done in a vacuum. Even in the highly agile, DevOps companies like Amazon and Facebook where developers release code many times a day, there is often some sort of approval process, whether implicit or explicit, before code goes out. Even if it’s just a peer that code reviews something, or a business person that examines a test version and pronounces it correct, another person often weighs in on our changes.

    That’s not always the case, as I know in emergencies, some of us make quick decisions and change or run code that only we have examined. I’d hope that’s the exception, rather than the rule, with most database changes.

    Today I’m curious. Who approves your database changes? Is there a formal process? An informal one? Does the person making the decision even understand the code or do they depend on you to have written and tested solid T-SQL?

    It’s been said that the person closest to the work is often the best person to judge if it should be released, but that’s only partially true. Deploying code is often disruptive. It introduces change, which customers may or may not like. There may be good reasons to release at discrete intervals, rather than whenever the developer things things are working. This may change with heavy use of feature flags or feature toggles, but in general, code releases are interruptions and we want to limit them.

    Unless something is broken, in which case, we often want the change as quick as it can be released. Does that mean we want a different change process? Let us know today.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • I Hate To Send This Email

    I use TrueDelta to report my car status every month. It’s a nice service, allowing car owners to see what experiences others have, and think about which models and years might be a good fit for me. I’ve tracked multiple cars with their service for the last 3-4 years, with reminders from them every quarter to update any repairs I’ve made.

    Recently I got an email from them with this opening: ” We hoped never to have to send this email. A few days ago we learned that TrueDelta has joined the increasingly long list of organizations whose server security has been breached by hackers.” The email went on to note that names and passwords to taken, and that everyone needed to perform a password reset. I applaud them for including “Security breached” in the subject as well as immediately changing everyone’s passwords so old ones wouldn’t work.

    I’ve been hacked at SQLServerCentral, though to our knowledge no data was stolen, merely vandalized. We haven’t ever been able to track suspected data breaches back to SSC, and I hope we never do, but I’m not naive to think that we never will. I hope we don’t, but hackers make determined efforts to gain access to data. At least we are aware of security measures, have a small staff with administrative access, and try to not allow any simple attack vectors.

    Not every company does a great job at securing their data, especially from phishing attacks. There’s a spectrum of how carefully data is protected by organizations, and as we’ve seen from haveibeenpwned.com and plenty of media reports, more and more companies lose data all the time. Some of those companies notify customers (some have to), and I would guess more than a few people have had to send out emails they never expected to send. More of us will dsend those emails in the future, and we should think about that today. Is there something we can do to avoid having to send those notifications?

    There probably isn’t something to ensure it never happens, but we can certainly work towards improving our security. As developers, we shouldn’t have short limits or character choices for passwords. If you wonder why, there’s a great answer at security.stackexchange. We shouldn’t be writing our own authentication schemes, but incorporating code that’s been written, vetted, and reviewed. And make sure we apply patches. Most of the security holes in software are known and patched, but without being deployed. Certainly if new patches become available, we should be able to incorporate them quickly. Above all, learn what SQL Injection is and don’t allow unvetted user input in queries, including those in hidden form fields.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.