Voice of the DBA

Category: Editorial

  • DevOps Can Help

    Amazon had a load balancer failure in 2012. The analysis of the event shows that there were missing data in the devices that caused issues. The restore of data from these devices is complex, way more complex with less mature tools than most database platforms. The result was a nearly 10 hour period of time when some customers were experiencing issues.

    In 2016, Gliffy had three days of downtime from a database error. In this case, an admin was updating a replicated system, but failed to sever a link with the primary node. Forgetting this step caused a data removal on the node, which replicated to the secondary nodes. They discovered the restore and replay of logs would take many days due to the size. They hadn’t practiced a DR situation in some time, and were not prepared for the delays.

    Digital Ocean received alerts earlier in 2017 that some services were not functioning. They traced this down to the primary database being deleted. The issue was a process used the wrong credentials for automated testing, and I’m guessing that part of the testing was removing and rebuilding a database. Five hours across the middle of the night resulted in the main database being restored, and a couple more hours to get replicas caught up.

    In the first two cases, there were issues with the deployment of changes to systems, as well as inadequate backup and restore processes. In both of these cases, I would argue that a good DevOps process would have automated the way the code was deployed, including ensuring that steps weren’t forgotten or predeployment backups captured the state of configuration. DevOps includes the “Ops” changes and should ensure that all state information is captured and stored in a VCS. If this had been done, it’s possible that these companies wouldn’t have had these issues.

    In the last case, certainly whoever sets up a system is responsible for using the correct credentials. While it’s easy to say that a developer or tester shouldn’t know the production credentials, but it’s entirely possible that the person that configured the process would have the credentials. I don’t know what to do here, as the first test of this might cause the issue. Maybe a second set of eyes is important for security changes in automated systems? That certainly could be part of your DevOps process. What I’d like here is two factor authentication for all security setup, including for SQL Server.

    DevOps isn’t a prescriptive set of things that someone does. Whenever I talk with people about DevOps and they give reasons why a particular step I’ve demonstrated won’t work for them, I tell them to stop doing that step. After all, the way you implement DevOps doesn’t have to match what I did. We each need to do what works for our environment, and ensure we have some consistency and repeatability in our process. Hopefully preventing downtime from simple mistakes.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Need for DRE

    “Today’s database professionals must be engineers, not administrators”

    That’s a quote from an interview at O’Reilly on Database Reliability Engineering. I don’t want to quibble about whether those of us working in technology are engineers in name or not, but in function, I do think the world is changing to one where we need to do more than be caretakers of systems. We need to be actively involved in ensuring we have very reliable, well-designed systems that use the best practices and patterns of highly available and reliable database systems.

    There is advice to learn and work closer with developers. I like that, and I do think the idea of learning to script and automate tasks is important, perhaps more important all the time, for data professionals. Some of the advice to look towards the quick moving, limited lifetime of many software components today doesn’t make sense. A database, at least some parts of a database, don’t fit within that model. There are valid reasons why a datastore must provide some stability and persistence for an application and won’t fit within a model of multiple systems. Or at least not practically within a model.

    However, there is good advice in the piece to learn more about different types of database services and patterns, and consider other ways of implementing datastores other than a RDBMS. I wouldn’t necessarily abandon a RDBMS for some NoSQL store just because developers think it’s easier, but I would consider whether I actually need to very tight coupling between different parts of the database and complete consistency. I think there are a fair number of domains where a CQRS pattern or some distributed store would work well. The move to microservices might be an enabler for your business if you consider the advantages for OLTP type transactions.

    The flip side of all the power that many stores provide is that many also make reporting and aggregating information more difficult. I would venture that in many cases, a data warehouse (perhaps even a RDBMS-based or columnar store) is necessary, along with the ETL process necessary to keep it up to date. These aren’t simple processes, and take resources to build. For most of us, because we don’t work at extreme scales, I’m not sure it’s worth leaving a relational platform, but I do think that we can learn to evolve and enhance our relational databases faster with DevOps ideas and techniques.

    Steve Jones

     

  • Your Job Description

    In every job I’ve ever held, there’s been some scope creep in my duties. No matter what I think my job is, I end up handling tasks in very different areas. While bartending, I’ve had to cook food a few times. While managing a network, I ended up performing some software development. As a DBA, I’ve assembled network cables. As a manager, I’ve had to arrange and setup dinner banquets for late working employees. Even at SQLServerCentral, I used to have to make sales calls when I expected to be more of a technical worker.

    That’s the sign of a good employee. Someone that pitches in and gets the work done as needed, doing their best if the task is out of their skill area. It’s what I’d expect from most employees, though I’ve never worked with a union. Rules might be different for union employees, though I’d hope most would pitch in in a crisis. I also think this is something I look for when I interview people. I hope to get the feeling from a candidate that they’d be willing to learn to handle tasks outside of their primary position.

    I was thinking about this as I read a post about job descriptions. It talks about the mess that many job postings contain, with a wide variety of requirements and duties. I think many of these descriptions, especially in technology are the result of too many chefs working on the final product. From what I’ve seen, most of the descriptions need a mythical candidate in order to meet all requirements. Companies will never find this person and instead compromise their choice based on who applies. So, always apply if you meet 50% of the requirements.

    This made me wonder about the descriptions that are out there. Some of them are really crazy, but I think some of the issue is there are people that want to make their jobs sound more important than they really are, just as they may do with their job title. Keeping this in mind, what would you say is your job description? Is that what you’d list in a job posting or would you add more or take away some of the items?

    I used to want a more important title, and a more complex job description as I was growing my career. These days I’m less concerned with pumping up my job, but I think that’s part of the cycle of a career. I’m curious where you are in your career and how you might describe what you actually do to others.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Securing the Things

    I would venture to guess that a lot of the data that will be produced in the next few years will come from various “things” that are implemented by companies, users, and governments. We tend to refer to the Internet of Things (IoT) as those computing devices that are small, specialized, and connected computing device that sends data elsewhere. Sensors, special purpose devices, and more make of the IoT spectrum, including many industrial and manufacturing hardware.

    Computers grew by leaps and bounds from approximately 250 worldwide in 1955 to one million in 1980 and 30 million by 1986. In 2015, the world saw 238 million computers sold worldwide. If we add smartphones, we had around 120 million sold in 2007 and 1.4 billion sold in 2015. Tablets add to this, but really, the large market is going to be IoT devices. 15 billion connected in 2015 with an estimate of 75 billion by 2025. I would wager that is going to be a low prediction.

    One of the major concerns with these devices is security, and with good reason. There have been hacks against many of these devices, and few have been designed or sold with security in mind. Quite a few devices assume a strong network perimeter inside of an organization, but history has shown us that these barriers aren’t always secure. What might be more disconcerting is that many of the hacks originate inside of the network boundary, whether from insiders or compromised devices. There’s a nice write up on the need for security with IoT. Microsoft recently proposed that we regulate privacy and security for the IoT devices.

    Many of these devices will generate data that we will use as data professionals. In fact, I expect a tremendous amount of data to be generated. If devices are hacked, we may end up with lots of suspect data. Even devices are secure, they may fail or have intermittent errors. How to we detect errant data and remove it from our data sets? How can we be sure anomalies aren’t just problems or failures in the device (or network)? These will be challenges for us moving forward.

    I don’t know that I think specific regulation is required, but I’d like to see some sort of framework devised. I think encryption should be required, as well as guidelines for the security of connections and data management. Perhaps a list of valid technologies could be used, growing and changing over time, including removing older items that might be outdated. For example, DES shouldn’t be allowed in any new deployments, and it really should be removed from old ones, though with some grace period. That means a regular investment in software development and upgrades that evolves our systems over time. Especially with regard to data security.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.4MB) podcast or subscribe to the feed at iTunes and Libsyn.