Voice of the DBA

Author: way0utwest

  • Why Have a Nom Com?

    I don’t want to get too deeply involved in the complaints and struggles of PASS, but I also want to ensure we have a healthy community. I want to have us continue to grow and prosper as data professionals who work with SQL Server. PASS has a mission, and it does help our user groups, SQL Saturdays, and members thrive as a group.

    I also want to say that I think PASS does lots of good things, and there are good people involved. Props for the good things, and I’ll make it a point to point them out more.

    Now…

    Every year we have a set of candidates for the PASS board, who will help guide the organization through the next few years. We have had board members who did  well, and board members who didn’t. Overall, the organization may move forward, or it may stall, but I haven’t really seen anyone damage PASS through their actions.

    To be clear, I have liked most of the people who have served and consider most of them friends. I’d be happy to sit down and have a drink with most of them, and I think they all had the community’s best interests at heart. No one made decisions to maliciously hurt the organization. I thank them all for their volunteer time and efforts.

    However let’s not confuse appreciation with acceptance. Your best intentions do not imply competence or success. Criticism is not a personal attack, but rather an understanding that the process and system have bugs. If you can’t handle that, don’t serve.

    I’ve wandered a bit from my title, but for a reason. The process for allowing candidates has changed over time and while I think the NomCom served a purpose when it was created, I wonder if that’s the case now. I saw a note recently that candidates need to meet a minimum criteria, and then they are evaluated by the committee.

    If someone passes the minimum criteria, shouldn’t they be on the ballot. That certainly hasn’t been the case, but really, why have a committee? It can serve no other purpose than to influence voting by ranking candidates or removing candidates from the process that the committee doesn’t like. That dislike can be for personal reasons, a non-disclosed issue about the candidate, or some reason they aren’t qualified, whatever that means.

    However if I certify I will travel, if I can speak English, work with SQL Server, and if I have some volunteer record, then stick me on. Well, not me, but anyone else.

    I know some people worry we might have 25 people running for 3 slots, and then oh no, what will we do? How can the voters decide? Listen, if we ever have 25 people running for 3 slots, I think that’s a good day in the community. I’d view that as a win, not as a problem.

    The point I’d make is there are no real decisions being made by the board that require some special training that the board members somehow have. We entrust the running of many civic decisions to people with no real training in some area, and I see no reason why the PASS board is any different. Any reasonably intelligent DBA can listen to information, ask questions, and make a decision.

    In fact, I’d argue that while everyone that has served on the board has worked in the technical field, and probably had some success, they aren’t necessarily qualified to lead a non-profit with $1m+ in revenue. At least not more qualified than you or any other member of the community.

    Even me.

    Let’s grow up a bit. Let’s recognize that the board is a part of the community, and keep it that way. If someone wants to run, meets the criteria, let them run. Anything else smacks of attempts to shape and control the organization in some way.

    Any way, whether good or bad, is unnecessary.

  • Still time to get a SQL Server tattoo

    Not for me, and perhaps not for you, but you can still encourage Jason Strate, Gareth Swanepoel, Ed Watson, and Kristen Benzel to go under the needle. If we raise $25,000 for Doctors without Borders, they’ll get tattoos.

    We’re over the $10,000 mark, so we’re already going to have these events at the PASS Summit

    That alone will make the event memorable, but we’d still like to do more. There are plenty of places around the world that have medical crisis underway, and this is a great charity that sends medical professionals to places that could use the help.

    If this cause speaks to you, then please think about donating for a great cause, and a bit of fun.

  • A SQLServerCentral DR Event

    We had a disaster at SQLServerCentral this past weekend. It wasn’t a big disaster, but it was an event that required a restore of data.

    An Administrative Error

    On Saturday, I was attending SQL Saturday #331 in Denver. In between my sessions, I was prepping a few demos and finished getting ready earlier than expected. Since I had a few minutes, I checked my email and immediately knew we had an issue on the site.

    We’ve been fighting spam for months, slowly tweaking our posting system in the forums. With the start of the American football season, we’ve been getting hundreds of posts every Friday and Saturday. I’ve tried to ensure these posts are removed before our newsletters are generated so that they aren’t filled with advertisements, but it’s been a chore.

    One of the things I can do in the forums is select a series of posts and mass delete (or open, close, hide, etc) the group. For most of the SPAM posts we receive, the posts all occur in the same few minutes and are grouped together. I’ve gotten in the habit of deleting these batches of posts, watching for a legitimate SQL Server post at the end.

    However on this day, one of the our regular threads was buried in the middle of all the SPAM posts. This was THE Thread, the most active and long lived discussion on the site with 45k posts. I inadvertently deleted the post and went on to give my presentation.

    Afterwards, I got a private message from the site, telling me the discussion had been deleted around 2:00pm MST.

    Quick Reaction

    The first thing I did when realizing what had happened was connect to our production database cluster through VPN. When I opened Management Studio, I ran a few queries to verify the discussion had been deleted, and not just “marked for deletion”. Logical deletes exist in many applications, and if this is the case in your own disaster, the last thing you want to do is initiate a database restore.

    In this case, the data was gone, so I immediately tried to initiate a restore. Since over an hour had passed, I didn’t want to restore over top of the current database. Instead, I wanted to restore a copy as a new database, as of 1:45pm or so.

    I selected the proper options, marking the full backup from overnight and the log backups throughout the day. I didn’t have time to worry about using STOPAT and trying to get close to the actual time of data modification, so I choose the last backup I knew would be good. Verifying the database name was a new name, I clicked OK.

    And nothing happened.

    Actually, that’s not true as I got an error. The backup system in use by Rackspace, our provider, doesn’t keep the files available from SSMS. I don’t have rights to work within the restore system, or even request one from Rackspace, so I opened a ticket with Red Gates support for a restore.

    Had this been a situation where the site was down or users were unable to read articles or post, I would have escalated this for immediate action. However since this was a restore of a single thread, and one that exists for entertainment more than education, I chose not to bother our IT staff on a Saturday night or Sunday.

    The Fix

    When I woke up Monday morning, I had a message that the restore had been completed to my specified new database (SSCForumsOld) as of 1:30pm MST on Saturday. I hadn’t asked anyone to do more than this, so this was the extent of actions taken by Red Gate.

    Again, I could have specified actions in more detail, but rather than try to explain to someone in email which thread, and which posts needed to be restored, I decided to handle this myself. After taking my children to school, I sat down and got to work.

    I’ve known the PK of this particular thread since I’ve had to work with in in the past. Connecting from Management Studio to the production instance, I verified I could see the 45k messages in the SSCForumsOld database. I ran the same query on the SQLServerCentralForums database, and validated the data was still missing. I then built a query that would perform an INSERT..SELECT of the parent posts from the restored database to the production system. This took longer than expected, with the table having a number of locks for about 2 minutes. However the post details had been moved.

    That left me with the need to move the actual words of each post, which are stored in a separate table (for some strange reason). Rather than lock up the forums for minutes, I spent time rewriting my next insert to use batches of 1000, and only insert those messages which had not already been moved. Since I could join on PKs, this went quickly, in a few seconds. I next changed my batch size to 5000, and this completed in about 15s.

    That seemed like enough time to run quickly, but also a good batch of data and manually executed this 9 times to move all the data. A quick check on the site showed THE Thread was back and I posted a few notes to let users know.

    Aftermath

    There are still a few issues with points for users that have posted to this topic being incorrect, but that is a lower priority item and I am letting our developers look at it. There is at least one known bug with points, and it’s possible we have another here.

    My personal lesson learned was that I need to move a little more slowly when removing SPAM. I also don’t want to trust myself to do it regularly, so I also spent part of Sunday morning writing a little code and scheduling a job to delete posts with certain patterns of titles that the spammers use. I tried to limit to those obvious subjects so that no legitimate posts are removed.

    We are also escalating some of the issues with SPAM, and with the return of my manager from sabbatical, I’m hoping we can build a few more filters to limit the disruptions in the forums.

    And of course, everyone that posts to THE Thread has my apologies for the mistake.

  • Testing in Production

    The recent Apple keynote announcing the iPhone 6 was very heavily watched. At least, I think it was heavily viewed because I had trouble connecting, numerous pauses and regular stuttering of the video. Despite the fact that the stream was limited to Apple operating systems (iOS/OSX), it appeared that many other people had trouble watching based on the tweets I saw. On top of the scale issues, there were also different language translations overlaid in the audio and crashes of the Safari browser. Overall, the live event was a disappointment to me, though it hasn’t stopped me from upgrading my iPhone.

    The numerous problems that occurred had me wondering if any system wide testing was previously conducted. Was it possible that Apple was actually having their full scale, end to end system test in production? During the live event? I guess it’s possible, though it would be  imprudent and foolish to do so. With all the effort and expense that goes into the “Apple show”, how could a complete system test not be managed?

    Certainly a one time event like a product launch can be hard to simulate. The scale alone is difficult to predict, but certainly there are things that can be simulated. The actual people and applications can be used to record, encode, broadcast, etc. As closely as possible to the conditions of the live event, with the same size, type, and configuration of equipment should be used. The same people who will operate it should participate. We know this as technologists, and most of us would perform testing like this if we could.

    I know that resources are often constrained, and time is precious. However we need to perform some testing prior to production if we are to have confidence that everything will work during a deployment of new bits. The best way to do this is to deploy often, to a variety of environments in the same way we will to production. Execute a variety of tests each time that ensure the application functions as expected. If we find problems, we shouldn’t fix them in that environment. We should start over, fixing the issue in development, and deploying the changes again through test, pre-production, and any other environments we have. Be repeating the process over and over, we can build confidence that our production environment will work as we expect. 

    I hope Apple did this, though the end result has me feeling a bit skeptical.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.9MB) podcast or subscribe to the feed at iTunes and LibSyn. feed