Tag: DevOps

  • Creating a Test Pipeline in Azure DevOps using a Windows Agent

    A customer was having some trouble getting started with Azure DevOps (AzDO) and building their database, so we took a step back and decided to create a simple test pipeline, so they could get a feel for how things work and then move on to more complex builds.

    This post looks at a basic pipeline on a Windows agent. I assume you have an Azure DevOps account and have created a project.

    This post is part of a series on Azure DevOps. You can click the link to see other posts.

    Setup

    We decided to start with a very simple test to start, using a task to get a directory listing. That’s a nice simple task, and one you can re-use a lot as you try to configure your pipeline.

    We created a new pipeline in our Azure DevOps project. To do this, we went to the Pipelines section of AzDO and selected “New”.

    2024-08_0004

    We next had a screen that showed choices. If you pick a repo, this wants to build a code-first, YAML pipeline. Great for coding and automation, bad for beginners. Select “Use the classic editor” at the bottom. Don’t worry, you can always move to YAML.

    2024-08_0005

    This takes us to a repository screen. We were working in Azure DevOps, so we can use Azure Repos, but if you used GitHub or some other Git repo on the Internet, you can choose that.

    You can choose to build from other branches, but I tend to start with main, though we often protect main to prevent direct commits. Once you select the repo and branch, click Continue.

    2024-08_0007

    We now get to choose a template. For teaching someone new, I Start with an empty one. Click Empty job to move on.

    2024-08_0008

    This gets us a basic pipeline. To orient you, there are multiple tabs, Tasks, Variables, Triggers, Options, History. We don’t need any of these to get started. The Tasks item is highlighted, so we see our tasks.

    The pipeline is the top level element, and most listed ot the left. The “Get Sources, is next, and is part of the pipeline as it downloads the repo. The Agent Job 1 is the container for a series of steps in our task, like a SQL Agent Job that contains steps. We’ll add different steps below here.

    2024-08_0009

    If you look to the right, you see some edit boxes. These are the top level pipeline settings. The Name is shown at the very top of the image, near the breadcrumb. If I edit this to say “Basic Directory” then this appears at the top.

    2024-08_0010

    The Agent pool is the list of possible agents. This defaults to using the Azure hosted agents from Microsoft, but you can create local agent pools if you want things to run inside your infrastructure on VMs instead. I’ll leave this, and I’ll also leave the Agent spec to Windows 2019. There are other options, which you can see.

    2024-08_0011

    Note, I am not a fan of using Latest agents. Pick a version that reduce debugging issues. Change this to a later version periodically when you are ready.

    If you click on the Agent job 1, you see some different values on the right. You can rename this agent, or specify a different agent from the pipeline. We’ll leave this alone.

    2024-08_0012

    On the line with Agent job 1, there is a + to the right. This is how we add tasks (steps) under our agent. Click this.

    2024-08_0013

    This brings up a list of tasks to the right. There are a lot of ones listed, and you can add more to your account. We’ll live with this list.

    2024-08_0014

    We could scroll or click the various categories (build, utility, test, etc.), but let’s search. Type “cmd” in the search box. We see a limited list of tasks. We’ll click Add next to the Command Line Script.

    2024-08_0015

    Once this chosen, it is added to the left under Agent job 1. Click this line to see the various settings on the right.

    2024-08_0017

    Let’s configure this. First, we’ll change the name to directory listing. Next, let’s delete everything in the script box and type “dir”. We should see this.

    2024-08_0018

    Now click “Save and queue” at the top. We get to enter a save comment and can configure the run. Just leave the defaults.

    Note: I grabbed this shot, and realized I’d accidentally set the agent to MacOS. I reset that before I continued.

    2024-08_0019

    Once we click Save and run, we should go to the screen that has meta data about this pipeline run. It looks like this. Click the line with the clue clock and Agent job 1.

    2024-08_0020

    That should bring up a log listing of the various steps. The listing on the left matches our pipeline steps. Whichever item is selected on the left filters the logs on the right to that info.

    2024-08_0021

    If we click the directory listing step, we see some info logged and then the command being run.

    2024-08_0022

    This directory listing matches what’s in our repo. We can see this easily by clicking on the repo. 2024-08_0024

    When the agent runs, the software runs inside the folder of the repo that is downloaded as part of the pipeline. If you want to get another folder listing, useful sometimes as you configure things, just change the dir command.

    Summary

    This is just a basic look at a pipeline and how to set up a simple task. It’s a useful task, and until you have your pipeline working, I’d leave this task in there. Being able to get visibility into the agent machine is often very helpful.

    I’ll look at a few other options in future posts.

  • SQL Saturday Denver 2024 Resources

    Here are the resources from my talks today.

    Best Practices for Seamless Database Deployments

    Architecting Zero Downtime Deployments

    Some good questions today, which I’ll write a few posts on:

    • What are feature flags
    • How do you handle adding a clustered index
    • How do you communicate with downstream groups on something like a rename

    Probably a few more, but it’s been a long day and I’m tired.

  • T-SQL Tuesday #177–Keeping track of database code


    This month’s invitation is from Mala Mahadevan, who has hosted 5 times. This latest one is one that is near and dear to my heart as I use source control most days and I think it’s important for database code.

    I’ll explain why below, but I’m glad to see this as a topic. We have a wide variety of technical topics being proposed, but not a lot about software engineering as a discipline, which is part of what version control is. Not building features, but better managing your process.

    If you want to host a T-SQL Tuesday, ping me.

    Capturing Database Code

    There are many ways to capture code, but I work for Redgate Software, so I use Flyway. Since I work there, I get a paid version, but I work with customers all the time and see a variety of things. If I didn’t have Redgate tools, I’d likely use something like SQL Compare to capture off database code, since that’s easy. Or SMO through SSMS.

    As for how I work with code, I use Git to store the code. Git is ubiquitous and I rarely find customers without Git. Sometimes the data teams aren’t using it or don’t know how, but that’s why I’ve written some Git articles on getting started.

    I also try to work in branches, with a protected main branch. This means no one can commit code to main, but rather need to commit it elsewhere and use a PR (pull request) to move the code into main. I do this with a lot of customers, helping them understand how to use version control to manage their code.

    For my git work, I primarily work in GitHub in public repos. I’m at https://github.com/way0utwest, where I keep a lot of sample projects for things I work on with customers to demonstrate how to use Redgate tools, or just manage code better.

    There are lots of ways to capture code, format it, and deploy it. However, you should use git and learn to manage your code within a team. I’d also suggest you use Flyway to deploy the code. There is an OSS version, and because it supports many platforms, if your company adopts PostgreSQL or DataBricks, you can still use a similar process to deploy code. Learn it and use it.

    But first, get code into a Git repo.

  • Send Metrics Not Logs

    This is part of a series on observability, a concept taking hold in modern software engineering.

    One of the interesting things I saw in an engineering presentation on Observability from Chik-Fil-A was that they are sometimes bandwidth-constrained at remote sites. In an early version of their platform, they sent logs back to HQ, and their logs used all the available bandwidth, so they were unable to process credit card transactions.

    While most of us don’t deal with lots of remote offices sending data back to a central data warehouse, we do often work in distributed environments, and we may send data to/from a cloud or even employees’ remote offices. Or maybe we send a lot of data between components. Bandwidth is very good in many parts of the world, but it isn’t infinite.

    In the presentation, they talked about a tool, called Vector, that can work with lots of data, slice/dice/aggregate/sample/etc. the data, and then send the results to a sink location. This works like many other ETL tools that have a source and sink, along with various transforms that operate on the data.

    It’s an interesting philosophy to try and send back metrics that might be useful to developers or Operations staff in understanding the performance of their system. By only sending metrics, the load on downstream systems is reduced. This also allows us to store less data and read metrics sooner rather than storing all the data and processing it each time someone needs a metric.

    The flip side of this is that taking this approach means that the consumers of the metrics need to ensure they are getting useful and actionable information. Determining what is needed will be like any development project, something built, iterated, re-tested, and repeated. This might even be an ongoing part of building software as new features and logging are added to your software or system.

    In general, I prefer to have more data over less, but the volumes of logging and instrumentation data have grown dramatically. Some systems are producing more log data than actual data on a daily basis. Like audit data, we likely need to reduce and limit the amount of data stored long-term. However, we want to keep the important data that we find useful.

    I am looking forward to trying out Vector and seeing what’s possible. Having good CLI-based tools that can work with data is becoming more important all the time, especially as more of us move to DevOps flows, coding our systems operation in text, storing it in version control, and deploying on demand.

    If you’ve used Vector, let us know what you think, and if you prefer another tool, share why today.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.