Voice of the DBA

Tag: Microsoft

  • Archive to the Lake

    Microsoft Fabric was announced at Build in May 2023. This is the next evolution of data warehousing from Microsoft, folding in Synapse and a number of other technologies to create a simpler location for storing and analyzing data. We’ve published some articles on the platform and there’s a great presentation from Mr. Paul Andrew on Linked In. It’s worth listening to, even in the background. Paul has a nice style and a great voice.

    Part of this platform is OneLake. This is a data lake for your org, just one of them, and while it’s able to store data in many formats, it’s mainly optimized to read tabular data in the delta parquet format. This is essentially a compressed text file that allows for some transactional changes to the original data in parquet format.

    I don’t do a lot of work with text files, and I’ve been suspicious of using lots of CSV or other text files in a warehouse environment, which is what a lot of people were advocating a few years ago. Exporting tables into lots of files split on some field, like date, while easy, didn’t seem like the best way to move data for reporting.

    Fabric, however, is optimized for reading delta files. A few presentations I’ve seen from people have advocated for exporting your data from SQL Server (or other platforms) into parquet. While I don’t know there’s a native way to do this (yet), I suspect one is coming. I’ve seen lots of articles (one, two, three, more) about how to do this now. We also have SQL Server able to read these files with external file formats already, so I’m sure we’ll have an easy way to write them soon.

    Many of us struggle with large systems, especially with query performance. We’d love to archive off data, though that’s often impractical. However, in an amazing, wonderful world, maybe we’ll get lots of people doing this, writing about it in the media, and our bosses will start to let us establish an archive in the lake. We could move some data there, especially old, unchanging data. We could delete that from source systems. We could have all our users happy.

    I don’t know if I see lots of data moving to the lake, but I certainly expect lots of it to be copied. If you haven’t thought about archives, data lakes, and text formats, it’s an area that seems to have a lot of growth. Perhaps it’s of interest to you and you might find a new career.

    Or maybe you just hope it gets widely adopted to relieve some pressure on your OLTP server.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • AI Thoughts on the Build Keynote

    If you haven’t seen the Build 2023 keynote, it’s, well, interesting. At a surface level, it’s focused on AI and delivers some demos that many of us might find to be useful and intriguing. I didn’t attend the event (or watch it live), but I did see it a bit later and I made some notes, pausing the 30-minute talk a few times to think about what I’d seen.

    The opening lightly glosses over some of the AI enhancements to development tools and the environments that can be created quickly in GitHub or Azure. Some of us will like those, and maybe they’ll grow on me, but I tend to prefer a development environment on my own hardware, where I have unlimited compute power at a fixed cost. The first big announcement is then showing Copilot technology, essentially some ChatGPT-like abilities, embedded into Windows 11. The demo shows asking Windows where settings are, with the response including buttons to take actions, like setting dark mode. Minorly useful, though I think Windows search works fine. I can type “env” and get the “edit environment variables” in the results. I still have to click through to change things, but this doesn’t seem like a better use of AI, especially if I need to type “set dark mode” instead of “dark”.

    To be fair, the demo has the user asking for ways to adjust the system to get more work done. The suggestions are for dark mode and a focus timer. I knew about the former, but not the latter. Perhaps being able to ask for general assistance with tasks is useful as there are likely lots of features I know nothing about and wouldn’t even think to look for. There is also the option to drop a document, like a PDF, in the chat and Windows asks if the user wants the system to “explain”, “rewrite”, or “summarize” the document. The user clicks summarize and gets a summary of the document.

    There is also a demo with plugins that developers can write for Bing, such as one that uses a legal package to make a change to a document. While lawyers might be worried about their practices (or paralegals about job prospects), I’m more worried about a fundamental problem that many of us data professionals have seen in the past: garbage in, garbage out.

    In this case, if the AI model isn’t well-trained, can I really trust it to summarize a PDF or change a legal document? How can I tell if it’s wrong, or slightly off? In some sense, this reminds me of a high school report. It might summarize some text at an A level, or a D level. It’s up to me to judge that, and I can’t assume the results are good or bad.

    The important thing to keep in mind, however, is that we aren’t in that place with AI. We can’t just trust the AI. We are in a time when AI is an assistant, where it can help us complete a task or get something done a little quicker. We are still responsible. We still have to verify and do some work, but if the Copilot can automatically launch Jira and navigate to a ticket, or attach a document and create a short message to our team, that saves us time. It saves us tedium. It can make our jobs easier. We are still needed, but we don’t do all the heavy lifting.

    I do worry about some of the opportunities for plugins that developers will write strictly to monetize their efforts. If I want a shopping list, I don’t want it to go to Instacart. I want a list I can use. I realize that doesn’t necessarily make Microsoft or a developer any money, but not all the tasks and advances are about profit. Or at least, I hope they all aren’t. I hope some are here to just make the world better. For a quick view of what that could be, watch the keynote closing video.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Sharing the Code

    I don’t know how many of you use the ScriptDOM. I haven’t really used it, but was very impressed with Mala Mahadevan’s Stairway Series on the topic. I have recommended this to a few customers that were looking for some complex code analysis features, which go beyond what SQL Prompt or SQL Fluff do.

    I noticed this week that ScriptDom has been open sourced by Microsoft. The code is available on GitHub, which means you can fork it and change it. Or submit PRs. No idea if Microsoft will take them, but if you write solid, useful code, they might.

    I like that more and more Microsoft is open-sourcing and sharing code that they write. Usually, their repos aren’t for software they sell, but maybe they will change that at some point.

    There are over 5000 repos in their account right now, including one for VSCode, which I use almost every day. While I don’t plan on contributing or even bug-fixing, I bet some of you might. I might contribute to the docs, which I do regularly for the SQL Server docs. There are a lot of changes here, but there are a few marked way0utwest.

    BTW, if you don’t want to do your own PRs, send me a note. I’m happy to edit the docs and submit changes.

    I am a fan of open-source projects, because I do think collaboration is useful in many situations. While I don’t expect many people to actually make changes to software, some will. Some, like me, will correct docs, and others will find issues in the code and report them. All of those efforts help us improve software, and I am all for higher quality software.

    Now if we could get Microsoft to open-source SSMS, maybe a few of you would find ways to improve that application.

    Steve Jones

  • Off to the 2023 Microsoft MVP Summit

    I’m off today to Seattle, Redmond actually, and the Microsoft MVP Summit. This is an annual conference Microsoft has run for their MVPs, allowing them to interact and learn from the developers for various products.

    I am a Microsoft Data Platform MVP, which means that I’ll mostly be seeing sessions and talking with the developers for SQL Server, Azure SQL, Synapse, etc. All the Data Platform stuff.

    The entire event is under NDA, so I can’t discuss anything that happens or what I will learn. At least not this week. A lot of this will come out from under NDA in the next few months, and this event gets me the chance to work with some things before they become public.

    It’s also the chance to get to know the developers and product managers better. I’ve become good friends with some over the years, and I look forward to the chance to not only learn from them, but share a meal or drink at some point this week.

    A quick trip, out today, back Thursday. Just in time to meet the tractor guy Friday.