Tag: T-SQL

  • Full-Text Search – Word Breakers and Stemmers

    There are numerous components to the Full-Text Search (FTS) subsystem in SQL Server that help provide efficient, relative answers to queries. Full-text Search is a little complex, and as I’ve been working with the system in an effort to learn more about it, I decided to document how a few things work.

    Word Breakers and stemmers are two interesting parts of the FTS system. They deal with certain language specific operations that help searches work better. They are related as the two items are loaded together if you use third party word breakers and stemmers.

    I haven’t seen third party word breakers, but some work from other products. As an example, here’s a post to load the Greek FTS search word breaker and stemmer from Sharepoint server if you are on SQL Server 2008. It’s included in SQL Server 2012.

    Word Breakers

    Let’s start with word breakers, which do just what the term implies: they break words. It would seem to be obvious that spaces are the word boundaries, and they are in English, but not necessarily in all languages. There are also the issues of characters in Asian languages like Japanese and Chinese. You can’t count spaces as the word boundaries on those languages.

    Word breakers use the lexical rules of the language to determine word boundaries. Essentially they find what the words are, and then further action can be taken in building the FTS index or processing the query.

    The “words” that the word breaker spits out are seen a “tokens” to the FTS index, and each can then be processed by stemmers, stoplists, thesaurus, etc.

    Stemmers

    Stemmers are an interesting part of the full-text search system. They remind me of my high school Latin classes, where we had to conjugate words. A stemmer takes a word and generates inflectional forms, or conjugations. The example in Books Online, and an easy one to understand is “run”. There are various forms of "run” that we would want to consider as equivalent when performing a search. For example, you would want to consider:

    • ran
    • running
    • runs
    • runner (perhaps)

    The same could be said for “lay”. That would generate

    • lie
    • laying
    • lain
    • lays

    This is one of the big advantages over the LIKE predicate in that stemmers can match these forms of the word being searched for. The index would relate all of these to the core, base word.

    More

    The Books Online page for Word Breakers and Stemmers has technical information on checking what’s installed, some troubleshooting, language settings, and some drier documentation on what you can do with word breakers, but not a lot of explanatory detail.

    I used a little of the information in Books Online, and some from Pro Full-Text Search in SQL Server 2008. You can read more, but unfortunately I haven’t found a lot more documentation on the details of how things work.

    You would probably learn more if you write your own Word Breaker and Stemmer, and there is a sample in the Windows SDK to get you started, but that’s beyond what I want to do.

  • T-SQL Tuesday #37 – A Month of Joins

    tsqltuesdayIt’s time once again for T-SQL Tuesday, and this month is hosted by SQLity.net, Sebastian Meine.

    If you want to know more or participate, read the invitation and write your own blog post.

    The topic this month is joins, in honor of Sebastian’s a-join-a-day series. He’s writing about various aspects of joins, and invites us all to do the same thing on this Tuesday.

    Writing Better Joins

    I’m not a T-SQL expert. I can write code, and understand many type of queries, but I’m not one to dazzle others with their code, like Jeff Moden can. Instead, I want to talk about how I’ve learned to ensure that my code makes sense, is understandable, and most importantly, easy to find the mistakes inside.

    I mainly do this by paying attention to the formatting of the code. I would say that once I started to get away from writing code like this, I found bugs easier, and understood the code better:

    WITH [EMP_cte]([BusinessEntityID], [OrganizationNode], [FirstName], [LastName], [RecursionLevel]) -- CTE name and columns
    AS (
    SELECT e.[BusinessEntityID], e.[OrganizationNode], p.[FirstName], p.[LastName], 0 -- Get the initial list of Employees for Manager n
    FROM [HumanResources].[Employee] e INNER JOIN [Person].[Person] p ON p.[BusinessEntityID] = e.[BusinessEntityID]
    WHERE e.[BusinessEntityID] = @BusinessEntityID
    UNION ALL SELECT e.[BusinessEntityID], e.[OrganizationNode], p.[FirstName], p.[LastName], [RecursionLevel] + 1 -- Join recursive member to anchor
    FROM [HumanResources].[Employee] e INNER JOIN [EMP_cte] ON e.[OrganizationNode].GetAncestor(1) = [EMP_cte].[OrganizationNode]
    INNER JOIN [Person].[Person] p ON p.[BusinessEntityID] = e.[BusinessEntityID]
    )
    SELECT [EMP_cte].[RecursionLevel], [EMP_cte].[OrganizationNode].ToString() as [OrganizationNode], p.[FirstName] AS 'ManagerFirstName', p.[LastName] AS 'ManagerLastName',
    [EMP_cte].[BusinessEntityID], [EMP_cte].[FirstName], [EMP_cte].[LastName] -- Outer select from the CTE
    FROM [EMP_cte] INNER JOIN [HumanResources].[Employee] e ON [EMP_cte].[OrganizationNode].GetAncestor(1) = e.[OrganizationNode]
    INNER JOIN [Person].[Person] p ON p.[BusinessEntityID] = e.[BusinessEntityID]
    ORDER BY [RecursionLevel], [EMP_cte].[OrganizationNode].ToString()
    OPTION (MAXRECURSION 25) 

    I often find code in forums, or sent to me and I need to reformat it so that it looks better. I prefer something like this:

    WITH    [EMP_cte] ( [BusinessEntityID], [OrganizationNode], [FirstName], [LastName], [RecursionLevel] )
              -- CTE name and columns
              AS (
                   SELECT
                    e.[BusinessEntityID]
                   ,e.[OrganizationNode]
                   ,p.[FirstName]
                   ,p.[LastName]
                   ,0 -- Get the initial list of Employees for Manager n
                   FROM
                    [HumanResources].[Employee] e
                    INNER JOIN [Person].[Person] p
                        ON p.[BusinessEntityID] = e.[BusinessEntityID]
                   WHERE
                    e.[BusinessEntityID] = @BusinessEntityID
                   UNION ALL
                   SELECT
                    e.[BusinessEntityID]
                   ,e.[OrganizationNode]
                   ,p.[FirstName]
                   ,p.[LastName]
                   ,[RecursionLevel] + 1 -- Join recursive member to anchor
                   FROM
                    [HumanResources].[Employee] e
                    INNER JOIN [EMP_cte]
                        ON e.[OrganizationNode].GetAncestor(1) = [EMP_cte].[OrganizationNode]
                    INNER JOIN [Person].[Person] p
                        ON p.[BusinessEntityID] = e.[BusinessEntityID]
                 )
        SELECT
            [EMP_cte].[RecursionLevel]
        ,   [EMP_cte].[OrganizationNode].ToString() AS [OrganizationNode]
        ,   p.[FirstName] AS 'ManagerFirstName'
        ,   p.[LastName] AS 'ManagerLastName'
        ,   [EMP_cte].[BusinessEntityID]
        ,   [EMP_cte].[FirstName]
        ,   [EMP_cte].[LastName] -- Outer select from the CTE
        FROM
            [EMP_cte]
            INNER JOIN [HumanResources].[Employee] e
                ON [EMP_cte].[OrganizationNode].GetAncestor(1) = e.[OrganizationNode]
            INNER JOIN [Person].[Person] p
                ON p.[BusinessEntityID] = e.[BusinessEntityID]
        ORDER BY
            [RecursionLevel]
        ,   [EMP_cte].[OrganizationNode].ToString()
    OPTION
            ( MAXRECURSION 25 ) 

    That actually came from reformatting the code using SQL Prompt, a product from my employer, Red Gate Software. I’m lucky in that SQL Prompt formats things as I’d prefer them, indenting and getting the JOIN and ON clauses onto separate lines.

    Having code with structure, where you can clearly see the tables being joined, the clauses in use, and not miss any of the columns being selected at a glance is important. When you’re under stress and trying to debug or develop something, it’s easy to miss something that’s happening in the code if it’s not formatted correctly.

    Whether you like commas before or after columns, or you want things indented so that the names of objects line up doesn’t really matter. What’s important is that you and your team agree on a set of formatting, or have tools that reformat things for each developer in a consistent way. You’ll spend less time trying to understand the code and more time building or fixing it, if it has a consistent layout.

  • Unprotected Queries

    SQL Injection
    SQL Injection is a constant problem in many applications.

    Today’s editorial was originally released on Dec 4, 2007. It is being republished as Steve is at the PASS Summit.

    This is absolutely amazing;over half a million database servers have no firewall. How can you put up a database server, SQL Server, Oracle, DB2, even MySQL, without a firewall?

    How can you put any server on the Internet without a firewall? Even most home routers enable a NAT router and basic firewall these days, not allowing connections in by default. In the last 5-6 years, the technology has been widely available, even to uninformed home users, to not deploy any system on the Internet without protection.

    So how do these servers get out there? Are these development systems? Are people opening 1433 so they can test an application or access their remote SQL Servers? That’s what I suspect. Many developers I know are optimists and they don’t expect people to be pinging their servers or accessing their systems in any way other than how it’s designed.

    We’ve been hacked here at SQLServerCentral.com a few times over the years with SQL Injection techniques, but never to my knowledge with an attack directly against our SQL Server. For a long time we did have our SQL Server exposed, but not on 1433. It was on a high, random port that was unused by any other service and we had strong passwords on accounts. It was a convenience service, we had login tracking, and I never saw an unexpected attempt in our logs.

    However if you run a corporate SQL Server and need to stick servers outside your firm’s firewall in some type of DMZ, at least close off port 1433 to anonymous access. Go spend the $100 out of your pocket for a small router that can at least protect your servers with basic NAT and prevent traffic from getting directly to your database server. It might not be the best solution, but it’s better than nothing.

    There’s no excuse these days for putting a server out on the Internet without at least basic NAT protection. Some type of router or firewall should protect every server, and probably every computer, and only allow those services that are really needed. For most servers, this is port 80 and nothing else. Allowing access to SQL Server, RPCs, or any other port that’s not meant for anonymous access, is really stupid.

    And if you can’t figure out a way to securely make your service available to partners or customers, then you should hire someone that can. There are plenty of networking professionals out there that can help you set things up correctly.

    Know your limits, ask for help, and don’t jeopardize your company’s security because of ignorance, pride, or laziness.

    Steve Jones


    The Voice of the DBA

    Wakamojo

    The podcast feeds are now available atsqlservercentral.podshow.comto get better bandwidth and maybe a little more exposure :). Comments are definitely appreciated and wanted. You can get feeds from there.

    Today’s podcast features music by Wakamojo, the Kansas band featuring our very own Adam Angelini, DBA from the heartland and SQLServerCentral.com community member.

    I really appreciate and value feedback on the podcasts. Let us know what you like, don’t like, or even send in ideas for the show. If you’d like to comment, post something here. The boss will be sure to read it.

  • Full Text Search – CONTAINS

    I’ve been working on a new presentation for full text search and brushing up on some of my T-SQL operators. Part of my talk goes into the CONTAINS operator, which is one of the full text search keywords you need to know.

    This operator is only used with full text indexes, so if you have a column that isn’t full-text indexed, it returns an error. If I issue this:

    SELECT *
     FROM dbo.salary
      WHERE CONTAINS(empname, 'Steve')
    
    

    I get this:

    Msg 7601, Level 16, State 2, Line 3

    Cannot use a CONTAINS or FREETEXT predicate on table or indexed view ‘dbo.salary’ because it is not full-text indexed.

    I have a table that is full text indexed and I can issue a basic query, which looks like so many other T-SQL queries.

    SELECT
     name
     FROM authordrafts
     WHERE CONTAINS(*, 'AlwaysOn')
     ;
     go
    

    This returns me all the rows where the columns in the full-text index (I used the star, *), have the term “AlwaysOn” in them. In this case, I’m hitting a FileTable table with lots of whitepapers in there.

    fts_1

    This query is essentially a LIKE search, but it isn’t doing character matching. Instead it is working with those keywords in the full text index. I’ve used a simple search above. I could replace the * with the column, in this case the file_stream column.

    SELECT
     name
     FROM authordrafts
     WHERE CONTAINS(file_stream, 'AlwaysOn')
     ;
     go
    

    I could also use a prefix term and the * wildcard, similar to LIKE.

    SELECT
     name
     FROM authordrafts
     WHERE CONTAINS(file_stream, 'Always*')
     ;
     go
    
    

    These match other rows where “always” is the document, which matches “always” as a standalone word as well as “alwayson” as a term.

    I could also limit the search to particular columns, using parenthesis and commas to separate them out. The BOL example from the CONTAINS page does a nice job of showing this.

    Use AdventureWorks2012;
    GO
    SELECT Name, Color
     FROM Production.Product
     WHERE CONTAINS((Name, Color), 'Red');
    
    

    This is just a very basic look at CONTAINS. In another post, I’ll look at a few more possibilities with this term.