Tag: syndicated

  • TechEd 2010 – BI Keynote Part 2

    In the customer demo, a few reports that were built with PowerPivot and Report Builder 3.0, which are helpful to business people in their hospital. The mapping control in RB3 is nice, and the customer said they built the report in minutes.

    There are tremendous features and capabilities in Reporting Services for SQL, and you can have end users handle a lot of their needs. However you will need to provide them guidance so that they aren’t just playing with reports. I can see data pros learning to teach people how to use reports, helping them schedule data refrehses, and find data, as important skills.

    The Alpha Geek Challenge, with Donald Farmer. Always a highlight of keynotes for me to see Donald. He ran this challenge, and they collected cool Powerpivot workbooks from various geeks.

    The most interesting analysis was by Dan Comingore, which used AdventureWorks for an employee moral.

    Dan English won the most interesting data set, with the flight information analyzed from attendees of the last BI conference.

    The third winner for Brian Fosse, for most interesting visualization, showing a circular graph visualization, the radar chart.

    The winners show some very cool capabilities of Excel, but I wouldn’t show them to end users as they might spend too much time actually reformatting their data, much like some people do with Word.

    The overall winner was Brian Fosse, who is actually a business person. That’s the point of these technologies according to Donald and Ted.

    Looking to the Future

    No commitments here, but these are ideas on their mind. Real or vaporware to gauge response? Who knows.

    The cloud is something they’re thinking about, and a good analogy to what is being done here. End users are using data and building applications, and someone else manages the data. That’s what happens in cloud computing. Not a bad analogy, but is it something that we want? I wonder.

    The idea is to provide all capabilities of SQL Server in SQL Azure, including reporting and analytics capabilities. Makes sense, and no timeline, but I suspect we’ll see something over the next year here.

    The consumization of IT. Things that used to be only for the public, search, social media features, etc. are making their way into IT. At least in MS products. I wonder if these are good or bad for business as some of these can end up being their own time sinks.

    Compliance is a big deal. In terms of BI, this means (to some extent) that data quality needs to be high. But what is the correct data? At least for reference data, MDM is designed to fill this need. It’s not a bad solution, and I saw an interesting session on that yesterday.

    Do BI people think of lineage? I haven’t, but it can be something that’s important.

    Dependencies are also an issue, especially as we start to share reports, and build on other reports. I can see this being very important, and potentially a problem in Sharepoint 2010.

    Data volumes are growing, both in size and in variety. More sources, more types, more data in absolute terms. Parallel Data Warehouse will come out this year, to allow 100TB+ for warehouses. CTP2 complete in that area, so they are working to get that finished as a product, and with reference configurations from vendors.

    Project Dallas, a data marketplace, looks cool. You can get public and commercial data sets from here. That’s worth checking out, and using where you can.

    “Stuff in code”, “hot off the developer machine”, glimpses of what’s coming. Amir Netz showing a few things, and he’s also one of the better speakers I’ve enjoyed over the years.

    Amir shows an application authored by a VP at MS, looking at accounts and sales. It has a waterfall chart for examining how the various units, managers, salespeople are doing. Names and $$ changed, but not bad. However the VP wanted something else, so the BI people went to redesign it.

    They changed the report to add the account person’s image, change the color of the background to imply the account status. Interesting idea, and then it’s extended to show everyone’s image. It uses a query, which is something that I think data pros will be writing.

    It’s a fun way to examine data, but the analyst seems to still require the person using the data to play with various drill in/out of data. It would be nice to quickly, and easily do some comparisons, like the developers can do as they check in/check out code. Compare two versions of the code. It would be great to be able to compare to reports, easily, especially at two levels.

    This technology, with the tile maps, should be available later this month, or next month.

    Powerpivot shipped without KPI features, but since it’s built on Analysis Services, they are there. So they have bene working on adding these, and they will be available soon, without requiring MDX skills.

    Amir also showed off a record view for a complex, wide table. Instead of seeing a long row, or a partial row, you see more of a report view with a single row moved into a multi-row record that fits on one screen, with labels for values. I can see this as being useful in some ways.

    There are some new capabilities to actually edit and program Powerpivot sheets in BIDS. There are some more developer oriented extensions, perhaps making it easier to build complex calculations or reports for end users.

    100 million rows aren’t enough sometimes. So in BIDS, Amir shows us a connection to a real SSAS instance. We see 2 billion rows of data being manipulated in Powerpivot (in BIDS). The sorting seems to work just as fast, same for filtering. Amir said that this “is beyond wicked fast. It’s the engine of the devil”

    A larger data set, showing refreshes, and an extrapolated scan rate of 2 trillion rows/sec.

    That is pretty amazing, and I’m sure it is great for most companies, many of whom have much smaller data sets.

  • TechEd 2010 – BI Keynote – Part 1

    The second day’s keynote is by Ted Kummert and focuses on BI.

    This keynote is in one of the smaller auditoriums, which makes sense as only a portion of the attendees are interested in BI. However there are a fair number of BI people in the conference that have a regular pass and aren’t allowed in. Apparently there’s an overflow room, but I found a number of people waiting in line for 5-10 minutes to get in, only to be turned away at the door because they didn’t have the BI, yellow bordered badge.

    Ted runs the business platform division, which includes SQL Server, and the application server technologies.

    Every day businesses are asking questions about their business. That’s true, they are always looking for more insight, more data to support or debunk a position. The dream of BI, making business more efficient, able to more forward, allowing people to make better decisions, that’s something we all want. I know that people at Microsoft want to make that owrk better, and they have, but there is also a need for us in IT to learn enough to effectively apply these technologies.

    The keynote looks back at the last BI conference, before all the BI enhancements were added and released in SQL Server 2008 R2. No committments to future released, but a look forward is coming.

    A slide showing that 20% of the end users have BI tools to use, and 80% do not have tools. It’s a similar story, and MS is looking to try and get more end user BI, self-service BI, to that 80%. BI for Everyone, for 100%, is the vision for MS.

    A nice admission that BI is too hard Too much terminology and technology to learn. I agree with that. BI is hard. So that the idea is to make BI more familiar, using familiar tools. That’s PowerPivot, and I agree with that move. I’ve written about it, and if you haven’t seen it, check it out.

    Collaboration is important, and that includes sharing BI reports, documents, etc. This is primarily with Sharepoint integration from MS.

    There is a note that BI for Everyone means that we have to have managed data. Not data in the wild, in Access, Excel, but in environments managed by IT. So it’s not a look to get fewer DBAs, but rather have DBAs become more managers of data, but allowing end users to access this data with new tools, with less IT involvement in the end user consumption.

    Column oriented, in memory store. That’s what PowerPivot is, and it’s one of the few times that I’ve heard someone actually note that.

    Moving the Excel sheets with Powerpivot into Sharepoint means that IT can manage it. That’s a good thing, so you might want to consider adding some Sharepoint skills. This allows the IT group to back up, and secure, the data that end users are actually compiling, and working with.

    Bi for everyone, sold as: Office 2010, Sharepoint 2010, SQL Server 2010. A great way for MS to sell more licenses, multiple products. Good business strategy, and it means that you’ll need to upgrade Office to get end users working. Tell system admins to get prepped for that, skill and budget-wise.

    Michael Tejedor doing a demo. Sharepoint 2010, searching, and then finding reports stored in Excel sheets. Looking at the data sources, SQL Azure is listed along with many that you expect to see. There are also sources for getting data from SSRS Reports, which is often where someone finds some data they want to scrape out.

    There is an extension to the Excel expression language for Powerpivot, DAX. There are also “social” additions to the collaborative items in Sharepoint. It’s a good marketing move, allowing people to do things like “Tag” or “comment” on a document at a high level, as opposed to a particular cell. you can even “rate” documents, which is something that I’d wonder if people used.

    There are more settings, in addition to things like permissions. You can set data refresh intervals, and workflows.  There are compliance features built in as well, allowing companies to better manage data and ensure it is not being released in violation of regulartory requirements. This is a good check outside of security

    There are good manageability features as well for IT, allowing you to see aggregate activity for the system as well as for individual reports, see the usage of queries, documents, etc. With the farms being built, this should allow you to capacity plan better than in the past.

    You can also see from where data is coming, which can allow you to find out if the sources of data are being overloaded, and perhaps allow you to denormalize, or partition out data sets that are important to end users.

    A customer story from CareGroup Healthcare. It’s a guy that was featured a bit last year at the BI conference at the PASS Summit. I had lunch with him, and he does like Microsoft technology. However that might be because he’s a featured customer, and perhaps he gets some benefits. He’s showing off some Powerpivot reports that I think he showed last year, which allow his end users to build reports that they need without coming to IT.

  • Getting a Page of Results

    The other day I was looking over a couple of articles on paging, looking to see if I could learn something new in T-SQL. I’ve implemented some SQL2000 era paging systems, none of which performed wonderfully, so I checked out Jacob Sebastian’s basic Server Side Paging and Paul White’s Optimizing Paging Part 1.

    I’ve done systems similar to Jacob’s, but he had an interesting use of the OVER clause in his code. He had this code:

    ;WITH emp AS (
      SELECT 
        CASE 
          WHEN @SortOrder = 'Title' THEN ROW_NUMBER()OVER (ORDER BY Title) 
          WHEN @SortOrder = 'HireDate' THEN ROW_NUMBER()OVER (ORDER BY HireDate) 
          WHEN @SortOrder = 'City' THEN ROW_NUMBER()OVER (ORDER BY City) 
              -- In all other cases, assume that @SortOrder = 'LastName' 
          ELSE ROW_NUMBER()OVER (ORDER BY LastName) 
         END AS RecID,   , LastName
       , FirstName
       , Title
       , HireDate
       , City
       , Country
       , PostalCode
     FROM employees

    This is a great solution in SQL Server 2005. It’s much different than what I had done in SQL 2000, where I’d typically approach the problem by using the sort key to get the next page.

    So say I had this data in a table (Customers):

    CustomerID    Customer

    ———–   ————

    1             Jones

    2             Smith

    3             Johnson

    4             Allen

    5             Gates     

    Then suppose I wanted page 1, 2 results per page, ordered by Customer. I would want to see “Allen, Gates” on page 1. I’d use this code.

    select top 2 Customer

    from Customers

    Order By Customer

    If I wanted page 2, I’d go here:

    select top 2 Customer

    from Customers

    where Customer > ‘Gates’

    Order By Customer

    And this would get me “Johnson” and “Jones” since the WHERE clause would reset results. If you have control of the application code, and you can pass in the previous values, you can easily build pages like this.

    If you switch orders, say to the CustomerID, then you can easily pass that in as well, and use that for ordering.

    There is a downside, however. Any ideas? I’ll post that in my next look at paging.

  • PCI and Encryption

    It surprises me how often I see people posting questions about what type of encryption to implement for credit card data. If you are processing credit cards yourself, and storing the data, you need to comply with the PIC regulations that exist. Here’s a good place to get started: https://www.pcisecuritystandards.org/.
    Actually the place you need to start is with your bank or processing company. They should be able to guide you in what requirements need to be met for safe data storage.
    However if you’re running some service and perhaps trying to store a credit card for a customer to make it easy to charge them over and over, that doesn’t mean you don’t need to comply. You are holding financial information about a customer and if something happens to the data, you’re at fault. Your company could be liable, and possibly even you personally if you make the recommendation to build something yourself.
    Good security isn’t magical, and it isn’t secret. It involves you using well known algorithms, protecting the keys, and following best practices. There are some great encryption technologies in SQL Server 2005/SQL Server 2008, but don’t just implement them without learning a few things about what best practices are and how these technologies work.