SQLBits V Goes West: Europe’s biggest SQL Server conference in November!

SQLBits V is now open for business: register here . If you are wondering what it is, UK SQL MVPs are arranging the largest SQL Server Conference in Europe, so it’s the event for SQL fans! It takes place on 19th– 21st November, in Newport (just outside Cardiff) in Wales.
If you’re going, please send me an email and let me know so I can be sure to say ‘hi’!

Add to Technorati Favorites

User Input in Business Intelligence projects

Recently, I read Simon Sabin’s blog on this importance of intelligence in BI, which prompted me to revisit the idea of user inclusiveness in the process. I originally posted a blog as part of the Microsoft ‘Industry Insiders‘ series a while ago, and as I continue on my BI journey, I’ve updated my thoughts on the importance of users in BI projects as a result of going through different project experiences.

It can often be easy to underestimate users, and I’ve even heard a perspective that users should be completely insulated from the process because they are not ‘technical’. However, users can sometimes surprise you when ‘water cooler’ conversations reveal interesting things about their business. I produced some geospatial mapping graphs recently using Tableau, and I must say that I was very pleased with the results! Being based in the UK, I had imported a series of UK postcodes along with their corresponding longitude and latitude for use in the graphical representation of the data. A ‘water cooler’ discussion revealed that I didn’t need to do this; in fact, a distinct work stream at a completely separate part of the business involved on delivering GIS software, so this gave me another opportunity to use the best possible data source for the display of the data.

Users can be very creative at generating the data that they need to do their everyday jobs, and it can be useful to harness this creativity to generate proper data warehouses. User creativity can be seen in situations where there are lots of Excel spreadsheets or Access books lurking around the business, individually maintained and updated by users in the absence of a central data source that meets their requirements. Additionally, this creativity can mean that they find workarounds in manipulating their systems to do what they need. When these situations arise, it is important to collude with users to ensure that all the necessary data is in the data warehouse, in the format that they need. Thus, the business teams more likely to adopt the solution, and the business benefits overall because the data is centrally stored, backed up, and has a proper lineage. Don’t get me wrong, I love Excel; but I don’t like the situation where there are lots of Excel spreadsheets floating around, unchecked, and not validated enough to be considered a proper authentic data source.

The initial blog grew out of a discussion I’d had, where a number of people were debating the need to have a business representative or business-focused Project Manager involved in the design stages of a data warehouse. My own view is that business users should have input at every stage of the development. User buy-in is absolutely critical to the success of a business intelligence project, since they are the ultimate consumers of the information. If they don’t like it, they won’t use it – it’s as simple as that! Users need to be involved right from the start. They can save you a lot of additional, unforeseen work since they will tell you directly if anything is missing from the warehouse. I came across an example of this, where the report consumers were not involved at all until the training. During training, the users noted with a great deal of disappointment that quite a few of their key criteria were simply missing from the warehouse. This led to discouragement and a loss of user confidence in the system. The project was ultimately leading to failure; since the users could obtain these key criteria outside of the warehouse, they saw no need to use the new warehouse.


My brief was to turn this situation around to ensure successful delivery of the project. The first step in doing so was ensuring that the users all had their say in the content of the data warehouse. Since these omissions had been discovered during the training phase, the project had been at a mature stage of development at that point of time. Thus, there was a considerable amount of re-work at every stage of the project to ensure that the warehouse contained the business users’ requirements to support their decision making processes. Needless to say, if the business users had been involved at the beginning, there would have been no need for the extensive re-work.
When building a data warehouse, it can be tempting to exclude users and to ultimately dictate a solution to them. This is particularly the case where users have never been exposed to a business intelligence solution and have a long journey ahead of them to understand the difference between a data warehouse, and a database. This situation is compounded by the fact that developers are not always the most gregarious people, and may possibly even be naturally disinclined to speak with customers.

I appreciate that it can be difficult to explain complex concepts, such as cubes, to business users. One customer told me that a ‘grey fog of confusion’ descended on his brain whenever he heard the word ‘cube’! However, I have found that using Microsoft BI products have helped business users to really see, understand and use their data have the ‘Aha!’ moment and to really understand. For example, showing business users the contents of a cube in Excel or in the Analysis Services browser supports their learning journey to the ‘Aha!’ moment.
The users’ journey to understanding and confidence can be supported by actively involving them at different stages. One useful way of doing this is to get the user’s input when creating cubes. I achieved this by listing out the possible hierarchies and attributes in an Excel spreadsheet, asking the business team to organise them to reflect their business structures. Since they knew the business well, they did this easily. When they saw the completed cube, it was a real confidence boost for them to see that they had had direct input into the creation of the cube and that it was a correct translation of the data into their business needs. The business users’ confidence levels were increased by the fact that they could interact with the cube in the comfort zone of Excel. Ultimately, involving the customer at all stages directly helped them to accept the system. Getting business users to accept, understand and interact with a business intelligence solution is often one of the hardest parts of a BI project. However, in my experience as a BI practitioner, the Microsoft BI stack of products makes this easier since it is directly aimed at this group of users who will ultimately determine the success – or otherwise – of the Business Intelligence project. What’s the point of delivering a technically perfect solution if nobody will use it or can understand it? Solutions like
Tableau, XLCubed are an excellent addition to existing solutions since they enhance the analysis of the underlying data, and so they’re also worth considering.

Using Kaizen, Kaikaku to deliver Analysis Services 2008

I have been interested recently in applying different methodologies to delivering Business Intelligence projects, particularly in Analysis Services 2008. In particular, Agile and Kaizen seem to be ‘hot topics’ right now, particularly given the credit crunch. It wasn’t clear to me that this approach would work well in Business Intelligence, so I’m currently exploring this idea as I progress on this journey!

Kaizen is derived from Japanese: kai, meaning change, and zen, meaning good. Kaizen deals with micro-improvements, which gradually increment to make larger changes. These changes are focused on eliminating waste from the process. By keeping progress under constant review, it means that problems are exposed much earlier on in the process. In terms of customer satisfaction and project delivery, Kaizen might work well since it focuses on constantly trying to make things better, and for the customer, that can only be a good thing. In particular, it focuses on looking at the facts, rather than a ‘hunch’; looking at data, rather than following a soundbite, should be the way forward in BI in any case. However, a process of constant change can have an effect on individuals on the team, since they are working under scenarios of constant change. Generally speaking, in my opinion, people are generally opposed to change in principle, so this might not suit everyone.

Related to Kaizen is the concept of Kaikaku, which can be translated as transformation, which is usually rapid in form, and it is also aimed at eliminating waste to create greater value. Kaizen tends to be associated with small, incremental changes which are more gradual in nature. Kaikaku is a deeper, more rapid change; this might mean something more drastic, such as making a decision on the inclusion or exclusion of a data source or Analysis Services 2008 cube. In order to eliminate waste, a Kaikaku way might be to introduce this radical change without excessive documentation to achieve it; instead, a lighter documentation effort, to note what is required and agreed before moving forward, may be more appropriate. This might differentiate these approaches from more traditional Business Intelligence projects, where a great deal of documentation may be produced (and perhaps not read!)

So, when implementing projects, I see it as a process of Kaikaku and Kaizen; a radical improvement might be required first (Kaikaku), which is then followed by a process of incremental improvement (Kaizen). This could be translated into the activities of managing a Microsoft Business Intelligence project; a radical improvement could be made by the inclusion of an Analysis Services 2008 cube, for example, followed by iterative amendments to the Analysis Services 2008 cube itself. Kaizen changes might involve something small, such as renaming calculated members so that the users can navigate their way around a cube more easily: thus reducing motion and wasted value. Thus, used together, these approaches are not inconsistent with the process of producing cubes in Analysis Services, for example. This traditionally requires a lot of ‘behind the scenes’ work in cleansing the data in a format that is comfortable to use in a cube, creating and developing the cube, and so on. These activities could be seen in the Kaikaku perspective rather than a Kaizen perspective: as radical changes rather than iterative changes, but equally aimed at producing value.

It is important to note that Kaizen may offer something more to BI rather than simply a gimmick. For example, the ‘headline’ had been that Scrum meetings are normally taken standing up, and should take no more than twenty minutes. At first, this sounds a bit gimmicky but then, the reason for this tactic comes clear; for many developers, meetings are often a ‘waste’. Given that Kaikaku and Kaizen are about eliminating waste, then it becomes apparent that meetings need to be kept to a minimum in order to eliminate waste as far as possible.

So it seems to me that more traditional approaches of Business Intelligence projects could be translated to work within a Kaizen environment. By detecting problems earlier upstream, this can reduce costs in the longer term since issues are fixed earlier, and have less ramifications. On the other hand, the customer has visibility of problems early on in the process. This can mean that the customer’s initial experiences of the project are not as positive as the perceptions that might be gained through producing a lot of project and requirements documentation in a more traditional Business Intelligence project, for example. One way which this might be tackled is by allowing the users to have early visibility of the data; for example, using data visualisation such as Tableau to see the results of Analysis Services cubes or even just SQL Server views. Thus, although the customer can see issues early on, they also see results earlier as well; this might mitigate the risk around problem perception early on in the process.

To summarise, the various management methodologies around project delivery and software development are useful to know, and might be additional tools to help BI projects go more smoothly. I’m all for that!