I’m excited to be attending ASG Technology’s Evolve19 event in Dallas, Texas on October 21 – 23.
I’m particularly interested in learning more about metadata management. As I’ve written previously, I believe that metadata management is often overlooked in data science. It’s an important part of obtaining full value from data, and it should be part of the data maturity model. I’m looking forward to seeing how businesses can securely use data to make better business decisions faster, and compliance teams can reduce data related risks and resolve any data problems.
It’s ironic that there are so many technical tools ways to communicate, there are still problems in achieving effective internal communication in companies. This is particularly evidence in relationships between Finance and IT. In this topic, the issues will be explored further The issue with having so many technical tools is that the quantity of communication may increase, so this can give the appearance that communication is going well. However, the quantity of communication does not always reflect the quality of communication. IT and Finance concepts are difficult for non-specialists to understand, and messages may not be received as they were intended when working across different teams. It is hard for each department to see through the lens of the other department, and this issue can be further exacerbated if there is a culture of blame in place.
What is Data Debt?
Data debt is like technical debt, only visible in bad data, poor data quality, and an underuse of data-oriented technology to help make better decisions. It is the overhang of projects, whether they went well or poorly.
Technical debt is the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer. It also applies to data debt; the cost of ignoring the data by choosing the quick or easy solution earlier at a previous point, which now takes longer to resolve.
At some point, however, we need to pay back data debt if we are to succeed in making better decisions.
Together, the issues all add up to make data debt, which is often a hot mess which nobody wants to resolve. Businesses can develop the Bystander stance; it is someone else’s problem.
What can you do about data debt?
Finance and IT need one another so that the whole organization can be successful, but there can be issues in communication and priorities that cause divisions. There can be seen very clearly in the sphere of business intelligence, where the business and IT need to work together to produce an essential component of the function of the organization: reporting, data and information flow.
Business Intelligence projects can inherit data debt from previous projects, which means that projects do not start from the starting line. Instead, the projects can start with a deficit before any progress can be made. In the long term, technical debt can exacerbate issues that exist between the Finance and IT departments. Take, for example, the situation where Finance don’t believe that their requirements and priorities are understood properly and that their voice isn’t being ‘heard’ in the IT strategy. From the Finance lens, this situation could be shown where IT won’t respond to support calls quickly enough, for example, or won’t allow the purchase of some new software. On the other hand, IT can hold the viewpoint that Finance are using uncontrolled data sources, such as Excel or Google Sheets spreadsheets. If something goes wrong with these artifacts, Finance team members involve IT to help fix something that they do not own, and inherit the technical debt for something that they did not create. This means that IT can end up owning ‘cottage industry’ artifacts, and this can create frustration. The ‘cottage industry’ artifacts arise partly as a result of Finance not getting the data or the software that they need from IT, so team members create their own data sources. The cycle of frustration between IT and Finance can continue without resolution, and since the pain isn’t owned by either department, there is no real ownership of resolving the issues. Instead, it simply gets absorbed into the business processes and it becomes part of everyday work. The real cost is never identified since it is a productivity cost, and it isn’t measured as stringently as direct costs that people can see or hold.
Conflicts between departments can arise out of differing priorities, but the overall vision should be the same for each department within the organization. As IT pushes forward into new projects, technical debt can be pushed to the side, simply to accumulate. For Finance, this issue can develop into unresolved conflict and ongoing productivity issues, as well as concerns over data governance and data veracity. The reality is that IT and Finance need to work together for the success of the whole organization. There are risks if the departments cannot work together. For example, if the departments cannot work together then this could put security at risk, for example, due to a failure to communicate successfully. Ultimately, this could land the company in court. Other consequences include an adversarial working environment which can make the company unable to recruit and retain key talent. In turn, this can impact technical debt as people leave without resolving it, and it just continues to accumulate. Although it’s hard to quantify, the difficulties in IT and Finance working together will cost the organization money, whether it is calculated in terms of lost productivity, or direct costs such as poor software selection. By contrast, if the teams can work together and work better, then both departments can increase their contribution to the success of the whole organization.
Blueprint for paying back data debt
How can the teams move forward from the impasse? We need to put the business back into business intelligence and move forward, working more collaboratively. Fundamentally, this is an issue of how information can be tailored to different audiences who are specialists in their own field, and need to work together for a specific outcome that is seen across the organization.
Both Finance and IT departments need to work together to improve understanding and collaboration between each department. Brown bag lunch sessions, where team members present bitesize summaries of relevant concepts to both teams, can provide a good experience of working together for all team members. This may also help to break down stereotypes of the characterization of each department. Further, it can help team members to enlarge their tribe to include people from other teams, as well as their own team.
Finance and IT leaders need to work together and set an example to the rest of the organization. Personal conflicts between department managers is not acceptable, and the dispute will affect processes and practices within the company and disrupt the entire workflow.
Set up an informal wiki or blog, where team members can add every day. This blog can be used to identify the data and the information each department needs every day. By identifying key information, the wiki can be used to help understand each department better, and to share this crucial information in a straightforward, recorded way. This will help the teams to see the impact that they can have on each other, and identify and prioritize effort to maximise benefit.
Creating a community of excellence as a cross-department initiative that has representatives from each department working on joint initiatives, with progress reports and clear objectives. This will take some time to set up, and there may be resistance due to previous projects that have failed or not delivered.
Inherit technical and data debt. Every new IT project should inherit some technical debt from previous It work. This could involve data cleansing, or improved reporting, or the review of a support contract of cloud vendors.
Requirements can be hard to write and accept. One way to increase teamwork is to accept that prototyping is a valid way of developing and articulating what Finance need. Further, it means that IT can take on board It’s considerations, such as security, governance and upgrades. When developing, IT will want to prototype; at first, the efforts will seem awry but constant communication and feedback will be a good investment of time.
Conclusion
It isn’t going to be easy, but it is achievable if the example and direction is set at the Csuite level. Communication is key.
And finally, you can see the end of the data debt.
Customers want their data good, fast, cheap and easy. A tall order, right?
One of the biggest challenges that I see with data warehousing in the cloud is that customers are concerned with cost. I was interested to see the Gigaom report on the topic of Data Warehousing in the cloud, which contained a number of benchmarks, including cost.
The study by GigaOm showed that Azure SQL Data Warehouse is now outperforming the competition up to a whopping 14x times at up to 17 times cheaper than the competition, namely, Google BigQuery and AWS Redshift. This is an incredible achievement and the Azure team should be proud!
As part of my work in Business Intelligence, often, this involves a move to the cloud by default. Simply put, customers want quick Business Intelligence and they don’t want to spend time or effort in looking after kit. They want to delegate the responsibility. This means that cost is a key differentiator, since they want their data, good, fast and cheap. I’m glad to see that the Azure SQL Datawarehouse is competing on cost and performance since customers do want their data good, fast, cheap and easy.
Customers also want their data easy and this is where Power BI comes in. If a customer wants to use Power BI, I generally recommend that they put their data into Azure so that the data is traversing the Azure network. This means that the customer is not paying to extract or access their data from another cloud system and then put it into Power BI.
I’m excited about Power BI dataflows, since I believe it’s a great way forward for companies to sort out the issue that nobody likes to talk about: cleaning data. I’m passionate about this topic since I believe in inheriting technical debt as much as I can, so I leave the organization’s technical debt in a better place than I found it. I’m taking the long view when I de
I tried to connect my Azure Data Lake gen 2 storage to Power BI, and I ran into this error message:
”Your Azure storage account must be in the same Azure Active Directory tenant as your Power BI tenant.’
I checked, and my Azure storage account was definitely in the same Azure Active Directory tenant as Power BI. So I was confused. What a pesky error! So I decided to investigate.
It turns out that I had missed a step when setting up the connectivity between Azure Data Lake gen 2 storage account and Power BI. I had missed assigning the Reader role to the Power BI service on the storage account.
Microsoft’s instructions are here and make sure you follow them to the letter. I missed a step because I rushed through it, and then spent time trying to figure it out, so it was my fault since I scrolled past a step. I’m recording this issue here in case anyone else gets this error message, and wonders what’s going on.
December is a great time for setting personal and business goals for next year. I’d hoped for some downtime at this time of year, but I’ve picked up a crucial data science project which needs to be delivered over Christmas so there isn’t much downtime. I’m also starting to write another book, which will be my fourth published book, and that will require focus. So I will be working every day except Christmas Day, and that includes weekends as usual.
As part of my MBA program, I’ve done lots of strategic analysis modelling for businesses, starting with the default SWOT and a PESTLE analysis through to different models: Porter, Business Canvas Model, SOAR, and so on. I am not going to share here. The reality is, with Brexit, it’s more difficult to plan. Markets, and businesses, do not like uncertainty. Businesses, like countries, benefit in the longer term from open trade. Increasingly, global competition will occur on a more level playing field but there has to be open access for that to occur. Brexit is a plan for the right by the right, and it is stopping the open access that we enjoyed previously.
For what it is worth, here is my take on Brexit: I believe that May’s Remainy Brexit plan will never get through Parliament because it pretty much keeps us in the Customs Union. This means that the People’s Vote option will become increasingly likely. We are no longer talking about the big red bus lie and we see the chaos. If there is another vote, I hope that we will stay in the EU as we are now. So I am going to assume that all other things remain equal.
More than talking a good game
I was contacted me recently, by a former colleague who explained to me that someone had spoken to them about me. This ‘someone’ had told them that I was too small to deliver a project, that I didn’t know what I was talking about but I talked a ‘good game’, and that I would never deliver on time or on budget. The former colleague was so concerned about what they’d heard, that they invited me to lunch to explain the accusations which were stated behind my back. I was so horrified and fairly mystified; I had never worked with this individual and I didn’t really know them. I just asked for specific examples where I had just ‘talked a good game’ and not delivered on time or on budget, because I had no idea where they got their evidence from. It was all fairly alarming but of course the ‘someone’ hadn’t given any basis for their assertions at all.
In any case, the former colleague simply said that they would continue to work with me and they enjoyed working with me in the past; they had their own evidence to go on, and it is a mark of our good relationship that they talked to me about it at all. I was hugely relieved but I do wonder how often these tactics work, so I’ve decided to explain myself here and discuss this in terms of my goals.
So if you’re here because you’ve heard something similar, please let me know and we can have an honest discussion. I’ve got over twenty years as a Consultant, and I have postgraduate degrees plus industry experience in delivering Artificial Intelligence solutions. I’m writing my fourth book in Business Intelligence and I’ve edited another two. I have presented to five thousand people in the past month. So I do talk but I also deliver.
How was 2018 for you?
Data Relish Ltd was originally set up as Copper Blue Consulting, and was rebranded three years ago in order to make the offerings more clear. I have employed people in the past, and I had a business partner for a few years but that relationship is no longer in place. I do work for myself but note the following points:
Contracts and Partnerships
Current Status for 2018: I have partners who help me to deliver effectively. I have contracts in place with large partner organizations which allow me to scale; I have partnerships for 24 by 7 support, for example. Partnerships allow me access to sales, marketing and legal backup when I need it. Some of these partnerships are with Azure resellers and organizations, and I do not directly sell Azure so that my customers can benefit from the support that my partner organizations offer.
I am not releasing partner names here; I have a degree of concern, originating from the conversation with my customer, that there is a strong need for confidentiality on the basis that I’ve had the aforementioned stunt pulled on me. I am not giving any names away so that I inadvertently give detractors the opportunity to repeat their performance.
Customers
Current Status for 2018: 50% repeat customers with 50% new customers generated by word–of-mouth from previous customers
I am able to provide references on request from organizations of different sizes, verticals and global impact. Some of my customers are multi-billion turnover, right down to startups. I don’t make a huge effort to market myself because all of my work comes from word-of-mouth. Since I am so dependent on word-of-mouth marketing, it can mean I’m susceptible to stunts like the one mentioned above since I don’t have the flashy marketing that might give some people comfort. I have started to look more closely at marketing and I have set up a partnership with a small consulting firm to help me. It works well since they give me help, but I also pass on leads that I receive, and we have jointly delivered projects on occasion.
Business Community
Current Status for 2018: I started more business networking last year but it tailed off in February. Reason: I was not picking the right events well enough and I got disheartened.
I have started to attend Chamber of Commerce events again, and I met with David Gauke MP (Minister for Justice) and Richard Harrington MP (Minister of Business and Industry) this year. I am interested in learning from the local business leaders here in Hertfordshire, and also looking at ways that I can feed back into the business community. Networking is important and it builds trust. Additionally, being part of the local Chamber groups is offering the route of having our voices heard jointly in Government, and I was surprised to learn that our local MPs have regular and frequent meetings with Chamber representatives. Business is crucial to the success of society by keeping cashflow moving, providing jobs and stability and homes. I advise businesses strategically because I believe in putting the business back into business intelligence, and at the front-and-centre of artificial intelligence efforts.
Charity
I was a Data Ambassador for DataKind this year. Wow! What a rewarding, creative experience and it was incredibly satisfying to work with people who love data and want to do good with data science to help charities.
Current Status for 2018: I got involved with DataKind. You know who you are and I love you all; you have brought such knowledge, fun and great company to my life and you shine your light far out. You cancel out the darkness where people behave as in the anecdote above. Thank you for your healing.
So what are my Goals?
Here is the list of goals for 2019:
Contracts and Partnerships
Goals for 2019: I have partners who help me to deliver effectively and I will continue to honour the trust that they have put in me.
In 2019, I do not intend to employ people now since it takes me away from doing things that I love. I may do so in 2020 but it is not the right thing for me at this time. I have previously lost sleep over making people redundant and I do not want that burden again. I work on projects, often, that are simply too big for me to run out and hire a whole team of people and it would be madness to try. How could I even support things like 24 hour support? These are big projects and I need help quickly. Customers need help with AI, BI and Data Science leadership. So it works well.
Customers
I have done extremely well this year and I am proud of my customer list. Through partnerships and on my own, I have punched well above my diminuitive size to have a customer list where my customers have billions and billions in turnover and it is a mind-boggling amount when I add up the turnover amount of all these customers combined.
I do help them to be even more successful and I do help them make technology choices as part of the vision and strategy that I help them to formulate and devise.
Goal: More repeat custom and generate new customers. 50%/50% split between regular customers and new customers is a good split, and it worked well for me this year. As I lead customers to success, they drop off my customer list because they don’t need me any more; but that’s what I want! Self-sufficient, satisfied customers who are enabled and empowered to move on, and who recommend me to their colleagues and community.
Business Community
I am going to set myself the target of attending more business events this year. I am currently researching the business events that have value, and the ones that do not offer value to me.
Goal: I intend to re-ignite my business networking in a more targeted way. I have been pleased to meet leaders that I can learn from, and i hope to contribute more.
Charity
I was a Data Ambassador for DataKind this year. Wow! What a rewarding, creative experience and it was incredibly satisfying to work with people who love data and want to do good with data science to help charities.
Goal for 2019: More of the same. I get enormous richness from philanthropic efforts and I want to help people.
I wish you all the best for 2019! And don’t let anyone steal your pixie dust.
I was very pleased to appear on the Izenda website along with five other Business Intelligence experts, discussing the past, present and future of self-service Business Intelligence. I was delighted and honoured to appear with luminaries such as Wayne Eckerson, John Myers, Kevin Smith, Rich Ghiossi, and Ron Powell.
Self-service Business Intelligence is a much larger topic than you might think, and it’s clear that some organizations who market themselves as ‘self-service’ aren’t really meeting the criteria. I recommend that you head over to the post in order to read it all. I’m interested in the idea of self-service analytics as well as self-service business intelligence, and I do think that will become increasingly relevant as the industry matures.
Thank you to Izenda for having me along. Please let me know what you think; I look forward to your comments.
I’m holding a series of training courses around the UK, more details will be published. In the first instance, on 15th September, I’ll be holding a day-long practical workshop on Working with Business Data for Busy Executives in SME Organisations in Hertfordshire, England. The cost will be £100 pounds plus VAT, food and workshop materials included, and you can also network and share experiences with other attendees who will also be running businesses, like you.
I don’t believe in a ‘stack ’em high’ approach, which doesn’t give a pleasant experience for learning. So, classes will be restricted to 12 people only, unless otherwise stated. This means that you will get a good amount of attention.
I’m doing the Executive MBA at the University of Hertfordshire Business School, I’ve also been a NED (Non Executive Director) for PASS, who are based in the United States. As I’ve been spending time leading organisations, I’m keen to share this knowledge and expertise with the community from a data-driven, data leader perspective. The following blog post will give you a flavor of the workshop. along with some of my thoughts on Microsoft Data Insights Summit. If you have any questions, please pop them in the comments box and I’ll read them from there.
Here are my slides from Microsoft Data Insights Summary, combined with some of the slides from the keynote by James Phillips, held in June 2017.
For those of you who know me, you’ll know that I have extensive experience in Tableau as well as Power BI. However, most of my consulting data visualisation is in Power BI suite of products. Why is that?
Tableau is wonderful at data visualisation, as is Power BI, of course. However, for enterprise customers, where I’m building a data warehouse, I prefer having analytics closer to the data source, perhaps in a data warehouse or data lake. I like to think about the overall business intelligence architecture. Tableau is superb at data visualisation and it also cleans and integrates data, but to a much lesser extend, which is why they partner so well with Alteryx. I don’t like cleaning data or doing repeatable analytics so close to the end reporting layer and business people seem to want to do it there, without thinking of issues such as robustness, repeat-ability and longevity in the analytical formula that they are creating. I prefer to hand off clean data and analytical formula to the reporting tool as far as possible.
I’m not thinking about Business Intelligence in terms of a spot solution for data visualisation or reporting, for example. I’m looking at the whole canvas. I prefer to clean the data and have it all fixed closer to the source, so that I can get the same number for the same report, regardless of the reporting technology that I use. With Power BI, I can stay within the Microsoft playpen of technologies. I do note however that Tableau Server is in Azure and if you are looking at analytics, that’s another option so that the analytics formula isn’t contained in disparate workbooks. Instead, they are published to Tableau Server and people can should download their workbooks there, for ‘one version of the truth’.
As an external consultant, I work with Power BI because I think it has an astonishing reach technically as well as geographically. Some of my customers are global and I really need the certainty of global resiliency.. Gone are the days when Microsoft had a lot of disparate reporting technologies that didn’t talk to one another very well and we had lots of different interfaces that used to overlap. Customers got really confused about what to use. For example, do you put your KPIs in Analysis Services, or in Reporting Services? Now:
The answer is always Power BI! Take a look:
Apart from Data Visualization, what is Power BI useful for?
Power BI is particularly useful for:
businesses that are acquiring other businesses and they need somewhere to put the data, and keep the business running in the meantime
cost savings
GDPR – if you don’t know what this is, you need to contact me to find out more. Microsoft are in the forefront of working with customers to make sure that they are compliant.
Do I still see Tableau?
Yes – some of my customers don’t need public cloud because they pop up their own data centres if and when and where they need them. So, for them, they tend to stick with what they know, and what works for them.
What Business Intelligence tools do I see less of?
I see Qlikview less and less, as customers look to align their reporting and they can replicate their Qlik scripts in SQL Server and SSRS.
I also don’t see Pyramid Analytics appearing much, and I don’t get asked often about them. According to the Gartner report, 2017 may represent a critical period for the company and, rightly or wrongly, the Gartner Magic Quadrant does carry enormous weight when customers are looking for solutions. With many solutions, customers don’t use the full range of features contained in any technical solution, and Pyramid are going to have to work hard to explain how they compare / compete with the Power BI on-premise solution, which is going to go from strength to strength.
However, for others, particularly in the SME market, the Azure offering is extremely compelling. Power BI and Azure together mean that you can focus on the business, rather than working on the technology to support the business. I can also see that more and more data is going into the cloud, and I am part of projects where I am doing exactly that – cloud business intelligence. Cloud Business Intelligence is a real growth offering for me and I plan to keep being ahead of the curve.
Are people using Power BI or is it simply good Microsoft Marketing?
People are using it, yes. Here are the numbers, produced by James Phillips during the Power BI Keynote:
Power BI and the C-Suite
Given it’s reach within the organisation, Power BI can reach the C-suite level as well as the rest of us, in the organisation. Before continuing, it’s probably worth reading about linear vs exponential business growth models e.g. HBR.
You can watch the video below, or read on for some of the headlines:
Here are some headlines:
Gross and Net Profit
Net profit
Progress Towards Targets
Revenues and revenue growth rate
Expenses
Employee Engagement
Let’s get started!
Gross and Net Profit
Why do CEOs care? As part of the Digital Transformation process, the CEO must develop a guiding philosophy about how he or she can best add value whilst showing ongoing strategic assessment and planning. However, it is difficult for them to allocate time to the collection, cultivation and analysis of data. Instead, they need to focus on strategic decisions, and they need data to run their business, to understand how their customers behave and measure what really matters to the organization. Power BI can help to bring clarity and predictability to the CEO, and this session is aimed at CEOs, and those who support them with data, in order to see how they can be empowered by Power BI, and see it as a key asset within the organisations short and long term future.
Net Profit
This goes without saying, but keeping an eye on net profit at all times is essential for business leaders. This might be visualized as a line graph or quarterly chart. However you decide to represent the data, it needs to provide detailed, regularly updated information. You can get added value by allowing this data to be broken down.
With Power BI, you’d be able to tap your chart and see real-time data on profits by region, product type or team. First you calculate your gross profit, then your expenses, subtract expenses from gross profit, and you have net profit.
Calculate Gross Profit first:
Gross profit, also called gross margin, shows you how much money you made from selling a product.
It subtracts the selling price from your wholesale cost to calculate the difference. It does not take into account expenses from rent, personnel, supplies, taxes or interest. Gross profit is a required step toward calculating the company’s income or net profit.
Progress Towards Targets
You can use EXPON.DIST function in Microsoft Excel to help measure progress towards your targets.
Use EXPON.DIST to model the time between events, such as how long from the order placement takes to actual delivery. For example, you can use EXPON.DIST to determine the probability that the process takes at most 1 minute.
Revenues and revenue growth rate
By being able to instantly visualize how fast (or otherwise) your business is growing its revenues, it’s much easier to find out what’s going right and what’s going wrong. Need to lose some dead weight? Invest in a growing department? Respond to a new trend among consumers? Tracking your revenues closely is crucial and will help with those decisions. A line graph would again be particularly clear in this instance.
Think of a Rubik’s cube – people instinctively know how to use them, and to arrange the cube into colors. We also interact with colour and data in the same way; intuitively and quickly.
Expenses
Whether it’s staff, machinery, IT or property, your expenses are one of the biggest drains on your long term success. A dashboard can break these down instantly so you can see where your biggest outgoings are, and then make decisions about what’s costing too much.
Revenue per employee
Revenue per employee is a little like Return on Investment. Are your people actually making enough revenue to justify hiring them? Are they working at 100% capacity or is there room for them to work more, instead of employing new workers? A revenue per employee dashboard helps you make these choices rationally.
Employee Engagement
Measured by an anonymous survey, employee engagement is a key BI factor for any CEO. If your people are motivated, enthusiastic and giving their work 100%, you can be sure your company will grow. By contrast, unengaged colleagues will be a detriment to productivity. It’s essential to keep regular tabs on how employees are feeling about their work.
Summary
As part of the Digital Transformation process, the CEO must develop a guiding philosophy about how he or she can best add value whilst showing ongoing strategic assessment and planning. However, it is difficult for them to allocate time to the collection, cultivation and analysis of data. Instead, they need to focus on strategic decisions, and they need data to run their business, to understand how their customers behave and measure what really matters to the organization.
Power BI can help to bring clarity and predictability to the CEO, and this session is aimed at CEOs, and those who support them with data, in order to see how they can be empowered by Power BI, and see it as a key asset within the organisations short and long term future.
