You can get the latest post from here, which gives you some ideas on communication for your virtual blended team, and some pointers towards handy Visio stencils that might help. You can also navigate my other IoT posts by going to the IoT Toolkit menu that I’m trying to keep updated.
I recently did a brief blog post for Izenda on IoT and business intelligence, and this part of my IoT series expands on some of the themes there.
The Internet of Things is a new phenomenon; that said, a simple search for ‘Internet of Things IoT’ brings back over 60 million search results in Bing. What is the Internet of Things? The Internet of Things Global Standard gives us the following definition: ‘The Internet of Things (IoT) is defined as Recommendation ITU-T Y.2060 (06/2012) as a global infrastructure for the information society, enabling advanced services by interconnecting (physical and virtual) things based on existing and evolving interoperable information and communication technologies.
Now, this definition is fine but it focuses on the ‘shiny’ aspect of IoT and, most importantly, it does not mention the data aspect of IoT. It emphasises the connectedness of the various gadgets and their interoperability. I prefer Peter Hinssen’s discussion, where he recommends that we talk about the value of the network of things. The connected devices, on their own, will fulfil their purpose. However, if you want real insights from these sources, then you need to splice the data together with other sources in order to get insights from it.
The thing is, the Internet of Things is really the Internet of You.
We are now heading towards the Zettabyte generation thanks to the Millennial generation. For example, the World Data Bank projects that 50% will have smartphone by 2016, and 80% by 2020. We sent 7.5 trillion SMS messages in 2014. In fact, one app, WhatsApp, sent 7.2 trillion messages. And that’s just data from one app. In 2000, Kodak processed 80 billion photos processed from camera film. In 2014, 800 billion photos from smartphones were shared on social networks. And that’s just the photos that were shared.
We are the Internet of Things.
From the business perspective, how do you make use of that IoT data? The consumerization of IT means that business users are often asked to manage and cleanse data, regardless of its size and nature. Research suggests that data is growing at a rate of 40% of each year into the next decade, driven by increased online activity and usage of smart devices. (ABI, 2013). (The New York Times, 2014 ). The consumerization of data means that business users should be able to access and analyze the data comfortably. When we introduce data that comes under the umbrella o the Internet of Things, business users will need to be able to access IoT data from devices as well as other data sources, to get insights from the data.
How can we harness the IoT phenomenon to understand and serve our customers better?
The addition of data from a variety of sources, including data from devices, means that IoT has a very wide scope and opportunity. IoT can focus on the devices themselves, or the network infrastructure connecting devices, or the analytics derived from the data which comes from the network and the devices. In order to get true insights, the IoT data would be deployed in tandem with other relevant data so that the business user obtains the context. The IoT would also introduce real time data, which would be mixed with historical data.
Customer expectations are rising; and customer-focused businesses will need to put analytics at the heart of their customer services. For example, customers do not distinguish between out of date data, and inaccurate data; for them, they are the same thing. The customer landscape is changing, and it includes the ‘millennials’ who expect technology to offer an unfailing, personal experience whilst being easy to use. This expectation extrapolates to data, and customers expect organizations to have their data correct, timely and personal.
For organizations who put customers front-and-center of their roadmap, management should encourage self-reliance in business users by ensuring that they have the right tools to provide customer-centered service. Unfortunately, business users can suffer from a split between business intelligence reporting, and the operation systems, as a result of decoupled processes and technology at the point at which they are trying to gain insights. Often, users have to move from one reporting technology to another operational system, and then back again, in order to get the information that they need. This issue can be disruptive in terms of the workflow, and it is an obstacle to insights. In terms of IoT data, business users may have to go and get data from yet another system, and that can be even more confusing.
What does IoT mean for BI? Business Intelligence has matured from the earlier centralized technology emphasis, to a more decentralized business-focused perspective which democratizes the data for the business users. With the advent of IoT technologies, issues on collecting, refining and understanding the data are exacerbated due to the introduction of a variety of structured and unstructured data sources. In the meantime, there is an increased interest in businesses to find insights in raw data. However, this remains a challenge with the introduction of IoT data from devices, creating a mélange of data that can be difficult for business users to assimilate and understand. Companies risk obtaining lower ROI in IoT projects, by focusing only on the data, rather than the insights.
How did the industry get to this point, with disjointed technology and processes, and disconnected users? How can we move forward from here, to including IoT data whilst resolving the issues of previous business intelligence iterations? To understand this unquenchable thirst for data by business users and what it means for the future, let’s start by taking stock of the history of Business Intelligence. What are users’ expectations about data and technology in general? Until recently, these expectations have been largely shaped by the technology. Let’s start with the history lesson. What are the historical stages of Business Intelligence?
The First Generation of Business Intelligence – change in the truth
First generation Business Intelligence is the world of corporate Business Intelligence, embodied by the phrase ‘the single source of truth’. This is a very technical discipline, which focused on the extract-transform-load processing of data into a data warehouse, and focused less on business user intervention. The net result is that the business seemed to be removed from Business Intelligence. In response, the users pushed for decentralization of the data, so that they could drive their own decision making using the data flexibly, and then confirm it independently in the context in which they are operating. In terms of technology, business users reverted to the default position of using Excel as a tool to work with Excel exports, and subverting the IT departments by storing data in email.
The Second Generation of Business Intelligence – change in the users
Second Generation Business Intelligence was the change was wrought by the business users, who demanded clean data sources on a strong technical apparatus that they could mash and merge together, and they were empowered to connect to these sources. In this stage, the industry pivoted to bring the Business back into Business Intelligence, and it is typified by the phrase self-service business intelligence. The intended end result is that the business has structured data sources that the users understand, and the technical teams have a robust technical structure for the architecture in place. As before, Excel remained the default position for working with data, but the business users had more power to mash data together. Self-service business intelligence was not faultless, however. Business users were still dissatisfied with the highly-centralized IT approach as they still relied on other people to give them the data that they need. This issue introduced a delay, which increased the ‘time to answer’ metric whilst simultaneously not recognizing that this feeds into the ‘time to question’ metric. It does not recognize that analytics is a journey, and users expect to ‘surf’ through data in the same way that they ‘surf’ through the Internet.
What problems does IoT introduce for businesses, and how can we resolve them?
Given that there are inefficiencies in the process of business intelligence in organizations at the moment, how is this exacerbated by the introduction of data from devices, otherwise known as the Internet of Things? IoT data introduces new issues for business users for a number of reasons. IoT devices will transmit large amounts of data at a velocity which cannot be simply reconciled with other time-based data. The velocity of the data will add in an additional complexity as business users need to understand ‘what happened when’, and how that marries with existing data sources which may even be legacy in nature. Business users will need that work to be presented to them simply. Further, IoT devices will transmit data in different formats, and this will need to be reconciled so that the actual meaningful data is married to the existing, familiar data. If the business users are moving around disparate technology in order to match the data together, then the disconnected technology presents an obstacle to understanding the data, and thereby obtaining the promised holy grail of insights.
IoT means that we can obtain a granular level of customer data which provides unique insights into customer behavior. However, it does not immediately follow that the data will bring insights on its own; interpretation and analysis will be vital in order to obtain insights. Businesses can interpret IoT as equivalent to data from devices, and it is easy to distracted by shiny gadgetry. The ‘shiny’ approach to IoT can mean that business users are ignored in the journey, thereby shearing their insights from the solution as a whole.
Helping Business Users along the IoT Journey
As internal and external expectations on data increase, the pressure on business users will increase accordingly. Business users will need help to travel along the user journey, to adapt these changes in the data landscape that include IoT data. One solution is to add a new application that will help the business users to work with the IoT data. However, adding a new application will exacerbate the existing issues that business users experience. This might be an easy option for IT, but it will add in a new complexity for the business user. The introduction of IoT data does not necessitate the introduction of new technology to analyze the data.
IoT data resides mainly in the cloud, which means that organization’s infrastructure is changing rapidly. It will need to be reconciled and understood, regardless of where it resides. Organizations can end up with a hybrid architecture of cloud and on premise solutions, and the midst of these complex and fast-moving architectures, business users are easily forgotten. The business users will need to have a seamless environment for resolving cloud and on premise systems to enable them to product the anticipated analysis and results. Business users will find it difficult to navigate the terrain between cloud and on premise data, which will aggravate existing issues in the putting together existing data sources.
Business users have a need for data to carry out a fundamental analytical activity: comparison. How does this data compare to that data? How did last year compare to this year? How did that customer compare with this customer? Answering these simple questions may mean that traditional analytical tools may not be able to cope with the new types of data that are generated by IoT technologies, because the data will be disconnected in terms of technology and process. Excel is excellent for rectangular data sources, but it is not designed for data sources where the data travels at such velocity, and in non-rectangular shapes. So, what’s next?
The Third Generation of Business Intelligence – change in the data
The Third Generation of Business Intelligence is where users work in the era of real change in data, and it is this change is wrought by changes in the data itself. The data has changed fundamentally; it now travels faster, has more shapes, and is bigger in size than ever before. Users are not finding it easy to compare data simply by farming data into Excel; they need to be empowered to tackle seemingly simple business questions, like comparison, in a way that fits with a fluid way of working whilst being empowered with data from the IoT sphere as well as more traditional sources. By tapping into new volumes and particularly new varieties of data, organizations can ask questions about their customers and their business in a way that they have never been able to do, previously. Further, when we add IoT into the mix, there is a promise of insights from customers and their environments, which can be incredibly valuable to companies. It is not all one way, however: In this era of tech-savvy consumers, customer relationships require planning, nurturing, and constant attention.
There should be an acceptance that business users will want access to IoT sources in the same way as any other source, but these can be exasperating and non-intuitive. Simplicity is vital in the race for the business analyst and user, and their goal is to reduce time and effort in getting the insights that they want, whilst increasing efficiency and value.
So, what gets forgotten in the IoT torrent of attention, and what can we do about it?
Simply put, the business users get lost. They are already getting lost frequently with BI projects, and this will only make matters worse for IoT projects.The ones who mash data together, clean it, make decisions on it, and put it next to other data sources in order to make sense of the data – these are the ones who should be using the data.
Given all of these issues, how do we bring the users back into an IoT architecture? I was faced with this issue recently, when designing an IoT architecture which had a focus on machine learning. IoT work involved a great deal of complexity, which is neatly hidden behind the buzzwords.
The changes in data now mean that there is a clear extension of where the industry has come from, and where it is headed. So what comes next? The third generation of business intelligence: ready to go analytics using data regardless of its shape and size.
Organisations will need to focus on the third generation of Business Intelligence if they are to be successful in facilitating users to have the access to data that they need. Users will want to try and analyse the data themselv es. Fast questions need fast answers, and businesses need to move from their initial business question through to the resulting insight quickly and accurately, in a way that they are comfortable. They also need results at the velocity of the business; answers when they need them. Remembering the users is a deceptively simple requirement that presents a number of challenges.
The dislocation between IT and the business is at its apex when we look at the opposing approaches to data. IT is still seen as a bottleneck rather than an enabler. Business users perceive IT departments as a lag in a process that needs to get from question to insight quickly, and they will look for ‘good enough’ data rather than ‘right data’ in order to get it. The way forward is to make the business users’ activities simpler whilst providing a solution that the IT department are closely involved and find the solution easier to support, so that both parties feel that they own the solution.
The solution should put the focus back on the business users who not on the humans who actually deliver service, create insights, and ultimately add business value. To do this, they need to be able to search for meaning in the data, via aggregation, broadcasting and consuming information in order to add the value that is expected of them.
To summarise, these issues were at the forefront of my mind, when I was architecting an IoT solution recently. In my next post, I will explain my technical choices, based on these considerations. On my survey, it was clear that IoT needs to be taken to a further stage so that it is usable, actionable and sensible; not just data about sensors, but data that is relevant and provides insights.
If you want to talk more about the IoT issues here, or you’re interested in having me come along and speak at your event or workplace, please email me at Jen.email@example.com
I organised an AzureCon viewing party tonight in Hertfordshire, with great team support from Team Awesome over at Cloudamour.
We watched a total of four keynotes, running back to back for almost four hours. The keynotes were all awesome and I’ve blogged some learnings here.
First up, was Scott Guthrie (t), igniting the keynotes and kicking off the event with the journey to the intelligent cloud. I missed some of this piece because I was welcoming guests as they arrived, making introductions and so on. If you want to see the video, you can catch Scott Guthrie here on Channel 9. The thrust of Scott’s session was about cloud energising business and technical leaders worldwide turn the digital disruption into their advantage. Scott led customers who used cloud to enable their business to break new ground, and share their best practices in using some of the latest Microsoft innovations in enabling their journey to the cloud.
My personal favourite part of this piece was seeing the inspirational Lara Rubbelke (t) up on stage. Lara is inspirational and she’s generous with her time, supporting SQLFamily members. Lara explained the SQL Data Warehouse very clearly in terms of its simplicity to set up, and it’s relevance to the business. I liked her piece because she talked tech and business equally and that’s hard. It’s something I find that I have to do in my role every day; basically, wearing different hats, and it’s not easy to accomplish. Lara achieves this with ease and I recommend that you watch her segment, which is about 32 minutes into the video She also makes you think about how this could be relevant in your environment and that is an important takeaway.
In Lara’s words, using the technology is a ‘zero risk’ decision which allows you to scale up, scale down as you need. We don’t need to move our data, it just works, thereby offering immediate ROI, visualised in PowerBI.
Next up was Bill Staples (t) the CVP for the Azure App Platform, and the focus here was in growing and expanding
businesses using Azure as a base for apps.
Since apps are so personal and based around customers’ experience, they can help accelerate their business transformation and driving rapid results which are customer-centric.
Bill had some pretty interesting case studies and you can find them over on his keynote session, which is over at Channel 9.
Next up, the session I’d looked forward to the most: T.K. Rengarajan (t), CVP Data Platform. Ranga talked about IoT – the Internet of Things with *your* things. As with IoT, there was a focus on Stream Processing and Predictive Analytics. How can we use that data properly? How can we use it for prescriptive analytics i.e. what can I do? What should I do? We should be able to drive intent on it, to derive intelligent action. Here are some use cases:
- Rockwell use it to manage gas dispensers.
- Ford are embedding IoT sensors in their cars, going forward.
- ThyssenKrup – leading elevator manufacturer. Track the health of their elevators’ health, around the globe. Optimise the service experience before it breaks down.
They have the ability to optimise their service experience in predicting failure before the elevator breaks down. Now, that’s predictive analytics in action, using Azure as a base!
- IoT Starts with your things
- Provide connectivity to both existing and new devices
- Facilitate new insights by garnessing power of untapped data
- Preconfigured Solutions
- Workflow Automation
- Device Connectivity
- Command and control
- Fully managed system for analytics. Analyse Data of any size, shape and speed
- Productive day one
- Build on open standards – YARN
- the ones with unstructured data
- u-sql ETL script
- Unstructured TSV in Data Lake store to structured tables in data lake store
- including JSON expansion and filtering
- Data lake can support both structured and unstructured data
- Its easy to submit a job, and there is even a slider for parallelism! We can slide up to 1000 levels of parallelism. Ranga asked people to submit a name. I like ‘Pixie Dust Slider’ because it’s sprinkling magic on your data, but I don’t think Microsoft marketing would ever go for that!
- We can see that U-SQL looks very similar to standard SQL
- We can make references in .NET
- One of our columns is a JSON object, but with data lake, we can take a function to extract out that column and work with it.
- The different jobs are broken down.
Finally, we moved on to Jason Zander (t) to talk about cloud infrastructure. More pixie dust to make it happen! Here’s a summary:
- 24 azure regions, more than Google and AWS combined. Welcome India #Azure data centers!
- Enough fibre to wrap around the globe, 56 times.
- 1.4 million miles of fiber in the DCs
- ExpressRoute – for Azure. Speeds of up to 10 Gigabits per second. 21 ExpressRoute locations worldwide, including London.
In this series, I’m writing a bunch of very practical posts on helping you through an IoT project. There are plenty of other posts about the ‘why’ and the marketing buzz, but this is about the ‘do’.
If you are using Azure, the chances are that you might be a Microsoft Partner already. There are some useful goodies in there, and you may not be aware of these opportunities. The benefits of the Microsoft Partner network can be found here. However, it can be hard to relate the list to actual projects, and this blog is aimed at translating these benefits into something tangible that can help you on your IoT project. Firstly, though, take a look at the Action Pack subscription video in order to get some background:
How can this help you to start on your IoT project? Well, if you are starting out on IoT and Azure, then the first thing you’ll need are some handy free Azure credits. Now, if you have an MSDN subscription, then you will also have free Azure credits. Did you know that you can get free credits as part of your Microsoft Partner Action Pack subscription as well? Members of the Microsoft Action Pack program receive monthly credits of £65 of Azure at no charge, and the terms and conditions can be found here.
In practice, these means that you can set up two subscriptions for your Azure account; one for MSDN, and the other for your Microsoft Partner Azure credits.
To help you start out on your Azure project with IoT, you can get five internal use licenses for Office365. This is extremely useful, because it means you can download the Office software. So, in my projects, I recommend my customers become a partner with the Action Pack subscription since they will get one the following:
- Microsoft Office 365—either five seats Office on-premises and five Microsoft Office 365, or 10 seats Office on-premises. You can earn more seats of Office 365 after an additional cloud sale.
- Microsoft Dynamics CRM—no Microsoft Dynamics CRM Online licenses are granted at the subscription point. These licenses are granted after you close one Microsoft Dynamics CRM Online deal or at least 50 seats of Office 365 in the previous 12 months.
For your IoT project, the first option is particularly useful in the following scenarios:
If you have taken on new team members to do an AzureML project, then you are going to need Office software such as Excel, in order to view data. If people are choosing a career in AzureML, then you can make a safe bet that these team members will want to use the latest and greatest technology. This means that giving them Excel 2007, for example, isn’t going to work. Happy team members produce better results, and it’s important to empower them with the tools that they need, and *want*, to do the a job that they are proud of doing.
If you have Office365, then you can hook up your data nicely so that you can see and share it in Power BI.
- What is your call to action?
- Sign up for the Microsoft Partner, and enrol for the cloud Programs
- Sign up as an Action Pack Subscriber
- Make sure to look at your benefits, and you’ll see the Azure subscription credits and your Office365 licence keys. To do this, go to Resources, and then look for ‘Access my software and cloud benefits’.
Using my Partner Azure credits, and my MSDN credits means that I have two separate subscriptions for paying for Azure. In my case, I have a subscription for my own Virtual Machines for development, and then a different Subscription for my Proof of Concept work and the portfolio I’m building for demonstrations. It helps me to keep an eye on how much credit I’m “spending” on development work on Azure VMs for development work. At the moment, I have a few physical servers which I *used* to use for development, but I like the portability of having everything in the cloud. It will mean I don’t have to lug my heavy Dell mobile workstation around with me. For demonstrations, I can video my demos in advance in case I can’t access the cloud for some reason. If I find I’m incurring a lot of Azure credits and paying money, then I need to decide whether to purchase another physical machine, or stick with Azure. So far, Azure is winning on cost, and on factors such as performance and reliability, and ease of use. Running a small business and being on the PASS Board mean that I’m incredibly busy, and I need to be careful how I spend my time and effort. As you can understand, doing a lot of tech support may not be the best use of my time – even though I do enjoy it!
Now, you’re ready to move to the next step! There are a range of choices for architecting an IoT project, and I will talk about some of these issues in my next blog post.
As some of you know, I’m part of an Internet of Things project (IoT). IoT is the latest buzzword, but honestly, in my opinion, it is all just data. The data may have a different velocity, and it may be fired at you in different shapes. A lot of the problems are still the same; how to store it, how to clean it, how to interpret it and analyse it.
The added complexity for me in IoT, from the perspective of a Business Intelligence specialist, is that I am not familiar with devices or any of the communication stacks for transferring data. I am learning very fast, and I’m glad to say that I’m surrounded by a great team who explain the details very clearly. I am learning about all sorts, from the details of electrical engineering through to protocols. I am enjoying the challenge. I have learned a lot, and I’ll try to share through my blog as the journey progresses.
In the meantime, I thought I’d share some of the tools and IP that I’ve been garnering as I go along this journey. This page is likely to be updated as we go along, so please keep checking back. Please note that I am not affiliated with any of these vendors in any way, and if you find other tools, please do share them with me.
Management of Azure for my IoT project
For the purposes of managing Azure storage, applications and diagnostics, I use the Azure Management Studio by Cerebrata Software, which is ultimately owned by Red Gate Software. Why do I like it? Well, you can find a detailed list of the features here but here’s why I like using it in practice:
- I find that the interface is clean and crisp, and I can navigate it easily. I don’t have to think about using the technology, to get it to do what I want.
- In an IoT project, particularly during the research phase, you’re not really sure how much data is being emitted. It may be more or less than predicted. With the Azure Management Studio, I can keep an eye on my storage – and therefore my costs.
- With any project which involves early adoption of new technology, it’s important that key stakeholders are reassured about the performance and reliability of the technology. The Azure Management Studio has a series of dashboards, and this helps me to tell the story of Azure to key stakeholders.
Management of IoT Projects
I prefer to use Microsoft Project to manage projects. I have built a default IoT Project Plan using Microsoft Project, and I tailor it for each project that I’m involved in. I use Project Online with Office365 because I like having everything in one place, and it is easy to share it. I use Excel to list out tasks for people who don’t have Project, or don’t know how to use it. If people are interested, I can share the files.
If you don’t have access to Microsoft Project, then I recommend Zoho Projects. Quite frankly it’s an undiscovered gem. It’s free for one project, or you can pay $20 per month for 20 projects. This is cloud software at it’s best; functional, cheap, flexible and pay as you go. At that price, you’d be crazy not to try at least the free version.
I also like JIRA to log bugs, workflows and so on. I’ve been using the online version for years, and there is really no substitute for it.
If you’re thinking of starting an IoT project, or want to know more, then please email us at IoT@DataRelish.com and I will see if I can help you, or I can perhaps put you in touch with other people who can help.