Life after serving on the PASS Board

I didn’t put myself forward for re-election to the PASS Board, and I officially step down on 31st December. I’ve always struggled with the question – why get involved in the tech community? If tech community is something that you think about, I can’t say it enough – it will change your life for the better. Please, don’t hesitate. I’m not leaving the community. I’m just going into a new phase and I’ll contribute in different ways.

About 8 years ago, I started to speak at technical events because I wanted to tackle my fears head on: standing up in a room full of people, everyone looking at me, in a very male oriented environment. So my first session was one hundred people, and since then, I’ve spoken all over the world, and my largest in-person audience was over six thousand people. I did lots of technical community work and I began to find my home there; I found friendship, and I’d specifically like to thank the community men who treated me as a person and an expert as I proved over and over again that I could teach and be relevant. I stopped being this afraid, shy, nervous thing. I also learned from other people and I found some friendship, simply from being part of the technical community. Gentlemen – you have such an important role to play in small and large ways and I thank you from the bottom of my heart. For those of you who care about these things, your voice and actions carry further than mine and, simply, thank you.

I want to deeply thank the informal tech community of Microsoft and the Tableau community for giving me the opportunities that you have, and for helping me to find some friendship there from the mostly great people I met. I have given a lot, but you have given me far more than you will ever know. Thank you.

Eventually I was elected to the board of PASS as a Non Executive Director and I hoped to make a difference, primarily in the fields of analytics and diversity. Thank you to everyone who voted for me. I thought that the influence would give me the ability to do good things for others. These are my passions. This didn’t work out as I’d hoped and, after four years of my life, I’ve decided that I won’t be putting myself forward for re-election. Hopefully the next person will find it easier to make a difference for EMEA, and be empowered to do things where I just could not have an impact. I’m going to do other things instead, and I’ll come on to those at the end.

I want to move past this, so let’s talk a little about what’s next for me?

  • MBA – I’m going to do my MBA. Academic success is something that I value. I learned a lot over the past four years but it’s time to spend time and effort on something that is for me. I’ll share my experiences in data-driven wisdom and I hope you’ll find it valuable as you grow in your careers.
  • Diversity Charter – I’m trying to set up a Diversity Charter so tech community orgs, such as user groups, can show that they are truly welcoming to people of different backgrounds. I’m hoping to be a part of the Diversity party that Microsoft are having. It’s easy for Diversity to start to mean ‘sticking pretty babes up on a stage’ and I’m hoping that the narrative doesn’t start to focus only on young women; it involves other aspects of people, such as age, disability, transgender identity, sexuality, race, faith and respect for other people.
  • Thought Leadership – I’ve also become attracted to thought leadership and I do industry analysis as a freelancer. PASS do Tech Leadership but that is not real Thought Leadership, which transcends technology. My efforts to have real Thought Leadership podcasts fell by the wayside and I only got two produced and the third one was never processed. I thought it would be a great way for PASS to connect, learn and share with industry thought leaders (not tech leaders) to promote the community at a more strategic level.
  • Events – I’ll continue speaking for as long as people want to hear me.

Diversity is important to me because it means I want to focus on something positive. I know what it feels like to be powerless and have your voice taken away from you. I want to be able to find some meaning in life and how we can help one another.  I’m doing it now because my voice is all I have. I want to try and make something good out of it. Diversity makes sense to make because it’s all about trying to make sure that everyone is included and they aren’t isolated from doing a job or a community activity that they love. And techies do love technology and everyone’s inner geek should be welcome.

 

Diversity Charter

For those of you looking for the Diversity Charter effort, you need to go here: https://diversitycharterblog.wordpress.com/

It is a team effort, not just me!

I’d love to see a Diversity Charter that user groups and communities could use, to show that they are welcoming and open to all members of the technical communities. I think that the charter could look something like this draft:

We believe that all members of the technical community are equally important.
We are part a tech community where we value a diverse network, and learn and share from one another:
regardless of age,
regardless of colour,
regardless of their ethnicity,
regardless of their religion or beliefs,
regardless of disability,
regardless of gender,
regardless of sexual orientation,
regardless of their race,
regardless of their ability or lack of ability,
regardless of nationality or accent.
We are a diverse tech community where we are all individuals with differences, but we are all members and we can all learn from each other.

I have other ideas:

  • A logo for the Charter. I have some ideas, but I’d be delighted for help
  • I have set up a Slack channel for people to discuss the Charter – please ping me on jen.stirrup@datarelish.com to find out more
  • I would love forums where people can ask questions and have community-led answers on how to be open and welcoming to people from different backgrounds. No question is a stupid question. The main thing is that you are talking about it with enquiry and openness in mind, in order to understand other people better. Nobody can fault you for having a kind heart that is trying to learn.

 

Azure Tools and Technologies Cheat Sheet

Don’t you think that the amount of Big Data technologies in Azure is very confusing? I’ve distilled some information from the official Microsoft Azure blog so it’s easier to read. Before we begin, here is a potted Hadoop history:

 

This cheatsheet contains a high level descriptions of the tools, APIs, SDKs, and technologies that you’ll see in Azure. Together, they are used in tandem with big data solutions, and they include proprietary Azure and open source technologies.

I hope that this cheat sheet will help you to more easily identify the tools and technologies you should investigate, depending on the function.

Function Description Tools
Data consumption Extracting and consuming the results from Hadoop-based solutions. Azure Intelligent Systems Service (ISS), Azure SQL Database,LINQ to Hive, Power BI, SQL Server Analysis Services (SSAS),SQL Server Database Engine,SQL Server Reporting Services (SSRS)
Data ingestion Extracting data from data sources and loading it into Hadoop-based solutions Aspera, Avro, AZCopy, Azure Intelligent Systems Service (ISS), Azure Storage Client Libraries, Azure Storage Explorer, Casablanca,Cloudberry Explorer, CloudXplorer, Cross-platform Command Line Interface (X-plat CLI),File Catalyst, Flume, Hadoop Command Line, HDInsight SDK and Microsoft .NET SDK for Hadoop, Kafka, PowerShell,Reactive Extensions (Rx), Signiant,SQL Server Data Quality Services (DQS),SQL Server Integration Services (SSIS),Sqoop,Storm,StreamInsight,Visual Studio Server Explorer
Data processing Processing, querying, and transforming data in Hadoop-based solutions Azure Intelligent Systems Service (ISS),Hcatalog, Hive,LINQ to Hive, Mahout,Map/reduce, Phoenix, Pig, Reactive Extensions (Rx), Samza, Solr,SQL Server Data Quality Services (DQS),Storm,StreamInsight
Data transfer Transfer  data between Hadoop and other data stores such as databases and cloud storage. Falcon,SQL Server Integration Services (SSIS)
Data visualization Visualizing and analyzing the results from Hadoop-based solutions. Azure Intelligent Systems Service (ISS), D3.jx, Microsoft Excel, Power BI, Power Map, Power Query, Power View, PowerPivot
Job submission Processing jobs  in Hadoop-based solutions. HDInsight SDK and Microsoft .NET SDK for Hadoop
Management Manage and monitor  Hadoop-based solutions. Ambari, Azure Storage Client Libraries, Azure Storage Explorer, Cerebrata Azure Management Studio, Chef, Chukwa, CloudXplorer, Ganglia, Hadoop command line,Knox,Azure Management Portal, Azure SDK for Node.js,Puppet, Remote Desktop Connection, REST APIs,System Center management pack for HDInsight,Visual Studio Server Explorer
Workflow Creating workflows and managing multi-step processing in Hadoop-based solutions. Azkaban, Cascading, Hamake, Oozie,SQL Server Integration Services (SSIS)

Any questions, please get in touch at hello@datarelish.com

SQL Server on Linux for the Business Intelligence Professional: Getting your Database on the VM

In this edition, we will look at getting the database up on the Virtual Machine as a first step. Then, we will restore it using SSMS. I’m expecting that you will have done the pre-requisites that I laid out yesterday. You will also need to have connected to your database on the Azure Virtual Machine.

For this purpose, we will use the WideWorldImporters sample databases provided by Microsoft.

If you are doing this activity on your own database: Make sure that the database backup type is Full. Also, make sure you are backing up to disk. which is most likely to be found in this location:

C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Backup\

Connecting via SSH to the Azure Virtual Machine

This is when we start to use SCP (Secure File Copy) and SSH (Secure Shell) to copy our BAK file up to the Linux Virtual Machine.

We will use Git for Windows in this example. Here is how the commands look:

Bash Open Connection

I’m going to deal out the commands here because it will help you.

Firstly, you need to copy the Azure connection command from an earlier step. Mine looks like this:

ssh datarelish@13.68.23.71

Type yes to continue, and hit return

Enter the password that you created when you set up the Azure Virtual Machine

Let’s execute the dir command so we can see what is up there.

From the last line, you can see that the directory contains one file: test_data.txt

Copying the file from your local Machine to the Azure Virtual Machine

I find it easier to have another Git window open here. This will point at the local machine, whereas the other will point at the Azure Linux VM.

In this new Git window, we are going to use SCP to copy the file from the local machine up to the remote Azure Virtual Machine.

Change your Directory to the location where you have stored the WideWorldImporters database. My location is D:\773WorkingDirectory

You can use commands such as dir and cd to help you to navigate.

To get to my D drive, I typed cd d:

I then used dir to read out the files and folders on the D drive

I then used cd 773* to get to the Directory I wanted, which is called 773WorkingDirectory

When I got there, I used the dir command to get to the databases I wanted.

To copy the file to the Azure Virtual Machine, I then executed the following command from the D:\773WorkingDirectory folder:

scp WideWorldImporters-DW-Full.bak datarelish@13.68.23.71:./

This landed the data to my home\datarelish directory. I could then use the cp command to copy the file so that the backup file ended up in the /var/opt/mssql/data directory.

In the next post, we will look at restoring the database onto SQL Server on Linux.

 

 

 

SQL Server on Linux for the Business Intelligence Professional: Connecting to your database on Azure Virtual Machine

What’s next for the Business Intelligence professional, as they learn about Linux? Here, we will connect SQL Server Management Studio to the Linux edition of SQL Server. I’m expecting you’ll have completed the pre-requisites laid out previously.

Connect SQL Server Management Studio to SQL Server Linux

Add in an inbound security rule which allows remote connection via TCP/1433.
  • In the Azure Portal, navigate to the SQL Server on Linux Virtual Machine. Mine is called 773SQL because I created it to prepare for the 773 exam.
  • Select the item Network Interfaces on the left hand side.
  • On the main panel, select the Network Interface
  • Next, look for the Network Security Group, which you can see under ‘Essentials’ on the right hand side.

Inbound Security Rules

  • Click on Add and Give the rule a name: SSMSRemoteConnections will be fine, if you can’t think of a name
  • Under Service, select MS SQL
  • Make sure TCP is selected
  • Port Range should read 1433
  • Select Allow and click OK

Your inbound Security Rule has been created. Now, let’s connect to your SQL Server Linux machine.

Get your Virtual Machine IP address from the Azure Portal

In this section, we will get the IP address of the Virtual Machine that’s running SQL Server on Linux.

In the Azure Portal, navigate to your Virtual Machine

If it isn’t up and running, you will need to start it. When it’s running successfully, click on the Connect button.

You will get a message that looks like the following item:

IP Address for Linux Connection

Make sure that you keep a note of the IP address. You will need it to connect SSMS to the SQL Server Linux Virtual Machine. In this example, the IP is 13.68.23.71. I have it set to Dynamic and the VM will most likely be gone by the time this blog is published…. so don’t bother trying to connect to this IP address here. You must use your own, obtained by following the previous step.

Connect SSMS to the SQL Server on Linux VM

Here is an example of how it should look – obviously you will need the password!

The IP Address was obtained from the Azure Portal

You will need to use SA

You will need to use the Password that you used when you set up the Azure Virtual Machine

Click connect and you should see your SQL Server now, as normal.

SSMS Side Panel

Let’s double check, shall we?

Checking the Version

Open a New Query window

Enter the Query: SELECT @@VERSION

Execute the query, and take a look at the results:

In the next blog in this series, let’s take a look at how we can get a database up to the SQL Server on Linux Virtual Machine, and then how we can restore it. We will also look at a potential pitfall or two.

The Previous Post is here

SQL Server on Linux for the Business Intelligence Professional: Prerequisites

In this blog series, I’ll talk about SQL Server on Linux for the Business Intelligence Professional. You’ll need some prequisites, and this blog post edition is dedicated to helping you to get set up.

It’s assumed that you have set up a SQL Server on Linux Virtual Machine on Azure.

To connect to it, you will need an SSH client. SSH stands for Secure Shell. SSH is used to log into a remote machine and execute commands. It also supports other features, such as tunneling, forwarding TCP ports and X11 connections. It can transfer files using the associated SSH file transfer (SFTP) or secure copy (SCP) protocols. SSH uses the client-server model.

I’ve put some instructions here so you have an SSH client installed.  The options are below:

Installing an SSH Client

If you don’t have a client installed, you have a lot of options for SSH Clients. For our purposes, there are no differences although I recommend Chocolatey because it has some great package features. These choices are listed here:

  • Choose Git for Windows I will go through the examples using this client
  • Use Chocolatey. SSH is also part of Chocolatey 
  • PuTTY is featured in the Microsoft documentation

Git for Windows

You can install Git for Windows, which has an SSH client.  There are a ton of front end tools.

Chocolatey

To install it, run a command console as Administrator. To do this, type CMD in the Cortana search window, and options will come up. Right click on the Console, and select the option Run as Administrator.
Copy and Paste the following command, taken from the Chocolatey Site:
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"
You can also install the Chocolatey GUI.  The command to do this is as follows, again, credit to the Chocolatey team over at the website itself:

choco install chocolateygui
You can use PuTTY, as Microsoft Books online show. I like using Git Windows simply because my customers usually have it installed already, and it’s not an extra step to install it. SSH is part of the Git for Windows bundle so you probably already have it.
When your SSH client is installed, then you can connect to your SQL Server Linux machine. I will cover this item in the next post.

How do you evaluate the performance of a Neural Network? Focus on AzureML

I read the Microsoft blog entitled ‘How to evaluate model performance in Azure Machine Learning‘. It’s a nice piece of work, and it got me thinking. I didn’t see that the blog post contained anything about neural network evaluation, so this topic is covered here.

How do you evaluate the performance of a Neural Network? This blog focuses on Neural Networks in AzureML, in order to help you to understand what they mean.

What are Neural Networks?

Would you like to know how to make predictions from a dataset? Alternatively, would you like to find exceptions, or outliers, that you need to watch out for? Neural networks are used in business to answer the business questions. They are used to make predictions from a dataset, or to find unusual patterns. They are best used for regression or classification business problems.

What are the different types of Neural Networks?

I’m going to credit the Asimov Institute with this amazing diagram:

Neural Network Types

In AzureML, we can review the output from a neural network experiment that we created previously. We can see the results by clicking on the Evaluation Model task, and clicking on the Visualise option.

Once we click on Visualise, we can see a number of charts, which are described here:

  • Receiver Operating Curve
  • Precision / Recall
  • Lift visualization

The Receiver Operating Curve

Here is an example:

ROC Curve

In our example, we can see that the curve well up into the left hand corner for the ROC curve. When we look on the precision and recall curve, we can see that precision and recall are high figures, and this leads to a high F1 score. This means that the model is effective in terms of how precisely it classifies the data, and that it covers a good proportion of the cases that it should have classified correctly.

Precision and Recall

Precision and recall are very useful for assessing models in terms of business questions. They offer more detail and insights into the model’s performance. Here is an example:

Precision and Recall Precision can be described as the fraction of times that the model classifies the number of cases correctly. It can be considered as a measure of confirmation, and it indicates how often the model is correct. Recall is a measure of utility, which means that it identifies how much that the model finds of all that there is to find within the search space. Both scores combine to make the F1 score. The F1 score combines Precision and Recall. If either precision and recall are small, then the F1 score value will be small.

Lift Visualisation

Lift Chart visually represents the improvement that a model provides when compared against a random guess.This is called a lift score. With a lift chart, you can compare the accuracy of predictions for the models that have the same predictable attribute. Lift Visualisation

Summary

In my next blog, I’ll talk a little about how we can make the Neural Network perform better.

To summarise, we have examined various key metrics in evaluating a neural network in AzureML. These scores also apply to other technologies, such as R.

These criteria can help us to evaluate our models, which, in turn, can help us to fundamentally evaluate our business questions. Understanding the numbers helps to drive the business forward, and visualizing these numbers helps to convey the message of the numbers.