Marlon Ribunal - Data Professionals At Work

11/02/2018
by Marlon Ribunal
Comments Off on How I Work: Data Professionals at Work

How I Work: Data Professionals at Work

It has been a busy year so far. Last March, I did a review, as a Technical Reviewer, for a SQL 2017 Machine Learning book. In the summer, the whole family went to the Philippines for a 3-week vacation. Between the Machine Learning book and the vacation, I was working with a book publisher for a possible contract for a SQL book. Well, I was just too busy at that point that I decided not to do it. I still have that sitting in my Someday list, though.

In the first quarter of the year, I was busy trying to check off the goals that I set earlier this year, then suddenly I got uber busy at work and had to put those goals in the backburner. But, then again, that’s not enough reason to not get involved in another book. I wanted to write another book but I cannot just do that at this point. So, I was very excited when Mala asked if I wanted to be a part of her book. I said Yes right away.

Data Professionals at Work is a collection of interviews of, well, data professionals working in the various aspects of the data industry – Database Administrator, Data Development, Data Analysis, Business Intelligence, Data Science, among others. You’ll read about how these professionals got their start in the data world. You’ll read about their opinion on what’s on the leading edge in the industry. You’ll get to read about their a-day-in-the-life-of.

Well, you’ll get a glimpse of what I do for a living in Chapter 26. You’ll read about what I have to say about Agile Methodologies, Best Practices, cloud adoption, etc.

All the interviewees were asked the same set of questions, so you’ll get different takes on the same topics, which is great!

If you’re thinking about moving into the data industry, this is a good book to pick up. You’ll get an idea of how everybody started in their field. This is a good book if you’re just starting out. You’ll get tips on how we approach our day-to-day work, tools that we use, work-life-balance, etc.

A note: My company in the book is showing as JustEnough Software Corp. That since changed to Mi9 Retail Co.

Cast Of Characters

(The following list is copied over from I’m in a book! Data Professionals at Work, sorry Kenneth. I was too lazy to write my own list)

Editor and interviewer: Malathi Mahadevan (blog|twitter)

Foreword: Kevin Kline (blog|twitter)

Interviewees:
Kirsten Benzel (blog|twitter)
Jes Borland (blog|twitter)
Tim Costello (blog|twitter)
Kevin Feasel (blog|twitter)
Kenneth Fisher (blog|twitter)
Drew Furgiuele (blog|twitter)
Ginger Grant (blog|twitter)
Kathi Kellenberger (blog|twitter)
Kendra Little (blog|twitter)
Andy Mallon (blog|twitter)
Jimmy May (blog|twitter)
Marlon Ribunal (blog|twitter)
Julie Smith (blog|twitter)
Dave Walden (blog|twitter)
Tracy Boggiano (blog|twitter)
Jason Brimhall (blog|twitter)
Mindy Curnutt (blog|twitter)
Argenis Fernandez(blog|twitter)
Joseph Fleming (twitter)
Matt Gordon (blog|twitter)
Vicky Harp (blog|twitter)
Andy Leonard (blog|twitter)
Steph Locke (blog|twitter)
John Q. Martin (blog|twitter)
John Morehouse (blog|twitter)
Joe Sack (blog|twitter)
Jonathan Stewart (blog|twitter)

The Data Professionals at Work book is now available at Amazon.

ggplot2() facet_grid

04/10/2018
by Marlon Ribunal
1 Comment

Creating Facets in SQL Server 2017 Using R

SQL Server 2017 In-Database Machine learning has brought the analytics closer to the data. It is now convenient to do data analytics either by using the programming language R or Python with the database engine.

In my previous post, I have demonstrated how easy it is to create a bar graph in SQL Server 2017 In-Database Machine Learning using  R.

We’re going to build upon that basic graph.

Sometimes doing data analysis would require us to look at an overview of our data across specific partitions, say a year. For example, we want to see how our product groups fare on month-to-month basis across the last 4 years.

In a data analytics perspective, there are quite a handful of data points in this requirement – data aggregate (quantity), monthly periods, and year partitions.

One of the approaches to handle such requirement is by using a facet. Faceting is a way of plotting subsets of data into a matrix of panels based on one or more variables – or facets.

Again, let’s begin with the end in mind. Here’s what faceting looks like:

Dataset

The quantity aggregates based on monthly period per product group are laid into the year partition facet in the following query. You can also set up your data like this in the R engine.

SELECT DISTINCT
SUM(il.Quantity) OVER (PARTITION BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate),
sg.StockGroupName
ORDER BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate)
) AS Quantity,
DATEPART(YEAR, i.InvoiceDate) AS InvoiceYear,
DATEPART(WEEK, i.InvoiceDate) AS InVoiceWeek,
DATEPART(MONTH, i.InvoiceDate) AS InvoiceMonthNo,
DATENAME(MONTH, i.InvoiceDate) AS InvoiceMonth,
sg.StockGroupName
FROM Sales.Invoices i
JOIN Sales.InvoiceLines il
ON i.InvoiceID = il.InvoiceID
JOIN Warehouse.StockItems si
ON il.StockItemID = si.StockItemID
JOIN Warehouse.StockItemStockGroups sis
ON si.StockItemID = sis.StockItemID
JOIN Warehouse.StockGroups sg
ON sis.StockGroupID = sg.StockGroupID
ORDER BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate);

Setup

Please refer to Creating Bar Graph In SQL Server 2017 Using R for the setup. We need two (2) libraries for this demonstration:

  • ggplot2()
  • scales

Again, if you get “missing package” error, just install it. If you get a “no function x” error, chances are you have not declared the function’s parent package. The data is based on the WideWorldImporters database.

Properties

The x-axis is ordered by DATEPART(MONTH, i.InvoiceDate) AS InvoiceMonthNo.The substring function in R is more intuitive than other programming languages. This expression is used as the label for x-axis: substr(InvoiceMonth, start = 1 , stop = 3). The theme setting is simply set by theme_dark() function.

Chart

The whole chart creation code looks like this:

quantitytschart <- ggplot(groupqty , aes(x = reorder(substr(InvoiceMonth, start = 1 , stop = 3), InvoiceMonthNo), y = Quantity, group = StockGroupName, colour = StockGroupName)) +
geom_line(size=1) + geom_point(size=3) +
facet_wrap( ~ InvoiceYear) + theme_dark() +
labs( x= "Month", y="Stock Group Quantity") +
scale_y_continuous(labels = comma)

That’s it.

Facets*

The above matrix is of type facet_wrap. Another option is facet_grid.

  • facet_grid(. ~ InvoiceYear) 
  • facet_wrap( ~ InvoiceYear)

facet_grid(row ~ col): this is what they call “bivariate”, which creates 2-D matrix of panels based on two factor.

facet_wrap(~cell): “univariate”, creates 1-D strip of panels, based on one factor, and wrap the strip into a 2-D matrix.

*Source: ggplot2 Quick Reference: facet

Here’s the facet_grid (See the screenshot of facet_wrap above):

Complete Code

Here’s the rest of the code:

EXEC sp_execute_external_script @language = N'R',
@script = N'
#Library
library(ggplot2);
library(scales);

chartfile <- "C:\\Marlon_test\\Groupqty.png"
png(filename=chartfile, width=1800, height=900)

#Data Frame
groupqty <- InputDataSet

quantitytschart <- ggplot(groupqty , aes(x = reorder(substr(InvoiceMonth, start = 1 , stop = 3), InvoiceMonthNo), y = Quantity, group = StockGroupName, colour = StockGroupName)) +
geom_line(size=1) + geom_point(size=3) +
facet_grid(. ~ InvoiceYear) + theme_dark() +
#facet_wrap( ~ InvoiceYear) + theme_dark() +
labs( x= "Month", y="Stock Group Quantity") +
scale_y_continuous(labels = comma)

print(quantitytschart)',
@input_data_1 = N'
SELECT DISTINCT
SUM(il.Quantity) OVER (PARTITION BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate),
sg.StockGroupName
ORDER BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate)
) AS Quantity,
DATEPART(YEAR, i.InvoiceDate) AS InvoiceYear,
DATEPART(WEEK, i.InvoiceDate) AS InvoiceWeek,
DATEPART(MONTH, i.InvoiceDate) AS InvoiceMonthNo,
DATENAME(MONTH, i.InvoiceDate) AS InvoiceMonth,
sg.StockGroupName
FROM Sales.Invoices i
JOIN Sales.InvoiceLines il
ON i.InvoiceID = il.InvoiceID
JOIN Warehouse.StockItems si
ON il.StockItemID = si.StockItemID
JOIN Warehouse.StockItemStockGroups sis
ON si.StockItemID = sis.StockItemID
JOIN Warehouse.StockGroups sg
ON sis.StockGroupID = sg.StockGroupID
ORDER BY DATEPART(YEAR, i.InvoiceDate),
DATEPART(MONTH, i.InvoiceDate);
';

Have fun and happy faceting!

04/03/2018
by Marlon Ribunal
3 Comments

Creating Bar Graph In SQL Server 2017 Using R

Creating a bar graph with R in SQL Server 2017 is relatively easy. It’s just a matter of finding the correct functions, components, and properties of the package of your choice. For our example here, I am using ggplot2 which is probably the most popular plotting system for R.

Let’s create a quick colorful bar graph. This should demonstrate how easy it is to use ggplot2 in SQL Server 2017 with R. “Easy”, of course, is a relative term. Well, it’s not easy on the onset as you have to get yourself familiarized with all the components of the package. But once you’re doing this on regular basis, you’ll get the hang of it. Think of the time when you’re still learning how to write your first Select statement in SQL.

Cheesy Bar Graph

Let’s begin with the end in mind. Here’s our cheesy colorful bar graph:

Install Package

Let’s install the package if it hasn’t been installed yet. The easiest way to do that is to run RGUI.exe that came with your SQL Server 2017 In-Database Machine Learning installation. You can find it here:

C:\Program Files\MSSQL14.MSSQLSERVER\R_SERVICES\bin\x64

Take note that you need to run the executable as Administrator. Also, if you’ve installed the R engine prior to your SQL Server 2017 In-Database Machine Learning with R, you have to explicitly tell the R package installer where you want your package installed.

> install.packages("ggplot2", lib="C:\\Program Files\\Microsoft SQL Server\\MSSQL14.MSSQLSERVER\\R_SERVICES\\library", dep = TRUE)

dep = TRUE tells the installer to install dependencies. ggplot2 depends on a lot of other packages. You can check dependencies using MiniCRAN.

Dataset

Our dataset is based on the WorldWideImporters database. Here we’ve got the Sum of Line Profit per Stock Group:

USE WideWorldImporters;
GO

SELECT sg.StockGroupName,
SUM(il.LineProfit) AS LineProfit
FROM Sales.InvoiceLines il
JOIN Warehouse.StockItems si
ON il.StockItemID = si.StockItemID
JOIN Warehouse.StockItemStockGroups sis
ON si.StockItemID = sis.StockItemID
JOIN Warehouse.StockGroups sg
ON sis.StockGroupID = sg.StockGroupID
GROUP BY sg.StockGroupName;

Setup

Let’s create a directory for the graphics output –C:\Marlon_Test.

Here’s the script that we pass to @script parameter for sp_execute_external_script:

library(ggplot2);
chartStockGroupProfit <- "C:\\Marlon_test\\StockGroupProfit.png"
png(filename=chartStockGroupProfit, width=1000, height=800)
StockGroupProfit <- InputDataSet
StockGroupProfitChart <- ggplot(data=StockGroupProfit, aes(x=StockGroupName, y=LineProfit)) +
geom_bar(stat="identity")
print(StockGroupProfitChart)

Where:

chartStockGroupProfit - Filename

StockGroupProfit - Data Frame based on the InputDataSet (see Select Statement above)

StockGroupProfitChart - The actual chart

Create the Bar Graph

Let’s put that together. Note: Although you specified you wanted to install the dependencies for the ggplot2 package (dep = TRUE), the installer in some cases might not install all of them. If the ggplot2 fails with “missing package” error, just install the specified package using RGui.exe (see above).

EXEC sp_execute_external_script
@language = N'R',
@script = N'
library(ggplot2);
chartStockGroupProfit <- "C:\\Marlon_test\\StockGroupProfit.png"
png(filename=chartStockGroupProfit, width=1000, height=800)
StockGroupProfit <- InputDataSet
StockGroupProfitChart <- ggplot(data=StockGroupProfit, aes(x=StockGroupName, y=LineProfit)) +
geom_bar(stat="identity")
print(StockGroupProfitChart)',
@input_data_1 = N'
SELECT sg.StockGroupName,
SUM(il.LineProfit) AS LineProfit
FROM Sales.InvoiceLines il
JOIN Warehouse.StockItems si
ON il.StockItemID = si.StockItemID
JOIN Warehouse.StockItemStockGroups sis
ON si.StockItemID = sis.StockItemID
JOIN Warehouse.StockGroups sg
ON sis.StockGroupID = sg.StockGroupID
GROUP BY sg.StockGroupName;';

Bar Graph Format

You may argue that the above graph looks a lot better than our colorful graph at the top of this blog. But arguments aside, let’s put on some formatting.

You may notice that the Y-Axis (Line Profit) is not in Dollar ($) format. The Packaging Material group is more than $50,000,000.00.

Well, you can argue about the aesthetic of the graph, but you may agree that the exponential format of the Line Profit is not quite intuitive.

If we divide the LineProfit by the exponential value of 1e6 (1 times 10 to the power 6, which is 1,000,000), then we get a readable metrics:

aes(x=StockGroupName, y=LineProfit/1e6)

To format that value in Dollars ($), we need to add another library to the script – library(scales).

Then we can do:

scale_y_continuous(labels = dollar)

Rename Labels

You may notice that the Y-Axis Label says “LineProfit/1e+06”. That can easily be renamed by using the lab() function:

labs(x="Product Groups", y="Line Profit (Millions)")

To format the data frame labels, use the theme() function:

theme(text = element_text(size=20, color="blue"), axis.text.y = element_text(size=18, face="bold"), axis.text.x = element_text(size=18, face="bold", angle=80, vjust = 0.5))

Then, we should have something like this:

Reorder Bar Graph

Just to have the graph easy to read, let’s reorder the bar in descending order.

Plus, we don’t really like to look at bars that seem to resemble that of a dirty finger gesture (you’ll see it).

aes(x=reorder(StockGroupName,-LineProfit / 1e6), y=LineProfit / 1e6))

Rainbow Magic

To have that rainbow effect on the bars, modify the geom_bar() like this:

geom_bar(stat="identity",fill = rainbow(n=length(StockGroupProfit$LineProfit)))

Putting it all together

Here’s the complete R code:

EXEC sp_execute_external_script
@language = N'R',
@script = N'
# Use ggplot2/scales
library(ggplot2);
library(scales)
# Create chart file (folder must be existing)
chartStockGroupProfit <- "C:\\Marlon_test\\StockGroupProfit.png"
png(filename=chartStockGroupProfit, width=1000, height=800)
# Define data frame
StockGroupProfit <- InputDataSet
# Build chart
StockGroupProfitChart <- ggplot(data=StockGroupProfit, aes(x=reorder(StockGroupName,-LineProfit / 1e6), y=LineProfit / 1e6)) +
geom_bar(stat="identity",fill = rainbow(n=length(StockGroupProfit$LineProfit))) +
theme(text = element_text(size=20, color="blue"), axis.text.y = element_text(size=18, face="bold"), axis.text.x = element_text(size=18, face="bold", angle=80, vjust = 0.5)) +
labs(x="Product Groups", y="Line Profit (Millions)") +
scale_y_continuous(labels = dollar)
print(StockGroupProfitChart)',
@input_data_1 = N'
SELECT sg.StockGroupName,
SUM(il.LineProfit) AS LineProfit
FROM Sales.InvoiceLines il
JOIN Warehouse.StockItems si
ON il.StockItemID = si.StockItemID
JOIN Warehouse.StockItemStockGroups sis
ON si.StockItemID = sis.StockItemID
JOIN Warehouse.StockGroups sg
ON sis.StockGroupID = sg.StockGroupID
GROUP BY sg.StockGroupName;'

There you go. Happy plotting!