Machine Learning with .NET: Model Builder

This is part three in a series on Machine Learning with .NET.

Last time on 36 Chambers, we looked at building our own model to explain why Kelvin Benjamin is awful. Today, we’re going to use the ML.NET Model Builder to understand how awful airlines are. The unofficial motto of this blog, after all, is “Parades need rain, too.”

The ML.NET Model Builder is still in preview, so expect things to change; some of those changes might be breaking changes, too. If you want to play along, download the Model Builder from the Visual Studio Marketplace. This works for Visual Studio 2017 and Visual Studio 2019, so you’ll need one of those. Community Edition is supported as well, so you don’t need a fancy version of Visual Studio.

After you install the extension, you’ll need to be in a solution with at least one project in it. The Model Builder creates new projects but is a context menu option off of a project itself. If you want to play along at home, clone my GitHub repo. I also have some instructions on the repo, which is rare for me.

First up, right-click on any of the projects and select Add --> Machine Learning.

We’ll add machine learning, all right, just not here. Not Europa.

This brings up a wizard tab to build a machine learning model.

Choose the form of the destructor.

You have four options from which to choose: two-class classification, multi-class classification, regression, or Choose Your Own Adventure. Today, we’re going to create a two-class classification model. Incidentally, they’re not kidding about things changing in preview—last time I looked at this, they didn’t have multi-class classifiers available.

Once you select Sentiment Analysis (that is, two-class classification of text), you can figure out how to feed data to this trainer.

Twitter is a garbage hole and airlines are great garbage magnets, so this should be grand.

As of right now, we can use delimited files or read from a table in SQL Server. Note the 1 GB limit for maximum file size; this is not a solution for very large data sets.

Once we select a file, we get the option to predict a label. As of the time of writing, this label must have a value of either 1 or 0. Our tweets are in the text column, so we’ll use that to predict sentiment.

From there, we move on to training.

This model is definitely a 10.

We don’t specify the algorithm or algorithms to use, but we do specify the number of seconds allowed for training. This is an interesting specification parameter; basically, the more seconds you allow for training, the more likely you are to find a good model. I don’t have any insight into the specifics of what happens during this training period; my expectation is that they iterate through a set of algorithms and try progressively more complicated sets of inputs and transformations. The ML.NET team does have a set of guidelines around how long you might want to train given data sizes. I’m going to train for 4 minutes because I didn’t plan well enough ahead to give myself ten minutes to let it wait. Here it is in progress:

Hard at work on training, boss. Training all kinds of stuff here.

Eventually, we run out of time and we have a winner:

Like a proper horse race, this one changed just as we ran out of time. The most exciting 4 minutes in sports.

Now it is time to evaluate. We get to see a table with the five best models on a few criteria: accuracy, Area Under the Curve (AUC), Area Under the Precision-Recall Curve (AUPRC), F1 score, and the amount of time it took to train.

One of these things is not like the others.

One thing I wish I could see here was a breakdown of our key correlation matrix measures—AUC, AUPRC, and F1 score are fine combination metrics and are very useful, but when you have major discrepancies in class makeup, recall by itself can mean a lot. For example, suppose we have a test for a medical condition which affects 1 in 100,000 people. We can get a 99.999% accuracy if we simply return “Negative result” for everybody, but that’s an awful test because the recall is zero: we never catch any of the positive results. It could be that I am not clever enough to interpret these results and infer the base values, though.

Regardless, our next step is to generate some code. Click on the “Code” link and you get the opportunity to add new projects. Note that you do not get to choose which of the models you want to use; the Model Builder picks the one with the highest accuracy.

Time to make the donuts.

We now have two new projects.

I made the donuts.

We have a model project and a console application. The model project contains our trained model (as a zip file) and input and output files. Here is the input file:

using Microsoft.ML.Data;

namespace DotNetMachineLearningML.Model.DataModels
{
    public class ModelInput
    {
        [ColumnName("airline_sentiment"), LoadColumn(0)]
        public bool Airline_sentiment { get; set; }

        [ColumnName("text"), LoadColumn(1)]
        public string Text { get; set; }
    }
}

And here is the output file:

using System;
using Microsoft.ML.Data;

namespace DotNetMachineLearningML.Model.DataModels
{
    public class ModelOutput
    {
        // ColumnName attribute is used to change the column name from
        // its default value, which is the name of the field.
        [ColumnName("PredictedLabel")]
        public bool Prediction { get; set; }

        public float Score { get; set; }
    }
}

The console application contains a model builder and a Program file. The ModelBuilder C# file allows us to modify or re-train our model should we desire, but is not needed for regular operation. The console application grabs a row at random from our tweet set and shows the prediction versus actual values. Instead of doing that, I’m going to take user input to classify a tweet. Replace the main method with this code:

static void Main(string[] args)
{
	MLContext mlContext = new MLContext();

	ITransformer mlModel = mlContext.Model.Load(GetAbsolutePath(MODEL_FILEPATH), out DataViewSchema inputSchema);
	var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(mlModel);

	Console.WriteLine("Enter a sample tweet:");
	var text = Console.ReadLine();

	// Create sample data to do a single prediction with it 
	ModelInput sampleData = new ModelInput { Text = text };

	// Try a single prediction
	ModelOutput predictionResult = predEngine.Predict(sampleData);
	string outcome = predictionResult.Prediction ? "Positive tweet" : "Negative tweet";
	Console.WriteLine($"Single Prediction --> Predicted value: {outcome}");

	Console.WriteLine("=============== End of process, hit any key to finish ===============");
	Console.ReadKey();
}

Change the console application to be the startup project and away we go:

Too soon for TWA jokes?

Conclusion

The Model Builder won’t replace an actual machine learning pipeline anytime soon, but it’s a good way to get started with processing clean data when you have no clue which algorithms you should try.