In the United States, by around 8th grade (ages 13–14), students are introduced to the concept of a function. This idea becomes foundational across nearly all of mathematics, including algebra, trigonometry, Euclidean geometry, calculus, differential equations, linear algebra, and abstract algebra.

A standard definition of a function is:

A function is a relation between two sets that assigns each element of the first set to exactly one element of the second set.

More formally, if:

then for every:

there exists exactly one:

such that:

This “exactly one output per input” property is the key structural constraint that defines a function.

A Critical Asymmetry

It is important to emphasize a subtle but critical asymmetry:

• Multiple inputs may map to the same output (many-to-one is allowed)
• A single input may not map to multiple outputs (one-to-many is not allowed)

So it is perfectly valid that:

and

with

However, the following is not valid for a function:

and

with

This deterministic mapping structure is what makes functions so powerful. It forms the foundation for derivatives, integrals, transformations, and essentially all of continuous mathematics.

Random Variables and Probability

In probability theory, we introduce a different kind of object: the random variable.

A random variable is often defined as a function:

In this sense, a random variable is still technically a function.

The key difference is that we do not typically focus on the deterministic mapping itself. Instead, we associate the variable with a probability distribution that governs which outcomes are likely.

Rather than asking:

we ask questions such as:

or

for some set $S$.

In other words, probability theory shifts the focus from deterministic outputs to distributions over possible outputs.

Statistics then reverses the direction of inference: we observe samples and attempt to infer properties of the underlying distribution.

Why LLMs Behave Like Stochastic Systems

This distinction becomes especially important when thinking about modern generative AI systems such as Large Language Models (LLMs).

At a computational level, an LLM can be viewed as a function:

where:

• $p$ is the prompt
• $\theta$ represents the learned model parameters

However, the key point is that the model does not directly produce a single fixed output.

Instead, it produces a probability distribution, and the final response is typically obtained by sampling from that distribution:

This means that for the same input prompt $p$, multiple different outputs $y$ can be generated across different runs.

So while the underlying model parameters are deterministic, the output generation process is stochastic.

A Simple Experiment

Consider the prompt:

“Write a paragraph about how to pick your favorite color.”

Running this prompt multiple times will produce different—but still valid—responses.

Each response is typically:

• Coherent
• Relevant
• Grammatically correct

Yet the responses will not necessarily be identical.

This variability is not noise in the traditional sense. It is a direct consequence of sampling from a learned probability distribution over language.

A More Complicated Experiment

The following plot shows how responses vary from LLM to LLM. This graph was generated by using the same prompt 100 times in each of the LLMs listed. The resulting language-related qualitative metrics were then calculated. Some models have a “temperature” parameter that can increase or decrease the amount of randomness or creativity in the LLM response. For this experiment, the temperature value was set to 0 for those models that have that parameter available. This encouraged the model to be as non-stochastic as possible when generating responses.

From the plot above, we can clearly see two things. First, across all of the LLMs, the response to the prompt was stochastic. Second, different models demonstrated different amounts of variance in the quantitative metrics of the text that was generated. (Note that this experiment was limited specifically to the English language.)

The prompt used for the experiment above was: “Write a concise paragraph explaining the significance of reproducibility in machine learning experiments.” The system prompt was: “You are a helpful assistant.”

The Value and Risk of Stochastic Generation

This stochasticity is central to both the power and the risk of LLMs.

Benefits

Stochastic generation enables:

• Creative variation
• Idea generation
• Exploration of alternative phrasings
• Discovery of alternative solutions

Risks

At the same time, it introduces:

• Non-zero probability of incorrect information
• Inconsistency across runs
• Difficulty in guaranteeing correctness without external verification

Importantly, these risks do not disappear with increased scale or capability.

They are structural properties of the sampling process itself.

Implications for Deployment

In practice, this means LLMs should not be treated as deterministic, truth-preserving systems.

They are better understood as probabilistic generators that require:

• Validation layers
• External grounding (retrieval systems, tools, databases, APIs)
• Testing across distributions of inputs
• Monitoring for failure modes over time

This is similar in spirit to how humans are evaluated in safety-critical roles.

We do not assume correctness from a single response or test. Instead, we rely on:

• Repeated evaluation
• Certification
• Ongoing oversight
• Performance monitoring

Closing Thought

LLMs are not simply “right or wrong” machines.

They are systems that generate outputs according to learned probability distributions.

That makes them powerful.

But it also means their outputs should be interpreted as:

Useful suggestions drawn from a distribution, not guaranteed facts.

How Artificial and Biological Neural Networks Differ

Miles Porter
Feb 22, 2026

At a recent AI summit that I organized for my company, it became clear to me that not everyone fully understands the basics of generative AI, or artificial neural networks in general. As a result, now seems like a good time to pause and put down some of my thoughts on how artificial neural network systems differ from biological systems.

Some background: I studied applied mathematics at Colorado State University in the early 90s. One of the areas I focused on was pattern analysis and neural networks. Neural nets had been around for a while, actually going back to the 60s or before. What had changed in the 90s was that new advances in computing power were making it possible to run larger and larger neural networks. This was a time before cloud computing, and really even before the internet. Most of the networks that I worked on consisted of a few layers of fully connected nodes. However, their power seemed really amazing. One experiment we did was to train a neural network to learn the logistic map. I recreated that work using TensorFlow later here: https://github.com/fractalbass/simple_neural_net

Back in grad school, we didn’t have TensorFlow and Keras, so we had to write the network algorithms by hand. I spent hours writing for-loops in C and dealing with how to numerically implement the gradient descent algorithm — literally to the point of tears.

And, aside from some mild emotional scarring, I was left with a huge realization: mathematically modeled neural networks are VERY DIFFERENT from biological “wet” neural networks that I had studied in my basic biology classes in some pretty fundamental and very important ways. One of those has to do with how signals flow through artificial vs biological networks.

Synchronized vs. Non-Synchronized Networks

In a numerical neural network, nodes and synapses are modeled, but the flow through the network is governed by strict timing. Signals flow from layer to layer and node to node in a preset order, and that order is critical. When training a neural network, a signal is passed into the network and the output is observed at the other end. That output is compared to some expected training value, and the error is then “back-propagated” back through the neural network and used to adjust the parameters of each node in order to reduce the error just a little bit. Then a new pattern is passed in and the output observed. That output is compared to the expected training value and the error is, again, back-propagated through the neural network. This process continues over and over and over. There are some slight variations on this process — such as saving up all the updates and applying them all at once (batch mode), or tweaking the method of calculating the errors — but essentially the process remains the same. The point here is that it is a VERY STRUCTURED process. That is necessary because the algorithm for minimizing the error coming out of the network depends on the chain rule in calculus.

That $f(g(x))$ part is really important. Let’s say that I have a 5-layer neural network, and I want to update the weights in the top layer based on some observed error. To do that, I would need to do the following:

Which is a friggin’ mess.

But the important thing is that in order to do this, I need to compute partial derivatives for the 5th layer, then the 4th, then the 3rd, and so on up to the first layer, in order to update the weights in each layer. This is the core concept behind “back”-propagation of errors. If the network didn’t have this preset structure, this becomes very, very difficult to do. The network needs to fire completely through one time, then everything stops and the error is computed before optimization can take place.

Biological neural networks differ from artificial neural networks in some other critical ways too:

1. Biological neural networks use both electrical and chemical signals.
2. Biological neural networks have extremely complex topologies that are constantly changing.
3. Biological neural networks don’t have strictly synchronized digital timing.
4. Biological neural networks include neurons that fire at different speeds and delays.
5. Biological neural networks are decentralized and involve local plasticity rules rather than centralized gradient-based optimization.
6. Biological neural networks are analog systems, artificial neural networks are digital.

All of these factors combine to make back-propagation of errors essentially impossible in biological neural networks.

Looking Ahead

Most artificial neural networks in practice follow what I described above in terms of their structure and algorithms. However, there are a few exceptions. One of these exceptions can be found in a paper written in 1997, which proposed a “third generation” of artificial neural networks called “spiking neural networks”: Maass (1997) — “Networks of Spiking Neurons: The Third Generation of Neural Networks”

Spiking neural networks differ from traditional artificial neural networks in a couple of critical ways:

1. The neurons in a spiking neural network store up input signals until they reach some critical threshold and then fire.
2. The firing of a spiking neural network is binary (0 or 1). This kind of binary function is not differentiable, which means it cannot be optimized by gradient descent (the chain rule) or any other similar process.

Because of point 2, optimizing (training) a spiking neural network becomes a non-convex optimization problem — in other words, they are very difficult to train. However, quantum computing might provide a potential solution. Quantum annealing is an approach specifically designed for optimization problems. However, we are a long way from having this approach enter the mainstream of computing.

Conclusion

What I hope readers take away from this post is that artificial neural networks — which are the foundation of the current “AI” buzz — differ in some really dramatic ways from biological neural networks.

Stay tuned for more articles on topics related to data science and AI.

Notice: The content of this article was initially written by Miles Porter — a human. AI technology has been used to proofread and format the content. AI systems can hallucinate and make mistakes. It is important to verify and double-check the above content.

Intro

Responsible AI, as I have been thinking of it, is made up of 6 pillars:

• Fairness / Bias
• Reliability / Safety
• Privacy / Security
• Inclusiveness
• Transparency
• Accountability

Originally, I was going make this about all of the above, but let’s focus in something a bit more specific: Accountability. Typically, when I think of Accountability in RAI, I tend to think of “Making people accountable for AI”. And, that is definitely part of that pillar. I seem to be reading more and more about sustainability and how it is also part of accountability. Now, a TON of discussion and papers have been written about the sustainability aspects of training large language models. This series from MIT is particularly informative:

But, I believe that sustainability concerns (and frankly business concerns) don’t end there. I would like to explore two specific examples of how sustainability can play a role in GenAI at inference time.

Simple Example 1. Regression

I recently wrote a Jupyter notebook that did a comparison between using the deterministic approach to linear regression vs using an LLM. Now, you may be thinking something along the lines of “That is just crazy!” …and you’d be correct. I was curious, however, just how “crazy it was.” My experiment involved creating 1000 points that were normally distributed around a line. I wanted to see just how much difference there would be in computation time between using a Scipy based algorithm vs using an LLM (OpenAI O3). For this experiment, I went with GPT5. One of the first problems that I ran into was dealing with getting all 1000 points stuffed into the LLM input tokens, so just to make things more simple, I decided to narrow down the problem to using 10 points. My prompt was, basically, “Please use linear regression to determine the best fit line through the following set of points [(-3.00, -2.412), …]”. The results were so lopsided that I then decided to handicap the Scipy approach and force it to use all 1000 points. The following graph shows the different between the Scipy model and the LLM for accuracy and computation time.

So, the LLM approach was actually 58950 slower than the Scipy approach! I knew that it would be bad, but I didn’t anticipate just HOW bad. (I should note here that some of the LLM compute time was clearly spent sending the data back and forth to Azure OpenAI Foundry. However, since there is no real option to run O3 on a local machine, that is no way around it.

But, let’s consider a second example… doing sentiment analysis on a dataset and see what we come up with…

Simple Example 2. Sentiment

Bottom line. Do you have to?

What next?

Attend my presentation at the Trimble Dimensions Conference in Las Vegas at the Venetian on Nov. 10th.

Conclusion

Thanks, and see ya’ in Vegas?

- Miles

;)

In the movie The Princess Bride there is a scene with Inigo and Dread Pirate Roberts having a sword fight on the Cliffs of Insanity. Inigo presses Roberts to remove his mask and reveal his identity. When Inigo says, “I must know,” Westley replies, “Get used to disappointment.”

This is a great metaphor for working in data science. If you really are doing data science, then you need to “Get used to disappointment.”

ROC and AUC

These three letter acronyms are important, particularly in the context of building classification models. ROC or Receiver Operating Characteristic is a graph that shows the impact of threshold values on a binary classifier. Area Under Curve, also often referred to as “lift”, corresponds to the improvement that a model has over using a naive guess for the classification. Any value of AUC that is below .5 shows that your model is actually WORSE that a random guess.

Yes. It happens.

This is an actual AUC from a project that I am working on.

Yes. I trained a binary classifier that is actually “dumber than dumb.”

The Point

Thomas Edison once said

“I have not failed 10,000 times—I’ve successfully found 10,000 ways that will not work.”

“Finding ways that will not work” is part of experimentation. It is a critical part of all science, it is how we find insights, and how we learn.

That is all for this extremely short blog post. Carry on and keep finding ways that will not work!

• Miles

;)

In my previous post, I wrote about how matrix based collaborative filtering works and went through a simple example implementation using the Surprise package in python. Continuing on with that topic, I wanted to explore an alternative approach to this problem that uses neural networks.

Recall that our previous approach was essentially about creating a matrix that maps users and items. This matrix is made up of previously known ratings from users. In our example, the items were movies and the ratings were scores from 0.5 to 5.0. Since not all users have watched all movies, the matrix in question is very sparse. The goal, then, is to attempt to find a solution that fills in the gaps.

Using matrix factorization and singular value decomposition is just one approach to solving this problem. Another approach involves using neural networks, or more accurately multi-layer perceptrons or “MLPs”.

The MovieLens Dataset

To demonstrate how this works, I am again using the MovieLens 100k dataset that consists of users, movies and rankings. More information about the MovieLens dataset can be found on the GroupLens website.

Network Structure

Neural networks similar to the one in this example are frequently referred to as “deep learning”, but IMO it really is not. As we will see below, the network in use here has only a few layers. Deep learning networks typically have dozens or more layers and lots of parameters. For example, Yolo V8 has 53 convolutional layers alone, and GPT3.5 has 96 layers and 175B trainable parameters. These types of networks are DEEP! As we will see, the neural network in this example is very basic and very shallow.

To understand how the network is structured, let’s start with a simple diagram. The following diagram was created from the torchviz packaged. We can see that the network starts with two main sets of inputs, user_factors and item_factors.

The key thing to understand here is that an initial “embedding” is taking place in order to feed information into the network. This is done by creating a mapping from a given user and an embedding vector of some length. In the case of my code for the movie example, the embedding vectors have a length of 20. Initially, the values of the embedding vectors are randomized. As the network continues to train, back propagation is used to update these values. As training progresses, the embedding vectors start to “understand the latent structure” hidden in the data.

As with the previous example with collaborative filtering, the neural network approach also suffers from the “cold-start problem.” The cold-start problem refers to the difficult of providing accurate recommendations for new or “cold” users that have limited historical data. (Yes, I did lift that from ChatGPT.) Simply put, in order for us to do inference, we are going to have to have trained our model on at least some information for a given user and item.

Introducing Factor Biases

One of the challenges in building models like this is that users and items can suffer from biases. For example, one user might be particularly generous in their rankings compared to others. Or, certain items may have been reviewed more times than others. In the paper “Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model.” (Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ‘08), pp. 426-434. DOI: 10.1145/1401890.1401944), author Yehuda Koren presents a way to deal with this situation by introducing a vector for user and item biases. The addition of these bias vectors can be seen in the next diagram.

Note that the bias vectors are of length 1. They are basically a trained value that boosts or discounts the overall value of the prediction for a given user and item. Like the embedding vectors, these values are updated as the network gets trained.

Some Code

The following includes some code that will train a neural network to make ratings predictions based on the MovieLens 100k dataset. In this code, I have configured the system to run 450 training epochs. With my Quadro GP100 (which is a slightly older GPU), this process took about an hour.

import os
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader, random_split
from torchviz import make_dot
import logging

# Set up logging
logging.basicConfig(level=logging.DEBUG)

# Set the device (CPU/GPU)
device = torch.device("cpu")
if torch.cuda.is_available():
    device = torch.device("cuda")
    print(f"PyTorch is using GPU: {torch.cuda.get_device_name(0)}")
else:
    print("PyTorch is using CPU")

# Define a custom dataset class for MovieLens data
class MovieLensDataset(Dataset):
    def __init__(self, data_path):
        # Load data from a CSV file into a Pandas DataFrame
        self.data = pd.read_csv(os.path.join(data_path, "u.data"), sep="\t", header=None, names=["user_id", "item_id", "rating", "timestamp"])

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data.iloc[idx]

# Define a recommender system model using PyTorch nn.Module
class RecommenderSystem(nn.Module):
    def __init__(self, n_users, n_items, n_factors=20):
        super().__init__()
        # Embedding layers for user and item factors
        self.user_factors = nn.Embedding(n_users, n_factors)
        self.item_factors = nn.Embedding(n_items, n_factors)
        # Embedding layers for user and item biases
        self.user_biases = nn.Embedding(n_users, 1)
        self.item_biases = nn.Embedding(n_items, 1)

    def forward(self, user, item):
        # Perform forward pass through the model
        user_factors = self.user_factors(user)
        item_factors = self.item_factors(item)
        user_biases = self.user_biases(user)
        item_biases = self.item_biases(item)
        
        rating = (user_factors * item_factors).sum(dim=1, keepdim=True)
        rating += user_biases + item_biases
        return rating.squeeze()
    
    def visualize_network(self, user, item):
        rating = self.forward(user, item)
        dot = make_dot(rating, params=dict(self.named_parameters()))
        return dot

# Method for training the model for one epoch
def train_epoch(model, train_loader, criterion, optimizer, device):
    model.train()
    for i in range(1, len(train_loader) + 1):
        row = train_loader.dataset.dataset.data.iloc[i - 1]
        try:
            user = torch.LongTensor([row["user_id"]])
            item = torch.LongTensor([row["item_id"]])
            rating = torch.FloatTensor([row["rating"]])
            
            user, item, rating = user.to(device), item.to(device), rating.to(device)

            optimizer.zero_grad()
            pred = model(user, item)
            loss = criterion(pred, rating.squeeze().float())
            loss.backward()
            optimizer.step()
        except Exception as e:
            print("Something blew up in training on row #{}!".format(i))
            print("Row: {}".format(row))
            exit(1)

# Method for evaluating the model
def evaluate(model, test_loader, device):
    model.eval()
    total_loss = 0
    total_count = 0

    with torch.no_grad():
        for i in range(1, len(test_loader) + 1):
            try:
                row = test_loader.dataset.dataset.data.iloc[i - 1]
                user = torch.LongTensor([row["user_id"]])
                item = torch.LongTensor([row["item_id"]])
                rating = torch.FloatTensor([row["rating"]])
                
                user, item, rating = user.to(device), item.to(device), rating.to(device)
                pred = model(user, item)
                total_loss += ((pred - rating) ** 2).sum().item()
                total_count += 1 #pred.size(0)
            except Exception as e:
                print("Something blew up in testing on row #{}!".format(i))
                print("Row: {}".format(row))
                exit(1)
    
    return total_loss / total_count

# Method to display predictions
def display_predictions(model, test_loader, item_names, device):
    model.eval()
    with torch.no_grad():
        for i in range(1, 10):
            row = test_loader.dataset.dataset.data.iloc[i - 1]
            user = torch.LongTensor([row["user_id"]])
            item = torch.LongTensor([row["item_id"]])
            rating = torch.FloatTensor([row["rating"]])

            user, item, rating = user.to(device), item.to(device), rating.to(device)
            pred = model(user, item)
            item_name = item_names.loc[item_names['item_id'] == row["item_id"], 'title'].values[0]
            print(f"Item Name: {item_name}, Actual Value: {rating.item()}, Predicted Value: {pred.item()}")

def get_item(loader, n):
    row = loader.dataset.dataset.data.iloc[n]
    user = torch.LongTensor([row["user_id"]])
    item = torch.LongTensor([row["item_id"]])
    rating = torch.FloatTensor([row["rating"]])
    return user.to(device), item.to(device), rating.to(device)

# Main function
def main():

    data_path = "./data/ml-100k"
    # Initialize MovieLens dataset and item names
    dataset = MovieLensDataset(data_path)
    item_names = pd.read_csv(os.path.join(data_path, "u.item"), sep="|", encoding="latin-1", header=None, names=["item_id", "title"], usecols=[0, 1])

    # Get the number of unique users and items
    n_users = dataset.data["user_id"].nunique()
    n_items = dataset.data["item_id"].nunique()

    # Split dataset into training and test sets
    train_size = int(0.9 * len(dataset))
    test_size = len(dataset) - train_size
    train_dataset, test_dataset = random_split(dataset, [train_size, test_size])

    # Create data loaders for training and testing
    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False)



    # Initialize the recommender system model, criterion, and optimizer
    model = RecommenderSystem(n_users, n_items).to(device)
    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    # # Visualize the network
    user, item, rating = get_item(train_loader, 0)
    dot = model.visualize_network(user, item)
    dot.render("network.gv", view=True)

    # Train the model for multiple epochs
    num_epochs = 10
    for epoch in range(1, num_epochs + 1):
        train_epoch(model, train_loader, criterion, optimizer, device)
        mse_train = evaluate(model, train_loader, device)
        mse_test = evaluate(model, test_loader, device)
        print(f"Epoch: {epoch}, Train MSE: {mse_train:.4f}, Test MSE: {mse_test:.4f}")

    # Display predictions for a subset of test data
    display_predictions(model, test_loader, item_names, device)

# Entry point
if __name__ == "__main__":
    main()

Results

The above code, which I ran on my desktop machine generated the following output:

(I have shorted the output here to make it a bit easier to consume.)

PyTorch is using GPU: Quadro GP100
Epoch: 1, Train MSE: 29.7299, Test MSE: 33.6222
Epoch: 2, Train MSE: 22.9194, Test MSE: 30.3332
Epoch: 3, Train MSE: 17.4325, Test MSE: 27.6297
...
Epoch: 450, Train MSE: 0.0414, Test MSE: 7.3477

Item Name: Kolya (1996), Actual Value: 3.0, Predicted Value: 2.773867130279541
Item Name: L.A. Confidential (1997), Actual Value: 3.0, Predicted Value: 3.0126101970672607
Item Name: Heavyweights (1994), Actual Value: 1.0, Predicted Value: 1.3654597997665405
Item Name: Legends of the Fall (1994), Actual Value: 2.0, Predicted Value: 1.8913805484771729
Item Name: Jackie Brown (1997), Actual Value: 1.0, Predicted Value: 0.6543256044387817
Item Name: Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963), Actual Value: 4.0, Predicted Value: 3.741678237915039
Item Name: Hunt for Red October, The (1990), Actual Value: 2.0, Predicted Value: 1.5932667255401611
Item Name: Jungle Book, The (1994), Actual Value: 5.0, Predicted Value: 5.35903263092041
Item Name: Grease (1978), Actual Value: 3.0, Predicted Value: 3.002894163131714

Here we can see that the network did a pretty good job of nailing the MovieLens 100K dataset. It should be noted that to train the model on the data took around an hour for 450 epochs. The collaborative filtering approach that I used in the previous post was significantly faster.

The following graph shows the training curve for the network. In the interest of time, I cut the training off at 450 epochs. The performance of the holdout test set was still improving at that point.

Discussion

The MovieLens dataset contains 943 users and 1682 items. Clearly, this is a pretty small dataset. There have been a number of different approaches to dealing with this problem including the release of the Torchrec domain library for Pytorch. Torchrec provides a way to handle embeddings such that they can be better parallelized across multiple machines. This allows network training to be scaled up to account for larger and larger datasets. In 2019, Meta (the parent company of Facebook) released an open source solution for dealing with the problem. Information about that approach can be found here.

Conclusion

In this post, I have discussed implementing a neural network based recommender system. I discussed, at a high level, how the neural network approach in this post differs from the singular value decomposition approach that I covered in my previous post. I included some examples of the network architecture both in it’s basic form, and with an added “bias” feature vector. I included code written in Python that uses the pytorch framework and that can leverage a GPU. I shared some initial results of the code showing how the network performed on a random hold-out test set. Finally, I included a short discussion on directions for neural network based recommender systems.

I hope you have enjoyed this post! Until next time…

Miles

I recently went thru an advanced course on Google on collaborative filtering. It was good, but DANG. It was a little mathematically intense. I thought I would go thru a similar exercise here to show how collaborative filtering can be used to make recommendations… and give a very basic code example. For my example, I will use a MovieLens dataset similar to the one in the Google example. I will use some comments from the approach outlined in “Data Mining for Business Analytics” by Shmueli, Bruce, Yahav, Patel, and Lichtendahl (ISBN 978-1-118-87936-8). The approach specified in that text has examples in R, which are great… if you like R. For those that don’t, I will conclude this post with an example that uses the open-source Surprise package (Simple Python Recommendation System Engine).

First, some background:

As outlined in the Google course, recommendation systems often typically have a three phase approach. These phases include candidate generating, scoring and re-ranking. In the candidate generation phase, two approaches; content-based filtering and collaborative filtering are used. The goal of the candidate phase is to reduce the number of possible matches down to some reasonable number so that they can be scored.

For the sake of this discussion, we will use the terms “users” and “items” to represent the things that we are recommending to, and the things that we are recommending respectively.

Content-based filtering focuses on similarities between items. The example used in the Google tutorial is “If user A watches two cute cat videos, then the system can recommend cute animal videos to that user.” This approach has one big advantage in that the model doesn’t need to worry about other users. However, the model also requires that all the “items” have been tagged, and the quality of the results depends heavily on that tagging.

Collaborative-based filtering, on the other hand doesn’t require prior tagging. In this approach similarities between users drives the recommendations. One of the big challenges of collaborative based filtering include the fact that the model cannot include new items. This is referred to as the cold-start problem. There are a few techniques that can help address this and other issues including WALS. I am going to defer discussion on that approach for another blog post. Ultimately, Collaborative based filtering has one huge advantage over content based filtering, and that is that it can make recommendations that seem “serendipitous.” For example, a user might enjoy watching cat videos and the system recommends other videos on knitting. That association was “discovered” because other users that liked cat videos also liked knitting videos. (It could happen. Just sayin’.)

The Basics of Collaborative Filtering:

There are two flavors of collaborative filtering, user-based and item-based. In User based collaborative the algorithm follows these steps:

User-based collaborative filtering:

1. Identify those users that are most like the user of interest.
2. Consider only those items that the user of interest has NOT purchased or ranked yet.
3. Use the similar users items to recommend possible items to the user of interest.

Item-based collaborative filtering:

The user-based approach is straight forward, but suffers from a big problem if the number of users is very large. Namely, the first step can be very costly in terms of compute. A less expensive approach is to do the filtering based on the items instead. That approach has the following steps:

1. Identify the ITEMS that were co-rated or co-purchased by any user with the item of interest.
2. Recommend the most popular item or correlated item(s) among the similar items.
3. For any given user, look at their items… and let those items recommend other items.

For the purpose of this exploration we will continue with the item-based collaborative filtering approach.

A Little Math:

In order to determine the items in step 1, we need to come up with some metric to measure how similar items are. The metric often used for this is referred to as the Pearson correlation metric. Euclidean can also be used, but “does not perform well for collaborative filtering as some other measures” (Shmueli, et al. Pg 345.) This is explained in some detail in the Google tutorial mentioned at the beginning of this post, and I won’t elaborate on it further here in the interest of time. (Not that you haven’t fallen asleep already unless you are a super data analytics nerd like me. :) )

The Pearson Correlation metric can be expressed as…

(‘cause it ain’t a good blog post without out some painful LaTeX, right?)

That is great, but what does it mean? It is important to understand that what we are really building behind the scenes is a matrix that associates users with items. Our matrix, initially, is a very sparse because not every user has ranked every item. The goal of the recommendation system, then, is to attempt to fill in the blanks in the matrix. In order to do that, we can use a singular value decomposition approach (SVD) that generates three matricies, U, S, and V. The U matrix represents the users, the V matrix represents the items, and the S matrix is diagonal matrix that is comprised of singular value such that the product of U,S and V approximates the original ratings matrix… with the gaps filled in.

Doing this requires a good deal of code that can potentially be very non-performant. But, fear not… there are packages that make this entire process easy, performant and less error prone. One such package for Python is the Surprise package.

Doing This With Code (the easy way):

Fortunately, a number of packages make this process much easier than having to do it by hand. The Surprise python package is a popular package for doing recommendation. The following example shows how it can be used. This example loads data from the ml-100k dataset, which can be found here: (https://grouplens.org/datasets/movielens/100k/0)

The following code loads the movie recommendations, uses the Singular Value Decomposition (SVD) algorithm to solve the recommendation matrix (this is also discussed in the Google tutorial), and displays the results of the predictions of the first 10 records in the test set. (Note that there is a test/train split involved in the code.)

The data in the training set has values of 0.5 - 5.0 This indicates the “score” that the given user gave the given item. In some recommendation systems, this score could be binary.

The results are pretty accurate, typically within 1 of the actual ranking provided by the user in the training set.

import os
import pandas as pd
from surprise import Dataset
from surprise import Reader
from surprise import SVD
from surprise.model_selection import cross_validate
from surprise.model_selection import train_test_split

class Recommender:

    def movie_rating_prediction(self):
        # Load the data
        reader = Reader(line_format='user item rating timestamp', sep='\t')
        movie_data = Dataset.load_from_file('data/ml-100k/u.data', reader=reader)

        # Perform cross-validation
        algo = SVD()
        cross_validate(algo, movie_data, measures=['RMSE', 'MAE'], cv=5, verbose=True)

        # Train and test the algorithm
        trainset = movie_data.build_full_trainset()
        algo.fit(trainset)
        testset = trainset.build_testset()
        predictions = algo.test(testset)

        # Load movie titles
        movies = pd.read_csv('./data/ml-100k/u.item', sep='|', encoding='latin-1', header=None, usecols=[0, 1], names=['item', 'title'])

        # Display the item name, predicted rating, and actual rating for the first 10 rows in the test set
        print("\nItem Name, Predicted Rating, Actual Rating")
        for idx, pred in enumerate(predictions[:10]):
            movie_title = movies[movies['item'] == int(pred.iid)]['title'].values[0]
            print(f"{movie_title}, {pred.est:.2f}, {pred.r_ui:.2f}")


if __name__ == '__main__':
    recommender = Recommender()
    recommender.movie_rating_prediction()

Running this code produces the following results:

Evaluating RMSE, MAE of algorithm SVD on 5 split(s).

                  Fold 1  Fold 2  Fold 3  Fold 4  Fold 5  Mean    Std     
RMSE (testset)    0.9432  0.9384  0.9431  0.9277  0.9324  0.9370  0.0061  
MAE (testset)     0.7432  0.7379  0.7443  0.7331  0.7341  0.7385  0.0046  
Fit time          1.02    1.04    1.02    1.01    1.19    1.06    0.07    
Test time         0.15    0.14    0.14    0.14    0.20    0.15    0.02    

Item Name, Predicted Rating, Actual Rating
Kolya (1996), 3.91, 3.00
Mrs. Doubtfire (1993), 3.48, 4.00
Muriel's Wedding (1994), 3.50, 4.00
Shall We Dance? (1996), 4.17, 3.00
Stand by Me (1986), 4.14, 5.00
Ace Ventura: Pet Detective (1994), 3.66, 5.00
Mrs. Brown (Her Majesty, Mrs. Brown) (1997), 3.71, 4.00
Raising Arizona (1987), 3.62, 4.00
Being There (1979), 4.30, 5.00
Truth About Cats & Dogs, The (1996), 3.61, 4.00

Conclusion:

In this post, I have outlined some of the steps used in typical recommendation systems. We have discussed the difference between content-based filtering and collaborative-based filtering. From there, we took a look at some of the math (Pearson’s correlation) and also how Singular Value Decomposition can be used to generate recommendations. Lastly, we took a look at how the Surprise package in python can be used to easily implement a recommender system based on movie scores.

It should be mentioned here that the approach above is based on solving the user item matrix. Other approaches also exist, including using deep neural networks. I will save that discussion for another day.

I hope you have enjoyed this post! Until next time…

Miles

Source

Recently, I came across a (very short) book by Andy Weir called Randomize. You can get if for free from Amazon. I won’t spoil it for you, but a main theme in the book is random numbers… or really the illusion of random numbers when it comes to computers.

When I was reading this book (short story might be a better description), it occurred to me that infinity is to calculus what randomness is to statistics and probability. To clarify, being able to compute integrals and derivatives require the ability to handle and manipulate calculations involving infinity. In probability and statistics, we are often faced with random variables, and chance. In order to compute probabilities and confidences, we must be able to handle and manipulate calculations involving randomness, and random distributions of various types. And, just as in calculus and mathematical analysis where we have different kinds of infinity (countable vs uncountable), there are also different kinds of distributions of random numbers.

Not All Random is the Same

Statistics is the study of the properties of samples and using that information to make statements about populations. When people say “Pick a random number between 1 and 100…”, what they typically mean is to pick an integer between 1 and 100, inclusive, where each number has the same chance of being chosen. It turns out that there are other ways to pick random numbers, however. Consider this variation on the “Pick a number between 1 and 100 game.”

Pick 30 random numbers between 1 and 100, and compute their average and do that over and over again 100 times and write down the averages.” The list that you write down will be random… but it is a special kind of random. The Central Limit Theorem of statistics tells us that the list of 100 averages will follow a normal distribution. (The CLT requires that the samples must have a sufficiently large sample size, usually n>=30.)

What we are touching on here is the idea of distributions of random numbers. If I say pick a random number between 1 and 100, that is called a “uniform” distribution. However, when I look at the means, I am starting to approximate a different kind of random distribution called a “normal distribution”. There are many other kinds of random distributions.

Source.

Stochastic vs Deterministic

Random numbers underlie the concept of stochastic models and optimization. Random, or “stochastic” models stand in contrast to what is often referred to as “deterministic” models, which use a direct approach. To understand the difference, consider two methods for estimating Pi. The method that Archimedes used, which involves circumscribing triangles inside of a circle would be considered deterministic. In the Archimedes algorithm, you don’t need any kind of randomness to do the calculations.

In contrast, the so-called “Monte Carlo” method involves randomly adding dots to a square with a circle enclosed inside.

In this approach, you add as many random dots as you would like, and then count those that fall inside the circle. You can then use this formula to get an estimate for Pi.

This approach is considered stochastic as it requires the use of random numbers for the simulation to work.

Similar techniques were used by John von Neumann and Stanislaw Ulam to solve problems related to radiation while working on the Manhattan project.

Good Random vs Bad Random

“True” random numbers are not predictable. Also, sequences of true random numbers are not generally repeatable on demand. Stochastic algorithms would not be repeatable if they relied on truly random numbers. And, without repeatability, algorithms that use stochastic processes would be very difficult to test and results would be impossible to confirm.

Digital computers are deterministic machines. Digital circuitry alone is not capable of creating truly random numbers (unless it is malfunctioning). Nature, however, is full of sources of randomness. To come up with true random numbers computers use devices called true random number generators or TRNGs. These devices usually involve some physical and analog source of entropy. True random number generators play a critical role when it comes to cybersecurity. An interesting example of one elaborate TRNG was created by the cybersecurity company Cloudflare in 2017. You can read more about it here. For simplicity, speed and repeatability, computers typically rely on algorithms called pseudo-random number generators, often called PRNGs. PRNGs involve a random seed to start with. We will see some examples next.

IBM developed a PRNG in the late 1960s called RANDU. This once highly regarded PRNG used this formula:

Known as a linear congruential generator, or LCG, the above sequence provides seemingly random numbers as shown below (scaled to between 0 and 1).

One advantage of RANDU was that it was very fast to compute. However, to see the weakness of RANDU, consider each three consecutive values as coordinates in 3 dimensional space. The following animation shows the obvious issue RANDU has.

In addition to the clearly anti-random 3-dimensional distribution of RANDU, the PRNG also suffers from other issues. The following article does into detail on examining RANDU.

https://bkamins.github.io/julialang/2020/12/31/randu.html

Randomness and Regression

Randomness and normal random variables play an important role in regression, particularly when it comes to goodness of fit. Simple linear regression can be thought of as a best fit line thru a collection of datapoints.

When we do a regression, one of the things that we pay special attention to is the difference between the original data points that we are trying to fit a line thru, and the difference between those points and the best fit line. These differences are referred to as the “residuals”. When we do regression, if we can show that the residuals are normally distributed, it opens up the opportunity to make statements about how well our regression line fits our data. To make our statements, we need the residuals to be independent from each other and all the residuals need to follow the same distribution. If this happens, we can state how well we expect our regression model to do in inference. This, by extension, makes it possible for us to create confidence intervals around our regression line. Having a “well fit model” also makes it possible for us to make statements about our confidence in the values of our regression model coefficients.

Random Means Not Predictable

At Georgia Tech, I took a class on business analytics. It was an interesting, if not a bit scattered course, that covered aspects of finance, digital marketing, and a number of other topics. One of the best things about the class was the book… that we never used. (Not sure why.) The book was “Data Mining for Business Analytics”. ISBN: 978-1-118-87936-8 by Shmueli, Bruce, Yahav, Patel, and Lichtendahl.

Towards the end of the book, there is this quote… “Before attempting to forecast a time series, it is important to determine whether it is predictable, in the sense that its past can be used to predict its future beyond the naive forecast.”

In other words, sometimes things are just random based on what you know. DON’T WASTE TIME TRYING TO PREDICT THEM!

A time series where each term is the previous term plus some random noise is known as a “random walk”.

It turns out that there is a really easy technique to determine if a time series is a random walk, which involves fitting an AR(1) model. An AR(1) model (or ARIMA(1,0,0)) is simply a regression model where each term in the sequence is regressed back on the previous term. In other words, take all the pairs in the time series defined by:

If the slope of a regression line fit to the above pairs is 1, we have shown that the sequence is a random walk. Since this is a regression model under the hood, we can also look at the P-value for the coefficient in the underlying model to get a sense of how reliable that coefficient is. As we mentioned before, making those “goodness of fit” statements relies on the fact that the residuals are independent and identically distributed. While the values in our time series are NOT independent (each step in the time series depends on the last one), the difference between two successive steps should be. To see why, look at the definition of a random walk, and move the $ X_{n-1} $ term to the left side of the equation.

In the above, $ \epsilon $ represents random noise.

An Example

Here is an interesting example. Below, I have taken the daily closing values of a company’s stock from Jan 1, 2021 thru Oct 10, 2023.

Next, using the R programming language, I have fit an ARIMA(1,0,0) model (aka AR(1) as mentioned above) to the data.

As illustrated above, the coefficient is really close to 1. Continuing on to check the statistical reliability of the coefficient in the model, the P-value is nearly zero. This means that the coefficient is highly reliable and we are looking at a random walk.

The bottom line here is that we have shown the stock above to be an essentially unpredictable random walk. Any efforts to fit a model based on JUST THIS DATA is not going to work. Now, that does not necessarily mean that we couldn’t find a different model with different predictor variables. It just means that the sequence itself is not enough to go on when trying to do predictions.

Conclusion

So, the takeaways from this post are:

1. Random is to stats what infinity is to calculus.
2. There are good random (estimate pi and lava lamps) and bad random RANDU.
3. There are different kinds of random (distributions).
4. Randomness underlies our ability to state the statistical significance and confidence intervals of regression.
5. Some stuff is just random, so don’t waste your time trying to predict it.

I hope you have enjoyed this long overdue post. I wish everyone a wonderful holiday season, and a happy new year!

I have been working full time on Data Science since mid 2017 (5 years). During that time, I have made just about every ML mistake you can think of.

• Use training data in the test set? Yup.
• Created a great deep learning model only to loose track of the model file and have to retrain? Sure.
• Tried to retrain a model, but couldn’t re-find the hyperparameters that worked well? Done that.
• Used an open source autoML package that was based on Java and used Log4J that was shown to have major security flaws and then forced to retrain the autoML model again, but I still cannot remember what the first autoML model settled on? Yeah.
• Ran an inference model on a bare-metal EC2 instance that mostly sat idle, but has to have a boatload of RAM so it is way more expensive that it should be, even if it only “wakes up” part of the time. Yeah.

Heck, I am not proud, I can even say that I wrote a MONSTER single-threaded-and-super-slow python app that sucks data out of a cloud data service, chugs on it, and then shoves the results back into the cloud data service… from scratch.

So, yeah. I have made a ton of mistakes. In December, I completed my MS in Analytics from Georgia Tech. (See my earlier post.) That program helped me begin to understand what professional analytics is about. It turned me onto techniques like EDA, simulation, cross-validation, and feature engineering. I learned and relearned about the math and statistics of why things work, and why they sometimes don’t. My masters courses also helped me learn about behaviors and best practices that help reduce the likelihood of making process and workflow mistakes. But, even after all that, doing analytics and data science is hard. Doing it well, is very hard.

I am so happy that Microsoft has helped make it a little easier.

Now, if you would have said to me that I would write that line 5 years ago, I would have said you were crazy. I have used Microsoft products in the past, but it has always been with some reluctance. I don’t typically run Windows unless I cannot avoid it (I prefer to use Linux, or a Mac.) Being a certified Novell Network Engineer (CNE… you young kids can just Google that), I never quite got over the fact that Microsoft basically killed Novell.

I was somewhat skeptical when, in my current job, I found myself faced with evaluating Azure ML. I must say that Microsoft has done a very nice job of Machine Learning Platform as a Service. Add to that how far Azure has come… Well, if you’re working in Analytics or Data Science, it might be well worth your time to check out the platform. And… you can try it out for free!

Still need to be convinced why MSFT has developed a winning solution for analytics? Ok. I respect that. Let me offer a few details to consider. I’ll break this down relative to how I tend to think of the analytics process…

• Data
• Exploratory Analysis
• Model Training
• Experiments
• Model Training With MASSIVE Data
• Model Management
• Model Deployment and…
• Data

That said, lets begin with data…

Data

As you maybe guessed from the above, I believe that data science and analytics literally begin and end with data. It is really crazy to me how so many data science platforms don’t really do data justice. Azure ML has this covered by seamlessly integrating data into the ML environment. You can see this in the Azure ML Studio. If you are not familiar with Azure, and are with AWS (as I was) it works in sort of the same paradigm. Azure has a TON of services that they offer through their portal. (My favorite joke about AWS is that if Voldermort really wanted to hide a horcrux, he would have done it in the AWS portal.) Azure ML Studio makes using the various services involved in Azure ML easy. They have really simplified the “searching for the horcrux” problem well with ML Studio.

Accessing data in Azure is done with datastores and datasets. Datastores provide seamless access to data hosted in Azure in the form of blob storage, files, data lakes, SQL, Postgres and MySQL databases. Datasets are versioned sets of data used primarily for training models. ML Studio makes it seamlessly easy to find and use data you need for training models. I really like the fact that can VERSION data sets. Knowing what DATA you used to train a model is critical. Change the training data, and of course you are going to change the model. This is something that I don’t think is addressed enough. Sure, it is important to know your model hyperparameters… but what DATA you use to train the model is an equally important question.

Exploratory Analysis

Once you have data, a critical first step is to actually look at it. This is something that was really stressed in several of my Masters classes, and I agree. Don’t just assume that your data file is correct. ACTUALLY LOOK AT IT. This is easy in Azure ML Studio.

If you’ve imported the data, you can simply view the data. You can also generate a dataset profile, which will provide some nice basics descriptive statistics and graphs for your data set. You could do this in a jupyter notebook easily enough… but with Azure, you don’t even have to do that.

Model Training

Once you have understood your data, you will eventually want to start building some models. Azure ML Studio uses the Jupyter notebook paradigm. You can mix code and documentation in these notebooks as you would expect. The other option with Azure is to use their “no code” (which is really low code) training pipeline “Designer”. I remember when we looked at this in grad school, and one classmate commented on just how easy and intuitive using the Azure ML designer was for this purpose.

If you are looking to build a simple regression model, or a K-MEANS, or even a boosted tree model, the designer works very well. It also helps prevent mistakes like forgetting to USE THE SAME DATA CLEANING on your inference data that you used on your training data. (This is another data science anti-pattern that I have screwed up on in the past.)

Experiments

As I mentioned before, one of the key behaviors of a data scientist is keeping good records of what version of a model was trained on what data with what hyperparameters and what the resulting accuracy metrics were for the testing data. IMO, this is a key differentiator between a professional data scientist, and a citizen/amature data scientist.

There is another really interesting aspect to experiments in Azure ML. An Azure ML experiment can be created, and then ran and re-ran programatically. This opens up a great way to “script” machine learning training. To make this work, the training process needs to be captured in an .py file.

from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.core.runconfig import DockerConfiguration
from azureml.widgets import RunDetails

# Get the workspace
ws = Workspace.from_config()

# Create a Python environment for the experiment (from a .yml file)
env = Environment.from_conda_specification("experiment_env", "environment.yml")

# Create a script config
script_config = ScriptRunConfig(source_directory=training_folder,
                                script='diabetes_training.py',
                                environment=env,
                                docker_runtime_config=DockerConfiguration(use_docker=True)) 

# submit the experiment run
experiment_name = 'mslearn-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)

# Show the running experiment run in the notebook widget
RunDetails(run).show()

# Block until the experiment run has completed
run.wait_for_completion()

Check out the full code here for an example: https://github.com/MicrosoftLearning/DP100/blob/master/03B%20-%20Training%20Models.ipynb

Azure ML allows you to use either Azure’s own experiments framework, or ML Flow. ML Flow is an opensource project developed by Databricks. That said, it seems to me that the Azure framework provides a nicer UI, and more complete functionality. I have attempted to use MLFlow in the past, and it doesn’t do well in corporate environments with complex network topologies. Plus, MLFlow requires you to run an MLFlow server… why do that when Azure essentially offers better functionality without the headache of running a python based server.

Model Training with MASSIVE Data

Training and working with big data is a special challenge in Analytics. There are a number of different approaches including downsampling the data, or trying to scale up the compute. But, what do you do if you come across a dataset that is multiple terabytes or petabytes? An answer to this problem can be found in cluster computing, and Apache Spark is the leading opensource project in this area. Unfortunately, Apache Spark is notoriously difficult to configure and run. The complexities of running computing clusters where worker nodes alll have to be coordinated is a challenging task. AWS attempts to address this problem with ther EMR (Elastic Map Reduce) service… unfortunately that too can be very challenging to get working and very difficult to troubleshoot when it doesn’t. Databricks is a Spark as a Service offering (and company) that makes running Spark very easy. Databricks also offers data storage solutions that work well with their enhanced Spark solution that makes a ton of sense.

Now, the really nice thing about Databricks is that it is actually contained in azure, and accessible from Azure ML Studio…

Azure Databricks is part of the Azure ecosystem, and so it is billed through your Azure account (essentially as a “pay as you go” databricks subscription.) I cannot overstate the value of Databricks when it comes to processing huge data. Historically, Databricks was used for ETL tasks and large scale data engineering. However, it can also be used for machine learning tasks. Databricks has its own ML libraries that take advantage of the underlying Spark infrastructure which makes training models on huge datasets practical. Databricks also works efficiently with the underlying Spark compute clusters (which it gets from Azure compute), and allows you to scale down these clusters when they are not used. But, don’t forget to make sure that option is set because, trust me on this, there is no faster way to create a big compute bill than bring up a gigantic cluster and forget to shut it down over the weekend!

Model Management

Training models is the fun part of data science and analytics, and models that have been trained are assets. A lot of time and effort go into creating models, and that investment is manifest in the trained model artifact. That is why, from a business perspective, it is absolutely imperative that trained models are organized, currated, and the data associated with those models isn’t lost. A huge part of doing data science is running experiments, and seeing if new approaches to solve a data problem result in better results. But, how can you know what a better result is if you cannot put your finger on what your current model’s accuracy was when you trained it? Azure ML does an excellent job of helping not only keep track of models, but providing a way to walk back from a model running in production, to the description of the how the model was trained, to the actual code that trained the model, the data that it was trained on, and the hyperparameters associated with the training. This is all part of Azure model management and azure deplopyments.

I think that model management is something that is difficult to fully appreciate until you are faced with it in real life. In my early days of messing around with deep learning, I trained a model that had some really good results. At least, I seem to remember that it had really good results. After running the model in production for a few months, someone asked me what the training accuracy was. (Precision, Recall, F1, etc.) I didn’t have that information stored anyway, but I didn’t think it would be a problem. I would just retrain the model. I was shocked when I saw my metrics after re-training. They were nowhere close to what I remembered. I tried and tried, but I could never find the right combination of learning rate, optimizer, etc. that gave me what I needed.

Again, don’t be “ME”! Keep track of your models and you results. This is the path to enlightenment, and Azure ML makes this so easy.

Model Deployment

Once you have a model trained, and the accuracy is acceptable, you are good to go! Except that you are not. You are actually not going anywhere. It is like running 26 miles in a marathon, and not finishing the last 0.2 miles. Deploying the model is how you leverage the investment that you made by collecting and cleaning data, running experiments, and keeping track of results. If you don’t somehow get that model into a workflow within your organization, you have essentially just wasted a TON of effort.

Unfortunately, for most data scientists, that last 0.2 miles of the marathon is a merciless uphill climb. And, for how clever we thought we were with our elastnet feature selection, and our autoregressive integrated moving averages, and our eigenvectors and principal component analysis, we are left staring blankly at a docker build file that just makes no damn sense. DevOPs is a (black) art… at least to a data scientist. And… thank the maker for Azure endpoints. Azure ML provides a very straight forward way to deploy models into production. This can be done on a single compute instance, or into a kubernetes cluster, all automatically. The management tools are all there for monitoring the models. The security protocols have all been worked out so you don’t have to reinvent the wheel and fight tooth and nail with JWTs and OAUTH… unless you want to. And if you do, bless you. The approach also allows you to deploy models not only into production but also on your own workstation via Docker.

It is important to keep in mind that when you deploy a model, someone is going to call it. And, if they call it, they are going to pass in data and expect a result. That input and output data is another resource. It contains information about your customer and may yield insights. And, so we return back right to where we started…

(And it all comes back to) DATA

When a model is deployed in Azure ML, you have the option to capture insights from the model running in production. Turning on insights will allow you to capture data and metrics for how people are using the endpoints you have created. This is incredibly valuable. The distribution of the OUTPUT of you models is also extremely important. Most models are trained once, and then used over and over. They essentially know their world for a single point in time. However, the universe continues to change. As a result, the longer you run a model in production, the results will start to drift. This means that your model will, over time, become less and less reliable at describing, predicting and prescribing things based on inputs. It is important to watch the distribution of our models outputs in order to know where and how things are changing. Azure ML application insights allows for data capture and monitoring of model inputs and outputs.

A Few Comments About “The Future”

Azure ML Studio isn’t the only way to work with Azure. Microsoft has developed a fully functional SDK that can be used to configure and manage ML data, components, models, experiments, etc. As companies develop more ML models, keeping track of those models will only increase in complexity. Leveraging scripts are a good way to help keep this managable. This is one of the main ideas behind MLOps. I think that the cohesive SDK that Microsoft has developed is particularly valuable in this regard. That said, I appreciate having a GUI available that allows me to focus on solving the problems and not on trying to remember SDK classes and methods. So, there is a balance that needs to be struck. From what I can see, it seems that Microsoft is doing a good job in that regard.

Conclusion

I hope this post has highlights some of the key benefits of running Azure ML. I only touched on a few of them here. I would encourage anyone who hasn’t done so and considers themself a data scientist to check out the Azure ML offering. You can sign up for a free account with $200 credit. The account gives you 30 days to play with Azure ML. The registration does require a credit card, but they will not automatically start charging you when you run out of credits. (At least that is the message you receive when you sign up.)

I think you’ll be impressed by Azure ML. Seriously.

Regards,

Miles

What is past is prologue.  (Shakespeare, The Tempest, 1.2)

Introduction:

Everything that happens in our reality happens in the context of time. And, knowing some discrete or continuous measure of something is only meaningful in the context of when that measurement was taken. For example, consider $1000. That number means something very different today than it did in 1822. Because we exist, trapped in the constant flow of time, knowing what is comming next, or classifying what is happening in the present are highly valuable to us. It is of little wonder that finance, autonomy, and natural science are filled with problems related to time series analysis. It is ironic that, despite the fact that we live surrounded by problems involving time series, no uniform python packages have existed for time series modeling. While scikit-learn does offer some tools, those tools are very limited. In 2019, Franz Király, Markus Löning, Anthony Bagnall and Jason Lines began work on a python framework intended to follow the scikit-learn interfaces. Their framework includes tools and models for classification, forecasting, annotation, regression, and more. This post explores some of the tools provided in the sktime package for time series classification. It is worth noting that sktime goes far beyond just time series classification and includes tools for time series transformers, forecasting, and more.

Time Series Classification

So, what is time series classification? To begin with, we should probably define what a time series is. The term is almost completely self defining. In their book “Time Series Analysis and Its Applications” by Shumway and Stoffer (ISBN: 978-3-319-52452-8), the authors refer to time series as “…data that have been observed at different points in time.” (Note, the entire first chapter of this book is an excellent resource covering many topics and examples in this field of study.) A slightly more detailed definition comes from the NIST Engineering Statistics Handbook: “An ordered sequence of values of a variable at equally spaced time intervals.”

Mathematically, the definition of a time series can be written as:

$ y_t $ where $ t=(…-2,-1,0,1,2…) $

Source: Time Series Modelling, Inference and Forecasting by Prado and West.

According to Alexandra Amindon, Time series classification asks the question “Given a set of time series with class labels, can we train a model to accurately predict the class of new time series?” Amindon has written an excellent blog post on this subject on towardsdatascience.com.

Our goal here, however, is not to re-hash this excellent work, but rather to apply several time series classification algorithms offered by sktime on a particular time series classification problem.

Simulated Problem:

Time series classification, as the name implies, is a machine learning task that involves creating a model that can classify different time series into specific sets. In the movie, “The Hunt for Red October”, the characters in a submarine identify a sound made by another submarine. Their task is to identify the type of submarine based on the sound it makes. In this case, analyzing a discrete Fourier transformation of the sound would probably work well because the sound of the submarine propellers are very cyclic.

Now, let’s consider a different, and likely more difficult problem. What if we had a spaceship that was attempting to land on the surface of the moon. Our goal was to identify who was flying the spaceship based on the trajectory that the ship had while landing. This problem would be significantly more difficult than the submarine example simply because the trajectory would appear much more varied. To explore this specific problem further, it is possible to simulate the scenario of landing a spaceship on the moon, and training different “agents” as pilots.

Reinforcement Learning:

The OpenAI Gym provides a rich set of simulated environments that can be used to simulate/generate time series data. Once such environment is the “Lunar Lander” game. In this game, the goal is to safely land a simulated lander module on the surface of the moon. The simulation takes place in a simplified 2-Dimensional space. The game starts with a lunar lander module some distance above the surface of the moon. The player (agent) then uses thrusters (left, right, and up) to guide the lander to a pre-determined landing zone.

For all those millennials out there, trust what many of us born before 1975 already know. The task is much harder than it sounds!

The objective of the OpenAI gym environment is to provide an environment that can be used to train reinforcement learning models. The reinforcement learning model used for this project is a Deep Queue Network or DQN.

It should be noted that, in this case, the DQNs fly the lander based on a parameterized subset of the environment “state”, which includes the location and angle of the lander. (Note that the lander will rotate when the lateral thrusters are applied). This approach of using parameterized values is somewhat different from other reinforcement learning models that use the graphical representation of the game “screen.” Using the parameterized values reduces the dimensionality of the problem space considerably, and reduces the time required to train the model.

The OpenAI Gym Lunar Lander code is part of the OpenAI Gym project and is opensource and freely available. More information about the PyTorch DQN is available on their website as well. While it would be tempting to dive into these resources further, we will continue to just focus on the time series data generated by the agents trained using these tools and techniques. Code for the neural networks is available at the link at the end of this post.

The Data

As missions are flown by the novice and expert agents, the state values of the game are saved. Each mission is saved as an individual file. Each of these files, then, can be thought of as a multidimensional time series. For the purposes of this paper, we looked specifically at just two parameters of the state, specifically the X and Y values of the lander. The goal of this paper was to focus just on univariant time series data. Therefore, it is necessary to take the X and Y values and somehow map them into a single value. In order to create this mapping, we first derive the velocity of the lander in the X and Y directions. Once the velocity has been calculated, the magnitude of the velocity M of the lander is calculated based on the simple formula

M = sqrt(X^2 + Y^2)

Where X and Y are the velocity in the X and Y directions.

It is possible to use the same approach on the change of the velocity of the lander. This, of course, is the lander’s acceleration. It is also possible to use the same approach on the change of acceleration of the lander. The change of acceleration is also commonly referred to as the “Jerk”. For this exercise, velocity returned the best results.

Here is an example of the velocity for a random novice and random expert flights. Here the novice flight was flown by an agent that had 300 epochs of training data vs the expert that that 700.

Another interesting visualization of the data is to look at the mean and standard deviation of the time series plotted on a graph where the x axis is the mean and the y axis is the standard deviation. Here, each dot represents one time series or “mission”.

Once the data has been prepared for all of the agents, an 80%/20% train/test split will be used to divide up the data. While it would be possible to perform some type of cross validation on the data, the simple 80%/20% split will do for now.

The process:

For the proposes of comparison, the following classification techniques were considered:

• IndividualBOSS
• KNeighborsTimeSeriesClassifier
• TimeSeriesForestClassifier
• MrSEQLClassifier
• MatrixProfileClassifier

The following code is the main cell in the jupyter notebook that is responsible for training and testing the different models. This code is also available in the URL mentioned at the end of this post.

from sktime.classification.all import ColumnEnsembleClassifier, RandomIntervalSpectralEnsemble, RandomIntervalSpectralEnsemble, IndividualBOSS, KNeighborsTimeSeriesClassifier, ShapeletTransformClassifier, MrSEQLClassifier
import sktime.classification.all
from sktime.classification.feature_based import MatrixProfileClassifier
from datetime import datetime

classifiers = ['IndividualBOSS', 
            'KNeighborsTimeSeriesClassifier', 
            'TimeSeriesForestClassifier', 
            'MrSEQLClassifier',
            'MatrixProfileClassifier']

def factory(classname):
    try:
        cls = getattr(sktime.classification.all, classname)
    except:  #Unfortunate kluge for matrix profiles.
        cls = getattr(sktime.classification.feature_based, classname)
    return cls()

for c in classifiers:
    _ = factory(c)


print("Classifiers appear to be present.")

results = []
for classifier_name in classifiers:
    start = datetime.now()
    print("Working on: {}".format(classifier_name))
    start = datetime.now()
    classifier = factory(classifier_name)
    
    # Train the model
    classifier.fit(X_train, y_train)
    
    trained_t = datetime.now()-start
    
    start = datetime.now()
    
    # Predict results on test data
    y_pred = classifier.predict(X_test)
    
    inference_t = datetime.now()-start
    
    # Measure accuracy of predictions against "ground" truth.
    a = accuracy_score(y_test, y_pred)
    
    # Save resultes
    results.append([classifier_name, trained_t.seconds/len(X_train), inference_t.seconds/len(X_test), a])

df = pd.DataFrame(data=results)
df.columns = ['Classifier', 'Training_Time', 'Inference_Time', 'Accuracy']

print("Models trained.  Results in tabular form:\n")
print(df)    

Results

The following is the results of the testing process:

                    Classifier  Training_Time  Inference_Time  Accuracy
0                  IndividualBOSS          0.000            0.00     0.544
1  KNeighborsTimeSeriesClassifier          0.000            0.02     0.736
2      TimeSeriesForestClassifier          0.004            0.00     0.752
3                MrSEQLClassifier          0.069            0.00     0.580
4         MatrixProfileClassifier          0.003            0.00     0.448

If we limit the traning and testing to just the best and worst agents, the accuracies are actually much better:

                    Classifier  Training_Time  Inference_Time  Accuracy
0                  IndividualBOSS         0.0000             0.0      0.70
1  KNeighborsTimeSeriesClassifier         0.0000             0.0      0.84
2      TimeSeriesForestClassifier         0.0000             0.0      0.86
3                MrSEQLClassifier         0.0125             0.0      0.73
4         MatrixProfileClassifier         0.0025             0.0      0.69

Conclusion

As can be seen from the above, the different time series classifiers in sktime offered different accuracies and different training and inference times. The TimeSeriesForestClassifier appeard to provide the best results for this dataset. Another interesting graph to consider in this experiment is the accuracy vs performance (training and inference time) for the different models.

This experiment only scrapes the surface of the tools available in the sktime toolbox. I encourage you to check out sktime on your next time series project!

References:

https://www.frontiersin.org/articles/10.3389/frai.2021.699448/full
https://www.sktime.org/en/stable/

Source Code

Source code is available at the url below. (Note the jupyter notebook for timeseries analysis is in the /game/examples/agents directory.):

https://github.com/fractalbass/lunar_lander_ml

When I started this blog, the first article I published was titled “What is data science?” Now that I have some experience in the field, I thought it would be interesting to explore my current perspective on some commonly used, but frequently undefined terms. The terms that I am choosing to focus on are the following:

• Analytics
• AI (Artificial Intelligence)
• Machine learning
• Supervised, unsupervised and semi-supervised learning
• Deep learning
• Simulation
• Probability
• Statistics

Since my first post, I have discovered that people have their own fairly unique perspectives and “mental map” of how the terms, techniques and disciplines above interrelate. The following is the mental map that I currently use. In many cases, the terms here overlap with each other. There are some terms that are, perhaps, missing altogether such as Optimization. I fully realize these facts. I further realize that, as these fields mature and new fields emerge (such as quantum computing), my map will likely change to reflect the “state of the art and science.”

A single image is worth a thousand words.

An Analytics Perspective

Comments about the diagram…

As mentioned above, I choose to use the all encompassing term of “Analytics”. My choice of this term is based on the fact that I have a Master of Science Degree from the Georgia Institute of Technology… in “Analytics”.

At the current time, I see Analytics consisting of Artificial Intelligence, Simulation, and Probability and Statistics. I am considering replacing “Simulation” with the broader term “Operations Research.” I think this substitution will allow me to more clearly place “Optimization” on the diagram.

I view Statistics and Probability as ways to describe the relationship between a population and samples of the population. In probability, we often make statements about the likelihood of some event occurring based on some preconditions. For example, what is the probability that I will pick a green marble out of a bag that contains 10 green marbles, and 10 red marbles? In the reverse case, we might have, say 20 individual samples that we have drawn at random for a big bag of marbles. From these samples, we may want to think about how many of a certain color of marbles there are in the overall population. When we talk about going from populations to samples, we use probability. When we make inferences about populations based on samples, we use statistics.

To me, the term Data Science is redundant. The scientific method involves collecting data by conducting experiments in order to prove or disprove a hypothesis. Based on this definition, it seems to me that for anything to be considered science, it is also data science. You cannot take the data out of science without it becoming something else.

Wikipedia defines data science as: “Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from noisy, structured and unstructured data,and apply knowledge and actionable insights from data across a broad range of application domains. Data science is related to data mining, machine learning and big data.”

Yeah, ok. To me, the above just seems like an attempt to glorify the term “data science.” That said, I am ok with someone saying “I am doing data science!”, if they are using data and following the scientific method.

The fact that I am being a bit negative about the term “data science” on a blog that is hosted at datascience.netlify.app isn’t lost on me. In full disclosure, I should probably also mention that my current job title is “Lead Data Scientist”. When I first started writing this blog, I was very excited about the term “Data Science”. As I have gained more knowledge in this space, I have also come to realize how huge it is and how much there is that I don’t know. I am tempted to say that, if I had it to do all over again, I may not use “datascience” in the name of my blog. That, however, wouldn’t be truthful. The term “data science”, for better or worse, seems to have stuck. Call it marketing, I guess. I plan on continuing to write about topics in Analytics on my blog: “datascience.netlify.app”!

Disclaimer

Again, this “mental map” is based on my academic work and what I see in my career. It likely will not fit with everyone else, and I am just fine with other definitions.