The Hierarchy of ML tooling on the Public Cloud

1 ML Services on the Public Cloud

Not all ML services are built the same. As a consultant working in the public cloud, I can tell you that you are spoilt for options for Artificial Intelligence (AI) / Machine Learning (ML) tooling on the 3 big public clouds - Azure, AWS, and GCP.

It can be overwhelming to process and synthesize the wave of information; especially when these services are constantly coming out with new features.

Just imagine how much of a nightmare it would be to explain to a layman which platform to choose, and why you chose to use this particular tool to solve your machine learning problem.

I’m writing this post to alleviate that problem statement for others, as well as for myself, so you walk away with a succinct and distilled understanding of what the public cloud has to offer. For the sake of simplicity, I will use the terms AI and ML interchangeably throughout this post.

2 Building a Custom ML System… should be a Last Resort

Before we jump into tooling comparison, let’s understand why we should even use managed services on the public cloud. It’s a valid assumption to question - Why not build your own custom infrastructure and ML model from scratch? To answer this question, let’s take a quick look at the ML lifecycle.

The below diagram depicts a typical ML lifecycle (the cycle is iterative):

As you can see, there are many parts to the entire lifecycle that must be considered.

A famous paper published by Google showed that a small fraction of the effort that goes into building maintainable ML models in production is writing the model training code.

This phenomenon is known as the hidden technical debt of ML systems in production, and also what has been termed by industry as Machine Learning Operations (MLOps), which has become an umbrella term to refer to the mentioned technical debt.

Below is a visual explanation to support the above statistics, adapted from Google’s paper:

I won’t go into a detailed explanation of each stage in the lifecycle, but here’s a summarized list of definitions. If you’re interested in learning more, I would recommend reading Machine Learning Design Patterns Chapter 9 on ML Lifecycle and AI Readiness for a detailed answer.

This section illustrates that writing the model training code is a relatively tiny part of the entire ML lifecycle, and actual data prep, evaluation, deployment, and monitoring of ML models in production is difficult.

Naturally, the conclusion is that building your own custom infrastructure and ML model takes considerable time and effort, and the decision to do so should be a last resort.

3 Hierarchy of ML tooling

Here is where leveraging public cloud services come in to fill the gap. There are broadly two offerings these hyperscalers package and provide to customers:

3.1 ML Platform 🪛

We will first discuss the ML Platform before discussing AI services. The platform provides auxiliary tooling required for MLOps.

Each public cloud has their own version of the ML Platform:

• Azure - Azure Machine Learning
• AWS - Amazon SageMaker
• GCP - Vertex AI

Who is it for?

Persona-wise, this is for the team who has internal data scientist resources, want to build custom state-of-the-art (SOTA) models with their own training data, and develop frameworks to do custom management of MLOps across the ML lifecycle.

How do I use it?

Requirement-wise, the business use case would need them to engineer a custom ML model implementation that AI services in Section 3.2 do not have the capabilities to meet.

What technology capabilities does the service provide?

The following table shows the technology features of each cloud ML platform:

3.2 AI Services 🧰

Next, we will discuss AI services. They enable ML development using a low-code / no-code approach, and mitigate the overhead of managing MLOps.

The over-arching argument for these services is neatly put below by Jeff Atwood:

Who is it for?

Persona-wise, these are for the teams who DO NOT HAVE EITHER:

1. Internal data scientist resources.
2. Own training data to train a custom ML model.
3. Investment of resources, effort, and time to engineer a custom ML model end-to-end.

How do I use it?

Requirement-wise, the ML business use case would be met by cloud provider AI service capabilities.

The goal is to add ML features into the product by leveraging hyperscaler base models and training data; so the team can prioritize core application development, integrate with the AI service via retrieving predictions from API endpoints, and ultimately spend minimal effort on model training and MLOps.

What technology capabilities does the service provide?

--- Speech ---

--- Natural Language ---

--- Vision ---

--- Decision ---

--- Search ---

4 Further Reading on Topics

We have covered considerable ground in this post regarding the spectrum of ML services the public cloud offers, however there are still other concepts that we have to consider when building an ML system.

I would encourage you to explore and find your own answers to these concepts that were not discussed as AI / ML become more deeply embedded within the products we use.

What ML tooling do the 3 public clouds offer to implement the following functionality?

• Model data lineage and provenance
• Model catalog
• Human review for post-prediction ground truth labeling
• Models that work on video data
• Models that do generic regression and classification

Special Thanks / References

A special mention and thanks to the authors and creators of the following resources, that helped me to write this post:

ML Tooling

• 📚 Pick your AI / ML path on Google Cloud

AI Services

• 📚 What is Azure Machine Learning?
• 📚 How Azure Machine Learning works: Architecture and Concepts
• 📚 Amazon SageMaker features
• 📚 Vertex AI
• 📚 Vertex AI Documentation

ML Platform

• 📚 Choose a Microsoft Cognitive Services Technology
• 📚 Azure Cognitive Services Documentation
• 📚 What is Azure AutoML?
• 🎦 Azure OpenAI Use Cases
• 📚 AWS Machine Learning Services
• 📚 AWS AutoML Solutions
• 📚 Google AI and Machine Learning Products
• 📚 Google AI and Machine Learning Solutions
• 📚 Google Cloud AutoML