Navigating the Complexities of ML-Powered Product Development
Written on
Chapter 1: The Challenge of Deploying ML Models
Deploying machine learning (ML) models can be an intimidating task, but it is essential for product development.
This article is part of a series that delves into one of the most daunting aspects of operational ML: successfully putting models into production. In earlier discussions, we explored two distinct strategies for creating ML-driven applications. Initially, we analyzed the benefits and drawbacks of hosting your model alongside your web server. While keeping the model integrated with your core business logic facilitates rapid experimentation and feedback from a select group of testers, this approach often falls short in a production environment.
The first method we reviewed is a straightforward technique that allows us to decouple the model from our web server. We examined the model-in-database strategy, highlighted suitable use cases, and weighed its advantages and disadvantages. This method is easy to implement, but it can lead to outdated models and may not consistently deliver value to users.
So, how can we address the limitations posed by these earlier methods? This article discusses the prevalent architectural design that involves deploying a dedicated inference server for your model.
Inference Service Architecture
Establishing a separate web service for our models introduces additional infrastructural complexity, yet, as we will discuss, it often represents the ideal solution for many ML-powered products.
Learning Rate is a newsletter for those curious about AI and MLOps. Subscribe to receive monthly insights and updates on the latest trends and articles.
The Model-as-a-Service Framework
The Model-as-a-Service (MaaS) framework enables us to operate the model through its own inference service, akin to any other backend system tailored for executing ML models. This service is supported by suitable infrastructure, comprising one or more machines capable of maintaining optimal performance for one or several ML models.
Our backend, which is our web server, functions as the client: it sends requests and receives responses. For instance, when it gets a request from a user to identify objects in an image, it forwards that request to the inference server. The model processes the request, returns a prediction, and the web server then relays that response back to the user.
As mentioned earlier, this approach seems to strike a balance for numerous ML-powered applications. But what makes it so effective? Let’s explore this method in detail, including when to utilize it and its various benefits and drawbacks.
Advantages of the MaaS Approach
The MaaS approach is arguably the most prevalent method for deploying ML models, and its popularity is well-founded.
Firstly, if there’s a bug within your ML model, it won’t necessarily compromise your entire application. This allows you to manage errors gracefully while continuing to provide reliable service to users.
Moreover, operating a separate inference service lets you select the best hardware for your ML model and scale it as necessary. Whether you need a GPU accelerator or plan to scale your server vertically or horizontally, this method allows for flexible scaling without disrupting other components of your application.
Lastly, the MaaS framework offers versatility. Your model becomes a standalone web service, enabling you to reuse it across various applications or even publicly. You can develop an API around it and potentially monetize it, similar to OpenAPI.
Drawbacks of the MaaS Approach
A significant downside of the MaaS framework is the increased latency it introduces. When your server communicates with your model, it does so over the network, which can slow down the system. It’s crucial to assess how this latency impacts your application.
Additionally, this method increases infrastructural complexity, as you must design, operate, and maintain a separate service.
Conclusion
In this article, we explored a third architectural paradigm that aids in the design and implementation of ML applications. The Model-as-a-Service approach represents an optimal solution for many ML-powered products, and there are compelling reasons for this.
The downsides of previous methods are substantial. It’s essential to scale your models independently and ensure they are adaptable enough to handle varying types of data requests.
In the next article, we will examine the requirements for running a robust, production-ready inference service on a Kubernetes cluster using KServe.
About the Author
My name is Dimitris Poulopoulos, and I work as a machine learning engineer at Arrikto. I have developed AI and software solutions for notable clients, including the European Commission, Eurostat, IMF, European Central Bank, OECD, and IKEA. If you're interested in more insights on Machine Learning, Deep Learning, Data Science, and DataOps, feel free to connect with me on Medium, LinkedIn, or Twitter @james2pl.
The first video, "658: How to Build Data and ML Products Users Love — with Brian T. O'Neill," discusses strategies for creating ML products that resonate with users.
The second video, "The framework for great ML products," outlines key principles for developing successful ML applications.