What is AI inference?

How does AI inference work?

AI inference is the part of the ML process where ML models ideally start delivering useful output to users. At a high level, the process of applying inference to a model input works through a three-step process. The first step is normalization, the second is execution, and the third is output.

1. Normalization

Before a model is ready to process an input, the model or the code that supports it needs to normalize the input into a format that it knows how to process. For instance, if the ML model is a LLM, then the data is converted into tokens via a tokenizer.

For an image-processing model, the size of the image may be converted to a standard size that is identical to what the model was provided with during the training process.

The normalization process prepares data so that the ML model can process it cleanly, quickly, and efficiently. By getting the data as close as possible to the format the model expects, the software maximizes the likelihood of returning high-quality, useful output for the user.

2. Execution

Once the data has been normalized into a format that is similar to what the model was trained on, the second step of the process is model execution.

The simplest way to describe this step of the process is that it is an extremely complicated game of pattern matching across a very high number of parameters. Parameters are decision nodes that are developed during the model training process.

Modern ML models can have trillions of these parameters. As the model executes, the data is passed to these parameters and evaluated to identify whether the input matches the pattern that led to that parameter during the training process.

It is important to note that at this phase, the model does not “learn” from any particular message. ML models primarily learn during the training phase.

Models can be improved during the inference phase by improving the context provided to the model or by updating model weights in the running model. But if neither the model nor a human changes the weights of the model based on responses to the outputs, the model’s output will not change substantially.

3. Output

This is the part of the AI inference process that is helpful to the user. After the model has gone through the execution phase by fully processing the input, the model now offers output to the user. Output can appear in several different formats.

LLMs often return text, familiar to many users through tools like ChatGPT. Other tools like Stable Diffusion instead offer an image in response to the user’s input. More traditional ML models offer more technical responses.

For instance, an insurance company may take dozens of inputs related to a customer requesting a quote and run those through a ML model to answer the question of how much lifetime profit selling a policy to that customer will return to the company.

As noted in the previous section, just because the training phase is over doesn’t mean that performance of the model can’t be improved during this phase. Model providers will often ask for feedback on the output of a model in order to further tune behaviors of the running model.

The four phases of AI models

Inference is a critical step of the AI model process, and understanding the whole ML model lifecycle is helpful to see where inference fits in the flow.

1. Training

The first step of any ML model’s lifetime is the training process. Training a ML model involves providing a very high amount of classified input. The classification part is important: that’s what the model uses to learn about the information it processes.

An easy example of training is to imagine a model used to classify images of cats. Training this model involves feeding the model many pictures, along with tags of what that image contains. Examples are simple tags like “cat” but also more specific tags like “calico” or “tabby.”

Training more capable models involves feeding as many instances of classified data as possible to the model.

2. Fine Tuning

After a model is trained, fine tuning the model output is a key part of the process. This involves tuning model weights to provide better output.

Training the model is the most computationally intensive phase of the operation, as the model must work through enormous amounts of data. But fine tuning is often the most time-consuming process of the model development process.

Fine tuning may continue even after the model is in production and serving results to users.

3. Inference

Inference is the point where the model is regularly operating on novel data that was not included in the training corpus. This doesn’t necessarily mean that the model is publicly available or serving responses in production.

As noted, the inference phase often overlaps with the fine-tuning phase, as different parts of the model’s capabilities are improved to provide better outputs.

4. Serving

This is the phase of the AI model where the model moves to production phase, or public availability. For teams that deploy AI models, the serving phase involves deploying and scaling these models in ways that make them as performant as possible for users.

For a production machine learning team inside an enterprise, this might be as simple as deploying the model to AWS SageMaker. For a team like Anthropic, deploying a new model of Claude might involve building massive new data centers to handle the demand for their model.

Types of AI inference

There are two primary forms of AI inference:

1. Real-time inference
2. Batch inference

Real-time inference is a process that happens, as the name suggests, in real time. A user feeds a set of inputs to the model, and those are processed immediately. Then the model returns an output.

Batch inference doesn’t happen in real-time but instead involves collecting a large number of inputs and then processing them in as a large batch.

Real-time inference is helpful for iterative work where a user is attempting to perform work by continually refining their inputs to the model. Batch inference is more useful when a user has a large amount of data to process and doesn’t need responses immediately.

Benefits and use cases

The use cases for AI inference are widely varied. Billions of people have interacted with chat interfaces from companies like OpenAI and Anthropic. Scientists use ML models to classify novel data while performing research or developing mathematical proofs.

Software developers the world over will talk at length about the way that AI inference has completely reshaped their workflows, allowing for developing more code, more quickly, in millions of contexts.

Machine learning is a relatively new technology compared to computing as a whole, so many use cases are still being discovered. What was possible ten years ago is a fraction of what we can do today. It’s likely that ten years from now, we’ll look back and think the same thing.

AI inference challenges

AI inference comes with many challenges for developers to overcome. Here are just a few:

1. Overfitting

Overfitting is the term we use when a model gives too much deference to training data during the inference phase. A model that overfits on training struggles to process novel information, reducing usefulness.

While the defect originates during the training phase, it is identified during the inference phase and leads to poor results for users.

2. Scaling

Model training is the most computationally-intensive part of the process, but inference still requires many computing resources. The more complicated and larger the model, the more resources required.

3. Hardware availability

AI inference requires specialized computing hardware which is costly and time-consuming to produce. As the demand for AI inference capability has exploded around the world, sourcing and deploying this hardware has been a major challenge for any company relying on ML models.

4. Latency

In any real-time inference context, latency is a key consideration. Modern AI models are tremendously complicated, and the more time spent computing a response, the more useful that response is likely to be.

However, a model which takes too long to respond is one that users will be reluctant to use. Finding the correct balance is a major challenge for developers.

AI inference is at the center of tomorrow’s technology

AI computing has transformed the way that billions of people interact with computers. That transformation is showing no signs of slowing down, as frontier models from companies like Google, OpenAI, and Anthropic are pushing the boundaries of what computers can do.

No matter how large or small the model is, inference is a critical step in every AI workflow, for every AI user. As we discover new use cases, inference will be at the forefront of every new advance.

Common questions about AI networking