What is multimodal AI? Definitions, use cases, and examples

Multimodal artificial intelligence (AI) is a form of artificial intelligence that enables the use of a combination of images, voice, text, video, and more to make predictions or generate entirely new content. This goes beyond the more prominent use cases relying primarily on text. Multimodal AI answers questions that require specialized answers in the moment, with the right media and in the right channel a customer chooses, with the same meaningful context one would expect from a human interaction.

How does multimodal AI differ from single-modal AI?

First, things first, what is modality? Modality in this case refers to data types – including many of the different types listed below:

• Images
• Voice
• Text
• Video

Single-modal AI is the application of AI along just a single format or type of data. For example, a common use case for single-modal AI is creating a chatbot that allows a customer to send text messages back and forth with a brand. These single-modal applications can commonly be found on websites, apps, and other branded properties.

Why is multimodal AI important?

Multimodal AI represents a significant shift in the natural and contextual aspects of communication with AI models. In other words, multimodal AI makes chatbots, knowledge assist tools, and many other AI applications feel that much more human.

In the modern communications environment, people have various senses involved at all times. Think of how you communicate with family and friends on your cellphone – seamlessly switching between calls, texts, videos, and pictures throughout the course of an ongoing conversation. Each one of these channels introduces valuable context to help process and understand conversations.

For example, imagine one of your friends recently adopted a new dog. First, you might excitedly call your friend as soon as you find out to learn more about the dog. While you’re on the phone, he sends you a picture so you can see what the dog looks like. Later on, maybe the dog does something cute, and your friend decides to send you a video.

In an ideal world, our customer support conversations would function the same way. Multimodal AI supports more robust communication with customer support AI interfaces – seamlessly moving between the modes that make the most sense for a specific use case or request. Picture this. Instead of describing the issue you are having with a new product, you can show it to the chatbot and receive direct feedback based on that image.

Let’s take a look at a quick example of a multimodal AI chat interface in action:

How multimodal AI works

When it comes to putting together a multimodal AI solution like the ones we’ve illustrated in this blog, there are three primary steps: input, model processing, and output. Let’s apply these steps to the bike example from before.

Step #1: Input

In this first step, the data is ingested and processed. In a multimodal solution, these data types would each be ingested by separate neural networks. In the case of the bike unboxing, the customer might prompt the virtual agent with a text-based question asking if the bike’s parts had been installed correctly or if changes needed to be made, along with a photo of the parts in question.

Step #2: Model Processing

In step two, the data is combined and the important aspects of each data type are pulled and compared to trained data models that allow them to generate a highly accurate assessment of the bike and create a relevant response.

Step #3: Output

In the final step, the model delivers predictions, decisions, and recommendations that leverage all of the data, and then present those outputs back to the customer. In this case the bike parts have been installed correctly, and the user is ready to go for a ride!

Where multimodal AI can support stronger customer experiences

While there are plenty of use cases for powerful multimodal AI strategies to have an impact on the customer experience, two use cases stand out: virtual agents and knowledge management.

Virtual Agents

Many of the examples we have shared so far fall into this category. Multimodal AI stands to make virtual agents much smarter and more versatile than their single-modal counterparts. In addition to pulling from a greater data set of information to help customers troubleshoot their problems, multimodal AI also empowers customers to ask questions in different ways, where text responses might not cut it.

For example, remember the bike unboxing video we watched earlier? If a chatbot is trying to assess that the chain and derailleur are installed correctly, it could walk through a set of step-by-step questions using the customer’s own eyes to verify the installation. Or it could request a picture that is then compared to correct installation pictures in its database to verify the workmanship quickly and seamlessly.

Knowledge Management

Multimodal knowledge management solutions function very similarly to this virtual agent example, with one small difference. In these cases, an agent likely operates as the intermediary between the customer and the solution. At the most basic level, a customer calls in with a question, the agent interacts with the knowledge management interface, and then pulls from a variety of information types to help triage and solve customer questions.

How to decide if multimodal AI is right for you

There are a few considerations that can be helpful to walk through before deciding if multimodal AI is right for your customer support environment. You will want to answer:

• Do your customers currently experience friction when they engage with your support operations? If so, what does that friction look like?
• Do you have access to enough robust data modalities to support this type of strategy successfully?
• Are your customers’ inquiries complex enough to require additional context beyond text interactions?

Thinking through questions like these can help to unearth opportunities where additional video and audio inputs could begin to enhance your customer experience by achieving outcomes like faster call resolutions, greater customer satisfaction, and more productive agents.

Why TTEC Digital is a strong partner for multimodal AI implementation in the contact center

TTEC Digital delivers powerful customer experience solutions at the point of conversation. By combining deep CX consulting expertise with decades of experience innovating on the leading contact center platforms, TTEC Digital is uniquely qualified to deliver custom AI strategies that drive agent productivity, customer satisfaction, and real business value.

Our data and analytics team offers an expansive knowledge of artificial intelligence and can provide a proven roadmap for implementation that supports contact center teams with all different levels of AI familiarity. No matter your objectives, our data and analytics experts will help you identify how prepared your company is to leverage AI, gain a foundational understanding of AI, build an action plan with our AI roadmap, and help guide you along the way.