Role

Product Designer

Duration

3 weeks

Team

Product Management

Data Science

IT 

Skills

UX research

XFN communication

Phone Ticketing Conversion

Adoption Experiments: Growth and Adoption Pod

When I began my time at Moveworks, I was tasked with conducting adoption experiments. What are adoption experiments? Fast-paced, impactful, low-cost, and scalable growth experiments that stem from adoption gaps in the product.

​

This case study focuses on increasing the adoption of Movework's ticket filing via the chatbot.

Moveworks is enterprise software that enables employees to receive instant IT, HR, and Finance help within chat platforms (e.g. Slack, Teams, etc.) via an NLU conversational AI chatbot experience.

Current State

Phone Ticketing 

At companies, employees can file IT tickets on many channels including email, phone, a self-service portal, walking up to an IT agent, or with Movework's bot. Habits and ticketing culture are different from company to company, but the phone is often one of the most popular methods of filing a ticket. This is because of the convenience and ease of talking to someone and explaining the problem verbally. One adoption focus is converting employees who file tickets on other channels (email, phone, a self-service portal, etc.) to file tickets via the Movework's bot.

Research Discovery

A key part of the discovery research for adoption experiments is studying user behavior. To get an understanding of users I read through hundreds of conversations between users and the bot at various companies. One pattern I noticed was that users were filing tickets for multifactor authentication (MFA) resets. Multifactor authentication reset is essentially a password reset. Instant MFA resets are one of the bot's main value propositions. This made me wonder why users were not utilizing the bot for this process. 

Building Conviction

Noticing the pattern of multifactor authentification resets filed via phone isn't enough to know if it has any impact on bot adoption and engagement. I looked at engagement patterns across users who have reset their MFA via the bot.

 

I focused on answering the questions:

  • Do these users engage more deeply than users who have not tried the MFA reset skill in the bot?

  • How many MFA reset tickets filed via the phone are actually simple enough to be resolved via the bot? 

Current State

Phone Ticketing 

At companies, employees can file IT tickets on many channels including email, phone, a self-service portal, walking up to an IT agent, or with Movework's bot. Habits and ticketing culture are different from company to company, but the phone is often one of the most popular methods of filing a ticket. This is because of the convenience and ease of talking to someone and explaining the problem verbally. One adoption focus is converting employees who file tickets on other channels (email, phone, a self-service portal, etc.) to file tickets via the Movework's bot.

Research Discovery

A key part of the discovery research for adoption experiments is studying user behavior. To get an understanding of users I read through hundreds of conversations between users and the bot at various companies. One pattern I noticed was that users were filing tickets for multifactor authentication (MFA) resets. Multifactor authentication reset is essentially a password reset. Instant MFA resets are one of the bot's main value propositions. This made me wonder why users were not utilizing the bot for this process. 

Building Conviction

Noticing the pattern of multifactor authentification resets filed via phone isn't enough to know if it has any impact on bot adoption and engagement. I looked at engagement patterns across users who have reset their MFA via the bot.

 

I focused on answering the questions:

  • Do these users engage more deeply than users who have not tried the MFA reset skill in the bot?

  • How many MFA reset tickets filed via the phone are actually simple enough to be resolved via the bot? 

Current State

Phone Ticketing 

At companies, employees can file IT tickets on many channels including email, phone, a self-service portal, walking up to an IT agent, or with Movework's bot. Habits and ticketing culture are different from company to company, but the phone is often one of the most popular methods of filing a ticket. This is because of the convenience and ease of talking to someone and explaining the problem verbally. One adoption focus is converting employees who file tickets on other channels (email, phone, a self-service portal, etc.) to file tickets via the Movework's bot.

Research Discovery

A key part of the discovery research for adoption experiments is studying user behavior. To get an understanding of users I read through hundreds of conversations between users and the bot at various companies. One pattern I noticed was that users were filing tickets for multifactor authentication (MFA) resets. Multifactor authentication reset is essentially a password reset. Instant MFA resets are one of the bot's main value propositions. This made me wonder why users were not utilizing the bot for this process. 

Building Conviction

Noticing the pattern of multifactor authentification resets filed via phone isn't enough to know if it has any impact on bot adoption and engagement. I looked at engagement patterns across users who have reset their MFA via the bot.

 

I focused on answering the questions:

  • Do these users engage more deeply than users who have not tried the MFA reset skill in the bot?

  • How many MFA reset tickets filed via the phone are actually simple enough to be resolved via the bot? 

Current State

Phone Ticketing 

At companies, employees can file IT tickets on many channels including email, phone, a self-service portal, walking up to an IT agent, or with Movework's bot. Habits and ticketing culture are different from company to company, but the phone is often one of the most popular methods of filing a ticket. This is because of the convenience and ease of talking to someone and explaining the problem verbally. One adoption focus is converting employees who file tickets on other channels (email, phone, a self-service portal, etc.) to file tickets via the Movework's bot.

Research Discovery

A key part of the discovery research for adoption experiments is studying user behavior. To get an understanding of users I read through hundreds of conversations between users and the bot at various companies. One pattern I noticed was that users were filing tickets for multifactor authentication (MFA) resets. Multifactor authentication reset is essentially a password reset. Instant MFA resets are one of the bot's main value propositions. This made me wonder why users were not utilizing the bot for this process. 

Building Conviction

Noticing the pattern of multifactor authentification resets filed via phone isn't enough to know if it has any impact on bot adoption and engagement. I looked at engagement patterns across users who have reset their MFA via the bot.

 

I focused on answering the questions:

  • Do these users engage more deeply than users who have not tried the MFA reset skill in the bot?

  • How many MFA reset tickets filed via the phone are actually simple enough to be resolved via the bot? 

Current State

Phone Ticketing 

At companies, employees can file IT tickets on many channels including email, phone, a self-service portal, walking up to an IT agent, or with Movework's bot. Habits and ticketing culture are different from company to company, but the phone is often one of the most popular methods of filing a ticket. This is because of the convenience and ease of talking to someone and explaining the problem verbally. One adoption focus is converting employees who file tickets on other channels (email, phone, a self-service portal, etc.) to file tickets via the Movework's bot.

Research Discovery

A key part of the discovery research for adoption experiments is studying user behavior. To get an understanding of users I read through hundreds of conversations between users and the bot at various companies. One pattern I noticed was that users were filing tickets for multifactor authentication (MFA) resets. Multifactor authentication reset is essentially a password reset. Instant MFA resets are one of the bot's main value propositions. This made me wonder why users were not utilizing the bot for this process. 

Building Conviction

Noticing the pattern of multifactor authentification resets filed via phone isn't enough to know if it has any impact on bot adoption and engagement. I looked at engagement patterns across users who have reset their MFA via the bot.

 

I focused on answering the questions:

  • Do these users engage more deeply than users who have not tried the MFA reset skill in the bot?

  • How many MFA reset tickets filed via the phone are actually simple enough to be resolved via the bot? 

Bot Multifactor Authentication Skill

Bot Multifactor Authentication Skill

Bot Multifactor Authentication Skill

Description: This is an example of how easy it is to reset a multifactor authentication password via the bot. A phone call with an IT agent can take 10+ minutes while chatting with the bot can take seconds to solve the problem.

Brainstorming

🙌 Goal

Educate users about the high value / low effort MFA reset skill, and as a result increase engagement.

Prioritizing Based on Risk Analysis

I started to think about different ways to introduce employees to the bot MFA reset skill. Would it be more effective to reach users through the bot or involve customers' IT agents in the experiment? What are the risks when including customer employees in the execution of the experiment? By thinking of possible risks and mitigants I was able to narrow the experiment down to one actionable solution.

Hypothesis

If users have a low-effort, high-value introduction to the bot by a trusted IT agent, they will be more likely to return to it for other use cases.

Solution

Agents Re-Directing to the Bot

After thinking about the most impactful solution, I landed on customer IT agents verbally redirecting users to the bot. This solution required a lot of perseverance on my part since collaboration with customer IT agents has never been done before at Moveworks. Convincing teammates and stakeholders that this was the most viable solution was difficult and required persuasion and persistence to get sign-off. 

Agent-Facing Script

I brainstormed ways that agents could effectively, efficiently, and quickly re-direct employees to the bot when the agent received a reset MFA phone call. I created a short script that outlined the possible outcomes and agent responses. It was important that the script wasn't cumbersome to the agent and was persuasive to the employee who might be hesitant to try something new when they are in need of help. 

Example Conversation with Agent-Facing Script Implementation 

Pitching the

Experiment

Since Moveworks is an enterprise product any customer-facing experiments need to be approved by numerous cross-functional groups. I pitched the idea to one of the Founders of Moveworks, Customer Success Managers, Customer Stakeholders, and Customer IT Agents.  

Pitching Per Audience

Depending on the audience I needed to tweak the presentation content. When internally presenting the idea it was important to focus on potential business impact and possible risks. 

​

On the other hand, when pitching the idea to customer stakeholders I focused on highlighting how it could help their bot engagement, as a result increasing their RIO. When discussing the experiment with agents I focused on how it could save them time to focus on more complex issues rather than tending to basic MFA resets throughout their day. With this strategy, I was able to get proper approvals internally and from customer stakeholders.

Results

Since Moveworks is an enterprise product any customer-facing experiments need to be approved by numerous cross-functional groups. I pitched the idea to one of the Founders of Moveworks, Customer Success Managers, Customer Stakeholders, and Customer IT Agents.  

📈 Positive Change

Agents conducted the experiment for 1 month. Impact metrics were focused on the depth of engagement after the user completes the MFA reset in the bot. Overall we saw great success. Customer #1's engagement nearly tripled and customer #2's engagement quadrupled.​

🔮 Future Impact

In addition to the impact metrics, I wanted to understand the IT agent's perspective on the experiment's success. 10/10 agents would like to re-direct employees to the bot for other problems.