Role

Product Designer

Duration

3 weeks

Team

Product Management

Data Science

Engineering

Translation

Skills

SQL

UX research

XFN communication

Multilingual Support Adoption

Adoption Experiments: Multilingual Adoption Pod

When I began my time at Moveworks, I was tasked with conducting adoption experiments. What are adoption experiments? Fast-paced, impactful, low-cost, and scalable growth experiments that stem from adoption gaps in the product.

This case study focuses on increasing the adoption of Movework's new multilingual chat capability.

Moveworks is enterprise software that enables employees to receive instant IT, HR, and Finance help within chat platforms (e.g. Slack, Teams, etc.) via an NLU conversational AI chatbot experience.

Email for Password

Full Case Study

Current State

Multilingual In-Bot Support

Moveworks expanded its AI conversational chatbot capabilities to include multilingual support. This means employees can receive instant help at work for IT, HR, and Finance needs by chatting with the Moveworks bot in Spanish, Deutsch, French, and other languages. The main problem is users who are multilingual are not setting their bot language to the newly supported languages.

How does a user change their language?

Typing in a non-English Language: The bot will detect the language and ask the user to select a language. Due to current product capabilities, this requires a few confirmation steps before the language setting changes which is where drop-off happens.

Interaction with promotions: Another way a user could change their language is if they receive an in-bot promotion. This also requires the user to type non-English utterances to trigger the flow and the same confirmation steps mentioned above.

Problem

Users tend to drop off before they set their non-English language. The examples below depict how drop-off is a common occurrence.

Bot Multifactor Authentication Skill

Description: On the right, the user typed a non-English utterance which the bot detected. The user doesn't see the message and fails to set their language to Spanish. On the left, the user receives a promotion from the bot notifying them about the multilingual support capability. The user interacts with the promotion by simply pressing a button, but doesn't follow the instructions in the promotion which state they must type "Hola" to set their language to Spanish.

Problem

Why is this a problem?

I started to think about different ways to introduce employees to the bot MFA reset skill. Would it be more effective to reach users through the bot or involve customers' IT agents in the experiment? What are the risks when including customer employees in the execution of the experiment? By thinking of possible risks and mitigants I was able to narrow the experiment down to one actionable solution.

🙌 Goal

How might we increase % of users who set their new preferred language within eligible multilingual speakers?

Building Conviction

I reviewed 200 conversation logs between a user and the bot. These conversations were selected without any specific criteria. About 30% of conversations users were active in the bot but not deeply engaging. These users tended to be Spanish speakers. Upon further investigation, I noticed they received a multilingual support promotion but didn’t engage or dropped off before completing the change flow.

Aha Moment

The patterns I noted from the conversation log reviewed ended up paralleling the current multilingual support activation problems. By validating the problem through my own research, I was able to strongly back up my solution when communicating the experiment to cross-functional product managers, engineers, and executives.

Hypothesis

If we remove barriers for entry,

then adoption of MLS will increase.

Solution

Auto-Activate and Awareness

By automatically changing the language for the user we are able to reduce dropoff and increase multilingual support activation. After changing their language for the user, we send them an in-bot promotion explaining the change.

User Selection Criteria

Previously the scope of multilingual support was focused on the user's locale. This is detrimental because, for example, not all people who live in Brazil speak Portuguese in their work environment.

I created new multilingual support eligibility criteria that focus on two trackable data points:

The user previously typed a non-English utterance in the bot
The user filed a non-English ticket in the past

By focusing on evidence of users speaking a non-English language we can select high confidence users for auto-activation and minimize the risk of users reverting back to English or getting frustrated by the auto-change.

New Auto-Activation Promotion

Results

📈 Positive Change

Impact metrics were focused on the depth of engagement and interactions with the promotional content send to users in the bot. The experiment would be deemed successful if users deeply engaged with the bot for the first time after the change. 20% or more of users pressed the "Thank you" button on the promotion, (with the assumption that the button click approves the change) and user engagement deepened after the change. 8-12% of users engaged with the bot deeply for the first time after the change. 50.4% of users engaged with the promotion positively by clicking the "Thank you" button.

📉 Negative Change

The gating criteria for the experiment was that no more than 40% of users who interact with the promotion should either press the "I prefer English" button or change the language back to English. The results were low compared to the prediction. Only 1% of users indicated they prefer English.