Role
Product Designer
Duration
3 weeks
Team
Product Management
Data Science
IT
Skills
UX research
XFN communication
Subject Line Experiments
Adoption Experiments: Growth and Adoption Pod
When I began my time at Moveworks, I was tasked with conducting adoption experiments. What are adoption experiments? Fast-paced, impactful, low-cost, and scalable growth experiments that stem from adoption gaps in the product.
This case study focuses on increasing the adoption of Movework's ticket filing via the chatbot.
Moveworks is enterprise software that enables employees to receive instant IT, HR, and Finance help within chat platforms (e.g. Slack, Teams, etc.) via an NLU conversational AI chatbot experience.
Current State
Big potential to improve customer’s open rate:
Subject lines have the highest correlation to a top customer benefit: open rate
Customers generate subject lines but don’t end up sending them
12% of users are interacting with existing subject line assistant (higher than any other inline product) and only 40% of those customers end up using the output generated for them
Writing content converts higher than rewriting content
Most customers tend to rewrite existing content (88% of generations) and 12% of generations are brand new. Rewritten generations convert lower on the apply step (38%) compared to 54% for the action of "write" even though it happens much less
Description: This is an example of how easy it is to reset a multifactor authentication password via the bot. A phone call with an IT agent can take 10+ minutes while chatting with the bot can take seconds to solve the problem.
Customer problem
I am an email specialist for an ecommerce business trying to write a punchy subject line that will drive opens for my email campaign, but I am struggling to write a subject line that will drive opens.
Research process
Goal
Discover the customer problems related to the end to end process of subject line and preview text creation
Steps
10 qualitative interviews
Led group problem statement brainstorm session
Validated qualitative data with a 300 participant survey
Led group hypothesis and solution brainstorm session
Customer
pain points
Biggest customer problems found in the interviews and validated by the survey
Repetition: It’s difficult to create email subject lines that are original and avoid repetition from past email campaigns
“Takes time to look back in the history of what was written in MC also in a word document.”
Predicting performance: Customers struggle to forecast how well a given email subject line will perform with their target audience before sending out each campaign
Synthesizing email content: Customers struggle to capture the essence of their email in fewer than 7-9 words
Competitor: Customers struggle to stand out or align to their competitors
Example Conversation with Agent-Facing Script Implementation
Pitching the
Experiment
Since Moveworks is an enterprise product any customer-facing experiments need to be approved by numerous cross-functional groups. I pitched the idea to one of the Founders of Moveworks, Customer Success Managers, Customer Stakeholders, and Customer IT Agents.
Pitching Per Audience
Depending on the audience I needed to tweak the presentation content. When internally presenting the idea it was important to focus on potential business impact and possible risks.
On the other hand, when pitching the idea to customer stakeholders I focused on highlighting how it could help their bot engagement, as a result increasing their RIO. When discussing the experiment with agents I focused on how it could save them time to focus on more complex issues rather than tending to basic MFA resets throughout their day. With this strategy, I was able to get proper approvals internally and from customer stakeholders.
Results
Since Moveworks is an enterprise product any customer-facing experiments need to be approved by numerous cross-functional groups. I pitched the idea to one of the Founders of Moveworks, Customer Success Managers, Customer Stakeholders, and Customer IT Agents.
📈 Positive Change
Agents conducted the experiment for 1 month. Impact metrics were focused on the depth of engagement after the user completes the MFA reset in the bot. Overall we saw great success. Customer #1's engagement nearly tripled and customer #2's engagement quadrupled.
🔮 Future Impact
In addition to the impact metrics, I wanted to understand the IT agent's perspective on the experiment's success. 10/10 agents would like to re-direct employees to the bot for other problems.