Case Studies
Launching a New Marketing Technology Platform through Programmatic Research
Crafting a design research strategy
At Vericast I owned research efforts supporting the design and development of a new marketing technology platform. Over the course of several months I led the creation and execution of a comprehensive product intelligence plan. By taking a programmatic approach, my work provided product leaders with a steady stream of timely insights to support evidence-based decision making.
My plan relied on two strategies to mitigate risks associated with new product development. First, multiple internal and external research efforts allowed for triangulating data to examine key questions from different points of view. For example, I helped concept-test the platform’s key features through interviews with target users as well as interviews with executive leaders from target clients.
Second, carefully selected tools and processes provided efficiency in the dissemination of actionable research insights. Partnerships with trusted vendors allowed us to recruit research participants on an ad hoc basis with little effort. Our new research tool Notably made it quick and easy to analyze data and create succinct, shareable insights. Finally, a biweekly insights sync with product leaders served as a routine opportunity to advocate for users and influence decision-making.
My successful implementation of the plan resulted in a feedback loop where research and design continuously informed one another. For example, one team effort required us to choose terminology for naming the platform’s key features. When I presented insights showing that users were easily confused by a term we were considering, product leaders agreed to throw it out and supported subsequent research on alternative terms.
Incorporating participatory design
One of the most exciting research efforts in the plan was participatory design with client advisors—early adopters who agreed to participate in extensive research activities. My vision for this component involved leading biweekly workshops with each client advisor, analyzing their feedback, and delivering insights to project stakeholders. The goal was to establish a routine cadence that would expand the concept of programmatic design research to include repeated, in-depth analysis of a small group of users over time.
Around the time our first client advisors were ready to engage in research, I stepped away from the project to take advantage of an exciting opportunity with another employer. My understanding is that other researchers have been successful at continuing to execute on the plan I created.
Creating New Tools to Support Inclusive Design in Financial Services
Defining objectives and exploring best practices
During my time at Ally Financial, I collaborated on several initiatives to make our UX design processes more inclusive. The bank had historically attracted customers with healthy finances who wanted to park their extra cash in no-fuss, high-interest savings accounts, but many of the new products and features being developed were intended for people who struggle with money and have different needs.
As an active member of the UX Inclusive Design Group, I collaborated with designers, researchers, and accessibility specialists to implement process improvements supporting more inclusive customer experiences. Our goal was to position inclusive design as a necessary practice, not an optional area of interest, so we looked for ways to integrate structured activities into our established routines. Rather than start from scratch, we dedicated several weeks to learning about best practices from industry leaders.
Inspiration for our own ideas came from two key sources. First was The Designer’s Critical Alphabet by Lesley-Ann Noel, PhD, which provides definitions and question prompts for key terms in critical theory. We were drawn to Noel’s work because it guides design teams through meaningful dialogue in a formal but approachable way. Noel was kind enough to meet with our group and advise on how to incorporate her cards into our project.
Our second source was the Conscious Design Canvas by Ben Evans, a template that asks teams to define both the obvious and non-obvious users they are designing for. After attending an InVision event where Evans explained how the tool challenges biases, we concluded that a similar canvas could help our teams develop empathy for marginalized users.
Designing a template for critical reflection
Next we adapted Noel’s cards and Evans’ canvas for an original workshop activity called the Inclusive Design Canvas. We concept-tested our initial prototype with a pilot project and used participant feedback to refine the design.
The Inclusive Design Canvas uses the critical alphabet to help participants identify specific types of unintended users for a given design. Then a series of discussion questions prompts participants to brainstorm unique pain points those users may experience as a result of having been marginalized during the design process. (The Inclusive Design Canvas is proprietary, so I can’t share images of it here.)
The outcome of an Inclusive Design Canvas workshop is twofold. First, design teams leave with a list of specific user pain points that need to be solved for, and second, they become more attuned to thinking about unintended users of products and services more generally.
Fostering empathy for a new target market
Each member of the Inclusive Design Group also helped lead initiatives within their respective functional teams as well. One UX research project involved creating a new enterprise persona to represent customers living on tight budgets. The process began with a collaboration among researchers from multiple departments who compiled data from various sources. For example, themes from qualitative interviews my team had conducted were validated with quantitative survey results from our partners in consumer research and innovation.
Based on these synthesized findings, we crafted a new persona that departed from our existing framework in innovative ways. First, we shifted from focusing on demographic to psychographic attributes. Whereas previous personas had been defined by their respective income brackets, this new persona was defined by the unique psychological traits resulting from a small delta between their income and expenses. This approach helped us clarify that customers with widely varying incomes nonetheless experience similar struggles when their living costs leave them with little wiggle room.
Second, we formatted the persona with a gender-neutral name and collage of diverse photos. Traditional personas typically include a name and photo representing a single social identity, but we wanted to convey how our new persona represents a population that is inclusive of various racial-ethnic, gender, and age groups. (The new persona is proprietary, so I can’t share images of it here.)
Putting new inclusive design tools to use
Incorporating the persona into various initiatives at the bank helped teams prioritize the unique needs of customers who are economically disadvantaged. For example, I helped write a set of screener questions aligned to the persona so that teams designing new budget tools could get feedback from appropriate usability testers.
The persona also played an important role in designing a new checking account feature called CoverDraft, which provides customers with up to $100 of fee-free overdraft coverage for qualified transactions. I led a series of journey mapping workshops with the persona to document paint points and moments of delight that our current customers might experience once CoverDraft went live.
I borrowed elements from the Inclusive Design Canvas to set up two journey maps. The first was for intended users, customers whose needs and preferences are aligned to the new feature. Workshop participants were ecstatic to document each touchpoint on this “happy path” and identified multiple moments of delight customers might encounter.
The second was for unintended users, customers who simply do not want or need CoverDraft despite the fact that it would be switched on by default. As we began to brainstorm customer pain points and frustrations, we realized that several serious “unhappy paths” had never been considered.
For example, it would be possible for a customer to ignore marketing emails about the new feature, never become aware about changes to their account, and then unintentionally complete debit transactions with money they don’t have and can’t afford to pay back. As one customer later commented on the webpage announcing CoverDraft, “How do I Opt Out of this service? If my account has items presented with insufficient funds, I do not want them paid.”
A major takeaway was the realization that some customers may not welcome the new feature, especially those who prefer overdraft transactions to be declined so they can avoid owing the bank money. Even though CoverDraft is meant to help people on the margins, we had to admit that it could actually create new problems for them instead. Product leaders were receptive to the feedback and agreed to prioritize functionality that will allow customers to easily turn the feature off. In the end, this project illustrates how inclusive design tools can help teams move beyond their own good intentions and critically evaluate the impact of their work on end users.
Designing Interactive Learning Experiences that Improve Student Outcomes
Focusing on the unique needs of extreme users drives innovation in curriculum development.
Understanding the unmet needs of students on the margins
When I taught Intro to Sociology at Tennessee Tech for the first time in Fall 2017, I quickly realized my biggest challenge was to keep students engaged throughout the entire fifteen-week course. Approximately 25% of students displayed a troubling pattern of behavior. They frequently skipped class, failed to submit low-value assignments, and rarely connected with me one-on-one, such as during office hours or through email correspondence.
When I took a closer look at their grades, I realized there were actually two different types of students involved: low-achieving students, who earned D’s and F’s, and high-achieving students, who earned A’s.
To better understand their experiences, I conducted one-on-one interviews with five low- and five high-achieving students. I used thematic analysis and affinity diagramming to synthesize this qualitative data alongside quantitative measures of academic performance. Finally, I created personas and journey maps to guide my student-centered redesign of the course.
This research revealed that low-achieving students tended to be freshmen with modest high school GPA’s and ACT scores, indicators that college-level work might be daunting for them. Many were first-generation college students who didn’t know what to expect from the course. They explained that when course content proved challenging early in the semester, they felt isolated and too embarrassed to ask for help. Continuing to attend class caused anxiety, so they unplugged from the course and fell further and further behind.
High-achieving students tended to be upperclassmen who were already succeeding in their major fields of study. Many benefited from the support of college-educated parents and had conquered college-prep courses in high school with relative ease. They viewed the course as neither challenging nor important and were confident in their ability to earn A’s without investing much effort. Skipping class and failing to submit low-value assignments were calculated choices, and they were able to score well on exams by quickly skimming textbook chapters on their own.
I used these insights to frame my design problem as involving two types of extreme users, each with their own unique needs. Low-achieving students needed more social support to voice their concerns and get help when they needed it. High-achieving students needed more meaningful connections to the course other than simply earning “easy A’s.”
Testing a possible solution for increasing student engagement
My goal was to structure class time around activities that would increase engagement, so I organized students into diverse groups of five and required them to work together on completing formative assessments and preparing for summative assessments.
Qualitative evidence collected through classroom observation showed that low-achieving students were more comfortable reaching out for help and high-achieving students were more invested in collaborating with their peers. Quantitative measures of class attendance and missing assignments both improved as well, and perhaps most importantly, low-achieving students were much more likely to earn a C or better in the course.
This new emphasis on group work also increased participation in class discussion more broadly, but unfortunately students were too often discussing unsubstantiated claims related to their own personal opinions. They were distracted by political ideology and began to lose sight of sociology as a science. Over time, I realized that my solution to the problem of keeping certain students engaged had created the new problem of keeping the entire class focused.
Iterating on the solution with a lab science approach
So, my next goal was to place more emphasis on how social scientific claims are supported by the careful collection and systematic analysis of evidence. Students needed to experience the research process themselves in order to better understand how sociologists arrive at their conclusions. They needed to learn about sociology by doing sociology.
In Spring 2019 I determined I would try teaching the course like a lab science. Just as natural science courses like chemistry and biology provide students with laboratory settings to develop familiarity with the scientific method, I would create lab assignments that give students the chance to work with real data. My idea was awarded a competitive curriculum grant through Tennessee Tech’s Quality Enhancement Plan, which involved an intensive two-day design sprint with other grant recipients.
On the first day, faculty each established criteria and constraints for our respective courses and then cycled through rounds of divergent and convergent thinking to brainstorm and subsequently edit our ideas. The following day, we used backward design to sketch out course objectives, assessments, and learning experiences for an iterative series of prototypes that were shared during group critiques. My final design included a series of ten lab assignments for students to complete with their group members during class.
Evaluating the positive impacts of innovative curriculum
These lab assignments were implemented in Fall 2019 and effectively oriented class time around the collection and analysis of empirical evidence, which ultimately kept students focused on the methods and theories that distinguish sociology as a social scientific discipline. Students rated me 4.9 out of 5.0 as an “excellent teacher,” and the course 4.6 out of 5.0 as an “excellent course.”
Later, a comparison of pre- and post-test data from a departmental general education assessment showed that students in my sections of the course were making the most significant gains. Based on these outcomes, my department chair nominated me for the university’s General Education Award for Outstanding Teaching.
After sharing these results with a colleague who specializes in the scholarship of teaching and learning, we decided to collaborate on a journal article illustrating how design thinking and user experience research can be used to improve course instruction. Our co-authored reflective essay was published in Transformative Dialogues: Teaching and Learning Journal.