Quantifying UX Maturity

My roles: Product Designer, UX Researcher

Prefer not to read? Watch an overview of this case study presentation on YouTube!

As a Senior Product Designer at GitLab (end to end DevOps solution application), part of my responsibilities include both identifying and driving new design opportunities and solutions for both the business and the user. In some cases, ideas are brought to me via my PM, other team members such as designers, engineers or leadership, and of course our customers. In other cases, I have the autonomy to identify and drive product improvements and do so when I see a valuable opportunity. My particular area of focus from the product and business standpoint is the Plan stage which works with use cases, users and flows related to Project Management.

We had a department goal to begin validating the maturity of the categories designers were responsible for in the product. This process hadn’t been done before so myself and one other designer were the first to lead such an effort, I over the category of “issue tracking” and them over the category of “merge requests”.

The Problem

At GitLab, we conduct what we refer to as a Category Maturity Scorecard process which is a summative evaluation of our current product categories to understand progress and opportunities surrounding usability. This is led by the Product Designer overseeing that category (in this case, myself) and collaboration often includes the stage UX Researcher and Product Manager. My group is Project Management within the Plan stage. The category within the stage group that we were analyzing was “issue tracking” which refers to the creation, evaluation and management of issues (similar to tickets or stories in tools like Jira and Asana) within GitLab.

At the time, our team had assigned our category a score which placed us in the Lovable group, a minimal number of 3.95. It was an arbitrary number based on assumptions as we’d not yet conducted the scorecard process to validate where we were.

The User

Issues are used at some point by pretty much every user of GitLab. However, for our particular category the primary users are the Product Manager and the Development Team Lead. We already have well defined personas for each and solid understanding of the day to day pain points associated with these roles, so our focus was on gauging the quality of the experience of using issue tracking for these particular users.

The Process

The features associated with issue tracking are some of the most used throughout the product. This was GitLab’s first time of working through the category maturity scorecard process and two of us were focused on it for separate categories. Because of that, I frequently collaborated with the other designer as well as my UX researcher and PM to get feedback on my progress.

First step – define the goal

The request was to analyze “issue tracking” as a category and gauge the level of maturity. In addition to that, we wanted to understand what low hanging fruit insights we could pull from this research to work on and help increase the score.

The research associated with the category needed to be built off of Jobs to be Done for the stage group. Myself, the other designer and three Product Managers associated with the stage worked on these initially with the intention of validating them at a future point (see case study for this recently completed work). The concept of issue tracking spanned multiple jobs though and the research scope was quickly growing.

I wanted to establish a North Star statement to help me focus and align the team on what we were striving to accomplish here based on an amalgamation of the jobs we’d defined that were associated with issue tracking. This required outlining what “issue tracking” is within the context of our product. The statement needed to be malleable enough to define the wide range of experiences associated with issue tracking but firm and clear enough that I could extract tasks related to it to test with users.

Alongside my PM, I created the following statement as the primary “job” associated with issue tracking: To provide a clear, single source of truth for an idea from inception to implementation that the members of a community can trust and effectively collaborate on.

Second step – create measurable tasks

Once the north star job statement was finalized, I created 14 scenarios for users to complete which I felt most represented the secondary jobs that could be extracted from the statement. This was done while collaborating with my PM to understand what they already knew about the user and the need and with Research to ensure the tasks were in alignment with what we needed to accomplish for the work as well as see if there were any opportunities to better articulate the task.

In addition to each scenario, success criteria had to be defined as there’s often more than one way to complete a single task or accomplish a goal within the product. I created this with feedback from Research.

An example of one of the scenarios was:”You’re working on a release plan for the month  surrounding a new onboarding feature for your product. Where would you go to start planning out the high level objectives you want to deliver for the release?“. The goal in this scenario was to see how the user would approach creating an initial larger idea. I wanted to understand where they would go, what they would do, without giving any specific direction as to what we would recommend.

Third step – conducting the tests

I conducted testing sessions with 11 participants – 6 who fell into the role of an “issue tracker” which included PM and engineering manager types and then 5 in the “issue worker” role such as an engineer or designer.

Each task was timed and the testing resulted for each in one of three cases:

• the user successfully completed it according to the success criteria (considered “success”)
• they didn’t complete it, expressing that they were not sure what else they would do or how to do what was being asked (considered “fail”)
• the user completed it according to their definition of success but not in alignment with our definition (considered “fail”)

For each successful task completion, we asked them to rank from 1-5 (with 1 being “completely agree” and 5 being “completely disagree”) how well the task met their requirements and then how easy the task was.

Fourth step – calculating the results

I then calculated the scores using the UMUX Lite approach as recommended in our documentation (note: the documentation appears to have evolved since we completed this initial research to include a third question).

The Results

As a result of the study, we determined that our category was actually a bit lower in maturity than we’d hoped at Complete rather than Lovable, but now we knew not only where we needed to improve but how.

The Future

We are planning to conduct another Category Maturity Scorecard evaluation soon using the same process to gauge progress made by iterating on issue tracking since the first CMS assessment.