The back side of the "usability test" that supports the strength of SmartHR. Faster, higher quality products!

[Mokuji] ・ User Research Promotion Office has started! -To incorporate "user research" into the development process as a matter of course.・ I want to deliver products that do not have usability issues to customers. ・ Practice of usability testing Design including "useful"

>>> [Related Articles] Click here for a list of articles related to "UX Research"

User Research Promotion Office started!

First of all, I was surprised that there was a team called "User Research Promotion Office" ...! What kind of organization is it?

I think it can be said that it is a team that specializes in "surveys to understand customers." There are four members: me, one UX designer, one product designer, and one PMM.

Specifically, we design and conduct surveys, summarize the results, and formulate guidelines for methods, and are working to contribute to improving the quality and productivity of product development.

Do you really want the product you have developed? We started in earnest in September 2020 with the desire to create a research system that can be determined here.

Profile ｜ Product Designer Tetsuhiro Sato Worked at a production company as a web designer since 2007. He has been involved in DTP, website, landing page production, logo production, etc., and has experience in direction, design, and coding. He joined SmartHR as a product designer in 2018 after launching a new service at a social media marketing company in 2013. As a launching member of the "Easy Analysis Report" function, he is engaged in general design work such as prototype production centering on information design and screen design. From 2020, in parallel with the charge of SmartHR itself, we started activities to spread user research within the company.

Incorporate "user research" into the development process as a matter of course.

The User Research Promotion Office is, so to speak, a Yokokushi team. What are you aiming for as an organizational role?

I agree. The ideal is to incorporate "user research" into the development process as a means that anyone can take for granted. For that reason, we are also promoting the creation of guidelines so that the knowledge we have accumulated can be easily utilized within the company.

There are various areas of user research. What are you currently working on?

At the moment, the main responsibility is "usability testing". Especially here, there were variations in whether or not it was implemented depending on the Scrum team.

As a premise, the merit of Scrum development is that improvements can be made quickly while repeating releases in a short period of time. At the same time, we wanted to improve the quality of our products by improving the small usability issues in the development process.

First of all, "user research" can be broadly classified as follows.

▼ Survey (Understanding the usage context and user experience) Questionnaire, depth interview, group interview, ethnography

▼ Verification (detailed product brush-up) Expert review, heuristic evaluation, prototype test

▼ Evaluation (evaluation of user experience of implemented product) Usability test

(Reference source: "UX Design Textbook" by Masaya Ando)

I joined the company in 2018, and since that time, "investigation" and "verification" such as "through interviews with customers and incorporating them into development requirements" have been practiced throughout the company.

I think this is the strength of SmartHR, but in a broad sense, "research" has taken root in the organization. I thought that the value of the company, "Let's make what people want," is embodied.

At the stage when the development is completed, we also carry out "expert reviews" to have professional members look at it from the user's perspective and find problems. Members of his Customer Success (CS), Customer Support (SP), and Human Resources Laboratories, who are familiar with the actual usage and needs of the customer, participate in product verification.

However, in recent years, the number of customer industries has increased, and it has become available to large-scale companies. We can certainly hear the voices of our customers, but in the development process, there was not enough process to evaluate the "usability" of a product from the user's perspective.

PdM, PMM, engineer, quality control engineer (QA), customer success (CS), customer support (SP) ... How can everyone develop from the same perspective and develop at high speed while improving the quality of the interface? I thought that an effective way to do this would be a "usability test" that verifies the usability of the product.

We want to deliver products that do not have usability issues to our customers

So to speak, I think it is a movement to standardize usability tests within the company, but what is the background to the fact that it is particularly important organizationally?

First of all, SmartHR is SaaS, so it is important to keep customers satisfied. We want to ensure the quality of the interface as well as the quick improvement of functions. Otherwise, not only existing customers but also new customers will be separated as "difficult to use". In the extreme, we want to deliver to our customers products that have no usability issues.

Speaking of "quality", tests for QA (Quality Assurance) and sprint reviews come to mind, but how do you separate them from usability tests?

I have a diagram I made to explain it in-house, but I am organizing it like this.

Based on Masaaki Kurosu's "UX Principles", it is illustrated with an interpretation so that it can be easily explained in-house. "Currently, the definition of the ISO standard is the mainstream, but I brought up and explained the definition of Nielsen as one definition of usability so that it is easy to understand even if you do not have specialized knowledge. The activity of the Research Promotion Office was a "usability test" aimed at verifying usability in a narrow sense, and it was suitable for comparing what verification of usability in a broad sense is done in the development process, including sprint reviews. "

Usability aims at "improvement to change from negative to normal value (0 level)", that is, "usable as a matter of course". On top of that, it is an image that there is a utility that "improves utility and convenience".

If there is a problem with usability, no matter how many functions are created, they will not be used. The "blue" in the upper part "Sprint Review" overlaps with "usability" because it inevitably comes from the perspective of bringing usability closer to zero. In the "Sprint Review", the main purpose is "utility", such as demonstrating the functions developed in the sprint to stakeholders and confirming the results.

In the situation where sprint reviews are routinely conducted in Scrum development, I wanted to convey the difference in purpose that is often confused with usability testing, so I used these diagrams to recognize them.

Usability testing practice

Is it okay to ask specifically how the test is conducted?

Now, mainly for hypothesis testing before release,

▼ Ask the user to operate the prototype (record the operation with Zoom) ▼ Observe the user's behavior and attach a sticky note to Miro * ▼ Hear the intention of the user's behavior and the feeling at that time ▼ Look at the sticky note While discovering and discussing usability issues with the entire Scrum team ▼ Ask them to consider improving the discovered issues before release

However, it is a general flow. * Miro is an online whiteboard tool

As a side note, the users to be tested here are looking for people who meet the requirements from the members of SmartHR as subjects and conducting it. (There are also SmartHR users within SmartHR.)

The purpose of the survey and the required method differ depending on the progress of the project as well as the function. The User Research Promotion Office supports the design of tests according to the purpose at that time.

SmartHR body development cycle

In the development cycle with SmartHR, after extracting the customer's request, it is selected and drafted according to the size of the function, and the priority of the function and the development of the new plus application are discussed at the Roadmap Mom Association. It will be discussed and decided. After that, the functions that were drafted were detailed and developed, and after verification, they were released. (Reference: https://tech.smarthr.jp/entry/2021/06/16/115550)

Usability test implementation calendar

First of all, regarding the implementation calendar, the theme recruitment period is from the first Friday to the next Thursday, with two weeks as one term. Furthermore, the test preparation period will be until Wednesday of the following week, and the test will be conducted on Thursday. Repeat this cycle.

1) Collect test themes

The "Theme Recruitment" is blue in the above calendar, but there is a meeting where the Scrum team gathers, and there is a form to publicize and solicit if there is a theme for usability testing. Apply from the Slack workflow

-Range: What test do you do? -Purpose: What are you testing for?・ Verification content: Points that we especially want you to see

Ask them to fill in these.

Conduct hearings when recruiting themes

If there is an application, we will conduct a specific hearing with that person and move on to designing the test.

2) Design the test

Fill in a more detailed template based on the content of the hearing. Based on the purpose of the test, we will drop it into scenarios and tasks. Here, the members of the Promotion Office will discuss and squeeze in.

Usability test templates include

Scope: What test do you do? Purpose: What are you testing for? Target: Who are you targeting? Verification content: What is the specific verification content? Scenario: Background to get the user to operate Task: Procedure to get the user to operate Verification method: What kind of equipment do you want the target person to test?

Is described.

To introduce an example of an actual test, there is an account lock function that "locks when login fails 10 times", and we performed its usability test.

This is a function created for security, but can the user correctly recognize and unlock the account lock function in the first place? Is the theme of verification.

Usability test design for account lock function

As specific verification contents,

・ Are you aware that there is an account lock function?・ Recognize the account locked state ・ Do you know how to unlock the account?・ Is it possible to unlock the account?

Items such as are set.

In this case, the account lock function can be unlocked by the employee himself or even by the administrator. This time, we prepared a test scenario in which "employees" and "administrators" were set as users, and asked them to play two roles. I try to write the test scenario concretely so that I can get into the situation as much as possible.

Account lock function test scenario

3) Test implementation

From here, it is the stage to actually perform the designed test. I will explain the day using the template above as a material for the test subjects.

Scenario A You are an employee working at SmartHR. When I tried to log in for the first time in a while to see my pay slip, I forgot my password and decided to try various passwords.

When I tried to log in for the first time in a while to see my pay slip, I forgot my password and decided to try various passwords.

Task A-1 (Employee Edition) Please lock your account by missing your password 10 times.

Task A-2 (Employee Edition) "Please restore the locked account.

4) Summarize improvement plans

Finally, I will summarize the test results.

We will summarize it in advance, but we will summarize the parts that are of concern for each task, such as the first person, the second person, and the third person who were tested while looking at the sticky note attached to Miro.

It seems that Miro is summarizing the sticky notes that I was interested in for each task.

Including the Scrum team and the members of the promotion office who are facilitating, we will pick up "What did you care about?" And organize the points that need to be improved.

It seems that the points that should be improved are summarized.

Capture the part that was lost on the actual screen.

At this time, I will talk about improvement plans together, but basically I leave it to the team when and how to improve.

There are still few achievements, but many of the things we have done have improved before the release. Of course, there are some that will improve after the release.

Design including "you can use it"

As a product / function, is it possible to re-develop from an almost completed state? It seems difficult to come to terms with the risk of so-called “rework”.

By the way, since we are working on Scrum development, the members are positive about "fixing". It is more risky not to use it after it is released. In fact, I witness scenes that users cannot use well during testing or recorded video, so I think that understanding the usability issues that users have will create a sense of conviction and lead to "fixing". ..

Mr. Sato: To be honest, isn't it difficult to take on the usability test part while having the main mission?

Certainly, testing takes time (laughs). However, although the main focus is on product development, the efforts of this promotion office are also one of my main businesses. What do you contribute to as a designer and receive money from customers? Can you use it properly? In essence, our product design group must ensure the quality of the interface. I think this is a major premise.

In particular, SmartHR has been used by major enterprises, so-called enterprise customers, in the last few years. Higher quality products are required. Under such circumstances, we would like to deliver products that do not have usability issues to our customers. I think this is also the significance of our efforts.

Finally, please tell us about your future efforts and goals.

This usability test has just begun. It's not yet organized. We are at the stage of making steady dissemination activities while making achievements. When I tell the members of the subjects who cooperated with the test how it actually improved, I am very pleased that it was useful for the product, but including that situation, I share the test results throughout the development with Slack etc. doing. With this kind of accumulation, I would like to show that usability testing is effective for the product.

Also, we haven't been able to quantitatively see how much usability has been improved by the interface after its release. We would like to expand our activities to such post-release verification. In addition, the SmartHR development organization has grown and new projects are emerging. I would like to create a system that can handle that as well.

Interview / text = Katsuya Shiraishi