Kindle iOS App Usability Testing

Download a pdf of the report below:

kindle_usability_reportDownload

Usability Test: iOS Kindle Application

Final Report

Elon University

Dr. Qian Xu – COM-660

Zach Henkels

Katie Jostes

Sneha Kulkarni

Juliana Walker

Executive Summary

In order to understand the user experience of the Amazon Kindle App on the iPhone, researchers conducted this study and analyzed the different ways that users complete tasks within the application. With a set of five graduate students from Elon University, whose ages ranged from 20-30 years old, researchers conducted 12 tasks with related scenarios. These tasks ranged from opening the book while in the application to changing the background color. Each session lasted between 6-8 minutes during which one group member was assigned the role of proctor and another member was assigned the role of note-taker. The data analyzed looked at the task completion rate, the number of errors made and their general reaction to each task.

Once the tasks were completed and the data was compiled, it was discovered that one task had a completion rate of zero. A simple solution to this would be if the icon was changed to something more recognizable. Additionally, this task, among others, was due in part to the navigation, which is comprised of two menus as well as other icons found at the top of the reading experience. Because of this, the researchers suggest a re-design of the overall navigation journey to something more clear and concise. With this study, researchers conducted tests, observed participants and analyzed data in an attempt to better the user experience of the Kindle reading experience on the iPhone mobile application. 

Table of Contents

Introduction & Purpose of Study

The Kindle mobile application is an alternative to an e-reader tablet and provides users with the opportunity of reading e-books sold by Amazon and previously purchased e-books. The application’s reading experience is intended to mimic the interface and functionality of Amazon’s Kindle e-reader tablets. The application has several functions, including a community feature linked through Goodreads, a library of downloaded books and previously purchased books, and a store where you can purchase new books and magazines. The primary function of the application is the reading experience. The purpose of our usability test was to understand and improve upon the overall reading experience of the Kindle mobile application. We focused on the ease of use the participants experienced while engaged with the reading portion of the application. This included highlighting, scanning the table of contents, changing the appearance of the screen and other features within the reading experience.

Methods

In our study, we tested five participants who were graduate students in the Interactive Media program between the ages of 20 and 30-years old. These participants were randomly selected from the common areas of the Long building on the campus of Elon University. As an incentive, we offered participants free donuts from NC Jelly Donut in Burlington, NC after they completed the test. We conducted the study in the Long building, using the innovation lab and primary classroom on the first floor. We preferred using these spaces because they were spacious and allowed for the moderator and note-taker to be farther away from the subject, making the user more comfortable with less of a chance to skew the results. The moderator’s task consisted of reading the orientation script to the subject at the beginning and sequentially reading the tasks out loud. The note-taker was required to jot down notes analyzing the subject’s reactions and nonverbal cues indicating emotions.

The sessions lasted between 6 to 8 minutes depending on the participant. We recorded subjects using the screen capture feature on the iPhone X along with recording subjects through audio and video recording via external Canon 7D Mark II. We chose to use the iPhone mainly because the screen capture made it easy to record participant’s interactions. Setting up the Canon at a 45-degree angle focusing on the participant allowed us to video record participants reactions and comments and accompanied well with the think aloud method. The think-aloud method allowed participants to use the system while continuously thinking and verbalizing their thoughts out loud as they moved through the interface.

We conceptualized tasks (listed in the Appendices section) for each participant to complete in the e-book 1984 by George Orwell. These tasks ranged from simple to complex following a series of steps. We provided scenarios for most tasks to help the users better understand the possible motivations and contexts behind the tasks they were asked to complete.

Within this study, we gathered solely quantitative data because we were interested in how the application functioned rather than what users thought about the experience.  The quantitative data collected in this study is based on the time used to complete each task, the number of steps taken to complete each task and the number of errors made in regard to each task.

Results

Task number two was disincluded from our results due to the varied nature of the proctor’s provided stopping point and the participants’ rates of page advancement. These factors provided no basis for an unbiased measure of time used to complete tasks and perceived errors.

Task completion rate

All tasks, with the exception of numbers four and ten, were completed by all participants.    Number four was a notable outlier, with none of the participants completing the task. The failure to complete this objective was a result of all participants becoming frustrated and voluntarily leaving the task undone. Number ten was missed by one participant due to an oversight rather than an inability to find a solution for the requested action.

Duration of Task

Number four was again an outlier in average time used to complete, or in this case, resign from a task. This is due to the majority of participants attempting nearly every option available, some of them multiple times. Tasks number six and seven also took an extended amount of time, but for varying reasons. While some participants’ times were extended due to errors, others took time to choose which content to utilize for completion of the prompt before attempting to locate the correct interface controls. The general increase of time taken to complete tasks eight through twelve were not necessarily due to an increased complexity, as these tasks also had a “long way” which was unaccounted for in the phrasing of the task.

Error rate

In defining what an “error” was in regard to this test, we decided upon “any action taken that did not go toward completing the given task.” If the user were to backtrack from their extraneous action, we did not count that as an error. For example, if the user opened the hamburger menu instead of turning a page, then closed the hamburger menu, we would only count this as one error.

Once again, task number four proved to be an anomaly when calculating the error rates. The lengthy exploration time on the task increased the number of accumulated errors. This exploration involved attempting multiple application features, some of which included: leaving the reading experience and attempting to change the application’s settings as a whole, choosing the X-Ray feature multiple times, using the search feature and scrolling through the hamburger menu extensively. Additionally, no user ever clicked on the feature that would have completed the task.

The average number of errors chart was mostly representative of our data, with a notable exception in task number six. While three of our five participants had zero errors while performing this task, one user committed seven errors, and this increased the average past a value that would be representative of the remaining participants. Because of these discrepancies, we have included a table of the raw data per user below.

Errors per Task

  1 3 4 5 6 7 8 9 10 11 12
User #1 0 0 14 0 7 2 0 0 0 2 0
User #2 0 0 16 0 0 3 1 1 2 0 1
User #3 0 0 15 1 2 4 0 1 3 0 1
User #4 0 0 18 0 0 1 2 0 2 0 1
User #5 0 0 8 0 0 0 0 1 0 0 0
Average # errors 0 0 14.2 0.2 1.8 2 0.6 0.6 1.4 0.4 0.6

Findings and Recommendations

Based on our results, we have a clear view of what aspects and features of the application are working and which are not. Features that are clear in their display and function, like the highlighting tool and the navigation between chapters and pages, are working very well. Because most users have an instinct to drag over text to highlight and copy, the highlighting function was one of the quickest and most successful features of the reading experience. Additionally, the navigation between pages and chapters works well, because of the horizontal scroll bar at the bottom and the clear navigation between sections and chapters in the hamburger menu.

It is evident that task number four, instructing participants to change the background of the e-reader to black, had the most number of errors. Participants ignored the icon “Aa” which, when clicked, allows the user to change several typographic settings as well as editing the color of the background. Participants explored all the other icons except for this one because universally the symbol “Aa” is known to change font size. To reduce ambiguity, we would like to propose replacing the “Aa” icon to be a settings or “gear” icon.

Another icon that could be updated is the “My Notebook” symbol. The current icon appears as a page in a book. This icon proved somewhat confusing to users, as it looks like a place to write notes, not a storage place where highlighted lines, notes and bookmarks are stored. We would suggest replacing the current icon with a symbol more commonly associated with storage, like a box or a file.

An additional recommendation would be to re-organize the navigation while in the reading experience. It was found that having multiple menus (both a hamburger on the left and a top navigation bar) became confusing to users and resulted in repeated errors in an attempt to complete the tasks. Our recommendation would be to compile certain features into one primary navigation.

One of the imperfections of the test was the wording of tasks. For tasks 10 and 11, when asking participants to locate where the bookmark and note were saved, our intention was to have users open these saved elements in the section “My Notebook.” Instead of opening the notebook, participants manually scrolled to find the bookmark and note. We later realized that the wording of tasks 10 and 11 should have mirrored the wording of task 12, replacing the word “saved” with “stored.”

There were several instances where the screen recording on the iPhone X abruptly stopped. For example, we tested a sixth user, but their screen recording stopped abruptly in the middle of the test, so we were unable to use their data. If we were to conduct this usability test again, we would use a different method for recording the screens that was more reliable. Additionally, the Kindle mobile application is moderately different on iOS (what we tested) compared to Android. If we were to conduct this test again or do further testing off this application in the future, we would like to test the Android version as well.

Another improvement would have been to choose random undergraduate participants for the study who were not from the Interactive Media program. Since we chose students from the Interactive Media program, there could have been potential bias throughout the process as we are students in the program as well. Additionally, this choice of participants made each session feel very casual. If the participants were strangers, we believe that each session could have been conducted in a less casual, more serious manner. In the early stages of planning the test, we had hoped to gather undergraduate students between the ages of 18-22. During this time, we created an online questionnaire that would gather background data about whether participants were familiar with e-reader technology and had used an e-reader tablet or the Kindle application before. When gathering undergraduate participants proved too difficult, we decided to gather our fellow Interactive Media students and did not gather background data about their experience with e-readers and the mobile Kindle application.

Appendices

In order to conduct our test, we used several instruments and materials. In order to record the  participant’s interactions with the Kindle application, we used an iPhone X and the screen recording feature. We also recorded video and audio of users via external Canon 7D Mark II. In order to introduce the test to participants, the proctor read the following script:

Hi, __________. My name is ________, and I’m going to be walking you through this experiment today.

As you already know, we’re testing the reading experience on the Kindle application to see what it’s like for people to use it.

I want to make it clear that we’re testing the application, not you. You can’t do anything wrong.

We want to hear exactly what you think about the experience. We want to improve it, so we’d like to hear your honest thoughts.

As we go along, I’m going to ask you to think out loud to tell me what’s going through your mind. For example, if you’re frustrated trying to complete a task, verbalize that frustration. This will help us with our research.

If you have questions, you can ask, but I probably won’t be able to answer them. We’re interested in how people do when they don’t have someone sitting in the room with them, so we don’t want to skew the results. We will answer any questions you still have when we’re done.

There’s a camera in the corner. With your permission, we’re going to videotape the computer screen and what you have to say. The video will be used only to help us figure out how to improve the app and it won’t be seen by anyone except the people working on this project.

If you would, I’m going to ask you to sign something for us. It simply says that we have your permission to tape you, but that it will only be seen by the people working on the project.

Do you have any questions before we begin?

In order to conduct our test, we crafted a list of 12 tasks and some scenarios for participants to complete in order to collect our data. Below we have listed out the 12 tasks and their related scenarios:

  • Open the book (1984 by George Orwell).
  • Turn to the next page.
    • Continue flipping through the pages as you wish. I’ll let you know when to stop.
  • Find the page number of the page you’re on.
    • You and your friend are reading the same book. They ask you a question and you refer to the current page you are on.
      • Let me know when you’re done.
  • Change the color of the background to be black.
    • It’s nighttime and the brightness of the white screen on your Kindle app is bothering your roommate.
      • Let me know when you’re done.
  • Add a bookmark.
    • You’re done reading for the day, but you want to save your place.
      • Let me know when you’re done.
    • Flip a few pages.
      • Let me know when you’re done.
  • Highlight a sentence in pink.
    • There’s an interesting quote on this page that you think you might want to come back to later.
      • Let me know when you’re done.
    • Flip a few pages.
      • Let me know when you’re done.
  • Make a note about an interesting quote.
    • There’s an interesting point made in the story that you’d like to tell your friend about later.
      • Let me know when you’re done.
  • Go from your current page to page 95.
    • Your friend found an interesting line on page 95.
      • Let me know when you’re done.
  • Get back to Chapter 1 so you can continue reading.
    • Before you skipped to page 95, you were in Chapter 1.
      • Let me know when you’re done.
  • Locate the bookmark you saved earlier.
    • Let me know when you find it.
  • Locate the note saved of the quote you’d like to tell your friend about.
    • Let me know when you find it.
  • Locate where the quote you highlighted in pink is stored.
    • Let me know when you find it.

Although we focused solely on quantitative data in our study, there were some interesting, qualitative comments recorded during the testing process. Below are some of the comments we believe captured some of the frustrations participants experienced: 

  • “I think I tried every option.”
    • “There’s got to be a quicker way…”
    • “I’m lost. I dunno.”
    • “It must be in settings, but I don’t see anything in settings.”