Usability Pilots - eGovernment benchmark

Usability indicators - 29/09

This project aims to identify new usability indicators for the European eGovernment benchmark. The benchmark is perhaps the most important EU digital government policy instrument since the European Union has no direct competencies in digital government. However, the eGovernment benchmark framework is under continuous development, and in particular, the usability indicators have shown limitations in terms of validity and comprehensiveness. Thus, this project aims to improve the eGovernment benchmark, with specific regard to measuring usability. The challenge is to identify feasible and effective indicators that can be integrated into the current data collection methodology.

This is the current list of candidate indicators for updating the usability perspective in the European eGovernment benchmark. The consortium produced a long list of indicators with inputs from interviews, expert workshops, desk research, literature review and stakeholder interactions. This list is constantly revised and can change according to the information gathered throughout the study.

Three perspectives need to be in mind when analysing this list. First, indicators must comply with usability theory, addressing important usability and user experience dimensions. Second, be aligned with policy priorities by complementing principles laid out in the Tallinn or Berlin declaration of eGovernment. Finally, indicators need to be technically feasible, indicators that cannot be measured by mystery shoppers, evaluated by visiting the website directly, or need inputs by asking for data from public administrations are excluded from the list.

Accuracy

Accuracy measures the number of errors users make either during the process of completing tasks or in the solution to the tasks. According to research done on the topic of user usability, measuring accuracy permits the design of services to be more efficient. In general, accuracy and completeness are essential to quantify how users achieve specified goals.

Error messages: For any page requiring user input, if an input error is detected, is the input identified as an error and described as text to notify the user?sentence permalink

Binary task completion

This section complements the accuracy measures. Binary task completion refers to measures of whether users complete tasks or not. Task completion includes the number of correct tasks and the number of tasks that users failed to finish within a set time. It is an essential metric to evaluate if the services are efficient in what they are proposing to do. To gather data for some of the indicators, it would be necessary to access the logs and databases where the digital public services are stored. This is the reason why some of them were excluded during the assessment process.

Ability to change inputs immediately: Is it possible for the user to make changes in their inputs while/after the service has been executed?sentence permalink

Online service completion: Can the service be completed entirely online?sentence permalink

Stop mid-transaction: Is it possible to stop the process and come back to continue it without having to restart from scratch?sentence permalink

Proprietary technology: Does the service need proprietary technology to be accessed? Does the service require proprietary technology to be completed? These could be an eID reader or a paid software.sentence permalink

Accessibility

Accessibility is an important aspect of usability as it can make the web pages easier to navigate and enable users with impairments to access the service when, if some measures were not taken, it would not be possible. Accessibility is already included in the monitoring exercise in the context of the accessibility directive. Some indicators are included here, in particular concerning less technical aspects of accessibility, to complement the ones already measured in the most recent data collection process of the eGovernment benchmark (2020). However, most of them were not scored.

Clear language: Is the language clear and easy to understand for a user with no prior knowledge as the accessibility standards of the World Wide Web Consortium (WCAG 2.0 SC 3.1.5)?sentence permalink

Translatability: Is the language correctly translated to English by machine learning tools such as Google Translate?sentence permalink

Speed and Performance

While it is complicated to measure how much users like a site's design with web APIs, measuring its speed and responsiveness is not. These indicators are imperative to understand the quality of the online services and how efficient they deliver excellent user usability. Knowing these variables will allow countries to optimise for quality of UX, which is key to the long-term success of any site on the web.

Input rate (clicks): How many clicks are necessary to finalise the service?sentence permalink

Cumulative Layout Shift (CLS): CLS measures the largest burst of layout shift scores for every unexpected layout shift that occurs during the entire lifespan of a page. A layout shift occurs any time a visible element changes its position from one rendered frame to the next. How long does it take for unexpected elements to stop moving before the page fully loads?sentence permalink

First Contentful Paint (FCP): First Contentful Paint marks the time at which the first text or image is painted/loaded. How long does it take for the first content to appear on the page?sentence permalink

First Input Delay (FID): An important user-centric metric for measuring load responsiveness because it quantifies the experience users feel when trying to interact with unresponsive pages. How long does it take for the webpage to start processing the first user interaction?sentence permalink

Largest Contentful Paint (LCP): Measures the perceived load speed because it marks the point in the page load timeline when the page's main content has likely loaded. How long does it take for the largest content on the page to load?sentence permalink

Page requests: How many requests (scripts) are there on the webpage?sentence permalink

Page size: How large (kb) is the page size?sentence permalink

Speed index: Speed Index measures how quickly content is visually displayed during page load. It measures how quickly the contents of a page are visibly populated. What is the speed index of the page (in seconds)?sentence permalink

Time to Interactive (TTI): Time to interactive is the amount of time it takes for the page to become fully interactive. What is the TTI of the webpage (in seconds)?sentence permalink

Total blocking time: Sum of all time periods between FCP and Time to Interactive. What is the total blocking time of the page (in milliseconds)?sentence permalink

Search Engine Optimisation

High SEO ranking is important for both findability and usability of the service. Pages with good performance, accessibility and accuracy will have a higher chance of scoring well and appearing first on search engines. Thus, it is also vital to keep track of SEO metrics to assess which aspects the online services can improve. It is also true that the findability of the service is an important part of the overall user experience online.

Average position: Average position on search engines for a specific keyword to find the service online. In this case, the keyword will be the name of the service. What is the average position of the webpage?sentence permalink

H1 tag: Each web page should have a title using a h1 tag that describes its topic or purpose. Does the webpage possess a single h1 tag depicting the title or purpose of the page?sentence permalink

Text on images: The text on the website can be readable by software and search engines. Is there any text embedded in images?sentence permalink

Visibility: This metric is based on CTR and shows a domain's progress. It scores the percentage of clicks received based on the organic rankings positions across all of the keywords tracked in a campaign.sentence permalink

URL structure: The user can easily identify the structure of the web page with an organised URL structure. This also helps Google to index the page easily. Is the URL structured in a categorised format?sentence permalink

Consistency and ease of use

Consistency and ease of use are founding concepts of good design. Ease of use is a fundamental notion that explains how easily users can use a service, and consistent user experience enhances ease of use and reduces the learning curve. However, measuring them objectively and comparatively can be complex. Some member states have made a substantial effort in providing a simple and consistent user experience, for instance by consolidating their online presence and creating guidelines, toolkits and templates with different levels of enforcement. The indicators in this section aim to detect to what extent the user experience is consistent, and member states have published instruments and tools to achieve it.

Service design template/handbooks: Does the central administration provide guidelines, standards, toolkits or design templates for other administrations to structure their own services or websites?sentence permalink

Consistent design across websites: Do the websites of different services have consistent branding and design elements?sentence permalink

Transparency of the process: Show the steps that are necessary to be executed until the service is completed. Does the web page depict each step of the process that the citizen needs to go through throughout the service execution?sentence permalink

Visibility of system status and visual aids: Does the service have loading bars, breadcrumbs, number of search results and other visual cues?sentence permalink

Testing and reporting

Metrics on usage are an important enabler of usability and a sign of its priority. If countries make the data on the adoption of their digital services publicly available, it is possible to measures many different KPIs within the user usability perspective and identify potential gaps to be corrected. Publicising data about the adoption of the digital service can be a great incentive to enhance usability and improve the user experience.

Reporting: Are there performance reports with data of the service publicly available?sentence permalink

Time

Completion time can be an indicator of how well the online service is structured. This is a standard metric in user usability, as the shorter the processing time, the better, and the better the user experience, although comparability remains a challenge. It is also recommended to set the right expectations and let the citizen know how long the execution of the service will take before starting the process, allowing the person to prepare more efficiently.

Task completion time: The time users take to complete a task as a whole. How long does it take to complete the service?sentence permalink

Time notification: Does the page inform how long it will take to complete the service before starting?sentence permalink

Support and help

Options for feedback and help are also long-standing metrics in user usability and are also present in the Berlin and Tallinn declaration on eGovernment and in the existing benchmarking methodology. Some new indicators were included to complement the ones from the last eGovernment benchmark.

Chatbot: Is there an automated chatbot to reply to questions from the user?sentence permalink

Existence of hints: Are there ways to help the user complete the service, such as examples, common mistakes to avoid, tips?sentence permalink