Report on the 2020/21 Monitoring Period

Analysis of Accessibility Requirements for Websites and Mobile Applications part of DL 83/2018

AMA, I.P. / April 2022

content of this report:

List of Graphics

• Graphic 01 - Stratification of the sample of the 281 websites selected for the 1st monitoring period 2020 - 2021 (by sector)
• Graphic 02 - Average rate of non-compliance of websites with the applicable clauses of the European Standard EN 301 549
• Graphic 03 - Average non-compliance rate of Apps by Operating System
• Graphic 04 - Average non-compliance rate of Websites and Apps – in-depth monitoring
• Graphic 05 - What are the most compromised Uses given the non-conforming clauses of EN 301 549?
• Graphic 06 - Website compliance with EN 301 549 level 'A' clauses tested
• Graphic 07 - Distribution of pages by website (Home+ sample)
• Graphic 08 - AccessMonitor score histogram by website (Home+ sample)
• Graphic 09 - AccessMonitor scores histogram per page (Home+ sample)
• Graphic 10 - Distribution of pages by website (2K sample)
• Graphic 11 - AccessMonitor Score Histogram by Website (2K sample)
• Graphic 12 - AccessMonitor scores histogram per page (2K sample)
• Graphic 13 - Website compliance with EN 301 549 level 'A' clauses tested
• Graphic 14 - Website compliance with EN 301 549 level 'AA' clauses tested
• Graphic 15 - Websites - Average compliance rate of the clauses that make up the Functional Performance Statements considering the primary relationships
• Graphic 16 - Websites - Average compliance rate of the clauses that make up the Functional Performance Statements considering the primary and secondary relationships

List of Tables

• Table 01 - Distribution of websites by administration level
• Table 02 - Distribution of websites by type of service
• Table 03 - Descriptive statistics of the sample of pages
• Table 04 - EN 301 549 clauses tested by AccessMonitor
• Table 05 - EN 301 549 clauses tested by QualWeb's ACT rules
• Table 06 - EN 301 549 clauses considered in the in-depth methodology for websites
• Table 07 - Demographic information of participants in the usability tests with websites
• Table 08 - Demographic information of participants in the usability tests with websites (continued)
• Table 09 - EN 301 549 clauses considered in the in-depth methodology for mobile applications
• Table 10 - Demographic information of participants in the usability tests with mobile applications
• Table 11 - Demographic information of participants in the usability tests with mobile applications
• Table 12 - Distribution of pages by website (Home+ sample)
• Table 13 - Complianc of Websites sample with the clauses of EN 301 549 tested (Home+ sample)
• Table 14 - Compliance of Websites sample with the Functional Performance Statements considering the primary relationships (Home+ sample)
• Table 15 - Compliance of Websites sample with the Functional Performance Statements considering the primary and secondary relationships (Home+ sample)
• Table 16 - Number of websites where each AccessMonitor test is true (Home+ sample)
• Table 17 - Number of websites where each QualWeb's ACT rule passes, fails, or needs manual validation (Home+ sample)
• Table 18 - Number of pages where each AccessMonitor test is true (Home+ sample)
• Table 19 - Number of pages where each QualWeb's ACT rule passes, fails, or needs manual validation (Home+ sample)
• Table 20 - Distribution of AccessMonitor score by website (Home+ sample)
• Table 21 - Distribution of AccessMonitor score by page (Home+ sample)
• Table 22 - Distribution of pages by website (2K sample)
• Table 23 - Compliance of Websites sample with the clauses of EN 301 549 tested (2K sample)
• Table 24 - Compliance of Websites sample with the Functional Performance Statements considering the primary relationships (2K sample)
• Table 25 - Compliance of Websites sample with the Functional Performance Statements considering the primary and secondary relationships (2K sample)
• Table 26 - Number of websites where each AccessMonitor test is true (2K sample)
• Table 27 - Number of websites where each QualWeb's ACT rule passes, fails, or needs manual validation (2K sample)
• Table 28 - Number of pages where each AccessMonitor test is true (2K sample)
• Table 29 - Number of pages where each QualWeb's ACT rule passes, fails, or needs manual validation (2K sample)
• Table 30 - Distribution of AccessMonitor score by website (2K sample)
• Table 31 - Distribution AccessMonitor score per page (2K sample)
• Table 32 - Compliance of the websites sample with the clauses of EN 301 549 tested
• Table 33 - Websites compliance with the Functional Performance Statements considering the primary relationships
• Table 34 - Average rate of the Websites compliance with the clauses of each Functional Performance Statement considering the primary relationships
• Table 35 - Websites compliance with the Functional Performance Statements considering primary and secondary relationships
• Table 36 - Average rate of the Websites compliance with the clauses of each Functional Performance Statement considering the primary and secondary relationships
• Table 37 - Compliance of mobile applications with the EN 301 549 clauses tested
• Table 38 - Compliance of mobile applications with the Functional Performance Statements considering the primary relationships
• Table 39 - Compliance of mobile applications with the Functional Performance Statements considering the primary and secondary relationships
• Table 40 - Compliance of mobile applications by operating system with the the EN 301 549 clauses tested
• Table 41 - Average rate of non-compliance of websites with the applicable EN 301 549 clauses
• Table 42 - Non-compliance rate of Apps by Operating System
• Table 43 - Non-compliance rate of Websites and Apps (in-depth monitoring)
• Table 44 - Non-compliance rate of each Functional Performance Statement
• Table 45 - List of sampled websites for the simplified monitoring method
• Table 46 - List of sampled websites for the in-depth monitoring method
• Table 47 - List of sampled mobile apps for in-depth monitoring method

Executive Summary

Under Decree-Law No 83/2018 of 19 October 2018 setting out accessibility requirements for websites and mobile applications, Agência para a Modernização Administrativa, I.P. (AMA) is responsible for submitting a report to the European Commission every three years on the outcome of the monitoring carried out, including measurement data. This document contains the results and measurement data for the 1st monitoring period from 2020 to 2021.

To carry out the analysis of this first monitoring period, 281 websites and 16 mobile applications were selected in accordance with the methodology proposed by Implementing Decision (EU) 2018/1524. The websites have been subjected to two types of monitoring: simplified monitoring and in-depth monitoring. The first is essentially based on checking a sample of pages with an automatic validator. The second is a manual human validation by an accessibility expert. Mobile applications have only been subjected to in-depth monitoring.

The selection of the sample of websites considered the geographical criterion (central, regional, and local) but also the type of service, the demand for it and the sector of activity. Both the sample of websites and the sample of mobile applications were submitted for consideration to organisations representing persons with disabilities via the Inclusion Policy Commission coordinated by the Secretariat of State for the Inclusion of Persons with Disabilities.

Note: if you are unable to consult the following chart for any reason, see the data in Table 02.

Websites

Note: if you are unable to consult the following chart for any reason, see the data in Table 41.

For the simplified monitoring analysis, we used two sample compositions to submit to automatic validators: one with fewer pages and one with more pages. The first based on the method that AMA is using at the Portuguese Web Accessibility Observatory, which consists of a sample composed of the 1st page plus all pages belonging to the domain hyperlinked from the first page, which we have designated as Home+ (H+). The second, which we have designated 2K because it consists of a collection with several levels of depth up to a maximum of 2,000 pages. On average, Home+ produced samples with an average of 80 pages per website while 2K produced samples of 1000 pages on average per website.

In essence, we wanted to answer the question: is the Home+ sample representative enough or will we have to increase the number of pages we are using at the Observatory? When analysing Graph 02 which shows us the average non-compliance rates of the clauses of the European Standard that make up each accessibility principle, we find that the non-compliance rates for the 4 principles follow similar trends in both automatically analysed samples (i.e. H+ and 2K).

Another of the questions at the outset was related to the effect on the results caused by expert validation (in-depth monitoring). To what extent does in-depth monitoring corroborate or not the results of the two-analysis carried out exclusively with automatic validators (simplified monitoring)? From the analysis of the Graph 02 we realise that the principles with the best and worst compliance states in simplified monitoring are also the worst and the best in in-depth monitoring, but the scenario found by in-depth monitoring is, on average, 1 ½ times worse than that revealed by simplified monitoring. For the “Understandable” principle, in-depth monitoring even reveals a picture three times worse than the simplified monitoring. This reflects the importance of manual expert assessments. There is, however, one aspect on which the two evaluation methods agree. When observing the non-compliance rates for each of the 4 Web accessibility principles listed by the W3C, both methods point to the ‘Perceivable’ principle and the ‘Robust’ principle as having the highest non-compliance rates — in the range of 78% and 94% respectively. It is also worth noting that the scenarios found by the automatic and manual evaluations in these two principles are only 10% apart, which leads us to conclude the good work of automatic tools in detecting non-compliances in the principles of “Perceivable” and “Robust” accessibility.

In-depth monitoring is more effective in detecting compliance levels than simplified monitoring, but this analysis consumes more resources and time. For this reason, it is applied to a small number of pages. In this analysis, the simplified monitoring was carried out in a universe of 281,706 pages (sample 2K) and 19,931 (sample Home+). In-depth monitoring has been applied to a universe of 477 pages.

Mobile applications

The selection of mobile applications consisted of 8 applications that made up the top of the most used applications present in the catalogue https://www.app.gov.pt, in its iOS version and its Android version, making, in practice, a total of 16 applications — it is proven that the iOS version and Android version of the same application require, in terms of accessibility, two independent reviews. 144 screens were analysed, averaging 9 screens per application. The average rate of non-compliance was 51% for iOS applications and 53% for Android applications, thus not allowing it to be clearly said that the applications of one version are better than the others. In the “Robust” accessibility principle, the average rate of non-compliance per version is equal — i.e., 87.5%. iOS applications are best presented in the “Operable” and “Understandable” principles, but on the “Perceivable” principle, Android applications are slightly better than iOS applications.

Of the 4 principles of accessibility, those with the highest levels of non-compliance are “Perceivable” and “Robust”. “Perceivable” is very penalising, for example, of the visually impaired users (blind or low vision) and deaf people. On the other hand, “Robust” penalises all users who make use of technologies to transform information to their reading and navigation capabilities. These technologies include assistive technologies — screen readers with speech synthesiser or braille displays, used by blind people — but also, globally, any indexing, navigating or visualisation technology that needs to interoperate with information. The “Perceivable” principle appears with non-conformity rates in the order of 65%. The ‘Robust’ principle, on the other hand, appears with very high rates of non-compliance, in the order of 90%.

Note: if you are unable to consult the following chart for any reason, see the data in Table 42.

Sites and Apps

The 4 accessibility principles are affected in the same way on websites and mobile applications — the best match the best and the worst match the worst. However, mobile applications are always better than websites in all 4 principles — on average 10 percentage points higher than websites. The ‘Robust’ principle is presented with rates of non-compliance around 90%, followed by the ‘Perceivable’ principle with values around 70%. On the other hand, the ‘Understandable’ and ‘Operable’ principles show significantly lower non-compliance rates, respectively around 15% and 40% for applications and 30% and 50% for websites.

Note: if you are unable to consult the following chart for any reason, see the data in Table 43.

Users and main problems

Which users face the highest number of barriers? In European Standard EN 301 549 a relationship is made between the clauses representing the good accessibility practices to be applied and the users benefiting from these practices named as Functional Performance Statements. This more elaborate name relates to the fact that, by referring, for example, to ‘Usage without vision’, it is intended to refer not only to people with vision disabilities, but to all those who, in a given situation, are unable or limited in the use of vision — e.g. someone who, due to the intense sunlight of a summer day, is unable or have more difficulty to use a smartphone on the street.

Thus, the data collected (see Graph 05) tell us that this sample has non-conformities in the clauses that primarily affect:

• ‘Usage with limited vision’ (46% of non-conformities);
• ‘Usage without perception of colour’ (45%), and;
• ‘Usage without vision’ (41%).

‘Usage with limited manipulation and strength’, ‘Usage with limited cognitive capabilities’ and ‘Usage without hearing’ appear with non-compliance rates in average between 30% and 40%.

Note: if you are unable to consult the following chart for any reason, see the data in Table 44.

From the analysis carried out on websites and mobile applications, high rates of non-compliance in the ‘Perceivable’ and ‘Robust’ principles between 80% and almost 100% — 97% of non-compliance of the ‘Robust’ principle in the in-depth monitoring of websites have been detected regardless of sample size and regardless of the monitoring method used.

It is therefore not surprising that the 11 clauses with the highest failure rate of websites in in-depth monitoring were:

• 9.1.1.1 Non-text content, with a failure rate of 100% compared to applicable websites;
• 9.1.2.1 Audio only and video only (pre-recorded), with a failure rate of 100% compared to applicable websites;
• 9.1.2.2 Subtitles (pre-recorded), with a failure rate of 100% compared to applicable websites;
• 9.1.2.3 Audio description or alternative multimedia (pre-recorded) with a failure rate of 100% compared to applicable websites;
• 9.1.2.5 Audio description (pre-recorded), with a failure rate of 100% compared to applicable websites;
• 9.1.3.1 Information and relations, with a failure rate of 100% compared to applicable websites;
• 9.1.4.2 Audio control, with a failure rate of 100% compared to applicable websites;
• 9.4.1.1 Syntactic analysis (parsing), with a failure rate of 100 % compared to the applicable websites;
• 9.1.4.10 Re-align, with a failure rate of 96% compared to applicable websites;
• 9.2.4.4 Purpose of the hyperlink (in context), with a failure rate of 96% compared to the applicable websites; e
• 9.4.1.2 Name, function, value, with a failure rate of 96% compared to applicable websites.

From this group of 11 clauses, 8 are part of the ‘Perceivable’ principle, where usage depends on vision and hearing, 2 for the ‘Robust’ principle, where there is interoperability with assistive technologies and 1 for the ‘Operable’ principle, where is the manipulation of digital interfaces with alternative technologies to the mouse, such as the keyboard and all their emulators.

It is also interesting to analyse the distribution of the clauses of EN 301 549 by WCAG 2.1 levels ‘A’ and ‘AA’. Although the European Standard does not make this distinction, considering all clauses equally, distribution can facilitate those facing the task of correcting the problems encountered. W3C says that “A” level problems affect more people and that it’s a good starting point for the rectification work out there. So, we included in the executive summary (see graphic below) Graph 13 which shows us the ‘A’ level clauses that were analysed in the present study. See also Graph 14 for the ‘AA’ level clauses.

Note: if you are unable to consult the following chart for any reason, see the data in Table 32.

The graph above shows us a scenario where it is possible to verify the average non-compliance rates of accessibility clauses. The highest non-compliance rates occupy the leftmost positions in the bar chart, and it is immediately possible to perceive that non-compliance affects 30% to 40% of the total clauses. On the contrary, 60% to 70% of the clauses that can be seen, are clauses that comply or clauses that do not apply. This type of data shows more clearly what work needs to be done. The data collected by the AccessMonitor – the Web Accessibility validator used and developed by AMA – also left us a positive track by finding that 78% of the sites have an ‘AccessMonitor score’ greater than 5 (this on a scale from 0 to 10, where 10 represents good accessibility practices).

Similar results were observed in the 24 mobile applications. Problems with:

• 11.1.4.4 Resize text — 16 applications (100%)
• 11.1.3.4 Guidance — 15 applications (94%)
• 11.2.4.4 Purpose of the link (in context) — 14 applications (88%)
• 11.4.1.2 Name, function, value — 14 applications (88%)
• 11.1.3.1 Information and relations — 13 applications (81%)

Once again, we are talking about clauses that are behind the ‘Perceivable’ principle (3 clauses), ‘Robust’ (1) and ‘Operable’ (1).

Accessibility Statements

On the 281 websites analysed, accessibility statements were found in 13 websites, corresponding to 4.6% of the sample. However, 29 other websites were identified with an accessibility page, but which did not correspond to a statement built in accordance with the model proposed in Decree-Law No 83/2018. If the entities to which these websites belong to, correct their statements, we will soon have 15% websites with Accessibility Statements.

Part I — Description of monitoring activities

General information

The monitoring activities took place in the period from October 2021 to March 2022, according to the following distribution:

• Simplified website monitoring
  • Sample definition: October and November 2021
  • Sample collection: December 2021
  • Sample evaluation: January 2022
• In-depth monitoring of websites
  • Sample definition: February 2022
  • Sample evaluation: February and March 2022
• In-depth monitoring of mobile applications
  • Sample definition: October and November 2021
  • Sample evaluation: February and March 2022
• Evaluation of websites and mobile applications through testing with users with disabilities: March 2022

Monitoring is the responsibility of the Agency for Administrative Modernisation, I.P. (AMA). To carry out the monitoring activities and to prepare this report, AMA has contracted the services of FCIÊNCIAS.ID — Associação para a Investigação e Desenvolvimento de Ciências.

The sample of websites was provided by AMA, comprising a total of 281 websites. The number of websites in the sample results from the minimum size set out in point 2.1 of Annex I of the Implementing Decision (EU) 2018/1524, the value being equivalent to two sites per 100,000 inhabitants plus 75 sites. Given that the population resident in Portugal in 2020 is 10,298,252 inhabitants (according to INE statistics) the minimum number is 281 sites. The initial sample was analysed to ensure that the sites included were accessible to the monitoring activities and that they actually corresponded to different domains (i.e. that they did not redirect to other domains in the sample). After some adjustments, the final sample is as attached in Table 45.

Sample composition

The sample consists of a total of 281 websites and 16 mobile applications. The sample includes websites of different levels of administration as can be seen in Table 01.

It was also tried to ensure that the websites represented the different services provided by public sector bodies. The distribution of websites by type of service, presented in Table 02, illustrates the diversity of services that were considered in the sample.

Sample of websites for the simplified monitoring method

All 281 websites were initially considered for the simplified monitoring method. For this monitoring period it was decided to build two samples of pages for simplified monitoring:

• Sample 'Home+' — This sample is composed, for each website, by the homepage of the site plus all pages belonging to the domain linked from the homepage.
• Sample '2K' — This sample is composed, for each website, of the first two thousand pages of the site, obtained by breadth-first searching starting on the homepage. If the site is less than 2,000 pages, then all pages of the site are included in the sample.

Table 03 presents descriptive statistics of the sample of pages. Analysing the sample it is possible to notice that, on average, the sampled sites have 79 pages linked from the homepage and that, on average, each site has more than a thousand pages. This figure will in fact be higher since we have limited the collection to two thousand pages per site and 63 sites have reached this limit. This also means that in 78% (218/281 sites) of the sampled websites we are working with the total number of pages. Table 03 also presents statistics linked to the size of the pages. For each page the number of HTML elements was counted and, on the basis of this number, sites with at least 10 pages having at least 100 HTML elements were identified. As a result of this analysis, it was observed that, in the Home+ sample, 15 % of the websites did not reach this limit. With the 2K sample, the percentage of websites not reaching this limit is 9%.

Sample of websites for the in-depth monitoring method

The sample of websites took into account the results of the simplified monitoring and was composed of the following criteria:

• 5 websites presenting the best results in simplified monitoring;
• 5 websites showing the worst results in simplified monitoring;
• 4 websites requiring authentication to access a significant part of their content, and;
• 10 websites of the set of sites that in the simplified monitoring were classified as having less than 10 pages with 100 or more HTML elements.

The final sample of websites for in-depth monitoring is annexed in Table 46.

Sample of mobile applications for the in-depth monitoring method

The sampled mobile applications were indicated by AMA. The top applications were chosen from the list of government applications available at https://www.app.gov.pt. The iOS and Android applications of the same entity were considered individually in this analysis, and the two versions of 8 applications were analysed, totalling 16 mobile applications.

The final sample of mobile applications for in-depth monitoring is annexed in Table 47.

Correlation with standards, technical specifications and tools used for monitoring

Methodology applied in the simplified monitoring of websites

The sample pages of the 281 websites were obtained through the web crawling mechanism. In order to ensure the best possible coverage, two tools have been used for this purpose:

• QualWeb Crawler was used as the main mechanism for obtaining pages, with the advantage of being able to process pages that use JavaScript, and;
• simplecrawler was used as a secondary mechanism, having the advantage of being able to search in subdomains.

For the evaluation of the accessibility of each page of the sample, two automatic accessibility assessment tools were considered:

• AccessMonitor — This tool has validated compliance with 37 tests based on different sufficient and advised techniques of the Web Content Accessibility Guidelines (WCAG).
• QualWeb — This tool validated compliance with 54 ACT rules. The ACT rules are an objective interpretation of WCAGs, obtained through a process of harmonisation conducted to ensure the participation of the community of accessibility testers.

The clauses of EN 301 549 tested by AccessMonitor are presented in Table 04, together with the corresponding WCAG success criteria.

The Table 05 presents the clauses of EN 301 549 as well as the corresponding success criteria, tested by QualWeb through ACT rules.

The combination of the clauses tested by AccessMonitor and QualWeb, presented in Table 04 and Table 05, respectively, shows that the coverage rate of the 2 tools compared to the total of the clauses in Chapter 9 of the European Standard is 65%. Of those tested, 75 % have already used harmonised ACT (Accessibility Conformance Testing) rules.

The evaluation process consisted of the collection of QualWeb evaluations followed by the calculation of AccessMonitor results from QualWeb results. This ensures that the results presented by the two tools relate to the same source code.

During the evaluation process there were several occurrences of pages that did not respond to the evaluator’s requests. Multiple attempts were made to obtain a response, separated by at least two days between them. Still, it was not possible to evaluate a total of 27.022 pages, which corresponds to 9.6 % of the pages in the sample. It should be noted that, as a result of the problems with the evaluations, it was not possible to evaluate any page of three sites in the sample Home+, as well as no page of two websites in the sample 2K.

Methodology applied for in-depth monitoring of websites

The testing methodology for Trusted Testermanual evaluations was applied. This methodology is based on Web Content Accessibility Guidelines (WCAG) 2.0. Considering that this analysis follows standard EN 301 549 based on WCAG 2.1, new tests have been added to the tests originally included in the Trusted Tester methodology, comprising the WCAG Success Criteria not covered. The added tests were based on the sufficient techniques provided in WCAG 2.1 and are presented in Table 48 in the annexes. Three auxiliary tools were used to support the tests performed:

• Andi (Accessible Name & Description Inspector);
• Color Contrast Analyser (CCA), and;
• Bookmarklet for testing Text Spacing.

The Table 06 presents the clauses of standard EN 301 549 that were considered in this evaluation, together with the corresponding WCAG success criteria.

For each sampled website, a set of pages to be included in each assessment was established, as defined in Commission Implementing Decision (EU) 2018/1524 of 11 October 2018. This representative sample considers the following pages:

• Homepage;
• Login;
• Sitemap;
• Contact;
• Help;
• Legal information;
• At least one page for each type of service provided by the website;
• Other intended primary uses, including search;
• Accessibility Statement;
• Pages containing the feedback mechanism;
• Pages with substantially distinct appearance or with different content;
• Relevant downloadable document for each type of service;
• Any page deemed relevant;
• All pages belonging to the same process; and
• Randomly chosen pages corresponding to 10% of the total number of previously selected pages.

Each page included in the representative sample of each website was evaluated according to the WCAG 2.1 AA success criteria. For each page, one of the following results was provided:

• Conforming — The page passes all the tests established for the corresponding success criterion.
• Violation — The page does not pass one or more tests established for the corresponding success criterion.
• Not applicable — The page does not have the element or elements covered by the corresponding success criterion.

Data collection was supported by the WCAG-EM Report Tool. At the end of the evaluation of each website, the report generated was recorded for future analysis.

The results of the in-depth monitoring of websites have been complemented by usability testing for people with disabilities.

For these tests, 6 participants were recruited, two with visual impairment, two with motor impairment and two with hearing impairment. All participants were recruited through contacts with local institutions. In Table 07 and Table 08 we can observe that all participants have been computer users for more than five years, and that they self-evaluate themselves with different levels of expertise in the use of computers (between 1 and 5, where 1 represents beginner and 5 specialist). All sessions were conducted remotely using Zoom and screen sharing. Each session took about 45 min. Each session began with a brief introduction to the study, followed by a demographic questionnaire with questions related to computer usage and experience. Each participant was instructed to perform one task at a time, being able to ask the evaluator at any time to repeat the task. Participants were asked to think aloud while performing the task. The evaluator present took notes while observing participants’ interactions with the websites. When a participant was trapped in one step of the task, the evaluator helped overcome the problem so that the participant could explore the rest of the task. At the end of every two tasks with each of the websites, a semi-structured interview was carried out with three initial questions focused on the experience with the website, its accessibility, and finally what could be improved. Participants were rewarded for their contribution with a gift card worth EUR 20.

For these tests, the 5 websites considered were the ones with the highest failure rates in the in-depth methodology according to the functional performance statement for each group of users recruited. Some websites obtained the same failure rate in their functional performance statement. In these cases, websites with the highest failure rate were considered considering all clauses. This sample consisted of 5 websites, and for each of them, 2 tasks were defined considering the primary use or essential functionalities of each one. The tasks were distributed in two groups, with equivalent effort. Each task was performed by one participant from each group of recruited users.

Methodology applied for in-depth monitoring of mobile applications

The iOS and Android versions of the same application were considered individually in this analysis. The apps were installed directly from the respective App Stores and standard user accounts were used for their evaluation. We have selected a set of screens to assess in each application following the methodology set out in points 2 and 3 of Annex I to the COMMISSION IMPLEMENTING DECISION (EU) 2018/1524 of 11 October 2018. This representative sample considers the following screens:

• Homepage (first screen of each application);
• Login;
• Sitemap;
• Contacts;
• Help;
• Legal information;
• At least one screen for each type of service;
• Other intended primary uses, including search;
• Accessibility statement;
• Screens containing the feedback mechanism;
• Screens with substantially distinct appearance or with different content;
• Relevant downloadable document for each type of service;
• Any other screen deemed relevant;
• All screens belonging to the same process; and
• Randomly chosen screens corresponding to 10% of the total number of previously selected screens.

In screen analysis, mobile interface components such as App Drawers that are present on multiple screens were considered to be part of all screens where they are present.

The following tools and devices were used to support the manual evaluation of mobile applications:

• Talkback and VoiceOver, the default screen readers available on Android and iOS operating systems, respectively;
• Bluetooth keyboard;
• Accessibility Scanner — application that allows semi-automatic contrast analysis;
• Colour Contrast Analyser — application that allows you to determine the contrast between two colours.

The collection of evaluation data was supported by the WCAG-EM Report Tool.

The testing methodology for manual evaluations was based on the Trusted Tester methodology. This methodology is based on Web Content Accessibility Guidelines (WCAG) 2.0. Considering that this analysis follows the standard EN 301 549 based on WCAG 2.1, new tests have been added to the tests originally included in the Trusted Tester methodology, comprising non-covered Success Criteria. The added tests were based on sufficient techniques provided in WCAG 2.1. All possible tests to perform or adapt them have been performed in all applications. The success criteria considered in this assessment correspond to the AA level of WCAG 2.1, as set out in standard EN 301 594, and are presented in Table 09. Criteria are marked ‘partially’ when it was not possible to apply the test to one or more applications. In the specific case of the clause ‘10.1.4.10 Reflow, the criterion was not possible to evaluate in iOS (because there is no equivalent functionality) being evaluated only on Android. The clauses ‘10.1.4.3 Contrast (minimum)’ and ‘10.1.4.11 Non-text contrast’ have not been evaluated in 3 Android applications because the screenshot functionality is locked in them which prevents contrast testing.

At the end of the evaluation of each application, a report was recorded for future analysis. After conducting the evaluations, the results obtained were analysed, considering the clauses that are verified or violated.

After finalising the expert evaluation of all Android applications, the two less compliant were selected to conduct a study with users.

Four participants were recruited, two visually impaired and screen reader users, one user with motor impairment and one deaf user. In Table 10 and Table 11 we can observe that all participants have been mobile users for more than three years, and that they self-evaluate themselves with different levels of expertise in the use of smartphone (between 1 and 5, where 1 represents beginner and 5 specialist). Two had previous experience with one of the applications and one had tried in the past, unsuccessfully, to use the second. Participants were recruited by contacts with local institutions. The session with the participant P3 was conducted in person, while with the participants P1, P2 and P4, the session was remote using Zoom and screen sharing. The sessions lasted about 45 min. Each session started with a short demographic questionnaire with questions related to smartphone usage and experience. The participants then installed the two applications selected for the study if they did not have them on their device. Each participant was instructed to perform one task at a time, being able to ask the evaluator at any time to repeat the task. Participants were asked to think aloud while performing the task. The evaluator took notes while observing participants’ interactions with their devices. When a participant was trapped in one step of the task, the evaluator helped overcome the problem so that the participant could explore the rest of the task. At the end of every two tasks with each of the applications, authorisation was requested to record audio, and a semi-structured interview was performed with three initial questions focused on the application experience: 1) “How was your experience with this application?”; 2) “What is your opinion on the accessibility of this application?”; and 3) “What can be improved?” Participants were rewarded for their contribution with a gift card worth EUR 20.

According to Statcounter Global, in Portugal, the percentage of Android users is 72%, compared to 27% iOS. Since Android is the most popular (more than double) operating system in Portugal, user evaluations were conducted on Android applications. It should be noted that there were no restrictions on recruitment and that all users contacted to participate in the study were found to be Android users.

The least compliant Android applications were selected. However, it should be noted that two others, if selected, would have been inaccessible to screen reader users as it is not possible to interact with the elements necessary for navigation using a screen reader. In the selected applications, two typical tasks were created for the objectives inherent to them.

Part II — Results of monitoring

Detailed results

Simplified website monitoring

In total, 281 websites were analysed, of which 281,706 pages were obtained. Of this total, 254,684 pages were evaluated, corresponding to 90.4 % of the sample. It can also be noted that on six websites only one page could be obtained.

Home+ Sample analysis

In the sample Home+, out of the 281 websites, 22,489 pages were collected, corresponding to an average of 80 pages per website. In Graph 07 it is possible to see the distribution of the number of pages per website and it is noticeable that for this sample most websites have less than 50 pages. The mode is 25 to 50-page sites. Please note that given the method of collecting pages from the Home+ sample, this number of pages also represents the number of links on the Homepage to the domain. The fact that three quarters of the sites have a number of links below 100 on the Homepage can be positive and beneficial for the accessibility of the site. On the contrary, 6.5 % of sites have more than 200 links on the Homepage alone and 1 % even reaches more than 500 hyperlinks, which leads to a complex homepage to navigate.

Note: if you are unable to consult the following chart for any reason, see the data in Table 12.

The Table 12 provides details on the distribution of pages by website in this sample.

Of the 281 websites, 3 failed to have at least one page evaluated, and 38 more failed the requirement to have at least 10 pages evaluated with 100 or more HTML elements, so they are not considered in the following automatic analysis. Thus, for this analysis only 240 websites with at least 10 pages evaluated with 100 or more HTML elements are considered. In total, in this sample, the results of the evaluations of 19,931 pages belonging to 240 sites were analysed.

Distribution of EN 301 549 clauses (H+)

The number (and percentage) of websites verifying or violating each clause is displayed in Table 13, as well as those requiring manual validation or not applicable on any website page.

From Table 13 we can conclude that there was a low compliance rate compared to the clauses of EN 301 549.

The clauses with the highest index of non-compliance were:

• 9.4.1.2 Name, role, value with 97.9 % of applicable websites not compliant;
• 9.1.4.3 Contrast (minimum) with 92.1 % of applicable websites not compliant;
• 9.2.4.4 Link Purpose (in context) with 89.2 % of the applicable websites not compliant;
• 9.1.3.1 Info and relationships with 74.6 % of applicable non-compliant websites;
• 9.1.1.1 Non-text content containing 72.1 % of applicable non-compliant websites;
• 9.4.1.1 Parsing with 69.6 % of applicable non-compliant websites.

Distribution of functional performance statements (H+)

Two analyses were carried out in relation to these statements:

• The first one considering the clauses supporting functional performance statements (primary relationships). For this analysis, based on the evaluated clauses, it was possible to consider 8 of the 11 functional performance statements.
• The second one considering all clauses supporting, in whole or in part, functional performance statements (primary and secondary relationships). For this analysis, based on the evaluated clauses, it was possible to consider 10 of the 11 functional performance statements.

The Table 14 presents the results obtained in the first analysis, in which only the clauses of the primary relationships were considered.

Observing Table 14 we can conclude that there is a large index of non-conformity in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statement with the lowest rate of non-compliance was:

• Usage with limited reach (LR) with a non-conforming rate of 33.9% in applicable websites.

The Table 15 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were considered.

Observing Table 15, we can conclude that there is a high index of non-compliance with functional performance statements, when assessing all clauses. The functional performance statement with the lowest rate of non-compliance was:

• Usage with limited reach (LR) with a non-compliance rate of 33.9% in applicable websites.

Distribution of AccessMonitor tests and ACT rules per website (H+)

The good and bad web accessibility practices identified by AccessMonitor on the websites are presented in Table 16.

The most frequent good practice on websites is “I found a title on the page, and it seems me correct”, applied on 99.6% of websites. The least common good practice is “I Located in the metadata of the page, X elements <link> that can be used to build a navigation system”, applied on only 0.4% of websites.

The most frequent bad practice on websites is “I located X colour combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG (i.e. 3 to 1 for large letter text and 4.5 to 1 for normal letter text”, verified at 92.1% of websites. The least frequent bad practice on websites is “I found X abbreviations in which you forgot to put the extension”, verified in 0.4% of websites.

The Table 17 displays the number of websites that pass, fail and require manual validation at least once by ACT rule tested. From the analysis of Table 17 we can verify that 33 ACT rules passed at least once, and 27 ACT rules failed at least once in the whole sample Home+. Six ACT rules have been found that fail on more websites than the ones where they pass:

• Link has non-empty accessible name
• ARIA required context role
• ARIA required owned elements
• Object element rendering non-text content has non-empty accessible name
• iframe with negative tabindex has no interactive elements
• First focusable element is link to non-repeated content

If we do not consider the rule “First focusable element is link to non-repeated content” because it can only automatically detect cases where the rule is not complied with, without being able to automatically detect cases where it is, we can verify that we are in the presence of four types of problems: (1) lack of names accessible in HTML elements, (2) incorrect use of ARIA, (3) incorrect attribution of focus to , and (4) use of colours with insufficient contrast.

Some of the rules feature a zero in all columns. This means that no page was found in the sample where the rule was applicable.

Distribution of AccessMonitor tests and ACT rules per page (H+)

The Table 18 presents the number of pages for which each AccessMonitor test is true, as well as the type of practice that the test validates. From the analysis of Table 18 we can verify that all good practices and all bad practices have been detected on at least one page. Of the 11 good practices analysed, all take place on at least 6 pages. Of the 28 bad practices, they all occur on more than one page. The average of pages that follow each good practice is 45%, while the average of pages that apply each bad practice is 18%.

The good practice applied by more pages is “I found a title on the page, and it seems me correct”, applied to 99.1% of the pages. The least applied good practice is “I located in the metadata of page X elements <link> that can be used to build a navigation system”, applied on only 0.03% of pages. The good practices under these conditions are the same as those found in the website analysis.

The bad practice applied on more pages is “I found that the first page link does not allow to jump directly to the main content area”, verified in 70% of the pages. The bad practices applied on fewer pages are “I found X abbreviations where you forgot to put the extension” and “I verified that this page has no title — you lack the element <title>”, verified in 0.01% of pages. Bad practices under these conditions vary from what was found in the website analysis.

The Table 19 displays the number of pages that pass, fail and require manual validation by ACT rule tested. From the analysis of Table 19 we can see that there is similarity to what happens to websites: 33 ACT rules have passed at least once and 27 ACT rules have failed at least once. Also, in this analysis were found six ACT rules that fail in more pages than those in which they pass. However, the six rules are not the same:

• Link has non-empty accessible name
• Visible label is part of accessible name
• headers attribute specified on a cell refers to cells in the same table element
• Object element rendering non-text content has non-empty accessible name
• MenuItem has non-empty accessible name
• First focusable element is link to non-repeated content

Continuing not to consider the rule “First focusable element is link to non-repeated content” due to the reasons presented above, we can verify that we are in the presence of three types of problems: (1) lack of accessible names in HTML elements, (2) incorrect use of the headers attribute in tables, and (3) use of colours with insufficient contrast.

AccessMonitor Score Distribution (H+)

The distribution of the scores of the 240 websites is presented in Graph 08. From the graph we can observe that the range of scores from 5 to 6 (inclusive) contains the highest concentration of websites. Additionally, only 1 website, corresponding to 0.42% of the sample, has a score of 10.

Note: if you are unable to consult the following chart for any reason, see the data in Table 20.

The distribution of the scores of the 240 websites is presented in Table 20. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

The distribution of the 19,931 evaluated pages of the 240 websites is presented in Graph 09. From the graph we can observe that the range of scores from 6 to 7 (inclusive) contains the highest concentration of pages. Additionally, only 41 pages, corresponding to 0.2% of the sample, have a score of 10.

Note: if you are unable to consult the following chart for any reason, see the data in Table 21.

The distribution of the 19,931 pages of the 240 websites is presented in Table 21. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive) despite the mode being the range from 6 to 7 (inclusive) with 31.3% of the pages.

2K Sample analysis

In the 2K sample, of the 281 websites, 281,706 pages were collected, corresponding to an average of 1003 pages per website. The Graph 10 presents the distribution of the number of pages per website. 30% of sites have more than 1750 pages. At the opposite end, 33% has less than 250 pages.

Note: if you are unable to consult the following chart for any reason, see the data in Table 22.

Table 22 provides details on the distribution of the number of pages per website in this sample.

Of the 281 websites, 2 failed the requirement to have at least one page evaluated, and 24 failed to have at least 10 pages with 100 or more elements evaluated, so they are not considered in the analysis below. Thus, for this analysis only 255 websites with at least 10 pages evaluated with 100 or more HTML elements are considered. In total, in this sample, 223,331 pages were evaluated.

Distribution of EN 301 549 clauses (2K)

The number (and percentage) of websites verifying or violating each clause of EN 301 549 is displayed in Table 23, as well as those requiring manual validation or not applicable on any page of the website.

From Table 23 we can conclude that there was a low compliance rate compared to the tested clauses.

The clauses with the highest index of non-compliance were:

• 9.4.1.2 Name, role, value with a failure rate of 99.2% compared to applicable websites;
• 9.2.4.4 Link Purpose (in context) with a failure rate of 95.3% compared to the applicable websites;
• 9.1.4.3 Contrast (minimum) with a failure rate of 94.9% compared to applicable websites;
• 9.1.3.1 Info and relationships with a failure rate of 84.3% compared to applicable websites;
• 9.1.1.1 Non-text content with a failure rate of 82% compared to applicable websites;
• 9.4.1.1 Parsing with a failure rate of 79.2% compared to applicable websites;

Distribution of EN 301 549 Functional Performance Statements (2K)

Two analyses were carried out in relation to these statements:

• The first one considering the clauses supporting functional performance statements (primary relationships). For this analysis, based on the evaluated clauses, it was possible to consider 9 of the 11 functional performance statements.
• The second one considering all clauses supporting, in whole or in part, functional performance statements (primary and secondary relationships). For this analysis, based on the evaluated clauses, it was possible to consider 10 of the 11 functional performance statements.

The Table 24 presents the results obtained in the first analysis, in which only the clauses of the primary relationships were considered.

Observing the Table 24, we conclude that there is a great failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statement with the lowest rate of non-compliance was:

• Usage with limited reach (LR) with a non-conforming rate of 34.9% in applicable websites.

The Table 25 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were considered.

Observing the Table 25, we can conclude that there is a high index of non-compliance with functional performance statements, when assessing all clauses of EN 301 549. The functional performance statement with the lowest rate of non-compliance was:

• Usage with limited reach (LR) with a non-conforming rate of 34.9% in applicable websites.

Distribution of AccessMonitor tests and ACT rules per website (2K)

The Table 26 displays the number of websites on which each AccessMonitor test is true, as well as the type of practice (good or bad practice) that the test validates. From the analysis of Table 26 we can verify that of the 12 good practices analysed, two are checked on only one website (“I identified X cases where redundant event handlers are used” and “I located in the metadata of the page X elements that can be used to build a navigation system”). On the other hand, of the 28 bad practices, they all occur on more than one website. The average of websites following each good practice is 56%, while the average of websites applying each bad practice is 46%.

The Table 27 displays the number of websites that pass, fail and require manual validation at least once by ACT rule tested. From the analysis of Table 27 we can verify that 33 ACT rules passed at least once and 28 ACT rules failed at least once in the entire 2K sample. Six ACT rules were found that fail in more websites than those in which they pass, these being the same 6 rules in which this was found in the Home+ sample:

• Link has non-empty accessible name
• ARIA required context role
• ARIA required owned elements
• Object element rendering non-text content has non-empty accessible name
• iframe with negative tabindex has no interactive elements
• First focusable element is link to non-repeated content

If we do not consider the rule “First focusable element is link to non-repeated content” due to the above reasons, we can see the same four types of problems: (1) lack of accessible names in HTML elements, (2) incorrect use of ARIA, (3) incorrect attribution of focus to iframes, and (4) use of colours with insufficient contrast.

Some of the rules have a zero in all columns. This means that no pages were found in the sample where the rule was applicable.

Distribution of Accessmonitor tests and ACT rules per page (2K)

The Table 28 displays the number of pages on which each AccessMonitor test is checked, as well as the type of practice that the test validates. By analysing the Table 28 we can verify that all good practices and all bad practices have been detected on at least one page. Of the 11 good practices analysed, all are checked on at least 5 pages. Of the 28 bad practices all occur in at least 30 pages. The average of pages that follow each good practice is 44%, while the average of pages that apply each bad practice is 18%.

The good practice applied by more pages is “I found a title on the page, and it seems me correct”, applied to 99.3% of the pages. The least applied good practice is “I’ve Identify X cases in which redundant event handlers are used”, applied in only 5 pages.

The bad practice applied on more pages is “I located X colour combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG (i.e. 3 to 1 for text with large letter and 4.5 for 1 for text with normal letter.”, verified in 72% of pages. The bad practice found in fewer pages is “I found X abbreviations in which you forgot to put the extension”, verified in 0.01% of the pages.

The Table 29 displays the number of pages that pass, fail and require manual validation by ACT rule tested. By analysing the Table 29, we can verify that 33 ACT rules passed at least once and 28 ACT rules failed at least once. Five ACT rules have been found that fail on more pages than those in which they pass:

• Link has non-empty accessible name
• Visible label is part of accessible name
• Object element rendering non-text content has non-empty accessible name
• iframe with negative tabindex has no interactive elements
• First focusable element is link to non-repeated content

Continuing not to consider the rule “First focusable element is link to non-repeated content” due to the reasons presented above, we can verify that we are in the presence of three types of problems: (1) lack of accessible names in HTML elements, (2) incorrect attribution of focus to iframes, and (3) use of colours with insufficient contrast.

AccessMonitor Score Distribution (2K)

The distribution of the AccessMonitor Scores on the 255 websites of the sample 2k is presented in Graph 11. From the graph we can observe that the range of scores ]5,6] contains the highest concentration of sites (30%). It should also be noted that 78% of websites have an AccessMonitor Score of more than 5 — this on a scale of 1 to 10. From the data collected we can also see that of the 8 sites in the scoring range ]9,10] there are only one site with a score of 10.

Note: if you are unable to consult the following chart for any reason, see the data in Table 30.

The distribution of the scores of the 255 websites is presented in Table 30. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

The distribution of the scores of the 223,331 pages evaluated of the 255 websites is presented in Graph 12. From the graph we can observe that the range of scores from 5 to 6 (inclusive) contains the highest concentration of pages. Additionally, only 44 pages, corresponding to 0.0018% of the sample, have a score of 10.

Note: if you are unable to consult the following chart for any reason, see the data in Table 31.

The distribution of the scores of the 223,331 pages of the 255 websites is presented in Table 31. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

Analysis of the results of simplified website monitoring

The simplified monitoring method identified the most frequent non-compliances with the clauses of EN 301 549 that can be assessed with the automated tools that have been used. The non-compliances that stand out:

• Two clauses associated with the WCAG “Robust” principle are on both lists with the highest non-compliance rate for the two samples. This is representative of the state of the web where content is not built considering the needs of assistive technologies. This prevents these technologies from being able to communicate correctly to their users the content of the pages they wish to consult. Problems relating to the communication of the accessible name or role of elements occur on more than 95% of the websites analysed. Problems associated with the correct use of HTML elements occur on at least 70% of the sampled websites.
• Content creators still do not create links with a description that allows users to understand the purpose of the hyperlinks. Pages with multiple links with the same text (typically a variation of “Read more” or “Click here”) exist on more than 95% of the websites analysed and are problematic for users browsing the pages using a list of links.
• Websites still do not respect the minimum contrast ratios that ensure the ability to read the text presented on their pages. This problem occurred on more than 90% of websites.
• There were no alternative descriptions for non-text content, mostly images, in more than 80% of the websites analysed. This bad practice prevents users who are unable to see the images from being able to use the website in full.
• The failure of more than 80% of websites to comply with the clause ‘9.1.3.1 Info and Relationships’ reveals various types of problems that prevent users of assistive technologies from having a correct perception of the content and structure of the page. Examples of issues associated with this clause include unnamed headers, table cells without headers, incorrect use of ARIA attributes, or unlabelled form fields.

The 2K sample, compared to the Home+ sample, shows that problems are even more frequent when extending the number of pages (the default rate for all problematic clauses grows in sample 2K). Several EN clauses show significant growth in the rate of non-compliance when considering the 2K sample. For example, 72% of the websites in the Home+ sample do not comply with the clause “9.1.1.1 Non-text content” while the percentage rises to 82% in the 2K sample. In clause “9.1.3.1 Info and relationship” the evolution is from 75% to 84%. The “9.2.1.1 Keyboard” increases from 8% to 13%. In “9.2.4.4 Link Purpose (in context)” rises from 89% to 95%. In “9.3.1.1 Language of the page” increases from 20% to 28%. In “9.4.1.1 Parsing” rises from 70% to 79%. The Home+ sample detects the diversity of the problems, but with a lower incidence rate. It does not provide a realistic coverage of the problems related to several clauses. This may be a result of the teams managing the websites paying more attention to the pages that are closer to the top of the hierarchy of the site because they are probably more visited. However, if this is the case, it is not a justification for not paying the same level of attention to all pages made available on the website. However, this logic does not hold for all the accessibility principles analysed. The average compliance of the Understandable principle shows that non-conformities detected are more easily diluted in larger samples such as 2K.

We remind you that the Home+ sample is the one used in the Portuguese Web Accessibility Observatory and that the logic of correcting the occurrences found should be applied not only to the sample but, in a cross-sectional way, to the entire website.

In-depth monitoring of websites

The results described in this section refer to the analysis of 24 websites. In total, 477 pages were analysed, corresponding to an average of 20 pages per website.

Distribution of EN 301 549 clauses per website (in-depth)

The Table 32 displays the number (and percentage) of websites verifying or violating each clause of EN 301 549 as well as those that do not apply to any page of the website.

The average non-compliance and compliance rates of the clauses of EN 301 549 applicable to each website are presented in Graph 13 and Graph 14. To facilitate the reading, the clauses corresponding to WCAG level ‘A’ were considered in Graph 13 and the clauses corresponding to WCAG level ‘AA’ were considered in Graph 14. Although the European Standard does not make this division by compliance levels, it proves useful for those who need to make corrections to the practices found. In the logic of WCAG 2.1 it will be rational to begin by eliminating the problems encountered at level ‘A’. A quick observation of both graphs shows us that the non-compliant clauses are around 30% for level ‘A’ and around 40% to 50% for the level ‘AA’ criteria. The number of clauses level ‘A’ is also significantly higher than the clauses expressing the level ‘AA’.

Note: if you are unable to consult the following chart for any reason, see the data in Table 32.

Note: if you are unable to consult the following chart for any reason, see the data in Table 32.

According to the data set out in Table 32, it can be concluded that there was a low compliance rate of the clauses.

The 11 clauses with the highest success rate are:

• 9.2.3.1 Three Flashes or Below with a 100% success rate in applicable websites;
• 9.2.5.2 Pointer Cancellation with a 100% success rate in applicable websites;
• 9.1.3.4 Orientation with a 96% success rate in applicable websites;
• 9.2.1.2 In Keyboard Trap with a 96% success rate in applicable websites;
• 9.3.2.1 On Focus with a success rate of 96% in applicable websites;
• 9.3.2.2 On Input with a 95% success rate in applicable websites;
• 9.3.3.4 Error Prevention (Legal, Financial, Data) with a 90% success rate in applicable websites;
• 9.2.5.1 Pointer Gestures with a success rate of 83% in applicable websites;
• 9.3.3.1 Error Identification with a success rate of 83% in applicable websites;
• 9.1.3.3 Sensory Characteristics with a success rate of 80% in applicable websites; and
• 9.3.2.3 Consistent Navigation with a success rate of 80% in applicable websites.

The remaining clauses achieved a success rate of 78% or less in the applicable websites.

The 11 clauses with the highest failure rate were:

• 9.1.1.1 Non-text Content with a failure rate of 100% in applicable websites;
• 9.1.2.1 Audio-only and Video-only (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.2 Captions (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.3 Audio Description or Media Alternative (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.5 Audio Description (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.3.1 Info and Relationships with a failure rate of 100% in applicable websites;
• 9.1.4.2 Audio Control with a failure rate of 100% in applicable websites;
• 9.4.1.1 Parsing with a failure rate of 100% in applicable websites;
• 9.1.4.10 Reflow with a failure rate of 96% in applicable websites;
• 9.2.4.4 Link Purpose (In Context) with a failure rate of 96% in applicable websites; and
• 9.4.1.2 Name, Role, Value with a 96% failure rate in applicable websites.

The remaining clauses obtained a failure rate of 95% or less.

Distribution of EN 301 549 Functional Performance Statements (in-depth)

Two analyses were carried out in relation to these statements:

• The first one considering the clauses supporting functional performance statements (primary relationships); and
• The second one considering all clauses supporting, in whole or in part, functional performance statements (primary and secondary relationships).

For this analysis, all 11 functional performance statements were considered.

Table 33 presents the results obtained in the first analysis, in which only the clauses of the primary relationships were considered. it can be concluded that there was a low compliance rate of the clauses.

The 11 clauses with the highest success rate are:

• 9.2.3.1 Three Flashes or Below with a 100% success rate in applicable websites;
• 9.2.5.2 Pointer Cancellation with a 100% success rate in applicable websites;
• 9.1.3.4 Orientation with a 96% success rate in applicable websites;
• 9.2.1.2 In Keyboard Trap with a 96% success rate in applicable websites;
• 9.3.2.1 On Focus with a success rate of 96% in applicable websites;
• 9.3.2.2 On Input with a 95% success rate in applicable websites;
• 9.3.3.4 Error Prevention (Legal, Financial, Data) with a 90% success rate in applicable websites;
• 9.2.5.1 Pointer Gestures with a success rate of 83% in applicable websites;
• 9.3.3.1 Error Identification with a success rate of 83% in applicable websites;
• 9.1.3.3 Sensory Characteristics with a success rate of 80% in applicable websites; and
• 9.3.2.3 Consistent Navigation with a success rate of 80% in applicable websites.

The remaining clauses achieved a success rate of 78% or less in the applicable websites.

The 11 clauses with the highest failure rate were:

• 9.1.1.1 Non-text Content with a failure rate of 100% in applicable websites;
• 9.1.2.1 Audio-only and Video-only (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.2 Captions (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.3 Audio Description or Media Alternative (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.2.5 Audio Description (Pre-recorded) with a failure rate of 100% in applicable websites;
• 9.1.3.1 Info and Relationships with a failure rate of 100% in applicable websites;
• 9.1.4.2 Audio Control with a failure rate of 100% in applicable websites;
• 9.4.1.1 Parsing with a failure rate of 100% in applicable websites;
• 9.1.4.10 Reflow with a failure rate of 96% in applicable websites;
• 9.2.4.4 Link Purpose (In Context) with a failure rate of 96% in applicable websites; and
• 9.4.1.2 Name, Role, Value with a 96% failure rate in applicable websites.

The remaining clauses obtained a failure rate of 95% or less.

Observing the Table 33, we can conclude that there is a great failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statements with the highest success rate were:

• Usage with limited reach with a rate of 21% compared to applicable websites;
• Minimize photosensitive seizure at a rate of 14% compared to the applicable websites, and;
• Usage with limited hearing at a rate of 6% compared to applicable websites.

The remaining functional performance statements achieved a success rate of 4% or less, with 3 of them having a 0% rate.

The Table 34 presents the results obtained in the first analysis, considering, however, the average compliance rate of the sample analysed.

The average rates of compliance with the clauses of each functional performance statement are presented in Graph 15, considering the primary relationships, representing the current state of the sample analysed.

Note: if you are unable to consult the following chart for any reason, see the data in Table 34.

Observing the Table 34 we can conclude that there is a large average failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statements with the highest index of non-compliance are:

• Usage with limited vision with a non-compliance rate of 46%;
• Usage without perception of colour with a non-compliance rate of 45%;
• Usage without vision with a non-compliance rate of 41%;
• Usage with limited manipulation capability and/or strength with a non-compliance rate of 36%;
• Usage with limited cognition capabilities with a non-compliance rate of 35%, and;
• Usage without hearing with a non-compliance rate of 34%.

The remaining functional performance statements achieved a non-compliance rate of less than 25%.

The Table 35 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were considered.

Observing the Table 35 we can conclude that there is a great failure rate in relation to functional performance statements, when evaluating all clauses. Only three functional performance statements have non-compliance rate different from 100%.

The Table 36 presents the results obtained in the second analysis, considering, however, the average compliance rate of the sample analysed.

The average rates of compliance with the clauses of each functional performance statement are presented in Graph 16 considering the primary and secondary relationships, representing the current state of the sample analysed.

Note: if you are unable to consult the following chart for any reason, see the data in Table 36.

Observing the Table 36 we can conclude that there is a large average failure rate in relation to functional performance statements, when evaluating all clauses. The functional performance statements with the highest level of non-compliance were:

• Usage with limited vision with a 47% non-compliance rate;
• Usage without voice capability with a 47% non-conformity rate, and;
• Usage without perception of colour with a 45% non-compliance rate.

The remaining functional performance statements obtained an average non-compliance rate of less than 40%.

Analysis of the results of in-depth monitoring of websites

The in-depth monitoring method identified the most frequent non-compliances with the clauses of EN 301 549 applicable to a manual website review. Non-compliances stand out:

• There were no alternative descriptions for non-text content, mostly images, on all websites analysed. This bad practice prevents users who are unable to see images from using the website in full.
• Media content, such as audio and video integrated in web pages, are constantly made available without an alternative version with subtitle or audio description, not considering users who need to access these alternative versions to understand the content correctly.
• Flaws were identified that prevents proper presentation whenever users need to expand the text. Some of the problems associated with this are the loss of information or functionality and a significant increase in the reading effort caused by the need to perform horizontal scrolling.
• Non-compliance with the clause “9.1.3.1 Info and Relationships” on all websites analysed reveals various types of problems that prevent users of assistive technologies from correctly perceiving the content and structure of the page. Examples of issues associated with this clause include unnamed headers, unlabelled form fields, or unidentified lists and tables.
• Content creators still do not create links with a description that allows users to understand the purpose of the link. Pages with hyperlinks without an out-of-context description are problematic for users browsing the pages through hyperlink lists — this is the case for many blind users who make use of screen readers.
• Two of the clauses linked to the WCAG ‘Robust’ principle are on the list of clauses with the highest rate of non-compliance. This is representative of the state in which the Web is — the content is not built taking into account the needs of assistive technologies. This prevents these technologies from being able to communicate correctly to their users the content of the pages they wish to consult. Problems relating to the communication of the accessible name or (semantic) role of elements occur on all the websites analysed. The same applies to problems associated with the correct use of HTML elements — all websites have grammatical HTML errors.

In-depth monitoring of mobile applications

16 applications were analysed, 8 Android applications and 8 iOS applications. In the total expert evaluations, 144 screens (70 in Android and 74 in iOS applications) were considered, corresponding to an average of 9 screens per application.

Mobile Applications Results by EN 301 549 Clauses

The number (and percentage) of applications verifying or violating each clause is displayed in Table 37, as well as those that do not apply to any of the screens evaluated or cannot be determined.

According to the data set out in Table 37, it can be concluded that compliance and non-compliance rates have similar values. The average compliance rate of the assessed clauses is 51% while the average non-compliance rate is 49%. Only 2 clauses passed through the 16 applications analysed and all applications failed at least one clause.

The 9 clauses with the highest success rate are:

• 11.2.5.2 Pointer Cancellation — 16 Applications (100%)
• 11.3.2.1 On focus — 16 applications (100%)
• 11.2.1.2 No keyboard trap — 14 applications (88%)
• 11.2.5.3 Label in Name — 14 applications (88%)
• 11.3.2.2 On input — 13 applications (81%)
• 11.2.4.3 Focus order — 12 applications (75%)
• 11.2.4.6 Headings and labels — 12 applications (75%)
• 11.2.4.7 Focus visible — 12 applications (75%)
• 11.3.3.1 Error identification — 12 applications (75%)

The remaining clauses achieved a success rate of less than 75%.

The clauses with the highest failure rate were:

• 11.1.4.4 Resize text — 16 applications (100%)
• 11.1.3.4 Orientation — 15 applications (94%)
• 11.2.4.4 Link purpose (in context) — 14 applications (88%)
• 11.4.1.2 Name, role, value — 14 applications (88%)
• 11.1.3.1 Info and relationships — 13 applications (81%)
• 11.1.1.1 Non-text content — 12 applications (75%)
• 11.1.4.3 Contrast (minimum) — 12 applications (75%)
• 11.2.4.2 Page titled — 12 applications (75%)

The remaining clauses obtained a failure rate of less than 75%.

It can be verified that none of the applications makes use of audio feedback, so the criteria associated with audio and subtitling were not analysed. It should be noted that two of the evaluated applications were totally inaccessible by screen reader.

Distribution of EN 301 549 Functional Performance Statements (Apps)

Two analyses were carried out in relation to these statements:

• The first taking into account the clauses supporting functional performance statements (primary relationships), and;
• The second one taking into account all clauses supporting, in whole or in part, functional performance statements (primary and secondary relationships).

The Table 38 presents the results obtained in the first analysis, in which only the clauses of the primary relationships were taken into account.

Observing the Table 38, we can conclude that there is a great failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. Four of the functional performance statements are non-compliant on all sites analysed. Another four manage to have a compliance rate different from zero:

• Usage with limited hearing with a compliance rate 50%;
• Usage without hearing with a compliance rate of 25%;
• Usage without perception of colour with a compliance rate of 6%, and;
• Usage with limited reach with a compliance rate of 6%.

The Table 39 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were taken into account.

A similar situation to the first analysis to apps was observed in Table 35 to websites. Thus, 6 functional performance statements are non-compliant in all the apps analysed, with the number of four functional performance statements remaining at a compliance rate above zero, although with lower values than in the first analysis:

• Usage without hearing with a compliance rate of 25%;
• Usage with limited hearing with a compliance rate 25%;
• Usage without perception of colour with a compliance rate of 6%, and;
• Usage with limited reach with a compliance rate of 6%.

Analysis of results of in-depth monitoring of mobile applications

The in-depth monitoring method identified the most frequent non-compliances with clauses EN 301 549 applicable to a manual analysis of mobile applications. It should be noted that only 59% of the clauses are compliant, many with a significant impact on the use of these applications. Mobile evaluations have been adapted from the techniques described in the Trusted Tester test methodology and benefit from a similar standardisation to that performed for website evaluations. Of the detected non-compliances, the following stand out:

• The frequent existence of content without textual alternative was verified.
• Applications regularly force the use of the device into a guideline, limiting access for disabled users who take advantage of other guidelines.
• Graphic and colour elements are regularly used as the only mechanism of assigning meaning to interactions.
• Non-compliance with the clause “10.1.3.1 Info and Relationships” by various applications analysed reveals various types of problems that prevent users of assistive technologies from having a correct perception of the content and structure of the screen. Examples of issues associated with this clause include unnamed headers, unlabelled form fields, or unidentified lists and tables.
• Content creators still do not create interactive components with a description that allows users to understand their purpose.
• All evaluated iOS applications revealed problems with compliance with contrast levels (10.1.4.3 Contrast (minimum), 10.1.4.11 Non-text contrast) and with the relationship between content (10.1.3.1 Info and relations).
• None of the evaluated applications allowed the font size to be adjusted, nor did they support the functionality of the operating system that automatically provides the service.
• There was the use of immense interactive elements without a concrete definition of function. For example, buttons were not advertised as such and texts were interactive, which led users to have to test virtually all the elements to discover those that were really interactive.

Complementary content

Usability tests' results with participants with disabilities

Websites' testing

In this study, 6 participants were present, two with visual impairment, two with motor impairment, and two with hearing impairment. The participants of this study were all experienced users. Despite the various problems encountered and observed during the session, all tasks were successfully completed without major interventions of the evaluator. The most common problems observed and reported by the participants were as follows:

• Various elements, such as images, hyperlinks and buttons, without an accessible name, with a non-descriptive accessible name, or, in the case of decorative images, without correct identification. In this case, the two screen reader users reported having to use different navigation strategies to better understand the context and possible information made available on the webpage, making this interaction much more costly;
• Section headings missing or inconsistent. In the first case, for screen reader users, difficulties arise to identify when they switched from one section of the webpage to another, as well as making it impossible to use these headings to find the content of interest faster, making this interaction much more costly again. In the second case, the structure of the web pages announced by the screen reader did not reflect the same visual hierarchical structure, causing participants to take longer to understand the structure of the webpage and thus find the desired information;
• Incorrect marking of states and properties of elements. This problem has two consequences for users. The first, reflected by the lack of information and inconsistency in the behavior of similar elements, such as hyperlinks and buttons directing the user to a new page or adding content on the same page, without correct notification to the user of a screen reader. The second consequence of this problem is in status messages that, although not visible, were still available to screen readers. In this case, the user could not distinguish which messages were actually valid for the interaction context;
• Video content without sufficient contextual information. In the context of this study, the lack of subtitling for synchronised media content was identified, leaving the user with hearing loss without any information about the content being transmitted by audio. In addition, video content used as alternatives to textual content was also identified, but without correct identification. In this case, a screen reader user could not identify the actual function of the element so as to understand that the element corresponded to an alternative to textual content already provided on the website;
• Finally, all participants reported the excess of textual information present on most websites analysed. While much of this content can be considered essential, in parallel with some of the previously mentioned problems, i.e. lack of navigation resources and structural elements, it represented an additional obstacle for users. It is noteworthy that some of the websites analysed provided a search functionality, explored by some of the participants who agreed to be a useful solution that they usually use whenever available. Another strategy used on a website to mitigate this problem, highlighted by a participant, was the use of visual and symbolic resources, helping to recognise the information provided also in textual mode. Additional suggestions provided by participants also included the provision of alternative sign language content, or the use of easier-to-understand language.

Mobile Applications' Testing

Participants in this study are all experienced smartphone users who have been using for more than three years. Despite the numerous problems encountered and observed during the session, except for two tasks with two different users, all the others were successfully completed without the intervention of the evaluator. The first task in question, in fact, was successfully completed from the perspective of the participant, but unfortunately the information provided by the application, which led him to believe that he was facing the right result, was incorrect. This could have had serious consequences if we were in a real context of use. The failure in the second task was due to a problem of content exploration that all participants felt when they were told to find a specific service. The most common problems observed and reported by the participants were as follows:

• The structure and graphics of the screen facilitated the understanding of the information, however, when the content of the screen was accessed by a screen reader, the relationships and descriptions of the content, although correct, were neither clear nor perceptible;
• The contents and navigation of one of the applications required the user to have prior knowledge of the services and their internal organisation, presenting themselves with acronyms and sequences of menus that made the choices not obvious. Aggravating the situation there was no search function available;
• Both applications have poorly identified components with no alternative text. Some components appeared interactively when they were not, and others appeared with untranslated texts, presenting themselves in different languages, which greatly increases the difficulty of access to those who use screen reader;
• Both applications had non-accessible elements, however they were not fundamental to the execution of the tasks as screen reader users were able to find alternative paths and proceed.

Comparison of results of simplified and in-depth monitoring of websites

The in-depth monitoring sample selected the five websites with the best and the five websites with the worst results in simplified monitoring. This makes it possible to compare the results of these websites in the two monitoring methodologies. For this analysis, the percentage of compliant and non-compliant clauses on each website was calculated, taking into account only the clauses present. The five websites with the best results in simplified monitoring are compliant with 38% of the clauses on average (better 57% and worst 24%), and are not compliant with 62% of the clauses (worst 76% and better 43%) in in-depth monitoring. The five websites with the lowest rankings in simplified monitoring comply with 33% of the clauses on average (better 42% and worst 17%), and are not compliant with 67% of the clauses (worst 83% and better 58%) in in-depth monitoring. Although we are faced with a small sample, with only 10 websites, it can be seen that the best ranked sites in simplified monitoring are also the most compliant with the clauses in in-depth monitoring. On average, the best ranked websites have a compliance rate for the clauses of the European standard 5 percentage points higher than the worst. Comparing the best of the top websites with the best of the worst website, it can be seen that what is in the best batch has an average compliance rate 15 percentage points higher than its worst counterpart.

We also analysed the part of the in-depth sample that corresponds to websites that were not evaluated in the simplified monitoring because they do not have at least 10 pages with more than 100 HTML elements. Although some of these websites have been classified as containing few pages due to problems with the crawling process, it was possible to see through in-depth monitoring that the sites possess good compliance rates. They even have better results than those best classified in simplified monitoring. These 10 websites showed an average compliance rate of 46% for the Standard clauses (better 56% and worse 40%) and a non-compliance with 54% of the clauses (worst 60% and better 44%).

Analysis of the accessibility performance of mobile applications by operating system

In this section we present the results by operating system as well as a comparison between the two versions of the applications of each public authority. The number (and percentage) of applications verifying or violating each clause is displayed in Table 40, as well as those that do not apply to any of the screens evaluated or cannot be determined.

According to the data set out in Table 40, it can be concluded that compliance rates with EN 301 549 clauses in both operating systems are close to 50%. Only four of the clauses assessed achieved 100% compliance in both operating systems. On Android the average compliance rate of the evaluated clauses is 53%, with only five clauses in full compliance (or not present), and in iOS 51% with nine clauses in full compliance (or not present):

Compliant clauses on Android and iOS:

• 11.1.4.5 Images of text
• 11.2.1.2 No keyboard trap
• 11.2.5.2 Pointer cancellation
• 11.3.2.1 on focus

Compliant clauses only on Android:

• 11.2.4.7 Focus visible

Compliant clauses only on iOS:

• 11.1.3.3 Sensory characteristics
• 11.1.4.1 Use of colour
• 11.2.4.6 Headings and labels
• 11.3.3.1 Error identification
• 11.3.3.4 Error prevention (legal, financial, data)

The five clauses with the greatest discrepancy between operating systems are:

• 11.1.4.1 Use of colour (75% Android vs 0% iOS non-compliant)
• 11.2.4.6 Headings and labels (50% Android vs 0% iOS non-compliant)
• 11.2.4.7 Focus visible focus (0% Android vs 50% iOS non-compliant)
• 11.2.1.1 Keyboard (50% Android vs 87.5% iOS non-compliant)
• 11.1.3.1 Info and relationships (62.5% Android vs 100% iOS non-compliant)

The eight clauses with the highest failure rate were:

On Android and iOS

• 11.1.4.4 Resize text (100%)
• 11.1.4.11 Non-text Contrast (87.5% Android, 100% iOS)
• 11.1.3.4 Orientation (87.5% Android, 100% iOS)
• 11.2.4.4 Link purpose (in context) (87.5%)
• 11.4.1.2 Name, role, value (87.5%)
• 11.1.4.3 Contrast (minimum) (60% Android, iOS 100%)

iOS only

• 11.1.3.1 Info and relationships (100%)
• 11.2.1.1 Keyboard (87.5%)

The remaining clauses obtained a failure rate of 75% or less.

Information about Accessibility Statements

On the 281 websites analysed, accessibility declarations were found in only 13, corresponding to 4.6% of the sample. 29 other websites were identified with an accessibility page but which did not correspond to a declaration built in accordance with the model proposed in Decree-Law No 83/2018 of 19 October 2018 transposing Directive (EU) 2016/2102 of the European Parliament and of the Council on the accessibility of websites and mobile applications. It would be enough for the entities to which these websites belong to correct their declarations and we would have 15% websites with Declaration of Accessibility.

The 13 accessibility declarations reported the following levels of compliance:

• 6 present the website as fully compliant;
• 6 present the website as partially compliant;
• 1 displays the website as non-compliant.

The average age (on 31 December 2021) of the identified accessibility declarations is 13.9 months, with the most recent declaration just under 4 months and the oldest approximately 2 years and 5 months. Taking this into account, it is normal that the websites of several of these accessibility statements have already changed since the date of publication of the declaration. Nevertheless, the score given by AccessMonitor to each of the websites was analysed and the following averages were obtained for each level of compliance reported:

• 7.7 for websites claiming to be fully compliant;
• 7.1 for websites claiming to be partially compliant;
• 5.6 for the website claiming to be non-compliant.

Use of the execution procedure and comments from end-users

In Portugal the entity responsible for receiving and collecting notifications from public sector bodies in breach of accessibility requirements is the National Institute for Rehabilitation, I.P. (INR). The INR provides a form to report discrimination situations by forwarding complaints to the competent authorities.

The information provided by the INR on complaints received for the year 2021 was as follows:

• Two complaints related to inaccessibility of websites
• Two complaints related to inaccessibility of mobile applications
• Two of the complaints were submitted by individual persons
• Two of the complaints were lodged by legal persons
• The four complaints report situations related to visual impairment
• None of the cases constituted an administrative offence and the INR did not refer them to the competent authorities for their instruction in accordance with Decree-Law No 34/2007.

In addition to the formal complaint procedure, in which users allege facts likely to constitute an administrative offence under the Non-Discrimination Law (Law No 46/2006) — formally claiming that they are being discriminated against — users can also contact the authorities, reporting difficulties in accessing certain documents or elements of the interface, suggestions for improvement, alerts of inaccessible situations, requesting access to alternative formats (e.g. requesting prints in braille, printouts in extended characters, or other) for specific documents, requesting accessible formats of specific documents that, because they are excepted (e.g. archival documents, with edit dates prior to September 2018), they are in formats that do not comply with European Standard EN 301 549. For this purpose, the Accessibility Statement, which each entity is required to make available on its website or mobile application, has in its section III the contact details of its respective entity.

Content related to additional measures

The following stakeholders were consulted for the monitoring and preparation of this report:

• Commission on Inclusion Policies, through its coordination — the Secretariat of State for the Inclusion of People with Disabilities.
• National Institute for Rehabilitation, I.P., to collect notifications regarding non-compliance with accessibility requirements by public sector bodies.

Annexes

Tables to support the Executive Summary

List of websites' sample to the simplified monitoring