Report on the 2022/24 Monitoring Period

Analysis of Accessibility Requirements for Websites and Mobile Applications part of DL 83/2018

AMA, I.P. / December 2024

Table of contents:

List of Graphs

• Graph 01 - Stratification of the sample of 657 websites selected for the 2nd monitoring period 2022–2024 (by sector)
• Graph 02 - Average non-compliance rate of websites with the applicable clauses of the European Norm EN 301 549
• Graph 03 - Average non-compliance rate of apps by operating system
• Graph 04 - Average non-compliance rate of websites and apps – in-depth monitoring
• Graph 05 - Which uses are most impacted by non-compliant clauses of EN 301 549?
• Graph 06 - Comparison of non-compliance rates by principle between websites in the first and second periods
• Graph 07 - Comparison of non-compliance rates by principle between mobile applications in the first and second periods

List of Tables

• Table 01 - Distribution of websites by type of service
• Table 02 - Descriptive statistics of the page sample
• Table 03 - EN 301 549 requirements tested by AccessMonitor
• Table 04 - EN 301 549 requirements considered in the in-depth methodology for websites
• Table 05 - EN 301 549 requirements considered in the in-depth methodology for mobile applications
• Table 06 - Compliance of websites with the tested EN 301 549 requirements
• Table 07 - Compliance of websites with functional performance statements considering primary relations
• Table 08 - Compliance of web pages with the tested EN 301 549 requirements
• Table 09 - Comparison of website compliance levels with EN 301 549 requirements evaluated in the two monitoring periods
• Table 10 - Compliance of websites with the tested requirements
• Table 11 - Compliance of websites with functional performance statements considering primary relations
• Table 12 - Compliance of websites with requirements tested in the two monitoring periods
• Table 13 - Compliance of mobile applications with the tested requirements
• Table 14 - Compliance of mobile applications by operating system with the tested requirements
• Table 15 - Compliance of mobile applications with functional performance statements considering primary relations
• Table 16 - Compliance of mobile applications with requirements tested in the two monitoring periods
• Table 17 - Average non-compliance rate of websites with the applicable requirements of the European Norm EN 301 549
• Table 18 - Non-compliance rate of apps by operating system
• Table 19 - Non-compliance rate of websites and apps – in-depth monitoring
• Table 20 - Non-compliance rate by Functional Performance Statement for websites and mobile applications
• Table 21 - List of websites included in the sample for the simplified monitoring method
• Table 22 - List of websites included in the sample for the in-depth monitoring method
• Table 23 - List of mobile applications included in the sample for the in-depth monitoring method

Executive Summary

Under DL No. 83/2018, of October 19, which establishes the accessibility requirements for websites and mobile applications, the Agência para a Modernização Administrativa, I.P. (AMA), is responsible for presenting a report to the European Commission every three years on the monitoring results, including measurement data. This document contains the results and measurement data for the 2nd monitoring period, spanning 2022 to 2024.

For the analysis during this monitoring period, a sample of 657 websites and 33 mobile applications was selected, following the methodology proposed by Implementing Decision (EU) 2018/1524. Websites underwent two types of monitoring: simplified monitoring, which mainly involves running an automatic validator through a sample of pages, and in-depth monitoring, involving manual validation by an accessibility expert. Mobile applications were only subject to in-depth monitoring.

When selecting the sample of websites, geographical criteria (central, regional, and local) as well as type of service, demand, and sector of activity were considered. Both the website and mobile application samples were submitted for review to organizations representing people with disabilities through the Inclusion Policy Committee coordinated by the Secretary of State for the Inclusion of Persons with Disabilities.

Note: if you are unable to consult the following chart for any reason, see the data in Table 1.

Websites

In the present analysis, we used a simplified monitoring approach with a sample consisting of the first page and all pages hyperlinked from the first page within the domain, which we designated as H+. Using the H+ methodology to collect pages across all websites, we obtained an average sample of 61 pages per website. For in-depth monitoring, we selected a subset of websites from the simplified monitoring process, totaling 54 out of 657 websites. A total of 424 pages from these 54 websites were manually evaluated.

Note: if you are unable to consult the following chart for any reason, see the data in Table 17.

In Graph 2, the average non-compliance rate for all websites is presented for each principle, covering all the clauses of that principle. It can be observed that the two methodologies yield similar, and even identical, results for three of the principles—namely, Operable, Understandable, and Robust. The only exception is the Perceivable principle, where the difference between the results of the two methodologies is significant—approximately 40 percentage points. For this principle, as well as for the others, the simplified methodology always results in equal or higher non-compliance rates.

The simplified methodology, being based on an automated evaluation tool, assesses a significantly smaller number of requirements - 13 for the simplified methodology vs. 56 for the in-depth methodology. For three principles, the simplified methodology approximates the results of the more reliable in-depth methodology. For the Perceivable principle, the three requirements assessed by the simplified methodology present a more negative picture, with high non-compliance rates. According to the in-depth methodology, those three requirements have a 100% non-compliance rate, even higher than the 86% reported by the simplified methodology.

Mobile Applications

The mobile application sample included 16 applications from various public entities in their iOS and Android versions (32 applications), plus one Android-only application, resulting in a total of 33 mobile applications analyzed. The average non-compliance rate for all requirements was 22% for iOS applications and 24% for Android applications, showing no clear advantage for either platform.

Note: if you are unable to consult the following chart for any reason, see the data in Table 18.

In Graph 3, the average non-compliance rates for all requirements, grouped by principle, are compared. Differences between the two operating systems are minor. iOS applications demonstrated higher compliance with the Perceivable, Operable, and Understandable principles, while Android applications performed better under the Robust principle.

Websites and Apps

Note: if you are unable to consult the following chart for any reason, see the data in Table 19.

In Graph 4, the performance of websites and mobile applications across the four accessibility principles is compared. The same trends affect both websites and mobile applications—the best and worst principles coincide. However, mobile applications consistently perform better than websites, except for the Robust accessibility principle.

In fact, for mobile applications, this is the principle that saw the greatest decline from the 1st to the 2nd monitoring report. In the 2024 report, the Robust principle for both websites and mobile applications shows non-compliance rates between 60% and 70%, with mobile apps surpassing website compliance levels for the first time.

At the root of this non-compliance are issues related to the correct handling of element semantics—buttons that function as links, links presented as buttons, and interactive elements that appear as simple text. These problems leave, for example, screen reader users unaware of how to interact with such elements.

Users and Main Issues

Which users face the most barriers? The European Norm EN 301 549 establishes a link between accessibility requirements and the users who benefit from these practices, referred to as Functional Performance Statements. These statements encompass not only people with disabilities but anyone facing temporary or situational limitations, such as difficulty using a smartphone in bright sunlight.

Note: if you are unable to consult the following chart for any reason, see the data in Table 20.

The data collected (Graph 5) indicates that nearly all websites and mobile applications have non-compliances across almost all Functional Performance Statements. The only exceptions are for the statement “Use with limited reach,” where 52% of websites and 11% of mobile applications are non-compliant, and “Use with limited cognition, language or learning,” where 64% of websites and 72% of mobile applications are non-compliant. For all other statements, non-compliance exceeds 80%, with rates above 90% for five statements.

Comparison with the Previous Monitoring Period

As this report concerns the second monitoring period, comparing its results with those of the first period helps identify trends in the accessibility of public websites and mobile applications in Portugal.

Note: if you are unable to consult the following chart for any reason, see the data by requirement in Table 12.

Graph 6 compares non-compliance rates for websites, grouped by accessibility principle. The values, derived from requirements assessed in both periods, show minor differences from earlier graphs. The accessibility level of websites has not improved over this period. Non-compliance for the Perceivable principle decreased from 52% to 47%. For the Operable principle, it increased from 40% to 42%. For the Understandable principle, it rose from 22% to 24%. For the Robust principle, it dropped from 72% to 64%. Calculating the average non-compliance rates for the four principles reveals a positive evolution, with the non-compliance rate decreasing from 46.5% in the first period to 45% in the second period. However, when comparing the average non-compliance rate for all clauses (without grouping by principle), it is observable that the value remained exactly the same, at 43%.

Note: if you are unable to consult the following chart for any reason, see the data by requirement in Table 16.

Graph 7 compares non-compliance rates for mobile applications, grouped by accessibility principle. As with websites, these values reflect requirements assessed in both periods and may slightly differ from earlier graphs. The accessibility level of mobile applications also showed no improvement. Non-compliance for the Perceivable principle remained at 30%. For the Operable principle, it decreased from 21% to 17%. For the Understandable principle, it increased from 13% to 14%. For the Robust principle, it rose from 44% to 73%. In fact, for mobile applications, this is the principle that saw the greatest decline from the 1st to the 2nd monitoring report. Calculating the average non-compliance rates for the four principles reveals that, for mobile applications, there was a negative evolution, with the non-compliance rate increasing from 27% in the first period to 33.5% in the second period. However, when comparing the average non-compliance rate for all clauses (without grouping by principle), the situation for mobile applications mirrors that of websites, as the average non-compliance rate remained exactly the same, at 25%.

Part I – Description of Monitoring Activities

General Information

The monitoring activities took place from February 2024 to December 2024, according to the following timeline:

• Simplified monitoring of websites
• Sample definition: March 2024
• Sample collection: March 2024
• Sample evaluation: March to August 2024
• In-depth monitoring of websites
• Sample definition: March 2024
• Sample evaluation: October to December 2024
• In-depth monitoring of mobile applications
• Sample definition: April 2024
• Sample evaluation: December 2024

The monitoring is conducted by the Agência para a Modernização Administrativa, I.P. (AMA). For carrying out monitoring activities and drafting this report, AMA enlisted the services of the Faculdade de Ciências da Universidade de Lisboa.

The sample provided by AMA includes 657 websites and 33 mobile applications. The number of websites in the sample exceeds the minimum required by point 2.1 of Annex I of Implementing Decision (EU) 2018/1524, equivalent to two sites per 100,000 inhabitants plus 75 sites. Considering Portugal's resident population in 2023 is 10,639,726 (according to INE statistics), the minimum required is 181 sites annually. For the three-year sample period, the minimum number is 543 sites. The initial sample was analyzed to ensure that included websites were accessible for monitoring activities and corresponded to unique domains (i.e., not redirecting to other domains in the sample). After adjustments, the final sample is presented in Table 21 annex.

Sample Composition

The website sample aims to represent the diverse services provided by public sector entities. The distribution of websites by service type, shown in Table 1, illustrates the variety of services considered in the sample.

Website Sample for Simplified Monitoring

All 657 websites were initially considered for the simplified monitoring method (see Table 21). This sample consists of each website's homepage and all pages linked to the homepage within the domain.

Table 2 provides descriptive statistics of the page sample. The analysis shows that, on average, sampled websites have 61 pages linked from their homepage.

Website Sample for In-Depth Monitoring

The final sample of websites for in-depth monitoring is presented in Table 22 in the annex. Websites for in-depth monitoring are a subset of websites used for simplified monitoring.

Mobile Application Sample for In-Depth Monitoring

The mobile applications included in the sample were selected by AMA. iOS and Android versions of the same entity's application were considered individually, analyzing both versions of 16 applications and one Android-only application, totaling 33 mobile applications.

The final mobile application sample for in-depth monitoring is shown in Table 23 in the annex.

Correlation with Standards, Technical Specifications, and Tools Used for Monitoring

Methodology for Simplified Monitoring of Websites

Pages from the 657 website sample were obtained through web crawling. The QualWeb Crawler was the primary tool used, capable of processing JavaScript-enabled pages.

Accessibility evaluation of each sampled page was conducted using the AccessMonitor automated accessibility evaluation tool. This tool validated compliance with 37 tests based on sufficient and recommended techniques from the Web Content Accessibility Guidelines (WCAG).

The requirements of EN 301 549 tested by AccessMonitor are listed in Table 3, along with corresponding WCAG success criteria.

Methodology for In-Depth Monitoring of Websites

For each sampled website, a representative set of pages was defined for evaluation, as specified in the Commission Implementing Decision (EU) 2018/1524. This representative sample includes:

• Homepage;
• Authentication page;
• Sitemap;
• Contact page;
• Help page;
• Legal notice;
• At least one page per service type provided by the website;
• Other primary uses, including search functionality;
• Accessibility statement;
• Feedback pages related to the statement;
• Pages with substantially distinct appearance or different content;
• Any relevant pages;
• All pages involved in a process.

Each page in the representative sample was evaluated against WCAG 2.1 AA success criteria. For each page, one of the following results was provided:

• Conforming – The page meets all tests for the corresponding success criterion.
• Violation – The page fails one or more tests for the corresponding success criterion.

Evaluation was conducted using the Accessibility Insights tool, which covers all success criteria of WCAG 2.2, exceeding WCAG 2.1 and EN 301 549 requirements.

Additional support was provided by the Color Contrast Analyzer (CCA) tool.

Table 4 lists the EN 301 549 requirements and corresponding WCAG success criteria considered in this evaluation.

Methodology for In-Depth Monitoring of Mobile Applications

Both iOS and Android versions of the same application were considered individually. Applications were installed directly from their respective app stores, using standard user accounts for evaluation, except for one case where an account could not be obtained. Screens selected for evaluation followed the methodology defined in points 2 and 3 of Annex I of the Commission Implementing Decision (EU) 2018/1524. This representative sample includes:

• Home screen (first screen of each application);
• Login screen;
• Sitemap;
• Contact screen;
• Help screen;
• Legal notice;
• At least one screen per service type;
• Other primary uses, including search functionality;
• Accessibility statement;
• Feedback screens related to the statement;
• Screens with substantially distinct appearance or different content;
• Relevant downloadable documents for each service type;
• Any other pertinent screen;
• If selected screens are part of a process, include all process screens.

Manual evaluation of mobile applications was supported by the following tools and devices:

• Talkback and VoiceOver screen readers for Android and iOS, respectively;
• Bluetooth keyboard;
• Accessibility Scanner;
• Accessibility Inspector;
• Colour Contrast Analyser.

Testing methodology for manual evaluations was based on the Trusted Tester methodology. While the Trusted Tester is based on WCAG 2.0, additional tests were added to cover WCAG 2.1 AA success criteria as required by EN 301 549. Added tests were based on sufficient techniques from WCAG 2.1.

The success criteria considered in this evaluation correspond to WCAG 2.1 AA compliance as outlined in EN 301 549, detailed in Table 5.

A report for each application was recorded for future analysis. After evaluations, the results were analyzed considering the verified and violated requirements.

Part II – Monitoring Results

Simplified Monitoring of Websites

In total, 657 websites were analyzed, encompassing 40,215 pages, with an average of 61 pages per website.

Distribution of EN 301 549 Requirements by Website in the Simplified Monitoring

Table 6 shows the number (and percentage) of websites violating each tested requirement.

From Table 6, we conclude that compliance with the tested requirements of EN 301 549 was low. The requirements with the highest non-compliance rates were:

• 9.1.3.1 Info and Relationships with 98% of websites non-compliant;
• 9.4.1.2 Name, Role, Value with 93% of websites non-compliant;
• 9.2.4.4 Link Purpose (in Context) with 91% of websites non-compliant;
• 9.2.4.1 Bypass Blocks with 91% of websites non-compliant;
• 9.1.4.3 Contrast (Minimum) with 88% of websites non-compliant;
• 9.1.1.1 Non-Text Content with 73% of websites non-compliant;
• 9.4.1.1 Parsing with 57% of websites non-compliant.

Distribution of EN 301 549 Functional Performance Statements by Website in the Simplified Monitoring

The performance of websites concerning the functional performance statements was analyzed, focusing on requirements that support these statements (primary relationships). Based on the evaluated requirements, 7 of the 11 functional performance statements were considered.

Table 7 presents the results.

From Table 7, we conclude that non-compliance with functional performance statements is significant when assessed based on the primary relationships. The functional performance statement with the lowest non-compliance rate was:

• Limited Reach (LR) with 26% non-compliance among applicable websites.

Distribution of EN 301 549 Requirements by Web Page

Table 8 shows the number (and percentage) of web pages violating each tested requirement, along with the average violations per page.

From Table 8, we conclude that compliance with the tested requirements of EN 301 549 was also low at the page level. The requirements with the highest non-compliance rates were:

• 9.1.3.1 Info and Relationships with 85% of pages non-compliant;
• 9.1.4.3 Contrast (Minimum) with 80% of pages non-compliant;
• 9.2.4.4 Link Purpose (in Context) with 65% of pages non-compliant;
• 9.4.1.2 Name, Role, Value with 59% of pages non-compliant;
• 9.2.4.1 Bypass Blocks with 55% of pages non-compliant.

Analysis of Simplified Website Monitoring Results

The simplified monitoring method identified the most frequent non-compliances with the EN 301 549 requirements detectable by the automated tool used. Key findings include:

• The two most common issues, present in at least 80% of tested pages, are:
• Non-compliance with the "9.1.3.1 Info and Relationships" requirement by 85% of pages (and 98% of websites), indicating various issues that prevent assistive technology users from correctly perceiving content and structure. Examples include inaccessible headers, table cells without headers, improper ARIA attributes, and unlabeled form fields.
• Websites not adhering to minimum contrast ratios, affecting text readability. This issue was present on over 88% of websites and 80% of web pages.
• A second set of three issues affects over 90% of websites, though with lower prevalence at the page level:
• The "9.2.4.1 Bypass Blocks" requirement, violated by 91% of websites and 55% of pages, causes difficulty for keyboard users who lack explicit mechanisms to skip repetitive menus and access main content directly.
• Content creators fail to provide descriptive links, with issues in 65% of pages (and 91% of websites). Examples include repeated link texts like "Read More" or "Click Here," complicating navigation for users relying on link context.
• Problems communicating accessible names or roles occur on 93% of websites and 59% of pages, highlighting a lack of consideration for assistive technology needs, leading to inadequate page interpretation.
• Lastly, the absence of alternative text descriptions for non-text content was observed on 73% of websites, though only 36% of pages. This impedes users who cannot view images from fully utilizing websites.

Comparison of Simplified Website Monitoring Results with Previous Period

Table 9 compares the percentage of websites violating the success criteria in both monitoring exercises.

From Table 9, we note a negative trend from the first to the second monitoring period. Violations increased for two requirements: "9.1.3.1 Info and Relationships" and "9.2.1.1 Keyboard." While the increase for the first may result from enhanced testing capabilities, the second reflects a lack of concern for keyboard accessibility.

On a positive note, compliance with requirements under the "Robust" WCAG principle, particularly "9.4.1.1 Parsing," improved. This may reflect the adoption of tools for syntactic validation. However, it's worth noting that modern browsers often correct such errors automatically, and this success criterion has been removed in the latest WCAG version.

In-Depth Monitoring of Websites

This section describes the results for 54 websites. A total of 424 pages were analyzed, averaging 8 pages per website.

Distribution of EN 301 549 Requirements by Website in the In-Depth Monitoring

Table 10 presents the number (and percentage) of websites violating each EN 301 549 requirement.

From Table 10, compliance was low, with the following requirements having over 75% of websites non-compliant:

• 9.1.1.1 Non-Text Content with a 100% failure rate;
• 9.1.3.1 Info and Relationships with a 100% failure rate;
• 9.1.3.2 Meaningful Sequence with an 81% failure rate;
• 9.1.4.3 Contrast (Minimum) with a 100% failure rate;
• 9.2.1.1 Keyboard with a 96% failure rate;
• 9.2.4.4 Link Purpose (in Context) with a 96% failure rate;
• 9.2.4.6 Headings and Labels with a 94% failure rate;
• 9.2.4.7 Focus Visible with an 83% failure rate;
• 9.2.5.3 Label in Name with a 78% failure rate;
• 9.4.1.1 Parsing with an 81% failure rate;
• 9.4.1.2 Name, Role, Value with a 100% failure rate.

On the positive side, the following requirements had less than 10% of websites non-compliant:

• 9.1.2.4 Captions (Live) with all websites compliant;
• 9.1.3.3 Sensory Characteristics with a 2% failure rate;
• 9.1.3.4 Orientation with a 6% failure rate;
• 9.1.4.2 Audio Control with a 2% failure rate;
• 9.2.1.2 No Keyboard Trap with a 9% failure rate;
• 9.2.1.4 Character Key Shortcuts with all websites compliant;
• 9.2.2.1 Timing Adjustable with a 6% failure rate;
• 9.2.3.1 Three Flashes or Below Threshold with all websites compliant;
• 9.2.4.5 Multiple Ways with a 7% failure rate;
• 9.2.5.2 Pointer Cancellation with a 4% failure rate;
• 9.2.5.4 Motion Actuation with all websites compliant;
• 9.3.2.1 On Focus with a 7% failure rate;
• 9.3.2.2 On Input with a 6% failure rate;
• 9.3.2.3 Consistent Navigation with a 7% failure rate;
• 9.3.2.4 Consistent Identification with a 2% failure rate;
• 9.3.3.4 Error Prevention (Legal, Financial, Data) with a 2% failure rate.

Distribution of EN 301 549 Functional Performance Statements in the In-Depth Monitoring

This analysis considered all functional performance statements through their primary relationships. Table 11 presents the results.

From Table 11, we conclude that non-compliance with functional performance statements is high, even when assessed based solely on primary relationships. The only three statements not violated by all websites were:

• Usage with Limited Hearing with 28% of websites compliant;
• Usage with Limited Reach with 19% of websites compliant;
• Minimize photosensitive seizure triggers with 89% of websites compliant.

Analysis of Results from the In-Depth Website Monitoring

The in-depth monitoring method allowed the identification of the most frequent non-compliances with the requirements of EN 301 549 applicable to a manual analysis of websites. The following non-compliances stand out:

• The absence of alternative descriptions for non-text content, mostly images, was found on all analyzed websites. This poor practice prevents users who cannot see the images from fully using the website.
• Non-compliance with the requirement “9.1.3.1 Info and Relationships” in all analyzed websites reveals several types of issues that prevent users of assistive technologies from correctly perceiving the content and structure of the page. Examples of problems associated with this requirement include headings without accessible names, form fields without labels, or lists and tables without proper identification.
• Non-compliance with the requirement “9.1.4.3 Contrast (minimum)” represents a significant issue for the digital accessibility of the analyzed websites. This requirement ensures that text has sufficient contrast with the background, allowing reading by people with visual impairments, including low vision or color blindness. Failing to comply with this requirement can make accessing information and online services difficult.
• Non-compliance with the requirement “9.2.1.1 Keyboard” is another recurring problem on the analyzed websites. This requirement demands that all interactive elements on a page be accessible and usable exclusively via the keyboard, without relying on pointing devices such as a mouse. Violating this requirement can exclude users with reduced mobility or other conditions that prevent them from using a mouse, compromising equity in access to digital services.
• Content creators continue to fail to create hyperlinks with descriptions that allow users to understand the purpose of the link. Pages with hyperlinks without a comprehensible description outside of context are problematic for users who navigate pages through lists of links—this is the case for many blind users who rely on screen readers.
• Non-compliance with the requirement “9.2.4.6 Headings and Labels” is also a relevant issue in the analyzed websites, directly affecting the navigation and understanding of page structure for users with cognitive or visual impairments. This requirement states that headings and labels must be used clearly and consistently, facilitating identification and navigation between different sections of content. Violating this requirement can result in disorganized pages, making it difficult to locate information and interact with content, especially on complex platforms.
• Non-compliance with the requirement “9.2.4.7 Focus Visible” is a common failure on the analyzed websites, especially affecting users of keyboards and assistive technologies. This requirement demands that the focus, when navigating with the keyboard or other forms of interaction, be clearly visible, allowing users to easily identify where they are on the page and which elements they can interact with. The lack of a visible focus can cause confusion and frustration, especially for people with visual or motor impairments who rely on this indicator to navigate correctly.
• Two of the requirements associated with the WCAG principle “Robust” are among the most non-compliant requirements. This reflects the current state of the web—content that is not built with assistive technologies in mind. This situation prevents these technologies from communicating correctly to their users the content of the pages they wish to consult. Problems related to the communication of accessible names or roles (semantics) of elements occur on all analyzed websites. The same goes for issues related to the proper use of HTML elements—81% of the websites present HTML syntax errors.

Comparison of Results from In-Depth Website Monitoring with Results from the Previous Period

In Table 12, the comparison of the percentage of websites violating the success criteria evaluated in both monitoring exercises is presented.

From the analysis of the results presented in Table 12, we can conclude that the evolution from the first to the second monitoring period shows both progress and setbacks.

Regarding aspects related to the visual design of web pages, an increase in non-compliance with the following requirements was observed: Use of Color (9.1.4.1), Contrast (minimum) (9.1.4.3), Resize Text (9.1.4.4), and Focus Visible (9.2.4.7). This increase can hinder the experience of users, especially those with visual impairments. The lack of proper contrast and inappropriate use of colors can make content unreadable for people with color blindness or low vision, compromising the site's accessibility. Additionally, the absence of text resizing can affect readability for users with vision difficulties. On the other hand, the decrease in non-compliance with Images of Text (9.1.4.5), Reflow (9.1.4.10), and Non-text Contrast (9.1.4.11) suggests an improvement in adapting content to the needs of users, especially on mobile devices and for those requiring more contrasting elements to better understand content structure.

Regarding keyboard interaction-related requirements, an increase in non-compliance was observed in the following cases: Content on Hover or Focus (9.1.4.13), Keyboard (9.2.1.1), Focus Visible (9.2.4.7), and Labels or Instructions (9.3.3.2). This increase can significantly harm the experience of users who rely solely on the keyboard to navigate, such as those with reduced mobility or visual impairments. The lack of visible focus, for example, makes efficient navigation difficult because users cannot identify where the focus is on the page, causing confusion and frustration. The increase in non-compliance with the requirement “9.2.1.1 Keyboard” suggests that many websites still do not allow full interaction via the keyboard, making them inaccessible to those unable to use a mouse. Additionally, the lack of clear labels and instructions (9.3.3.2) can make filling out forms or understanding functionalities more difficult, directly impacting the user experience. On the other hand, the decrease in non-compliance with the requirements for Identify Input Purpose (9.1.3.5) and Timing Adjustable (9.2.2.1) indicates an improvement in accessibility in terms of usability and time control, benefiting users who need more time or clear instructions to interact with the site's elements. The decrease in non-compliance with the requirement Multiple Ways (9.2.4.5) is also a positive sign, as it indicates that more sites are offering various alternatives for navigation and content searching, facilitating accessibility for users with different needs and preferences.

The analysis of the requirements related to page semantics reveals a mixed trend. The increase in non-compliance with the criteria Headings and Labels (9.2.4.6), Language of Parts (9.3.1.2), and Labels or Instructions (9.3.3.2) is concerning, as it suggests that more sites are not ensuring a clear and intuitive structure for users, which can make navigation and understanding content more difficult. The lack of proper headings and labels can make the navigation experience confusing, especially for screen reader users or those with cognitive impairments. On the other hand, the decrease in non-compliance with the criteria Identify Input Purpose (9.1.3.5), Page Titled (9.2.4.2), Language of Page (9.3.1.1), and Consistent Identification (9.3.2.4) is a positive development, as it indicates that websites are improving how they present content and ensure more predictable and understandable navigation, benefiting accessibility, particularly for users with cognitive and language impairments.

In-Depth Monitoring of Mobile Applications

A total of 33 applications were analyzed, consisting of 17 Android applications and 16 iOS applications.

Results by EN 301 549 Requirement of Mobile Applications

In Table 13, the number (and percentage) of applications violating each requirement is presented for all the applications, i.e., from both operating systems.

By analyzing the data presented in Table 13, it is possible to determine the requirements most frequently violated:

• 11.1.3.4 Orientation - 97% of applications
• 11.1.4.3 Contrast (minimum) - 91% of applications
• 11.4.1.2 Name, function, value - 90% of applications
• 11.2.1.1 Keyboard - 88% of applications
• 11.1.3.1 Info and relationships - 84% of applications

Accessibility of Mobile Applications by Operating System

This section presents the results organized by operating system. In Table 14, the percentage of applications violating each requirement per operating system is shown.

According to the data presented in Table 14, it can be observed that both operating systems have similar non-compliance levels (24% for Android and 22% for iOS). However, it is important to highlight that some requirements show significant variations in their non-compliance levels between the two operating systems. These are:

• 11.1.3.1 Info and relationships - 94% of Android applications and 73% of iOS applications
• 11.1.4.4 Resize text - 50% of Android applications and 81% of iOS applications
• 11.2.4.7 Focus visible - 59% of Android applications and 7% of iOS applications

The differences observed in compliance with requirements 11.1.3.1 and 11.2.4.7 may be attributed to several factors. First, Apple’s design guidelines, such as the Human Interface Guidelines, more explicitly emphasize accessibility, including visible focus indicators and semantic structures, compared to Android's Material Design, which may allow more flexibility and inconsistency. Second, Apple's native tools and frameworks, like Xcode and UIKit, offer more integrated and easier-to-apply accessibility features, while Android developers may need to put in more effort to achieve similar results. Lastly, in multi-platform frameworks like Flutter or React Native, default configurations often favor iOS requirements, requiring additional customizations to ensure equivalent accessibility on Android.

On the other hand, development guidelines can also benefit the Android platform, as seen in the non-compliance results for requirement 11.1.4.4. Android promotes the use of scalable units like “sp” in Material Design guidelines, encouraging responsive design practices, whereas iOS developers may neglect dynamic text support due to less emphasis in the Human Interface Guidelines. Furthermore, frameworks like Jetpack Compose in Android provide native support for text scaling, making its application easier, while in iOS, developers need to manually configure Dynamic Type and ensure proper testing.

Distribution of Functional Performance Statements

Based on the results of the mobile application monitoring, support for functional performance statements was also analyzed, considering primary relationships. Table 15 presents the results obtained from the first analysis, where only the requirements of primary relationships were considered.

By analyzing Table 15, we can conclude that there is a high rate of non-compliance with the functional performance statements. Three of the functional performance statements are non-compliant in all of the mobile applications analyzed. Of the remaining ones, three statements are non-compliant in more than 85% of the applications, and the other two statements are non-compliant in more than 50% of the applications.

Analysis of the Results from In-Depth Mobile Application Monitoring

The in-depth monitoring method allowed for the identification of the most frequent non-compliances with the EN 301 549 requirements applicable to a manual analysis of mobile applications. Some notable non-compliances detected include:

• Applications regularly force the device into one orientation, limiting access for users with motor disabilities who benefit from other orientations.
• A large number of evaluated applications revealed issues with compliance with contrast levels (11.1.4.3 Contrast (minimum) and 11.1.4.11 Non-textual contrast). This means that many of these applications may not provide adequate support for users with visual disabilities, as the contrast between text and background or other elements may not meet the necessary standards for readability and accessibility.
• Content creators continue to fail to create interactive components with descriptions that allow users to understand their purpose. In particular, many interactive elements were used without a concrete definition of their function. For example, buttons were not announced as such, and texts were presented as interactive, which led users to test virtually every element to discover which ones were actually interactive.
• Most applications do not comply with the requirement “11.2.1.1 Keyboard,” meaning that many features are not fully accessible for people who rely on keyboards as their primary input device, making navigation and interaction difficult.
• Non-compliance with the requirement “11.1.3.1 Info and relationships” in several applications analyzed reveals various types of issues that prevent users of assistive technologies from having an accurate perception of the content and structure of the screen. Examples of problems associated with this requirement include headers without accessible names, form fields without labels, or lists and tables without proper identification.

Comparison of In-Depth Mobile Application Monitoring Results with the Previous Period's Results

Table 16 presents a comparison of the percentage of mobile applications violating the success criteria evaluated in both monitoring periods.

By analyzing the results presented in Table 16, we can conclude that the evolution from the first to the second monitoring period shows few changes in most requirements. However, there are notable changes in some requirements, both positive and negative. The requirements that evolved positively during this period are:

• 11.1.4.4 Resize text saw a decrease of 34% in non-compliant applications;
• 11.2.4.4 Link purpose (in context) saw a decrease of 72% in non-compliant applications;
• 11.2.5.1 Pointer gestures saw a decrease of 41% in non-compliant applications.

The requirements that evolved negatively during this period are:

• 11.1.3.2 Meaningful sequence saw an increase of 26% in non-compliant applications;
• 11.2.5.3 Label in name saw an increase of 40% in non-compliant applications;
• 11.4.1.3 Status messages saw an increase of 55% in non-compliant applications.

The positive aspects seem to focus on flexibility offered to users. In particular, the ability to resize text and not rely solely on pointer gestures. It is also positive to note an increased concern for the application's semantics, with a significant decrease in the number of applications non-compliant with requirement 11.2.4.4. However, a contradictory indication must be mentioned, observed in the increase of applications non-compliant with requirement 11.4.1.3. This requirement also negatively impacts assistive technology users and exemplifies what seems to be a negative trend of applications becoming less compatible with assistive technologies. Besides requirement 11.4.1.3, which reflects many cases where error messages are not announced by screen readers, issues are also evident in requirements 11.2.5.3 (where labels for various fields are presented visually but not available for assistive technologies) and 11.1.3.2 (where the sequence in which content is presented by assistive technologies does not match the sequence in which it is presented visually). This scenario appears to represent a reality in which mobile app developers do not know how to properly make the content of these applications available to the assistive technologies their users rely on.

Annexes

Tables to support the Executive Summary

List of websites' sample for the simplified monitoring

List of websites' sample for the in-depth monitoring

List of mobile apps' sample for the in-depth monitoring