Report on the 2020-2021 monitoring period

Index

List of tables

  1. Distribution of websites by level of administration
  2. Distribution of websites by type of service
  3. Descriptive statistics of the sample of pages
  4. Clauses of EN 301 549 tested by AccessMonitor
  5. Clauses of EN 301 549 tested by QualWeb’s ACT rules
  6. Clauses of the EN 301 549 standard considered in the in-depth methodology
  7. Demographic information of test participants for website
  8. Clauses of EN 301 549 considered in the in-depth methodology for mobile applications
  9. Demographic information of test participants with mobile application users
  10. Distribution of the number of pages per website for the Home+ sample
  11. Compliance of websites with the clauses of EN 301 549 tested for the sample Home+
  12. Compliance of websites with functional performance declarations considering primary relationships for the Home+ sample
  13. Compliance of websites with functional performance statements considering primary and secondary relationships for the Home+ sample
  14. Number of websites where each AccessMonitor test is true in the Home+ sample
  15. Number of websites that pass, fail or need manual validation for each QualWeb ACT rule in the Home+ sample
  16. Number of pages where each AccessMonitor test is true in the Home+ sample
  17. Number of pages that pass, fail or need manual validation for each QualWeb ACT rule in the Home+ sample
  18. Distribution of AccessMonitor scores for websites in the Home+ sample
  19. Distribution of AccessMonitor scores for pages in the Home+ sample
  20. Distribution of the number of pages per website for the sample 2K
  21. Compliance of websites with the tested clauses for the 2K sample
  22. Compliance of websites with functional performance statements considering primary relationships for the 2K sample
  23. Compliance of websites with functional performance statements considering primary and secondary relationships for the 2K sample
  24. Number of websites where each AccessMonitor test is true in the 2K sample
  25. Number of websites that pass, fail or need manual validation for each QualWeb ACT rule in sample 2K
  26. Number of pages where each AccessMonitor test is true in the 2K sample
  27. Number of pages that pass, fail or need manual validation for each QualWeb ACT rule in the 2K sample
  28. Distribution of AccessMonitor scores for the websites in the 2K sample
  29. Distribution of AccessMonitor scores for the pages in the 2K sample
  30. Compliance of websites with tested clauses
  31. Compliance of websites with functional performance statements considering primary relations
  32. Average compliance rate with the clauses of each functional performance statement considering primary relations
  33. Compliance of websites with functional performance statements considering primary and secondary relations
  34. Average compliance rate with the clauses of each functional performance statement considering primary and secondary relations
  35. Compliance of mobile applications with tested clauses
  36. Compliance with the tested EN clauses of mobile applications per operating system
  37. List of sampled websites for the simplified monitoring method
  38. List of sampled websites for the in-depth monitoring method
  39. List of sampled mobile applications for the in-depth monitoring method

List of figures

  1. Page distribution bar chart for the Home+ sample.
  2. Score distribution bar chart of the websites in the Home+ sample.
  3. Score distribution bar chart of the pages in the Home+ sample.
  4. 2K sample’s page distribution bar chart.
  5. Score distribution bar chart of sample 2K websites.
  6. Bar chart of the score distribution for the sample of homepage plus linked pages.
  7. Conformity of websites with tested clauses.
  8. Average rate of compliance with the clauses of each functional performance statement considering primary relationships.
  9. Average rate of compliance with the clauses of each functional performance statement considering primary and secondary relationships.

Executive summary

This document presents the results of the simplified and in-depth monitoring of websites and mobile applications of Portuguese public sector bodies. Monitoring activities considered 281 websites (24 for in-depth monitoring) and 16 mobile applications. Monitoring activities were complemented by tests with persons with disabilities. The results show that there are multiple accessibility problems, without any website or mobile application analysed in the in-depth monitoring being fully compliant with the clauses set out in European standard EN 301 549.

Description of the monitoring activities

General information

The monitoring activities took place in the period from October 2021 to March 2022, according to the following distribution:

Monitoring is the responsibility of the Agência para a Modernização Administrativa, I.P. (AMA). To carry out the monitoring activities and prepare this report, AMA has contracted the services of FCIÊNCIAS.ID — Associação para a Investigação e Desenvolvimento de Ciências.

The sample of websites was provided by AMA, comprising a total of 281 websites. The number of websites in the sample is the equivalent of the minimum size set out in point 2.1 of Annex I of the Implementing Decision (EU) 2018/1524, with a value equivalent to two websites per 100,000 inhabitants plus 75 sites. With the population residing in Portugal in 2020 of 10.298,252 inhabitants (according to the INE statistics), the minimum number is 281 sites. The initial sample was analysed to ensure that the sites included were accessible to the monitoring activities and that they actually corresponded to different domains (i.e., that they did not redirect to other domains in the sample). After some adjustments, the final sample is that presented in Annex in table 37.

Sample composition

The sample consists of a total of 281 websites and 16 mobile applications. The sample includes websites of different levels of administration as can be seen in table 1.

Table 1: Distribution of websites by level of administration
Levels of administration Number of sites Percentage of sites
State websites 75 27%
Regional Websites 20 7%
Local Websites 66 23%
Other websites of bodies governed by public law 80 28%
Other websites 40 14%

It was also tried to ensure that websites represent the different services provided by public sector bodies. The distribution of websites by type of service, presented in table 2, illustrates the diversity of services that were considered in the sample.

Table 2: Distribution of websites by type of service
Type of Service Number of sites Percentage of sites
Central Administration 31 11%
Basic and Secondary Education 20 7%
Higher Education 20 7%
Hospitals 20 7%
Neighborhood Boards 30 11%
Municipalities 36 13%
Museums 20 7%
Non-Governmental Organisations 20 7%
Sovereign Bodies and Independent Entities 19 7%
Most popular portals and services 25 9%
Autonomous Region of Madeira 12 4%
Autonomous Region of the Azores 8 3%
State Public Business Sector 20 7%

Sample of websites for the simplified monitoring method

All 281 websites were initially considered for the simplified monitoring method. For this monitoring period it was decided to build two samples of pages for simplified monitoring:

The table 3 presents descriptive statistics of the sample of pages. Analysing the sample it is possible to notice that, on average, the sampled sites have 79 pages linked from the homepage and that, on average, each site has more than a thousand pages. This number will in fact be higher since we have limited the collection to two thousand pages per site and 63 sites have reached this limit. The table 3 also presents statistics associated with the size of the pages. For each page the number of HTML elements was counted and, on the basis of this number, sites with at least 10 pages with at least 100 HTML elements were identified. As a result of this analysis, it was observed that, in the Home+ sample, 15% of the websites did not reach this limit. With the 2K sample, the percentage of websites not reaching this limit decreases to 9%.

Table 3: Descriptive statistics of the sample of pages
Sample Home+ 2K
Websites 281 281
Total pages 22,489 281,706
Pages by website 80 1,003
Pages with 100 or more HTML elements 20,054 (89%) 223,398 (79%)
Websites with 10 or more pages with 100 or more HTML elements 239 (85%) 255 (91%)

Sample of websites for the in-depth monitoring method

The sample of websites took into account the results of the simplified monitoring and was composed taking into account the following criteria:

The final sample of websites for in-depth monitoring is presented in Annex in the table 38.

Sample of mobile applications for in-depth monitoring method

The sampled mobile applications were indicated by AMA, having selected the top applications from the list of government applications available at https://www.app.gov.pt. The iOS and Android applications of the same entity were considered individually in this analysis, and the two versions of 8 applications were analysed, totaling 16 mobile applications.

The final sample of mobile applications for in-depth monitoring is presented in Annex in table 39.

Correlation with the standards, technical specifications and tools used for monitoring

Methodology applied in simplified website monitoring

The sample pages of the 281 websites were obtained through web crawling. In order to ensure the best possible coverage, two tools have been used for this purpose:

For the evaluation of the accessibility of each page of the sample, two automated accessibility assessment tools were considered:

The clauses of EN 301 549 tested by AccessMonitor are presented in table 4, together with the corresponding WCAG success criteria.

Table 4: Clauses of EN 301 549 tested by AccessMonitor
EN Clause Clause description WCAG SC SC description
9.1.1.1 Non-text content 1.1.1 Non-text content
9.1.2.1 Audio-only and video-only (pre-recorded) 1.2.1 Audio-only and video-only (pre-recorded)
9.1.3.1 Info and relationships 1.3.1 Info and relationships
9.1.3.2 Meaningful sequence 1.3.2 Meaningful sequence
9.1.4.3 Contrast (minimum) 1.4.3 Contrast (minimum)
9.1.4.4 Resise text 1.4.4 Resise text
9.1.4.5 Images of text 1.4.5 Images of text
9.2.1.1 Keyboard 2.1.1 Keyboard
9.2.2.1 Timing adjustable 2.2.1 Timing adjustable
9.2.2.2 Pause, stop, hide 2.2.2 Pause, stop, hide
9.2.4.1 Bypass blocks 2.4.1 Bypass Blocks
9.2.4.2 Page titled 2.4.2 Page titled
9.2.4.4 Link purpose (in context) 2.4.4 Link purpose (in context)
9.2.4.5 Multiple ways 2.4.5 Multiple ways
9.2.4.6 Headings and labels 2.4.6 Heading and labels
9.2.4.7 Focus visible 2.4.7 Focus visible
9.3.1.1 Language of page 3.1.1 Language of page
9.3.2.1 On focus 3.2.1 On focus
9.3.2.2 On input 3.2.2 On input
9.3.3.2 Labels or instructions 3.3.2 Labels or instructions
9.4.1.1 Parsing 4.1.1 Parsing
9.4.1.2 Name, role, value 4.1.2 Name, role, value

The table 5 presents the clauses of EN 301 549 as well as the corresponding WCAG success criteria, tested by QualWeb through ACT rules.

Table 5: Clauses of EN 301 549 tested by QualWeb’s ACT rules
EN Clause Clause description WCAG SC SC description
9.1.1.1 Non-text content 1.1.1 Non-text content
9.1.2.1 Audio-only and video-only (pre-recorded) 1.2.1 Audio-only and video-only (pre-recorded)
9.1.2.2 Captions (pre-recorded) 1.2.2 Captions (pre-recorded)
9.1.2.3 Audio description or media alternative (pre-recorded) 1.2.3 Audio description or media alternative (pre-recorded)
9.1.2.5 Audio description (pre-recorded) 1.2.5 Audio description (pre-recorded)
9.1.3.1 Info and relationships 1.3.1 Info and relationships
9.1.3.4 Orientation 1.3.4 Orientation
9.1.3.5 Identify input purpose 1.3.5 Identify input purpose
9.1.4.2 Audio control 1.4.2 Audio control
9.1.4.3 Contrast (minimum) 1.4.3 Contrast (minimum)
9.1.4.4 Resise text 1.4.4 Resise text
9.1.4.12 Text spacing 1.4.12 Text spacing
9.2.1.1 Keyboard 2.1.1 Keyboard
9.2.2.1 Timing adjustable 2.2.1 Timing adjustable
9.2.4.1 Bypass blocks 2.4.1 Bypass Blocks
9.2.4.2 Page titled 2.4.2 Page titled
9.2.4.4 Link purpose (in context) 2.4.4 Link purpose (in context)
9.2.4.7 Focus visible 2.4.7 Focus visible
9.2.5.3 Label in name 2.5.3 Label in name
9.3.1.1 Language of page 3.1.1 Language of page
9.3.1.2 Language of parts 3.1.2 Language of parts
9.3.3.1 Error identification 3.3.1 Error identification
9.4.1.1 Parsing 4.1.1 Parsing
9.4.1.2 Name, role, value 4.1.2 Name, role, value

The evaluation process consisted of the collection of QualWeb evaluations followed by the calculation of AccessMonitor results from QualWeb results, thus ensuring that the results presented by the two tools relate to the same source code.

During the evaluation process there were several occurrences of pages that did not respond to the evaluating tool’s requests. Multiple attempts were made to obtain a response, separated by at least two days. Still, it was not possible to evaluate a total of 27,022 pages, which corresponds to 9.6% of the pages in the sample. It should be noted that, as a result of the problems with the evaluations, it was not possible to evaluate any page of three sites in the Home+ sample, as well as no page of two websites in the sample 2K.

Methodology applied for in-depth monitoring of websites

The Trusted Tester methodology was applied. This methodology is based on the Web Content Accessibility Guidelines (WCAG) 2.0. Considering that this analysis follows the EN 301 549 standard, based on WCAG 2.1, new tests have been added to the tests originally included in the Trusted Tester methodology, comprising the WCAG Success Criteria not covered. The added tests were based on sufficient techniques provided in WCAG 2.1 and are presented in Annex. Three auxiliary tools were used to support the tests performed:

The table 6 presents the clauses of the EN 301 549 that were considered in this evaluation, together with the corresponding WCAG success criteria.

Table 6: Clauses of the EN 301 549 standard considered in the in-depth methodology
EN Clause Clause description WCAG SC SC description
9.1.1.1 Non-text Content 1.1.1 Non-text Content
9.1.2.1 Audio-only and Video-only (Pre-recorded) 1.2.1 Audio-only and Video-only (Pre-recorded)
9.1.2.2 Captions (Pre-recorded) 1.2.2 Captions (Pre-recorded)
9.1.2.3 Audio Description or Media Alternative (Pre-recorded) 1.2.3 Audio Description or Media Alternative (Pre-recorded)
9.1.2.5 Audio Description (Pre-recorded) 1.2.5 Audio Description (Pre-recorded)
9.1.3.1 Info and Relationships 1.3.1 Info and Relationships
9.1.3.2 Meaningful Sequence 1.3.2 Meaningful Sequence
9.1.3.3 Sensory Characteristics 1.3.3 Sensory Characteristics
9.1.3.4 Orientation 1.3.4 Orientation
9.1.3.5 Identify Input Purpose 1.3.5 Identify Input Purpose
9.1.4.1 Use of Colour 1.4.1 Use of Colour
9.1.4.2 Audio Control 1.4.2 Audio Control
9.1.4.3 Contrast (Minimum) 1.4.3 Contrast (Minimum)
9.1.4.4 Resize Text 1.4.4 Resize Text
9.1.4.5 Images of Text 1.4.5 Images of Text
9.1.4.10 Reflow 1.4.10 Reflow
9.1.4.11 Non-Text Contrast 1.4.11 Non-Text Contrast
9.1.4.12 Text Spacing 1.4.12 Text Spacing
9.1.4.13 Content on Hover or Focus 1.4.13 Content on Hover or Focus
9.2.1.1 Keyboard 2.1.1 Keyboard
9.2.1.2 No Keyboard Trap 2.1.2 No Keyboard Trap
9.2.1.4 Character Key Shortcuts 2.1.4 Character Key Shortcuts
9.2.2.1 Timing Adjustable 2.2.1 Timing Adjustable
9.2.2.2 Pause, Stop, Hide 2.2.2 Pause, Stop, Hide
9.2.3.1 Three Flashes or Below 2.3.1 Three Flashes or Below
9.2.4.1 Bypass Blocks 2.4.1 Bypass Blocks
9.2.4.2 Page Titled 2.4.2 Page Titled
9.2.4.3 Focus Order 2.4.3 Focus Order
9.2.4.4 Link Purpose (In Context) 2.4.4 Link Purpose (In Context)
9.2.4.5 Multiple Ways 2.4.5 Multiple Ways
9.2.4.6 Headings and Labels 2.4.6 Headings and Labels
9.2.4.7 Focus Visible 2.4.7 Focus Visible
9.2.5.1 Pointer Gestures 2.5.1 Pointer Gestures
9.2.5.2 Pointer Cancellation 2.5.2 Pointer Cancellation
9.2.5.3 Label in Name 2.5.3 Label in Name
9.2.5.4 Motion Actuation 2.5.4 Motion Actuation
9.3.1.1 Language of Page 3.1.1 Language of Page
9.3.1.2 Language of Parts 3.1.2 Language of Parts
9.3.2.1 On Focus 3.2.1 On Focus
9.3.2.2 On Input 3.2.2 On Input
9.3.2.3 Consistent Navigation 3.2.3 Consistent Navigation
9.3.2.4 Consistent Identification 3.2.4 Consistent Identification
9.3.3.1 Error Identification 3.3.1 Error Identification
9.3.3.2 Labels or Instructions 3.3.2 Labels or Instructions
9.3.3.3 Error Suggestion 3.3.3 Error Suggestion
9.3.3.4 Error Prevention (Legal, Financial, Data) 3.3.4 Error Prevention (Legal, Financial, Data)
9.4.1.1 Parsing 4.1.1 Parsing
9.4.1.2 Name, Role, Value 4.1.2 Name, Role, Value
9.4.1.3 Status Messages (WCAG 2.1) 4.1.3 Status Messages (WCAG 2.1)
9.6 WCAG Conformance Requirements 5.2 WCAG Conformance Requirements
10.1.1.1 Non-text Content 1.1.1 Non-text Content
10.1.2.1 Audio-only and Video-only (Pre-recorded) 1.2.1 Audio-only and Video-only (Pre-recorded)
10.1.2.2 Captions (Pre-recorded) 1.2.2 Captions (Pre-recorded)
10.1.2.3 Audio Description or Media Alternative (Pre-recorded) 1.2.3 Audio Description or Media Alternative (Pre-recorded)
10.1.2.5 Audio Description (Pre-recorded) 1.2.5 Audio Description (Pre-recorded)
10.1.3.1 Info and Relationships 1.3.1 Info and Relationships
10.1.3.2 Meaningful Sequence 1.3.2 Meaningful Sequence
10.1.3.3 Sensory Characteristics 1.3.3 Sensory Characteristics
10.1.3.4 Orientation 1.3.4 Orientation
10.1.3.5 Identify Input Purpose 1.3.5 Identify Input Purpose
10.1.4.1 Use of Colour 1.4.1 Use of Colour
10.1.4.2 Audio Control 1.4.2 Audio Control
10.1.4.3 Contrast (Minimum) 1.4.3 Contrast (Minimum)
10.1.4.4 Resize Text 1.4.4 Resize Text
10.1.4.5 Images of Text 1.4.5 Images of Text
10.1.4.10 Reflow 1.4.10 Reflow
10.1.4.11 Non-Text Contrast 1.4.11 Non-Text Contrast
10.1.4.12 Text Spacing 1.4.12 Text Spacing
10.1.4.13 Content on Hover or Focus 1.4.13 Content on Hover or Focus
10.2.1.1 Keyboard 2.1.1 Keyboard
10.2.1.2 No Keyboard Trap 2.1.2 No Keyboard Trap
10.2.1.4 Character Key Shortcuts 2.1.4 Character Key Shortcuts
10.2.2.1 Timing Adjustable 2.2.1 Timing Adjustable
10.2.2.2 Pause, Stop, Hide 2.2.2 Pause, Stop, Hide
10.2.3.1 Three Flashes or Below 2.3.1 Three Flashes or Below
10.2.4.2 Page Titled 2.4.2 Page Titled
10.2.4.3 Focus Order 2.4.3 Focus Order
10.2.4.4 Link Purpose (In Context) 2.4.4 Link Purpose (In Context)
10.2.4.6 Headings and Labels 2.4.6 Headings and Labels
10.2.4.7 Focus Visible 2.4.7 Focus Visible
10.2.5.1 Pointer Gestures 2.5.1 Pointer Gestures
10.2.5.2 Pointer Cancellation 2.5.2 Pointer Cancellation
10.2.5.3 Label in Name 2.5.3 Label in Name
10.2.5.4 Motion Actuation 2.5.4 Motion Actuation
10.3.1.1 Language of Page 3.1.1 Language of Page
10.3.1.2 Language of Parts 3.1.2 Language of Parts
10.3.2.1 On Focus 3.2.1 On Focus
10.3.2.2 On Input 3.2.2 On Input
10.3.3.1 Error Identification 3.3.1 Error Identification
10.3.3.2 Labels or Instructions 3.3.2 Labels or Instructions
10.3.3.3 Error Suggestion 3.3.3 Error Suggestion
10.3.3.4 Error Prevention (Legal, Financial, Data) 3.3.4 Error Prevention (Legal, Financial, Data)
10.4.1.1 Parsing 4.1.1 Parsing
10.4.1.2 Name, Role, Value 4.1.2 Name, Role, Value
10.4.1.3 Status Messages (WCAG 2.1) 4.1.3 Status Messages (WCAG 2.1)

For each sampled website, a set of pages to be included in each assessment was established, as defined in the Commission Implementing Decision (EU) 2018/1524 of 11 October 2018. This representative sample considers the following pages:

Each page included in the representative sample of each website was evaluated according to the WCAG 2.1 AA success criteria. For each page, one of the following results was provided:

Data collection was supported by the WCAG-EM Report Tool. At the end of the evaluation of each website, the report generated was recorded for future analysis.

The results of the in-depth monitoring of websites have been complemented with usability testing by people with disabilities.

For these tests, 6 participants were recruited, two with visual impairment, two with motor impairment, and two with hearing impairment. All participants were recruited through contacts with local institutions. In table 7 we can observe that all have been computer users for more than five years, and that they self-evaluate with different levels of expertise in the use of computers (between 1 and 5 where 1 represents beginner and 5 specialist). All sessions were conducted remotely using Zoom and screen sharing. Each session took about 45 min. Each session began with a brief introduction to the study, followed by a demographic questionnaire with questions related to computer usage and experience. Each participant was instructed to perform one task at a time, being able to ask the evaluator at any time to repeat the task. Participants were asked to think aloud while performing the task. The evaluator present took notes while observing participants’ interactions with the websites. When a participant was trapped in one step of the task, the evaluator helped overcome the problem so that the participant could explore the rest of the task. At the end of every two tasks with each of the websites, a semi-structured interview was carried out with three initial questions focused on the experience with the website, its accessibility, and finally what could be improved. Participants were rewarded for their contribution with a gift card worth EUR 20.

Table 7: Demographic information of test participants for website
ID Age Visual Capability Motor Capability Hearing Capability Assistive Technology Computer Use Expertise
PC1 35 Totally blind (no light perception) No limitations No limitations Screen reader Over 5 years 3
PC2 34 Severe low vision (visual acuity less than 6/60) No limitations No limitations Screen reader Over 5 years 4
PDM1 52 Sighted With limited handling and/or strength No limitations None Over 5 years 3
PDM2 25 Sighted With limited handling and/or strength No limitations None Over 5 years 5
PS1 42 Sighted No limitations Deep Deafness None Over 5 years 5
PS2 28 Sighted No limitations Deep Deafness None Over 5 years 5

For these tests, the 5 websites considered where the ones with the highest failure rates in the in-depth methodology according to the declaration of functional performance for each group of users recruited. Some websites obtained the same failure rate in their declaration of functional performance. In these cases, websites with the highest failure rate were considered considering all clauses. This sample consisted of 5 websites, and for each of them, 2 tasks were defined considering the primary use or essential functionalities of each one. The tasks were distributed in two groups, with equivalent effort. Each task was performed by one participant from each group of recruited users.

Methodology applied to in-depth monitoring of mobile applications

The iOS and Android applications of the same entity were considered individually in this analysis. The applications were installed directly from the official App Stores and standard user accounts were used for the evaluation. We have selected a set of screens to assess in each application following the methodology set out in points 2 and 3 of Annex I to the COMMISSION IMPLEMENTING DECISION (EU) 2018/1524 of 11 October 2018. This representative sample considers the following screens:

In screen analysis, mobile interface components such as App Drawers that are present on multiple screens were considered to be part of all screens where they are present.

The following tools and devices were used to support the manual evaluation of mobile applications:

The collection of evaluation data was supported by the WCAG-EM Report Tool.

The testing methodology for manual evaluations was based on the Trusted Tester methodology. This methodology is based on the Web Content Accessibility Guidelines (WCAG) 2.0. Considering that this analysis follows the EN 301 549 standard, based on WCAG 2.1, new tests have been added to the tests originally included in the Trusted Tester methodology, comprising non-covered Success Criteria. The added tests were based on sufficient techniques provided in WCAG 2.1. All possible tests to perform or adapt them have been performed in all applications. The success criteria considered in this assessment correspond to the AA level of WCAG 2.1, as set out in the EN 301 594 standard, and are presented in table 8. Criteria marked as Partially represent the instances where it was not possible to apply the test to one or more applications. In the specific case of clause 10.1.4.10 Reflow, the criterion was not possible to evaluate in iOS (because there is no equivalent functionality) being evaluated only on Android. Clauses 10.1.4.3 Contrast (Minimum) and 10.1.4.11 Non-text Contrast have not been evaluated in 3 Android applications because the screen capture feature is locked in them which prevents contrast testing.

Table 8: Clauses of EN 301 549 considered in the in-depth methodology for mobile applications
EN Clause Clause description WCAG SC SC description Assessed
10.1.1.1 Non-text Content 1.1.1 Non-text Content Yes
10.1.2.1 Audio-only and Video-only (Pre-recorded) 1.2.1 Audio-only and Video-only (Pre-recorded) Yes
10.1.2.2 Captions (Pre-recorded) 1.2.2 Captions (Pre-recorded) Yes
10.1.2.3 Audio Description or Media Alternative (Pre-recorded) 1.2.3 Audio Description or Media Alternative (Pre-recorded) Yes
10.1.2.5 Audio Description (Pre-recorded) 1.2.5 Audio Description (Pre-recorded) Yes
10.1.3.1 Info and Relationships 1.3.1 Info and Relationships Yes
10.1.3.2 Meaningful Sequence 1.3.2 Meaningful Sequence No
10.1.3.3 Sensory Characteristics 1.3.3 Sensory Characteristics Yes
10.1.3.4 Orientation 1.3.4 Orientation Yes
10.1.3.5 Identify Input Purpose 1.3.5 Identify Input Purpose No
10.1.4.1 Use of Colour 1.4.1 Use of Colour Yes
10.1.4.2 Audio Control 1.4.2 Audio Control Yes
10.1.4.3 Contrast (Minimum) 1.4.3 Contrast (Minimum) Partially
10.1.4.4 Resise Text 1.4.4 Resise Text Yes
10.1.4.5 Images of Text 1.4.5 Images of Text Yes
10.1.4.10 Reflow 1.4.10 Reflow Partially
10.1.4.11 Non-Text Contrast 1.4.11 Non-Text Contrast Partially
10.1.4.12 Text Spacing 1.4.12 Text Spacing No
10.1.4.13 Content on Hover or Focus 1.4.13 Content on Hover or Focus No
10.2.1.1 Keyboard 2.1.1 Keyboard Yes
10.2.1.2 No Keyboard Trap 2.1.2 No Keyboard Trap Yes
10.2.1.4 Character Key Shortcuts 2.1.4 Character Key Shortcuts No
10.2.2.1 Timing Adjustable 2.2.1 Timing Adjustable Yes
10.2.2.2 Pause, Stop, Hide 2.2.2 Pause, Stop, Hide Yes
10.2.3.1 Three Flashes or Below 2.3.1 Three Flashes or Below Yes
10.2.4.2 Page Titled 2.4.2 Page Titled Yes
10.2.4.3 Focus Order 2.4.3 Focus Order Yes
10.2.4.4 Link Purpose 2.4.4 Link Purpose Yes
10.2.4.6 Headings and Labels 2.4.6 Headings and Labels Yes
10.2.4.7 Focus Visible 2.4.7 Focus Visible Yes
10.2.5.1 Pointer Gestures 2.5.1 Pointer Gestures Yes
10.2.5.2 Pointer Cancellation 2.5.2 Pointer Cancellation Yes
10.2.5.3 Label in Name 2.5.3 Label in Name Yes
10.2.5.4 Motion Actuation 2.5.4 Motion Actuation Yes
10.3.1.1 Language of Page 3.1.1 Language of Page No
10.3.1.2 Language of Parts 3.1.2 Language of Parts No
10.3.2.1 On Focus 3.2.1 On Focus Yes
10.3.2.2 On Input 3.2.2 On Input Yes
10.3.3.1 Error Identification 3.3.1 Error Identification Yes
10.3.3.2 Labels or Instructions 3.3.2 Labels or Instructions Yes
10.3.3.3 Error Suggestion 3.3.3 Error Suggestion Yes
10.3.3.4 Error Prevention (Legal, Financial, Date) 3.3.4 Error Prevention (Legal, Financial, Date) Yes
10.4.1.1 Parsing 4.1.1 Parsing No
10.4.1.2 Name, Role, Value 4.1.2 Name, Role, Value Yes
10.4.1.3 Status Messages (WCAG 2.1) 4.1.3 Status Messages (WCAG 2.1) No
11.1.1.1 Non-text Content 1.1.1 Non-text Content Yes
11.1.2.1 Audio-only and Video-only (Pre-recorded) 1.2.1 Audio-only and Video-only (Pre-recorded) Yes
11.1.2.2 Captions (Pre-recorded) 1.2.2 Captions (Pre-recorded) Yes
11.1.2.3 Audio Description or Media Alternative (Pre-recorded) 1.2.3 Audio Description or Media Alternative (Pre-recorded) Yes
11.1.2.5 Audio Description (Pre-recorded) 1.2.5 Audio Description (Pre-recorded) Yes
11.1.3.1 Info and Relationships 1.3.1 Info and Relationships Yes
11.1.3.2 Meaningful Sequence 1.3.2 Meaningful Sequence No
11.1.3.3 Sensory Characteristics 1.3.3 Sensory Characteristics Yes
11.1.3.4 Orientation 1.3.4 Orientation Yes
11.1.3.5 Identify Input Purpose 1.3.5 Identify Input Purpose No
11.1.4.1 Use of Colour 1.4.1 Use of Colour Yes
11.1.4.2 Audio Control 1.4.2 Audio Control Yes
11.1.4.3 Contrast (Minimum) 1.4.3 Contrast (Minimum) Partially
11.1.4.4 Resise Text 1.4.4 Resise Text Yes
11.1.4.5 Images of Text 1.4.5 Images of Text Yes
11.1.4.10 Reflow 1.4.10 Reflow Partially
11.1.4.11 Non-Text Contrast 1.4.11 Non-Text Contrast Partially
11.1.4.12 Text Spacing 1.4.12 Text Spacing No
11.1.4.13 Content on Hover or Focus 1.4.13 Content on Hover or Focus No
11.2.1.1 Keyboard 2.1.1 Keyboard Yes
11.2.1.2 No Keyboard Trap 2.1.2 No Keyboard Trap Yes
11.2.1.4 Character Key Shortcuts 2.1.4 Character Key Shortcuts No
11.2.2.1 Timing Adjustable 2.2.1 Timing Adjustable Yes
11.2.2.2 Pause, Stop, Hide 2.2.2 Pause, Stop, Hide Yes
11.2.3.1 Three Flashes or Below 2.3.1 Three Flashes or Below Yes
11.2.4.2 Page Titled 2.4.2 Page Titled Yes
11.2.4.3 Focus Order 2.4.3 Focus Order Yes
11.2.4.4 Link Purpose 2.4.4 Link Purpose Yes
11.2.4.6 Headings and Labels 2.4.6 Headings and Labels Yes
11.2.4.7 Focus Visible 2.4.7 Focus Visible Yes
11.2.5.1 Pointer Gestures 2.5.1 Pointer Gestures Yes
11.2.5.2 Pointer Cancellation 2.5.2 Pointer Cancellation Yes
11.2.5.3 Label in Name 2.5.3 Label in Name Yes
11.2.5.4 Motion Actuation 2.5.4 Motion Actuation Yes
11.3.1.1 Language of Page 3.1.1 Language of Page No
11.3.1.2 Language of Parts 3.1.2 Language of Parts No
11.3.2.1 On Focus 3.2.1 On Focus Yes
11.3.2.2 On Input 3.2.2 On Input Yes
11.3.3.1 Error Identification 3.3.1 Error Identification Yes
11.3.3.2 Labels or Instructions 3.3.2 Labels or Instructions Yes
11.3.3.3 Error Suggestion 3.3.3 Error Suggestion Yes
11.3.3.4 Error Prevention (Legal, Financial, Date) 3.3.4 Error Prevention (Legal, Financial, Date) Yes
11.4.1.1 Parsing 4.1.1 Parsing No
11.4.1.2 Name, Role, Value 4.1.2 Name, Role, Value Yes
11.4.1.3 Status Messages (WCAG 2.1) 4.1.3 Status Messages (WCAG 2.1) No

At the end of the evaluation of each application, a report was recorded for future analysis. After conducting the evaluations, the results obtained were analysed, considering the clauses that are verified or violated.

After finalising the expert evaluation of all Android applications, the two less compliant were selected to conduct a study with users.

Four participants were recruited, two visually impaired screen reader users, one motor-disabled user and one deaf user. In table 9 we can observe that all have been mobile device users for more than three years, and that they self-evaluate with different levels of expertise in the use of smartphone (between 1 and 5 where 1 represents beginner and 5 specialist). Two had previous experience with one of the applications and one had tried in the past, unsuccessfully, to use the second. Participants were recruited by contacts with local institutions. The session with the participant P3 was conducted in person, while with the participants P1, P2 and P4 the session was remote using Zoom and screen sharing. The sessions lasted about 45 min. Each session started with a brief demographic questionnaire with questions related to smartphone usage and experience. The participants then installed the two applications selected for the study if they did not have them on their device. Each participant was instructed to perform one task at a time, being able to ask the evaluator at any time to repeat the task. Participants were asked to think aloud while performing the task. The evaluator present took notes while observing participants’ interactions with their devices. When a participant was trapped in one step of the task, the evaluator helped overcome the problem so that the participant could explore the rest of the task. At the end of every two tasks with each of the applications, authorisation was requested to record audio, and a semi-structured interview was performed with three initial questions focused on the application experience: 1) “How was your experience with this application?”; 2) “What is your opinion on the accessibility of this application?”; and 3) “What can be improved?” Participants were rewarded for their contribution with a gift card worth EUR 20.

Table 9: Demographic information of test participants with mobile application users
ID Age Vision Motor Capacity Hearing Assistive Technology Smartphone Use Expertise App 1 Experience App 2 Experience
P1 42 Sighted With limited handling ability and/or force No limitations No No Over 3 years 3 Never used
P2 34 Severe low vision (visual acuity less than 6/60) No limitations No limitations Screen reader Over 3 years Use occasionally Tried
P3 42 Severe low vision (visual acuity less than 6/60) No limitations No limitations Screen reader Over 3 years 5 I’ve used it before Never used
P4 44 Sighted No limitations Deep Deaf None Over 3 years 1 Never used Never used

According to Statcounter Global, in Portugal the percentage of Android users is 72%, compared to 27% for iOS. Since Android is the most popular (more than double) operating system in Portugal, user reviews were conducted on Android applications. It should be noted that no restrictions were imposed on recruitment and that all users contacted to participate in the study were found to be Android users.

The least compliant Android applications were selected. However, it should be noted that two others, if selected, would have been inaccessible to screen reader users as it is not possible to interact with the elements necessary for navigation using a screen reader. In the selected applications, two typical tasks were created for the objectives inherent to them.

Outcome of the monitoring

Detailed outcome

Simplified website monitoring

In total, 281 websites were analysed, from which 281,706 pages were obtained. Of this total, 254,684 pages were evaluated, corresponding to 90.4% of the sample. For six websites it was only possible to access one page.

Home+ sample analysis

In the Home+ sample, of the 281 websites, 22,489 pages were collected, corresponding to an average of 80 pages per website. In figure 1 it is possible to see the distribution of the number of pages per website and it is noticeable that for this sample most websites have 50 or fewer pages, with more than three quarters of the sites having 100 or fewer pages.

Figure 1: Page distribution bar chart for the Home+ sample.

The table 10 provides details on the distribution of the number of pages per website in this sample.

Table 10: Distribution of the number of pages per website for the Home+ sample
Page range Frequency (number of websites) Frequency (number of websites) (%) Accumulated frequency Accumulated frequency (%)
[1,25] 69 24.8% 69 24.8%
]25,50] 78 28.1% 147 52.9%
]50,75] 43 15.5% 190 68.3%
]75,100] 26 9.4% 216 77.7%
]100,125] 13 4.7% 229 82.4%
]125,150] 12 4.3% 241 86.7%
]150,175] 13 4.7% 254 91.4%
]175,200] 2 0.7% 256 92.1%
]200,225] 4 1.4% 260 93.5%
]225,250] 3 1.1% 263 94.6%
]250,275] 1 0.4% 264 95.0%
]275,300] 2 0.7% 266 95.7%
]300,325] 6 2.2% 272 97.8%
]325,350] 0 0.0% 272 97.8%
]350,375] 0 0.0% 272 97.8%
]375,400] 0 0.0% 272 97.8%
]400,425] 2 0.7% 274 98.6%
]425,450] 1 0.4% 275 98.9%
]450,475] 0 0.0% 275 98.9%
]475,500] 0 0.0% 275 98.9%
]500,525] 1 0.4% 276 99.3%
]525,550] 1 0.4% 277 99.6%
]550,575] 0 0.0% 277 99.6%
]575,600] 1 0.4% 278 100.0%

Of the 281 websites, 3 failed to have at least one page evaluated, and 38 more failed the requirement to have at least 10 pages evaluated with 100 or more elements, so they are not considered in the analysis below. Thus, for this analysis only 240 websites with at least 10 pages evaluated with 100 or more elements are considered. In total, in this sample, the results of the evaluations of 19,931 pages were analysed.

Clause distribution

In table 11 the number (and percentage) of websites verifying or violating each clause, as well as those requiring manual validation or not applicable on any page of the website, is displayed.

Table 11: Compliance of websites with the clauses of EN 301 549 tested for the sample Home+
Clause EN 301 549 WCAG Compliance Level Non-conforming Needing Manual Validation Not Applicable
9.1.1.1 Non-text Content A 173 (72.1%) 0 (0%) 0 (0%)
9.1.2.1 Audio-only and Video-only (prerecorded) A 0 (0%) 33 (13.8%) 207 (86.3%)
9.1.2.2 Captions (prerecorded) A 0 (0%) 33 (13.8%) 207 (86.3%)
9.1.2.3 Audio Description or Media Alternative (prerecorded) A 0 (0%) 33 (13.8%) 207 (86.3%)
9.1.3.1 Info and Relationships A 179 (74.6%) 0 (0%) 1 (0.4%)
9.1.4.2 Audio Control A 0 (0%) 4 (1.7%) 236 (98.3%)
9.2.1.1 Keyboard A 20 (8.3%) 0 (0%) 151 (62.9%)
9.2.2.1 Timing Adjustable A 4 (1.7%) 0 (0%) 221 (92.1%)
9.2.4.1 Bypass Blocks A 0 (0%) 179 (74.6%) 0 (0%)
9.2.4.2 Page Titled A 8 (3.3%) 0 (0%) 0 (0%)
9.2.4.4 Link Purpose A 214 (89.2%) 21 (8.8%) 0 (0%)
9.2.5.3 Label in Name A 37 (15.4%) 0 (0%) 131 (54.6%)
9.3.1.1 Language of Page A 48 (20.0%) 0 (0%) 0 (0%)
9.3.3.1 Error Identification A 0 (0%) 233 (97.1%) 7 (2.9%)
9.4.1.1 Parsing A 167 (69.6%) 0 (0%) 0 (0%)
9.4.1.2 Name, Role, Value A 235 (97.9%) 1 (0.4%) 0 (0%)
9.1.2.5 Audio Description (prerecorded) AA 0 (0%) 33 (13.8%) 207 (86.3%)
9.1.3.4 Orientation AA 0 (0%) 0 (0%) 240 (100.0%)
9.1.3.5 Identify Input Purpose AA 2 (0.8%) 0 (0%) 161 (67.1%)
9.1.4.3 Contrast (Minimum) AA 221 (92.1%) 10 (4.2%) 0 (0%)
9.1.4.4 Resise text AA 63 (26.3%) 169 (70.4%) 5 (2.1%)
9.3.1.2 Language of Parts AA 0 (0%) 0 (0%) 159 (66.3%)

Observing the table 11 we can conclude that there was low compliance to the clauses of EN 301 549.

The clauses with the highest index of non-compliance were:

— 9.4.1.2 Name, Role, Value with 97.9% of applicable websites not compliant; — 9.1.4.3 Contrast (Minimum) with 92.1% of applicable websites not compliant; — 9.2.4.4 Link Purpose (In Context) with 89.2% of applicable websites not compliant; — 9.1.3.1 Info and Relationships with 74.6% of applicable websites not compliant; — 9.1.1.1 Non-text Content with 72.1% of applicable websites not compliant;
— 9.4.1.1 Parsing with 69.6% of applicable websites not compliant.

Distribution of Functional Performance Statements

Two analyses were carried out in relation to these statements:

The table 12 presents the results obtained in the first analysis, in which only the clauses of primary relationships were taken into account.

Table 12: Compliance of websites with functional performance declarations considering primary relationships for the Home+ sample
Functional Performance Statement Non-conforming Needing Manual Validation Not Applicable
Usage without vision (WV) 278 (100%) 0 (0%) 0 (0%)
Usage with limited vision (LV) 277 (99.6%) 1 (0.4%) 0 (0%)
Usage without perception of colour (WPC) 246 (88.5%) 28 (10.1%) 1 (0.4%)
Usage without hearing (WH) 227 (81.7%) 18 (6.54%) 4 (1.4%)
Usage with limited hearing (LH) 86 (30.9%) 19 (6.8%) 167 (60.1%)
Usage with limited manipulation or strength (LMS) 276 (99.3%) 2 (0.7%) 0 (0%)
Usage with limited reach (LR) 39 (14%) 0 (0%) 163 (58.6%)
Usage with limited cognition, language or learning (LC) 276 (99.2%) 2 (0.7%) 0 (0%)

Observing the table 12 we can conclude that there is a large non-compliance rate with functional performance statements, when evaluating only the clauses of primary relationships. The functional performance statement with the lowest non-conforming rate was:

The table 13 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were taken into account.

Table 13: Compliance of websites with functional performance statements considering primary and secondary relationships for the Home+ sample
Functional Performance Statement Non-conforming Needing Manual Validation Not Applicable
Usage without vision (WV) 240 (100.0%) 0 (0%) 0 (0%)
Usage with limited vision (LV) 240 (100.0%) 0 (0%) 0 (0%)
Usage without perception of colour (WPC) 221 (92.1%) 19 (7.9%) 0 (0%)
Usage without hearing (WH) 209 (87.1%) 15 (6.3%) 0 (0%)
Usage with limited hearing (LH) 209 (87.1%) 15 (6.3%) 0 (0%)
Usage with no or limited vocal capability (LVC) 238 (99.2%) 2 (0.8%) 0 (0%)
Usage with limited manipulation or strength (LMS) 239 (99.6%) 1 (0.4%) 0 (0%)
Usage with limited reach (LR) 37 (15.4%) 0 (0%) 131 (54.6%)
Usage with limited cognition, language or learning (LC) 240 (100.0%) 0 (0%) 0 (0%)
Privacy (P) 206 (85.8%) 11 (4.6%) 0 (0%)

Observing the table 13 we can conclude that there is a great non-compliance rate in relation to functional performance statements when evaluating all clauses. The functional performance statement with the lowest non-compliance rate was:

Distribution of tests by website

The table 14 displays the number of websites on which each AccessMonitor test is verified, as well as the type of practice (good or bad practice) that the test validates. From the analysis of table 14 we can verify that of the 11 good practices analysed, one is verified only on one website (“I located in the metadata of page X elements <link> which could be used to build a navigation system”). On the other hand, of the 28 bad practices, only one is verified on a single website (I found X abbreviations where you forgot to put the extension”). The average of websites following each good practice is 59%, while the average of websites applying each bad practice is 38%.

The best practice applied by more websites is “I found a title on the page and it seems correct”, applied to 99.6% of websites. The least applied good practice is “I located in the metadata of page X elements <link> which could be used to build a navigation system”, applied to only 0.4% of websites.

The bad practice applied on more websites is “I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for large letter text and 4.5 for 1 for normal letter text”, verified in 92.1% of websites. The bad practice applied on fewer websites is “I found X abbreviations in which you forgot to put the extension”, verified at 0.4% of websites.

Table 14: Number of websites where each AccessMonitor test is true in the Home+ sample
Assertion Number of websites Type of practice
I noticed that all images on the page have the required alternative equivalent in text. 195 Good practice
I found X images on the page that do not have the required alternative equivalent in text. 169 Bad practice
I found X images on the page with alt=“” (above null). 221 To validate manually
I found X images on the page with alt that does not serve as an alternative equivalent. 79 Bad practice
I found X images on the page where alt has more than 100 characters. 73 To validate manually
I noticed that all <area> active page image maps make use of the alt attribute. 17 Good practice
I found X elements <area> without attribute alt or with alt=“”. 35 Bad practice
I have determined that all graphic buttons on the page make use of the alt attribute. 25 Good practice
I located X graphic buttons on the page that does not have the alt attribute. 12 Bad practice
I found X links whose content is empty. Or rather, it is composed of only one image and the image has an empty character as an alternative textual equivalent (i.e. alt=“). 173 Bad practice
I have identified X cases where the title attribute of the link element is limited to repeating the text in the link. 164 Bad practice
I found X groups of links with the same text but whose destination is different. 233 To validate manually
I found that the first link of the page allows us to jump to the main content. 74 To validate manually
I found that the first page link does not allow to jump directly to the main content area. 198 Bad practice
I found X links to bypass blocks of content. 184 To validate manually
I found X headers on the page. 238 To validate manually
I did not find a main header <H1> on this page it’s marked. 138 Bad practice
I located X occurrences of empty lists or with <li> off the lists. 136 Bad practice
I found X elements <fieldset> off the form. 26 Bad practice
I found X elements <fieldset> No description. 55 Bad practice
I located X elements <label> which are not visible or incorrectly positioned. 105 Bad practice
I have determined that all form controls have an accessible name. 190 Good practice
I found X form controls with no access name. 144 To validate manually
I found X cases where javascript is used to remove focus from the field, whenever the field receives the focus. 65 Bad practice
I noticed that all forms have a button to submit the data to the server. 186 Good practice
I have identified X forms without the button to submit the data to the server. 120 Bad practice
Locate X data tables without the <caption> element. 191 Bad practice
I found X tables without identified headers but that use the <caption> element. 19 To validate manually
I found X complex data tables — where multiple rows configure header cells — in which data cells were found to exist without the <headers> attribute. 69 Bad practice
I found X tables without marked headers. 175 Bad practice
I found X elements <iframe> No title. 154 Bad practice
I found X abbreviations in which you forgot to put the extension. 1 Bad practice
I have identified X cases in which CSS specifies a line spacing of less than 1.5. 3 To validate manually
I found X cases where you use justified text via HTML. 38 Bad practice
I have identified X cases where justified text is used via CSS. 147 To validate manually
I have identified X cases where the font size is expressed in absolute units of measurement. 53 Bad practice
I have identified X cases in the CSS where absolute units of measurement are used to define the width of the contents boxes. 75 To validate manually
I checked that there are X cases, where the units of measurement that define the width of the content elements existing in HTML are expressed in absolute values. 188 To validate manually
I checked that there is X case, where the units of measurement that define the width of the content elements existing in HTML are expressed in relative values. 90 Good practice
I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for large letter text and 4.5 to 1 for normal letter text. 221 Bad practice
I have identified X CSS rules where the color of the letter or background color is not specified. 73 To validate manually
I have identified X cases where redundant event handlers are not used. 81 Bad practice
I have identified X cases where event handlers are associated with non-interactive elements. 49 Bad practice
I noticed that there are no obsolete elements used for visual control of the presentation. 238 Good practice
I checked that the main language of the page is marked “X”. 216 To validate manually
I noticed that the X attribute is missing. 45 Bad practice
I checked that the main language of the page is incorrectly marked. “X” is there? 3 Bad practice
I checked that this page has no title — you lack the <title> element. 2 Bad practice
I checked that this page has no title — the <title> element; it’s empty. 6 Bad practice
I found a title on the page and it seems correct. 239 Good practice
It seems to me that the page title is too large — it contains X characters. 82 To validate manually
I found in the metadata page X elements <link> they could be used to build a navigation system. 1 Good practice
I have found that the page will automatically restart through the <meta http-equiv=“refresh”> element. 3 To validate manually
I found that the page is automatically redirected through the <meta http-equiv=“redirect”> element. 1 To validate manually
I noticed that all headers on this page have an accessible name 234 Good practice

The table 15 displays the number of websites that pass, fail and require manual validation by ACT rule tested. From the analysis of table 15 we can verify that 33 ACT rules passed at least once and 27 ACT rules failed at least once in the entire Home+ sample. Seven ACT rules have been found that fail on more websites than they pass:

— Link has non-empty accessible name — Aria required context role — Aria required owned elements — Object element rendering non-text content has non-empty accessible name — iframe with negative tabindex has no interactive elements — First focusable element is link to non-repeated content — Text has enhanced contrast

If we do not consider the rule “First focusable element is link to non-repeated content”, because it can only automatically detect cases in which rule is not met, without being able to automatically detect cases where it is, we can verify that we are in the presence of four types of problems: lack of accessible names in HTML elements, incorrect use of ARIA, incorrect attribution of focus to iframes, and use of colors with insufficient contrast.

Table 15: Number of websites that pass, fail or need manual validation for each QualWeb ACT rule in the Home+ sample
ACT Rule Passing Failing Needing Manual Validation
Orientation of the page is not restricted using CSS transform property 0 0 0
iframe elements with identical accessible names have equivalent purpose 0 0 22
Button has non-empty accessible name 218 49 0
Link has non-empty accessible name 150 214 0
Element with aria-hidden has no focusable content 180 60 0
meta viewport allows for zoom 171 63 0
audio or video avoids automatically playing audio 0 0 4
id attribute value is unique 203 167 0
role attribute has valid value 196 54 0
svg element with explicit role has non-empty accessible name 21 1 0
Element with lang attribute has valid language tag 81 0 0
video element visual content has accessible alternative 0 0 33
autocomplete attribute has valid value 78 2 0
ARIA state or property is permitted 190 68 0
video element auditory content has accessible alternative 0 0 33
aria-* attribute is defined in WAI-ARIA 215 1 0
Element with role attribute has required states and properties 204 17 0
audio element content has text alternative 0 0 0
Visible label is part of accessible name 82 37 0
video element visual-only content has accessible alternative 0 0 33
video element visual content has strict accessible alternative 0 0 33
ARIA required context role 49 59 0
ARIA state or property has valid value 202 6 0
headers attribute specified on a cell refers to cells in the same table element 15 5 0
ARIA required owned elements 45 75 0
Table header cell has assigned cells 92 18 0
Zoomed text node is not clipped with CSS overflow 0 0 231
Error message describes invalid form field value 0 0 233
Object element rendering non-text content has non-empty accessible name 0 6 0
Scrollable element is keyboard accessible 76 20 0
Links with identical accessible names and context serve equivalent purpose 120 0 210
Element marked as decorative is not exposed 219 74 0
Audio or video that plays automatically has no audio that lasts more than 3 seconds 0 0 4
audio or video that plays automatically has a control mechanism 0 0 4
video element visual-only content is media alternative for text 0 0 33
video element visual-only content has description track 0 0 0
video element visual-only content has transcript 0 0 33
video element visual-only content has audio track alternative 0 0 33
video element visual content has audio description 0 0 33
video element content is media alternative for text 0 0 33
video element visual content has description track 0 0 33
audio element content has transcript 0 0 0
audio element content is media alternative for text 0 0 0
video element auditory content has captions 0 0 33
Audio and visuals of video element have transcript 0 0 33
Element in sequential focus order has visible focus 0 0 240
Document has a landmark with non-repeated content 91 0 203
Document has heading for non-repeated content 91 0 203
Element with presentational children has no focusable content 238 26 0
Menuitem has non-empty accessible name 10 5 0
Letter spacing in style attributes is not !important 89 1 0
Word spacing in style attributes is not !important 31 0 0
iframe with negative tabindex has no interactive elements 14 71 0
meta element has no refresh delay (no exception) 15 4 0
First focusable element is link to non-repeated content 0 206 34
Block of repeated content is collapsible 91 0 203
Document has an instrument to move focus to non-repeated content 91 0 203
Bypass Blocks of Repeated Content 178 0 179
Text has enhanced contrast 63 236 2
Distribution of tests per page

In the table 16 the number of pages on which each AccessMonitor test is true, as well as the type of practice the test validates, is shown. From the analysis of table 16 we can verify that all good practices and all bad practices have been detected on at least one page. Of the 11 good practices analysed, all take place on at least 6 pages. Of the 28 bad practices, they all occur on more than one page. The average of pages that follow each good practice is 45%, while the average of pages that apply each bad practice is 18%.

The good practice applied by more pages is “I found a title on the page and it seems correct”, applied to 99.1% of the pages. The least applied good practice is “I located in the metadata of page X elements <link> which can be used to build a navigation system”, applied on only 0.03% of pages. Good practices under these conditions are the same as in the website analysis.

The bad practice applied on more pages is “I found that the first page link does not allow to jump directly to the main content area”, verified in 70% of the pages. The bad practices applied on fewer pages are “I found X abbreviations where you forgot to put the extension” and “I verified that this page has no title — it lacks the element <title>”, verified in 0.01% of pages. Bad practices under these conditions vary from what was found in the website analysis.

Table 16: Number of pages where each AccessMonitor test is true in the Home+ sample
Assertion Number of pages Type of practice
I noticed that all images on the page have the required alternative equivalent in text. 14145 Good practice
I found X images on the page that do not have the required alternative equivalent in text. 5565 Bad practice
I found X images on the page with alt=“” (above null). 8794 To validate manually
I found X images on the page with alt that does not serve as an alternative equivalent. 1561 Bad practice
I found X images on the page where alt has more than 100 characters. 875 To validate manually
I noticed that all <area> active page image maps make use of the alt attribute. 217 Good practice
I found X elements <area> without attribute alt or with alt=“”. 259 Bad practice
I have determined that all graphic buttons on the page make use of the alt attribute. 3477 Good practice
Locate X graphic buttons on the page that does not have the alt attribute. 120 Bad practice
I found X links whose content is empty. Or rather, it is composed of only one image and the image has an empty character as an alternative textual equivalent (i.e. alt=“). 6031 Bad practice
I have identified X cases where the title attribute of the link element is limited to repeating the text in the link. 10497 Bad practice
I found X groups of links with the same text but whose destination is different. 11791 To validate manually
I found that the first link of the page allows us to jump to the main content. 5982 To validate manually
I found that the first page link does not allow to jump directly to the main content area. 13949 Bad practice
I found X links to bypass blocks of content. 13067 To validate manually
I found X headers on the page. 18842 To validate manually
I did not find a main header <H1> on this page it’s marked. 4008 Bad practice
Locate X occurrences of empty lists or with <li> off the lists. 4168 Bad practice
I found X elements <fieldset> off the form. 671 Bad practice
I found X elements <fieldset> No description. 618 Bad practice
I located X elements <label> which are not visible or incorrectly positioned. 6739 Bad practice
I have determined that all form controls have an accessible name. 11384 Good practice
I found X form controls with no access name. 3235 To validate manually
I found X cases where javascript is used to remove focus from the field, whenever the field receives the focus. 2983 Bad practice
I noticed that all forms have a button to submit the data to the server. 12130 Good practice
I have identified X forms without the button to submit the data to the server. 4730 Bad practice
Locate X data tables without the <caption> element. 6411 Bad practice
I found X tables without identified headers but that use the <caption> element. 128 To validate manually
I found X complex data tables — where multiple rows configure header cells — in which data cells were found to exist without the <headers> attribute. 1256 Bad practice
I found X tables without marked headers. 5837 Bad practice
I found X elements <iframe> No title. 2103 Bad practice
I found X abbreviations in which you forgot to put the extension. 2 Bad practice
I have identified X cases in which CSS specifies a line spacing of less than 1.5. 27 To validate manually
I found X cases where you use justified text via HTML. 161 Bad practice
I have identified X cases where justified text is used via CSS. 4248 To validate manually
I have identified X cases where the font size is expressed in absolute units of measurement. 4462 Bad practice
I have identified X cases in the CSS where absolute units of measurement are used to define the width of the contents boxes. 5853 To validate manually
I checked that there are X cases, where the units of measurement that define the width of the content elements existing in HTML are expressed in absolute values. 3931 To validate manually
I checked that there is X case, where the units of measurement that define the width of the content elements existing in HTML are expressed in relative values. 1579 Good practice
I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for large letter text and 4.5 to 1 for normal letter text. 13875 Bad practice
I have identified X CSS rules where the color of the letter or background color is not specified. 3785 To validate manually
I have identified X cases where redundant event handlers are not used. 2439 Bad practice
I have identified X cases where event handlers are associated with non-interactive elements. 1017 Bad practice
I noticed that there are no obsolete elements used for visual control of the presentation. 18772 Good practice
I checked that the main language of the page is marked “X”. 18085 To validate manually
I noticed that the X attribute is missing. 1846 Bad practice
I checked that the main language of the page is incorrectly marked. “X” is there? 38 Bad practice
I checked that this page has no title — you lack the <title> element. 2 Bad practice
I checked that this page has no title — the <title> element; it’s empty. 186 Bad practice
I found a title on the page and it seems correct. 19743 Good practice
It seems to me that the page title is too large — it contains X characters. 430 To validate manually
I found in the metadata page X elements <link> which could be used to build a navigation system.. 6 Good practice
I have found that the page will automatically restart through the <meta http-equiv=“refresh”> element. 224 To validate manually
I found that the page is automatically redirected through the <meta http-equiv=“redirect”> element. 1 To validate manually
I noticed that all headers on this page have an accessible name 16868 Good Practice

In the table 17 the number of pages that pass, fail and require manual validation by ACT rule tested is displayed. From the analysis of table 17 we can verify that there is similarity to what happens to websites: 33 ACT rules passed at least once and 27 ACT rules failed at least once. Also in this analysis, seven ACT rules were found that fail in more pages than those in which they pass. However, the seven rules are not the same:

— Link has non-empty accessible name — Visible label is part of accessible name — headers attribute specified on a cell refers to cells in the same table element — Object element rendering non-text content has non-empty accessible name — MenuItem has non-empty accessible name — First focusable element is link to non-repeated content — Text has enhanced contrast

Continuing not to consider the rule “First focusable element is link to non-repeated content” due to the reasons presented above, we can verify that we are in the presence of three types of problems: lack of accessible accessible for HTML elements, incorrect use of the headers attribute in tables, and use of colors with insufficient contrast.

Table 17: Number of pages that pass, fail or need manual validation for each QualWeb ACT rule in the Home+ sample
ACT Rule Passing Failing Needing Manual Validation
Orientation of the page is not restricted using CSS transform property 0 0 0
iframe elements with identical accessible names have equivalent purpose 0 0 42
Button has non-empty accessible name 13175 1433 0
Link has non-empty accessible name 9277 10652 0
Element with aria-hidden has no focusable content 10777 729 0
meta viewport allows for zoom 12505 5549 0
audio or video avoids automatically playing audio 0 0 8
id attribute value is unique 14704 5220 0
role attribute has valid value 12464 613 0
svg element with explicit role has non-empty accessible name 2100 41 0
Element with lang attribute has valid language tag 2267 0 0
video element visual content has accessible alternative 0 0 105
autocomplete attribute has valid value 3411 2 0
ARIA state or property is permitted 11720 1728 0
video element auditory content has accessible alternative 0 0 105
aria-* attribute is defined in WAI-ARIA 16053 1 0
Element with role attribute has required states and properties 12454 54 0
audio element content has text alternative 0 0 0
Visible label is part of accessible name 1069 1982 0
video element visual-only content has accessible alternative 0 0 105
video element visual content has strict accessible alternative 0 0 105
ARIA required context role 1159 738 0
ARIA state or property has valid value 13422 26 0
headers attribute specified on a cell refers to cells in the same table element 70 99 0
ARIA required owned elements 1106 906 0
Table header cell has assigned cells 806 30 0
Zoomed text node is not clipped with CSS overflow 0 0 18114
Error message describes invalid form field value 0 0 17881
Object element rendering non-text content has non-empty accessible name 0 114 0
Scrollable element is keyboard accessible 3695 346 0
Links with identical accessible names and context serve equivalent purpose 1608 0 5206
Element marked as decorative is not exposed 10900 700 0
Audio or video that plays automatically has no audio that lasts more than 3 seconds 0 0 8
audio or video that plays automatically has a control mechanism 0 0 8
video element visual-only content is media alternative for text 0 0 105
video element visual-only content has description track 0 0 0
video element visual-only content has transcript 0 0 105
video element visual-only content has audio track alternative 0 0 105
video element visual content has audio description 0 0 105
video element content is media alternative for text 0 0 105
video element visual content has description track 0 0 105
audio element content has transcript 0 0 0
audio element content is media alternative for text 0 0 0
video element auditory content has captions 0 0 105
Audio and visuals of video element have transcript 0 0 105
Element in sequential focus order has visible focus 0 0 19930
Document has a landmark with non-repeated content 3411 0 16520
Document has heading for non-repeated content 3411 0 16520
Element with presentational children has no focusable content 19441 481 0
Menuitem has non-empty accessible name 364 742 0
Letter spacing in style attributes is not !important 2125 1 0
Word spacing in style attributes is not !important 83 0 0
iframe with negative tabindex has no interactive elements 622 593 0
meta element has no refresh delay (no exception) 956 225 0
First focusable element is link to non-repeated content 0 14544 5387
Block of repeated content is collapsible 3411 0 16520
Document has an instrument to move focus to non-repeated content 3411 0 16520
Bypass Blocks of Repeated Content 6685 0 13246
Text has enhanced contrast 1020 17994 916
Score Distribution

In the figure 2 the distribution of scores for the 240 websites is displayed. From the graph we can observe that the range of scores from 5 to 6 (inclusive) contains the highest concentration of websites. Additionally, only 1 website, corresponding to 0.42% of the sample, has a score of 10.

Figure 2: Score distribution bar chart of the websites in the Home+ sample.

In the table 18 the distribution of scores for the 240 websites is displayed. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

Table 18: Distribution of AccessMonitor scores for websites in the Home+ sample
Score range Frequency (number of websites) Frequency (number of websites) (%) Accumulated frequency Accumulated frequency (%)
[1,2] 0 0% 0 0%
]2.3] 3 1.3% 3 1.3%
]3.4] 10 4.2% 13 5.4%
]4,5] 50 20.8% 63 26.3%
]5,6] 67 27.9% 130 54.2%
]6,7] 59 24.6% 189 78.8%
]7,8] 32 13.3% 221 92.1%
]8,9] 10 4.2% 231 96.3%
]9,10] 9 3.8% 240 100%

In the figure 3 the distribution of the scores of the 19,931 pages evaluated of the 240 websites is shown. From the graph we can observe that the range of scores from 6 to 7 (inclusive) contains the highest concentration of pages. Additionally, only 41 pages, corresponding to 0.2% of the sample, have a score of 10.

Figure 3: Score distribution bar chart of the pages in the Home+ sample.

In the table 19 the distribution of scores for the 19,931 pages of the 240 websites is displayed. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive), even tough the mode being the range from 6 to 7 (inclusive) with 31.3% of the pages.

Table 19: Distribution of AccessMonitor scores for pages in the Home+ sample
Score interval Frequency (page number) Frequency (page number) (%) Accumulated frequency Accumulated frequency (%)
[1,2] 0 0% 0 0%
]2,3] 190 1% 190 1%
]3,4] 817 4.1% 1007 5.1%
]4,5] 4435 22.3% 5442 27.3%
]5,6] 5033 25.3% 10475 52.6%
]6,7] 6241 31.3% 16716 83.9%
]7,8] 1911 9.6% 18627 93.5%
]8,9] 626 3.1% 19253 96.6%
]9,10] 678 3.4% 19931 100%

2K sample analysis

In the 2K sample, of the 281 websites, 281,706 pages were collected, corresponding to an average of 1003 pages per website. In figure 4 it is possible to view the distribution of the number of pages per website and it is noticeable that for this sample most websites have 600 or fewer pages, but the mode is 2,000 pages (number of pages when the crawler was interrupted).

Figure 4: 2K sample’s page distribution bar chart.

The table 20 provides details on the distribution of the number of pages per website in this sample.

Table 20: Distribution of the number of pages per website for the sample 2K
Page range Frequency (number of websites) Frequency (number of websites) (%) Accumulated frequency Accumulated frequency (%)
[1,25] 24 8.6% 24 8.6%
]25,50] 15 5.4% 39 14.0%
]50,75] 9 3.2% 48 17.2%
]75,100] 6 2.2% 54 19.4%
]100,125] 10 3.6% 64 22.9%
]125,150] 4 1.4% 68 24.4%
]150,175] 11 3.9% 79 28.3%
]175,200] 3 1.1% 82 29.4%
]200,225] 6 2.2% 88 31.5%
]225,250] 7 2.5% 95 34.1%
]250,275] 4 1.4% 99 35.5%
]275,300] 2 0.7% 101 36.2%
]300,325] 3 1.1% 104 37.3%
]325,350] 6 2.2% 110 39.4%
]350,375] 3 1.1% 113 40.5%
]375,400] 7 2.5% 120 43.0%
]400,425] 1 0.4% 121 43.4%
]425,450] 1 0.4% 122 43.7%
]450,475] 5 1.8% 127 45.5%
]475,500] 3 1.1% 130 46.6%
]500,525] 0 0.0% 130 46.6%
]525,550] 4 1.4% 134 48.0%
]550,575] 2 0.7% 136 48.7%
]575,600] 4 1.4% 140 50.2%
]600,625] 1 0.4% 141 50.5%
]625,650] 2 0.7% 143 51.3%
]650,675] 0 0.0% 143 51.3%
]675,700] 5 1.8% 148 53.0%
]700,725] 1 0.4% 149 53.4%
]725,750] 3 1.1% 152 54.5%
]750,775] 2 0.7% 154 55.2%
]775,800] 0 0.0% 154 55.2%
]800,825] 1 0.4% 155 55.6%
]825,850] 2 0.7% 157 56.3%
]850,875] 1 0.4% 158 56.6%
]875,900] 0 0.0% 158 56.6%
]900,925] 0 0.0% 158 56.6%
]925,950] 2 0.7% 160 57.3%
]950,975] 1 0.4% 161 57.7%
]975,1000] 1 0.4% 162 58.1%
]1000,1025] 2 0.7% 164 58.8%
]1025,1050] 2 0.7% 166 59.5%
]1050,1075] 2 0.7% 168 60.2%
]1075,1100] 1 0.4% 169 60.6%
]1100,1125] 0 0.0% 169 60.6%
]1125,1150] 0 0.0% 169 60.6%
]1150,1175] 1 0.4% 170 60.9%
]1175,1200] 2 0.7% 172 61.6%
]1200,1225] 1 0.4% 173 62.0%
]1225,1250] 0 0.0% 173 62.0%
]1250,1275] 2 0.7% 175 62.7%
]1275,1300] 0 0.0% 175 62.7%
]1300,1325] 1 0.4% 176 63.1%
]1325,1350] 2 0.7% 178 63.8%
]1350,1375] 2 0.7% 180 64.5%
]1375,1400] 0 0.0% 180 64.5%
]1400,1425] 0 0.0% 180 64.5%
]1425,1450] 1 0.4% 181 64.9%
]1450,1475] 0 0.0% 181 64.9%
]1475,1500] 2 0.7% 183 65.6%
]1500,1525] 0 0.0% 183 65.6%
]1525,1550] 0 0.0% 183 65.6%
]1550,1575] 1 0.4% 184 65.9%
]1575,1600] 1 0.4% 185 66.3%
]1600,1625] 2 0.7% 187 67.0%
]1625,1650] 0 0.0% 187 67.0%
]1650,1675] 0 0.0% 187 67.0%
]1675,1700] 7 2.5% 194 69.5%
]1700,1725] 1 0.4% 195 69.9%
]1725,1750] 0 0.0% 195 69.9%
]1750,1775] 4 1.4% 199 71.3%
]1775,1800] 2 0.7% 201 72.0%
]1800,1825] 2 0.7% 203 72.8%
]1825,1850] 0 0.0% 203 72.8%
]1850,1875] 3 1.1% 206 73.8%
]1875,1900] 3 1.1% 209 74.9%
]1900,1925] 4 1.4% 213 76.3%
]1925,1950] 4 1.4% 217 77.8%
]1950,1975] 8 2.9% 225 80.6%
]1975,2000] 54 19.4% 279 100.0%

Of the 281 websites, 2 failed the requirement to have at least one page evaluated, and 24 failed to have at least 10 pages evaluated with 100 or more elements, so they are not considered in the analysis below. Thus, for this analysis, only 255 websites with at least 10 pages evaluated with 100 or more elements are considered. In total, in this sample, 223,331 pages were evaluated.

Clause distribution

In table 21 the number (and percentage) of websites verifying or violating each clause of EN 301 549, as well as those requiring manual validation or not applicable on any website page, is displayed.

Table 21: Compliance of websites with the tested clauses for the 2K sample
Clause EN 301 549 WCAG Compliance Level Non-conforming Needing Manual Validation Not Applicable
9.1.1.1 Non-text Content A 209 (82.0%) 0 (0%) 0 (0%)
9.1.2.1 Audio-only and Video-only (prerecorded) A 0 (0%) 55 (21.6%) 200 (78.4%)
9.1.2.2 Captions (prerecorded) A 0 (0%) 55 (21.6%) 200 (78.4%)
9.1.2.3 Audio Description or Media Alternative (prerecorded) A 0 (0%) 55 (21.6%) 200 (78.4%)
9.1.3.1 Info and Relationships A 214 (83.9%) 0 (0%) 1 (0.4%)
9.1.4.2 Audio Control A 0 (0%) 9 (3.5%) 246 (96.5%)
9.2.1.1 Keyboard A 33 (12.9%) 0 (0%) 139 (54.5%)
9.2.2.1 Timing Adjustable A 5 (2.0%) 0 (0%) 228 (89.4%)
9.2.4.1 Bypass Blocks A 0 (0%) 199 (78.0%) 0 (0%)
9.2.4.2 Page Titled A 19 (7.5%) 0 (0%) 0 (0%)
9.2.4.4 Link Purpose A 243 (95.3%) 11 (4.3%) 0 (0%)
9.2.5.3 Label in Name A 45 (17.6%) 0 (0%) 126 (49.4%)
9.3.1.1 Language of Page A 72 (28.2%) 0 (0%) 0 (0%)
9.3.3.1 Error Identification A 0 (0%) 249 (97.6%) 6 (2.4%)
9.4.1.1 Parsing A 202 (79.2%) 0 (0%) 0 (0%)
9.4.1.2 Name, Role, Value A 253 (99.2%) 1 (0.4%) 0 (0%)
9.1.2.5 Audio Description (prerecorded) AA 0 (0%) 55 (21.6%) 200 (78.4%)
9.1.3.4 Orientation AA 0 (0%) 0 (0%) 255 (100.0%)
9.1.3.5 Identify Input Purpose AA 3 (1.2%) 0 (0%) 144 (56.5%)
9.1.4.3 Contrast (Minimum) AA 242 (94.9%) 6 (2.4%) 0 (0%)
9.1.4.4 Resise text AA 77 (30.2%) 172 (67.5%) 2 (0.8%)
9.3.1.2 Language of Parts AA 3 (1.2%) 0 (0%) 132 (51.8%)

Observing the table 21 we can conclude that there was a low compliance rate of the tested clauses.

The clauses with the highest non-conforming rate were:

Distribution of Functional Performance Statements

Two analyses were carried out in relation to these statements:

The table 22 presents the results obtained in the first analysis, where only the clauses of the primary relationships were taken into account.

Table 22: Compliance of websites with functional performance statements considering primary relationships for the 2K sample
Functional Performance Statement Conforming Infringing Needing Manual Validation Not Applicable
Usage without vision (WV) 255 (100.0%) 0 (0%) 0 (0%)
Usage with limited vision (LV) 254 (99.6%) 1 (0.4%) 0 (0%)
Usage without perception of colour (WPC) 242 (94.9%) 13 (5.1%) 0 (0%)
Usage without hearing (WH) 230 (90.2%) 13 (5.1%) 0 (0%)
Usage with limited hearing (LH) 127 (49.8%) 22 (8.6%) 104 (40.8%)
Usage with limited manipulation or strength (LMS) 255 (100.0%) 0 (0%) 0 (0%)
Usage with limited reach (LR) 45 (17.6%) 0 (0%) 126 (49.4%)
Usage with limited cognition, language or learning (LC) 254 (99.6%) 1 (0.4%) 0 (0%)

Observing the table 22 we can conclude that there is a large failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statements with the lowest non-conforming rate was:

The table 23 presents the results obtained in the second analysis, in which all clauses of primary and secondary relationships were taken into account.

Table 23: Compliance of websites with functional performance statements considering primary and secondary relationships for the 2K sample
Functional Performance Statement Conforming Infringing Needing Manual Validation Not Applicable
Usage without vision (WV) 255 (100.0%) 0 (0%) 0 (0%)
Usage with limited vision (LV) 255 (100.0%) 0 (0%) 0 (0%)
Usage without perception of colour (WPC) 242 (94.9%) 13 (5.1%) 0 (0%)
Usage without hearing (WH) 235 (92.2%) 10 (3.9%) 0 (0%)
Usage with limited hearing (LH) 233 (91.4%) 12 (4.7%) 0 (0%)
Usage with no or limited vocal capability (WVC) 254 (99.6%) 1 (0.4%) 0 (0%)
Usage with limited manipulation or strength (LMS) 255 (100.0%) 0 (0%) 0 (0%)
Usage with limited reach (LR) 45 (17.6%) 0 (0%) 126 (49.4%)
Usage with limited cognition, language or learning (LC) 255 (100.0%) 0 (0%) 0 (0%)
Privacy (P) 230 (90.2%) 10 (3.9%) 0 (0%)

Observing the table 23 we can conclude that there is a high failure rate in relation to functional performance statements, when evaluating all the clauses of EN 301 549. The functional performance statement with the lowest non-conforming rate was:

Distribution of tests per website

The table 24 displays the number of websites on which each AccessMonitor test is true, as well as the type of practice (good or bad practice) that the test validates. From the analysis of table 24 we can verify that of the 12 good practices analysed, two are verified on only one website (“I identified X cases where redundant event handlers are used” and “I located in the metadata of page X elements <link> which could be used to build a navigation system”). On the other hand, of the 28 bad practices, they all occur on more than one website. The average of websites following each good practice is 56%, while the average of websites applying each bad practice is 46%.

Table 24: Number of websites where each AccessMonitor test is true in the 2K sample
Assertion Number of websites Type of practice
I noticed that all images on the page have the necessary alternative equivalent in text. 218 Good practice
I found X images on the page that do not have the necessary alternative equivalent in text. 205 Bad practice
I found X images on the page with alt=” (alt null). 240 To validate manually
I found X images on the page with alt that does not serve as an alternative equivalent. 127 Bad practice
I found X images on the page where alt has more than 100 characters. 122 To validate manually
I found that all <area> active page image maps make use of the alt attribute. 21 Good practice
I found X elements <area> No attribute alt or with alt=”. 48 Bad practice
I noticed that all graphic buttons on the page make use of the alt attribute. 32 Good practice
I‘ve located X graphic buttons on the page that doesn’t have the alt attribute. 20 Bad practice
I found X links whose content is empty. Or rather, it is composed of only one image and the image has an empty character as an alternative textual equivalent (i.e. alt=“). 207 Bad practice
I have identified X cases where the title attribute of the link element is limited to repeating the text in the link. 196 Bad practice
I found X groups of links with the same text but whose destination is different. 250 To validate manually
I noticed that the first link of the page allows us to jump to the main content. 91 To validate manually
I noticed that the first link of the page does not allow to jump directly to the main content area. 223 Bad practice
I found X links to bypass content blocks. 203 To validate manually
I found X headers on the page. 252 To validate manually
I did not find a main header <H1> on this page it’s marked. 172 Bad practice
I located X occurrences of empty lists or with <li> off the lists. 177 Bad practice
I found X elements <fieldset> off the form. 40 Bad practice
I found X elements <fieldset> No description. 76 Bad practice
I located X elements <label> which are not visible or incorrectly positioned. 135 Bad practice
I’ve found that all form checks have an accessible name. 211 Good practice
I found X form controls with no access name. 171 To validate manually
I found X cases where javascript is used to remove the focus from the field, whenever the field receives the focus. 93 Bad practice
I noticed that all forms have a button to submit the data to the server. 205 Good practice
I identified X forms without the button to submit the data to the server. 152 Bad practice
I’ve located X data tables without the <caption> element. 221 Bad practice
I found X tables without identified headers but that use the <caption> element. 32 To validate manually
I found X complex data tables — where multiple rows configure header cells — in which data cells were found to exist without the <headers> attribute. 102 Bad practice
I found X tables without marked headers. 211 Bad practice
I found X elements <iframe> No title. 197 Bad practice
I found X abbreviations in which you forgot to put the extension. 8 Bad practice
I identified X cases where the CSS specifies a line spacing of less than 1.5. 3 To validate manually
I found X cases where we use justified text via HTML. 80 Bad practice
I have identified X cases where justified text is used via CSS. 185 To validate manually
I’ve identified X cases where the font size is expressed in absolute units of measurement. 69 Bad practice
I identified X cases in the CSS where absolute units of measurement are used to define the width of the contents boxes. 96 To validate manually
I checked that there are X cases, where the units of measurement that define the width of the content elements existing in HTML are expressed in absolute values. 228 To validate manually
I checked that there is X case, where the units of measurement that define the width of the content elements existing in HTML are expressed in relative values. 123 Good practice
I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for large letter text and 4.5 to 1 for normal letter text. 242 Bad practice
I have identified X CSS rules where the color of the letter or background color is not specified. 114 To validate manually
I’ve identified X cases where redundant event handlers are used. 1 Good practice
I’ve identified X cases where redundant event handlers are not used. 98 Bad practice
I’ve identified X cases where event handlers are associated with non-interactive elements. 64 Bad practice
I asked the W3C HTML validator and found that there are no HTML errors. 255 Good practice
I found that there are no obsolete elements used for visual control of the presentation. 253 Good practice
I checked that the main language of the page is marked”X“. 232 To validate manually
I found that the X attribute is missing. 70 Bad practice
I checked that the main language of the page is incorrectly marked. “X” exists? 6 Bad practice
I noticed that this page is untitled — you lack the <title> element. 6 Bad practice
I noticed that this page has no title — the <title> element; it’s empty. 13 Bad practice
I found a title on the page and it seems right to me. 255 Good practice
It seems to me that the page title is too large — it contains X characters. 173 To validate manually
I found in the metadata of page X elements <link> which could be used to build a navigation system.. 1 Good practice
I found that the page will automatically restart through the <meta http-equiv=“refresh”> element. 4 To validate manually
I found that the page is automatically redirected through the <meta http-equiv=“redirect”> element. 1 To validate manually
I noticed that all headers on this page have an accessible name 250 Good practice

The table 25 displays the number of websites that pass, fail and require manual validation by ACT rule tested. From the analysis of table 25 we can verify that 33 ACT rules passed at least once and 28 ACT rules failed at least once in the entire 2K sample. Seven ACT rules were found that fail in more websites than those in which they pass, these being the same 7 rules in which this was found in the Home+ sample:

— Link has non-empty accessible name — Aria required context role — Aria required owned elements — Object element rendering non-text content has non-empty accessible name — iframe with negative tabindex has no interactive elements — First focusable element is link to non-repeated content — Text has enhanced contrast

If we do not consider the rule “First focusable element is link to non-repeated content” due to the above reasons, we can see the same four types of problems: lack of accessible names in HTML elements, incorrect use of ARIA, incorrect attribution of focus to iframes, and use of colors with insufficient contrast.

Table 25: Number of websites that pass, fail or need manual validation for each QualWeb ACT rule in sample 2K
ACT Rule Passing Failing Needing Manual Validation
Orientation of the page is not restricted using CSS transform property 0 0 0
iframe elements with identical accessible names have equivalent purpose 0 0 36
Button has non-empty accessible name 237 71 0
Link has non-empty accessible name 186 243 0
Element with aria-hidden has no focusable content 201 73 0
meta viewport allows for zoom 191 77 0
audio or video avoids automatically playing audio 0 0 9
id attribute value is unique 225 202 0
role attribute has valid value 215 69 0
svg element with explicit role has non-empty accessible name 34 2 0
Element with lang attribute has valid language tag 123 3 0
video element visual content has accessible alternative 0 0 55
autocomplete attribute has valid value 111 3 0
ARIA state or property is permitted 217 84 0
video element auditory content has accessible alternative 0 0 55
aria-* attribute is defined in WAI-ARIA 237 3 0
Element with role attribute has required states and properties 224 27 0
audio element content has text alternative 0 0 1
Visible label is part of accessible name 99 45 0
video element visual-only content has accessible alternative 0 0 55
video element visual content has strict accessible alternative 0 0 55
ARIA required context role 63 70 0
ARIA state or property has valid value 229 11 0
headers attribute specified on a cell refers to cells in the same table element 17 5 0
ARIA required owned elements 61 92 0
Table header cell has assigned cells 120 31 0
Zoomed text node is not clipped with CSS overflow 0 0 248
Error message describes invalid form field value 0 0 249
Object element rendering non-text content has non-empty accessible name 0 12 0
Scrollable element is keyboard accessible 98 33 0
Links with identical accessible names and context serve equivalent purpose 173 0 237
Element marked as decorative is not exposed 237 99 0
Audio or video that plays automatically has no audio that lasts more than 3 seconds 0 0 9
audio or video that plays automatically has a control mechanism 0 0 9
video element visual-only content is media alternative for text 0 0 55
video element visual-only content has description track 0 0 0
video element visual-only content has transcript 0 0 55
video element visual-only content has audio track alternative 0 0 55
video element visual content has audio description 0 0 55
video element content is media alternative for text 0 0 55
video element visual content has description track 0 0 55
audio element content has transcript 0 0 1
audio element content is media alternative for text 0 0 1
video element auditory content has captions 0 0 55
Audio and visuals of video element have transcript 0 0 55
Element in sequential focus order has visible focus 0 0 255
Document has a landmark with non-repeated content 136 0 224
Document has heading for non-repeated content 136 0 224
Element with presentational children has no focusable content 253 37 0
Menuitem has non-empty accessible name 18 6 0
Letter spacing in style attributes is not !important 128 1 0
Word spacing in style attributes is not !important 66 0 0
iframe with negative tabindex has no interactive elements 22 86 0
meta element has no refresh delay (no exception) 22 5 0
First focusable element is link to non-repeated content 0 233 22
Block of repeated content is collapsible 136 0 224
Document has an instrument to move focus to non-repeated content 136 0 224
Bypass Blocks of Repeated Content 215 0 199
Text has enhanced contrast 117 252 2
Distribution of tests per page

In the table 26 the number of pages on which each AccessMonitor test is true, as well as the type of practice the test validates. From the analysis of table 26 we can verify that all good practices and all bad practices have been detected on at least one page. Of the 11 good practices analysed, all are true on at least 5 pages. Of the 28 bad practices all occur in at least 30 pages. The average of pages that follow each good practice is 44%, while the average of pages that apply each bad practice is 18%.

The good practice applied by more pages is “I found a title on the page and it seems correct”, applied to 99.3% of the pages. The least applied good practice is “Identified X cases in which redundant event handlers are used”, applied in only 5 pages.

The bad practice applied on more pages is “I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for text with large letter and 4.5 for 1 for text with normal letter.”, verified in 72% of pages. The bad practice found in fewer pages is “I found X abbreviations in which you forgot to put the extension”, verified in 0.01% of the pages.

Table 26: Number of pages where each AccessMonitor test is true in the 2K sample
Assertion Number of pages Type of practice
————————————————————————————————————————————————————————————— ——– ———————–
I noticed that all images on the page have the necessary alternative equivalent in text. 166184 Good practice
I found X images on the page that do not have the necessary alternative equivalent in text. 53125 Bad practice
I found X images on the page with alt=” (alt null). 98620 To validate manually
I found X images on the page with alt that does not serve as an alternative equivalent. 11463 Bad practice
I found X images on the page where alt has more than 100 characters. 12035 To validate manually
I found that all <area> active page image maps make use of the alt attribute. 1376 Good practice
I found X elements <area> No attribute alt or with alt=”. 1609 Bad practice
I noticed that all graphic buttons on the page make use of the alt attribute. 27591 Good practice
I‘ve located X graphic buttons on the page that doesn’t have the alt attribute. 1155 Bad practice
I found X links whose content is empty. Or rather, it is composed of only one image and the image has an empty character as an alternative textual equivalent (i.e. alt=“). 65081 Bad practice
I have identified X cases where the title attribute of the link element is limited to repeating the text in the link. 116078 Bad practice
I found X groups of links with the same text but whose destination is different. 128555 To validate manually
I noticed that the first link of the page allows us to jump to the main content. 64891 To validate manually
I noticed that the first link of the page does not allow to jump directly to the main content area. 158442 Bad practice
I found X links to bypass content blocks. 136622 To validate manually
I found X headers on the page. 212994 To validate manually
I did not find a main header <H1> on this page it’s marked. 43060 Bad practice
I located X occurrences of empty lists or with <li> off the lists. 38817 Bad practice
I found X elements <fieldset> off the form. 10133 Bad practice
I found X elements <fieldset> No description. 8870 Bad practice
I located X elements <label> which are not visible or incorrectly positioned. 65204 Bad practice
I’ve found that all form checks have an accessible name. 120360 Good practice
I found X form controls with no access name. 33684 To validate manually
I found X cases where javascript is used to remove the focus from the field, whenever the field receives the focus. 22892 Bad practice
I noticed that all forms have a button to submit the data to the server. 134286 Good practice
I identified X forms without the button to submit the data to the server. 52573 Bad practice
I’ve located X data tables without the <caption> element. 70883 Bad practice
I found X tables without identified headers but that use the <caption> element. 2193 To validate manually
I found X complex data tables — where multiple rows configure header cells — in which data cells were found to exist without the <headers> attribute. 22437 Bad practice
I found X tables without marked headers. 61032 Bad practice
I found X elements <iframe> No title. 23862 Bad practice
I found X abbreviations in which you forgot to put the extension. 30 Bad practice
I identified X cases where the CSS specifies a line spacing of less than 1.5. 166 To validate manually
I found X cases where we use justified text via HTML. 1731 Bad practice
I have identified X cases where justified text is used via CSS. 43765 To validate manually
I’ve identified X cases where the font size is expressed in absolute units of measurement. 42197 Bad practice
I identified X cases in the CSS where absolute units of measurement are used to define the width of the contents boxes. 48951 To validate manually
I checked that there are X cases, where the units of measurement that define the width of the content elements existing in HTML are expressed in absolute values. 52796 To validate manually
I checked that there is X case, where the units of measurement that define the width of the content elements existing in HTML are expressed in relative values. 17147 Good practice
I located X color combinations whose contrast ratio is lower than the minimum contrast ratio allowed by WCAG, i.e. 3 to 1 for large letter text and 4.5 to 1 for normal letter text. 160985 Bad practice
I have identified X CSS rules where the color of the letter or background color is not specified. 39852 To validate manually
I’ve identified X cases where redundant event handlers are used. 5 Good practice
I’ve identified X cases where redundant event handlers are not used. 28517 Bad practice
I’ve identified X cases where event handlers are associated with non-interactive elements. 11737 Bad practice
I asked the W3C HTML validator and found that there are no HTML errors. 223333 Good practice
I found that there are no obsolete elements used for visual control of the presentation. 214197 Good practice
I checked that the main language of the page is marked”X“. 204711 To validate manually
I found that the X attribute is missing. 18622 Bad practice
I checked that the main language of the page is incorrectly marked. “X” exists? 482 Bad practice
I noticed that this page is untitled — you lack the <title> element. 91 Bad practice
I noticed that this page has no title — the <title> element; it’s empty. 1458 Bad practice
I found a title on the page and it seems right to me. 221784 Good practice
It seems to me that the page title is too large — it contains X characters. 15411 To validate manually
I found in the metadata of page X elements <link> which could be used to build a navigation system.. 8 Good practice
I found that the page will automatically restart through the <meta http-equiv=“refresh”>” element. 1547 To validate manually
I found that the page is automatically redirected through the <meta http-equiv=“redirect”>” element. 1 To validate manually
I noticed that all headers on this page have an accessible name 186303 Good practice

In the table 27 the number of pages that pass, fail and require manual validation by ACT rule tested is displayed. From the analysis of table 27 we can verify that 33 ACT rules passed at least once and 28 ACT rules failed at least once. Six ACT rules have been found that fail on more pages than those in which they pass:

— Link has non-empty accessible name — Visible label is part of accessible name — Object element rendering non-text content has non-empty accessible name — iframe with negative tabindex has no interactive elements — First focusable element is link to non-repeated content — Text has enhanced contrast

Continuing not to consider the rule “First focusable element is link to non-repeated content” due to the reasons presented above, we can verify that we are in the presence of three types of problems: lack of accessible names in HTML elements, incorrect attribution of focus to iframes, and use of colors with insufficient contrast.

In the table 27 the number of pages that pass, fail and require manual validation by ACT rule tested is displayed.

Table 27: Number of pages that pass, fail or need manual validation for each QualWeb ACT rule in the 2K sample
ACT Rule Passing Failing Needing Manual Validation
Orientation of the page is not restricted using CSS transform property 0 0 0
iframe elements with identical accessible names have equivalent purpose 0 0 179
Button has non-empty accessible name 151452 17888 0
Link has non-empty accessible name 100878 122392 0
Element with aria-hidden has no focusable content 106321 10044 0
meta viewport allows for zoom 153320 39199 0
audio or video avoids automatically playing audio 0 0 32
id attribute value is unique 161006 61962 0
role attribute has valid value 137089 6814 0
svg element with explicit role has non-empty accessible name 16479 2078 0
Element with lang attribute has valid language tag 22182 4 0
video element visual content has accessible alternative 0 0 1037
autocomplete attribute has valid value 28937 263 0
ARIA state or property is permitted 122192 13195 0
video element auditory content has accessible alternative 0 0 1037
aria-* attribute is defined in WAI-ARIA 169517 19 0
Element with role attribute has required states and properties 137092 452 0
audio element content has text alternative 0 0 10
Visible label is part of accessible name 16116 18510 0
video element visual-only content has accessible alternative 0 0 1037
video element visual content has strict accessible alternative 0 0 1037
ARIA required context role 10501 8967 0
ARIA state or property has valid value 135242 144 0
headers attribute specified on a cell refers to cells in the same table element 610 315 0
ARIA required owned elements 10834 7218 0
Table header cell has assigned cells 12252 1939 0
Zoomed text node is not clipped with CSS overflow 0 0 187516
Error message describes invalid form field value 0 0 195831
Object element rendering non-text content has non-empty accessible name 0 1878 0
Scrollable element is keyboard accessible 29550 2192 0
Links with identical accessible names and context serve equivalent purpose 24129 0 50732
Element marked as decorative is not exposed 125040 6245 0
Audio or video that plays automatically has no audio that lasts more than 3 seconds 0 0 32
audio or video that plays automatically has a control mechanism 0 0 32
video element visual-only content is media alternative for text 0 0 1037
video element visual-only content has description track 0 0 0
video element visual-only content has transcript 0 0 1037
video element visual-only content has audio track alternative 0 0 1037
video element visual content has audio description 0 0 1037
video element content is media alternative for text 0 0 1037
video element visual content has description track 0 0 1037
audio element content has transcript 0 0 10
audio element content is media alternative for text 0 0 10
video element auditory content has captions 0 0 1037
Audio and visuals of video element have transcript 0 0 1037
Element in sequential focus order has visible focus 0 0 223291
Document has a landmark with non-repeated content 39351 0 183982
Document has heading for non-repeated content 39351 0 183982
Element with presentational children has no focusable content 219007 4154 0
Menuitem has non-empty accessible name 7608 6078 0
Letter spacing in style attributes is not !important 22573 4 0
Word spacing in style attributes is not !important 828 0 0
iframe with negative tabindex has no interactive elements 3624 5298 0
meta element has no refresh delay (no exception) 7966 1548 0
First focusable element is link to non-repeated content 0 169562 53771
Block of repeated content is collapsible 39351 0 183982
Document has an instrument to move focus to non-repeated content 39351 0 183982
Bypass Blocks of Repeated Content 70298 0 153034
Text has enhanced contrast 11620 198836 12800
Score Distribution

In the figure 5 the distribution of scores for the 255 websites is displayed. From the graph we can observe that the range of scores from 5 to 6 (inclusive) contains the highest concentration of pages. Additionally, only 1 website, corresponding to 0.39% of the sample, has a score of 10.

Figure 5: Score distribution bar chart of sample 2K websites.

In the table 28 the distribution of scores for the 255 websites is displayed. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

Table 28: Distribution of AccessMonitor scores for the websites in the 2K sample
Score interval Frequency (page number) Frequency (page number) (%) Accumulated frequency Accumulated frequency (%)
[1,2] 0 0% 0 0%
]2,3] 4 1.6% 4 1.6%
]3,4] 9 3.5% 13 5.1%
]4,5] 43 16.9% 56 22%
]5,6] 75 29.4% 131 51.4%
]6,7] 70 27.5% 201 78.8%
]7,8] 32 12.5% 233 91.4%
]8,9] 14 5.5% 247 96.9%
]9,10] 8 3.2% 255 100%

In the figure 6 the distribution of scores of the 223,331 evaluated pages of the 255 websites is displayed. From the graph we can observe that the range of scores from 5 to 6 (inclusive) contains the highest concentration of pages. Additionally, only 44 pages, corresponding to 0.0018% of the sample, have a score of 10.

Figure 6: Bar chart of the score distribution for the sample of homepage plus linked pages.

In the table 29 the distribution of scores for the 223,331 pages of 255 websites is displayed. From the table we can observe that the 50% percentile is in the range of scores from 5 to 6 (inclusive).

Table 29: Distribution of AccessMonitor scores for the pages in the 2K sample
Score interval Frequency (page number) Frequency (page number) (%) Accumulated frequency Accumulated frequency (%)
[1,2] 0 0% 0 0%
]2,3] 4845 2.2% 4845 2.2%
]3,4] 1592 0.7% 6437 2.9%
]4,5] 36502 16.3% 42939 19.2%
]5,6] 75878 34% 118817 53.2%
]6,7] 62803 28.1% 181620 81.3%
]7,8] 23796 10.7% 205416 92%
]8,9] 9932 4.4% 215348 96.4%
]9,10] 7983 3.6% 223331 100%

Analysis of the results of the simplified website monitoring

The simplified monitoring method identified the most frequent non-compliances with the clauses of EN 301 549 that can be assessed with the automated tools that have been used. The non-compliances the stand out include:

The 2K sample, compared to the Home+ sample, shows that problems are even more frequent when extending the number of pages (the default rate for all problematic clauses grows in the 2K sample). Several EN clauses show significant growth in the rate of non-compliance when considering the 2K sample. For example, 72% of the websites in the Home+ sample do not comply with the clause “9.1.1.1 Non-text Content” while the percentage rises to 82% in the 2K sample. In clause “9.1.3.1 Info and Relationships”, the evolution is from 75% to 84%. In the “9.2.1.1 Keyboard” it increases from 8% to 13%. In “9.2.4.4 Link Purpose (In Context)” it rises from 89% to 95%. In the “9.3.1.1 Language of Page” it increases from 20% to 28%. In “9.4.1.1 Parsing” it increases from 70% to 79%. Clearly, it can be seen that the Home+ sample does not provide realistic coverage of the problems related to several clauses. This may be a result of the teams developing the websites paying more attention to the pages that are closer to the top of the hierarchy of the site because they are probably more visited. However, if this is the situation, it is not a justification for not paying the same level of attention to all pages made available on the website.

In-depth monitoring of websites

The results described in this section refer to the analysis of 24 websites. In total, 477 pages were analysed, corresponding to an average of 20 pages per website.

Clause distribution by website

In table 30, the number (and percentage) of websites verifying or violating each clause of EN 301 549 is displayed, as well as those that do not apply on any website page.

Table 30: Compliance of websites with tested clauses
Clause EN 301 549 WCAG Level Conforming Non-compliant Not applicable
9.1.1.1 Non-text Content A 0 (0%) 24 (100%) 0 (0%)
9.1.2.1 Audio-only and Video-only (Pre-recorded) A 0 (0%) 8 (33%) 16 (67%)
9.1.2.2 Captions (Pre-recorded) A 0 (0%) 7 (29%) 17 (71%)
9.1.2.3 Audio Description or Media Alternative (Pre-recorded) A 0 (0%) 6 (25%) 18 (75%)
9.1.2.5 Audio Description (Pre-recorded) AA 0 (0%) 7 (29%) 17 (71%)
9.1.3.1 Info and Relationships A 0 (0%) 24 (100%) 0 (0%)
9.1.3.2 Meaningful Sequence A 7 (29%) 17 (71%) 0 (0%)
9.1.3.3 Sensory Characteristics A 8 (33%) 2 (8%) 14 (58%)
9.1.3.4 Orientation AA 23 (96%) 1 (4%) 0 (0%)
9.1.3.5 Identify Input Purpose AA 1 (4%) 18 (75%) 5 (21%)
9.1.4.1 Use of Colour A 3 (13%) 10 (42%) 11 (46%)
9.1.4.2 Audio Control A 0 (0%) 2 (8%) 22 (92%)
9.1.4.3 Contrast (Minimum) AA 4 (17%) 20 (83%) 0 (0%)
9.1.4.4 Resise Text AA 13 (54%) 11 (46%) 0 (0%)
9.1.4.5 Images of Text AA 2 (8%) 14 (58%) 8 (33%)
9.1.4.10 Reflow AA 1 (4%) 23 (96%) 0 (0%)
9.1.4.11 Non-Text Contrast AA 3 (13%) 21 (88%) 0 (0%)
9.1.4.12 Text Spacing AA 10 (42%) 14 (58%) 0 (0%)
9.1.4.13 Content on Hover or Focus AA 7 (29%) 7 (29%) 10 (42%)
9.2.1.1 Keyboard A 6 (25%) 18 (75%) 0 (0%)
9.2.1.2 In Keyboard Trap A 23 (96%) 1 (4%) 0 (0%)
9.2.1.4 Character Key Shortcuts A 2 (8%) 1 (4%) 21 (88%)
9.2.2.1 Timing Adjustable A 1 (4%) 6 (25%) 17 (71%)
9.2.2.2 Pause, Stop, Hide A 2 (8%) 12 (50%) 10 (42%)
9.2.3.1 Three Flashes or Below A 1 (4%) 0 (0%) 23 (96%)
9.2.4.1 Bypass Blocks A 7 (29%) 15 (63%) 2 (8%)
9.2.4.2 Page Titled A 8 (33%) 16 (67%) 0 (0%)
9.2.4.3 Focus Order A 7 (29%) 17 (71%) 0 (0%)
9.2.4.4 Link Purpose (In Context) A 1 (4%) 23 (96%) 0 (0%)
9.2.4.5 Multiple Ways AA 10 (42%) 12 (50%) 2 (8%)
9.2.4.6 Headings and Labels AA 16 (67%) 8 (33%) 0 (0%)
9.2.4.7 Focus Visible AA 8 (33%) 16 (67%) 0 (0%)
9.2.5.1 Pointer Gestures A 5 (21%) 1 (4%) 18 (75%)
9.2.5.2 Pointer Cancellation A 23 (96%) 0 (0%) 1 (4%)
9.2.5.3 Label in Name A 8 (33%) 16 (67%) 0 (0%)
9.2.5.4 Motion Actuation A 0 (0%) 0 (0%) 24 (100%)
9.3.1.1 Language of Page A 11 (46%) 13 (54%) 0 (0%)
9.3.1.2 Language of Parts AA 2 (8%) 9 (38%) 13 (54%)
9.3.2.1 On Focus A 23 (96%) 1 (4%) 0 (0%)
9.3.2.2 On Input A 20 (83%) 1 (4%) 3 (13%)
9.3.2.3 Consistent Navigation AA 16 (67%) 4 (17%) 4 (17%)
9.3.2.4 Consistent Identification AA 14 (58%) 7 (29%) 3 (13%)
9.3.3.1 Error Identification A 10 (42%) 2 (8%) 12 (50%)
9.3.3.2 Labels or Instructions A 9 (38%) 12 (50%) 3 (13%)
9.3.3.3 Error Suggestion AA 7 (29%) 2 (8%) 15 (63%)
9.3.3.4 Error Prevention (Legal, Financial, Date) AA 9 (38%) 1 (4%) 14 (58%)
9.4.1.1 Parsing A 0 (0%) 24 (100%) 0 (0%)
9.4.1.2 Name, Role, Value A 1 (4%) 22 (92%) 1 (4%)
9.4.1.3 Status Messages (WCAG 2.1) AA 1 (4%) 6 (25%) 17 (71%)

In figure 7, the total number of websites per result of compliance with the clauses of EN 301 549 is presented as compliant, non-compliant and non-applicable clauses.

Figure 7: Conformity of websites with tested clauses.

According to the data set out in table 30, it can be concluded that there was a low compliance rate of the clauses.

The 11 clauses with the highest success rate are:

The remaining clauses achieved a success rate of 78% or less in the applicable websites.

The 11 clauses with the highest failure rate were:

The remaining clauses obtained a failure rate of 95% or less.

Distribution of Functional Performance Statements

Two analyses were carried out in relation to these statements:

For this analysis, all 11 functional performance statements were considered.

The table 31 presents the results obtained in the first analysis, in which only the clauses of primary relationships were taken into account.

Table 31: Compliance of websites with functional performance statements considering primary relations
Functional Performance Statement Conforming Infringing Needing Manual Validation
Use in the absence of vision 0 (0%) 24 (100%) 0 (0%)
Use with limited vision 0 (0%) 24 (100%) 0 (0%)
Use in the absence of color perception 1 (4%) 23 (96%) 0 (0%)
Use in the absence of hearing 1 (4%) 23 (96%) 0 (0%)
Use with limited hearing 1 (4%) 18 (75%) 5 (21%)
Use with limited handling and/or strength 1 (4%) 23 (96%) 0 (0%)
Use with limited range of motion 5 (21%) 19 (79%) 0 (0%)
Limitation of the risk of triggering photosensitive reactions 1 (4%) 6 (25%) 17 (71%)
Use with limited cognitive capabilities 0 (0%) 24 (100%) 0 (0%)

Observing the table 31 we can conclude that there is a large failure rate in relation to functional performance statements, when only the clauses of primary relationships are evaluated. The functional performance statements with the highest success rate were:

The remaining functional performance statements achieved a success rate of 4% or less, with 3 of them having a 0% rate.

The table 32 presents the results obtained in the first analysis, but considering the average compliance rate of the sample analysed.

Table 32: Average compliance rate with the clauses of each functional performance statement considering primary relations
Functional Performance Statement Conforming Infringing Needing Manual Validation
Use in the absence of vision 32% 41% 27%
Use with limited vision 33% 46% 21%
Use in the absence of color perception 18% 45% 37%
Use in the absence of hearing 6% 34% 60%
Use with limited hearing 6% 22% 71%
Use with limited handling capacity and/or strength 40% 36% 24%
Use with limited range of motion 37% 15% 48%
Limitation of the risk of triggering photosensitive reactions 4% 13% 83%
Use with limited cognitive capabilities 41% 35% 25%

In the figure 8 the average compliance rate with the clauses of each functional performance statement considering the primary relationships is presented, representing the current state of the analysed sample.

Figure 8: Average rate of compliance with the clauses of each functional performance statement considering primary relationships.

Observing the table 32 we can conclude that there is a large average failure rate for the functional performance statements, when only the primary relations clauses are evaluated. The functional performance statements with the highest level of average success rate were:

The remaining functional performance statements achieved an average success rate of 18% or less.

The table 33 presents the results obtained in the second analysis, in which all the clauses of primary and secondary relationships were taken into account.

Table 33: Compliance of websites with functional performance statements considering primary and secondary relations
Functional Performance Statement Conforming Infringing Needing Manual Validation
Use in the absence of vision 0 (0%) 24 (100%) 0 (0%)
Use with limited vision 0 (0%) 24 (100%) 0 (0%)
Use in the absence of color perception 1 (4%) 23 (96%) 0 (0%)
Use in the absence of hearing 0 (0%) 24 (100%) 0 (0%)
Use with limited hearing 0 (0%) 24 (100%) 0 (0%)
Use in the absence of vocal capacity 0 (0%) 24 (100%) 0 (0%)
Use with limited handling and/or strength 0 (0%) 24 (100%) 0 (0%)
Use with limited range of motion 5 (21%) 19 (79%) 0 (0%)
Limitation of the risk of triggering photosensitive reactions 1 (4%) 6 (25%) 17 (71%)
Use with limited cognitive capabilities 0 (0%) 24 (100%) 0 (0%)
Privacy 0 (0%) 24 (100%) 0 (0%)

Observing the table 33 we can conclude that there is a great failure rate for the functional performance statements when evaluating all clauses. The functional performance statements with the highest success rate were:

The remaining functional performance statements achieved a success rate of 0%.

The table 34 presents the results obtained in the second analysis, but considering the average compliance rate of the sample analysed.

Table 34: Average compliance rate with the clauses of each functional performance statement considering primary and secondary relations
Functional Performance Statement Conforming Infringing Needing Manual Validation
Use in the absence of vision 31% 39% 29%
Use with limited vision 28% 47% 24%
Use in the absence of color perception 18% 45% 37%
Use in the absence of hearing 16% 36% 48%
Use with limited hearing 9% 34% 57%
Use in the absence of vocal capacity 37% 47% 16%
Use with limited handling and/or strength 38% 36% 26%
Use with limited range of motion 29% 16% 55%
Limitation of the risk of triggering photosensitive reactions 29% 16% 55%
Use with limited cognitive capabilities 28% 38% 33%
Privacy 0% 100% 0%

In the figure 9 the average compliance rate with the clauses of each functional performance statement considering the primary and secondary relationships is presented.

Figure 9: Average rate of compliance with the clauses of each functional performance statement considering primary and secondary relationships.

Observing the table 34 we can conclude that there is a large average failure rate for the functional performance statements, when evaluating all clauses. The functional performance statements with the highest average success rate were:

The remaining functional performance statements achieved an average success rate of less than 29%, with one of them having a 0% rate.

Analysis of the results of the in-depth monitoring of websites

The in-depth monitoring method identified the most frequent non-compliances with the clauses of EN 301 549 applicable to a manual website review. From the non-compliances found, the following stand out:

In-depth monitoring of mobile applications

In the expert evaluations, a total of 144 screens were analysed (70 in Android applications, 74 in iOS applications), corresponding to an average of 9 screens per application.

Results by Clause

In table 35 the number (and percentage) of applications verifying or violating each clause is displayed, as well as those that are not applicable on any of the assessed screens or that cannot be determined.

Table 35: Compliance of mobile applications with tested clauses
EN 301 549 Clause WCAG Level Compliant Non-compliant Not Applicable
10.1.1.1 Non-text Content A 4 (25%) 12 (75%) 0 (0%)
10.1.2.1 Audio-only and Video-only (Pre-recorded) A 0 (0%) 0 (0%) 16 (100%)
10.1.2.2 Captions (Pre-recorded) A 0 (0%) 0 (0%) 16 (100%)
10.1.2.3 Audio Description or Media Alternative (Pre-recorded) A 0 (0%) 0 (0%) 16 (100%)
10.1.2.5 Audio Description (Pre-recorded) AA 0 (0%) 0 (0%) 16 (100%)
10.1.3.1 Info and Relationships A 3 (18.75%) 13 (81.25%) 0 (0%)
10.1.3.2 Meaningful Sequence A 0 (0%) 0 (0%) 0 (0%)
10.1.3.3 Sensory Characteristics A 8 (50%) 1 (6.25%) 7 (43.75%)
10.1.3.4 Orientation AA 1 (6.25%) 15 (93.75%) 0 (0%)
10.1.3.5 Identify Input Purpose AA 0 (0%) 0 (0%) 0 (0%)
10.1.4.1 Use of Colour A 10 (62.5%) 6 (37.5%) 0 (0%)
10.1.4.2 Audio Control A 0 (0%) 0 (0%) 16 (100%)
10.1.4.3 Contrast (Minimum) AA 0 (0%) 13 (81.25%) 0 (0%)
10.1.4.4 Resise Text AA 0 (0%) 16 (100%) 0 (0%)
10.1.4.5 Images of Text AA 2 (12.5%) 0 (0%) 14 (87.5%)
10.1.4.10 Reflow AA 5 (31.25%) 3 (18.75%) 0 (0%)
10.1.4.11 Non-Text Contrast AA 1 (6.25%) 12 (75%) 0 (0%)
10.1.4.12 Text Spacing AA 0 (0%) 0 (0%) 0 (0%)
10.1.4.13 Content on Hover or Focus AA 0 (0%) 0 (0%) 0 (0%)
10.2.1.1 Keyboard A 5 (31.25%) 11 (68.75%) 0 (0%)
10.2.1.2 In Keyboard Trap A 14 (87.5%) 0 (0%) 2 (12.5%)
10.2.1.4 Character Key Shortcuts A 0 (0%) 0 (0%) 0 (0%)
10.2.2.1 Timing Adjustable A 0 (0%) 0 (0%) 16 (100%)
10.2.2.2 Pause, Stop, Hide A 0 (0%) 2 (12.5%) 14 (87.5%)
10.2.3.1 Three Flashes or Below A 0 (0%) 0 (0%) 16 (100%)
10.2.4.2 Page Titled A 3 (18.75%) 12 (75%) 0 (0%)
10.2.4.3 Focus Order A 12 (75%) 4 (25%) 0 (0%)
10.2.4.4 Link Purpose (In Context) A 2 (12.5%) 14 (87.5%) 0 (0%)
10.2.4.6 Headings and Labels AA 12 (75%) 4 (25%) 0 (0%)
10.2.4.7 Focus Visible AA 12 (75%) 4 (25%) 0 (0%)
10.2.5.1 Pointer Gestures A 7 (43.75%) 7 (43.75%) 2 (12.5%)
10.2.5.2 Pointer Cancellation A 16 (100%) 0 (0%) 0 (0%)
10.2.5.3 Label in Name A 14 (87.5%) 2 (12.5%) 0 (0%)
10.2.5.4 Motion Actuation A 0 (0%) 0 (0%) 16 (100%)
10.3.1.1 Language of Page A 0 (0%) 0 (0%) 0 (0%)
10.3.1.2 Language of Parts AA 0 (0%) 0 (0%) 0 (0%)
10.3.2.1 On Focus A 16 (100%) 0 (0%) 0 (0%)
10.3.2.2 On Input A 13 (81.25%) 3 (18.75%) 0 (0%)
10.3.3.1 Error Identification A 12 (75%) 2 (12.5%) 2 (12.5%)
10.3.3.2 Labels or Instructions A 10 (62.5%) 6 (37.5%) 0 (0%)
10.3.3.3 Error Suggestion AA 11 (68.75%) 3 (18.75%) 2 (12.5%)
10.3.3.4 Error Prevention (Legal, Financial, Date) AA 2 (12.5%) 0 (0%) 14 (87.5%)
10.4.1.1 Parsing A 0 (0%) 0 (0%) 0 (0%)
10.4.1.2 Name, Role, Value A 2 (12.5%) 14 (87.5%) 0 (0%)
10.4.1.3 Status Messages (WCAG 2.1) AA 0 (0%) 0 (0%) 0 (0%)

According to the data presented in table 35, it is possible to conclude that there was a low compliance rate of the tested clauses. Only 2 clauses are conforming in the 16 applications analysed and all applications failed at least one clause. The average compliance rate of the assessed clauses is 57%.

The 9 clauses with the highest success rate are:

The remaining clauses achieved a success rate of less than 75%.

The clauses with the highest failure rate were:

The remaining clauses obtained a failure rate of 75% or less.

It can be verified that none of the applications makes use of audio feedback, so the criteria associated with audio and subtitling were not analysed. It should be noted that two of the evaluated applications were totally inaccessible by screen reader.

Analysis of results of in-depth monitoring of mobile applications

The in-depth monitoring method identified the most frequent non-compliances with the EN 301 549 clauses applicable to a manual analysis of mobile applications. It should be noted that only 59% of the clauses are compliant, many with a significant impact on the use of these applications. Mobile evaluations have been adapted from the techniques described in the Trusted Tester test methodology and benefit from the adaptations performed for website evaluation. Of the detected non-compliances, the following stand out:

Additional content

Results of the tests with users with impairments

Tests of websites

The participants of this study were all experienced users. Despite the various problems encountered and observed during the session, all tasks were successfully completed without major interventions of the evaluator present. The most common problems observed and reported by the participants were as follows:

Mobile application testing

Participants in this study are all experienced smartphone users who have been using them for more than three years. Despite the numerous problems encountered and observed during the session, with the exception of two tasks with two different users, all the others were successfully completed without the intervention of the evaluator present. The first task in question, in fact, was successfully completed from the perspective of the participant, but unfortunately the information that the application led him to believe as a result was incorrect, which could have led to serious consequences if it was being used outside the context of this study. The failure in the second task was due to a content exploration problem that all participants felt when they were told to find a specific service. The most common problems observed and reported by the participants were as follows:

Comparison of results from simplified and in-depth monitoring of websites

The in-depth monitoring sampled the five websites with the best and the five websites with the worst results in simplified monitoring. This makes it possible to compare the results of these websites in the two monitoring methodologies. For this analysis, the percentage of compliant and non-compliant clauses on each website was calculated, taking into account only the applicable clauses. The five websites with the best results in simplified monitoring are compliant with 38% of the clauses on average (best 57% and worst 24%), and are not compliant with 62% of the clauses (worst 76% and best 43%) in in-depth monitoring. The five websites with the lowest rankings in simplified monitoring are compliant with 33% of the clauses on average (best 42% and worst 17%), and are not compliant with 67% of the clauses (worst 83% and best 58%) in in-depth monitoring. This sample, although small with only 10 websites, shows that the best classified sites in the simplified monitoring are more compliant with the clauses of the EN 301 549 in in-depth monitoring. These websites are on average compliant with 5% more of the clauses than the worst ranked websites. The best result of each set of websites represents a 15% improvement in compliance for the best rated websites in simplified monitoring.

We also analysed the part of the in-depth sample that corresponds to websites that were not assessed in the simplified monitoring because they have less than 10 pages with more than 100 elements. Although some of these websites have been classified as “small” due to crawling problems, it is possible to see smaller sites being more compliant, even when compared with the websites that have been best rated in the simplified monitoring. These 10 websites showed compliance with 46% of EN clauses (best 56% and worst 40%) and non-compliance with 54% of clauses (worst 60% and best 44%).

Analysis of mobile application accessibility performance by operating system

In this section we present the results by operating system as well as a comparison between the two application versions of each public entity. In the table 36 the number (and percentage) of applications that check or breach each clause, as well as those that do not apply to any of the assessed screens or that cannot be determined, is displayed.

Table 36: Compliance with the tested EN clauses of mobile applications per operating system
EN 301 549 Clause WCAG Level Android Compliant iOS Compliant Android Non-compliant iOS Non-compliant
10.1.1.1 Non-text Content A 2 (25%) 2 (25%) 6 (75%) 6 (75%)
10.1.2.1 Audio-only and Video-only (Pre-recorded) A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.2.2 Captions (Pre-recorded) A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.2.3 Audio Description or Media Alternative (Pre-recorded) A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.2.5 Audio Description (Pre-recorded) AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.3.1 Info and Relationships A 3 (37.5%) 0 (0%) 5 (62.5%) 8 (100%)
10.1.3.2 Meaningful Sequence A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.3.3 Sensory Characteristics A 0 (0%) 8 (100%) 1 (12.5%) 0 (0%)
10.1.3.4 Orientation AA 1 (12.5%) 0 (0%) 7 (87.5%) 8 (100%)
10.1.3.5 Identify Input Purpose AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.4.1 Use of Colour A 2 (25%) 8 (100%) 6 (75%) 0 (0%)
10.1.4.2 Audio Control A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.4.3 Contrast (Minimum) AA 0 (0%) 0 (0%) 5 (62.5%) 8 (100%)
10.1.4.4 Resise Text AA 0 (0%) 0 (0%) 8 (100%) 8 (100%)
10.1.4.5 Images of Text AA 1 (12.5%) 1 (12.5%) 0 (0%) 0 (0%)
10.1.4.10 Reflow AA 5 (62.5%) 0 (0%) 3 (37.5%) 0 (0%)
10.1.4.11 Non-Text Contrast AA 1 (12.5%) 0 (0%) 4 (50%) 8 (100%)
10.1.4.12 Text Spacing AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.1.4.13 Content on Hover or Focus AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.2.1.1 Keyboard A 4 (50%) 1 (12.5%) 4 (50%) 7 (87.5%)
10.2.1.2 In Keyboard Trap A 8 (100%) 6 (75%) 0 (0%) 0 (0%)
10.2.1.4 Character Key Shortcuts A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.2.2.1 Timing Adjustable A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.2.2.2 Pause, Stop, Hide A 0 (0%) 0 (0%) 0 (0%) 2 (25%)
10.2.3.1 Three Flashes or Below A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.2.4.2 Page Titled A 2 (25%) 1 (12.5%) 6 (75%) 6 (75%)
10.2.4.3 Focus Order A 7 (87.5%) 5 (62.5%) 1 (12.5%) 3 (37.5%)
10.2.4.4 Link Purpose (In Context) A 1 (12.5%) 1 (12.5%) 7 (87.5%) 7 (87.5%)
10.2.4.6 Headings and Labels AA 4 (50%) 8 (100%) 4 (50%) 0 (0%)
10.2.4.7 Focus Visible AA 8 (100%) 4 (50%) 0 (0%) 4 (50%)
10.2.5.1 Pointer Gestures A 3 (37.5%) 4 (50%) 4 (50%) 3 (37.5%)
10.2.5.2 Pointer Cancellation A 8 (100%) 8 (100%) 0 (0%) 0 (0%)
10.2.5.3 Label in Name A 7 (87.5%) 7 (87.5%) 1 (12.5%) 1 (12.5%)
10.2.5.4 Motion Actuation A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.3.1.1 Language of Page A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.3.1.2 Language of Parts AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.3.2.1 On Focus A 8 (100%) 8 (100%) 0 (0%) 0 (0%)
10.3.2.2 On Input A 7 (87.5%) 6 (75%) 1 (12.5%) 2 (25%)
10.3.3.1 Error Identification A 5 (62.5%) 7 (87.5%) 2 (25%) 0 (0%)
10.3.3.2 Labels or Instructions A 4 (50%) 6 (75%) 4 (50%) 2 (25%)
10.3.3.3 Error Suggestion AA 5 (62.5%) 6 (75%) 2 (25%) 1 (12.5%)
10.3.3.4 Error Prevention (Legal, Financial, Date) AA 0 (0%) 2 (25%) 0 (0%) 0 (0%)
10.4.1.1 Parsing A 0 (0%) 0 (0%) 0 (0%) 0 (0%)
10.4.1.2 Name, Role, Value A 1 (12.5%) 1 (12.5%) 7 (87.5%) 7 (87.5%)
10.4.1.3 Status Messages (WCAG 2.1) AA 0 (0%) 0 (0%) 0 (0%) 0 (0%)

According to the data presented in the table 36, it can be concluded that there was a low compliance rate with the EN 301 549 clauses in both operating systems. Only four of the clauses assessed achieved 100% compliance in both operating systems. On Android the average compliance rate of the evaluated clauses is 50%, with only five clauses in full compliance (or not present), and in iOS 58% with only nine clauses in full compliance (or not present):

Compliant clauses on Android and iOS

Compliant clauses only on Android

Compliant clauses only on iOS

The five clauses with the greatest discrepancy between operating systems are:

The eight clauses with the highest failure rate were:

On Android and iOS

IOS only

The remaining clauses obtained a failure rate of 75% or less.

In general, iOS applications when meeting the criteria tend to all comply, possibly due to obligations imposed by the operating system in the development of applications. However, similarly, when one application does not meet one criterion, the rest also tend not to do so. Android applications have a higher variance in meeting the criteria and a lower average compliance than iOS (50% and 58% respectively).

Information on accessibility statements

On the 281 websites analysed, accessibility statements were found in only 13, corresponding to 4.6% of the sample. 29 other websites were identified with an accessibility page but which did not correspond to a statement built in accordance with the model proposed in Decree-Law No 83/2018 of 19 October 2018 transposing Directive (EU) 2016/2102 of the European Parliament and of the Council on the accessibility of websites and mobile applications.

The 13 accessibility statements reported the following levels of compliance:

The average age (at 31 December 2021) of the identified accessibility statements is 13.9 months, with the most recent statement just under 4 months and the oldest approximately 2 years and 5 months. Taking this into account, it is normal that the websites of several of these accessibility statements have already changed since the date of publication of the statement. Nevertheless, the score given by AccessMonitor to each of the websites was analysed and the following averages were obtained for each level of compliance reported:

Use of the enforcement procedure and end-user feedback

In Portugal, the entity responsible for receiving and collecting notifications from public sector bodies in breach of accessibility requirements is the Instituto Nacional para a Reabilitação, IP (INR). The INR provides a form to report discrimination situations by forwarding complaints to the competent authorities.

The information provided by the INR on complaints received for the year 2021 was as follows:

Content related to additional measures

The following stakeholders were consulted for the monitoring and preparation of this report:

Annexes

Sample of websites for simplified monitoring

Table 37: List of sampled websites for the simplified monitoring method
Entity Website Type of Service
Administração Regional de Saúde de Lisboa e Vale do Tejo, I.P. https://www.arslvt.min-saude.pt Central Administration
Administração Regional de Saúde do Alentejo, I.P http://www.arsalentejo.min-saude.pt Central Administration
Administração Regional de Saúde do Algarve, I.P http://www.arsalgarve.min-saude.pt Central Administration
Administração Regional de Saúde do Centro, I.P http://www.arscentro.min-saude.pt Central Administration
Administração Regional de Saúde do Norte, I.P http://www.arsnorte.min-saude.pt Central Administration
Agência para a Competitividade e Inovação, I.P. https://www.iapmei.pt Central Administration
Agência para a Modernização Administrativa, I.P. https://www.ama.gov.pt Central Administration
Alto Comissariado para as Migrações, I.P. https://www.acm.gov.pt Central Administration
Biblioteca Nacional de Portugal http://www.bnportugal.gov.pt Central Administration
INA - Instituto Nacional de Administração, I.P. http://www.ina.gov.pt/ Central Administration
Direção-Geral da Saúde https://www.dgs.pt Central Administration
Direção-Geral das Artes https://www.dgartes.gov.pt Central Administration
Direção-Geral das Autarquias Locais http://www.portalautarquico.pt Central Administration
Direção-Geral de Estatísticas da Educação e Ciência https://www.dgeec.mec.pt/np4/home Central Administration
Direção-Geral do Consumidor https://www.consumidor.gov.pt Central Administration
Direção-Geral do Emprego e das Relações de Trabalho https://www.dgert.gov.pt Central Administration
Direção-Geral do Higher Education https://www.dges.gov.pt Central Administration
Direção-Geral do Livro, dos Arquivos e das Bibliotecas https://dglab.gov.pt Central Administration
Direção-Geral do Património http://www.patrimoniocultural.gov.pt Central Administration
Direção-Geral do Território https://www.dgterritorio.gov.pt Central Administration
Fundação para a Ciência e a Tecnologia, I.P https://www.fct.pt Central Administration
Infarmed - Autoridade Nacional do Medicamento e Produtos de Saúde, I.P. http://www.infarmed.pt Central Administration
Instituto da Conservação da Natureza e das Florestas, I.P. https://www.icnf.pt Central Administration
Instituto da Habitação e da Reabilitação Urbana, I.P. https://www.portaldahabitacao.pt Central Administration
Instituto de Financiamento da Agricultura e Pescas, I.P. https://www.ifap.pt Central Administration
Instituto Nacional para a Reabilitação, I.P. https://www.inr.pt/inicio Central Administration
Instituto Português do Sangue e da Transplantação, IP http://www.ipst.pt/ Central Administration
Programa SIMPLEX https://www.simplex.gov.pt Central Administration
Serviços Partilhados do Ministério da Saúde, E.P.E. https://spms.min-saude.pt/ Central Administration
Turismo de Portugal, I.P. http://www.turismodeportugal.pt Central Administration
Academia de Música de Santa Cecília https://www.am-santacecilia.pt Basic and Secondary Education
Colégio Arautos do Evangelho https://www.colegioarautos.net Basic and Secondary Education
Colégio D. Diogo de Sousa https://cdds.pt/~wp/ Basic and Secondary Education
Colégio Efanor https://www.colegioefanor.pt/pt/ Basic and Secondary Education
Colégio Nossa Senhora do Rosário https://www.colegiodorosario.pt Basic and Secondary Education
Escola Básica e Secundária de Arga e Lima https://www.agescolasargaelima.pt/ Basic and Secondary Education
Escola Básica e Secundária de Vila Cova http://www.aevc.edu.pt Basic and Secondary Education
Escola Básica e Secundária Dr. Machado de Matos https://aemachadodematos.pt/agrupamento/ Basic and Secondary Education
Escola Básica e Secundária Dr. Manuel Ribeiro Ferreira http://agalvaiazere.ccems.pt Basic and Secondary Education
Escola Básica e Secundária Henrique Sommer http://aehenriquesommer.ccems.pt Basic and Secondary Education
Escola Secundária Adolfo Portela http://www.esap.edu.pt Basic and Secondary Education
Escola Secundária Carlos Amarante https://aecarlosamarante.pt Basic and Secondary Education
Escola Secundária de Penafiel http://www.espenafiel.org Basic and Secondary Education
Escola Secundária do Restelo https://www.aerestelo.pt Basic and Secondary Education
Escola Secundária Dr. Joaquim Gomes Ferreira Alves https://www.esdjgfa.org Basic and Secondary Education
Escola Secundária Infanta D. Maria https://www.esidm.pt Basic and Secondary Education
Escola Secundária João Silva Correia http://www.aejsc.pt Basic and Secondary Education
Escola Secundária Manuel da Fonseca https://www.aesc.edu.pt Basic and Secondary Education
Escola Secundária Quinta das Palmeiras https://agrupamento.espjs.edu.pt Basic and Secondary Education
Escola Secundária Tomaz Pelayo https://home.tomazpelayo.com Basic and Secondary Education
Instituto Piaget https://ipiaget.org Higher Education
Instituto Politécnico da Guarda http://www.ipg.pt Higher Education
Instituto Politécnico de Beja https://www.ipbeja.pt/Paginas/default.aspx Higher Education
Instituto Politécnico de Leiria https://www.ipleiria.pt Higher Education
Instituto Politécnico do Porto https://www.ipp.pt Higher Education
Técnico Lisboa https://tecnico.ulisboa.pt Higher Education
Universidade Aberta https://portal.uab.pt Higher Education
Universidade Autónoma de Lisboa https://autonoma.pt Higher Education
Universidade Católica Portuguesa https://www.ucp.pt Higher Education
Universidade da Madeira https://www.uma.pt Higher Education
Universidade de Coimbra https://www.uc.pt Higher Education
Universidade de Évora https://www.uevora.pt Higher Education
Universidade de Lisboa https://www.ulisboa.pt Higher Education
Universidade do Algarve https://www.ualg.pt Higher Education
Universidade do Minho https://www.uminho.pt Higher Education
Universidade do Porto https://sigarra.up.pt Higher Education
Universidade dos Açores https://www.uac.pt Higher Education
Universidade Lusíada - Lisboa https://www.lis.ulusiada.pt Higher Education
Universidade Lusófona de Humanidades e Tecnologia https://www.ulusofona.pt Higher Education
Universidade Nova de Lisboa https://www.unl.pt Higher Education
Centro Hospitalar de Setúbal, EPE http://www.chs.min-saude.pt Hospitals
Centro Hospitalar do Médio Tejo, EPE http://www.chmt.min-saude.pt Hospitals
Centro Hospitalar e Universitário de Coimbra, EPE https://www.chuc.min-saude.pt Hospitals
Centro Hospitalar Universitário de Lisboa Central, EPE http://www.chlc.min-saude.pt Hospitals
Centro Hospitalar Universitário de São João, EPE https://portal-chsj.min-saude.pt Hospitals
Centro Hospitalar Universitário do Algarve, EPE http://www.chualgarve.min-saude.pt Hospitals
Centro Hospitalar Universitário do Porto, EPE https://www.chporto.pt Hospitals
Hospital Beatriz Ângelo http://www.hbeatrizangelo.pt Hospitals
Hospital da Senhora da Oliveira Guimarães, EPE https://www.hospitaldeguimaraes.min-saude.pt Hospitals
Hospital de Braga, EPE https://www.hospitaldebraga.pt Hospitals
Hospital de Cascais Dr. José de Almeida https://www.hospitaldecascais.pt Hospitals
Hospital de Magalhães Lemos, EPE https://www.hmlemos.min-saude.pt Hospitals
Hospital de Vila Franca de Xira, EPE https://www.hospitalvilafrancadexira.pt Hospitals
Hospital do Espírito Santo de Évora, EPE https://www.hevora.min-saude.pt Hospitals
Hospital Garcia de Orta, EPE https://www.hgo.min-saude.pt Hospitals
Hospital Professor Doutor Fernando Fonseca, EPE https://hff.min-saude.pt Hospitals
Instituto de Oftalmologia Dr. Gama Pinto https://www.igpinto.min-saude.pt Hospitals
Instituto Português de Oncologia de Coimbra Francisco Gentil, EPE https://www.ipocoimbra.min-saude.pt Hospitals
Instituto Português de Oncologia de Lisboa Francisco Gentil, EPE https://www.ipolisboa.min-saude.pt Hospitals
Instituto Português de Oncologia do Porto Francisco Gentil, EPE https://ipoporto.pt Hospitals
Algueirão-Mem Martins https://www.jfamm.pt Neighborhood Boards
Arrifes https://www.arrifes.pt Neighborhood Boards
Braga (São Vítor) http://www.juntasvictor.pt Neighborhood Boards
Buarcos e São Julião https://www.buarcosesaojuliao.pt Neighborhood Boards
Castelo Branco https://jf-castelobranco.pt Neighborhood Boards
Fafe http://www.jf-fafe.pt Neighborhood Boards
Odivelas http://jf-odivelas.pt Neighborhood Boards
Pinhel https://freguesiadepinhel.net Neighborhood Boards
Portimão http://www.jf-portimao.pt Neighborhood Boards
Porto Santo http://www.jfportosanto.com Neighborhood Boards
Praia da Vitória (Santa Cruz) https://freguesiasantacruz.pt Neighborhood Boards
Rabo de Peixe http://www.jf-rabodepeixe.pt Neighborhood Boards
Rio Tinto https://www.riotinto.pt Neighborhood Boards
Santa Maria Maior https://www.jf-santamariamaior.pt Neighborhood Boards
Santo António http://www.jf-santoantonio.pt Neighborhood Boards
Santo António dos Olivais https://jfsao.pt Neighborhood Boards
São João da Madeira https://www.fsjm.pt Neighborhood Boards
São Martinho https://jf-saomartinho.pt Neighborhood Boards
Setúbal (São Sebastião) https://www.jfss.pt Neighborhood Boards
União das freguesias da Sé e São Lourenço http://www.junta-se-slourenco.pt Neighborhood Boards
União das freguesias de Beja (Santiago Maior e São João Baptista) https://www.ufsmaiorsjbaptista.pt Neighborhood Boards
União das freguesias de Cascais e Estoril https://jf-cascaisestoril.pt Neighborhood Boards
União das freguesias de Leiria, Pousos, Barreira e Cortes https://uf-lpbc.pt Neighborhood Boards
União das freguesias de Mafamude e Vilar do Paraíso https://www.mafamudevilarparaiso.pt Neighborhood Boards
União das freguesias de Malagueira e Horta das Figueiras https://uniaof-malagueirahfigueiras.pt Neighborhood Boards
União das freguesias de São Mamede de Infesta e Senhora da Hora https://www.uf-smish.pt Neighborhood Boards
União das freguesias de Sé, Santa Maria e Meixedo https://ufssmm.pt Neighborhood Boards
União das freguesias de Viana do Castelo (Santa Maria Maior e Monserrate) e Meadela http://santamariamaior-monserrate-meadela.com Neighborhood Boards
União de freguesias da cidade de Santarém https://www.uf-cidadesantarem.pt Neighborhood Boards
Viseu http://freguesiadeviseu.pt/portal/ Neighborhood Boards
Almada https://www.cm-almada.pt/ Municipalities
Aveiro https://www.cm-aveiro.pt Municipalities
Beja https://cm-beja.pt Municipalities
Braga https://www.cm-braga.pt Municipalities
Bragança https://www.cm-braganca.pt Municipalities
Castelo Branco https://www.cm-castelobranco.pt/ Municipalities
Campo Maior https://www.cm-campo-maior.pt/pt/ Municipalities
Coimbra https://www.cm-coimbra.pt Municipalities
Covilhã http://www.cm-covilha.pt/ Municipalities
Estremoz https://www.cm-estremoz.pt Municipalities
Évora https://www.cm-evora.pt Municipalities
Faro https://www.cm-faro.pt Municipalities
Funchal https://www.funchal.pt/pt/ Municipalities
Guarda https://www.mun-guarda.pt Municipalities
Guimarães https://www.cm-guimaraes.pt Municipalities
Leiria https://www.cm-leiria.pt Municipalities
Lisboa https://www.lisboa.pt/ Municipalities
Mirandela https://www.cm-mirandela.pt Municipalities
Mortágua https://www.cm-mortagua.pt Municipalities
Nazaré https://www.cm-nazare.pt Municipalities
Odemira https://www.cm-odemira.pt Municipalities
Oliveira do Hospital https://www.cm-oliveiradohospital.pt Municipalities
Ponta Delgada https://www.cm-pontadelgada.pt/ Municipalities
Ponte de Lima https://www.cm-pontedelima.pt Municipalities
Portalegre http://www.cm-portalegre.pt Municipalities
Portimão https://www.cm-portimao.pt Municipalities
Porto https://www.cm-porto.pt/ Municipalities
Sabrosa https://www.sabrosa.pt Municipalities
Santa Maria da Feira https://cm-feira.pt Municipalities
Santarém https://www.cm-santarem.pt Municipalities
Seia https://www.cm-seia.pt Municipalities
Setúbal https://www.mun-setubal.pt Municipalities
Torres Novas https://www.cm-torresnovas.pt Municipalities
Viana do Castelo http://www.cm-viana-castelo.pt Municipalities
Vila Real https://www.cm-vilareal.pt Municipalities
Viseu https://www.cm-viseu.pt/ Municipalities
Casa Colombo - Museu de Porto Santo http://www.museucolombo-portosanto.com/home.html Museums
Cultura Madeira - Museus https://cultura.madeira.gov.pt Museums
Museu das Flores http://www.museu-flores.azores.gov.pt Museums
Museu de Angra do Heroísmo http://museu-angra.azores.gov.pt/ Museums
Museu do Pico / Museu dos Baleeiros http://www.museu-pico.azores.gov.pt Museums
Museu Nacional da Imprensa http://www.museudaimprensa.pt Museums
Museu Nacional da Música http://www.museunacionaldamusica.gov.pt Museums
Museu Nacional de Arqueologia http://www.museunacionalarqueologia.gov.pt/ Museums
Museu Nacional de Arte Antiga http://www.museudearteantiga.pt/ Museums
Museu Nacional de Arte Contemporânea - Museu do Chiado http://museuartecontemporanea.pt/ Museums
Museu Nacional de Etnologia / Museu de Arte Popular https://mnetnologia.wordpress.com/ Museums
Museu Nacional de História Natural e da Ciência http://www.museus.ulisboa.pt Museums
Museu Nacional de Soares dos Reis http://www.museusoaresdosreis.gov.pt Museums
Museu Nacional do Azulejo http://www.museudoazulejo.pt/ Museums
Museu Nacional dos Coches http://museudoscoches.gov.pt/ Museums
Museu Nacional Ferroviário http://www.fmnf.pt/ Museums
Museu Nacional Grão Vasco http://www.museunacionalgraovasco.gov.pt/ Museums
Palácio Nacional da Ajuda http://www.palacioajuda.gov.pt Museums
Palácio Nacional da Pena https://www.parquesdesintra.pt Museums
Palácio Nacional de Mafra http://www.palaciomafra.gov.pt Museums
ACAPO - Associação de Cegos e Amblíopes de Portugal http://www.acapo.pt/ Non-Governmental Organisations
APCL Lisboa - Associação de Paralisia Cerebral de Lisboa http://www.apcl.org.pt/ Non-Governmental Organisations
APD – Associação Portuguesa de Deficientes http://www.apd.org.pt/ Non-Governmental Organisations
APELA - Associação Portuguesa de Esclerose Lateral Amiotrófica http://www.apela.pt/ Non-Governmental Organisations
APN – Associação Portuguesa de Doentes Neuromusculares http://www.apn.pt/ Non-Governmental Organisations
APPACDM Lisboa - Associação Portuguesa de Pais e Amigos do Cidadão Deficiente Mental de Lisboa http://www.appacdm-lisboa.org/ Non-Governmental Organisations
APPACDM Porto - Associação Portuguesa de Pais e Amigos do Cidadão Deficiente Mental do Porto http://www.appacdmporto.com/ Non-Governmental Organisations
APPDA Lisboa - Associação Portuguesa para a Perturbações do Desenvolvimento e Autismo http://www.appda-lisboa.org.pt/ Non-Governmental Organisations
APS – Associação Portuguesa de Surdos - Delegação de Lisboa https://apsurdos.org.pt Non-Governmental Organisations
ARCIL - Associação para Recuperação de Cidadãos Inadaptados da Lousã http://www.arcil.org Non-Governmental Organisations
CADIn - Centro de Apoio ao Desenvolvimento Infantil http://www.cadin.net/ Non-Governmental Organisations
CECD - Mira Sintra, CRL http://www.cecdmirasintra.org/ Non-Governmental Organisations
CERCILISBOA - Cooperativa de Ensino e Reabilitação de Cidadãos Inadaptados de Lisboa https://www.cercilisboa.org.pt Non-Governmental Organisations
CNOD – Confederação Nacional de Organizações de Pessoas com Deficiência https://cnod.pt Non-Governmental Organisations
Comité Paralimpico de Portugal http://www.comiteparalimpicoportugal.pt/ Non-Governmental Organisations
CRID - Centro de Reabilitação e Integração de Deficientes http://www.crid.pt/ Non-Governmental Organisations
FENACERCI - Federação Nacional das Cooperativas de Solidariedade Social http://www.fenacerci.pt/ Non-Governmental Organisations
FPAS - Federação Portuguesa das Associações de Surdos https://fpasurdos.pt Non-Governmental Organisations
FPDA - Federação Portuguesa de Autismo http://www.fpda.pt/ Non-Governmental Organisations
Pais em Rede http://paisemrede.pt/ Non-Governmental Organisations
Assembleia da República http://www.parlamento.pt Sovereignty Bodies and Independent Entities
Autoridade da Mobilidade e dos Transportes https://www.amt-autoridade.pt Sovereignty Bodies and Independent Entities
Banco de Portugal https://www.bportugal.pt Sovereignty Bodies and Independent Entities
Conselho Económico e Social https://www.ces.pt Sovereignty Bodies and Independent Entities
Entidade Reguladora da Saúde https://www.ers.pt Sovereignty Bodies and Independent Entities
Entidade Reguladora dos Serviços de Águas e Resíduos http://www.ersar.pt/pt Sovereignty Bodies and Independent Entities
Entidade Reguladora dos Serviços Energéticos https://www.erse.pt Sovereignty Bodies and Independent Entities
Entidade Reguladora para a Comunicação Social https://www.erc.pt Sovereignty Bodies and Independent Entities
Presidência da República https://www.presidencia.pt Sovereignty Bodies and Independent Entities
Procuradoria Geral da República https://www.ministeriopublico.pt Sovereignty Bodies and Independent Entities
Procuradoria-Geral Distrital de Lisboa https://www.pgdlisboa.pt/home.php Sovereignty Bodies and Independent Entities
Provedoria de Justiça https://www.provedor-jus.pt Sovereignty Bodies and Independent Entities
Supremo Tribunal Administrativo https://www.stadministrativo.pt Sovereignty Bodies and Independent Entities
Direção-geral da Administração da Justiça https://dgaj.justica.gov.pt Directorate-General
Justiça Mais Próxima 20/23 https://justicamaisproxima.justica.gov.pt Sovereignty Bodies and Independent Entities
Tribunal Constitucional http://www.tribunalconstitucional.pt Sovereignty Bodies and Independent Entities
Tribunal da Relação de Coimbra https://trc.pt Sovereignty Bodies and Independent Entities
Tribunal da Relação de Lisboa http://www.trl.mj.pt Sovereignty Bodies and Independent Entities
Tribunal da Relação do Porto https://www.trp.pt Sovereignty Bodies and Independent Entities
Tribunal de Contas https://www.tcontas.pt Sovereignty Bodies and Independent Entities
Autenticação Gov https://www.autenticacao.gov.pt Most Searched Portals and Services
Autoridade Nacional de Comunicações https://www.anacom.pt Most Searched Portals and Services
Comissão Nacional de Eleições https://www.cne.pt Most Searched Portals and Services
Diário da República https://dre.pt Most Searched Portals and Services
Direção-Geral da Educação https://www.dge.mec.pt Most Searched Portals and Services
ePortugal https://eportugal.gov.pt Most Searched Portals and Services
Instituto da Mobilidade e dos Transportes, I.P. https://www.imt-ip.pt Most Searched Portals and Services
Instituto do Emprego e Formação Profissional, I.P. https://www.iefp.pt/ Most Searched Portals and Services
Instituto Nacional de Emergência Médica, I.P. https://www.inem.pt Most Searched Portals and Services
Instituto Nacional de Estatística https://www.ine.pt Most Searched Portals and Services
Instituto Português do Mar e da Atmosfera https://www.ipma.pt Most Searched Portals and Services
Mapa de Cidadão https://mapa.eportugal.gov.pt Most Searched Portals and Services
Portal CITIUS da Justiça https://www.citius.mj.pt Most Searched Portals and Services
Portal da Justiça https://justica.gov.pt Most Searched Portals and Services
Instituto da Segurança Social, I.P. / Portal da Segurança Social https://www.seg-social.pt Most Searched Portals and Services
Portal das Comunidades https://www.portaldascomunidades.mne.pt Most Searched Portals and Services
Portal das Finanças https://www.portaldasfinancas.gov.pt Most Searched Portals and Services
Portal do Governo de Portugal https://www.portugal.gov.pt Most Searched Portals and Services
Portal do SNS https://www.sns.gov.pt Most Searched Portals and Services
Portal do SNS 24 https://www.sns24.gov.pt Most Searched Portals and Services
Portal dos Serviços de Estrangeiros e Fronteiras https://www.sef.pt Most Searched Portals and Services
Portal eFatura https://faturas.portaldasfinancas.gov.pt Most Searched Portals and Services
Portal Mais Transparência https://transparencia.gov.pt Most Searched Portals and Services
Recuperar Portugal https://recuperarportugal.gov.pt Most Searched Portals and Services
Segurança Social Direta https://app.seg-social.pt Most Searched Portals and Services
Assembleia Legislativa da Região Autónoma da Madeira https://www.alram.pt/pt Autonomous Region of Madeira
Autoridade Tributária e Assuntos Fiscais da Região Autónoma da Madeira https://at.madeira.gov.pt Autonomous Region of Madeira
Direção Regional de Educação https://www.madeira.gov.pt/dre Autonomous Region of Madeira
Direção Regional de Estatística da Madeira https://estatistica.madeira.gov.pt Autonomous Region of Madeira
Governo Regional da Região Autónoma da Madeira https://www.madeira.gov.pt Autonomous Region of Madeira
Instituto das Florestas e Conservação da Natureza, IP-RAM https://ifcn.madeira.gov.pt Autonomous Region of Madeira
Instituto de Administração da Saúde, IP-RAM https://iasaude.pt Autonomous Region of Madeira
Instituto de Emprego da Madeira, IP-RAM https://www.iem.madeira.gov.pt Autonomous Region of Madeira
Portal Madeira. Toda Sua. DG Turismo https://www.visitmadeira.pt Autonomous Region of Madeira
SDM - Sociedade de Desenvolvimento da Madeira, S.A. https://www.ibc-madeira.com Autonomous Region of Madeira
Serviço de Saúde da Região Autónoma da Madeira, E.P.E. http://www.sesaram.pt Autonomous Region of Madeira
Serviço Regional de Proteção Civil da Madeira https://www.procivmadeira.pt Autonomous Region of Madeira
Agência para a Modernização e Qualidade do Serviço ao Cidadão, I.P. https://www.riac.azores.gov.pt Autonomous Region of the Azores
Assembleia Legislativa da Região Autónoma dos Açores http://www.alra.pt Autonomous Region of the Azores
Conservatório Regional de Ponta Delgada https://crpd.edu.azores.gov.pt Autonomous Region of the Azores
Direção Regional do Turismo https://www.visitazores.com Autonomous Region of the Azores
Governo dos Açores https://portal.azores.gov.pt Autonomous Region of the Azores
Observatório do Turismo dos Açores https://otacores.com Autonomous Region of the Azores
Serviço Regional de Estatística dos Açores https://srea.azores.gov.pt Autonomous Region of the Azores
Serviço Regional de Proteção Civil e Bombeiros dos Açores https://www.prociv.azores.gov.pt Autonomous Region of the Azores
CAIXA GERAL DE DEPOSITOS S.A. https://www.cgd.pt Public Business Sector of the State
CARRISBUS-MANUTENÇÃO, REPARAÇÃO E TRANSPORTES,S.A. https://www.carris.pt Public Business Sector of the State
CP - COMBOIOS DE PORTUGAL, E.P.E https://www.cp.pt Public Business Sector of the State
EGEAC - EMPRESA DE GESTÃO DE EQUIPAMENTOS E ANIMAÇÃO CULTURAL, EM, S.A. https://egeac.pt Public Business Sector of the State
EMEL - EMPRESA MUNICIPAL DE MOBILIDADE E ESTACIONAMENTO DE LISBOA, E.M., S.A. https://www.emel.pt Public Business Sector of the State
ENATUR-EMPRESA NACIONAL DE TURISMO S.A. http://www.enatur.pt Public Business Sector of the State
FUNDAÇÃO INATEL https://www.inatel.pt Public Business Sector of the State
INFRAESTRUTURAS DE PORTUGAL, S.A. https://www.infraestruturasdeportugal.pt Public Business Sector of the State
LUSA - AGÊNCIA DE NOTÍCIAS DE PORTUGAL S.A. https://www.lusa.pt Public Business Sector of the State
METRO DO PORTO S.A. https://www.metrodoporto.pt Public Business Sector of the State
METROPOLITANO DE LISBOA, E.P.E. https://www.metrolisboa.pt Public Business Sector of the State
OPART - ORGANISMO DE PRODUÇÃO ARTÍSTICA, E.P.E. https://tnsc.pt Public Business Sector of the State
POLO CIENTÍFICO E TECNOLÓGICO DA MADEIRA, MADEIRA TECNOPOLO S.A. http://www.madeiratecnopolo.pt Public Business Sector of the State
SANTA CASA DA MISERICÓRDIA DE LISBOA - DEPARTAMENTO DE JOGOS https://www.scml.pt Public Business Sector of the State
SATA AIR AÇORES - SOCIEDADE AÇORIANA DE TRANSPORTES AÉREOS S.A. https://www.azoresairlines.pt/pt-pt Public Business Sector of the State
SERVIÇOS MUNICIPALIZADOS DE TRANSPORTES URBANOS DE COIMBRA https://www.smtuc.pt Public Business Sector of the State
STCP SERVIÇOS - TRANSPORTES URBANOS, CONSULTORIA E PARTICIPAÇÕES, UNIPESSOAL LDA https://www.stcp.pt Public Business Sector of the State
TEATRO MICAELENSE - CENTRO CULTURAL E DE CONGRESSOS S.A. https://www.teatromicaelense.pt Public Business Sector of the State
TRANSPORTES AÉREOS PORTUGUESES, S.A. https://www.flytap.com Public Business Sector of the State
TRATOLIXO - TRATAMENTO DE RESÍDUOS SÓLIDOS E.I.M. - EMPRESA INTERMUNICIPAL, S.A. https://www.tratolixo.pt Public Business Sector of the State

Sample of websites for in-depth monitoring

Table 38: List of sampled websites for the in-depth monitoring method
Entity Website Type of Service
Agrupamento de Escolas de Santiago do Cacém https://www.aesc.edu.pt Basic and Secondary Education
Associação de Paralisia Cerebral de Lisboa http://www.apcl.org.pt Non-Governmental Organisations
Autoridade da Mobilidade e dos Transportes https://www.amt-autoridade.pt Sovereign Bodies and Independent Entities
Casa Colombo - Museu de Porto Santo http://www.museucolombo-portosanto.com/home.html Museums
Centro Hospitalar e Universitário de Coimbra, EPE https://www.chuc.min-saude.pt Hospitals
CM Guarda https://www.mun-guarda.pt Municipalities
CM Mortágua https://www.cm-mortagua.pt Municipalities
CM Setúbal https://www.mun-setubal.pt Municipalities
CM Torres Ns https://www.cm-torresnovas.pt Municipalities
Comissão Nacional de Eleições https://www.cne.pt Most Searched Portals and Services
Comité Paralímpico de Portugal http://www.comiteparalimpicoportugal.pt/ Non-Governmental Organisations
Entidade Reguladora para a Comunicação Social https://www.erc.pt Sovereign Bodies and Independent Entities
ePortugal https://eportugal.gov.pt Most Searched Portals and Services
Federação Portuguesa das Associações de Surdos https://fpasurdos.pt Non-Governmental Organisations
Infraestruturas de Portugal, S.A. https://www.infraestruturasdeportugal.pt Public Sector Business of the State
Instituto Politécnico de Beja https://www.ipbeja.pt/Paginas/default.aspx Higher Education
Mapa de Cidadão https://mapa.eportugal.gov.pt Most Searched Portals and Services
Museu de Angra do Heroísmo http://museu-angra.azores.gov.pt/ Museums
Museu Nacional da Imprensa http://www.museudaimprensa.pt Museums
Museu Nacional do Azulejo http://www.museudoazulejo.gov.pt Museums
Portal das Finanças https://www.portaldasfinancas.gov.pt Most Searched Portals and Services
Portal do SNS 24 https://www.sns24.gov.pt Most Searched Portals and Services
Segurança Social Direta https://app.seg-social.pt Most Searched Portals and Services
União de freguesias da cidade de Santarém https://www.uf-cidadesantarem.pt Neighborhood Boards

Sample of mobile applications for in-depth monitoring

Table 39: List of sampled mobile applications for the in-depth monitoring method
Application Operating System URL
Comboios de Portugal Android https://play.google.com/store/apps/details?id=pt.cp.mobiapp&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
Autenticação GOV Android https://play.google.com/store/apps/details?id=pt.ama.autenticacaogov&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
id.gov.pt Android https://play.google.com/store/apps/details?id=id.gov.pt&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
sigaApp Android https://play.google.com/store/apps/details?id=pt.segsocial.iies.sigaapp.prod&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
EMEL ePark Android https://play.google.com/store/apps/details?id=pt.emel.epark&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
Mapa de Cidadão Android https://play.google.com/store/apps/details?id=pt.ama.mapadocidadao&referrer=utm_source%3D42matters.com%26utm_medium%3Dapi
e-fatura Android https://play.google.com/store/apps/details?id=pt.gov.efatura.mobille.dev.app&hl=en_US&gl=US
SNS 24 Android https://play.google.com/store/apps/details?id=pt.minsaude.spms.ces&hl=pt_PT&gl=US
Comboios de Portugal iOS https://apps.apple.com/pt/app/comboios-de-portugal/id1105415627
Autenticação GOV iOS https://apps.apple.com/pt/app/autenticação-gov/id1291777170
id.gov.pt iOS https://apps.apple.com/pt/app/id-gov-pt/id1384884826
sigaApp iOS https://apps.apple.com/pt/app/sigaapp/id1127868225
EMEL ePark iOS https://apps.apple.com/pt/app/epark-emel/id909274823
Mapa de Cidadão iOS https://apps.apple.com/us/app/mapa-de-cidadão/id966526205
e-fatura iOS https://apps.apple.com/pt/app/e-fatura/id887477687
SNS 24 iOS https://apps.apple.com/pt/app/sns-24/id1192353854

Does not restrict device orientation

Identify Input Purpose

Reflow

Contrast of active components

Contrast of component states

Contrast of graphic objects

Text spacing

Content on Hover or Focus

Shortcuts

Pointers and movement

Label in Name

Status Messages