---
title: 2nd Monitoring Period - Part I  
author: Agência para a Modernização Administrativa, I.P
---

## Part I – Description of Monitoring Activities  

### General Information  

The monitoring activities took place from February 2024 to December 2024, according to the following timeline:  

- Simplified monitoring of websites  
  - Sample definition: March 2024  
  - Sample collection: March 2024  
  - Sample evaluation: March to August 2024  
- In-depth monitoring of websites  
  - Sample definition: March 2024  
  - Sample evaluation: October to December 2024  
- In-depth monitoring of mobile applications  
  - Sample definition: April 2024  
  - Sample evaluation: December 2024  

The monitoring is conducted by the Agência para a Modernização Administrativa, I.P. (AMA). For carrying out monitoring activities and drafting this report, AMA enlisted the services of the Faculdade de Ciências da Universidade de Lisboa.  

The sample provided by AMA includes 657 websites and 33 mobile applications. The number of websites in the sample exceeds the minimum required by point 2.1 of Annex I of [Implementing Decision (EU) 2018/1524](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32018D1524), equivalent to two sites per 100,000 inhabitants plus 75 sites. Considering Portugal's resident population in 2023 is 10,639,726 (according to [INE statistics](https://www.ine.pt/xportal/xmain?xpid=INE&xpgid=ine_indicadores&indOcorrCod=0008273&contexto=pi&selTab=tab0)), the minimum required is 181 sites annually. For the three-year sample period, the minimum number is 543 sites. The initial sample was analyzed to ensure that included websites were accessible for monitoring activities and corresponded to unique domains (i.e., not redirecting to other domains in the sample). After adjustments, the final sample is presented in [Table 21](#T21) annex.  

### Sample Composition  

The website sample aims to represent the diverse services provided by public sector entities. The distribution of websites by service type, shown in [Table 1](#T01), illustrates the variety of services considered in the sample.  

table01.txt  

#### Website Sample for Simplified Monitoring  

All 657 websites were initially considered for the simplified monitoring method (see [Table 21](#T21)). This sample consists of each website's homepage and all pages linked to the homepage within the domain.  

[Table 2](#T02) provides descriptive statistics of the page sample. The analysis shows that, on average, sampled websites have 61 pages linked from their homepage.  

table02.txt  

#### Website Sample for In-Depth Monitoring  

The final sample of websites for in-depth monitoring is presented in [Table 22](#T22) in the annex. Websites for in-depth monitoring are a subset of websites used for simplified monitoring. 

#### Mobile Application Sample for In-Depth Monitoring  

The mobile applications included in the sample were selected by AMA. iOS and Android versions of the same entity's application were considered individually, analyzing both versions of 16 applications and one Android-only application, totaling 33 mobile applications.  

The final mobile application sample for in-depth monitoring is shown in [Table 23](#T23) in the annex.  

### Correlation with Standards, Technical Specifications, and Tools Used for Monitoring  

#### Methodology for Simplified Monitoring of Websites  

Pages from the 657 website sample were obtained through web crawling. The [_QualWeb Crawler_](https://github.com/qualweb/qualweb/tree/main/packages/crawler) was the primary tool used, capable of processing JavaScript-enabled pages.  

Accessibility evaluation of each sampled page was conducted using the [AccessMonitor](https://accessmonitor.acessibilidade.gov.pt) automated accessibility evaluation tool. This tool validated compliance with 37 tests based on sufficient and recommended techniques from the [Web Content Accessibility Guidelines](https://www.w3.org/TR/WCAG21/) (WCAG).  

The requirements of EN 301 549 tested by _AccessMonitor_ are listed in [Table 3](#T03), along with corresponding WCAG success criteria.  

table03.txt  

#### Methodology for In-Depth Monitoring of Websites  

For each sampled website, a representative set of pages was defined for evaluation, as specified in the [Commission Implementing Decision (EU) 2018/1524](https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32018D1524). This representative sample includes:  

- Homepage;  
- Authentication page;  
- Sitemap;  
- Contact page;  
- Help page;  
- Legal notice;  
- At least one page per service type provided by the website;  
- Other primary uses, including search functionality;  
- Accessibility statement;  
- Feedback pages related to the statement;  
- Pages with substantially distinct appearance or different content;  
- Any relevant pages;  
- All pages involved in a process.  

Each page in the representative sample was evaluated against WCAG 2.1 AA success criteria. For each page, one of the following results was provided:  

- **Conforming** – The page meets all tests for the corresponding success criterion.  
- **Violation** – The page fails one or more tests for the corresponding success criterion.  

Evaluation was conducted using the [Accessibility Insights](https://accessibilityinsights.io) tool, which covers all success criteria of WCAG 2.2, exceeding WCAG 2.1 and EN 301 549 requirements.  

Additional support was provided by the [_Color Contrast Analyzer_ (CCA)](https://www.tpgi.com/color-contrast-checker) tool.  

[Table 4](#T04) lists the EN 301 549 requirements and corresponding WCAG success criteria considered in this evaluation.  

table04.txt  

#### Methodology for In-Depth Monitoring of Mobile Applications  

Both iOS and Android versions of the same application were considered individually. Applications were installed directly from their respective app stores, using standard user accounts for evaluation, except for one case where an account could not be obtained. Screens selected for evaluation followed the methodology defined in points 2 and 3 of Annex I of the Commission Implementing Decision (EU) 2018/1524. This representative sample includes:  

- Home screen (first screen of each application);  
- Login screen;  
- Sitemap;  
- Contact screen;  
- Help screen;  
- Legal notice;  
- At least one screen per service type;  
- Other primary uses, including search functionality;  
- Accessibility statement;  
- Feedback screens related to the statement;  
- Screens with substantially distinct appearance or different content;  
- Relevant downloadable documents for each service type;  
- Any other pertinent screen;  
- If selected screens are part of a process, include all process screens.  

Manual evaluation of mobile applications was supported by the following tools and devices:  

- Talkback and VoiceOver screen readers for Android and iOS, respectively;  
- Bluetooth keyboard;  
- [_Accessibility Scanner_](https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.auditor&hl=en&gl=US);  
- [_Accessibility Inspector_](https://developer.apple.com/documentation/accessibility/accessibility-inspector);  
- [_Colour Contrast Analyser_](https://www.tpgi.com/color-contrast-checker/).  

Testing methodology for manual evaluations was based on the [_Trusted Tester_](https://section508coordinators.github.io/TrustedTester/) methodology. While the Trusted Tester is based on WCAG 2.0, additional tests were added to cover WCAG 2.1 AA success criteria as required by EN 301 549. Added tests were based on sufficient techniques from WCAG 2.1.  

The success criteria considered in this evaluation correspond to WCAG 2.1 AA compliance as outlined in EN 301 549, detailed in [Table 5](#T05).  

table05.txt  

A report for each application was recorded for future analysis. After evaluations, the results were analyzed considering the verified and violated requirements.  