ISO/IEC 25040:2024: ISO/IEC 25040:2024 – Tutorial (about 60 minutes)

ISO/IEC 25040:2024 is the international framework for evaluating quality: it tells you how to plan, run and close a quality evaluation of an ICT product (including software), a set of data or an IT service, so that the result is fair, repeatable and useful for a decision. Development teams, quality assurance groups, buyers of software and services, IT service providers and independent test laboratories use it to turn "is this good enough?" into a structured, evidence-based judgement. After this tutorial you will be able to explain the concepts behind a quality evaluation, walk through the five activities of the evaluation process, choose the right type of evaluation for a business situation, and design a small evaluation of your own.

Free tutorial · about 60 minutes · 7 lessons · 5 diagrams · then a 25-question knowledge check and a certificate.

Course overview

Edition and status: The current edition is ISO/IEC 25040:2024, the second edition, published on 16 September 2024 by ISO and IEC through Joint Technical Committee ISO/IEC JTC 1, Subcommittee SC 7 (Software and systems engineering). Its full title is "Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Quality evaluation framework". It cancels and replaces the first edition, ISO/IEC 25040:2011, which carried the narrower title "Evaluation process" and focused on software products. The 2024 edition is published and current (ISO stage 60.60, International Standard published); no newer edition or revision project had been announced when this course was checked. Like all ISO and IEC standards it will go through periodic systematic review, so check the ISO catalogue entry for its status before relying on it in a contract. The year in the course title, 2024, is therefore the edition taught here. Verified 2026-10-29 against ISO's own catalogue data, the publisher's free preview of the standard (foreword, introduction, scope and table of contents) and national standards body listings that show the 2024 edition as the current one.

A note on copyright: the standard is a paid, copyrighted document. This tutorial explains its ideas in its own words and uses the names of its clauses, activities and concepts, which are publicly listed in the table of contents and in public overviews. It does not reproduce the standard's requirements or definitions. If you must claim conformance, write contract clauses or run an accredited evaluation, buy the standard from ISO, IEC or your national standards body.

Who it is for: test managers and testers, quality assurance and quality engineering staff, architects, product owners, procurement and vendor managers, IT service managers, data owners, auditors and anyone asked to judge whether a system, a dataset or a service is good enough to accept, buy, release or certify. No earlier contact with ISO standards is assumed.

Prerequisites: none. It helps to know roughly what a quality characteristic is (for example reliability or security). If you have taken the ISO/IEC 25010:2023 course on this site, you already have that background, but Lesson 2 recaps what you need.

Learning objectives. After this course you will be able to:

Explain what a quality evaluation is, why it needs a defined framework, and what ISO/IEC 25040:2024 covers and deliberately leaves out.
Use the core vocabulary correctly: target entity, quality evaluation criteria, rigor, quality measure, quality rating module, rating result, evaluation result and assessment.
Describe the structure of the standard and the five activities of its evaluation process reference model, with the main tasks and outputs of each.
Choose between the four types of quality evaluation for a given business situation and explain how the choice changes the criteria and the report.
Plan and run a small evaluation for a real product, dataset or service, including staffing, consensus, reporting and the secure handling of evaluation data.
Explain how evaluation results are used for acceptance and release decisions, and how product certification schemes and accredited laboratories relate to the standard.
Relate ISO/IEC 25040 to the quality models, measures and requirements standards of the SQuaRE family and to testing, life cycle and management system standards.

How to use this tutorial: read the seven lessons in order (about 45 minutes), study the five diagrams, then answer the reflection questions with one product, dataset or service from your own work in mind. The knowledge check at the end has 25 questions; you pass at 70 %.

Lesson 1 – Origins, purpose and scope

Why quality needs to be evaluated, not just tested

Every organisation eventually has to answer a quality question that matters to money, safety or reputation. A hospital must decide which of two appointment-booking services to buy. A bank must decide whether an outsourced mobile app is ready to release. A data team must decide whether a purchased customer dataset is accurate enough to drive marketing. A software vendor wants to show the market that its product is maintainable and secure. In each case someone runs tests, collects numbers and writes a report, and someone else makes a decision based on that report.

The trouble is that these reports are often hard to trust or compare. One supplier measures response time on an empty test server, another under realistic load. One evaluator rates security as "good" after a checklist review, another after a penetration test. Nobody wrote down beforehand what "good enough" meant, so the decision becomes an argument. A quality evaluation framework fixes this by agreeing, before any measurement is taken, why the evaluation is being done, what exactly is being evaluated, against which criteria, with which measures and rating rules, how thoroughly, by whom, and how the results will be reported. Testing produces evidence; evaluation turns that evidence into a defensible judgement about quality.

From ISO/IEC 14598 to SQuaRE

International standards for software product evaluation go back to the 1990s, when the ISO/IEC 9126 series described a software quality model and the ISO/IEC 14598 series described how to evaluate software products against it. In the 2000s, ISO/IEC JTC 1/SC 7 merged and modernised both into one coherent family called SQuaRE (Systems and software Quality Requirements and Evaluation), numbered ISO/IEC 25000 to 25099. Inside SQuaRE, the numbers 25040 to 25049 form the quality evaluation division. ISO/IEC 25040 is the anchor document of that division: it describes the evaluation framework that the other SQuaRE parts plug into.

The first edition, ISO/IEC 25040:2011, was titled "Evaluation process". It concentrated on software products and on one kind of evaluation: checking whether a product conforms to its quality requirements. Over the next decade the rest of SQuaRE grew. A data quality model already existed (ISO/IEC 25012), and new models followed for IT services, for quality in use and for artificial intelligence systems, while the product quality model itself was revised in 2023. The evaluation standard had to catch up.

What changed in the 2024 edition

The second edition, published in September 2024, was retitled "Quality evaluation framework" and technically revised. Based on its foreword, the main changes are:

alignment with the other SQuaRE divisions (quality management, models, measurement and requirements), so that terms and concepts match across the family;
alignment with the life cycle process standards for systems and software and with requirements engineering standards;
a wider set of things that can be evaluated: not only software, but ICT products in general, data and IT services;
four types of quality evaluation instead of just conformity to requirements (you will meet them in Lesson 3);
clearer concepts of what a quality evaluation is, and more practical guidance on planning one.

The process was also restructured. The 2011 edition began with establishing evaluation requirements and then specifying, designing, executing and concluding the evaluation. The 2024 edition keeps five activities but names them define, design, plan, execute and conclude, with a dedicated planning activity that reflects the new emphasis on practical planning.

Scope: what it covers and what it leaves out

ISO/IEC 25040:2024 provides the concepts of quality evaluation and a process reference model with requirements and recommendations, so that organisations can implement an evaluation process and improve it over time. It is meant for anyone who needs to evaluate the quality of a target: development organisations (including integrators, developers and quality assurance groups), acquirers, IT service providers and independent evaluators such as test laboratories.

Two exclusions are important. First, the standard does not give specific test methods. It will not tell you how to run a load test or a usability study; for that you use testing standards and techniques. Second, it does not tell you how to use the evaluation results once you have them, for example how to negotiate with a supplier or how to set a release policy. It stops at producing a valid, well-reported evaluation result. Keeping these limits in mind prevents a common misunderstanding: ISO/IEC 25040 is a framework for organising evaluations, not a test manual and not a decision-making policy.

Figure 1 – The SQuaRE divisions: models, measures and requirements all feed the quality evaluation division, where ISO/IEC 25040 sets the framework.

Figure 1 shows why the evaluation standard sits near the end of the chain. The management division supplies common terms. The model division says what quality means for a type of target. The measurement division says how to quantify it. The requirements division says how to state what is needed. The evaluation division brings all of this together to judge a real target and produce results that people can act on.

Lesson 2 – Key concepts and vocabulary

ISO/IEC 25040 relies on terms defined in ISO/IEC 25000 (the guide to SQuaRE) and adds a few of its own. The explanations below are plain-language paraphrases written for this course, not the official definitions.

Quality evaluation – a systematic examination of how far a target satisfies stated quality criteria, carried out with defined measures, rating rules and analysis, and producing documented results.
Target entity – the thing being evaluated. Under the 2024 edition this can be an ICT product such as a system or software product, a set of data, or an IT service. A target can contain several components that are evaluated separately.
Evaluation requester – the person or organisation that asks for the evaluation and will use its results, such as an acquirer, a product manager or a regulator. It is not always the same party as the developer.
Evaluator – the person, team or organisation that performs the evaluation. It can be internal (a quality assurance group) or independent (a third-party laboratory).
Quality model – a structured set of quality characteristics and subcharacteristics that defines what quality means for a kind of target. Examples are the product quality model of ISO/IEC 25010, the data quality model of ISO/IEC 25012 and the IT service quality model of ISO/IEC TS 25011.
Quality characteristic and subcharacteristic – named categories of quality, such as security or reliability, broken into narrower aspects such as confidentiality or availability.
Quality measure – a defined way of quantifying a characteristic or subcharacteristic, usually a formula built from simpler counts or times; the 2502n standards define many of them.
Quality measure element (QME) – a basic measured value, such as the number of failures or the total operating time, used to compute a quality measure.
Measurement source – where the data for a measure comes from: for example the target itself, its documentation, its behaviour during tests or operation, or the people who use it. The standard discusses measurement sources in clause 4 and in an informative annex.
Quality evaluation criteria – the specific quality requirements the target is judged against, expressed with characteristics, measures and target values. They come from requirements specifications, contracts, quality standards or the needs of a specific use.
Rigor – how thorough, precise and strict the evaluation must be. Higher rigor means more evidence, stricter methods and more independence, and costs more time and money.
Quality rating module – a packaged, reusable description of how to rate one aspect of quality: which measures to use, how to collect the data and how to convert measured values into ratings. Think of it as a recipe for evaluating, say, availability or maintainability.
Implementation rating module – a rating module that has been tailored so it can be applied directly to the actual target entity, with concrete tools, environments and thresholds.
Quality rating result – the outcome of applying a rating module to a target, for example "availability measured at 99.7 %, rated acceptable".
Evaluation result – the analysed combination of rating results for subcharacteristics and characteristics, showing to what degree the target meets the evaluation criteria.
Assessment – a comprehensive judgement of the target against documented criteria for a specific purpose, such as accepting a delivery or releasing a product. In the standard's terms, an assessment uses evaluation results as its input.
Evaluation report – the formal record of the evaluation: what was evaluated and why, how, by whom, with which results and limitations.

How the concepts connect

These terms form a chain from the abstract to the concrete. A quality model gives you characteristics and subcharacteristics. For each relevant subcharacteristic you choose quality measures, which are computed from quality measure elements gathered from a measurement source. A quality rating module packages the measures, the measurement method and the rating rules; once tailored to your target it becomes an implementation rating module. Applying it produces rating results. Analysing the rating results across subcharacteristics and characteristics produces evaluation results. An assessment then uses those results, together with other information such as cost or strategy, to reach a decision.

Figure 2 – From quality model to decision: measures and rating modules turn raw data into rating results, evaluation results and finally an assessment.

Evaluation versus testing, measurement and assessment

Beginners often use these words interchangeably, so it helps to separate them. Testing exercises a target to find defects or gather evidence. Measurement assigns numbers to properties, such as response time or defect density. Evaluation, in the ISO/IEC 25040 sense, uses measurement and other evidence, applies agreed rating rules and analysis, and states how well the target meets defined criteria. Assessment uses evaluation results to reach a comprehensive conclusion for a purpose, such as "accept the delivery" or "do not release yet". A single test campaign can feed an evaluation, and one evaluation can feed several assessments.

Lesson 3 – Structure of the standard and the four types of evaluation

The clauses and annexes

ISO/IEC 25040:2024 is a compact document of about 30 pages. Its table of contents, which is publicly visible in the preview, has this shape:

Clauses 1 to 3 – scope, normative references (ISO/IEC 25000, the guide to SQuaRE, and ISO/IEC 25030, the quality requirements framework) and terms and definitions.
Clause 4, Concepts of quality evaluation – what a quality evaluation is (4.1); how quality models and quality measures are used in evaluation (4.2); measurement sources (4.3); the tasks of quality evaluation (4.4); the quality rating module (4.5); and assessment using evaluation (4.6).
Clause 5, Quality evaluation process reference model – an overview (5.1) followed by five activities: define the evaluation (5.2), design the evaluation (5.3), plan the evaluation (5.4), execute the evaluation (5.5) and conclude the evaluation (5.6). Each activity is described in the same pattern: its purpose, its expected outcomes, and the activities and tasks that achieve them.
Annex A (informative) – more on measurement sources.
Annex B (informative) – examples of the four types of quality evaluation.
Annex C (informative) – how the evaluation process can be implemented through the system and software life cycle processes.

The purpose-outcomes-activities pattern in clause 5 is the same style used by the life cycle process standards ISO/IEC/IEEE 12207 and 15288. That is deliberate: it lets organisations slot the evaluation process into their existing process frameworks and assess it like any other process.

The five activities of the evaluation process

The process reference model is the heart of the standard. In plain words, the five activities and their main tasks are as follows. The task names below follow public overviews of the 2024 edition and are paraphrased.

Define the evaluation. Set the scope. Establish the purpose of the evaluation, identify all the target entities involved, define the quality evaluation criteria (which characteristics and subcharacteristics matter, with what targets), and define how rigorous the evaluation must be.
Design the evaluation. Decide how it will be done. Identify the components that need separate evaluation, select quality rating modules and tailor them to the target, decide the analysis methods that will combine rating results, and define the outputs, for example a pass or fail, a score, a profile of strengths and weaknesses, or recommendations.
Plan the evaluation. Organise the work. List the activities needed, decide staffing and schedule based on the availability of people, tools and computing resources, record an evaluation plan (scope, purpose, activities, methods, tools, organisation and resources) and get all relevant parties to agree to it.
Execute the evaluation. Do the work as planned. Apply the implementation rating modules to the target entities, record the measurement and rating results, and analyse them to derive results for subcharacteristics and characteristics.
Conclude the evaluation. Make the results usable and close properly. Review the results together with the requester, write the evaluation report (including limitations and the evaluators' qualifications), review the evaluation process itself to improve it, and dispose of evaluation data and items as the requester requires: return, archive or securely destroy them.

Figure 3 – The five activities of the ISO/IEC 25040:2024 evaluation process and the main output of each, with feedback into future evaluations.

Figure 3 shows the activities as a sequence, but real evaluations iterate. Designing the evaluation may reveal that a criterion cannot be measured with the available sources, which sends you back to redefine it. Planning may show that the required rigor is unaffordable, which forces an explicit trade-off with the requester. The feedback arrow from concluding back to defining represents the process review: each evaluation should make the next one better, especially the rating modules.

The four types of quality evaluation

The biggest conceptual addition in 2024 is the recognition that people evaluate quality for different reasons, and the reason shapes everything else. The standard distinguishes four types, illustrated with examples in Annex B:

Evaluation for suitability to a specific use – does this target fit our particular context? A typical case is an acquirer comparing candidate products or services for its own users, environment and constraints. The criteria come from the intended use.
Evaluation for qualification to a quality standard – does this target meet the quality level that a published standard or scheme demands? The criteria come from the external standard, such as a requirements standard for ready-to-use software products or a sector rule.
Evaluation for conformity checking to requirements – does this target satisfy the quality requirements that were specified for it? This is the classic acceptance or release check, and the only type the 2011 edition focused on. The criteria come from the requirements specification or the contract.
Evaluation for suitability to the market – is this target competitive for a market segment? A vendor might evaluate its product against what customers in a segment typically expect. The criteria come from market expectations and comparisons.

Figure 4 – The four types of quality evaluation in the 2024 edition and where each type takes its evaluation criteria from.

The type is chosen in the very first task, when the purpose is established. Getting it right matters. A conformity check can pass a product that is still a poor fit for a specific user group, because the requirements never mentioned that group. A suitability-to-market evaluation can show a product is competitive while it still fails one contractual threshold. Naming the type makes the meaning of the result clear to everyone who reads the report.

Lesson 4 – How it works in practice

Roles and responsibilities

Most evaluations involve at least three parties. The requester owns the purpose and will use the results. The evaluator owns the method and the quality of the evidence. The provider of the target (a developer, supplier, data provider or service provider) supplies the product, data, service, documentation and access. Other stakeholders, such as end users, security officers or regulators, contribute criteria and review results. In a small internal evaluation one person may hold several roles, but it is still worth naming them, because the standard expects the plan to be agreed by all relevant parties and the results to be reviewed together with the requester.

Figure 5 – Typical interactions between requester, evaluator and provider across the evaluation, from defining the purpose to disposing of evaluation data.

Step by step through an evaluation

Defining. Start by writing a one-paragraph statement of purpose and the evaluation type. Then list every target entity. For a modern system this often includes more than the software: the data it relies on, the cloud service it runs on, and the support service behind it. Next, set the criteria. Use the relevant quality model as a checklist so nothing important is forgotten, pick the characteristics and subcharacteristics that matter, and give each a measurable target. Where possible, derive these from existing quality requirements written with ISO/IEC 25030; if none exist, the definition activity is where you create them with the requester. Finally, agree the rigor. A useful way to think about rigor is to ask what happens if the evaluation is wrong. A wrong rating on a marketing website costs a little; a wrong rating on a medication dosing service could harm patients. Higher risk justifies more rigor: more samples, stricter environments, more independence of evaluators and more formal records.

Designing. Break the target into components that need separate evaluation, such as a mobile app, a web portal, an API and a data store. For each criterion, select a quality rating module. Some organisations maintain a library of these modules, for example one for availability based on monitored uptime, one for maintainability based on static analysis of source code, one for data completeness based on field-level checks. Tailor each module into an implementation rating module: name the tool, the environment, the sample, the formula and the rating thresholds. Then decide how results will be combined. Will you use simple pass or fail per criterion, a weighted score, or knock-out criteria that fail the whole evaluation if missed? Finally, define the outputs the requester actually needs.

Planning. Turn the design into a schedule of activities with named people, tools and machines. Check availability early: test environments, licences and access to the provider's systems are frequent bottlenecks. Record the plan, including scope, purpose, methods, tools, organisation and resources, and get it agreed by the requester, the provider and other relevant parties. This consensus step is easy to skip and expensive to miss: a provider who never agreed to the test conditions will dispute the results.

Executing. Apply each implementation rating module as designed, record every measurement and rating result with enough detail to reproduce it, and log deviations from the plan. Then analyse: combine rating results into results per subcharacteristic and characteristic, and state how far the target meets the criteria. Keep raw data separate from interpretation so reviewers can check both.

Concluding. Review the results with the requester before finalising them, to confirm they are justified and fit the intended use. Write the report: the evaluation requirements, the methods and measures, the results and analysis, limitations and constraints (for example a test environment smaller than production), and who performed the evaluation with their qualifications. Review the evaluation process itself and capture improvements, especially to rating modules. Finally, handle the evaluation data and items as agreed: return the provider's software, archive records for the agreed period and securely destroy sensitive data such as personal records used in testing.

Artefacts you will typically produce

Activity	Typical artefacts	Typical owner
Define	Evaluation purpose and type, list of target entities, evaluation criteria with targets, rigor statement	Requester with evaluator
Design	Component list, selected and tailored rating modules, analysis method, output definition	Evaluator
Plan	Evaluation plan with schedule, staffing, tools and environments; record of agreement	Evaluator, agreed by all parties
Execute	Measurement records, rating results, deviation log, analysed results	Evaluator
Conclude	Results review minutes, evaluation report, process review notes, data disposal record	Evaluator with requester

Fitting evaluation into the life cycle

Annex C of the standard shows how the evaluation process can be carried out through the system and software life cycle processes of ISO/IEC/IEEE 12207 and 15288. In practice this means you do not need a separate bureaucracy. Defining evaluation criteria fits naturally into requirements definition; designing and planning the evaluation fit into verification and validation planning; executing it uses the verification, validation and quality assurance processes; and acquirers can embed the whole thing in the acquisition process, with the evaluation plan referenced in the contract. Evaluations can also be repeated at several points: an early evaluation of a prototype, a conformity evaluation at acceptance, and periodic evaluations of a live IT service.

Lesson 5 – Assessment, certification and compliance

Is there an ISO/IEC 25040 certificate?

No organisation is "certified to ISO/IEC 25040" in the way companies are certified to ISO 9001 or ISO/IEC 27001. ISO/IEC 25040 is not a management system standard with an accredited certification scheme run by ISO. ISO itself never issues certificates. What you can do is conform to the standard: implement an evaluation process that meets its requirements, and state in an evaluation report that the evaluation followed ISO/IEC 25040. A customer or auditor can then check that the report shows the purpose, criteria, rigor, methods, results, limitations and evaluator qualifications that the framework expects.

Product evaluation and certification schemes

Independent evaluations of products do exist and can lead to certificates, but they are run by third parties, not by ISO. The best-known example built on SQuaRE is a Spanish-origin ecosystem in which a software product quality laboratory, AQCLab, became accredited in 2012 to ISO/IEC 17025 (the standard for the competence of testing and calibration laboratories) for evaluations based on the ISO/IEC 25000 series. In that model, an organisation asks the accredited laboratory to evaluate chosen characteristics such as maintainability or functional suitability; the laboratory produces an evaluation report; the organisation improves the product if needed and re-evaluates; and a certification body then reviews the laboratory report, audits the organisation and issues a product certificate that names the product, its version and the characteristics evaluated. Such certificates say something about a specific product version and specific characteristics, not about the organisation as a whole.

The lesson for beginners is to read any "ISO 25000 certified" claim carefully. Ask which product and version, which characteristics, which laboratory, under which accreditation, and when. A certificate for maintainability tells you nothing about security.

How evaluation results are expressed

The standard leaves the form of results to the design activity, so you will see several styles:

Pass or fail per criterion or overall, typical for conformity checks and acceptance.
Scores or levels, for example a rating on a scale per characteristic, typical of laboratory evaluations and market comparisons.
Profiles of strengths and weaknesses, useful when selecting between candidates.
Recommendations, useful when the requester wants to improve the target.

Whatever the form, the rating rules must be fixed before execution. Setting thresholds after seeing the numbers turns an evaluation into a negotiation.

From evaluation to assessment

Clause 4.6 links evaluation to assessment. An assessment is a broader judgement, for a purpose such as acceptance or release, that uses evaluation results as evidence. The distinction matters because evaluators and decision makers have different jobs. An evaluator can state that a product met 11 of 12 criteria and missed the availability target by 0.2 points. Whether to accept it anyway, with a penalty or a fix date, is an assessment decision for the requester, who may weigh cost, schedule and risk. Recall that ISO/IEC 25040 deliberately does not tell you how to use results; it makes sure the results are trustworthy enough to decide on.

What makes an evaluation trustworthy

Whether you are an internal team or an accredited laboratory, reviewers will look for the same qualities:

Objectivity – results depend on evidence and agreed rules, not on the evaluator's opinion of the supplier.
Repeatability – the same evaluator repeating the evaluation of the same target with the same method gets the same result.
Reproducibility – a different evaluator using the same specification gets the same result.
Impartiality – the evaluator has no interest that biases the result; independence grows with rigor.
Traceability – every result can be traced back to its criterion, measure, data and measurement source.
Transparency about limits – the report states what was not evaluated and under what constraints.

These properties are what turn a set of test logs into an evaluation that a buyer, a regulator or a court could rely on.

Lesson 6 – Worked example: a hospital group selects a booking service

The following example is fictional, and the numbers in it are invented to illustrate the method. It shows how the five activities fit together for an evaluation that covers software, data and an IT service at once.

The situation

A regional hospital group with eight sites wants a cloud-based patient appointment booking service to replace phone booking. Two suppliers, called Supplier A and Supplier B here, have passed a commercial pre-selection. The chief operating officer asks the group's quality and testing team to evaluate both and recommend one. Patients will use the service on phones; staff will use a web console; the service must import the group's existing appointment data and connect to the patient administration system.

Activity 1 – Define

The team writes the purpose: compare two candidate services for suitability to the hospital group's specific use, so that the chief operating officer can choose one. They record the evaluation type as suitability to a specific use, not market suitability, because the question is "which fits us?", not "which is best in general?".

They identify the target entities for each supplier: the booking software (patient app and staff console), the data produced when the existing appointment records are migrated, and the IT service that hosts and supports the software. Using the product quality model of ISO/IEC 25010 as a checklist, they select interaction capability (including accessibility for older patients), security (confidentiality of patient data), reliability (availability during clinic hours), compatibility (interoperability with the patient administration system) and performance efficiency (response time at the Monday-morning peak). For the migrated data they use the data quality model of ISO/IEC 25012 and pick accuracy, completeness and currentness. For the service they pick incident response time and support availability.

They agree rigor with the requester. Security and data accuracy get high rigor because errors could expose patient data or cause missed appointments: independent penetration testing and full-record comparison. Interaction capability gets medium rigor: a moderated session with representative patients. Other criteria get standard rigor based on supplier evidence checked by sampling.

Activity 2 – Design

The team splits each target into components: patient app, staff console, integration interface, migrated dataset and support service. For each criterion they select a rating module from their library and tailor it:

Criterion	Implementation rating module (summary)	Target	Rule
Availability in clinic hours	Synthetic monitoring every minute for 14 days on the supplier's trial tenant	at least 99.8 %	Knock-out
Response time at peak	Load test replaying the Monday peak profile, 95th percentile booking time	under 2 seconds	Weighted
Confidentiality	Independent penetration test of app, console and interface	no high or critical findings open	Knock-out
Accessibility and learnability	Moderated session with 12 patients over 65; task success rate	at least 90 %	Weighted
Interoperability	Exchange of 200 test appointments with the patient administration system	all exchanged correctly	Knock-out
Data accuracy and completeness	Full field-level comparison of a migrated copy against source records	at least 99.5 % of records exact	Knock-out
Support responsiveness	Ten test incidents of mixed priority raised during the trial	within contracted response times	Weighted

The analysis method is a two-stage rule: any missed knock-out criterion makes a candidate unsuitable; among the remaining candidates, a weighted score decides, with weights agreed with the requester. The defined output is a comparative profile of strengths and weaknesses per characteristic, a pass or fail on each knock-out criterion, the weighted score, and a short recommendation.

Activity 3 – Plan

The team lists the activities: set up trial tenants, configure monitoring, prepare the load profile, contract the penetration testers, recruit patients for the sessions, prepare synthetic test patients for the interface test, and run a trial migration on a de-identified copy of the appointment data. They assign people, book the load-test tool and set a six-week schedule. The plan records scope, purpose, criteria, methods, tools, staffing and resources, and states that no real patient data leaves the hospital: the migration test uses de-identified data and the interface test uses synthetic patients.

Both suppliers, the hospital's information security officer and the data protection officer review the plan. Supplier B objects that 14 days of monitoring on a trial tenant does not reflect its production platform. After discussion, both suppliers agree to run trial tenants on production infrastructure. The agreed plan is signed off by all parties.

Activity 4 – Execute

Over six weeks the team applies each rating module and records the results in a shared evaluation log. Supplier A achieves 99.93 % availability, a 95th percentile booking time of 1.4 seconds, no open high findings after one fix retest, 92 % task success with older patients, a fully correct interface exchange and 99.7 % exact migrated records. Supplier B achieves 99.95 % availability and 1.1 seconds, but the penetration test finds a high-severity access control weakness in the staff console that is not fixed within the evaluation window, and only 83 % of older patients complete the booking task without help.

In the analysis, Supplier B misses a knock-out criterion (confidentiality) and therefore is rated unsuitable for this use, even though it is faster. Supplier A passes all knock-out criteria and has a good weighted score, with one weakness: two of the ten support incidents were answered slightly late.

Activity 5 – Conclude

The team reviews the results with the chief operating officer and the information security officer. They confirm that the results are justified and fit the decision. The report states the purpose and type, target entities, criteria and rigor, methods and tools, all rating results, the analysis, the recommendation (Supplier A), and limitations: the evaluation used a trial tenant with synthetic load, accessibility was tested with 12 patients, and the security result for Supplier B reflects its state during the evaluation window only. It names the evaluators and their qualifications, including the external penetration testing firm.

In the process review, the team notes that their data-accuracy rating module needed rework because date formats differed between systems, and they update the module in their library. Finally, they ask both suppliers to delete the de-identified dataset from trial tenants, obtain written confirmation, return supplier documentation, and archive the evaluation records for the period required by hospital policy.

What the example teaches

The chief operating officer then makes the assessment: accept Supplier A, with a contract clause tightening support response times. That decision is outside ISO/IEC 25040, but it rests on an evaluation that was defined, designed, planned, executed and concluded in a way both suppliers had agreed to, which makes it far harder to dispute. Notice also how the 2024 scope helped: data and the IT service were evaluated alongside the software, rather than being forgotten because "the product" was taken to mean only the code.

Lesson 7 – Relationship to other standards and frameworks

ISO/IEC 25040 depends on its SQuaRE neighbours and works alongside testing, life cycle and management standards. The table summarises the most important relationships.

Standard	What it provides	How it relates to ISO/IEC 25040
ISO/IEC 25000 (Guide to SQuaRE)	Common terms, architecture and overview of the series	Normative reference; supplies most of the vocabulary used in evaluations
ISO/IEC 25010 (product quality model)	Nine product quality characteristics and their subcharacteristics	Supplies characteristics for evaluating ICT and software products
ISO/IEC 25012 and ISO/IEC TS 25011	Data quality model and IT service quality model	Supply characteristics when the target is data or an IT service
ISO/IEC 25019 and ISO/IEC 25059	Quality-in-use model and quality model for AI systems	Supply criteria for evaluating outcomes in use and AI-based products
ISO/IEC 2502n (for example 25023, 25024)	Quality measures and measure elements	Supply the measures that rating modules package
ISO/IEC 25030	Quality requirements framework	Normative reference; requirements become evaluation criteria
ISO/IEC 25041	Evaluation guide for developers, acquirers and independent evaluators	Companion in the evaluation division with role-specific guidance
ISO/IEC/IEEE 29119	Software testing concepts, processes, documentation and techniques	Supplies test methods, which 25040 deliberately leaves out
ISO/IEC/IEEE 12207 and 15288	Software and system life cycle processes	Host processes through which evaluations are implemented (Annex C)
ISO/IEC 17025	Competence of testing and calibration laboratories	Accreditation basis for independent evaluation laboratories
ISO 9001	Quality management system requirements	Management system that can use evaluations for quality goals and verification
ISO/IEC 330xx	Process assessment	Assesses how well processes are performed; 25040 evaluates products, data and services

Three relationships worth remembering

Models and measures are inputs, not the evaluation. Choosing ISO/IEC 25010 characteristics and ISO/IEC 25023 measures does not by itself give you an evaluation. Without a defined purpose, rigor, rating rules, an agreed plan and a report, you only have numbers. ISO/IEC 25040 supplies the missing framework.

Testing provides evidence; evaluation judges it. ISO/IEC/IEEE 29119 tells you how to organise and design tests. ISO/IEC 25040 tells you how to define what the evidence must show, how to rate it and how to report it. Many organisations run their test process under 29119 and their acceptance and selection evaluations under 25040, with the test results flowing into the rating modules.

Product evaluation is not process assessment. Process assessment standards such as the ISO/IEC 330xx series, and maturity models built on them, judge how well an organisation performs its processes. ISO/IEC 25040 judges the quality of a specific target. A mature organisation can still ship a poor product, and a strong product can come from an immature process, so the two views complement each other.

Common pitfalls and best practices

Common pitfalls

Measuring before defining. Teams start load tests and scans, then try to decide afterwards what the numbers mean. Results become arguable and incomparable.
Moving the goalposts. Rating thresholds are adjusted after results are known, usually to make a favoured candidate pass. This destroys objectivity.
Treating "the product" as only the code. Data quality and the supporting IT service are left out, although users experience all three together.
Wrong evaluation type. A conformity check is used to choose between products, or a market comparison is presented as proof of contractual compliance.
Rigor that does not match risk. Safety- or privacy-critical criteria are checked by reading supplier brochures, while low-risk features receive exhaustive testing.
No agreement on the plan. Providers are not consulted about test conditions and later reject the results.
Silent limitations. Reports omit the test environment, sample sizes or what was not covered, so readers overestimate what the results prove.
Forgetting data disposal. Real personal or confidential data used in evaluation stays on trial systems after the evaluation ends.

Best practices

Write the purpose and evaluation type in one paragraph first, and check every later decision against it.
Use the relevant SQuaRE quality model as a checklist to select criteria, and give each criterion a measurable target agreed in advance.
Set rigor per criterion according to the consequences of a wrong result, and record why.
Build a library of rating modules and tailor them per evaluation; improve them after every process review.
Fix analysis rules, such as knock-out criteria and weights, before execution and record them in the plan.
Obtain written agreement on the plan from the requester, providers and other relevant parties.
Report limitations and evaluator qualifications as carefully as the results.
Prefer synthetic or de-identified data for evaluation, and confirm return, archiving or destruction of evaluation data at the end.

Summary and key takeaways

ISO/IEC 25040:2024 is the current SQuaRE quality evaluation framework; the second edition replaced the 2011 "Evaluation process" edition and widened the scope to ICT products, data and IT services.
It provides concepts and a process reference model, but no test methods and no guidance on how to use the results once produced.
The core chain runs from quality model to measures, rating modules, rating results, evaluation results and finally an assessment decision made by the requester.
The process has five activities: define, design, plan, execute and conclude, each described by purpose, outcomes and tasks.
There are four evaluation types: suitability to a specific use, qualification to a quality standard, conformity checking to requirements and suitability to the market; the type decides where the criteria come from.
Rigor, agreed criteria, rating rules fixed in advance, consensus on the plan and honest reporting of limits make results trustworthy.
There is no ISO certificate for ISO/IEC 25040 itself; third-party schemes with ISO/IEC 17025-accredited laboratories can certify specific characteristics of specific product versions.
The standard works together with ISO/IEC 25010, 25012, 25030, the 2502n measures, ISO/IEC/IEEE 29119 for testing and ISO/IEC/IEEE 12207 and 15288 for life cycle processes.

Reflection questions

Think of a product, dataset or service your organisation recently accepted or bought. Which of the four evaluation types did that decision really need, and which type was actually performed?
For the same example, list the target entities. Did the evaluation cover the data and the supporting service, or only the software?
Pick two quality criteria that matter in your work. How much rigor would you assign to each, and what would a wrong result cost if the evaluation were mistaken?
Design one implementation rating module for a criterion you care about: name the measure, the measurement source, the tool or method, the target and the rating rule.
Your evaluation shows a supplier narrowly missed one important target. What belongs in the evaluation report, and what belongs to the requester's assessment decision?