Computer Science software engineering – assignment

> Consider a mobile app that you make use of on your personal phone. List and discuss 3 to 5 security risks that developers should consider when developing apps like this one.

Outline your plan addressing these issues and other issues. Need a 7-9 page APA-formatted paper with minimum of 9 peer-reviewed citations. Need proper introduction and conclusion included.

NO AI WORK!

310

C H A P T E R

What is it? The answer isn’t as easy as you
might think. You know quality when you see it,
and yet, it can be an elusive thing to define.
But for computer software, quality is some-
thing that we must define, and that’s what we’ll
do in this chapter

.

Who does it? Everyone—software engineers,
managers, all stakeholders—involved in the
software process is responsible for quality.

Why is it important? You can do it right, or you
can do it over again. If a software team
stresses quality in all software engineering ac-
tivities, it reduces the amount of rework that it
must do. That results in lower costs, and more
importantly, improved time to market.

What are the steps? To achieve high-quality
software, four activities must occur: proven
software engineering process and practice,
solid project management, comprehensive
quality control, and the presence of a quality
assurance infrastructure.

What is the work product? Software that
meets its customer’s needs, performs accu-
rately and reliably, and provides value to all
who use it.

How do I ensure that I’ve done it right? Track
quality by examining the results of all quality
control activities, and measure quality by
examining errors before delivery and defects
released to the field.

Q u i c k L o o k

cost of quality . . . . . . . . . . . . . . . . . . . . . . . . . 3

17

good enough . . . . . . . . . . . . . . . . . . . . . . . . . . 3

16

liability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .320
machine learning . . . . . . . . . . . . . . . . . . . . . . .322
management actions . . . . . . . . . . . . . . . . . . . 321
quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

quality dilemma . . . . . . . . . . . . . . . . . . . . . . . . 315
quality dimensions . . . . . . . . . . . . . . . . . . . . . 314
quality factors . . . . . . . . . . . . . . . . . . . . . . . . . 312
quantitative quality assessment . . . . . . . . . . . 315
risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .320

k e y
c o n c e p t

s

15 Quality
Concep

ts

The drumbeat for improved software quality began in earnest as software became
increasingly integrated in every facet of our lives. By the 1990s, major corpora-
tions recognized that billions of dollars each year were being wasted on software
that didn’t deliver the features and functionality that were promised. Worse, both
government and industry became increasingly concerned that a major software
fault might cripple important infrastructure, costing tens of billions more. By the
turn of the century, CIO Magazine trumpeted the headline, “Let’s Stop Wasting
$78 Billion a Year,” lamenting the fact that “American businesses spend billions
for software that doesn’t do what it’s supposed to do” [Lev01]. Sadly, at least
one survey of the state of software quality practices done in 2014 suggests that
maintenance and software evolution activities make up as much as 90 percent of
the total software development costs [Nan14]. Poor software quality caused by
a rush to release products without adequate testing continues to plague the soft-
ware industry.

CHAPTER 15 QUALITY CONCEPTS 311

Today, software quality remains an issue, but who is to blame? Customers blame
developers, arguing that sloppy practices lead to low-quality software. Developers
blame customers (and other stakeholders), arguing that irrational delivery dates and a
continuing stream of changes force them to deliver software before it has been fully
validated. Who’s right? Both—and that’s the problem. In this chapter, we consider
software quality as a concept and examine why it’s worthy of serious consideration
whenever software engineering practices are applied.

15.1 Wh at is Qua L i t y?
In his mystical book Zen and the Art of Motorcycle Maintenance, Robert Persig
[Per74] commented on the thing we call quality:

Quality . . . you know what it is, yet you don’t know what it is. But that’s self-contradictory.
But some things are better than others; that is, they have more quality. But when you try
to say what the quality is, apart from the things that have it, it all goes poof! There’s
nothing to talk about  .  .  . Obviously some things are better than others  .  .  . but what’s
the betterness?  .  .  . So round and round you go, spinning mental wheels and nowhere
finding anyplace to get traction. What the hell is Quality? What is it?

Indeed—what is it?
At a somewhat more pragmatic level, David Garvin [Gar84] of the Harvard Busi-

ness School suggests that “quality is a complex and multifaceted concept” that can
be described from five different points of view. The transcendental view argues (like
Persig) that quality is something you immediately recognize but cannot explicitly
define. The user view sees quality in terms of an end user’s specific goals. If a prod-
uct meets those goals, it exhibits quality. The manufacturer’s view defines quality in
terms of the original specification of the product. If the product conforms to the spec,
it exhibits quality. The product view suggests that quality can be tied to inherent
characteristics (e.g., functions and features) of a product. Finally, the value-based view
measures quality based on how much a customer is willing to pay for a product. In
reality, quality encompasses all these views and more.

Quality of design refers to the characteristics that designers specify for a product.
The grade of materials, tolerances, and performance specifications all contribute to
the quality of design. As higher-grade materials are used, tighter tolerances and greater
levels of performance are specified, and the design quality of a product increases if
the product is manufactured according to specifications.

In software development, quality of design encompasses the degree to which the
design meets the functions and features specified in the requirements model. Quality
of conformance focuses on the degree to which the implementation follows the design
and the resulting system meets its requirements and performance goals.

But are quality of design and quality of conformance the only issues that software
engineers must consider? Robert Glass [Gla98] argues that a more “intuitive” relation-
ship is in order:

user satisfaction = compliant product + good quality
+ delivery within budget and schedule

312 PART THREE QUALITY AND SECURITY

At the bottom line, Glass contends that quality is important, but if the user isn’t
satisfied, nothing else really matters. DeMarco [DeM98] re

info

rces this view when
he states: “A product’s quality is a function of how much it changes the world for the
better.” This view of quality contends that if a software product provides substantial
benefit to its end users, they may be willing to tolerate occasional reliability or per-
formance problems. A modern view of software quality requires attention to customer
satisfaction as well as conformance to the product requirements [Max16].

15.2 so f t Wa r e Qua L i t

y

Even the most jaded software developers will agree that high-quality software is an
important goal. But how do we define software quality? In the most general sense,
software quality can be defined as: An effective software process applied in a manner
that creates a useful product that provides measurable value for those who produce
it and those who use it.

There is little question that the preceding definition could be modified or extended
and debated endlessly. For the purposes of this book, the definition serves to empha-
size three important points:

1. An effective software process establishes the infrastructure that supports any
effort at building a high-quality software product. The management aspects of
process create the checks and balances that help avoid project chaos—a key
contributor to poor quality. Software engineering practices allow the developer
to analyze the problem and design a solid solution—both critical to building
high-quality software. Finally, umbrella activities such as change management
and technical reviews have as much to do with quality as any other part of
software engineering practice.

2. A useful product delivers the content, functions, and features that the end user
desires, but as important, it delivers these assets in a reliable, error-free way.
A useful product always satisfies those requirements that have been explicitly
stated by stakeholders. In addition, it satisfies a set of implicit requirements
(e.g., ease of use) that are expected of all high-quality software.

3. By adding value for both the producer and user of a software product, high-
quality software provides benefits for the software organization and the end-
user community. The software organization gains added value because
high-quality software requires less maintenance effort, fewer bug fixes, and
reduced customer support. This enables software engineers to spend more time
creating new applications and less on rework. The user community gains added
value because the application provides a useful capability in a way that expe-
dites some business process. The end result is (1) greater software product rev-
enue, (2) better profitability when an application supports a business process,
and/or (3) improved availability of information that is crucial for the business.

15.2.1 Quality Factors
Several software quality models and standards have been proposed in the software
engineering literature. David Garvin [Gar84] writes that quality is a multifaceted

CHAPTER 15 QUALITY CONCEPTS 313

phenomena and requires the use of multiple perspectives to assess it. McCall and
Walters [McC77] proposed a useful way to think about and organize factors affecting
software quality. Their software quality factors (shown in Figure 15.1) focus on three
software product aspects: its operation characteristics, its ability to undergo change,
and its adaptability to new environments. McCall’s quality factors provide a basis for
engineering software that provides high levels of user satisfaction by focusing on the
overall user experience delivered by the software product. This cannot be done unless
developers ensure that the requirements specification is correct and that defects are
removed early in the software development process [Max16].

The ISO 25010 quality model is the newest standard (created in 2011 and revised
in 2017).1 This standard defines two quality models. The quality in use model describes
five characteristics that are appropriate when considering using the product in a
particular context (e.g., using the product on a specific platform by a human). The
product quality model describes eight characteristics that focus on both the static and
dynamic nature of computer systems.

∙ Quality in Use Model
∙ Effectiveness. Accuracy and completeness with which users achieve goals
∙ Efficiency. Resources expended to achieve user goals completely with

desired accuracy
∙ Satisfaction. Usefulness, trust, pleasure, comfor

t

∙ Freedom from risk. Mitigation of economic, health, safety, and environ-

mental risks
∙ Context coverage. Completeness, flexibility

∙ Product Quality
∙ Functional suitability. Complete, correct, appropriat

e

∙ Performance efficiency. Timing, resource utilization, capacity
∙ Compatibility. Coexistence, interoperability

Figure 15.1
McCall’s
software
quality factors

Product Operati

on

Correctness       Reliability       Usability       Integrity       Efficiency

Te
sta

b

ilit
y  

   
Flexib

ilit
y  

   
Te

sta
bilit

y

Product
Revisi

on

Portability       Reusability       Interoperability

Product Transition

1 The ISO 25010 can be found at https://www.iso.org/standard/35733.html.

314 PART THREE QUALITY AND SECURITY

∙ Usability. Appropriateness, learnability, operability, error protection,
aesthetics, accessibility

∙ Reliability. Maturity, availability, fault tolerance, recoverability
∙ Security. Confidentiality, integrity, accountability, authenticity
∙ Maintainability. Modularity, reusability, modifiability, testability
∙ Portability. Adaptability, installability, replaceability

The addition of the quality in use model helps to emphasize the importance of
customer satisfaction in the assessment of software quality. The product quality model
points out the importance of assessing both the functional and nonfunctional require-
ments for the software product [Max16].

15.2.2 Qualitative Quality Assessment
The quality dimensions and factors presented in Section 15.2.1 focus on the complete
software product and can be used as a generic indication of the quality of an applica-
tion. Your software team can develop a set of quality characteristics and associated
questions that would probe the degree to which each factor has been satisfied.2 For
example, ISO 25010 identifies usability as an important quality factor. If you were
asked to evaluate a user interface and assess its usability, how would you proceed?

Although it’s tempting to develop quantitative measures for the quality factors noted
in Section 15.2.1, you can also create a simple checklist of attributes that provide a
solid indication that the factor is present. You might start with the sub-characteristics
suggested for usability in ISO 25010: appropriateness, learnability, operability, error
protection, aesthetics, and accessibility. You and your team might decide to create a
user questionnaire and a set of structured tasks for users to perform. You might
observe the users while they perform these tasks and have them complete the question-
naire when they finish. We will discuss usability testing in more detail in Chapter 21.

To conduct your assessment, you and your team will need to address specific,
measurable (or at least, recognizable) attributes of the interface. Your tasks might be
focused on answering the following questions:

∙ How quickly can users determine whether the software product can be used to
help them complete their task or not? (appropriateness)

∙ How long does it take users to learn how to use the system functions needed
to complete their task? (learnability)

∙ Is the user able to recall how to use system functions in subsequent testing
sessions without having to relearn them? (learnability)

∙ How long does it take users to complete tasks using the system? (operability)
∙ Does the system try to prevent users from making errors? (error protection)
∙ Does the system allow users to undo operations that may have resulted in

errors? (error protection)
∙ Do answers give favorable responses to questions about the appearance of the

user interface? (aesthetics)

2 These characteristics and questions would be addressed as part of a software review
(Chapter 16).

CHAPTER 15 QUALITY CONCEPTS 315

∙ Does the interface conform to the expectations set forth by the golden rules
from Chapter 12? (accessibility)

∙ Does the user interface conform to the accessibility checklist items required
for the intended users? (accessibility)

As the interface design is developed, the software team would review the design
prototype and ask the questions noted. If the answer to most of these questions is yes,
it is likely that the user interface exhibits high quality. A collection of questions
similar to these would be developed for each quality factor to be assessed. In the case
of usability, it is always important to observe representative users interact with the
system. For some other quality factors it may be important to test the software in
the wild (or at least in the production environment).

15.2.3 Quantitative Quality Assessment
In the preceding subsections, we have presented a set of qualitative factors for the
“measurement” of software quality. The software engineering community strives to
develop precise measures for software quality and is sometimes frustrated by the
subjective nature of the activity. Cavano and McCall [Cav78] discuss this situation:

Subjectivity and specialization .  .  . apply to determining software quality. To help solve
this problem, a more precise definition of software quality is needed as well as some
way to derive quantitative measurements of software quality for objective analysis  .  .  .
Since there is no such thing as absolute knowledge, one should not expect to measure
software quality exactly, for every measurement is partially imperfect.

Several software design defects can be detected using software metrics. The process
consists of finding code fragments that suggest the presence of things like high cou-
pling or unnecessary levels of complexity. Internal code attributes can be described
quantitatively using software metrics. Any time software metric values computed for
a code fragment fall outside the range of acceptable values, it signals the existence
of a quality problem that should be investigated [Max16].

In Chapter 23, we’ll present a set of software metrics that can be applied to the
quantitative assessment of software quality. In all cases, the metrics represent indirect
measures; that is, we never really measure quality but rather some manifestation of
quality. The complicating factor is the precise relationship between the variable that
is measured and the quality of software.

15.3 th e so f t Wa r e Qua L i t y Di L e m m a

In an interview [Ven03] published on the Web, Bertrand Meyer discusses what we
call the quality dilemma:

If you produce a software system that has terrible quality, you lose because no one will
want to buy it. If on the other hand you spend infinite time, extremely large effort, and huge
sums of money to build the absolutely perfect piece of software, then it’s going to take so
long to complete and it will be so expensive to produce that you’ll be out of business any-
way. Either you missed the market window, or you simply exhausted all your resources. So
people in industry try to get to that magical middle ground where the product is good enough
not to be rejected right away, such as during evaluation, but also not the object of so much
perfectionism and so much work that it would take too long or cost too much to complete.

316 PART THREE QUALITY AND SECURITY

It’s fine to state that software engineers should strive to produce high-quality sys-
tems. It’s even better to apply good practices in your attempt to do so. But the situ-
ation discussed by Meyer is real life and represents a dilemma for even the best
software engineering organizations. When you’re faced with the quality dilemma (and
everyone is faced with it at one time or another), try to achieve balance—enough
effort to produce acceptable quality without burying the project.

15.3.1 “Good Enough” Software
Stated bluntly, if we are to accept the argument made by Meyer, is it acceptable to
produce “good enough” software? The answer to this question must be yes, because
software companies do it every day [Rod17]. They create software with known bugs
and deliver it to a broad population of end users. They recognize that some of the
functions and features delivered in version 1.0 may not be of the highest quality and
plan for improvements in version 2.0. They do this knowing that some customers will
complain, but they recognize that time to market may trump better quality as long as
the delivered product is “good enough.”

Exactly what is “good enough”? Good enough software delivers high-quality func-
tions and features that end users desire, but at the same time it delivers other more
obscure or specialized functions and features that contain known bugs. The software
vendor hopes that the vast majority of end users will overlook the bugs because they
are so happy with other application functionality.

This idea may resonate with many readers. If you’re one of them, we can only ask
you to consider some of the arguments against “good enough.” It is true that “good
enough” may work in some application domains and for a few major software com-
panies. After all, if a company has a large marketing budget and can convince enough
people to buy version 1.0, it has succeeded in locking them in. As we noted earlier,
it can argue that it will improve quality in subsequent versions. By delivering a good
enough version 1.0, it has cornered the market.

If you work for a small company, be wary of this philosophy. When you deliver a
good enough (buggy) product, you risk permanent damage to your company’s reputa-
tion. You may never get a chance to deliver version 2.0 because bad buzz may cause
your sales to plummet and your company to fold.

If you work in certain application domains (e.g., real-time embedded software)
or build application software that is integrated with hardware (e.g., automotive
software, telecommunications software), delivering software with known bugs can
be negligent and open your company to expensive litigation. In some cases, it can
even be criminal. No one wants good enough aircraft avionics software! Ebert
writes that the software process model should provide clear criteria to guide devel-
opers to determine what is really “good enough” for the intended application
domain [Ebe14].

So, proceed with caution if you believe that “good enough” is a shortcut that can
solve your software quality problems. It can work, but only for a few and only in a
limited set of application domains.3

3 A worthwhile discussion of the pros and cons of “good enough” software can be found in
[Bre02].

CHAPTER 15 QUALITY CONCEPTS 317

15.3.2 The Cost of Quality
The argument goes something like this: We know that quality is important, but it costs
us time and money—too much time and money to get the level of software quality we
really want. On its face, this argument seems reasonable (see Meyer’s comments
earlier in this section). There is no question that quality has a cost, but lack of qual-
ity also has a cost—not only to end users who must live with buggy software, but
also to the software organization that has built and must maintain it. The real question
is this: Which cost should we be worried about? To answer this question, you must
understand both the cost of achieving quality and the cost of low-quality software.

The cost of quality includes all costs incurred in the pursuit of quality or in per-
forming quality-related activities and the downstream costs of lack of quality. To
understand these costs, an organization should collect metrics to provide a baseline
for the current cost of quality, identify opportunities for reducing these costs, and
provide a normalized basis of comparison. The cost of quality can be divided into
costs associated with prevention, appraisal, and failure.

Prevention costs include (1) the cost of management activities required to plan and
coordinate all quality control and quality assurance activities, (2) the cost of added
technical activities to develop complete requirements and design models, (3) test plan-
ning costs, and (4) the cost of all training associated with these activities. Don’t be
afraid to incur significant prevention costs. Rest assured that your investment will
provide an excellent retur

n.

Appraisal costs include activities to gain insight into product condition the “first
time through” each process. Examples of appraisal costs include: (1) the cost of
conducting technical reviews (Chapter 16) for software engineering work products,
(2) the cost of data collection and metrics evaluation (Chapter 23), and (3) the cost
of testing and debugging (Chapters 19 through 21).

Failure costs are those that would disappear if no errors appeared before shipping
a product to customers. Failure costs may be subdivided into internal failure costs and
external failure costs. Internal failure costs are incurred when you detect an error in
a product prior to shipment. Internal failure costs include: (1) the cost required to
perform rework (repair) to correct an error, (2) the cost that occurs when rework
inadvertently generates side effects that must be mitigated, and (3) the costs associated
with the collection of quality metrics that allow an organization to assess the modes
of failure. External failure costs are associated with defects found after the product
has been shipped to the customer. Examples of external failure costs are complaint
resolution, product return and replacement, help line support, and labor costs associ-
ated with warranty work. A poor reputation and the resulting loss of business is
another external failure cost that is difficult to quantify but nonetheless very real. Bad
things happen when low-quality software is produced.

In an indictment of software developers who refuse to consider external failure
costs, Cem Kaner [Kan95] states:

Many of the external failure costs, such as goodwill, are difficult to quantify, and many
companies therefore ignore them when calculating their cost-benefit tradeoffs. Other
external failure costs can be reduced (e.g. by providing cheaper, lower-quality, post-sale
support, or by charging customers for support) without increasing customer satisfaction.
By ignoring the costs to our customers of bad products, quality engineers encourage
quality-related decision-making that victimizes our customers, rather than delighting them.

318 PART THREE QUALITY AND SECURITY

As expected, the relative costs to find and repair an error or defect increase dra-
matically as we go from prevention to detection to internal failure to external failure
costs. Figure 15.2, based on data collected by Boehm and Basili [Boe01b] and illus-
trated by Cigital Inc. [Cig07], illustrates this phenomenon.

The industry average cost to correct a defect during code generation is approxi-
mately $977 per error. The industry average cost to correct the same error if it is
discovered during system testing is $7,136 per error. Cigital Inc. [Cig07] considers a
large application that has 200 errors introduced during coding.

According to industry average data, the cost of finding and correcting defects during the
coding phase is $977 per defect. Thus, the total cost for correcting the 200 “critical”
defects during this phase (200 × $977) is approximately $195,400.

Industry average data shows that the cost of finding and correcting defects during the
system testing phase is $7,136 per defect. In this case, assuming that the system testing
phase revealed approximately 50 critical defects (or only 25% of those found by Cigital
in the coding phase), the cost of finding and fixing those defects (50 × $7,136) would
have been approximately $356,800. This would also have resulted in 150 critical errors
going undetected and uncorrected. The cost of finding and fixing these remaining 150
defects in the maintenance phase (150 × $14,102) would have been $2,115,300. Thus,
the total cost of finding and fixing the 200 defects after the coding phase would have
been $2,472,100 ($2,115,300 + $356,800).

Even if your software organization has costs that are half of the industry average
(most have no idea what their costs are!), the cost savings associated with early qual-
ity control and assurance activities (conducted during requirements analysis and
design) are compelling.

Figure 15.2
Relative cost
of correcting
errors and
defects
Source: Boehm,
Barry and Basili,
Victor R., “Soft-
ware Defect
Reduction Top 10
List,” IEEE
Computer,
vol. 34, no. 1,
January 2001.

$139

CodingDesignRequirements MaintenanceTesting

$16,000

$14,000

$12,000

$10,000

$8,000

$-

$6,000

$4,000

$2,000

$14,102

$7,136

$977
$455

CHAPTER 15 QUALITY CONCEPTS 319

15.3.3 Risks
In Chapter 1 of this book, we wrote, “people bet their jobs, their comforts, their safety,
their entertainment, their decisions, and their very lives on computer software. It bet-
ter be right.” The implication is that low-quality software increases risks for both the
developer and the end user.4 In the preceding subsection, we discussed one of these
risks (cost). But the downside of poorly designed and implemented applications does
not always stop with dollars and time. An extreme example [Gag04] might serve to
illustrate.

Throughout the month of November 2000 at a hospital in Panama, 28 patients
received massive overdoses of gamma rays during treatment for a variety of cancers.
In the months that followed, 5 of these patients died from radiation poisoning and 15
others developed serious complications. What caused this tragedy? A software pack-
age, developed by a U.S. company, was modified by hospital technicians to compute
modified doses of radiation for each patient.

Quality Issu

es

The scene: Doug Miller’s office
as the SafeHome software
project begins.

The players: Doug Miller, manager of the
SafeHome software engineering team, and
other members of the product software engi-
neering team.

The conversation:
Doug: I was looking at an industry report on
the costs of repairing software defects. It’s
pretty sobering.

Jamie: We’re already working on developing
test cases for each functional requirement.

Doug: That’s good, but I was noticing that it
costs eight times as much to repair a defect
that is discovered in testing than it does if the
defect is caught and repaired during coding.

Vinod: We’re using pair programming, so we
should be able to catch most of the defects
during coding.

Doug: I think you’re missing the point. Quality
is more than simply removing coding errors.
We need to look at the project quality goals
and ensure that the evolving software prod-
ucts are meeting them.

Jamie: Do you mean things like usability,
security, and reliability?

Doug: Yes, I do. We need to build checks into
the software process to monitor our progress
toward meeting our quality goals.

Vinod: Can’t we finish the first prototype and
then check it for quality?

Doug: I’m afraid not. We must establish a
culture of quality early in the project.

Vinod: What do you want us to do, Doug?

Doug: I think we will need to find a technique
that will allow us to monitor the quality of the
SafeHome products. Let’s think about this and
revisit this again tomorrow.

safehome

4 In an article titled “And the ‘Most Shocking Software Failure’ Award Goes To . . .” Chelsea
Frischnecht provides a few brief examples of what can go wrong. The article can be found
at https://www.tricentis.com/blog/2017/03/01/software-fail-awards/.

320 PART THREE QUALITY AND SECURITY

The three Panamanian medical physicists, who tweaked the software to provide
additional capability, were charged with second-degree murder. The U.S. company
was faced with serious litigation in two countries. Gage and McCormick comment:

This is not a cautionary tale for medical technicians, even though they can find them-
selves fighting to stay out of jail if they misunderstand or misuse technology. This also
is not a tale of how human beings can be injured or worse by poorly designed or poorly
explained software, although there are plenty of examples to make the point. This is a
warning for any creator of computer programs: that software quality matters, that appli-
cations must be foolproof, and that—whether embedded in the engine of a car, a robotic
arm in a factory or a healing device in a hospital—poorly deployed code can kill.

Poor quality leads to risks, some of them very serious.5

15.3.4 Negligence and Liability
The story is all too common. A governmental or corporate entity hires a major soft-
ware developer or consulting company to analyze requirements and then design and
construct a software-based “system” to support some major activity. The system might
support a major corporate function (e.g., pension management) or some governmental
function (e.g., health care administration or homeland security).

Work begins with the best of intentions on both sides, but by the time the system is
delivered, things have gone bad. The system is late, fails to deliver desired features and
functions, is error-prone, and does not meet with customer approval. Litigation ensues.

In most cases, the customer claims that the developer has been negligent (in the
manner in which it has applied software practices) and is therefore not entitled to
payment. The developer often claims that the customer has repeatedly changed its
requirements and has subverted the development partnership in other ways. In every
case, the quality of the delivered system comes into question.

15.3.5 Quality and Security
As the criticality of Web-based and mobile systems grows, application security has
become increasingly important. Stated simply, software that does not exhibit high
quality is easier to hack, and as a consequence, low-quality software can indirectly
increase the security risk with all its attendant costs and problems.

In an interview in ComputerWorld, author and security expert Gary McGraw
comments [Wil05]:

Software security relates entirely and completely to quality. You must think about secu-
rity, reliability, availability, dependability—at the beginning, in the design, architecture,
test, and coding phases, all through the software life cycle [process]. Even people aware
of the software security problem have focused on late life-cycle stuff. The earlier you
find the software problem, the better. And there are two kinds of software problems. One
is bugs, which are implementation problems. The other is software flaws—architectural
problems in the design. People pay too much attention to bugs and not enough on flaws.

To build a secure system, you must focus on quality, and that focus must begin
during design. The concepts and methods discussed in Part Two of this book lead to

5 In early 2019, an error in the flight control software produced by a major aircraft manufac-
turer was directly linked to two crashes and the deaths of 346 people.

CHAPTER 15 QUALITY CONCEPTS 321

a software architecture that reduces “flaws.” A more detailed discussion of security
engineering is presented in Chapter 18.

15.3.6 The Impact of Management Actions
Software quality is often influenced as much by management decisions as it is by
technology decisions. Even the best software engineering practices can be subverted
by poor business decisions and questionable project management actions.

In Part Four of this book we discuss project management within the context of the
software process. As each project task is initiated, a project leader will make decisions
that can have a significant impact on product quality.

Estimation Decisions. A software team is rarely given the luxury of providing an
estimate for a project before delivery dates are established and an overall budget is
specified. Instead, the team conducts a “sanity check” to ensure that delivery dates
and milestones are rational. In many cases there is enormous time-to-market pressure
that forces a team to accept unrealistic delivery dates. As a consequence, shortcuts
are taken, activities that lead to higher-quality software may be skipped, and product
quality suffers. If a delivery date is irrational, it is important to hold your ground.
Explain why you need more time, or alternatively, suggest a subset of functionality
that can be delivered (with high quality) in the time allotted.

Scheduling Decisions. When a software project schedule is established (Chapter 25),
tasks are sequenced based on dependencies. For example, because component A depends
on processing that occurs within components B, C, and D, component A cannot be
scheduled for testing until components B, C, and D are fully tested. A project schedule
would reflect this. But if time is very short, and A must be available for further critical
testing, you might decide to test A without its subordinate components (which are run-
ning slightly behind schedule), so that you can make it available for other testing that
must be done before delivery. After all, the deadline looms. As a consequence, A may
have defects that are hidden, only to be discovered much later. Quality suffers.

Risk-Oriented Decisions. Risk management (Chapter 26) is one of the key attributes
of a successful software project. You really do need to know what might go wrong
and establish a contingency plan if it does. Too many software teams prefer blind
optimism, establishing a development schedule under the assumption that nothing will
go wrong. Worse, they don’t have a way of handling things that do go wrong. As a
consequence, when a risk becomes a reality, chaos reigns, and as the degree of crazi-
ness rises, the level of quality invariably falls.

The software quality dilemma can best be summarized by stating Meskimen’s law:
There’s never time to do it right, but always time to do it over again. Our advice:
Taking the time to do it right is almost never the wrong decision.

15.4 ac h i e v i ng so f t Wa r e Qua L i t y

Software quality doesn’t just appear. It is the result of good project management and
solid software engineering practice. Management and practice are applied within the
context of four broad activities that help a software team achieve high software quality:
software engineering methods, project management techniques, quality control actions,
and software quality assurance.

322 PART THREE QUALITY AND SECURITY

15.4.1 Software Engineering Methods
If you expect to build high-quality software, you must understand the problem to be
solved. You must also be capable of creating a design that conforms to the problem
while at the same time exhibiting characteristics that lead to software that exhibits the
quality dimensions and factors discussed in Section 15.2.

In Part Two of this book, we presented a wide array of concepts and methods that
can lead to a reasonably complete understanding of the problem and a comprehensive
design that establishes a solid foundation for the construction activity. If you apply
those concepts and adopt appropriate analysis and design methods, the likelihood of
creating high-quality software will increase substantially.

15.4.2 Project Management Techniques
The impact of poor management decisions on software quality has been discussed in
Section 15.3.6. The implications are clear: If (1) a project manager uses estimation to
verify that delivery dates are achievable, (2) schedule dependencies are understood
and the team resists the temptation to use shortcuts, (3) risk planning is conducted so
problems do not breed chaos, software quality will be affected in a positive way.

In addition, the project plan should include explicit techniques for quality and
change management. Techniques that lead to good project management practices are
discussed in Part Four of this book.

15.4.3 Machine Learning and Defect Prediction
Defect prediction [Mun17] is an important part of identifying software components
that may have quality concerns. Defect prediction models use statistical techniques to
examine the relationships among combinations of software metrics and software com-
ponents containing known software defects. They can be an efficient and effective
way for software developers to quickly identify defect-prone classes. This can reduce
costs and development times [Mal16].

Machine learning is an application of artificial intelligence (AI) techniques that
provide systems with the ability to learn and improve from experience without being
explicitly programmed. Stated another way, machine learning focuses on the develop-
ment of computer programs that can access data and use the data to learn for them-
selves. Machine learning techniques can be used to automate the process of
discovering predictive relationships between software metrics and defective compo-
nents [Ort17], [Li16], [Mal16].

Machine learning systems process large data sets containing representative combi-
nations of metrics for defective and nondefective software components. These data
are used to tune classification algorithms. Once the system has built a prediction
model though this type of training, it can be used for quality assessment and defect
prediction on data associated with future software products. Building these types of
classifiers is a big part of what modern data scientists do. More discussion on the use
of data science and software engineering appears in Appendix 2 of this book.

15.4.4 Quality Control
Quality control encompasses a set of software engineering actions that help to ensure
that each work product meets its quality goals. Models are reviewed to ensure that

CHAPTER 15 QUALITY CONCEPTS 323

they are complete and consistent. Code may be inspected to uncover and correct errors
before testing commences. A series of testing steps is applied to uncover errors in
processing logic, data manipulation, and interface communication. A combination of
measurement and feedback allows a software team to tune the process when any of
these work products fails to meet quality goals. Quality control activities are discussed
in detail throughout the remainder of Part Three of this book.

15.4.5 Quality

Assurance

Quality assurance establishes the infrastructure that supports solid software engineering
methods, rational project management, and quality control actions—all pivotal if you
intend to build high-quality software. In addition, quality assurance consists of a set of
auditing and reporting functions that assess the effectiveness and completeness of qual-
ity control actions. The goal of quality assurance is to provide management and tech-
nical staff with the data necessary to be informed about product quality, thereby
gaining insight and confidence that actions to achieve product quality are working. Of
course, if the data provided through quality assurance identifies problems, it is manage-
ment’s responsibility to address the problems and apply the necessary resources to
resolve quality issues. Software quality assurance is discussed in detail in Chapter 17.

15.5 su m m a ry

Concern for the quality of the software-based systems has grown as software becomes
integrated into every aspect of our daily lives. But it is difficult to develop a compre-
hensive description of software quality. In this chapter, quality has been defined as an
effective software process applied in a manner that creates a useful product that pro-
vides measurable value for those who produce it and those who use it.

A wide variety of software quality dimensions and factors has been proposed over
the years. All try to define a set of characteristics that, if achieved, will lead to high
software quality. McCall’s and the ISO 25010 quality factors establish characteristics
such as reliability, usability, maintainability, functionality, and portability as indicators
that quality exists.

Every software organization is faced with the software quality dilemma. In essence,
everyone wants to build high-quality systems, but the time and effort required to
produce “perfect” software are simply unavailable in a market-driven world. The ques-
tion becomes, Should we build software that is “good enough”? Although many com-
panies do just that, there is a significant downside that must be considered.

Regardless of the approach that is chosen, quality does have a cost that can be
discussed in terms of prevention, appraisal, and failure. Prevention costs include all
software engineering actions that are designed to prevent defects in the first place.
Appraisal costs are associated with those actions that assess software work products
to determine their quality. Failure costs encompass the internal price of failure and
the external effects that poor quality precipitates.

Software quality is achieved through the application of software engineering meth-
ods, solid management practices, and comprehensive quality control—all supported
by a software quality assurance infrastructure. In the chapters that follow, quality
control and assurance are discussed in some detail.

324 PART THREE QUALITY AND SECURITY

Pro b l e m s a n d Po i n t s to Po n d e

r

15.1. Describe how you would assess the quality of a university before applying to it. What
factors would be important? Which would be critical?

15.2. Using the definition of software quality proposed in Section 15.2, do you think it’s pos-
sible to create a useful product that provides measurable value without using an effective
process? Explain your answer.

15.3. Using the subattributes noted for the ISO 25010 quality factor “maintainability” in Sec-
tion 15.2.1, develop a set of questions that explore whether or not these attributes are present.
Follow the example shown in Section 15.2.2.

15.4. Describe the software quality dilemma in your own words.

15.5. What is “good enough” software? Name a specific company and specific products that
you believe were developed using the good enough philosophy.

15.6. Considering each of the four aspects of the cost of quality, which do you think is the
most expensive and why?

15.7. Do a Web search and find three other examples of “risks” to the public that can be
directly traced to poor software quality. Consider beginning your search at http://catless.ncl
.ac.uk/risks.

15.8. Are quality and security the same thing? Explain.

15.9. Explain why it is that many of us continue to live by Meskimen’s law. What is it about
the software business that causes this?

Design element: Quick Look icon magnifying glass: © Roger Pressman

325

C H A P T E R

16

What is it? You’ll make mistakes as you de-
velop software engineering work products.
There’s no shame in that—as long as you try
hard, very hard, to find and correct the mis-
takes before they are delivered to end users.
Technical reviews are the most effective
mechanism for finding mistakes early in the
software process.

Who does it? Software engineers perform
technical reviews, also called peer reviews,
with their colleagues. As we discussed in
Chapters 3 and 4, sometimes it is wise to
include other stakeholders in these reviews.

Why is it important? If you find an error early
in the process, it is less expensive to correct.
In addition, errors have a way of amplifying as
the process proceeds. So a relatively minor
error left untreated early in the process can be
amplified into a major set of errors later in the
project. Finally, reviews save time by reducing

the amount of rework that will be required late
in the project.

What are the steps? Your approach to reviews
will vary depending on the type of review you
select. In general, six steps are employed,
although not all are used for every type of
review: planning, preparation, structuring the
meeting, noting errors, making corrections
(done outside the review), and verifying that
corrections have been performed properly.

What is the work product? The output of a
review is a list of issues and/or errors that
have been uncovered. In addition, the techni-
cal status of the work product is also indicated.

How do I ensure that I’ve done it right? First,
select the type of review that is appropriate for
your development culture. Follow the guide-
lines that lead to successful reviews. If the re-
views that you conduct lead to higher-quality
software, you’ve done it right.

Q u i c k L o o k

Reviews—A Recommended
Approach

bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326
cost effectiveness . . . . . . . . . . . . . . . . . . . . . .329
defect amplification . . . . . . . . . . . . . . . . . . . .327
defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326
error density . . . . . . . . . . . . . . . . . . . . . . . . . .328

errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .326
informal reviews . . . . . . . . . . . . . . . . . . . . . . . 331
record keeping . . . . . . . . . . . . . . . . . . . . . . . .333
review reporting . . . . . . . . . . . . . . . . . . . . . . .333
technical reviews . . . . . . . . . . . . . . . . . . . . . .326

k e y
c o n c e p t s

Software reviews are a “filter” for the software process work flow. Too few, and
the flow is “dirty.” Too many, and the flow slows to a trickle. Reviews are
applied at various points during software engineering and serve to uncover errors
and defects. Software reviews “purify” software engineering work products,
including requirements and design models, code, and testing data. Using metrics,
you can determine which reviews work and emphasize them and at the same
time remove ineffective reviews from the flow to accelerate the process.

326 PART THREE QUALITY AND SECURITY

Many different types of reviews can be conducted as part of software engineering.
Each has its place. An informal meeting around the coffee machine is a form of
review, if technical problems are discussed. A formal presentation of software archi-
tecture to an audience of customers, management, and technical staff is also a form
of review. In this book, however, we focus on technical or peer reviews, exemplified
by casual reviews, walkthroughs, and inspections. A technical review (TR) is the most
effective filter from a quality control standpoint. Conducted by software engineers and
other stakeholders for all project team members, the TR is an effective means for
uncovering errors and improving software quality.

16.1 co st im pac t o f so f t wa r e De f e c t s

Within the context of the software process, the terms defect and fault are synonymous.
Both imply a quality problem that is discovered after the software has been released
to end users (or to another framework activity in the software process). In earlier
chapters, we used the term error to depict a quality problem that is discovered by
software engineers (or others) before the software is released to the end user (or to
another framework activity in the software process).

Bugs, Errors, and Defects
The goal of software quality control, and
in a broader sense, quality management

in general, is to remove quality problems in the
software. These problems are referred to by vari-
ous names—bugs, faults, errors, or defects, to
name a few. Are these terms synonymous, or are
there subtle differences between them?

In this book we make a clear distinction be-
tween an error (a quality problem found before
the software is released to other stakeholders
or end users) and a defect (a quality problem
found only after the software has been re-
leased to end users or other stakeholders).1
We make this distinction because errors and
defects have very different economic, business,
psychological, and human impacts. As software
engineers, we want to find and correct as many
errors as possible before the customer and/or
end user encounter them. We want to avoid

defects—because defects ( justifiably) make
software people look bad.

It is important to note, however, that the tempo-
ral distinction made between errors and defects in
this book is not mainstream thinking. The general
consensus within the software engineering com-
munity is that defects and errors, faults, and bugs
are synonymous. That is, the point in time that the
problem was encountered has no bearing on the
term used to describe the problem. Part of the ar-
gument in favor of this view is that it is sometimes
difficult to make a clear distinction between pre-
and postrelease (e.g., consider an incremental
process used in agile development).

Regardless of how you choose to interpret
these terms, recognize that the point in time at
which a problem is discovered does matter and
that software engineers should try hard—very
hard—to find problems before their customers
and end users encounter them.

info

1 If software process improvement is considered, a quality problem that is propagated from
one process framework activity (e.g., modeling) to another (e.g., construction) can also
be called a “defect” because the problem should have been found before a work product
(e.g., a design model) was “released” to the next activity.

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 327

The primary objective of a formal technical review (FTR) is to find errors before
they are passed on to another software engineering activity or released to the end user.
The obvious benefit of technical reviews is the early discovery of errors so that they
do not propagate to the next step in the software process.

A number of industry studies indicate that design activities introduce between 50
and 65 percent of all errors (and ultimately, all defects) during the software process.
However, review techniques have been shown to be up to 75 percent effective [Jon86]
in uncovering design flaws. By detecting and removing a large percentage of these
errors, the review process substantially reduces the cost of subsequent activities in the
software process. We have known this for decades, and yet there are still many devel-
opers who do not believe the time spent on reviews is almost always less than the
time required to rewrite bad code [Yad17].

16.2 De f e c t am p L i f i cat i o n a n D re m ova L

Defect amplification is a concept originally proposed almost four decades ago [IBM81].
It helps to justify the effort expended on software reviews. In essence, defect ampli-
fication makes the following argument—an error introduced early in the software
engineering work flow (e.g., during requirement modeling) and undetected, can and
often will be amplified into multiple errors during design. If those errors are not
uncovered (using effective reviews), they themselves may be further amplified into
still more errors during coding. A single error introduced early and not uncovered and
corrected can amplify into multiple errors later in the process. Defect propagation is
a term used to describe the impact an undiscovered error has on future development
activities or product behavior [Vit17].

As a development team moves deeper into the software process, the cost of find-
ing and fixing an error grows. This simple reality is exacerbated by defect amplifi-
cation and propagation because a single error may become multiple errors
downstream. The cost of finding and fixing a single error can be signficant, but the
cost to find and fix multiple errors propagated by a single earlier error is substan-
tially more significant.

To conduct reviews, you must expend time and effort, and your development orga-
nization must spend money. However, the reality of defect amplification and propaga-
tion leaves little doubt that you can pay now or pay much more later. This is what
technical debt (Chapter 11) is all about [Xia16] [Vit17].

16.3 re v i e w me t r i c s a n D th e i r us e

Technical reviews are one of many actions that are required as part of good software
engineering practice. Each action requires dedicated human effort. Because available
project effort is finite, it is important for a software engineering organization to under-
stand the effectiveness of each action by defining a set of metrics (Chapter 23) that
can be used to assess their efficacy.

328 PART THREE QUALITY AND SECURITY

Although many metrics can be defined for technical reviews, a relatively small
subset can provide useful insight. The following review metrics can be collected for
each review that is conducted:

∙ Preparation effort, Ep. The effort (in person-hours) required to review a
work product prior to the actual review meeting

∙ Assessment effort, Ea. The effort (in person-hours) that is expended during
the actual review

∙ Rework effort, Er. The effort (in person-hours) that is dedicated to the
correction of those errors uncovered during the review

∙ Review effort, Ereview. Represents the sum of effort measures for reviews:

Ereview = Ep + Ea + Er

∙ Work product size (WPS). A measure of the size of the work product that
has been reviewed (e.g., the number of UML models, the number of docu-
ment pages, or the number of lines of code)

∙ Minor errors found, Errminor. The number of errors found that can be catego-
rized as minor (requiring less than some prespecified effort to correct)

∙ Major errors found, Errmajor. The number of errors found that can be catego-
rized as major (requiring more than some prespecified effort to correct)

∙ Total errors found, Errtot. Represents the sum of the errors found:

Errtot = Errminor + Errmajor

∙ Error density. Represents the errors found per unit of work product
reviewed:

Error density =
Errtot

WPS

How might these metrics be used? As an example, consider a requirements model
that is reviewed to uncover errors, inconsistencies, and omissions. It would be pos-
sible to compute the error density in several different ways. Assume the requirements
model contains 18 UML diagrams as part of 32 overall pages of descriptive materials.
The review uncovers 18 minor errors and 4 major errors. Therefore, Errtot = 22. Error
density is 1.2 errors per UML diagram, or 0.68 errors per requirements model page.

If reviews are conducted for a number of different types of work products (e.g.,
requirements model, design model, code, test cases), the percentage of errors uncov-
ered for each review can be computed against the total number of errors found for all
reviews. In addition, the error density for each work product can be computed.

Once data are collected for many reviews conducted across many projects, average
values for error density enable you to estimate the number of errors to be found in a
new document before it is reviewed. For example, if the average error density for a
requirements model is 0.68 errors per page, and a new requirements model is 40 pages
long, a rough estimate suggests that your software team will find around 27 errors
during the review of the document. If you find only 9 errors, you’ve either done an
extremely good job in developing the requirements model or your review approach
was not thorough enough.

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 329

It is difficult to measure the cost effectiveness of any technical review in real time.
A software engineering organization can assess the effectiveness of reviews and their
cost benefit only after reviews have been completed, review metrics have been col-
lected, average data have been computed, and then the downstream quality of the
software is measured (via testing).

Returning to the previous example, the average error density for requirements mod-
els was determined to be 0.68 per page. The effort required to correct a minor model
error (immediately after the review) has been found to require 4 person-hours. The
effort required for a major requirement error has been found to be 18 person-hours.
Examining the review data collected, you find that minor errors occur about six times
more frequently than major errors. Therefore, you can estimate that the average effort
to find and correct a requirements error during review is about 6 person-hours.

Requirements-related errors uncovered during testing require an average of
45  person-hours to find and correct (no data are available on the relative severity of
the error). Using the averages noted, we get:

Effort saved per error = Etesting − Ereviews

= 45 − 6 = 39 person-hours/error

Because 22 errors were found during the review of the requirements model, a savings
of about 858 person-hours of testing effort would be achieved. And that’s just for
requirements-related errors. Errors associated with design and code would add to the
overall benefit.

The bottom line—effort saved leads to shorter delivery cycles and improved time
to market. The example presented in this section suggests this may be true. More
importantly, industry data for software reviews has been collected for more than three
decades and is summarized qualitatively using the graphs shown in Figure 16.1.

Referring to the figure, the effort expended when reviews are used does increase
early in the development of a software increment, but this early investment for reviews
pays dividends because testing and corrective effort is reduced. It is important to note

Planning Requirements Design Code Test Deployment

Time

E�
or

t

Without inspection

With inspection

Figure 16.1
Effort
expended with
and without
reviews
Source: Fagan,
Michael E.,
“Advances in Soft-
ware Inspections,”
IEEE Transactions
on Software
Engineering,
vol. SE-12, no. 7,
July 1986,
744–751.

330 PART THREE QUALITY AND SECURITY

that the deployment date for development with reviews is sooner than the deployment
date without reviews. Reviews don’t take time; they save it!

16.4 Cr i t e r i a f o r ty p e s o f re v i e ws

Technical reviews can be classified as either formal or informal or somewhere in
between these two extremes. The level of formality is chosen to match the type of
product to be built, the project time line, and the people who are doing the work.
Figure 16.2 depicts a reference model for technical reviews [Lai02] that identifies four
characteristics that contribute to the formality with which a review is conducted.

Each of the reference model characteristics helps to define the level of review
formality. The formality of a review increases when (1) distinct roles are explicitly
defined for the reviewers, (2) there is a sufficient amount of planning and preparation
for the review, (3) a distinct structure for the review (including tasks and internal work
products) is defined, and (4) follow-up by the reviewers occurs for any corrections
that are made.

An element that is not presented in this model is the frequency of the reviews them-
selves. If you are using an agile prototyping model (Chapter 4) that contains relatively
short sprints, your team may opt for less formal reviews because the reviews are hap-
pening fairly often. This usually means that defects are caught sooner and more often.

To understand the reference model, let’s assume that you’ve decided to review the
interface design for SafeHomeAssured.com. You can do this in a variety of different
ways that range from relatively casual to extremely rigorous. If you decide that the casual
approach is most appropriate, you ask a few colleagues (peers) to examine the interface
prototype in an effort to uncover potential problems. All of you decide that there will
be no advance preparation, but that you will evaluate the prototype in a reasonably
structured way—looking at layout first, aesthetics next, navigation options after that, and
so on. As the designer, you decide to take a few notes, but nothing formal.

Review

Planning &
Preparation

Correction &
Verification

Meeting
Structure

Roles
Individuals

Play

Figure 16.2
Reference
model for
technical
reviews

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 331

But what if the interface is pivotal to the success of the entire project? What if
human lives depended on an interface that was ergonomically sound? You might
decide that a more rigorous approach was necessary. A review team would be formed.
Each person on the team would have a specific role to play—leading the team, record-
ing findings, presenting the material, and so on. Each reviewer would be given access
to the work product (in this case, the interface prototype) before the review and would
spend time looking for errors, inconsistencies, and omissions. A set of specific tasks
would be conducted based on an agenda that was developed before the review occurred.
The results of the review would be formally recorded, and the team would decide on
the status of the work product based on the outcome of the review. Members of the
review team might also verify that the corrections made were done properly.

In this book we consider two broad categories of technical reviews: informal
reviews and more formal technical reviews. Within each category, a number of dif-
ferent approaches can be chosen. These are presented in the sections that follow.

16.5 in f o r m a L re v i e ws

Informal reviews include a simple desk check of a software engineering work product
with a colleague, a casual meeting (involving more than two people) for the purpose
of reviewing a work product, or the review-oriented aspects of pair programming
(Chapter 3).

A simple desk check or a casual meeting conducted with a colleague is a review.
However, because there is no advance planning or preparation, no agenda or meeting
structure, and no follow-up on the errors that are uncovered, the effectiveness of
such reviews is considerably lower than more formal approaches. But a simple desk
check can and does uncover errors that might otherwise propagate further into the
software process.

One way to improve the efficacy of a desk check review is to develop a set of
simple review checklists2 for each major work product produced by the software team.
The questions posed within the checklist are generic, but they will serve to guide the
reviewers as they check the work product. For example, let’s reexamine a desk check
of the interface prototype for SafeHomeAssured.com. Rather than simply playing
with the prototype at the designer’s workstation, the designer and a colleague examine
the prototype using a checklist for interfaces:

∙ Is the layout designed using standard conventions? Left to right? Top to bottom?
∙ Does the presentation need to be scrolled?
∙ Are color and placement, typeface, and size used effectively?
∙ Are all navigation options or functions represented at the same level of

abstraction?
∙ Are all navigation choices clearly labeled?

2 Literally hundreds of technical review checklists can be found via a web search. For exam-
ple, a useful code review checklist can be downloaded from https://courses.cs.washington.
edu/courses/cse403/12wi/sections/12wi_code_review_checklist .

332 PART THREE QUALITY AND SECURITY

and so on. Any errors or issues noted by the reviewers are recorded by the designer
for resolution at a later time. Desk checks may be scheduled in an ad hoc manner, or
they may be mandated as part of good software engineering practice. In general, the
amount of material to be reviewed is relatively small and the overall time spent on a
desk check spans little more than 1 or 2 hours.

In Chapter 3, we described pair programming in the following manner: XP recom-
mends that two people work together at one computer workstation to create code for
a story. This provides a mechanism for real-time problem solving (two heads are often
better than one) and real-time quality assurance.

Pair programming (Section 3.5.1) can be characterized as a continuous desk check.
Rather than scheduling a review at some point in time, pair programming encourages
continuous review as a work product (design or code) is created. The benefit is imme-
diate discovery of errors and better work product quality.

Some software engineers argue that the inherent redundancy built into pair pro-
gramming is wasteful of resources. After all, why assign two people to a job that one
person can accomplish? The answer to this question can be found in Section 16.3. If
the quality of the work product produced as a consequence of pair programming is
significantly better than the work of an individual, the quality-related savings can more
than justify the “redundancy” implied by pair programming.

16.6 fo r m a L te c h n i ca L re v i e ws

A formal technical review (FTR) is a software quality control activity performed by
software engineers (and others). The objectives of an FTR are: (1) to uncover errors
in function, logic, or implementation for any representation of the software; (2) to
verify that the software under review meets its requirements; (3) to ensure that the
software has been represented according to predefined standards; (4) to achieve soft-
ware that is developed in a uniform manner; and (5) to make projects more manage-
able. In addition, the FTR serves as a training ground, enabling junior engineers to
observe different approaches to software analysis, design, and implementation. The
FTR also serves to promote backup and continuity because a number of people
become familiar with parts of the software that they may not have otherwise seen.

The FTR is actually a class of reviews that includes walkthroughs and inspections.
Each FTR is conducted as a meeting and will be successful only if it is properly
planned, controlled, and attended. In the sections that follow, guidelines similar to
those for a walkthrough are presented as a representative formal technical review. If
you have interest in software inspections, as well as additional information on walk-
throughs, see [Rad02], [Wie02], or [Fre90].

16.6.1 The Review Meeting
Regardless of the FTR format that is chosen, every review meeting should abide by
the following constraints:

∙ Between three and five people (typically) should be involved in the review.
∙ Advance preparation should occur but should require no more than 2 hours of

work for each person.
∙ The duration of the review meeting should be less than 2 hours.

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 333

Given these constraints, it should be obvious that an FTR focuses on a specific
(and small) part of the overall software. For example, rather than attempting to review
an entire design, walkthroughs are conducted for each component or small group of
components. By narrowing the focus, the FTR has a higher likelihood of uncovering
errors.

The focus of the FTR is on a work product (e.g., a self-contained portion of a
requirements model, a detailed component design, or source code for a component).
The individual who has developed the work product—the producer—informs the proj-
ect leader that the work product is complete and that a review is required. The project
leader contacts a review leader, who evaluates the product for readiness, generates
copies of product materials, and distributes them to two or three reviewers for advance
preparation. Each reviewer is expected to spend between 1 and 2 hours reviewing the
product, making notes, and otherwise becoming familiar with the work. Concurrently,
the review leader also reviews the product and establishes an agenda for the review
meeting, which is typically scheduled for the next day.

The review meeting is attended by the review leader, all reviewers, and the pro-
ducer. One of the reviewers takes on the role of a recorder, that is, the individual
who records (in writing) all important issues raised during the review. The FTR
begins with an introduction of the agenda and a brief introduction by the producer.
The producer then proceeds to “walk through” the work product, explaining the
material, while reviewers raise issues based on their advance preparation. When valid
problems or errors are discovered, the recorder notes each. Don’t point out errors
too harshly. One way to be gentle is to ask a question that enables the producer to
discover the error.

At the end of the review, all attendees of the FTR must decide whether to:
(1) accept the product without further modification, (2) reject the product due to
severe errors (once corrected, another review must be performed), or (3) accept the
product provisionally (minor errors have been encountered and must be corrected, but
no additional review will be required). After the decision is made, all FTR attendees
complete a sign-off, indicating their participation in the review and their concurrence
with the review team’s findings.

16.6.2 Review Reporting and Record Keeping
During the FTR, a reviewer (the recorder) actively records all issues that have been
raised. These are summarized at the end of the review meeting, and a review issues
list is produced. In addition, a formal technical review summary report is completed.
A review summary report answers three questions:

1. What was reviewed?
2. Who reviewed it?
3. What were the findings and conclusions?

The review summary report is a single-page form (with possible attachments). It
becomes part of the project historical record and may be distributed to the project
leader and other interested parties.

The review issues list serves two purposes: (1) to identify problem areas within
the product and (2) to serve as an action item checklist that guides the producer as
corrections are made. An issues list is normally attached to the summary report.

334 PART THREE QUALITY AND SECURITY

You should establish a follow-up procedure to ensure that items on the issues list
have been properly corrected. Unless this is done, it is possible that issues raised can
“fall between the cracks.” One approach is to assign the responsibility for follow-up
to the review leader.

16.6.3 Review Guidelines
Guidelines for conducting formal technical reviews must be established in advance,
distributed to all reviewers, agreed upon, and then followed. A review that is uncon-
trolled can often be worse than no review at all. The following represents a minimum
set of guidelines for formal technical reviews:

1. Review the product, not the producer. An FTR involves people and egos.
Conducted properly, the FTR should leave all participants with a warm feel-
ing of accomplishment. Conducted improperly, the FTR can take on the aura
of an inquisition. Errors should be pointed out gently; the tone of the meeting
should be loose and constructive; the intent should not be to embarrass or
belittle. The review leader should conduct the review meeting to ensure that
the proper tone and attitude are maintained and should immediately halt a
review that has gotten out of control.

2. Set an agenda and maintain it. One of the key maladies of meetings of all
types is drift. An FTR must be kept on track and on schedule. The review
leader is chartered with the responsibility for maintaining the meeting sched-
ule and should not be afraid to nudge people when drift sets in.

3. Limit debate and rebuttal. When an issue is raised by a reviewer, there may
not be universal agreement on its impact. Rather than spending time debating
the question, the issue should be recorded for further discussion off-line.

4. Enunciate problem areas, but don’t attempt to solve every problem noted.
A review is not a problem-solving session. The solution of a problem can often
be accomplished by the producer alone or with the help of only one other indi-
vidual. Problem solving should be postponed until after the review meeting.

5. Take written notes. It is sometimes a good idea for the recorder to make
notes on a wall board, so that wording and priorities can be assessed by other
reviewers as information is recorded.

6. Limit the number of participants and insist upon advance preparation.
Two heads are better than one, but 14 are not necessarily better than 4. Keep
the number of people involved to the necessary minimum. However, all
review team members must prepare in advance.

7. Develop a checklist for each product that is likely to be reviewed. A
checklist helps the review leader to structure the FTR meeting and helps each
reviewer to focus on important issues. Checklists should be developed for
analysis, design, code, and even testing work products.

8. Allocate resources and schedule time for FTRs. For reviews to be effective,
they should be scheduled as a task during the software process. In addition,
time should be scheduled for the inevitable modifications that will occur as
the result of an FTR.

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 335

9. Conduct meaningful training for all reviewers. The training should stress
both process-related issues and the human psychological side of reviews.

10. Review your early reviews. Debriefing can be beneficial in uncovering prob-
lems with the review process itself. The very first product to be reviewed
should be the review guidelines themselves.

Because many variables (e.g., number of participants, type of work products,
timing and length, specific review approach) have an impact on a successful review,
a software organization should experiment to determine what approach works best in
a local context.

In an ideal setting, every software-engineering work product would undergo a for-
mal technical review. In the real world of software projects, resources are limited and
time is short. As a consequence, reviews are often skipped, even though their value
as a quality control mechanism is recognized. However, full FTR resources should be
used on those work products that are likely to be error prone.

Quality Issues

The scene: Doug Miller’s office
as the SafeHome software
project begins.

The players: Doug Miller, manager of the
SafeHome software engineering team, and
other members of the product software
engineering team.

The conversation:
Doug: I know we didn’t spend time develop-
ing a quality plan for this project, but we’re
already into it and we have to consider
quality . . . right?

Jamie: Sure. We’ve already decided that as
we develop the requirements model
[Chapter 8], Ed has committed to develop a
testing procedure for each requirement.

Doug: That’s really good, but we’re not going
to wait until testing to evaluate quality, are we?

Vinod: No! Of course not. We’ve got reviews
scheduled into the project plan for this soft-
ware increment. We’ll begin quality control
with the reviews.

Jamie: I’m a bit concerned that we won’t have
enough time to conduct all the reviews. In fact,
I know we won’t.

Doug: Hmmm. So what do you propose?

Jamie: I say we select those elements
of the requirements and design model that
are most critical to SafeHome and review
them.

Vinod: But what if we miss something in a part
of the model we don’t review?

Jamie: Maybe . . . but I’m not sure we even
have time to review every element of the
models.

Vinod: What do you want us to do, Doug?

Doug: Let’s steal something from Extreme
Programming [Chapter 3]. We’ll develop the
elements of each model in pairs—two
people—and conduct an informal review of
each as we go. We’ll then target “critical”
elements for a more formal team review, but
keep those reviews to a minimum. That way,
everything gets looked at by more than one
set of eyes, but we still maintain our delivery
dates.

Jamie: That means we’re going to have to
revise the schedule.

Doug: So be it. Quality trumps schedule on
this project.

safehome

336 PART THREE QUALITY AND SECURITY

16.7 po st m o rt e m eva Luat i o n s

Many lessons can be learned if a software team takes the time to evaluate the results
of a software project after the software has been delivered to end users. Baaz and his
colleagues [Baa10] suggest the use of a postmortem evaluation (PME) as a mechanism
to determine what went right and what went wrong when software engineering process
and practice is applied in a specific project.

Unlike an FTR that focuses on a specific work product, it is more like a Scrum
retrospective (Section 3.4.5). A PME examines the entire software project, focusing
on both “excellences (that is, achievements and positive experiences) and challenges
(problems and negative experiences)” [Baa10]. Often conducted in a workshop format,
a PME is attended by members of the software team and stakeholders. The intent is
to identify excellences and challenges and to extract lessons learned from both. The
objective is to suggest improvements to both process and practice going forward.
Many software engineers regard PME documents as some of the most valuable docu-
ments to save in the project archive.

16.8 ag i L e re v i e ws

It’s not surprising that some software engineers balk at the thought of including
reviews of any kind in agile development processes. Yet failing to catch defects early
can be costly in terms of time and resources. Ignoring technical debt does not make
it go away. Agile developers have the same need to find defects early (and often)
that all software developers have. Admittedly, it may be somewhat more difficult to
get agile developers to make use of metrics, but many of them can be collected
unobtrusively.

If we take a closer look at the Scrum framework (Section 3.4), there are several
places where informal and formal reviews take place. During the sprint planning
meeting, user stories are reviewed and ordered according to priority, before selecting
the user stories to include in the next sprint. The daily Scrum meeting is an informal
way to ensure that the team members are all working on the same priorities and to
catch any defects that may prevent completing the sprint on time. Agile developers
often use pair programming, another informal review technique. The sprint review
meeting is often conducted using guidelines similar to those discussed for a formal
technical review. The code producers walk through the user stories selected for the
sprint and demonstrate to the product owner that all functionality is present. Unlike
the FTR, the product owner has the final word on whether to accept the sprint proto-
type or not.

If we look at the evaluate prototype portion of our recommended process model
(Section 4.5), this task is also likely to be conducted as a formal technical review with
development risk assessment added to it. We mentioned previously (Section 16.7) that
the sprint retrospective meeting is really similar to the project postmortem meeting in
that the development team is trying to capture its lessons learned. A major aspect of
software quality assurance is being able to repeat your successes and avoid repeating
your mistakes.

CHAPTER 16 REVIEWS—A RECOMMENDED APPROACH 337

16.9 su m m a ry

The intent of every technical review is to find errors and uncover issues that would
have a negative impact on the software to be deployed. The sooner an error is uncovered
and corrected, the less likely that error will propagate to other software engineering
work products and amplify itself, resulting in significantly more effort to correct it.

To determine whether quality control activities are working, a set of metrics should
be collected. Review metrics focus on the effort required to conduct the review and
the types and severity of errors uncovered during the review. Once metrics data are
collected, they can be used to assess the efficacy of the reviews you do conduct.
Industry data indicate that reviews provide a significant return on investment.

A reference model for review formality identifies the roles people play, planning
and preparation, meeting structure, correction approach, and verification as the char-
acteristics that indicate the degree of formality with which a review is conducted.
Informal reviews are casual in nature but can still be used effectively to uncover errors.
Formal reviews are more structured and have the highest probability of leading to
high-quality software.

Informal reviews are characterized by minimal planning and preparation and little
record keeping. Desk checks and pair programming fall into the informal review
category.

A formal technical review is a stylized meeting that has been shown to be extremely
effective in uncovering errors. Formal technical reviews establish defined roles for
each reviewer, encourage planning and advance preparation, require the application
of defined review guidelines, and mandate record keeping and status reporting.

pro b L e m s a n D po i n t s to po n D e r

16.1. Explain the difference between an error and a defect.

16.2. Why can’t we just wait until testing to find and correct all software errors?

16.3. Assume that 10 errors have been introduced in the requirements model and that each
error will be amplified by a factor of 2:1 into design and an additional 20 design errors are
introduced and then amplified 1.5:1 into code where an additional 30 errors are introduced.
Assume further that all unit testing will find 30 percent of all errors, integration will find
30 percent of the remaining errors, and validation tests will find 50 percent of the remaining
errors. No reviews are conducted. How many errors will be released to the end users?

16.4. Reconsider the situation described in Problem 16.3, but now assume that requirements,
design, and code reviews are conducted and are 60 percent effective in uncovering all errors at
that step. How many errors will be released to the field?

16.5. Reconsider the situation described in Problems 16.3 and 16.4. If each of the errors
released to the field costs $4800 to find and correct and each error found in review costs $240
to find and correct, how much money is saved by conducting reviews?

16.6. Describe the meaning of Figure 16.1 in your own words.

16.7. Can you think of a few instances in which a desk check might create problems rather
than provide benefits?

338 PART THREE QUALITY AND SECURITY

16.8. A formal technical review is effective only if everyone has prepared in advance. How do
you recognize a review participant who has not prepared? What do you do if you’re the review
leader?

16.9. How is technical debt addressed in agile process models?

16.10. Consider the review guidelines presented in Section 16.6.3. Which do you think is most
important and why?

Design element: Quick Look icon magnifying glass: © Roger Pressman

339

C H A P T E R

17

What is it? It’s not enough to talk the talk by
saying that software quality is important. You
have to (1) explicitly define what is meant when
you say “software quality,” (2) create a set of
activities that will help ensure that every soft-
ware engineering work product exhibits high
quality, (3) perform quality control and assur-
ance activities on every software project,
(4)  use metrics to develop strategies for
improving your software process and, as a
consequence, the quality of the end product.

Who does it? Everyone involved in the software
engineering process is responsible for quality.

Why is it important? You can do it right, or you
can do it over again. If a software team
stresses quality in all software engineering ac-
tivities, it reduces the amount of rework that it
must do. That results in lower costs, and more
importantly, improved time to market.

What are the steps? Before software quality
assurance (SQA) activities can be initiated, it is

important to define software quality at several
different levels of abstraction. Once you un-
derstand what quality is, a software team must
identify a set of SQA activities that will filter
errors out of work products before they are
passed on.

What is the work product? A Software Qual-
ity Assurance Plan is created to define a soft-
ware team’s SQA strategy. During modeling
and coding, the primary SQA work product is
the output of technical reviews (Chapter 16).
During testing (Chapters 19 through 21), test
plans and procedures are produced. Other
work products associated with process
improvement may also be generated.

How do I ensure that I’ve done it right? Find
errors before they become defects! That is,
work to improve your defect removal effi-
ciency (Chapter 23), thereby reducing the
amount of rework that your software team
must perform.

Q u i c k L o o k

Software Quality
Assurance

Bayesian inference . . . . . . . . . . . . . . . . . . . . . 351
elements of software quality assurance . . . . 341
formal approaches . . . . . . . . . . . . . . . . . . . . . 347
genetic algorithms . . . . . . . . . . . . . . . . . . . . .352
goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .346
ISO 9001:2015 standard . . . . . . . . . . . . . . . . .354

Six Sigma . . . . . . . . . . . . . . . . . . . . . . . . . . . . .349
software reliability . . . . . . . . . . . . . . . . . . . . .350
software safety . . . . . . . . . . . . . . . . . . . . . . . .352
SQA plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . .354
SQA tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .343
statistical software quality assurance . . . . . . 347

k e y
c o n c e p t s

The software engineering approach described in this book works toward a single
goal: to produce on-time, high-quality software. Yet many readers will be
challenged by the question: “What is software quality?”

340 PART 3 QUALITY AND SECURITY

Philip Crosby [Cro79], in his landmark book on quality, provides a wry answer to
this question:

The problem of quality management is not what people don’t know about it. The problem
is what they think they do know .  .  .

In this regard, quality has much in common with sex. Everybody is for it. (Under
certain conditions, of course.) Everyone feels they understand it. (Even though they
wouldn’t want to explain it.) Everyone thinks execution is only a matter of following
natural inclinations. (After all, we do get along somehow.) And, of course, most people
feel that problems in these areas are caused by other people. (If only they would take
the time to do things right.)

Indeed, quality is a challenging concept—one that we addressed in some detail in
Chapter 15.1

Some software developers continue to believe that software quality is something
you begin to worry about after code has been generated. Nothing could be further
from the truth! Software quality assurance (often called quality management) is an
umbrella activity (Chapter 2) that is applied throughout the software process.

Software quality assurance (SQA) encompasses (Figure 17.1): (1) an SQA process,
(2) specific quality assurance and quality control tasks (including technical reviews
and a multitiered testing strategy), (3) effective software engineering practice (meth-
ods and tools), (4) control of all software work products and the changes made to
them (Chapter 22), (5) a procedure to ensure compliance with software development
standards (when applicable), and (6) measurement and reporting mechanisms.

In this chapter, we focus on the management issues and the process-specific
activities that enable a software organization to ensure that it does “the right things
at the right time in the right way.”

1 If you have not read Chapter 15, you should do so now.

Figure 17.1
Software
quality
assurance

Software
Quality

Assurance

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 341

17.1 Bac kg ro u n d is s u e s

Quality control and assurance are essential activities for any business that produces
products to be used by others. Prior to the twentieth century, quality control was the
sole responsibility of the craftsperson who built a product. As time passed and mass
production techniques became commonplace, quality control became an activity
performed by people other than the ones who built the product.

The first formal quality assurance and control function was introduced at Bell Labs
in 1916 and spread rapidly throughout the manufacturing world. During the 1940s,
more formal approaches to quality control were suggested. These relied on measure-
ment and continuous process improvement [Dem86] as key elements of quality
management.

The history of quality assurance in software development parallels the history of
quality in hardware manufacturing. During the early days of computing (1950s and
1960s), quality was the sole responsibility of the programmer. Standards for quality
assurance for software were introduced in military contract software development
during the 1970s and have spread rapidly into software development in the commer-
cial world [IEE17]. Extending the definition presented earlier, software quality assur-
ance is a “planned and systematic pattern of actions” [Sch01] that are required to
ensure high quality in software. The scope of quality assurance responsibility might
best be characterized by paraphrasing a once-popular automobile commercial: “Quality
Is Job #1.” The implication for software is that many different constituencies have
software quality assurance responsibility—software engineers, project managers,
customers, salespeople, and the individuals who serve within an SQA group.

The SQA group serves as the customer’s in-house representative. That is, the peo-
ple who perform SQA must look at the software from the customer’s point of view.
Does the software adequately meet the quality factors noted in Chapter 15? Have
software engineering practices been conducted according to preestablished standards?
Have technical disciplines properly performed their roles as part of the SQA activity?
The SQA group attempts to answer these and other questions to ensure that software
quality is maintained.

17.2 eL e m e n t s o f so f t wa r e Qua L i t y as s u r a nc e

Software quality assurance encompasses a broad range of concerns and activities that
focus on the management of software quality. These can be summarized in the
following manner [Hor03]:

Standards. The IEEE, ISO, and other standards organizations have pro-
duced a broad array of software engineering standards and related documents.
Standards may be adopted voluntarily by a software engineering organization
or imposed by the customer or other stakeholders. The job of SQA is to
ensure that standards that have been adopted are followed and that all work
products conform to them.
Reviews and audits. Technical reviews are a quality control activity per-
formed by software engineers for software engineers (Chapter 16). Their

342 PART 3 QUALITY AND SECURITY

intent is to uncover errors. Audits are a type of review performed by SQA
personnel with the intent of ensuring that quality guidelines are being fol-
lowed for software engineering work. For example, an audit of the review
process might be conducted to ensure that reviews are being performed in a
manner that will lead to the highest likelihood of uncovering errors.
Testing. Software testing (Chapters 19 through 21) is a quality control func-
tion that has one primary goal—to find errors. The job of SQA is to ensure
that testing is properly planned and efficiently conducted so that it has the
highest likelihood of achieving its primary goal.
Error/defect collection and analysis. The only way to improve is to mea-
sure how you’re doing. SQA collects and analyzes error and defect data to
better understand how errors are introduced and what software engineering
activities are best suited to eliminating them.
Change management. Change is one of the most disruptive aspects of any
software project. If it is not properly managed, change can lead to confusion,
and confusion almost always leads to poor quality. SQA ensures that adequate
change management practices (Chapter 22) have been instituted.
Education. Every software organization wants to improve its software engi-
neering practices. A key contributor to improvement is education of software
engineers, their managers, and other stakeholders. The SQA organization
takes the lead in software process improvement (Chapter 28) and is a key
proponent and sponsor of educational programs.
Vendor management. Three categories of software are acquired from exter-
nal software vendors—shrink-wrapped packages (e.g., Microsoft Office), a
tailored shell [Hor03] that provides a basic skeletal structure that is custom
tailored to the needs of a purchaser, and contracted software that is custom
designed and constructed from specifications provided by the customer orga-
nization. The job of the SQA organization is to ensure that high-quality soft-
ware results by suggesting specific quality practices that the vendor should
follow (when possible) and incorporating quality mandates as part of any
contract with an external vendor.
Security management. With the increase in cyber crime and new govern-
ment regulations regarding privacy, every software organization should insti-
tute policies that protect data at all levels, establish firewall protection for
mobile apps, and ensure that software has not been tampered with internally.
SQA ensures that appropriate process and technology are used to achieve
software security (Chapter 18).
Safety. Because software is almost always a pivotal component of human-
rated systems (e.g., automotive or aircraft applications), the impact of hidden
defects can be catastrophic. SQA may be responsible for assessing the impact
of software failure and for initiating those steps required to reduce risk.
Risk management. Although the analysis and mitigation of risk (Chapter 26)
is the concern of software engineers, the SQA organization ensures that risk
management activities are properly conducted and that risk-related contingency
plans have been established.

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 343

In addition to each of these concerns and activities, SQA works to ensure that
software support activities (e.g., maintenance, help lines, documentation, and manuals)
are conducted or produced with quality as a dominant concern.

17.3 sQa pro c e s s e s a n d pro d u c t ch a r ac t e r i st i c s

As we begin a discussion of software quality assurance, it’s important to note that SQA
procedures and approaches that work in one software environment may not work as
well in another. Even within a company that adopts a consistent approach2 to software
engineering, different software products may exhibit different levels of quality [Par11].

The solution to this dilemma is to understand the specific quality requirements for
a software product and then select the process and specific SQA actions and tasks
that will be used to meet those requirements. The Software Engineering Institute’s
CMMI and ISO 9000 standards are the most commonly used process frameworks.
Each proposes “a syntax and semantics” [Par11] that will lead to the implementation
of software engineering practices that improve product quality. Rather than instantiat-
ing either framework in its entirety, a software organization can “harmonize” the two
models by selecting elements of both frameworks and matching them to the quality
requirements of an individual product.

17.4 sQa ta s k s, goa L s, a n d me t r i c s

Software quality assurance is composed of a variety of tasks associated with two dif-
ferent constituencies—the software engineers who do technical work and an SQA
group that has responsibility for quality assurance planning, oversight, record keeping,
analysis, and reporting.

Modern software quality assurance is often data driven, as shown in Figure 17.2.
The product stakeholders define goals and quality measures, problem areas are
identified, indicators are measured, and a determination is made as to whether or not
process changes are needed. Software engineers address quality (and perform quality
control activities) by applying solid technical methods and measures, conducting tech-
nical reviews, and performing well-planned software testing.

17.4.1 SQA Tasks
The charter of the SQA group is to assist the software team in achieving a high-
quality end product. The Software Engineering Institute recommends a set of SQA
activities that address quality assurance planning, oversight, record keeping, analysis,
and reporting. These activities are performed (or facilitated) by an independent SQA
group that:

Prepares an SQA plan for a project. The plan is developed as part of
project planning and is reviewed by all stakeholders. Quality assurance activi-
ties performed by the software engineering team and the SQA group are gov-
erned by the plan. The plan identifies evaluations to be performed, audits and

2 For example, CMMI-defined process and practices (Chapter 28).

344 PART 3 QUALITY AND SECURITY

reviews to be conducted, standards that are applicable to the project, proce-
dures for error reporting and tracking, work products that are produced by the
SQA group, and feedback that will be provided to the software team.
Participates in the development of the project’s software process
description. The software team selects a process for the work to be
performed. The SQA group reviews the process description for compliance
with organizational policy, internal software standards, externally imposed
standards (e.g., ISO-9001), and other parts of the software project plan.
Reviews software engineering activities to verify compliance with the
defined software process. The SQA group identifies, documents, and tracks
deviations from the process and verifies that corrections have been made.
Audits designated software work products to verify compliance with those
defined as part of the software process. The SQA group reviews selected
work products; identifies, documents, and tracks deviations; verifies that cor-
rections have been made; and periodically reports the results of its work to
the project manager.
Ensures that deviations in software work and work products are docu-
mented and handled according to a documented procedure. Deviations
may be encountered in the project plan, process description, applicable
standards, or software engineering work products.
Records any noncompliance, and reports to senior management.
Noncompliance items are tracked until they are resolved.

In addition to these activities, the SQA group coordinates the control and manage-
ment of change (Chapter 22) and helps to collect and analyze software metrics.

Figure 17.2
Software
quality
assurance

Analyze the
Assessment Software

Quality
Assurance

Process

Compare
Resu

lts

Prepare
Criteria

Identify
Indicators

Identify a
Problem Area

Take Measures
to Improve

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 345

17.4.2 Goals, Attributes, and Metrics
The SQA activities described in the preceding section are performed to achieve a set
of pragmatic goals:

Requirements quality. The correctness, completeness, and consistency of
the requirements model will have a strong influence on the quality of all work
products that follow. SQA must ensure that the software team has properly
reviewed the requirements model to achieve a high level of quality.
Design quality. Every element of the design model should be assessed by
the software team to ensure that it exhibits high quality and that the design
itself conforms to requirements. SQA looks for attributes of the design that
are indicators of quality.
Code quality. Source code and related work products (e.g., other descriptive
information) must conform to local coding standards and exhibit characteris-
tics that will facilitate maintainability. SQA should isolate those attributes that
allow a reasonable analysis of the quality of code.

Software Quality Assurance

The scene: Doug Miller’s office
as the SafeHome software
project begins.

The players: Doug Miller, manager of the
SafeHome software engineering team, and
other members of the product software
engineering team.

The conversation:
Doug: How are things going with the informal
reviews?

Jamie: We’re conducting informal reviews of
the critical project elements in pairs as we
code but before testing. It’s going faster than
I thought.

Doug: That’s good, but I want to have Bridget
Thorton’s SQA group conduct audits of our
work products to ensure that we’re following
our processes and meeting our quality goals.

Venod: Aren’t they already doing the bulk of
the testing?

Doug: Yes, they are. But QA is more than test-
ing. We need to be sure that our documents
are evolving along with our code and that

we’re making sure we don’t introduce errors
as we integrate new components.
Jamie: I really don’t want to be evaluated
based on their findings.
Doug: No worries. The audits are focused on
conformance of our work products to the re-
quirements and the process activities we’ve
defined. We’ll only be using audit results to try
to improve our processes as well as our
software products.
Vinod: I must believe it’s going to take more
of our time.
Doug: In the long run it will save us time when
we find defects earlier. It also costs less to fix
defects if they’re caught early.
Jamie: That sounds like a good thing then.
Doug: It’s also important to identify the activi-
ties where defects were introduced and add
review tasks to catch them in the future.
Vinod: That’ll help us determine if we’re sam-
pling carefully enough with our review activities.
Doug: I think SQA activities will make us a
better team in the long run.

safehome

346 PART 3 QUALITY AND SECURITY

Quality control effectiveness. A software team should apply limited
resources in a way that has the highest likelihood of achieving a high-quality
result. SQA analyzes the allocation of resources for reviews and testing to
assess whether they are being allocated in the most effective manner.

Table 17.1 (adapted from [Hya96]) identifies the attributes that are indicators for
the existence of quality for each of the goals discussed. Metrics that can be used to
indicate the relative strength of an attribute are also shown.

Table 17.1
Software
quality goals,
attributes,
and metrics
Source: Adapted
from Hyatt, L.,
and Rosenberg, L.,
“A Software
Quality Model
and Metrics for
Identifying Project
Risks and As-
sessing Software
Quality,” NASA
SATC, 1996.

Goal Attribute Metric

Requirement
quality

Ambiguity Number of ambiguous modifiers
(e.g., many, large, human-friendly)

Completeness Number of TBA, TBD

Understandability Number of sections/subsections

Volatility Number of changes per requirement

Time (by activity) when change is requested

Traceability Number of requirements not traceable to
design/code

Model clarity Number of UML models

Number of descriptive pages per model

Number of UML errors

Design quality Architectural integrity Existence of architectural model

Component completeness Number of components that trace to
architectural model

Complexity of procedural design

Interface complexity Average number of pick to get to a typical
function or content

Layout appropriateness

Patterns Number of patterns used

Code quality Complexity Cyclomatic complexity

Maintainability Design factors

Understandability Percent internal comments

Variable naming conventions

Percent reused components

Reusability Percent reused component

Documentation Readability index

QC effectiveness Resource allocation Staff hour percentage per activity

Completion rate Actual vs. budgeted completion time

Review effectiveness See review metrics

Testing effectiveness Number of errors found and criticality

Effort required to correct an error

Origin of error

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 347

17.5 fo r m a L ap p roac h e s to sQa
In the preceding sections, we have argued that software quality is everyone’s job and
that it can be achieved through competent software engineering practice as well as
through the application of technical reviews, a multitiered testing strategy, better con-
trol of software work products and the changes made to them, and the application of
accepted software engineering standards and process frameworks. In addition, quality
can be defined in terms of a broad array of quality attributes and measured (indirectly)
using a variety of indices and metrics.

Over the past three decades, a small, but vocal, segment of the software engineer-
ing community has argued that a more formal approach to software quality assurance
is required. It can be argued that a computer program is a mathematical object. A
rigorous syntax and semantics can be defined for every programming language, and
a rigorous approach to the specification of software requirements is available. If the
requirements model (specification) and the programming language can be represented
in a rigorous manner, it should be possible to apply mathematic proof of correctness
to demonstrate that a program conforms exactly to its specifications.

Attempts to prove programs correct are not new. Dijkstra [Dij76a] and Linger,
Mills, and Witt [Lin79], among others, advocated proofs of program correctness and
tied these to the use of structured programming concepts. Although formal methods
are interesting to some software engineering researchers, commercial developers rarely
make use of formal methods in 2018.

17.6 stat i st i ca L so f t wa r e Qua L i t y as s u r a nc e

Statistical quality assurance reflects a growing trend throughout the industry to become
more quantitative about quality. For software, statistical quality assurance implies the
following steps:

1. Information about software errors and defects is collected and categorized.
2. An attempt is made to trace each error and defect to its underlying cause

(e.g., nonconformance to specifications, design error, violation of stan-
dards, poor communication with the customer).

3. Using the Pareto principle (80 percent of the defects can be traced to 20 percent
of all possible causes), isolate the 20 percent (the vital few).

4. Once the vital few causes have been identified, move to correct the problems
that have caused the errors and defects.

This relatively simple concept represents an important step toward the creation of
an adaptive software process in which changes are made to improve those elements
of the process that introduce error.

17.6.1 A Generic Example
To illustrate the use of statistical methods for software engineering work, assume that
a software engineering organization collects information on errors and defects for a
period of 1 year. Some of the errors are uncovered as software is being developed.
Other defects are encountered after the software has been released to its end users.

348 PART 3 QUALITY AND SECURITY

Although hundreds of different problems are uncovered, all can be tracked to one
(or more) of the following causes:

∙ Incomplete or erroneous specifications (IES)
∙ Misinterpretation of customer communication (MCC)
∙ Intentional deviation from specifications (IDS)
∙ Violation of programming standards (VPS)
∙ Error in data representation (EDR)
∙ Inconsistent component interface (ICI)
∙ Error in design logic (EDL)
∙ Incomplete or erroneous testing (IET)
∙ Inaccurate or incomplete documentation (IID)
∙ Error in programming language translation of design (PLT)
∙ Ambiguous or inconsistent human/computer interface (HCI)
∙ Miscellaneous (MIS)

To apply statistical SQA, a table is built (see Table 17.2). The table indicates that IES,
MCC, and EDR are the vital few causes that account for 53 percent of all errors. It should
be noted, however, that IES, EDR, PLT, and EDL would be selected as the vital few
causes if only serious errors are considered. Once the vital few causes are determined, the
software engineering organization can begin corrective action. For example, to correct
MCC, you might implement requirements gathering techniques (Chapter 7) to improve
the quality of customer communication and specifications. To improve EDR, you might
acquire tools for data modeling and perform more stringent data design reviews.

Table 17.2
Data
collection for
statistical
SQA

Total Serious Moderate Minor

Error No. % No. % No. % No. %

IES 205 22% 34 27% 68 18% 103 24%

MCC 156 17% 12 9% 68 18% 76 17%

IDS 48 5% 1 1% 24 6% 23 5%

VPS 25 3% 0 0% 15 4% 10 2%

EDR 130 14% 26 20% 68 18% 36 8%

ICI 58 6% 9 7% 18 5% 31 7%

EDL 45 5% 14 11% 12 3% 19 4%

IET 95 10% 12 9% 35 9% 48 11%

IID 36 4% 2 2% 20 5% 14 3%

PLT 60 6% 15 12% 19 5% 26 6%

HCI 28 3% 3 2% 17 4% 8 2%

MIS 56 6% 0 0% 15 4% 41 9%

Totals 942 100% 128 100% 379 100% 435 100%

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 349

It is important to note that corrective action focuses primarily on the vital few.
As the vital few causes are corrected, new candidates pop to the top of the stack.

Statistical quality assurance techniques for software have been shown to provide
substantial quality improvement (e.g., [Rya11], [Art97]). In some cases, software orga-
nizations have achieved a 50 percent reduction per year in defects after applying these
techniques.

The application of the statistical SQA and the Pareto principle can be summarized
in a single sentence: Spend your time focusing on things that really matter, but first
be sure that you understand what really matters!

17.6.2 Six Sigma for Software Engineering
Six Sigma is the most widely used strategy for statistical quality assurance in industry
today. Originally popularized by Motorola in the 1980s, the Six Sigma strategy “is a
business-management strategy designed to improve the quality of process outputs by
minimizing variation and causes of defects in processes. It is a subset of the Total
Quality Management (TQM) methodology with a heavy focus on statistical applica-
tions used to reduce costs and improve quality” [Voe18]. The term Six Sigma is
derived from six standard deviations—3.4 instances (defects) per million occurrences—
implying an extremely high-quality standard. The Six Sigma methodology defines
three core steps:

∙ Define customer requirements and deliverables and project goals via
well-defined methods of customer communication.

∙ Measure the existing process and its output to determine current quality
performance (collect defect metrics).

∙ Analyze defect metrics and determine the vital few causes.

If an existing software process is in place, but improvement is required, Six Sigma
suggests two additional steps:

∙ Improve the process by eliminating the root causes

of defects.

∙ Control the process to ensure that future work does not reintroduce the causes

of defects.

These core and additional steps are sometimes referred to as the DMAIC (define,
measure, analyze, improve, and control) method.

If an organization is developing a software process (rather than improving an
existing process), the core steps are augmented as follows:

∙ Design the process to (1) avoid the root causes of defects and (2) to meet
customer requirements.

∙ Verify that the process model will, in fact, avoid defects and meet customer
requirements.

This variation is sometimes called the DMADV (define, measure, analyze, design,
and verify) method.

A comprehensive discussion of Six Sigma is best left to resources dedicated to the
subject. If you have further interest, see [Voe18], [Pyz14], and [Sne18].

350 PART 3 QUALITY AND SECURITY

17.7 so f t wa r e re L i a B i L i t y

There is no doubt that the reliability of a computer program is an important element
of its overall quality. If a program repeatedly and frequently fails to perform, it mat-
ters little whether other software quality factors are acceptable.

Software reliability, unlike many other quality factors, can be measured directly
and estimated using historical and developmental data. Software reliability is defined
in statistical terms as “the probability of failure-free operation of a computer program
in a specified environment for a specified time” [Mus87]. To illustrate, program X is
estimated to have a reliability of 0.999 over eight elapsed processing hours. In other
words, if program X were to be executed 1000 times and require a total of 8 hours
of elapsed processing time (execution time), it is likely to operate correctly (without
failure) 999 times.

Whenever software reliability is discussed, a pivotal question arises: What is
meant by the term failure? In the context of any discussion of software quality and
reliability, failure is nonconformance to software requirements. Yet, even within this
definition, there are gradations. Failures can be only annoying or catastrophic. One
failure can be corrected within seconds, while another requires weeks or even
months to correct. Complicating the issue even further, the correction of one failure
may in fact result in the introduction of other errors that ultimately result in other
failures.

17.7.1 Measures of Reliability and Availability
Early work in software reliability attempted to extrapolate the mathematics of hard-
ware reliability theory to the prediction of software reliability. Most hardware-related
reliability models are predicated on failure due to wear rather than failure due to
design defects. In hardware, failures due to physical wear (e.g., the effects of tem-
perature, corrosion, shock) are more likely than a design-related failure. Unfortunately,
the opposite is true for software. In fact, all software failures can be traced to design
or implementation problems; wear (see Chapter 1) is not a factor.

There has been an ongoing debate over the relationship between key concepts in
hardware reliability and their applicability to software. Although an irrefutable link
has yet to be established, it is worthwhile to consider a few simple concepts that apply
to both system elements.

If we consider a computer-based system, a simple measure of reliability is mean
time between failure (MTBF):3

MTBF = MTTF + MTTR

where the acronyms MTTF and MTTR are mean time to failure and mean time to
repair,4 respectively.

3 It is important to note that MTBF and related measures are based on CPU time, not wall
clock time.

4 Although debugging (and related corrections) may be required following a failure, in many
cases the software will work properly after a restart with no other change.

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 351

Many researchers argue that MTBF is a far more useful measure than other
quality-related software metrics discussed in Chapter 23. Stated simply, an end user
is concerned with failures, not with the total defect count. Because each defect contained
within a program does not have the same failure rate, the total defect count provides
little indication of the reliability of a system. For example, consider a program that has
been in operation for 3000 processor hours without failure. Many defects in this pro-
gram may remain undetected for tens of thousands of hours before they are discovered.
The MTBF of such obscure errors might be 30,000 or even 60,000 processor hours.
Other defects, not yet discovered, might have a failure rate of 4000 or 5000 hours. Even
if every one of the first category of errors (those with long MTBF) is removed, the
impact on software reliability is negligible.

However, MTBF can be problematic for two reasons: (1) it projects a time span
between failures but does not provide us with a projected failure rate, and (2) MTBF
can be misinterpreted to mean average life span even though this is not what it
implies.

An alternative measure of reliability is failures in time (FIT)—a statistical measure
of how many failures a component will have over 1 billion hours of operation. There-
fore, 1 FIT is equivalent to one failure in every billion hours of operation.

In addition to a reliability measure, you should also develop a measure of avail-
ability. Software availability is the probability that a program is operating according
to requirements at a given point in time and is defined as

Availability =
MTTF

MTTF + MTTR
× 100%

The MTBF reliability measure is equally sensitive to MTTF and MTTR. The avail-
ability measure is somewhat more sensitive to MTTR, an indirect measure of the
maintainability of software. Of course, some aspects of availability have nothing to
do with failure. For example, scheduling downtime (for support functions) causes the
software to be unavailable. For a comprehensive discussion of software reliability
measures, see [Laz11].

17.7.2 Use of AI to Model Reliability
Some software engineers view data science as the application of artificial intelli-
gence techniques to solve software engineering problems. One of the things artificial
intelligence methods attempt to do is provide reasonable solutions to problems
where the needed data may be incomplete. Software reliability is defined as the
probability of failure-free software operation for a specified time period in a spec-
ified environment. This means that we can never know the exact moment when a
software product will fail because we will never have the complete data needed to
calculate the probability.

Software engineers have used statistical techniques based on Bayes’ theorem5 in
quantitative decision-making situations for several years. Bayesian inference is a
method of statistical inference in which Bayes’ theorem is used to update the probability

5 Bayes’ theorem for conditional probabilities is P(A|B) = (P(B|A) * P(A)) / P(B). For more
details, see http://www.statisticshowto.com/bayes-theorem-problems/.

352 PART 3 QUALITY AND SECURITY

for a hypothesis (such as system reliability) as more evidence or information becomes
available. Bayesian inference can be used to estimate probabilistic quantities using
historic data even when some information is missing. Using Bayesian techniques has
allowed for real-time solutions to probability estimation problems that are beyond
human reasoning [Tos17].

Proactive failure prediction using machine learning was discussed briefly in
Section 15.4.3. It would be nice to be able to predict system failures in subsequent
sprints before you deliver the prototype being developed in the current sprint. Mak-
ing use of predictive data analytics such as a regression model involving MTBF has
been used to estimate where and what types of defects might occur in future
prototypes [Bat18].

A genetic algorithm is a heuristic search method used in artificial intelligence and
computing. It is used for finding near-optimal solutions to search problems based on
the theory of natural selection and evolutionary biology. Genetic algorithms can be
used to grow reliability models by discovering relationships in historic system data.
These models have been used to identify software components that may fail in the
future. Sometimes these models have been created based on metrics estimated from
UML models before any codes has been written [Pad17]. This type of work is very
important to developers interested in refactoring software products or reusing software
components in other products.

17.7.3 Software Safety
Software safety is a software quality assurance activity that focuses on the identifica-
tion and assessment of potential hazards that may affect software negatively and cause
an entire system to fail. If hazards can be identified early in the software process,
software design features can be specified that will either eliminate or control potential
hazards.

A modeling and analysis process is conducted as part of software safety. Ini-
tially, hazards are identified and categorized by criticality and risk. For example,
some of the hazards associated with a computer-based cruise control for an auto-
mobile might be: (1) causes uncontrolled acceleration that cannot be stopped,
(2) does not respond to depression of brake pedal (by turning off), (3) does not
engage when switch is activated, and (4) slowly loses or gains speed. Once these
system-level hazards are identified, analysis techniques are used to assign severity
and probability of occurrence.6 To be effective, software must be analyzed in the
context of the entire system. For example, a subtle user input error (people are
system components) may be magnified by a software fault to produce control data
that improperly positions a mechanical device. If and only if a set of external
environmental conditions is met, the improper position of the mechanical device
will cause a disastrous failure. Analysis techniques [Eri15] such as fault tree anal-
ysis, real-time logic, and Petri net models can be used to predict the chain of events
that can cause hazards and the probability that each of the events will occur to
create the chain.

6 This approach is similar to the risk analysis methods described in Chapter 26. The primary
difference is the emphasis on technology issues rather than project-related topics.

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 353

Once hazards are identified and analyzed, safety-related requirements can be spec-
ified for the software. That is, the specification can contain a list of undesirable events
and the desired system responses to these events. The role of software in managing
undesirable events is then indicated.

Although software reliability and software safety are related to one another, it is
important to understand the subtle difference between them. Software reliability uses
statistical analysis to determine the likelihood that a software failure will occur. How-
ever, the occurrence of a failure does not necessarily result in a hazard or mishap.
Software safety examines the ways in which failures result in conditions that can lead
to a mishap. That is, failures are not considered in a vacuum but are evaluated in the
context of an entire computer-based system and its environment.

A comprehensive discussion of software safety is beyond the scope of this book.
If you have further interest in software safety and related system issues, see [Fir13],
[Har12a], and [Lev12].

17.8 th e iso 9000 Qua L i t y sta n da r d s 7

A quality assurance system may be defined as the organizational structure, responsi-
bilities, procedures, processes, and resources for implementing quality management
[ANS87]. Quality assurance systems are created to help organizations ensure their
products and services satisfy customer expectations by meeting their specifications.
These systems cover a wide variety of activities encompassing a product’s entire life
cycle including planning, controlling, measuring, testing, and reporting, and improv-
ing quality levels throughout the development and manufacturing process. ISO 9000
describes quality assurance elements in generic terms that can be applied to any busi-
ness regardless of the products or services offered.

To become registered to one of the quality assurance system models contained in
ISO 9000, a company’s quality system and operations are scrutinized by third-party
auditors for compliance to the standard and for effective operation. Upon successful
registration, a company is issued a certificate from a registration body represented by
the auditors. Semiannual surveillance audits ensure continued compliance to the
standard.

The requirements delineated by ISO 9001:2015 address topics such as management
responsibility, quality system, contract review, design control, document and data con-
trol, product identification and traceability, process control, inspection and testing,
corrective and preventive action, control of quality records, internal quality audits,
training, servicing, and statistical techniques. For a software organization to become
registered to ISO 9001:2015, it must establish policies and procedures to address each
of the requirements just noted (and others) and then be able to demonstrate that these
policies and procedures are being followed. If you desire further information on ISO
9001:2015, see [ISO14].

7 This section, written by Michael Stovsky, has been adapted from Fundamentals of ISO 9000,
a workbook developed for Essential Software Engineering, a video curriculum developed
by R. S. Pressman & Associates, Inc. Reprinted with permission.

354 PART 3 QUALITY AND SECURITY

17.9 th e sQa pL a n

The SQA plan provides a road map for instituting software quality assurance. Devel-
oped by the SQA group (or by the software team if an SQA group does not exist),
the plan serves as a template for SQA activities that are instituted for each software
project.

A standard for SQA plans has been published by the IEEE [IEE17]. The standard
recommends a structure that identifies: (1) the purpose and scope of the plan, (2) a
description of all software engineering work products (e.g., models, documents, source
code) that fall within the purview of SQA, (3) all applicable standards and practices
that are applied during the software process, (4) SQA actions and tasks (including
reviews and audits) and their placement throughout the software process, (5) the tools
and methods that support SQA actions and tasks, (6) software configuration manage-
ment (Chapter 22) procedures, (7) methods for assembling, safeguarding, and main-
taining all SQA-related records, and (8) organizational roles and responsibilities
relative to product quality.

The ISO 9001:2015 Standard
The following outline defines the basic
elements of the ISO 9001:2015 standard.

Comprehensive information on the standard can
be obtained from the International Organization
for Standardization (www.iso.ch) and other Internet
sources (e.g., www.praxiom.com).

Establish the elements of a quality management
system.

Develop, implement, and improve the
system.

Define a policy that emphasizes the
importance of the system.

Document the quality system.

Describe the process.

Produce an operational manual.

Develop methods for controlling (updating)
documents.

Establish methods for record keeping.

Support quality control and assurance.

Promote the importance of quality among all
stakeholders.

Focus on customer satisfaction.

Define a quality plan that addresses objectives,
responsibilities, and authority.

Define communication mechanisms among
stakeholders.

Establish review mechanisms for the quality
management system.

Identify review methods and feedback
mechanisms.

Define follow-up procedures.

Identify quality resources including personnel,
training, and infrastructure elements.

Establish control mechanisms.

For planning.

For customer requirements.

For technical activities (e.g., analysis, design,
testing).

For project monitoring and management.

Define methods for remediation.

Assess quality data and metrics.

Define approach for continuous process and
quality improvement.

info

CHAPTER 17 SOFTWARE QUALITY ASSURANCE 355

17.10 Su m m a ry

Software quality assurance is a software engineering umbrella activity that is applied
at each step in the software process. SQA encompasses procedures for the effective
application of methods and tools, oversight of quality control activities such as tech-
nical reviews and software testing, procedures for change management, procedures for
assuring compliance to standards, and measurement and reporting mechanisms.

To properly conduct software quality assurance, data about the software engineer-
ing process should be collected, evaluated, and disseminated. Statistical SQA helps
to improve the quality of the product and the software process itself. Software reli-
ability models extend measurements, enabling collected defect data to be extrapolated
into projected failure rates and reliability predictions.

In summary, you should note the words of Dunn and Ullman [Dun82]: “Software
quality assurance is the mapping of the managerial precepts and design disciplines of
quality assurance onto the applicable managerial and technological space of software
engineering.” The ability to ensure quality is the measure of a mature engineering
discipline. When the mapping is successfully accomplished, mature software engineer-
ing is the result.

Pro b l e m S a n d Po i n t S to Po n d e r

17.1. Some people say that “variation control is the heart of quality control.” Because every
program that is created is different from every other program, what are the variations that we
look for and how do we control them?

17.2. Is it possible to assess the quality of software if the customer keeps changing what it is
supposed to do?

17.3. Quality and reliability are related concepts but are fundamentally different in a number
of ways. Discuss the differences.

17.4. Can a program be correct and still not be reliable? Explain.

17.5. Can a program be correct and still not exhibit good quality? Explain.

17.6. Why is there often tension between a software engineering group and an independent
software quality assurance group? Is this healthy?

17.7. You have been given the responsibility for improving the quality of software across your
organization. What is the first thing that you should do? What’s next?

17.8. Besides counting errors and defects, are there other countable characteristics of software
that imply quality? What are they and can they be measured directly?

17.9. The MTBF concept for reliability is open to criticism. Explain why.

17.10. Consider two safety-critical systems that are controlled by computer. List at least three
hazards for each that can be directly linked to software failures.

Design element: Quick Look icon magnifying glass: © Roger Pressman

356

Software Security
Engineering
Contributed by: Nancy Mead Carnegie Mellon University
Software Engineering Institute

18
C H A P T E R

What is it? Software security engineering en-
compasses a set of techniques that can im-
prove the security of software while it is under
development.

Who does it? Although software engineers do
not need to become security experts, they
need to collaborate with security experts. The
security experts may be members of the soft-
ware team, on a separate specialized team, or
they may be outside consultants.

Why is it important? The media continually re-
ports instances of hacking—whether by gang-
sters, corporate competitors, hostile nations,
or any other bad actor. The consequences for
critical infrastructure, financial institutions,
health care, and all aspects of modern life are
significant.

What are the steps? There are a number of
steps that can be taken to ensure that software
is secure. We will discuss some of them here
and provide pointers to resources for further
exploration.

What is the work product? As you will see,
there are many work products that are devel-
oped in the process of secure software engi-
neering. The ultimate work product, or course,
is the software that you have developed using
secure software engineering practices.

How do I ensure that I’ve done it right?
Everything that we will discuss as a method to
improve software security, whether at the or-
ganizational or project level, can and should
be reviewed by the interested parties. In addi-
tion, secure development processes can be
improved, if they are found to be lacking.

Q u i c k L o o k

attack patterns . . . . . . . . . . . . . . . . . . . . . . . .363
attack surface . . . . . . . . . . . . . . . . . . . . . . . . .366
maturity models . . . . . . . . . . . . . . . . . . . . . . .370
measurement . . . . . . . . . . . . . . . . . . . . . . . . .368
misuse and abuse cases . . . . . . . . . . . . . . . . .363
requirements engineering . . . . . . . . . . . . . . .360
secure coding . . . . . . . . . . . . . . . . . . . . . . . . .367
secure development life-cycle activities . . . .359

security life-cycle models . . . . . . . . . . . . . . . .357
security process improvement . . . . . . . . . . . .370
security risk analysis . . . . . . . . . . . . . . . . . . . .364
software security engineering,
importance of . . . . . . . . . . . . . . . . . . . . . . . . .357
threat modeling, prioritization,
and mitigation . . . . . . . . . . . . . . . . . . . . . . . . .365

k e y
c o n c e p t s

Stop and take a look around. Where do you see software being deployed? Sure,
it’s in your laptop, tablet, and cell phone. What about your appliances—refrigera-
tor, dishwasher, and so forth? How about your car? Financial transactions—ATM,
online banking, financial software, tax software? Your supplier of electricity?
Definitely using software. Do you have any wearable devices on? A fit-bit? Maybe
you have medical devices, like a pacemaker. The bottom line is: Software is all
around us, on us, and sometimes in us. Every software product has the potential
to be hacked, sometimes with dire consequences. This is the reason that we, as
software engineers, need to be concerned about software security.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 357

18.1 Wh y so f t Wa r e se c u r i t y eng i n e e r i ng
is im p o rta n t

Software security is about more than just securing operational software with firewalls,
strong passwords, and encryption. It’s also about developing software in such a way
that it is more secure from the outset. Techniques are now available that will help us
develop software that is significantly more secure than it would be otherwise.

In this chapter, we’re going to look as some of the models and techniques that can
help us achieve better software security. We’ll start by looking at security process
models. Then we’ll examine specific process activities, including such activities as
requirements engineering, misuse or abuse cases, security risk analysis, threat model-
ing, attack surface, secure coding, and measurement. We’ll also consider security
process improvement models. Finally, we’ll summarize and provide you with a list of
references so that you can dive into any of these topics in more depth.

Software security engineering research is a very active area. In this book we’re
only providing an overview of methods and tools to support actual practice. There are
many books (for example, [Mea16], [Shu13], and [Hel18]) and other resources devoted
exclusively to software security engineering, and we will point you to some of them.

18.2 se c u r i t y Li f e-cyc L e mo d e L s

The Microsoft Security Development Lifecycle (SDL) [Mea16] [Mic18] is an industry-
leading software security process. A Microsoft-wide initiative and a mandatory policy
since 2004, the SDL enabled Microsoft to embed security and privacy in its software
and culture. The SDL introduces security and privacy early and throughout all phases
of the development process and is without question the most widely known and used
security development life-cycle model.

Microsoft defined a collection of principles it calls Secure by Design, Secure by
Default, Secure in Deployment, and Communications (SD3+C) to help determine
where security efforts are needed. These are as follows [Mic10]:

Secure by Design
Secure architecture, design, and structure. Developers consider security issues part of

the basic architectural design of software development. They review detailed designs
for possible security issues, and they design and develop mitigations for all threats.

Threat modeling and mitigation. Threat models are created, and threat mitigations are
present in all design and functional specifications.

Elimination of vulnerabilities. No known security vulnerabilities that would present a sig-
nificant risk to the anticipated use of the software remain in the code after review. This
review includes the use of analysis and testing tools to eliminate classes of vulnerabilities.

Improvements in security. Less secure legacy protocols and code are deprecated, and,
where possible, users are provided with secure alternatives that are consistent with
industry standards.

Secure by Default
Least privilege. All components run with the fewest possible permissions.
Defense in depth. Components do not rely on a single threat mitigation solution that leaves

users exposed if it fails.

358 PART THREE QUALITY AND SECURITY

Conservative default settings. The development team is aware of the attack surface for the
product and minimizes it in the default configuration.

Avoidance of risky default changes. Applications do not make any default changes
to the operating system or security settings that reduce security for the host com-
puter. In some cases, such as for security products, it is acceptable for a software
program to strengthen (increase) security settings for the host computer. The most
common violations of this principle are games that either open firewall ports without
informing the user or instruct users to open firewall ports without informing users of
possible risks.

Less commonly used services off by default. If fewer than 80 percent of a program’s
users use a feature, that feature should not be activated by default. Measuring
80 percent usage in a product is often difficult because programs are designed for
many different personas. It can be useful to consider whether a feature addresses a
core/primary use scenario for all personas. If it does, the feature is sometimes referred
to as a P1 feature.

Secure in Deployment
Deployment guides. Prescriptive deployment guides outline how to deploy each feature

of a program securely, including providing users with information that enables them
to assess the security risk of activating non-default options (and thereby increasing the
attack surface).

Analysis and management tools. Security analysis and management tools enable admin-
istrators to determine and configure the optimal security level for a software release.

Patch deployment tools. Deployment tools aid in patch deployment.

Communications
Security response. Development teams respond promptly to reports of security vulner-

abilities and communicate information about security updates.
Community engagement. Development teams proactively engage with users to answer

questions about security vulnerabilities, security updates, or changes in the security
landscape.

The secure software development process model looks like the one shown in
Figure 18.1.

The Microsoft SDL documentation describes what architects, designers, develop-
ers, and testers are required to do for each of the 16 recommended practices. The data
that Microsoft collected after implementing the SDL shows a significant reduction in

Figure 18.1 Secure Software Development Process Model at Microsoft
Adapted from Shunn, A., et al. Strengths in Security Solutions, Software Engineering Institute, Carnegie
Mellon University, 2013. Available at http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=77878.

Training Design Implementation VerificationRequirements Release Response

Core
Security
Training

Execute
Incident

Response
Plan

Established
Security

Requirements

Create Quality
Gates/Bug Bars

Security &
Privacy Risk
Assessment

Established
Design

Requirements

Analyze Attack
Surface

Threat Modeling

Use Approved
Tools

Deprecate
Unsafe

Functions

Static Analysis

Dynamic
Analysis

Fuzz Testing

Attack Surface
Review

Incident
Response Plan

Final Security
Review

Release
Archive

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 359

vulnerabilities, which led to a need for fewer patches, and thus a significant cost
savings. We recommend that you browse the SDL website to learn more about these
practices. Since the SDL was developed, there have been numerous papers, books,
training, and so on, to go with the SDL model.1

18.3 se c u r e de v e L o p m e n t Li f e-cyc L e ac t i v i t i e s

A different approach that is independent of a life-cycle model is the touchpoints for
software security [McG06], which argues that the activities (touchpoints) are what
matter, not the model. The activities can be incorporated into any life-cycle model,
and thus are considered to be process agnostic. The touchpoints later formed the basis
for BSIMM, a maturity model that will be discussed later in this chapter. Some orga-
nizations consider the touchpoints to be the minimum set of activities that should be
performed in secure software development. A pictorial version of the touchpoints is
shown in Figure 18.2. In this diagram, the recommended security activities appear
above the corresponding software development activity, or life-cycle phase:

With the SDL and touchpoints in mind, we’ll look at some of the important secure
software development activities that are associated with them.

1 More recently, Microsoft has shown how the SDL activities can be integrated with an agile
development approach: https://www.microsoft.com/en-us/SDL/Discover/sdlagile.aspx.

Figure 18.2
Software
security
touchpoints

Investor
Feedback

Tests and
Test Results

Code

Test Plans

Architecture
and Design

Requirements
and Use Cases

External Review

Abuse
Cases

Security
Requirements

Risk-Based
Security Tests

Security
Operations

Penetration
Testing

Code Review
(Tools)

Risk
Analysis

360 PART THREE QUALITY AND SECURITY

18.4 se c u r i t y re Q u i r e m e n t s eng i n e e r i ng

Although security requirements are an important part of secure software develop-
ment, in practice they are often neglected. When they exist, they are often an add-
on, copied from a generic list of security features. The requirements engineering
that is needed to get a better set of security requirements seldom takes place [All08].

Requirements engineering practice typically addresses desired user features. There-
fore, attention is given to the functionality of the system from the user’s perspective,
but little attention is given to what the system should not do [Bis02]. Users expect
systems to be secure, and these assumptions need to make their way into security
requirements for software systems before they are developed, not after the fact. Often
the users’ assumptions about security are overlooked because system features are the
primary focus.

In addition to security life-cycle models, there are many process models that are
specific to security requirements. These include: core security requirements artifacts
[Mof04], Software Cost Reduction (SCR) [Hei02], SQUARE (Security QUAlity
Requirements Engineering) [Mea05], and Security Requirements Engineering Process
(SREP) [Mel06]. For the remainder of this section, we’ll consider SQUARE as a
representative example of security life-cycle models.

18.4.1 SQUARE
SQUARE is a representative security requirements engineering process model, but
it’s important to keep in mind that if you already have a development process model,
such as the one presented in Chapter 4, you can just pick up some of the SQUARE
steps to enhance your existing model. There’s no need to develop a whole new
process to address security in your software development activities. We suggest that
you add security definitions to your glossary; perform risk analysis, including iden-
tification of potential attacks via misuse cases or threat modeling; develop mitigation
strategies; and perform categorization and prioritization of candidate security
requirements.

The SQUARE process model provides for eliciting, categorizing, and prioritizing
security requirements for software-intensive systems. Its focus is to build security
concepts into the early stages of the development life cycle. It can also be used for
fielded systems and those undergoing improvements and modifications. The process
is shown in Table 18.1, followed by brief descriptions of each step.

18.4.2 The SQUARE Process
The SQUARE process is best applied by the project’s requirements engineers and
security experts with supportive executive management and stakeholders. Let’s take a
look at the steps.2

Step 1. Agree on definitions. So that there is not semantic confusion, this
step is needed as a prerequisite to security requirements engineering. On a

2 To dig deeper, see the resources available at the SEPA 9/e website.

361

N
um

be
r

St
ep

In
pu

t
Te

ch
ni

qu
es

Pa
rt

ic
ip

an
ts

O
ut

pu
t

1
A

gr
ee

o
n

de
fin

iti
on

s

.
C

an
di

da
te

d
efi

ni
tio

ns
f

ro
m

IE

EE
a

nd
o

th

er

s
ta

nd
ar

d

s
St

ru
ct

ur
ed

in
te

rv
ie

w
s,

f
oc

us
g

ro
up

St
ak

eh
ol

de
rs

,

re

qu
ire

m
en

ts
t

ea
m

A
gr

ee
d-

to
d

efi
ni

tio
ns

2
Id

en
tif

y
as

se
ts

a
nd

se

cu
rit

y
go

al
s.

D
efi

ni
tio

ns
, c

an
di

da
te

a
ss

et
s

an
d

go
al

s,
b

us
in

es
s

dr
iv

er
s,

po

lic
ie

s

an

d
pr

oc
ed

ur
es

,
ex

a

m
pl

es

Fa
ci

lit
at

ed
w

or
k

se
ss

io
n,

s
ur

ve
ys

,
in

te
rv

ie
w

s
St

ak
eh

ol
de

rs
,

re
qu

i

re
m

en
ts

en

gi
ne

er

A
ss

et
s

an
d

se
cu

rit
y

go
al

s

3
D

ev
el

op
a

rti
fa

ct
s

to

su
pp

or
t

se
cu

rit
y

re
qu

ire
m

en
ts

d
efi

ni
tio

n.

Po
te

nt
ia

l a
rti

fa
ct

s
(e

.g
.,

sc
en

ar
io

s,
m

is
us

e
ca

se
s,

te

m
pl

at
es

, f
or

m
s)

W
or

k
se

ss
io

n
Re

qu
ire

m
en

ts

en
gi

ne
er

N
ee

de
d

ar
tif

ac
ts

:

s
ce

na
rio

s,

m
is

us
e

ca
se

s,
m

od
el

s,

te
m

pl
at

es
, f

or
m

s

4
Pe

r

fo
rm

(s
ec

ur
ity

) r
is

k
as

se
ss

m
en

t.
M

is
us

e
ca

se
s,

s
ce

na
rio

s,

se
cu

rit
y

go
al

s
Ri

sk
a

ss
es

sm
en

t
m

et
ho

d,
a

na
ly

si
s

of

an
tic

ip
at

ed
r

is
k

ag
ai

ns
t

or
ga

ni
za

t

io
na

l
ris

k
to

le
ra

nc
e,

in
cl

ud
in

g
th

re
at

a
na

ly
si

s

Re
qu

ire
m

en
ts

en

gi
ne

er
, r

is
k

ex
pe

rt,

st
ak

eh
ol

de
rs

Ri
sk

a
ss

es
sm

en
t

re
su

lts

5
Se

le
ct

e
lic

ita
tio

n
te

ch
ni

qu
es

.
G

oa
ls

, d
efi

ni
tio

ns
, c

an
di

da
te

te

ch
ni

qu
es

, e
xp

er
tis

e
of

st

ak
eh

ol
de

rs
, o

rg
an

iz
at

io
na

l
st

yl
e,

c
ul

tu
re

, l
ev

el
o

f
se

cu
rit

y
ne

ed
ed

, c
os

t-b
en

efi
t

an
al

ys
is

,
et

c.

W
or

k
se

ss
io

n
Re

qu
ire

m
en

ts

en
gi

ne
er

Se
le

ct
ed

e
lic

ita
tio

n
te

ch
ni

qu
es

6
El

ic
it

se
cu

rit
y

re

qu
ire

m
en

ts
.

A
rti

fa
ct

s,
r

is
k

as
se

ss
m

en
t

re
su

lts
, s

el
ec

te
d

te
ch

ni
qu

es
A

cc
el

er
at

ed
R

eq
ui

re
m

en
ts

M
et

ho
d,

Jo

in
t

A
pp

lic
at

io
n

D
ev

el
op

m
en

t,
in

te
rv

ie
w

s,
s

ur
ve

ys
, m

od
el

-b
as

ed

an
al

ys
is

, c
he

ck
lis

ts
, l

is
ts

o
f r

eu
sa

bl
e

re
qu

ire
m

en
ts

t
yp

es
, d

oc
um

en
t

re
vi

ew
s

St
ak

eh
ol

de
rs

fa

ci
lit

at
ed

b
y

re
qu

ire
m

en
ts

en

gi
ne

er

In
iti

al
c

ut
a

t
se

cu
rit

y
re

qu
ire

m
en

ts

7
C

at
eg

or
iz

e
re

qu
ire

m
en

ts

as
t

o
le

ve
l (

sy
st

em
,

so
ftw

ar
e,

e
tc

.)
an

d
w

he
th

er

th
ey

a
re

r
eq

ui
re

m
en

ts
o

r
ot

he
r

ki
nd

s
of

c
on

st
ra

in
ts

.

In
iti

al
r

eq
ui

re
m

en
ts

,
ar

ch
ite

ct
ur

e
W

or
k

se
ss

io
n

us
in

g
a

st
an

da
rd

s
et

o
f

ca
te

go
rie

s
Re

qu
ire

m
en

ts

en
gi

ne
er

, o
th

er

sp
ec

ia
lis

ts
a

s
ne

ed
ed

C
at

eg
or

iz
ed

r
eq

ui
re

m
en

ts

8
Pr

io
rit

iz
e

re
qu

ire
m

en
ts

.
C

at
eg

or
iz

ed
r

eq
ui

re
m

en
ts

an

d
ris

k
as

se
ss

m
en

t
re

su
lts

Pr
io

rit
iz

at
io

n
m

et
ho

ds
s

uc
h

as

A
na

ly
tic

al
H

ie
ra

rc
hy

P
ro

ce
ss

(A
H

P)
,

Tr
ia

ge
, W

in
-W

in
, e

tc
.

St
ak

eh
ol

de
rs

fa

ci
lit

at
ed

b
y

re
qu

ire
m

en
ts

e
ng

in
ee

r

Pr
io

rit
iz

ed
r

eq
ui

re
m

en
ts

9
In

sp
ec

t
re

qu
ire

m
en

ts
.

Pr
io

rit
iz

ed
r

eq
ui

re
m

en
ts

,
ca

nd
id

at
e

fo
rm

al
in

sp
ec

tio
n

te
ch

ni
qu

e

In
sp

ec
tio

n
m

et
ho

d
su

ch
a

s
Fa

ga
n

an
d

pe
er

r
ev

ie
w

s
In

sp
ec

tio
n

te
am

In
iti

al
s

el
ec

te
d

re
qu

ire
m

en
ts

,
do

cu
m

en
ta

tio
n

of
d

ec
is

io
n-

m
ak

in
g

pr
oc

es
s

an
d

ra
tio

na
le

Ta
bl

e
18

.1
Th

e
SQ

U
A

R
E

pr
oc

es
s

362 PART THREE QUALITY AND SECURITY

given project, team members tend to have definitions in mind, based on
their prior experience, but those definitions are often different from one
another [Woo05]. Sources such as the Institute for Electrical and Electron-
ics Engineers (IEEE) and the Software Engineering Body of Knowledge
(SWEBOK) provide a range of definitions to select from or tailor
[SWE14].

Step 2. Identify assets and security goals. This step occurs at the project’s
organizational level and is needed to support software development. Different
stakeholders usually have concerns about different assets, and thus have dif-
ferent goals. For example, a stakeholder in human resources may be con-
cerned about maintaining the confidentiality of personnel records, whereas a
stakeholder in a research area may be concerned with ensuring that research
project information is not accessed, modified, or stolen.

Step 3. Develop artifacts. This step is necessary to support all subsequent
security requirements engineering activities. Often, organizations do not have
key documents needed to support requirements definition, or they may not be
up to date. This means that a lot of time may be spent backtracking to try to
obtain documents, or the team will have to bring them up to date before
going further.

Step 4. Perform risk assessment. This step requires an expert in risk assess-
ment methods, the support of the stakeholders, and the support of a security
requirements engineer. There are a number of risk assessment methods, but
regardless of the one that you choose, the outcomes of risk assessment can
help in identifying the high-priority security exposures.

Step 5. Select elicitation technique. This step becomes important when there
are diverse stakeholders. A more formal elicitation technique, such as the
Accelerated Requirements Method [Hub99], Joint Application Design
[Woo89], or structured interviews, can be effective in overcoming communica-
tion issues when there are stakeholders with different cultural backgrounds. In
other cases, elicitation may simply consist of sitting down with a primary
stakeholder to try to understand that stakeholder’s security requirements needs.

Step 6. Elicit security requirements. This step encompasses the actual
elicitation process using the selected technique. Most elicitation techniques
provide detailed guidance on how to perform elicitation. This builds on the
artifacts that were developed in earlier steps.

Step 7. Categorize requirements. This step allows the security requirements
engineer to distinguish among essential requirements, goals (desired require-
ments), and architectural constraints that may be present. This categorization
also helps in the prioritization activity that follows.

Step 8. Prioritize requirements. This step depends on the prior step and
may also involve performing a cost-benefit analysis to determine which
security requirements have a high payoff relative to their cost. Of course
prioritization may also depend on other consequences of security breaches,
such as loss of life, loss of reputation, and loss of consumer confidence.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 363

Step 9. Requirements inspection. This review activity can be accomplished
at varying levels of formality, discussed in Chapter 16. Once inspection is
complete, the project team should have an initial set of prioritized security
requirements that can be revisited as needed later in the project.

18.5 mi s u s e a n d ab u s e ca s e s a n d at tac k pat t e r n s

Misuse (or abuse) cases can help you view your software in the same way that
attackers do. By thinking about negative events, you can better understand how to
develop secure software. A misuse case can be thought of as a use case that the
attacker initiates.

One of the goals of misuse cases [Sin00] is to decide up front how software should
react to potential attacks. You can also use misuse and normal use cases together to
conduct threat and hazard analysis [Ale03].

We suggest creating misuse cases through brainstorming. Teaming security experts
with subject matter experts (SMEs) covers a lot of ground quickly. During brainstorm-
ing, software security experts ask many questions of developers to help identify the
places where the system is likely to have weaknesses. This involves a careful look at
all user interfaces and considers events that developers assume can’t happen, but that
attackers can actually cause to happen.

Here are some questions that need to be considered: How can the system distin-
guish between valid and invalid input data? Can it tell whether a request is coming
from a legitimate application or a rogue application? Can an insider cause a system
to malfunction? Trying to answer such questions helps developers to analyze their
assumptions and allows them to fix problems up front.

Misuse cases can be in table or diagram form. Figure 18.3 provides an example
misuse case that shows how DroidCleaner malware can successfully attack a cell
phone using an open-source e-mail application called K-9. This is extracted from a
much larger report that you may wish to study [Ali14].

In this misuse case, the user keeps e-mail on the phone’s external storage area. The
attacker gains access to the phone’s storage by compromising the operating system.
A common way for the attacker to gain access to the phone is by tricking the user
into installing a Trojan, to which the user unwittingly grants access to the drive dur-
ing the install process. The attacker is then able to use the Trojan to download files,
including the e-mail contents file.

Attack patterns can provide some help by providing a blueprint for creating an
attack. For example, buffer overflow is one type of security exploitation. Attackers
trying to capitalize on a buffer overflow make use of similar steps [OWA16].
Attack patterns can document these steps (e.g., timing, resources, techniques) as
well as practices software developers can use to prevent or mitigate their success
[Hog04]. When you’re trying to develop misuse and abuse cases, attack patterns
can help.

Misuse cases need to be prioritized as they are generated. In addition, they need
to strike the right balance between cost and benefit. The project budget may not allow

364 PART THREE QUALITY AND SECURITY

a software team to implement all defined mitigation strategies at once. In such cases,
the strategies can be prioritized and implemented incrementally. The team can also
exclude certain cases as being extremely unlikely.

Templates for misuse and abuse cases appear in a number of references. They can
be text or diagrams and may be supported by tools. Good sources for templates are
in materials by Sindre and Opdahl [Sin01] and Alexander [Ale02].

18.6 se c u r i t y ri s k ana Lys i s

A wide variety of security risk assessment methods have been proposed. Typical
examples include SEI CERT’s Security Engineering Risk Analysis (SERA) method3
and the NIST Risk Management Framework (RMF).4

RMF has emerged as an approach that is widely used, providing guidelines for the
users. The RMF steps for security are:

∙ Categorize the information system and the information processed, stored, and
transmitted by that system based on an impact analysis.

∙ Select an initial set of baseline security controls for the information system based
on the security categorization; using an organizational assessment of risk and
local conditions, tailor and supplement the security control baseline as needed.

∙ Implement the security controls, and describe how the controls are employed
within the information system and its operational environment.

∙ Assess the security controls using appropriate assessment procedures to deter-
mine the extent to which the controls are implemented correctly, operating as
intended, and producing the desired outcome with respect to meeting the
security requirements for the system.

Figure 18.3 Misuse case (exploited by DroidCleaner): Data in an e-mail stored on the smartphone
is stolen

User Attacker

Android

Save E-mail
Contents

Download E-mail
Contents

Grant Access
to File

Gain Access to
E-mail Contents

Access E-mail Access E-mail<>

Manage Access <>

<> <> Compromise Phone Security

3 See https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=485410.
4 See https://csrc.nist.gov/publications/detail/sp/800-37/rev-1/final.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 365

∙ Authorize the information system operation based on a determination of the
risk to organizational operations and assets, individuals, or other organizations
(including national defense), from the operation of the information system that
this risk is acceptable.

∙ Monitor the security controls in the information system on an ongoing basis
including assessing control effectiveness, documenting changes to the system
or its environment of operation, conducting security impact analyses of the
associated changes, and reporting the security state of the system to desig-
nated organizational officials.

It’s important to note that NIST also provides a set of security controls to select
from, thus simplifying the risk assessment work. Recently, the RMF has been modi-
fied to include privacy concerns.

18.7 th r e at mo d e L i ng, pr i o r i t i z at i o n,
a n d mi t i gat i o n

A threat modeling method (TMM) is an approach for creating an abstraction of a soft-
ware system, aimed at identifying attackers’ abilities and goals, and using that abstrac-
tion to generate and catalog possible threats that the system must mitigate [Shu16].

STRIDE (an acronym for six threat categories) is representative of a number of
threat modeling methods [Mea18] and is the most well-established TMM, representing
the state of the practice. At its core, STRIDE requires breaking down a system into
its various elements, assessing each of these elements for their vulnerability to threats,
and then mitigating these threats [Her06]. In practice, a typical STRIDE implementa-
tion includes modeling a system with data flow diagrams (DFDs),5 mapping the DFD
elements to the six threat categories, determining the specific threats via checklists or
threat trees, and documenting the threats and steps for their prevention [Sca15].
STRIDE can be implemented manually; however, a free Microsoft Secure Develop-
ment Lifecycle (SDL) Threat Modeling Tool [Mic17] can also be used. Table 18.2
identifies the security property associated with each of the six threat categories.

Table 18.2
Threat
categories
and security
properties

Threat Security Property

Spoofing Authentication

Tampering Integrity

Repudiation Nonrepudiation

Information disclosure Confidentiality

Denial of service Availability

Elevation of privilege Authorization

5 A brief tutorial on data flow diagrams can be downloaded from https://ratandon.mysite.syr
.edu/cis453/notes/DFD_over_Flowcharts .

366 PART THREE QUALITY AND SECURITY

DFDs are designed to show how a system works by using standard symbols to
graphically represent the interaction between data stores (e.g., databases, files, regis-
tries), processes (e.g., DLLs, Web services), data flows (e.g., function calls, remote
procedure calls), and external entities (e.g., people, other systems) [Sho14]. Once
complete, each of these system elements in turn can be associated with one or more
relevant threat categories, as depicted in Table 18.3.

In the next stage, the typical STRIDE user works through a checklist (that may be
in the form of a threat tree) of specific threats that are associated with each match
between a DFD element and threat category. Such checklists are accessible through
STRIDE reference books or tools.

Once the threats have been identified, mitigation strategies can be developed and
prioritized. Typically, prioritization is based on cost and value considerations. Consid-
ering the cost of implementing the mitigation strategy is important, but it’s equally
important to also consider the cost of not implementing it, which is reflected in value.
Remember that risks that are realized result in costs that are not only expressed in terms
of dollars, but could also reflect loss of reputation, loss of trust, and even loss of life.

18.8 at tac k su r fac e

An attack surface can be defined6 in the following manner:

The attack surface describes all of the different points where an attacker could get into
a system, and where they could get data out.

The attack surface of an application is:

1. the sum of all paths for data/commands into and out of the application, and
2. the code that protects these paths (including resource connection and authentica-

tion, authorization, activity logging, data validation and encoding), and
3. all valuable data used in the application, including secrets and keys, intellectual

property, critical business data, personal data and PII, and
4. the code that protects these data (including encryption and checksums, access

auditing, and data integrity and operational security controls). [OWA18]

Table 18.3
Threat
categories of
DFD system
elements

Element Spoofing Tampering Repudiation
Information
Disclosure

Denial of
Service

Elevation of
Privilege

Data flows X X X

Data stores X X X

Processes X X X X X X

External entity X X

6 See https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Attack_Surface_
Analysis_Cheat_Sheet.md.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 367

The OWASP Foundation [OWA18] states that attack surface analysis is:

. . . targeted to be used by developers to understand and manage application security risks
as they design and change an application, as well as by application security specialists
doing a security risk assessment. The focus here is on protecting an application from
external attack—it does not take into account attacks on the users or operators of the
system (e.g., malware injection, social engineering attacks), and there is less focus on
insider threats, although the principles remain the same. The internal attack surface is likely
to be different to the external attack surface and some users may have a lot of access.

Attack Surface Analysis is about mapping out what parts of a system need to be
reviewed and tested for security vulnerabilities. The point of attack surface analysis is to
understand the risk areas in an application, to make developers and security specialists
aware of what parts of the application are open to attack, to find ways of minimizing
this, and to notice when and how the Attack Surface changes and what this means from
a risk perspective.

18.9 se c u r e co d i ng

Secure coding is just what the name implies—coding in such a way that vulnerabili-
ties are not introduced as a result of coding errors. It’s not surprising that most soft-
ware vulnerabilities occur because of sloppy and erroneous coding practices, many of
which can be easily avoided.

For example, a condition known as buffer overflow results from one of the most
well-known and common coding errors. OWASP7 describes it as follows:

A buffer overflow condition exists when a program attempts to put more data in a buf-
fer than it can hold or when a program attempts to put data in a memory area past a
buffer. In this case, a buffer is a sequential section of memory allocated to contain
anything from a character string to an array of integers. Writing outside the bounds of
a block of allocated memory can corrupt data, crash the program, or cause the execution
of malicious code.

Buffer overflow is just one example of coding errors that can result in vulnerabil-
ities. Fortunately, a number of coding standards now exist to provide guidance on
secure coding. The SEI/CERT website8 provides a list of the top-10 secure coding
practices:

1. Validate input. Validate input from all untrusted data sources.
2. Heed compiler warnings. Compile code using the highest warning level

available for your compiler and eliminate warnings by modifying the code.
3. Architect and design for security policies. Create a software architecture

and design your software to implement and enforce security policies.
4. Keep it simple. Keep the design as simple and as small as possible.
5. Default deny. Base access decisions on permission rather than exclusion.

7 See https://www.owasp.org/index.php/Buffer_overflow_attack.
8 See https://wiki.sei.cmu.edu/confluence/display/seccode/Top+10+Secure+Coding+Practices.

368 PART THREE QUALITY AND SECURITY

6. Adhere to the principle of least privilege. Every process should execute
with the least set of privileges necessary to complete the job.

7. Sanitize data sent to other systems. Sanitize all data passed to complex
subsystems such as command shells, relational databases, and commercial
off-the-shelf (COTS) components.

8. Practice defense in depth. Manage risk with multiple defensive strategies.
9. Use effective quality assurance techniques.
10. Adopt a secure coding standard.

SEI CERT and others also provide secure coding standards.9 In addition to using
a secure coding standard, you should inspect for coding errors that lead to vulnera-
bilities. This is just a natural add-on to your normal code inspection and review
process (Chapter 16). Static analysis tools10 are used to automatically analyze code and
are another mechanism for detecting vulnerabilities due to coding errors.

18.10 me a s u r e m e n t

Developing adequate measures of software security is a difficult problem, and one for
which there are differing viewpoints. On the one hand, you can look at the develop-
ment processes followed and assess whether the resultant software is likely to be
secure. On the other hand, you can look at the incidence of vulnerabilities and suc-
cessful break-ins and measure those as a way of assessing software security. However,
neither of these measurement approaches will allow you to say with 100 percent cer-
tainty that our software is secure. When you add supporting software such as operating
systems and external interoperable systems, the measurement of software security
becomes even more difficult. Nevertheless some progress has been made.

Measures of software quality can go a long way toward measuring software secu-
rity. Specifically, vulnerabilities invariably point to software defects. Although not all
software defects are security problems, vulnerabilities in software generally result
from a defect of some kind, whether it is in the requirements, architecture, or code.
Therefore, measures such as defect and vulnerability counts [Woo14] are useful.
Microsoft uses measures such as attack surface analysis and tries to keep the attack
surface (places where software can be compromised) to a minimum.

Just as use of maturity models such as CMMI (Chapter 28) suggests that higher-
quality software will result, mature security development processes, such as those
emphasized by BSIMM,11 will result in more secure software. In some cases, organi-
zations are encouraged to identify the unique set of security metrics that are relevant
to them. BSIMM makes reference to this, as does SAMM.12

9 See https://wiki.sei.cmu.edu/confluence/display/seccode/SEI+CERT+Coding+Standards.
10 A list of commercially available tools can be found at https://en.wikipedia.org/wiki/List_of_

tools_for_static_code_analysis.
11 See https://www.bsimm.com/.
12 See https://www.owasp.org/index.php/OWASP_SAMM_Project.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 369

It’s important to note that none of the measurement characteristics implied by
various maturity models is perfect. If you follow good secure software development
processes, does that guarantee that the software is secure? No! If you find a lot of
vulnerabilities, does that mean that most of them have been found or that there are
still more to be found because this is a particularly poor piece of software? There are
no simple answers. However, to assess vulnerabilities and associated software security,
we have to collect data so that patterns can be analyzed over time. If we don’t collect
data about software security, we will never be able to measure its improvement.

Tables 18.4 and 18.5 provide examples of how to assess software security during
each life-cycle phase. The full tables and discussion can be found in [Mea17] and
[Alb10].

Table 18.4
Examples of
life-cycle-
phase
measures

Life-Cycle Phase Example Software Security Measures

Requirements engineering Percentage of relevant software security principles reflected in
requirements-specific actions (assuming security principles
essential for a given development project have been selected)

Percentage of security requirements that have been subject to
analysis (risk, feasibility, cost-benefit, performance trade-offs)
prior to being included in the specification

Percentage of security requirements covered by attack patterns,
misuse and abuse cases, and other specified means of threat
modeling and analysis

Architecture and design Percentage of architectural and design components subject to
attack surface analysis and measurement

Percentage of architectural and design components subject to
architectural risk analysis

Percentage of high-value security controls covered by a security
design pattern

Table 18.5
Example
measures
based on
the seven
principles
of evidence

Principle Description

Risk Number of active and latent threats, categorized

Incidents reported by category of threat

Likelihood of occurrence for each threat category

Financial and/or human safety estimate of impact for each
threat category

Trusted dependencies Number of levels of subcontracting in the supply chain (in other
words, have the subcontractors, in turn, executed subcontracts,
and what is the depth of this activity?)

Number of suppliers by level

Hierarchical and peer dependencies between suppliers by level

Number of (vetted) trusted suppliers in the supply chain by level

370 PART THREE QUALITY AND SECURITY

18.11 se c u r i t y pro c e s s im p rov e m e n t
a n d mat u r i t y mo d e L s

A number of process improvement and maturity models are available for software
development in general, such as the Capability Maturity Model Integration (CMMI).13
For cyber security maturity, the CMMI Institute offers a newer product, the Cyber
Capability Maturity Management platform.14 OWASP offers the Software Assurance
Maturity Model (SAMM).15 SAMM is an open framework to help organizations for-
mulate and implement a strategy for software security that is tailored to the specific
risks facing the organization.

A comprehensive discussion of these models is beyond the scope of this book. To
provide a simple overview, consider the overall objective of SAMM:

∙ Evaluate an organization’s existing software security practices.
∙ Build a balanced software security assurance program in well-defined iterations.
∙ Demonstrate concrete improvements to a security assurance program.
∙ Define and measure security-related activities throughout an organization.

Perhaps the most well-known maturity model that is specifically for software secu-
rity is the Building Security in Maturity Model (BSIMM). BSIMM has periodic
releases, typically every year or two. The BSIMM model and its recent summarized
assessment results can be downloaded from the BSIMM website.16 According to the
BSIMM developers, BSIMM is meant to be used by those who create and execute a
software security initiative.

All the maturity models mentioned here (and others) have benefits, and the essen-
tial elements of the models are freely available. However, sometimes the assessment
is done by external entities. It’s possible to do an internal self-assessment and define
the associated improvement program, but it requires dedicated resources and effort.
Alternatively, some of these organizations offer assessment programs, thus providing
an external view of the strengths and areas for improvement within a software
organization.

18.12 su m m a ry

All software engineers should have an understanding of what it takes to develop secure
software. The steps that are needed to improve the security of your software products
are relevant in each activity in the software process, regardless of which process model
is used.

Although there are still many open questions, and technologies that need additional
research, there are many resources available today to assist with this challenge. For
every activity that normally takes place in the software process, try to incorporate

13 See https://cmmiinstitute.com/.
14 See https://cmmiinstitute.com/products/cybermaturity.
15 See https://www.owasp.org/index.php/OWASP_SAMM_Project.
16 See https://www.bsimm.com/.

CHAPTER 18 SOFTWARE SECURITY ENGINEERING 371

security aspects. Models such as Microsoft SDL and the SQUARE model can be
assessed to determine which steps you can incorporate into your development process.

Add security to a risk analysis activity, especially using detailed guidance available
from NIST. Given the number of secure coding standards already available, it is cer-
tainly possible for anyone to learn how to code securely. Inspect your code for remain-
ing vulnerabilities. Learn how to identify security holes and develop and prioritize
mitigation strategies. Perform static analysis testing on your code. Visit the OWASP
and BSIMM websites, among others, to learn about maturity in software security
engineering.

As software becomes ever more ubiquitous, the number of vulnerabilities and suc-
cessful hacks grow as well. It will take all our efforts to stem the tide, but many of
the tools already exist to tackle this problem. The consequences of failing to address
software security are high, and the benefits of developing secure software are huge.

Pro b l e m s a n d Po i n t s to Po n d e r

18.1. What is the most important thing that a software team can do to improve software
security?

18.2. If you were recommending one activity for your organization to improve software secu-
rity, what would it be? If you were recommending multiple activities, what are they, and what
would be the priorities, considering that it’s not likely that all of them will be implemented
at once?

18.3. How could you incorporate software security into your existing process model or into a
new process model?

18.4. Sit down with a colleague and identify security risks on a software project that is in
development. Come up with mitigation strategies and prioritize them.

18.5. Are you collecting measurement data that could be used or repurposed to help measure
software security? If not, is there data that could easily be collected for this purpose?

18.6. Use the Internet to find out the details needed to create a phishing attack pattern.

18.7. Explain some of the problems that might be encountered if you wait until after the system
is completed to address security risks.

18.8. Use the Internet to determine the average cost to the consumer of a single incidence of
identity theft.

18.9. Consider a MobileApp that you make use of on your personal phone. List three to five
security risks that developers should consider when developing apps like this one.

18.10. Determine the security requirements of a bill-paying wallet-type MobileApp.

Design element: Quick Look icon magnifying glass: © Roger Pressman