OpenDSA TDDI16F25 Data Structures and Algorithms

This document is a translation of the lecture notes from Linköping University. It can be accessed at https://www.ida.liu.se/opendsa/Books/TDDI16F25/html/. The formulas are presented in KaTeX format, and the code syntax follows Java (Generic).

Chapter 0: Preface

0.1. How to Use this System

Welcome to OpenDSA! OpenDSA is an open source project whose goal is to provide online course materials for a wide range of Computer Science courses.

Learning Management Systems and OpenDSA Book Instances OpenDSA materials are most often presented in the form of a “textbook instance” as content within some Learning Management System (LMS), such as Canvas or Moodle. Your OpenDSA “textbook” is made up of a series of modules. Each module corresponds to roughly one section in a regular textbook, or what might get presented in part of a one-hour lecture. OpenDSA content combines text and images with algorithm visualizations for all of the algorithms, along with lots of interactive assessment exercises. The goal of the exercisees is to let you know if you really understand what you have been reading.

Registration and Accounts: This is an open source project, and in keeping with the spirit of our commitment to openness, our materials are free for use. We maintain versions of our materials that are publicly accessible. However, in order to receive credits for completed exercises, you need to sign in. This is done using the “register” or “login” buttons at the top of the page.

Overview of major elements: An OpenDSA module contains a number of interactive components intended to help you learn the material, in addition to regular text and graphics. Most modules include short slideshows that help explain steps of an algorithm. OpenDSA also includes full Algorithm Visualizations that go through a complete example of the algorithm (these will usually let you set the input data if you like). There are many “proficiency exercises” where you show the steps taken by an algorithm. There are also many sets of multiple choice, fill-in-the-blanks, and True/False questions. There are also various calculators, simulations, and other types of interactive exercises.

Getting credit: You can get “completion” for doing the various activities. When any of the following activities are completed, the interface should give you some type of feedback such as showing a green checkmark or shading an associated line in the gradebook. Furthermore, you can always check your progress by clicking on your name in the top right of the page to access the gradebook. Depending on how your book is configured, completing an activity might or might not give assignment points, as shown in the gradebook.

Seeing your credit and seeing your current score: Usually an exercise or slideshow will give you visual confirmation when you have completed it. If you complete all exercises and slideshows for a module, the title line should show “Module Complete” in green.

Slideshows: Slideshows are controlled by the buttons at the top of the slideshow. “<” and “>” buttons will back up or advance by one slide, while “<<” and “>>” will take you to the beginning or end of the slideshow. You get “completion” for a slideshow just for progressing to the end (one step at a time). A green checkmark should show up on the right when you complete it. Depending on how your book instance is configured, completing a slideshow might give you points or it might not.

Algorithm Visualizations: Slideshows demonstrate a part of an algorithm on a fixed input. The Algorithm Visualizations let you run the whole algorithm, typically on input that you can choose. The Algorithm visualizations will have some parameters that you can set (such as number of elements in an input array, and the values of the array elements). The “Run” button will start (or restart) the Algorithm Visualization. It will also begin the visualization with new, randomly selected data values if you did not specify your own input. “Reset” will typically clear all of the input fields and restart the visualization. Many visualizations have a “Help” button to give detailed information about that particular visualization.

Question Sets: Most modules finish with a collection of multiple choice, True/False, or type-a-number questions. To get credit for a question set, you will have to answer some number of the questions correctly (the exact number required can be different for each question set). Once you have credit, the interface should indicate this. You can still get more questions at that point if you would like more practice. Above the question on the right-hand side is a counter to indicate your current number of questions correct out of the total number needed to complete the exercise. If you answer a question wrong, your progress toward the completion threshold will go backwards by one point. (And you will still have to answer that question before you can continue!) Note that once you have been given completion credit for the question set, you cannot lose that credit by answering more questions, even if you then get some wrong.

The question sets work by randomly selecting from the questions available to that set. Typically, once you correctly answer a question, you will not see it again (or at least not with the same inputs). But if you answer it incorrectly (and then clear it with the correct answer), it might appear again.

The questions will have “hints” that you can use to help you figure out the answer. If you take a hint, you will not get credit for that question toward completing the exercise. However, you will not lose a point on that question, either.

Interactive Questions: Some of the question are interactive, in that that you will click on an array, tree node, or other visual element to answer the question. They work the same way as the multiple choice questions, in that there is a required number that you must get right to complete the exercise successfully.

Proficiency Exercises: Proficiency exercises are graded on a point system. Each (logical) step that you do correctly in the proficiency exercise gives you a point. You need to get a set fraction of the total possible points to complete the exercise (typically 90% of the possible points). Generally, if you make a mistake on a proficiency exercise, it will tell you that you did the step wrong and correct it for you so that you can continue on. You will not get credit for that particular step when you do it wrong, but you can still get credit for the remaining steps. Once you get completion credit for a proficiency exercise, the LMS interface should give some indication of this.

Note that the “points” as counted by the proficiency exercise is different from the credit assigned by the LMS for completion of the exercise. For example, to complete the proficiency exercise for a particular sorting algorithm, you might need to complete 18 of 20 steps in the exercise correctly. Once you have done this, the LMS might award you (for example) 2 points of completion credit. You can always practice an exercise as many times as you like, whether you already have completion credit or not.

Chapter 1: Introduction

1.1. Data Structures and Algorithms

1.1.1. Introduction

How many cities with more than 250,000 people lie within 500 miles of Dallas, Texas? How many people in my company make over $100,000 per year? Can we connect all of our telephone customers with less than 1,000 miles of cable? To answer questions like these, it is not enough to have the necessary information. We must organize that information in a way that allows us to find the answers in time to satisfy our needs.

Representing information is fundamental to computer science. The primary purpose of most computer programs is not to perform calculations, but to store and retrieve information—usually as fast as possible. For this reason, the study of data structures and the algorithms that manipulate them is at the heart of computer science. And that is what this book is about—helping you to understand how to structure information to support efficient processing.

Any course on Data Structures and Algorithms will try to teach you about three things:

1. It will present a collection of commonly used data structures and algorithms. These form a programmer’s basic “toolkit”. For many problems, some data structure or algorithm in the toolkit will provide a good solution. We focus on data structures and algorithms that have proven over time to be most useful.
2. It will introduce the idea of tradeoffs, and reinforce the concept that there are costs and benefits associated with every data structure or algorithm. This is done by describing, for each data structure, the amount of space and time required for typical operations. For each algorithm, we examine the time required for key input types.
3. It will teach you how to measure the effectiveness of a data structure or algorithm. Only through such measurement can you determine which data structure in your toolkit is most appropriate for a new problem. The techniques presented also allow you to judge the merits of new data structures that you or others might invent.

There are often many approaches to solving a problem. How do we choose between them? At the heart of computer program design are two (sometimes conflicting) goals:

1. To design an algorithm that is easy to understand, code, and debug.
2. To design an algorithm that makes efficient use of the computer’s resources.

Ideally, the resulting program is true to both of these goals. We might say that such a program is “elegant.” While the algorithms and program code examples presented here attempt to be elegant in this sense, it is not the purpose of this book to explicitly treat issues related to goal (1). These are primarily concerns for the discipline of Software Engineering. Rather, we mostly focus on issues relating to goal (2).

How do we measure efficiency? Our method for evaluating the efficiency of an algorithm or computer program is called asymptotic analysis. Asymptotic analysis also gives a way to define the inherent difficulty of a problem. Throughout the book we use asymptotic analysis techniques to estimate the time cost for every algorithm presented. This allows you to see how each algorithm compares to other algorithms for solving the same problem in terms of its efficiency.

1.1.2. A Philosophy of Data Structures

You might think that with ever more powerful computers, program efficiency is becoming less important. After all, processor speed and memory size still continue to improve. Won’t today’s efficiency problem be solved by tomorrow’s hardware?

As we develop more powerful computers, our history so far has always been to use that additional computing power to tackle more complex problems, be it in the form of more sophisticated user interfaces, bigger problem sizes, or new problems previously deemed computationally infeasible. More complex problems demand more computation, making the need for efficient programs even greater. Unfortunately, as tasks become more complex, they become less like our everyday experience. So today’s computer scientists must be trained to have a thorough understanding of the principles behind efficient program design, because their ordinary life experiences often do not apply when designing computer programs.

In the most general sense, a data structure is any data representation and its associated operations. Even an integer or floating point number stored on the computer can be viewed as a simple data structure. More commonly, people use the term “data structure” to mean an organization or structuring for a collection of data items. A sorted list of integers stored in an array is an example of such a structuring. These ideas are explored further in a discussion of Abstract Data Types.

Given sufficient space to store a collection of data items, it is always possible to search for specified items within the collection, print or otherwise process the data items in any desired order, or modify the value of any particular data item. The most obvious example is an unsorted array containing all of the data items. It is possible to perform all necessary operations on an unsorted array. However, using the proper data structure can make the difference between a program running in a few seconds and one requiring many days. For example, searching for a given record in a hash table is much faster than searching for it in an unsorted array.

A solution is said to be efficient if it solves the problem within the required resource constraints. Examples of resource constraints include the total space available to store the data—possibly divided into separate main memory and disk space constraints—and the time allowed to perform each subtask. A solution is sometimes said to be efficient if it requires fewer resources than known alternatives, regardless of whether it meets any particular requirements. The cost of a solution is the amount of resources that the solution consumes. Most often, cost is measured in terms of one key resource such as time, with the implied assumption that the solution meets the other resource constraints.

1.1.3. Selecting a Data Structure

It should go without saying that people write programs to solve problems. However, sometimes programmers forget this. So it is crucial to keep this truism in mind when selecting a data structure to solve a particular problem. Only by first analyzing the problem to determine the performance goals that must be achieved can there be any hope of selecting the right data structure for the job. Poor program designers ignore this analysis step and apply a data structure that they are familiar with but which is inappropriate to the problem. The result is typically a slow program. Conversely, there is no sense in adopting a complex representation to “improve” a program that can meet its performance goals when implemented using a simpler design.

When selecting a data structure to solve a problem, you should follow these steps.

1. Analyze your problem to determine the basic operations that must be supported. Examples of basic operations include inserting a data item into the data structure, deleting a data item from the data structure, and finding a specified data item.
2. Quantify the resource constraints for each operation.
3. Select the data structure that best meets these requirements.

This three-step approach to selecting a data structure operationalizes a data-centered view of the design process. The first concern is for the data and the operations to be performed on them, the next concern is the representation for those data, and the final concern is the implementation of that representation.

Resource constraints on certain key operations, such as search, inserting data records, and deleting data records, normally drive the data structure selection process. Many issues relating to the relative importance of these operations are addressed by the following three questions, which you should ask yourself whenever you must choose a data structure.

1. Are all data items inserted into the data structure at the beginning, or are insertions interspersed with other operations? Static applications (where the data are loaded at the beginning and never change) typically get by with simpler data structures to get an efficient implementation, while dynamic applications often require something more complicated.
2. Can data items be deleted? If so, this will probably make the implementation more complicated.
3. Are all data items processed in some well-defined order, or is search for specific data items allowed? “Random access” search generally requires more complex data structures.

Each data structure has associated costs and benefits. In practice, it is hardly ever true that one data structure is better than another for use in all situations. If one data structure or algorithm is superior to another in all respects, the inferior one will usually have long been forgotten. For nearly every data structure and algorithm presented in this book, you will see examples of where it is the best choice. Some of the examples might surprise you.

A data structure requires a certain amount of space for each data item it stores, a certain amount of time to perform a single basic operation, and a certain amount of programming effort. Each problem has constraints on available space and time. Each solution to a problem makes use of the basic operations in some relative proportion, and the data structure selection process must account for this. Only after a careful analysis of your problem’s characteristics can you determine the best data structure for the task.

Example 1.1.1

A bank must support many types of transactions with its customers, but we will examine a simple model where customers wish to open accounts, close accounts, and add money or withdraw money from accounts. We can consider this problem at two distinct levels: (1) the requirements for the physical infrastructure and workflow process that the bank uses in its interactions with its customers, and (2) the requirements for the database system that manages the accounts.

The typical customer opens and closes accounts far less often than accessing the account. Customers are willing to spend many minutes during the process of opening or closing the account, but are typically not willing to wait more than a brief time for individual account transactions such as a deposit or withdrawal. These observations can be considered as informal specifications for the time constraints on the problem.

It is common practice for banks to provide two tiers of service. Human tellers or automated teller machines (ATMs) support customer access to account balances and updates such as deposits and withdrawals. Special service representatives are typically provided (during restricted hours) to handle opening and closing accounts. Teller and ATM transactions are expected to take little time. Opening or closing an account can take much longer (perhaps up to an hour from the customer’s perspective).

From a database perspective, we see that ATM transactions do not modify the database significantly. For simplicity, assume that if money is added or removed, this transaction simply changes the value stored in an account record. Adding a new account to the database is allowed to take several minutes. Deleting an account need have no time constraint, because from the customer’s point of view all that matters is that all the money be returned (equivalent to a withdrawal). From the bank’s point of view, the account record might be removed from the database system after business hours, or at the end of the monthly account cycle.

When considering the choice of data structure to use in the database system that manages customer accounts, we see that a data structure that has little concern for the cost of deletion, but is highly efficient for search and moderately efficient for insertion, should meet the resource constraints imposed by this problem. Records are accessible by unique account number (sometimes called an exact-match query). One data structure that meets these requirements is the hash table. Hash tables allow for extremely fast exact-match search. A record can be modified quickly when the modification does not affect its space requirements. Hash tables also support efficient insertion of new records. While deletions can also be supported efficiently, too many deletions lead to some degradation in performance for the remaining operations. However, the hash table can be reorganized periodically to restore the system to peak efficiency. Such reorganization can occur offline so as not to affect ATM transactions.

Example 1.1.2

A company is developing a database system containing information about cities and towns in the United States. There are many thousands of cities and towns, and the database program should allow users to find information about a particular place by name (another example of an exact-match query). Users should also be able to find all places that match a particular value or range of values for attributes such as location or population size. This is known as a range query.

A reasonable database system must answer queries quickly enough to satisfy the patience of a typical user. For an exact-match query, a few seconds is satisfactory. If the database is meant to support range queries that can return many cities that match the query specification, the user might tolerate the entire operation to take longer, perhaps on the order of a minute. To meet this requirement, it will be necessary to support operations that process range queries efficiently by processing all cities in the range as a batch, rather than as a series of operations on individual cities.

The hash table suggested in the previous example is inappropriate for implementing our city database, because it cannot perform efficient range queries. The B-tree supports large databases, insertion and deletion of data records, and range queries. However, a simple linear index would be more appropriate if the database is created once, and then never changed, such as an atlas distributed on a CD or accessed from a website.

Chapter 2: Mathematical Background

2.1. Chapter Introduction

This chapter presents mathematical notation, background, and techniques used throughout the modules. This material is provided primarily for review and reference. You might wish to return to the relevant sections when you encounter unfamiliar notation or mathematical techniques in later chapters.

The concept of estimation might be unfamiliar to many readers. Estimation is not a mathematical technique, but rather a general engineering skill. It is enormously useful to computer scientists doing design work, because any proposed solution whose estimated resource requirements fall well outside the problem’s resource constraints can be discarded immediately, allowing time for greater analysis of more promising solutions.

2.2. Sets and Relations

2.2.1. Set Notation

The concept of a set in the mathematical sense has wide application in computer science. The notations and techniques of set theory are commonly used when describing and implementing algorithms because the abstractions associated with sets often help to clarify and simplify algorithm design.

A set is a collection of distinguishable members or elements. The members are typically drawn from some larger population known as the base type. Each member of a set is either a primitive element of the base type or is a set itself. There is no concept of duplication in a set. Each value from the base type is either in the set or not in the set. For example, a set named might consist of the three integers 7, 11, and 42. In this case, ’s members are 7, 11, and 42, and the base type is integer.

The following table shows the symbols commonly used to express sets and their relationships.

Table 2.2.1

Here are some examples of this notation in use. First define two sets, and .

(because has three members) and (because has two members). The union of and , written , is the set of elements in either or , which is {2, 3, 5, 10}. The intersection of and , written , is the set of elements that appear in both and , which is {5}. The set difference of and , written , is the set of elements that occur in but not in , which is {2, 3}. Note that and that , but in general . In this example, . Finally, the set {5, 3, 2} is indistinguishable from set , because sets have no concept of order. Likewise, set {2, 3, 2, 5} is also indistinguishable from , because sets have no concept of duplicate elements.

The set product or Cartesian product of two sets is a set of ordered pairs. For our example sets, the set product would be

The powerset of a set (denoted ) is the set of all possible subsets for . Consider the set . The powerset of is

A collection of elements with no order (like a set), but with duplicate-valued elements is called a bag [^1]. To distinguish bags from sets, we will use square brackets [] around a bag’s elements. For example, bag [3, 4, 5, 4] is distinct from bag [3, 4, 5], while set {3, 4, 5, 4} is indistinguishable from set {3, 4, 5}. However, bag [3, 4, 5, 4] is indistinguishable from bag [3, 4, 4, 5].

A sequence is a collection of elements with an order, and which may contain duplicate-valued elements. A sequence is also sometimes called a tuple or a vector. In a sequence, there is a 0th element, a 1st element, 2nd element, and so on. We will use angle brackets to enclose the elements of a sequence. For example, is a sequence. Note that sequence is distinct from sequence , and both are distinct from sequence .

[^1]: The object referred to here as a bag is sometimes called a multilist. But, the term multilist also refers to a list that may contain sublists.

2.2.1.1. Relations

A relation over set is a set of ordered pairs from . As an example of a relation, if is , then

is a relation, and

is a different relation. If tuple is in relation , we may use the infix notation . We often use relations such as the less than operator () on the natural numbers, which includes ordered pairs such as and , but not or . Rather than writing the relationship in terms of ordered pairs, we typically use an infix notation for such relations, writing .

Define the properties of relations as follows, with a binary relation over set .

• is reflexive if for all .
• is irreflexive if is not true for all .
• is symmetric if whenever , then , for all .
• is antisymmetric if whenever and , then , for all .
• is transitive if whenever and , then , for all .

As examples, for the natural numbers, is irreflexive (because aRa is never true), antisymmetric (because there is no case where and ), and transitive. Relation is reflexive, antisymmetric, and transitive. Relation is reflexive, symmetric (and antisymmetric!), and transitive. For people, the relation “is a sibling of” is symmetric and transitive. If we define a person to be a sibling of themself, then it is reflexive; if we define a person not to be a sibling of themself, then it is not reflexive.

2.2.2. Equivalence Relations

is an equivalence relation on set if it is reflexive, symmetric, and transitive. An equivalence relation can be used to partition a set into equivalence classes. If two elements and are equivalent to each other, we write . A partition of a set is a collection of subsets that are disjoint from each other and whose union is . An equivalence relation on set partitions the set into disjoint subsets whose elements are equivalent. The UNION/FIND algorithm efficiently maintains equivalence classes on a set. One application for such disjoint sets computing a minimal cost spanning tree.

Example 2.2.1

For the integers, is an equivalence relation that partitions each element into a distinct subset. In other words, for any integer , three things are true.

1. ,
2. if then , and
3. if and , then .

Of course, for distinct integers , , and there are never cases where , , or . So the requirements for symmetry and transitivity are never violated, and therefore the relation is symmetric and transitive.

Example 2.2.2

If we clarify the definition of sibling to mean that a person is a sibling of themself, then the sibling relation is an equivalence relation that partitions the set of people.

Example 2.2.3

We can use the modulus function to define an equivalence relation. For the set of integers, use the modulus function to define a binary relation such that two numbers and are in the relation if and only if . Thus, for , is in the relation because . We see that modulus used in this way defines an equivalence relation on the integers, and this relation can be used to partition the integers into equivalence classes. This relation is an equivalence relation because

1. for all ;
2. if , then ; and
3. if and , then .

2.2.3. Partial Orders

A binary relation is called a partial order if it is antisymmetric and transitive. If the relation is reflexive, it is called a non-strict partial order. If the relation is irreflexive, it is called a strict partial order. The set on which the partial order is defined is called a partially ordered set or a poset. Elements and of a set are comparable under a given relation if either or . If every pair of distinct elements in a partial order are comparable, then the order is called a total order or linear order.

Example 2.2.4

For the integers, relations and define partial orders. Operation is a total order because, for every pair of integers and such that , either or . Likewise, is a total order because, for every pair of integers and such that , either or .

Example 2.2.5

For the powerset of the integers, the subset operator defines a partial order (because it is antisymmetric and transitive). For example, . However, sets {1, 2} and {1, 3} are not comparable by the subset operator, because neither is a subset of the other. Therefore, the subset operator does not define a total order on the powerset of the integers.

2.3. Miscellaneous Notation

This module collects together definitions for a number of mathematical terms and concepts, as a place for reference when needed.

Units of measure: OpenDSA modules use the following notation for units of measure. “B” will be used as an abbreviation for bytes, “b” for bits, “KB” for kilobytes bytes), “MB” for megabytes bytes) “GB” for gigabytes bytes) and “ms” for milliseconds (a millisecond is 1/1000 of a second). Spaces are not placed between the number and the unit abbreviation when a power of two is intended. Thus a disk drive of size 25 gigabytes (where a gigabyte is intended as bytes) will be written as “25GB”. Spaces are used when a decimal value is intended. An amount of 2000 bits would therefore be written “2 Kb” while “2Kb” represents 2048 bits. 2000 milliseconds is written as 2000 ms. Note that in this book large amounts of storage are nearly always measured in powers of two and times in powers of ten.

Factorial function: The factorial function, written for an integer greater than 0, is the product of the integers between 1 and , inclusive. Thus, . As a special case, . The factorial function grows quickly as becomes larger. Because computing the factorial function directly is a time-consuming process, it can be useful to have an equation that provides a good approximation. Stirling’s approximation states that , where ( is the base for the system of natural logarithms) [^1]. Thus we see that while grows slower than (because ), it grows faster than for any positive integer constant .

Permutations: A permutation of a sequence is simply the members of arranged in some order. For example, a permutation of the integers 1 through would be those values arranged in some order. If the sequence contains distinct members, then there are different permutations for the sequence. This is because there are choices for the first member in the permutation; for each choice of first member there are choices for the second member, and so on. Sometimes one would like to obtain a random permutation for a sequence, that is, one of the possible permutations is selected in such a way that each permutation has equal probability of being selected. A simple function for generating a random permutation is as follows. Here, the values of the sequence are stored in positions 0 through of array A, function swap(A, i, j) exchanges elements i and j in array A, and Random(n) returns an integer value in the range 0 to .

// Randomly permute the values in array A
static <T> void permute(T[] A) {
  for (int i = A.length; i > 0; i--) // for each i
    swap(A, i-1, random(i));         //   swap A[i-1] with a random
}                                    //   position in the range 0 to i-1.

Boolean variables: A Boolean variable is a variable that takes on one of the two values True and False. These two values are often associated with the values 1 and 0, respectively, although there is no reason why this needs to be the case. It is poor programming practice to rely on the correspondence between 0 and False, because these are logically distinct objects of different types.

Logic Notation: We will occasionally make use of the notation of symbolic or Boolean logic. means “ implies ” or “If then ”. means “ if and only if ” or “ is equivalent to ”. means “ or ” (useful both in the context of symbolic logic or when performing a Boolean operation). means “ and ”. and both mean “not ” or the negation of where is a Boolean variable.

Floor and ceiling: The floor of (written ) takes real value and returns the greatest integer . For example, , as does , while and . The ceiling of (written ) takes real value and returns the least integer . For example, , as does , while .

Modulus function: The modulus (or mod) function returns the remainder of an integer division. Sometimes written in mathematical expressions, the syntax in many programming languages is n % m. From the definition of remainder, is the integer such that for an integer, and . Therefore, the result of must be between 0 and when and are positive integers. For example, ; , , and .

There is more than one way to assign values to and , depending on how integer division is interpreted. The most common mathematical definition computes the mod function as . In this case, . However, Java and C++ compilers typically use the underlying processor’s machine instruction for computing integer arithmetic. On many computers this is done by truncating the resulting fraction, meaning . Under this definition, . Another language might do something different.

Unfortunately, for many applications this is not what the user wants or expects. For example, many hash systems will perform some computation on a record’s key value and then take the result modulo the hash table size. The expectation here would be that the result is a legal index into the hash table, not a negative number. Implementers of hash functions must either insure that the result of the computation is always positive, or else add the hash table size to the result of the modulo function when that result is negative.

[^1]: The symbol “” means “approximately equal.”

2.4. Logarithms

2.4.1. Logarithms

The logarithm of base for value is the power to which is raised to get . Normally, this is written as . Thus, if then , and .

Logarithms are used frequently by programmers. Here are two typical uses.

Example 2.4.1

Many programs require an encoding for a collection of objects. What is the minimum number of bits needed to represent distinct code values? The answer is bits. For example, if you have 1000 codes to store, you will require at least bits to have 1000 different codes (10 bits provide 1024 distinct code values).

Example 2.4.2

Consider the binary search algorithm for finding a given value within an array sorted by value from lowest to highest. Binary search first looks at the middle element and determines if the value being searched for is in the upper half or the lower half of the array. The algorithm then continues splitting the appropriate subarray in half until the desired value is found. How many times can an array of size (n) be split in half until only one element remains in the final subarray? The answer is times.

In OpenDSA, nearly all logarithms used have a base of two. This is because data structures and algorithms most often divide things in half, or store codes with binary bits. Whenever you see the notation in OpenDSA, either is meant or else the term is being used asymptotically and so the actual base does not matter. For logarithms using any base other than two, we will show the base explicitly.

Logarithms have the following properties, for any positive values of , , and , and any positive integers and .

1. .
2. .
3. .
4. .

The first two properties state that the logarithm of two numbers multiplied (or divided) can be found by adding (or subtracting) the logarithms of the two numbers. [^1] Property (3) is simply an extension of property (1). Property (4) tells us that, for variable and any two integer constants and , and differ by the constant factor , regardless of the value of . Most runtime analyses we use are of a type that ignores constant factors in costs. Property (4) says that such analyses need not be concerned with the base of the logarithm, because this can change the total cost only by a constant factor.

A useful identity to know is:

To give some intuition for why this is true: What does it mean to take the log (base 2) of ? If , then is the power to which you need to raise 2 to get back to . So of course, when the base of the log is 2.

When discussing logarithms, exponents often lead to confusion. Property (3) tells us that . How do we indicate the square of the logarithm (as opposed to the logarithm of )? This could be written as , but it is traditional to use . On the other hand, we might want to take the logarithm of the logarithm of . This is written .

A special notation is used in the rare case when we need to know how many times we must take the log of a number before we reach a value . This quantity is written . For example, because , , , and , which is a total of 4 log operations.

[^1]: These properties are the idea behind the slide rule. Adding two numbers can be viewed as joining two lengths together and measuring their combined length. Multiplication is not so easily done. However, if the numbers are first converted to the lengths of their logarithms, then those lengths can be added and the inverse logarithm of the resulting length gives the answer for the multiplication (this is simply logarithm property (1)). A slide rule measures the length of the logarithm for the numbers, lets you slide bars representing these lengths to add up the total length, and finally converts this total length to the correct numeric answer by taking the inverse of the logarithm for the result.

Here is some practice with manipulating logarithms.

2.5. Summations

2.5.1. Summations

Most programs contain loop constructs. When analyzing running time costs for programs with loops, we need to add up the costs for each time the loop is executed. This is an example of a summation. Summations are simply the sum of costs for some function applied to a range of parameter values. Summations are typically written with the following “Sigma” notation:

This notation indicates that we are summing the value of over some range of (integer) values. The parameter to the expression and its initial value are indicated below the symbol. Here, the notation indicates that the parameter is and that it begins with the value 1. At the top of the symbol is the expression . This indicates the maximum value for the parameter . Thus, this notation means to sum the values of as ranges across the integers from 1 through . This can also be written . Within a sentence, Sigma notation is typeset as .

Given a summation, you often wish to replace it with an algebraic equation with the same value as the summation. This is known as a closed-form solution, and the process of replacing the summation with its closed-form solution is known as solving the summation. For example, the summation is simply the expression “1” summed times (remember that ranges from 1 to ). Because the sum of 1s is , the closed-form solution is .

Here is an explanation about the closed form solution of one summation that you will see many times in this book. Since this appears so often, it will help you later if you can get comfortable with it.

Here is a list of useful summations, along with their closed-form solutions.

As special cases to this last summation, we have the following two:

As a corollary to (7),

Finally,

The sum of reciprocals from 1 to , called the Harmonic Series and written , has a value between and . To be more precise, as grows, the summation grows closer to

where is Euler’s constant and has the value 0.5772…

Most of these equalities can be proved easily by a proof by induction. Unfortunately, induction does not help us derive a closed-form solution. Induction only confirms when a proposed closed-form solution is correct.

2.6. Recurrence Relations

2.6.1. Recurrence Relations

The running time for a recursive algorithm is most easily expressed by a recursive expression because the total time for the recursive algorithm includes the time to run the recursive call(s). A recurrence relation defines a function by means of an expression that includes one or more (smaller) instances of itself. A classic example is the recursive definition for the factorial function:

Another standard example of a recurrence is the Fibonacci sequence:

From this definition, the first seven numbers of the Fibonacci sequence are

Notice that this definition contains two parts: the general definition for and the base cases for and . Likewise, the definition for factorial contains a recursive part and base cases.

Recurrence relations are often used to model the cost of recursive functions. For example, the number of multiplications required by a recursive version of the factorial function for an input of size will be zero when or (the base cases), and it will be one plus the cost of calling fact on a value of . This can be defined using the following recurrence:

As with summations, we typically wish to replace the recurrence relation with a closed-form solution. One approach is to expand the recurrence by replacing any occurrences of on the right-hand side with its definition.

A slightly more complicated recurrence is

Again, we will use expansion to help us find a closed form solution.

2.7. Mathematical Proof Techniques

2.7.1. Mathematical Proof Techniques

Solving any problem has two distinct parts: the investigation and the argument. Students are too used to seeing only the argument in their textbooks and lectures. But to be successful in school (and in life after school), one needs to be good at both, and to understand the differences between these two phases of the process. To solve the problem, you must investigate successfully. That means engaging the problem, and working through until you find a solution. Then, to give the answer to your client (whether that “client” be your instructor when writing answers on a homework assignment or exam, or a written report to your boss), you need to be able to make the argument in a way that gets the solution across clearly and succinctly. The argument phase involves good technical writing skills—the ability to make a clear, logical argument.

Being conversant with standard proof techniques can help you in this process. Knowing how to write a good proof helps in many ways. First, it clarifies your thought process, which in turn clarifies your explanations. Second, if you use one of the standard proof structures such as proof by contradiction or an induction proof, then both you and your reader are working from a shared understanding of that structure. That makes for less complexity to your reader to understand your proof, because the reader need not decode the structure of your argument from scratch.

This section briefly introduces three commonly used proof techniques:

1. deduction, or direct proof;
2. proof by contradiction and
3. proof by mathematical induction.

In general, a direct proof is just a “logical explanation”. A direct proof is sometimes referred to as an argument by deduction. This is simply an argument in terms of logic.

2.7.1.1. Direct Proof

Example 2.7.1

Here is a direct proof that . If we take the first and last terms of the series, since they are 1 and , of course they sum to . If we take the second term and next-to-last term, since they are 2 and , they also sum to . Likewise for the third term and third-from-the-end term. We can go on and pair up terms like this, such that there are pairs that each sum to , for a total sum of . You can check for yourself that this is true even if is odd (and so the middle value of the series has no partner).

Many direct proofs are written in English with words such as “if … then”. In this case logic notation such as can often help express the proof. Even if we don’t wish to use symbolic logic notation, we can still take advantage of fundamental theorems of logic to structure our arguments. For example, if we want to prove that and are equivalent, we can first prove and then prove .

In some domains, proofs are essentially a series of state changes from a start state to an end state. Formal predicate logic can be viewed in this way, with the various “rules of logic” being used to make the changes from one formula or combining a couple of formulas to make a new formula on the route to the destination. Symbolic manipulations to solve integration problems in introductory calculus classes are similar in spirit, as are high school geometry proofs.

2.7.1.2. Proof by Contradiction

The simplest way to disprove a theorem or statement is to find a counter-example to the theorem. Unfortunately, no number of examples supporting a theorem is sufficient to prove that the theorem is correct. However, there is an approach that is vaguely similar to disproving by counter-example, called proof by contradiction. To prove a theorem by contradiction, we first assume that the theorem is false. We then find a logical contradiction stemming from this assumption. If the logic used to find the contradiction is correct, then the only way to resolve the contradiction is to recognize that the assumption that the theorem is false must be incorrect. That is, we conclude that the theorem must be true.

Example 2.7.2

Here is a simple proof by contradiction.

Theorem: There is no largest integer.

Proof by contradiction:

Step 1. Contrary assumption: Assume that there is a largest integer. Call it (for “biggest”).

Step 2. Show this assumption leads to a contradiction: Consider . is an integer because it is the sum of two integers. Also, , which means that is not the largest integer after all. Thus, we have reached a contradiction. The only flaw in our reasoning is the initial assumption that the theorem is false. Thus, we conclude that the theorem is correct.

A related proof technique is proving the contrapositive. We can prove that by proving . This technique works because the truth table for the two logical statements are the same.

2.7.1.3. Proof by Mathematical Induction

Mathematical induction can be used to prove a wide variety of theorems. Induction also provides a useful way to think about algorithm design, because it encourages you to think about solving a problem by building up from simple subproblems. Induction can help to prove that a recursive function produces the correct result. Understanding recursion is a big step toward understanding induction, and vice versa, since they work by essentially the same process.

Within the context of algorithm analysis, one of the most important uses for mathematical induction is as a method to test a hypothesis. When seeking a closed-form solution for a summation or recurrence, we might first guess or otherwise acquire evidence that a particular formula is the correct solution. If the formula is indeed correct, it is often an easy matter to prove that fact with an induction proof.

Let Thrm be a theorem to prove, and express Thrm in terms of a positive integer parameter . Mathematical induction states that Thrm is true for any value of parameter (for , where c is some constant) if the following two conditions are true:

1. Base Case: Thrm holds for , and
2. Induction Step: If Thrm holds for , then Thrm holds for .

Proving the base case is usually easy, typically requiring that some small value such as 1 be substituted for in the theorem and applying simple algebra or logic as necessary to verify the theorem. Proving the induction step is sometimes easy, and sometimes difficult. An alternative formulation of the induction step is known as strong induction. The induction step for strong induction is:

2a. Induction Step:

If Thrm holds for all , then Thrm holds for .

Proving either variant of the induction step (in conjunction with verifying the base case) yields a satisfactory proof by mathematical induction.

The two conditions that make up the induction proof combine to demonstrate that Thrm holds for as an extension of the fact that Thrm holds for . This fact, combined again with condition (2) or (2a), indicates that Thrm also holds for , and so on. Thus, Thrm holds for all values of (larger than the base cases) once the two conditions have been proved.

What makes mathematical induction so powerful (and so mystifying to most people at first) is that we can take advantage of the assumption that Thrm holds for all values less than as a tool to help us prove that Thrm holds for . This is known as the induction hypothesis. Having this assumption to work with makes the induction step easier to prove than tackling the original theorem itself. Being able to rely on the induction hypothesis provides extra information that we can bring to bear on the problem.

Recursion and induction have many similarities. Both are anchored on one or more base cases. A recursive function relies on the ability to call itself to get the answer for smaller instances of the problem. Likewise, induction proofs rely on the truth of the induction hypothesis to prove the theorem. The induction hypothesis does not come out of thin air. It is true if and only if the theorem itself is true, and therefore is reliable within the proof context. Using the induction hypothesis it do work is exactly the same as using a recursive call to do work.

Example 2.7.3

Here is a sample proof by mathematical induction. Call the sum of the first positive integers .

Theorem: .

Proof: The proof is by mathematical induction.

1. Check the base case. For , verify that . is simply the sum of the first positive number, which is 1. Because , the formula is correct for the base case.
2. State the induction hypothesis. The induction hypothesis is

3. Use the assumption from the induction hypothesis for to show that the result is true for . The induction hypothesis states that , and because , we can substitute for to get

Thus, by mathematical induction,

Note carefully what took place in this example. First we cast in terms of a smaller occurrence of the problem: . This is important because once comes into the picture, we can use the induction hypothesis to replace with . From here, it is simple algebra to prove that equals the right-hand side of the original theorem.

We can compare the induction proof of Example 2.7.3 with the direct proof in Example 2.7.1. Different people might think one is easier to understand than the other, but certainly the writer of the direct proof version had to discover an insight unique to that problem that might not be helpful or relevant when proving other summations.

Example 2.7.4

Here is another simple proof by induction that illustrates choosing the proper variable for induction. We wish to prove by induction that the sum of the first positive odd numbers is . First we need a way to describe the ’th odd number, which is simply . This also allows us to cast the theorem as a summation.

Theorem: .

Proof: The base case of yields , which is true. The induction hypothesis is

We now use the induction hypothesis to show that the theorem holds true for . The sum of the first odd numbers is simply the sum of the first odd numbers plus the ’th odd number. In the second line below, we will use the induction hypothesis to replace the partial summation (shown in brackets in the first line) with its closed-form solution. After that, algebra takes care of the rest.

Thus, by mathematical induction,

Example 2.7.5

This example shows how we can use induction to prove that a proposed closed-form solution for a recurrence relation is correct.

Theorem: The recurrence relation has closed-form solution .

Proof: To prove the base case, we observe from the definition that . From the proposed closed-form solution we get , which matches the definition.

The induction hypothesis is that . Combining the definition of the recurrence with the induction hypothesis, we see immediately that

for . Thus, we have proved the theorem correct by mathematical induction.

Example 2.7.6

This example uses induction without involving summations or other equations. It also illustrates a more flexible use of base cases.

Theorem: 2 cent and 5 cent stamps can be used to form any value (for values ).

Proof: The theorem defines the problem for values because it does not hold for the values 1 and 3. Using 4 as the base case, a value of 4 cents can be made from two 2 cent stamps. The induction hypothesis is that a value of can be made from some combination of 2 cent and 5 cent stamps. We now use the induction hypothesis to show how to get the value from 2 cent and 5 cent stamps. Either the makeup for value includes a 5 cent stamp, or it does not. If so, then replace a 5 cent stamp with three 2 cent stamps. If not, then the makeup must have included at least two 2 cent stamps (because it is at least of size 4 and contains only 2 cent stamps). In this case, replace two of the 2 cent stamps with a single 5 cent stamp. In either case, we now have a value of n made up of 2 cent and 5 cent stamps. Thus, by mathematical induction, the theorem is correct.

Example 2.7.7

Here is an example using strong induction.

Theorem: For is divisible by some prime number.

Proof: For the base case, choose . 2 is divisible by the prime number 2. The induction hypothesis is that any value , is divisible by some prime number. There are now two cases to consider when proving the theorem for . If is a prime number, then is divisible by itself. If is not a prime number, then for and , both integers less than but greater than 1. The induction hypothesis tells us that is divisible by some prime number. That same prime number must also divide . Thus, by mathematical induction, the theorem is correct.

Our next example of mathematical induction proves a theorem from geometry. It also illustrates a standard technique of induction proof where we take objects and remove some object to use the induction hypothesis.

Figure 2.7.1: A two-coloring for the regions formed by three lines in the plane.

Example 2.7.8

Define a two-coloring for a set of regions as a way of assigning one of two colors to each region such that no two regions sharing a side have the same color. For example, a chessboard is two-colored. Figure 2.7.1 shows a two-coloring for the plane with three lines. We will assume that the two colors to be used are black and white.

Theorem: The set of regions formed by infinite lines in the plane can be two-colored.

Proof:

Compare the proof in Example 2.7.8 with that in Example 2.7.6. For Example 2.7.6, we took a collection of stamps of size (which, by the induction hypothesis, must have the desired property) and from that “built” a collection of size that has the desired property. We therefore proved the existence of some collection of stamps of size with the desired property.

For Example 2.7.8 we must prove that any collection of lines has the desired property. Thus, our strategy is to take an arbitrary collection of lines, and “reduce” it so that we have a set of lines that must have the desired property because it matches the induction hypothesis. From there, we merely need to show that reversing the original reduction process preserves the desired property.

In contrast, consider what is required if we attempt to “build” from a set of lines of size to one of size . We would have great difficulty justifying that all possible collections of lines are covered by our building process. By reducing from an arbitrary collection of lines to something less, we avoid this problem.

Another advantage to thinking in terms of “reducing from ” rather than “building up from ” is that reducing is more like what we do when we write a recursive function. In recursion, we would naturally compute some function of by calling the function (recursively) on and then using the result to compute the value for .

This section’s final example shows how induction can be used to prove that a recursive function produces the correct result.

Example 2.7.9

We would like to prove that function fact does indeed compute the factorial function. There are two distinct steps to such a proof. The first is to prove that the function always terminates. The second is to prove that the function returns the correct value.

Theorem: Function fact will terminate for any value of .

Proof: For the base case, we observe that fact will terminate directly whenever . The induction hypothesis is that fact will terminate for . For , we have two possibilities. One possibility is that . In that case, fact will terminate directly because it will fail its assertion test. Otherwise, fact will make a recursive call to fact(n-1). By the induction hypothesis, fact(n-1) must terminate.

Theorem: Function fact does compute the factorial function for any value in the range 0 to 12.

Proof: To prove the base case, observe that when or , fact(n) returns the correct value of 1. The induction hypothesis is that fact(n-1) returns the correct value of (n-1)!. For any value n within the legal range, fact(n) returns fact(n-1). By the induction hypothesis, fact(n-1) , and because , we have proved that fact(n) produces the correct result.

We can use a similar process to prove many recursive programs correct. The general form is to show that the base cases perform correctly, and then to use the induction hypothesis to show that the recursive step also produces the correct result. Prior to this, we must prove that the function always terminates, which might also be done using an induction proof.

2.8. Estimation

One of the most useful life skills that you can gain from your computer science training is the ability to perform quick estimates. This is sometimes known as “back of the napkin” or “back of the envelope” calculation. Both nicknames suggest that only a rough estimate is produced. Estimation techniques are a standard part of engineering curricula but are often neglected in computer science. Estimation is no substitute for rigorous, detailed analysis of a problem, but it can help to decide when a rigorous analysis is warranted: If the initial estimate indicates that the solution is unworkable, then further analysis is probably unnecessary.

Estimation can be formalized by the following three-step process:

1. Determine the major parameters that affect the problem.
2. Derive an equation that relates the parameters to the problem.
3. Select values for the parameters, and apply the equation to yield an estimated solution.

When doing estimations, a good way to reassure yourself that the estimate is reasonable is to do it in two different ways. In general, if you want to know what comes out of a system, you can either try to estimate that directly, or you can estimate what goes into the system (assuming that what goes in must later come out). If both approaches (independently) give similar answers, then this should build confidence in the estimate.

When calculating, be sure that your units match. For example, do not add feet and pounds. Verify that the result is in the correct units. Always keep in mind that the output of a calculation is only as good as its input. The more uncertain your valuation for the input parameters in Step 3, the more uncertain the output value. However, back of the envelope calculations are often meant only to get an answer within an order of magnitude, or perhaps within a factor of two. Before doing an estimate, you should decide on acceptable error bounds, such as within 25%, within a factor of two, and so forth. Once you are confident that an estimate falls within your error bounds, leave it alone! Do not try to get a more precise estimate than necessary for your purpose.

Example 2.8.1

How many library bookcases does it take to store books containing one million pages? I estimate that a 500-page book requires one inch on the library shelf (it will help to look at the size of any handy book), yielding about 200 feet of shelf space for one million pages. If a shelf is 4 feet wide, then 50 shelves are required. If a bookcase contains 5 shelves, this yields about 10 library bookcases. To reach this conclusion, I estimated the number of pages per inch, the width of a shelf, and the number of shelves in a bookcase. None of my estimates are likely to be precise, but I feel confident that my answer is correct to within a factor of two. (After writing this, I went to Virginia Tech’s library and looked at some real bookcases. They were only about 3 feet wide, but typically had 7 shelves for a total of 21 shelf-feet. So I was correct to within 10% on bookcase capacity, far better than I expected or needed. One of my selected values was too high, and the other too low, which canceled out the errors.)

Example 2.8.2

Is it more economical to buy a car that gets 20 miles per gallon, or one that gets 30 miles per gallon but costs 3/gallon, then the yearly gas bill is 1200 for the more efficient car. If we ignore issues such as the payback that would be received if we invested $3000 in a bank, it would take 5 years to make up the difference in price. At this point, the buyer must decide if price is the only criterion and if a 5-year payback time is acceptable. Naturally, a person who drives more will make up the difference more quickly, and changes in gasoline prices will also greatly affect the outcome.

Example 2.8.3

When at the supermarket doing the week’s shopping, can you estimate about how much you will have to pay at the checkout? One simple way is to round the price of each item to the nearest dollar, and add this value to a mental running total as you put the item in your shopping cart. This will likely give an answer within a couple of dollars of the true total.

2.9. Chapter Summary Questions

Here are some practice questions for the modules in this chapter.

Chapter 3: Algorithm Analysis

3.1. Chapter Introduction

How long will it take to process the company payroll once we complete our planned merger? Should I buy a new payroll program from vendor X or vendor Y? If a particular program is slow, is it badly implemented or is it solving a hard problem? Questions like these ask us to consider the difficulty of a problem, or the relative efficiency of two or more approaches to solving a problem.

This chapter introduces the motivation, basic notation, and fundamental techniques of algorithm analysis. We focus on a methodology known as asymptotic algorithm analysis, or simply asymptotic analysis. Asymptotic analysis attempts to estimate the resource consumption of an algorithm. It allows us to compare the relative costs of two or more algorithms for solving the same problem. Asymptotic analysis also gives algorithm designers a tool for estimating whether a proposed solution is likely to meet the resource constraints for a problem before they implement an actual program. After reading this chapter, you should understand

• the concept of a growth rate, the rate at which the cost of an algorithm grows as the size of its input grows;
• the concept of an upper bound and lower bound for a growth rate, and how to estimate these bounds for a simple program, algorithm, or problem; and
• the difference between the cost of an algorithm (or program) and the cost of a problem.

The chapter concludes with a brief discussion of the practical difficulties encountered when empirically measuring the cost of a program, and some principles for code tuning to improve program efficiency.

3.2. Problems, Algorithms, and Programs

3.2.1. Problems, Algorithms, and Programs

3.2.1.1. Problems

Programmers commonly deal with problems, algorithms, and computer programs. These are three distinct concepts.

As your intuition would suggest, a problem is a task to be performed. It is best thought of in terms of inputs and matching outputs. A problem definition should not include any constraints on how the problem is to be solved. The solution method should be developed only after the problem is precisely defined and thoroughly understood. However, a problem definition should include constraints on the resources that may be consumed by any acceptable solution. For any problem to be solved by a computer, there are always such constraints, whether stated or implied. For example, any computer program may use only the main memory and disk space available, and it must run in a “reasonable” amount of time.

Problems can be viewed as functions in the mathematical sense. A function is a matching between inputs (the domain) and outputs (the range). An input to a function might be a single value or a collection of information. The values making up an input are called the parameters of the function. A specific selection of values for the parameters is called an instance of the problem. For example, the input parameter to a sorting function might be an array of integers. A particular array of integers, with a given size and specific values for each position in the array, would be an instance of the sorting problem. Different instances might generate the same output. However, any problem instance must always result in the same output every time the function is computed using that particular input.

This concept of all problems behaving like mathematical functions might not match your intuition for the behavior of computer programs. You might know of programs to which you can give the same input value on two separate occasions, and two different outputs will result. For example, if you type date to a typical Linux command line prompt, you will get the current date. Naturally the date will be different on different days, even though the same command is given. However, there is obviously more to the input for the date program than the command that you type to run the program. The date program computes a function. In other words, on any particular day there can only be a single answer returned by a properly running date program on a completely specified input. For all computer programs, the output is completely determined by the program’s full set of inputs. Even a “random number generator” is completely determined by its inputs (although some random number generating systems appear to get around this by accepting a random input from a physical process beyond the user’s control). The limits to what functions can be implemented by programs is part of the domain of Computability.

3.2.1.2. Algorithms

An algorithm is a method or a process followed to solve a problem. If the problem is viewed as a function, then an algorithm is an implementation for the function that transforms an input to the corresponding output. A problem can be solved by many different algorithms. A given algorithm solves only one problem (i.e., computes a particular function). OpenDSA modules cover many problems, and for several of these problems we will see more than one algorithm. For the important problem of sorting there are over a dozen commonly known algorithms!

The advantage of knowing several solutions to a problem is that solution might be more efficient than solution for a specific variation of the problem, or for a specific class of inputs to the problem, while solution might be more efficient than for another variation or class of inputs. For example, one sorting algorithm might be the best for sorting a small collection of integers (which is important if you need to do this many times). Another might be the best for sorting a large collection of integers. A third might be the best for sorting a collection of variable-length strings.

By definition, something can only be called an algorithm if it has all of the following properties.

1. It must be correct. In other words, it must compute the desired function, converting each input to the correct output. Note that every algorithm implements some function, because every algorithm maps every input to some output (even if that output is a program crash). At issue here is whether a given algorithm implements the intended function.
2. It is composed of a series of concrete steps. Concrete means that the action described by that step is completely understood — and doable — by the person or machine that must perform the algorithm. Each step must also be doable in a finite amount of time. Thus, the algorithm gives us a “recipe” for solving the problem by performing a series of steps, where each such step is within our capacity to perform. The ability to perform a step can depend on who or what is intended to execute the recipe. For example, the steps of a cookie recipe in a cookbook might be considered sufficiently concrete for instructing a human cook, but not for programming an automated cookie-making factory.
3. There can be no ambiguity as to which step will be performed next. Often it is the next step of the algorithm description. Selection (e.g., the if statement) is normally a part of any language for describing algorithms. Selection allows a choice for which step will be performed next, but the selection process is unambiguous at the time when the choice is made.
4. It must be composed of a finite number of steps. If the description for the algorithm were made up of an infinite number of steps, we could never hope to write it down, nor implement it as a computer program. Most languages for describing algorithms (including English and “pseudocode”) provide some way to perform repeated actions, known as iteration. Examples of iteration in programming languages include the while and for loop constructs. Iteration allows for short descriptions, with the number of steps actually performed controlled by the input.
5. It must terminate. In other words, it may not go into an infinite loop.

3.2.1.3. Programs

We often think of a computer program as an instance, or concrete representation, of an algorithm in some programming language. Algorithms are usually presented in terms of programs, or parts of programs. Naturally, there are many programs that are instances of the same algorithm, because any modern computer programming language can be used to implement the same collection of algorithms (although some programming languages can make life easier for the programmer). To simplify presentation, people often use the terms “algorithm” and “program” interchangeably, despite the fact that they are really separate concepts. By definition, an algorithm must provide sufficient detail that it can be converted into a program when needed.

The requirement that an algorithm must terminate means that not all computer programs meet the technical definition of an algorithm. Your operating system is one such program. However, you can think of the various tasks for an operating system (each with associated inputs and outputs) as individual problems, each solved by specific algorithms implemented by a part of the operating system program, and each one of which terminates once its output is produced.

To summarize: A problem is a function or a mapping of inputs to outputs. An algorithm is a recipe for solving a problem whose steps are concrete and unambiguous. Algorithms must be correct, of finite length, and must terminate for all inputs. A program is an instantiation of an algorithm in a programming language. The following slideshow should help you to visualize the differences.

3.3. Comparing Algorithms

3.3.1. Comparing Algorithms

3.3.1.1. Introduction

How do you compare two algorithms for solving some problem in terms of efficiency? We could implement both algorithms as computer programs and then run them on a suitable range of inputs, measuring how much of the resources in question each program uses. This approach is often unsatisfactory for four reasons. First, there is the effort involved in programming and testing two algorithms when at best you want to keep only one. Second, when empirically comparing two algorithms there is always the chance that one of the programs was “better written” than the other, and therefore the relative qualities of the underlying algorithms are not truly represented by their implementations. This can easily occur when the programmer has a bias regarding the algorithms. Third, the choice of empirical test cases might unfairly favor one algorithm. Fourth, you could find that even the better of the two algorithms does not fall within your resource budget. In that case you must begin the entire process again with yet another program implementing a new algorithm. But, how would you know if any algorithm can meet the resource budget? Perhaps the problem is simply too difficult for any implementation to be within budget.

These problems can often be avoided by using asymptotic analysis. Asymptotic analysis measures the efficiency of an algorithm, or its implementation as a program, as the input size becomes large. It is actually an estimating technique and does not tell us anything about the relative merits of two programs where one is always “slightly faster” than the other. However, asymptotic analysis has proved useful to computer scientists who must determine if a particular algorithm is worth considering for implementation.

The critical resource for a program is most often its running time. However, you cannot pay attention to running time alone. You must also be concerned with other factors such as the space required to run the program (both main memory and disk space). Typically you will analyze the time required for an algorithm (or the instantiation of an algorithm in the form of a program), and the space required for a data structure.

Many factors affect the running time of a program. Some relate to the environment in which the program is compiled and run. Such factors include the speed of the computer’s CPU, bus, and peripheral hardware. Competition with other users for the computer’s (or the network’s) resources can make a program slow to a crawl. The programming language and the quality of code generated by a particular compiler can have a significant effect. The “coding efficiency” of the programmer who converts the algorithm to a program can have a tremendous impact as well.

If you need to get a program working within time and space constraints on a particular computer, all of these factors can be relevant. Yet, none of these factors address the differences between two algorithms or data structures. To be fair, if you want to compare two programs derived from two algorithms for solving the same problem, they should both be compiled with the same compiler and run on the same computer under the same conditions. As much as possible, the same amount of care should be taken in the programming effort devoted to each program to make the implementations “equally efficient”. In this sense, all of the factors mentioned above should cancel out of the comparison because they apply to both algorithms equally.

If you truly wish to understand the running time of an algorithm, there are other factors that are more appropriate to consider than machine speed, programming language, compiler, and so forth. Ideally we would measure the running time of the algorithm under standard benchmark conditions. However, we have no way to calculate the running time reliably other than to run an implementation of the algorithm on some computer. The only alternative is to use some other measure as a surrogate for running time.

3.3.1.2. Basic Operations and Input Size

Of primary consideration when estimating an algorithm’s performance is the number of basic operations required by the algorithm to process an input of a certain size. The terms “basic operations” and “size” are both rather vague and depend on the algorithm being analyzed. Size is often the number of inputs processed. For example, when comparing sorting algorithms the size of the problem is typically measured by the number of records to be sorted. A basic operation must have the property that its time to complete does not depend on the particular values of its operands. Adding or comparing two integer variables are examples of basic operations in most programming languages. Summing the contents of an array containing integers is not, because the cost depends on the value of (i.e., the size of the input).

Example 3.3.1

Consider a simple algorithm to solve the problem of finding the largest value in an array of integers. The algorithm looks at each integer in turn, saving the position of the largest value seen so far. This algorithm is called the largest-value sequential search and is illustrated by the following function:

// Return position of largest value in integer array A
static int largest(int[] A) {
  int currlarge = 0;             // Position of largest element seen
  for (int i=1; i<A.length; i++) // For each element
    if (A[currlarge] < A[i])     //   if A[i] is larger
       currlarge = i;            //     remember its position
  return currlarge;              // Return largest position
}

Here, the size of the problem is A.length, the number of integers stored in array A. The basic operation is to compare an integer’s value to that of the largest value seen so far. It is reasonable to assume that it takes a fixed amount of time to do one such comparison, regardless of the value of the two integers or their positions in the array.

Because the most important factor affecting running time is normally size of the input, for a given input size we often express the time to run the algorithm as a function of , written as . We will always assume is a non-negative value.

Let us call the amount of time required to compare two integers in function largest. We do not care right now what the precise value of might be. Nor are we concerned with the time required to increment variable because this must be done for each value in the array, or the time for the actual assignment when a larger value is found, or the little bit of extra time taken to initialize currlarge. We just want a reasonable approximation for the time taken to execute the algorithm. The total time to run largest is therefore approximately , because we must make comparisons, with each comparison costing time. We say that function largest (and by extension, the largest-value sequential search algorithm for any typical implementation) has a running time expressed by the equation

This equation describes the growth rate for the running time of the largest-value sequential search algorithm.

Example 3.3.2

The running time of a statement that assigns the first value of an integer array to a variable is simply the time required to copy the value of the first array value. We can assume this assignment takes a constant amount of time regardless of the value. Let us call the amount of time necessary to copy an integer. No matter how large the array on a typical computer (given reasonable conditions for memory and array size), the time to copy the value from the first position of the array is always . Thus, the equation for this algorithm is simply

indicating that the size of the input has no effect on the running time. This is called a constant running time.

Example 3.3.3

Consider the following code:

sum = 0;
for (i=1; i<=n; i++)
  for (j=1; j<=n; j++)
    sum++;

What is the running time for this code fragment? Clearly it takes longer to run when is larger. The basic operation in this example is the increment operation for variable sum. We can assume that incrementing takes constant time; call this time . (We can ignore the time required to initialize sum, and to increment the loop counters i and j. In practice, these costs can safely be bundled into time .) The total number of increment operations is . Thus, we say that the running time is .

3.3.1.3. Growth Rates

The growth rate for an algorithm is the rate at which the cost of the algorithm grows as the size of its input grows. The following figure shows a graph for six equations, each meant to describe the running time for a particular program or algorithm. A variety of growth rates that are representative of typical algorithms are shown.

Figure 3.3.2: Two views of a graph illustrating the growth rates for six equations. The bottom view shows in detail the lower-left portion of the top view. The horizontal axis represents input size. The vertical axis can represent time, space, or any other measure of cost.

The two equations labeled and are graphed by straight lines. A growth rate of (for any positive constant) is often referred to as a linear growth rate or running time. This means that as the value of grows, the running time of the algorithm grows in the same proportion. Doubling the value of roughly doubles the running time. An algorithm whose running-time equation has a highest-order term containing a factor of is said to have a quadratic growth rate. In the figure, the line labeled represents a quadratic growth rate. The line labeled represents an exponential growth rate. This name comes from the fact that appears in the exponent. The line labeled also grows exponentially.

As you can see from the figure, the difference between an algorithm whose running time has cost and another with cost becomes tremendous as grows. For , the algorithm with running time is already much slower. This is despite the fact that has a greater constant factor than . Comparing the two curves marked and shows that changing the constant factor for one of the equations only shifts the point at which the two curves cross. For , the algorithm with cost is slower than the algorithm with cost . This graph also shows that the equation grows somewhat more quickly than both and , but not nearly so quickly as the equation . For constants grows faster than either or . Finally, algorithms with cost or are prohibitively expensive for even modest values of . Note that for constants grows faster than .

We can get some further insight into relative growth rates for various algorithms from the following table. Most of the growth rates that appear in typical algorithms are shown, along with some representative input sizes. Once again, we see that the growth rate has a tremendous effect on the resources consumed by an algorithm.

Table 3.3.1

Costs for representative growth rates.

3.3.2. Growth Rates Ordering Exercise

3.4. Best, Worst, and Average Cases

3.4.1. Best, Worst, and Average Cases

When analyzing an algorithm, should we study the best, worst, or average case? Normally we are not interested in the best case, because this might happen only rarely and generally is too optimistic for a fair characterization of the algorithm’s running time. In other words, analysis based on the best case is not likely to be representative of the behavior of the algorithm. However, there are rare instances where a best-case analysis is useful—in particular, when the best case has high probability of occurring. The Shellsort and Quicksort algorithms both can take advantage of the best-case running time of Insertion Sort to become more efficient.

How about the worst case? The advantage to analyzing the worst case is that you know for certain that the algorithm must perform at least that well. This is especially important for real-time applications, such as for the computers that monitor an air traffic control system. Here, it would not be acceptable to use an algorithm that can handle airplanes quickly enough most of the time, but which fails to perform quickly enough when all airplanes are coming from the same direction.

For other applications—particularly when we wish to aggregate the cost of running the program many times on many different inputs—worst-case analysis might not be a representative measure of the algorithm’s performance. Often we prefer to know the average-case running time. This means that we would like to know the typical behavior of the algorithm on inputs of size . Unfortunately, average-case analysis is not always possible. Average-case analysis first requires that we understand how the actual inputs to the program (and their costs) are distributed with respect to the set of all possible inputs to the program. For example, it was stated previously that the sequential search algorithm on average examines half of the array values. This is only true if the element with value is equally likely to appear in any position in the array. If this assumption is not correct, then the algorithm does not necessarily examine half of the array values in the average case.

The characteristics of a data distribution have a significant effect on many search algorithms, such as those based on hashing and search trees such as the BST. Incorrect assumptions about data distribution can have disastrous consequences on a program’s space or time performance. Unusual data distributions can also be used to advantage, such as is done by self-organizing lists.

In summary, for real-time applications we are likely to prefer a worst-case analysis of an algorithm. Otherwise, we often desire an average-case analysis if we know enough about the distribution of our input to compute the average case. If not, then we must resort to worst-case analysis.

3.5. Faster Computer, or Faster Algorithm?

3.5.1. Faster Computer, or Faster Algorithm?

Imagine that you have a problem to solve, and you know of an algorithm whose running time is proportional to where is a measure of the input size. Unfortunately, the resulting program takes ten times too long to run. If you replace your current computer with a new one that is ten times faster, will the algorithm become acceptable? If the problem size remains the same, then perhaps the faster computer will allow you to get your work done quickly enough even with an algorithm having a high growth rate. But a funny thing happens to most people who get a faster computer. They don’t run the same problem faster. They run a bigger problem! Say that on your old computer you were content to sort 10,000 records because that could be done by the computer during your lunch break. On your new computer you might hope to sort 100,000 records in the same time. You won’t be back from lunch any sooner, so you are better off solving a larger problem. And because the new machine is ten times faster, you would like to sort ten times as many records.

If your algorithm’s growth rate is linear (i.e., if the equation that describes the running time on input size is for some constant ), then 100,000 records on the new machine will be sorted in the same time as 10,000 records on the old machine. If the algorithm’s growth rate is greater than , such as , then you will not be able to do a problem ten times the size in the same amount of time on a machine that is ten times faster.

How much larger a problem can be solved in a given amount of time by a faster computer? Assume that the new machine is ten times faster than the old. Say that the old machine could solve a problem of size in an hour. What is the largest problem that the new machine can solve in one hour? The following table shows how large a problem can be solved on the two machines for five running-time functions.

Table 3.5.1

The increase in problem size that can be run in a fixed period of time on a computer that is ten times faster. The first column lists the right-hand sides for five growth rate equations. For the purpose of this example, arbitrarily assume that the old machine can run 10,000 basic operations in one hour. The second column shows the maximum value for that can be run in 10,000 basic operations on the old machine. The third column shows the value for , the new maximum size for the problem that can be run in the same time on the new machine that is ten times faster. Variable is the greatest size for the problem that can run in 100,000 basic operations. The fourth column shows how the size of changed to become on the new machine. The fifth column shows the increase in the problem size as the ratio of to .

This table illustrates many important points. The first two equations are both linear; only the value of the constant factor has changed. In both cases, the machine that is ten times faster gives an increase in problem size by a factor of ten. In other words, while the value of the constant does affect the absolute size of the problem that can be solved in a fixed amount of time, it does not affect the improvement in problem size (as a proportion to the original size) gained by a faster computer. This relationship holds true regardless of the algorithm’s growth rate: Constant factors never affect the relative improvement gained by a faster computer.

An algorithm with time equation does not receive nearly as great an improvement from the faster machine as an algorithm with linear growth rate. Instead of an improvement by a factor of ten, the improvement is only the square root of that: . Thus, the algorithm with higher growth rate not only solves a smaller problem in a given time in the first place, it also receives less of a speedup from a faster computer. As computers get ever faster, the disparity in problem sizes becomes ever greater.

The algorithm with growth rate improves by a greater amount than the one with quadratic growth rate, but not by as great an amount as the algorithms with linear growth rates.

Note that something special happens in the case of the algorithm whose running time grows exponentially. If you look at its plot on a graph, the curve for the algorithm whose time is proportional to goes up very quickly as grows. The increase in problem size on the machine ten times as fast is about (to be precise, it is ). The increase in problem size for an algorithm with exponential growth rate is by a constant addition, not by a multiplicative factor. Because the old value of was 13, the new problem size is 16. If next year you buy another computer ten times faster yet, then the new computer (100 times faster than the original computer) will only run a problem of size 19. If you had a second program whose growth rate is and for which the original computer could run a problem of size 1000 in an hour, than a machine ten times faster can run a problem only of size 1003 in an hour! Thus, an exponential growth rate is radically different than the other growth rates shown in the table. The significance of this difference is an important topic in computational complexity theory.

Instead of buying a faster computer, consider what happens if you replace an algorithm whose running time is proportional to with a new algorithm whose running time is proportional to . In a graph relating growth rate functions to input size, a fixed amount of time would appear as a horizontal line. If the line for the amount of time available to solve your problem is above the point at which the curves for the two growth rates in question meet, then the algorithm whose running time grows less quickly is faster. An algorithm with running time requires time steps for an input of size . An algorithm with running time requires time steps for an input of size , which is an improvement of much more than a factor of ten when compared to the algorithm with running time . Because whenever , if the typical problem size is larger than 58 for this example, then you would be much better off changing algorithms instead of buying a computer ten times faster. Furthermore, when you do buy a faster computer, an algorithm with a slower growth rate provides a greater benefit in terms of larger problem size that can run in a certain time on the new computer.

3.6. Asymptotic Analysis and Upper Bounds

3.6.1. Asymptotic Analysis and Upper Bounds

Figure 3.6.2: Two views of a graph illustrating the growth rates for six equations. The bottom view shows in detail the lower-left portion of the top view. The horizontal axis represents input size. The vertical axis can represent time, space, or any other measure of cost.

Despite the larger constant for the curve labeled in the figure above, crosses it at the relatively small value of . What if we double the value of the constant in front of the linear equation? As shown in the graph, is surpassed by once . The additional factor of two for the linear growth rate does not much matter. It only doubles the -coordinate for the intersection point. In general, changes to a constant factor in either equation only shift where the two curves cross, not whether the two curves cross.

When you buy a faster computer or a faster compiler, the new problem size that can be run in a given amount of time for a given growth rate is larger by the same factor, regardless of the constant on the running-time equation. The time curves for two algorithms with different growth rates still cross, regardless of their running-time equation constants. For these reasons, we usually ignore the constants when we want an estimate of the growth rate for the running time or other resource requirements of an algorithm. This simplifies the analysis and keeps us thinking about the most important aspect: the growth rate. This is called asymptotic algorithm analysis. To be precise, asymptotic analysis refers to the study of an algorithm as the input size “gets big” or reaches a limit (in the calculus sense). However, it has proved to be so useful to ignore all constant factors that asymptotic analysis is used for most algorithm comparisons.

In rare situations, it is not reasonable to ignore the constants. When comparing algorithms meant to run on small values of , the constant can have a large effect. For example, if the problem requires you to sort many collections of exactly five records, then a sorting algorithm designed for sorting thousands of records is probably not appropriate, even if its asymptotic analysis indicates good performance. There are rare cases where the constants for two algorithms under comparison can differ by a factor of 1000 or more, making the one with lower growth rate impractical for typical problem sizes due to its large constant. Asymptotic analysis is a form of “back of the envelope” estimation for algorithm resource consumption. It provides a simplified model of the running time or other resource needs of an algorithm. This simplification usually helps you understand the behavior of your algorithms. Just be aware of the limitations to asymptotic analysis in the rare situation where the constant is important.

3.6.1.1. Upper Bounds

Several terms are used to describe the running-time equation for an algorithm. These terms—and their associated symbols—indicate precisely what aspect of the algorithm’s behavior is being described. One is the upper bound for the growth of the algorithm’s running time. It indicates the upper or highest growth rate that the algorithm can have.

Because the phrase “has an upper bound to its growth rate of ” is long and often used when discussing algorithms, we adopt a special notation, called big-Oh notation. If the upper bound for an algorithm’s growth rate (for, say, the worst case) is (f(n)), then we would write that this algorithm is “in the set in the worst case” (or just “in in the worst case”). For example, if grows as fast as (the running time of our algorithm) for the worst-case input, we would say the algorithm is “in in the worst case”.

The following is a precise definition for an upper bound. represents the true running time of the algorithm. is some expression for the upper bound.

For a non-negatively valued function, is in set if there exist two positive constants and such that for all .

Constant is the smallest value of for which the claim of an upper bound holds true. Usually is small, such as 1, but does not need to be. You must also be able to pick some constant , but it is irrelevant what the value for actually is. In other words, the definition says that for all inputs of the type in question (such as the worst case for all inputs of size ) that are large enough (i.e., ), the algorithm always executes in less than or equal to steps for some constant .

Example 3.6.1

Consider the sequential search algorithm for finding a specified value in an array of integers. If visiting and examining one value in the array requires steps where is a positive number, and if the value we search for has equal probability of appearing in any position in the array, then in the average case . For all values of , . Therefore, by the definition, is in for and .

Example 3.6.2

For a particular algorithm, in the average case where and are positive numbers. Then,

for all . So, for , and . Therefore, is in by the second definition.

Example 3.6.3

Assigning the value from the first position of an array to a variable takes constant time regardless of the size of the array. Thus, (for the best, worst, and average cases). We could say in this case that is in . However, it is traditional to say that an algorithm whose running time has a constant upper bound is in .

If someone asked you out of the blue “Who is the best?” your natural reaction should be to reply “Best at what?” In the same way, if you are asked “What is the growth rate of this algorithm”, you would need to ask “When? Best case? Average case? Or worst case?” Some algorithms have the same behavior no matter which input instance of a given size that they receive. An example is finding the maximum in an array of integers. But for many algorithms, it makes a big difference which particular input of a given size is involved, such as when searching an unsorted array for a particular value. So any statement about the upper bound of an algorithm must be in the context of some specific class of inputs of size . We measure this upper bound nearly always on the best-case, average-case, or worst-case inputs. Thus, we cannot say, “this algorithm has an upper bound to its growth rate of ” because that is an incomplete statement. We must say something like, “this algorithm has an upper bound to its growth rate of in the average case“.

Knowing that something is in says only how bad things can be. Perhaps things are not nearly so bad. Because sequential search is in in the worst case, it is also true to say that sequential search is in . But sequential search is practical for large in a way that is not true for some other algorithms in . We always seek to define the running time of an algorithm with the tightest (lowest) possible upper bound. Thus, we prefer to say that sequential search is in . This also explains why the phrase “is in ” or the notation “” is used instead of “is ” or “”. There is no strict equality to the use of big-Oh notation. is in , but is not in .

3.6.1.2. Simplifying Rules

Once you determine the running-time equation for an algorithm, it really is a simple matter to derive the big-Oh expressions from the equation. You do not need to resort to the formal definitions of asymptotic analysis. Instead, you can use the following rules to determine the simplest form.

1. If is in and is in , then is in .
2. If is in for any constant , then is in .
3. If is in and is in , then is in .
4. If is in and is in , then is in .

The first rule says that if some function is an upper bound for your cost function, then any upper bound for is also an upper bound for your cost function.

The significance of rule (2) is that you can ignore any multiplicative constants in your equations when using big-Oh notation.

Rule (3) says that given two parts of a program run in sequence (whether two statements or two sections of code), you need consider only the more expensive part.

Rule (4) is used to analyze simple loops in programs. If some action is repeated some number of times, and each repetition has the same cost, then the total cost is the cost of the action multiplied by the number of times that the action takes place.

Taking the first three rules collectively, you can ignore all constants and all lower-order terms to determine the asymptotic growth rate for any cost function. The advantages and dangers of ignoring constants were discussed near the beginning of this section. Ignoring lower-order terms is reasonable when performing an asymptotic analysis. The higher-order terms soon swamp the lower-order terms in their contribution to the total cost as (n) becomes larger. Thus, if , then is in . The term contributes relatively little to the total cost for large .

From now on, we will use these simplifying rules when discussing the cost for a program or algorithm.

3.6.1.5. Practice Questions

3.7. Lower Bounds and Notation

3.7.1. Lower Bounds and Theta Notation

3.7.1.1. Lower Bounds

Big-Oh notation describes an upper bound. In other words, big-Oh notation states a claim about the greatest amount of some resource (usually time) that is required by an algorithm for some class of inputs of size (typically the worst such input, the average of all possible inputs, or the best such input).

Similar notation is used to describe the least amount of a resource that an algorithm needs for some class of input. Like big-Oh notation, this is a measure of the algorithm’s growth rate. Like big-Oh notation, it works for any resource, but we most often measure the least amount of time required. And again, like big-Oh notation, we are measuring the resource required for some particular class of inputs: the worst-, average-, or best-case input of size .

The lower bound for an algorithm (or a problem, as explained later) is denoted by the symbol , pronounced “big-Omega” or just “Omega”. The following definition for is symmetric with the definition of big-Oh.

For a non-negatively valued function, is in set if there exist two positive constants and such that for all . [^1]

Example 3.7.1

Assume for and . Then,

for all . So, for and . Therefore, is in by the definition.

It is also true that the equation of the example above is in . However, as with big-Oh notation, we wish to get the “tightest” (for notation, the largest) bound possible. Thus, we prefer to say that this running time is in .

Recall the sequential search algorithm to find a value within an array of integers. In the average and worst cases this algorithm is in , because in both the average and worst cases we must examine at least values (where is 1/2 in the average case and 1 in the worst case).

[^1]: An alternate (non-equivalent) definition for is

is in the set if there exists a positive constant such that for an infinite number of values for . This definition says that for an “interesting” number of cases, the algorithm takes at least time. Note that this definition is not symmetric with the definition of big-Oh. For to be a lower bound, this definition does not require that for all values of greater than some constant. It only requires that this happen often enough, in particular that it happen for an infinite number of values for . Motivation for this alternate definition can be found in the following example. Assume a particular algorithm has the following behavior:

$$
\begin{aligned}\mathbf{T}(n) = \left{
\right.\end{aligned}
$$

From this definition, for all even . So, for an infinite number of values of (i.e., for all even ) for . Therefore, is in by the definition. For this equation for , it is true that all inputs of size take at least time. But an infinite number of inputs of size take time, so we would like to say that the algorithm is in . Unfortunately, using our first definition will yield a lower bound of because it is not possible to pick constants and such that for all . The alternative definition does result in a lower bound of for this algorithm, which seems to fit common sense more closely. Fortunately, few real algorithms or computer programs display the pathological behavior of this example. Our first definition for generally yields the expected result. As you can see from this discussion, asymptotic bounds notation is not a law of nature. It is merely a powerful modeling tool used to describe the behavior of algorithms.

3.7.1.2. Theta Notation

The definitions for big-Oh and give us ways to describe the upper bound for an algorithm (if we can find an equation for the maximum cost of a particular class of inputs of size ) and the lower bound for an algorithm (if we can find an equation for the minimum cost for a particular class of inputs of size ). When the upper and lower bounds are the same within a constant factor, we indicate this by using (big-Theta) notation. An algorithm is said to be if it is in and it is in . Note that we drop the word “in” for notation, because there is a strict equality for two equations with the same . In other words, if is , then is .

Because the sequential search algorithm is both in and in in the average case, we say it is in the average case.

Given an algebraic equation describing the time requirement for an algorithm, the upper and lower bounds always meet. That is because in some sense we have a perfect analysis for the algorithm, embodied by the running-time equation. For many algorithms (or their instantiations as programs), it is easy to come up with the equation that defines their runtime behavior. The analysis for most commonly used algorithms is well understood and we can almost always give a analysis for them. However, the class of NP-Complete problems all have no definitive analysis, just some unsatisfying big-Oh and analyses. Even some “simple” programs are hard to analyze. Nobody currently knows the true upper or lower bounds for the following code fragment.

while (n > 1)
  if (ODD(n))
    n = 3 * n + 1;
   else
     n = n / 2;

While some textbooks and programmers will casually say that an algorithm is “order of” or “big-Oh” of some cost function, it is generally better to use notation rather than big-Oh notation whenever we have sufficient knowledge about an algorithm to be sure that the upper and lower bounds indeed match. OpenDSA modules use notation in preference to big-Oh notation whenever our state of knowledge makes that possible. Limitations on our ability to analyze certain algorithms may require use of big-Oh or notations. In rare occasions when the discussion is explicitly about the upper or lower bound of a problem or algorithm, the corresponding notation will be used in preference to notation.

3.7.1.3. Classifying Functions

Given functions and whose growth rates are expressed as algebraic equations, we might like to determine if one grows faster than the other. The best way to do this is to take the limit of the two functions as grows towards infinity,

If the limit goes to , then is in because grows faster. If the limit goes to zero, then is in because grows faster. If the limit goes to some constant other than zero, then because both grow at the same rate.

Example 3.7.2

If and , is in , , or ? Since

we easily see that

because grows faster than . Thus, is in .

3.8. Calculating Program Running Time

3.8.1. Calculating Program Running Time

This modules discusses the analysis for several simple code fragments. We will make use of the algorithm analysis simplifying rules:

1. If is in and is in , then is in .
2. If is in for any constant , then is in .
3. If is in and is in , then is in .
4. If is in and is in , then is in .

Example 3.8.1

We begin with an analysis of a simple assignment to an integer variable.

a = b;

Because the assignment statement takes constant time, it is .

Example 3.8.2

Consider a simple for loop.

sum = 0;
for (i=1; i<=n; i++)
   sum += n;

The first line is . The for loop is repeated times. The third line takes constant time so, by simplifying rule (4), the total cost for executing the two lines making up the for loop is . By rule (3), the cost of the entire code fragment is also .

Example 3.8.3

We now analyze a code fragment with several for loops, some of which are nested.

sum = 0;
for (j=1; j<=n; j++)     // First for loop
   for (i=1; i<=j; i++)  //   is a double loop
      sum++;
for (k=0; k<n; k++)      // Second for loop
   A[k] = k;

This code fragment has three separate statements: the first assignment statement and the two for loops. Again the assignment statement takes constant time; call it . The second for loop is just like the one in Example 3.8.2 and takes time.

The first for loop is a double loop and requires a special technique. We work from the inside of the loop outward. The expression sum++ requires constant time; call it . Because the inner for loop is executed times, by simplifying rule (4) it has cost . The outer for loop is executed times, but each time the cost of the inner loop is different because it costs with changing each time. You should see that for the first execution of the outer loop, is 1. For the second execution of the outer loop, is 2. Each time through the outer loop, becomes one greater, until the last time through the loop when . Thus, the total cost of the loop is times the sum of the integers 1 through . We know that

which is . By simplifying rule (3), is simply .

Example 3.8.4

Compare the asymptotic analysis for the following two code fragments.

sum1 = 0;
for (i=1; i<=n; i++)     // First double loop
   for (j=1; j<=n; j++)  //   do n times
      sum1++;

sum2 = 0;
for (i=1; i<=n; i++)     // Second double loop
   for (j=1; j<=i; j++)  //   do i times
      sum2++;

In the first double loop, the inner for loop always executes times. Because the outer loop executes times, it should be obvious that the statement sum1++ is executed precisely times. The second loop is similar to the one analyzed in the previous example, with cost . This is approximately . Thus, both double loops cost , though the second requires about half the time of the first.

Example 3.8.5

Not all doubly nested for loops are . The following pair of nested loops illustrates this fact.

sum1 = 0;
for (k=1; k<=n; k*=2)    // Do log n times
   for (j=1; j<=n; j++)  // Do n times
      sum1++;

sum2 = 0;
for (k=1; k<=n; k*=2)    // Do log n times
   for (j=1; j<=k; j++)  // Do k times
      sum2++;

When analyzing these two code fragments, we will assume that is a power of two. The first code fragment has its outer for loop executed times because on each iteration is multiplied by two until it reaches . Because the inner loop always executes times, the total cost for the first code fragment can be expressed as

So the cost of this first double loop is . Note that a variable substitution takes place here to create the summation, with .

In the second code fragment, the outer loop is also executed times. The inner loop has cost , which doubles each time. The summation can be expressed as

where is assumed to be a power of two and again .

What about other control statements? While loops are analyzed in a manner similar to for loops. The cost of an if statement in the worst case is the greater of the costs for the then and else clauses. This is also true for the average case, assuming that the size of does not affect the probability of executing one of the clauses (which is usually, but not necessarily, true). For switch statements, the worst-case cost is that of the most expensive branch. For subroutine calls, simply add the cost of executing the subroutine.

There are rare situations in which the probability for executing the various branches of an if or switch statement are functions of the input size. For example, for input of size , the then clause of an if statement might be executed with probability . An example would be an if statement that executes the then clause only for the smallest of values. To perform an average-case analysis for such programs, we cannot simply count the cost of the if statement as being the cost of the more expensive branch. In such situations, the technique of amortized analysis can come to the rescue.

Determining the execution time of a recursive subroutine can be difficult. The running time for a recursive subroutine is typically best expressed by a recurrence relation. For example, the recursive factorial function calls itself with a value one less than its input value. The result of this recursive call is then multiplied by the input value, which takes constant time. Thus, the cost of the factorial function, if we wish to measure cost in terms of the number of multiplication operations, is one more than the number of multiplications made by the recursive call on the smaller input. Because the base case does no multiplications, its cost is zero. Thus, the running time for this function can be expressed as

The closed-form solution for this recurrence relation is .

3.8.1.1. Case Study: Two Search Algorithms

The final example of algorithm analysis for this section will compare two algorithms for performing search in an array. Earlier, we determined that the running time for sequential search on an array where the search value is equally likely to appear in any location is in both the average and worst cases. We would like to compare this running time to that required to perform a binary search on an array whose values are stored in order from lowest to highest. Here is a visualization of the binary search method.

3.8.1.2. Binary Search Practice Exercise

3.8.1.3. Analyzing Binary Search

Function binarySearch is designed to find the (single) occurrence of and return its position. A special value is returned if does not appear in the array. This algorithm can be modified to implement variations such as returning the position of the first occurrence of in the array if multiple occurrences are allowed, and returning the position of the greatest value less than when is not in the array.

Comparing sequential search to binary search, we see that as grows, the running time for sequential search in the average and worst cases quickly becomes much greater than the running time for binary search. Taken in isolation, binary search appears to be much more efficient than sequential search. This is despite the fact that the constant factor for binary search is greater than that for sequential search, because the calculation for the next search position in binary search is more expensive than just incrementing the current position, as sequential search does.

Note however that the running time for sequential search will be roughly the same regardless of whether or not the array values are stored in order. In contrast, binary search requires that the array values be ordered from lowest to highest. Depending on the context in which binary search is to be used, this requirement for a sorted array could be detrimental to the running time of a complete program, because maintaining the values in sorted order requires a greater cost when inserting new elements into the array. This is an example of a tradeoff between the advantage of binary search during search and the disadvantage related to maintaining a sorted array. Only in the context of the complete problem to be solved can we know whether the advantage outweighs the disadvantage.

3.9. Analyzing Problems

3.9.1. Analyzing Problems

You most often use the techniques of “algorithm” analysis to analyze an algorithm, or the instantiation of an algorithm as a program. You can also use these same techniques to analyze the cost of a problem. The key question that we want to ask is: How hard is a problem? Certainly we should expect that in some sense, the problem of sorting a list of records is harder than the problem of searching a list of records for a given key value. Certainly the algorithms that we know for sorting some records seem to be more expensive than the algorithms that we know for searching those same records.

What we need are useful definitions for the upper bound and lower bound of a problem.

One might start by thinking that the upper bound for a problem is how hard any algorithm can be for the problem. But we can make algorithms as bad as we want, so that is not useful. Instead, what is useful is to say that a problem is only as hard as what we CAN do. In other words, we should define the upper bound for a problem to be the best algorithm that we know for the problem. Of course, whenever we talk about bounds, we have to say when they apply. We we really should say something like the best algorithm that we know in the worst case, or the best algorithm that we know in the average case.

But what does it mean to give a lower bound for a problem? Lower bound refers to the minimum that any algorithm MUST cost. For example, when searching an unsorted list, we MUST look at every record. When sorting a list, we MUST look at every record (to even know if it is sorted).

It is much easier to show that an algorithm (or program) is in than it is to show that a problem is in . For a problem to be in means that every algorithm that solves the problem is in , even algorithms that we have not thought of! In other words, EVERY algorithm MUST have at least this cost. So, to prove a lower bound, we need an argument that is true, even for algorithms that we don’t know.

So far all of our examples of algorithm analysis give “obvious” results, with big-Oh always matching . To understand how big-Oh, , and notations are properly used to describe our understanding of a problem or an algorithm, it is best to consider an example where you do not already know a lot about the problem.

Let us look ahead to analyzing the problem of sorting to see how this process works. What is the least possible cost for any sorting algorithm in the worst case? The algorithm must at least look at every element in the input, just to determine that the input is truly sorted. Thus, any sorting algorithm must take at least time. For many problems, this observation that each of the inputs must be looked at leads to an easy lower bound.

In your previous study of computer science, you have probably seen an example of a sorting algorithm whose running time is in in the worst case. The simple Bubble Sort and Insertion Sort algorithms typically given as examples in a first year programming course have worst case running times in . Thus, the problem of sorting can be said to have an upper bound in . How do we close the gap between and ? Can there be a better sorting algorithm? If you can think of no algorithm whose worst-case growth rate is better than , and if you have discovered no analysis technique to show that the least cost for the problem of sorting in the worst case is greater than , then you cannot know for sure whether or not there is a better algorithm.

Many good sorting algorithms have running time that is in in the worst case. This greatly narrows the gap. With this new knowledge, we now have a lower bound in and an upper bound in . Should we search for a faster algorithm? Many have tried, without success. Fortunately (or perhaps unfortunately?), we can prove that any sorting algorithm must have running time in in the worst case. [^1] This proof is one of the most important results in the field of algorithm analysis, and it means that no sorting algorithm can possibly run faster than for the worst-case input of size . Thus, we can conclude that the problem of sorting is in the worst case, because the upper and lower bounds have met.

Knowing the lower bound for a problem does not give you a good algorithm. But it does help you to know when to stop looking. If the lower bound for the problem matches the upper bound for the algorithm (within a constant factor), then we know that we can find an algorithm that is better only by a constant factor.

So, to summarize: The upper bound for a problem is the best that you CAN do, while the lower bound for a problem is the least work that you MUST do. If those two are the same, then we say that we really understand our problem.

[^1]: While it is fortunate to know the truth, it is unfortunate that sorting is rather than .

3.10. Common Misunderstandings

3.10.1. Common Misunderstandings

Asymptotic analysis is one of the most intellectually difficult topics that undergraduate computer science majors are confronted with. Most people find growth rates and asymptotic analysis confusing and so develop misconceptions about either the concepts or the terminology. It helps to know what the standard points of confusion are, in hopes of avoiding them.

One problem with differentiating the concepts of upper and lower bounds is that, for most algorithms that you will encounter, it is easy to recognize the true growth rate for that algorithm. Given complete knowledge about a cost function, the upper and lower bound for that cost function are always the same. Thus, the distinction between an upper and a lower bound is only worthwhile when you have incomplete knowledge about the thing being measured. If this distinction is still not clear, then you should read about analyzing problems. We use -notation to indicate that there is no meaningful difference between what we know about the growth rates of the upper and lower bound (which is usually the case for simple algorithms).

It is a common mistake to confuse the concepts of upper bound or lower bound on the one hand, and worst case or best case on the other. The best, worst, or average cases each define a cost for a specific input instance (or specific set of instances for the average case). In contrast, upper and lower bounds describe our understanding of the growth rate for that cost measure. So to define the growth rate for an algorithm or problem, we need to determine what we are measuring (the best, worst, or average case) and also our description for what we know about the growth rate of that cost measure (big-Oh, , or ).

The upper bound for an algorithm is not the same as the worst case for that algorithm for a given input of size . What is being bounded is not the actual cost (which you can determine for a given value of ), but rather the growth rate for the cost. There cannot be a growth rate for a single point, such as a particular value of . The growth rate applies to the change in cost as a change in input size occurs. Likewise, the lower bound is not the same as the best case for a given size .

Another common misconception is thinking that the best case for an algorithm occurs when the input size is as small as possible, or that the worst case occurs when the input size is as large as possible. What is correct is that best- and worse-case instances exist for each possible size of input. That is, for all inputs of a given size, say , one (or more) of the inputs of size is the best and one (or more) of the inputs of size is the worst. Often (but not always!), we can characterize the best input case for an arbitrary size, and we can characterize the worst input case for an arbitrary size. Ideally, we can determine the growth rate for the characterized best, worst, and average cases as the input size grows.

Example 3.10.1

What is the growth rate of the best case for sequential search? For any array of size , the best case occurs when the value we are looking for appears in the first position of the array. This is true regardless of the size of the array. Thus, the best case (for arbitrary size ) occurs when the desired value is in the first of positions, and its cost is 1. It is not correct to say that the best case occurs when .

3.11. Amortized Analysis

This module presents the concept of amortized analysis, which is the analysis for a series of operations taken as a whole. In particular, amortized analysis allows us to deal with the situation where the worst-case cost for operations is less than times the worst-case cost of any one operation. Rather than focusing on the individual cost of each operation independently and summing them, amortized analysis looks at the cost of the entire series and “charges” each individual operation with a share of the total cost.

We can apply the technique of amortized analysis in the case of a series of sequential searches in an unsorted array. For random searches, the average-case cost for each search is , and so the expected total cost for the series is . Unfortunately, in the worst case all of the searches would be to the last item in the array. In this case, each search costs for a total worst-case cost of . Compare this to the cost for a series of searches such that each item in the array is searched for precisely once. In this situation, some of the searches must be expensive, but also some searches must be cheap. The total number of searches, in the best, average, and worst case, for this problem must be . This is a factor of two better than the more pessimistic analysis that charges each operation in the series with its worst-case cost.

As another example of amortized analysis, consider the process of incrementing a binary counter. The algorithm is to move from the lower-order (rightmost) bit toward the high-order (leftmost) bit, changing 1s to 0s until the first 0 is encountered. This 0 is changed to a 1, and the increment operation is done. Below is an implementation for the increment operation, assuming that a binary number of length is stored in array A of length .

for (i=0; ((i<A.length) && (A[i] == 1)); i++)
  A[i] = 0;
if (i < A.length)
  A[i] = 1;

If we count from 0 through , (requiring a counter with at least bits), what is the average cost for an increment operation in terms of the number of bits processed? Naive worst-case analysis says that if all bits are 1 (except for the high-order bit), then bits need to be processed. Thus, if there are increments, then the cost is . However, this is much too high, because it is rare for so many bits to be processed. In fact, half of the time the low-order bit is 0, and so only that bit is processed. One quarter of the time, the low-order two bits are 01, and so only the low-order two bits are processed. Another way to view this is that the low-order bit is always flipped, the bit to its left is flipped half the time, the next bit one quarter of the time, and so on. We can capture this with the summation (charging costs to bits going from right to left)

In other words, the average number of bits flipped on each increment is 2, leading to a total cost of only for a series of increments.

A useful concept for amortized analysis is illustrated by a simple variation on the stack data structure, where the pop function is slightly modified to take a second parameter indicating that pop operations are to be performed.

The “local” worst-case analysis for multipop is for elements in the stack. Thus, if there are calls to push and calls to multipop, then the naive worst-case cost for the series of operation is . This analysis is unreasonably pessimistic. Clearly it is not really possible to pop elements each time multipop is called. Analysis that focuses on single operations cannot deal with this global limit, and so we turn to amortized analysis to model the entire series of operations.

The key to an amortized analysis of this problem lies in the concept of potential. At any given time, a certain number of items may be on the stack. The cost for multipop can be no more than this number of items. Each call to push places another item on the stack, which can be removed by only a single multipop operation. Thus, each call to push raises the potential of the stack by one item. The sum of costs for all calls to multipop can never be more than the total potential of the stack (aside from a constant time cost associated with each call to multipop itself).

The amortized cost for any series of push and multipop operations is the sum of three costs. First, each of the push operations takes constant time. Second, each multipop operation takes a constant time in overhead, regardless of the number of items popped on that call. Finally, we count the sum of the potentials expended by all multipop operations, which is at most , the number of push operations. This total cost can therefore be expressed as

A similar argument was used in our analysis for the partition function in the Quicksort algorithm. While on any given pass through the while loop the left or right pointers might move all the way through the remainder of the partition, doing so would reduce the number of times that the while loop can be further executed.

Our final example uses amortized analysis to prove a relationship between the cost of the move-to-front self-organizing list heuristic and the cost for the optimal static ordering of the list.

Recall that, for a series of search operations, the minimum cost for a static list results when the list is sorted by frequency of access to its records. This is the optimal ordering for the records if we never allow the positions of records to change, because the most-frequently accessed record is first (and thus has least cost), followed by the next most frequently accessed record, and so on.

Theorem 3.11.1

Theorem: The total number of comparisons required by any series of or more searches on a self-organizing list of length using the move-to-front heuristic is never more than twice the total number of comparisons required when series is applied to the list stored in its optimal static order.

Proof: Each comparison of the search key with a record in the list is either successful or unsuccessful. For searches, there must be exactly successful comparisons for both the self-organizing list and the static list. The total number of unsuccessful comparisons in the self-organizing list is the sum, over all pairs of distinct keys, of the number of unsuccessful comparisons made between that pair.

Consider a particular pair of keys: and . For any sequence of searches , the total number of (unsuccessful) comparisons between and is identical to the number of comparisons between and required for the subsequence of made up only of searches for or . Call this subsequence . In other words, including searches for other keys does not affect the relative position of and and so does not affect the relative contribution to the total cost of the unsuccessful comparisons between and .

The number of unsuccessful comparisons between and made by the move-to-front heuristic on subsequence is at most twice the number of unsuccessful comparisons between and required when is applied to the optimal static ordering for the list. To see this, assume that contains s and s, with . Under the optimal static ordering, unsuccessful comparisons are required because must appear before in the list (because its access frequency is higher). Move-to-front will yield an unsuccessful comparison whenever the request sequence changes from to or from to . The total number of such changes possible is because each change involves an and each can be part of at most two changes.

Because the total number of unsuccessful comparisons required by move-to-front for any given pair of keys is at most twice that required by the optimal static ordering, the total number of unsuccessful comparisons required by move-to-front for all pairs of keys is also at most twice as high. Because the number of successful comparisons is the same for both methods, the total number of comparisons required by move-to-front is less than twice the number of comparisons required by the optimal static ordering.

3.12. Multiple Parameters

Sometimes the proper analysis for an algorithm requires multiple parameters to describe the cost. To illustrate the concept, consider an algorithm to compute the rank ordering for counts of all pixel values in a picture. Pictures are often represented by a two-dimensional array, and a pixel is one cell in the array. The value of a pixel is either the code value for the color, or a value for the intensity of the picture at that pixel. Assume that each pixel can take any integer value in the range 0 to . The problem is to find the number of pixels of each color value and then sort the color values with respect to the number of times each value appears in the picture. Assume that the picture is a rectangle with pixels. A pseudocode algorithm to solve the problem follows.

for (i=0; i<C; i++)   // Initialize count
   count[i] = 0;
for (i=0; i<P; i++)   // Look at all of the pixels
   count[value(i)]++; // Increment a pixel value count
sort(count);          // Sort pixel value counts

In this example, count is an array of size C that stores the number of pixels for each color value. Function value(i) returns the color value for pixel .

The time for the first for loop (which initializes count) is based on the number of colors, . The time for the second loop (which determines the number of pixels with each color) is . The time for the final line, the call to sort, depends on the cost of the sorting algorithm used. We will assume that the sorting algorithm has cost if items are sorted, thus yielding as the total algorithm cost.

Is this a good representation for the cost of this algorithm? What is actually being sorted? It is not the pixels, but rather the colors. What if is much smaller than ? Then the estimate of is pessimistic, because much fewer than items are being sorted. Instead, we should use as our analysis variable for steps that look at each pixel, and as our analysis variable for steps that look at colors. Then we get for the initialization loop, for the pixel count loop, and for the sorting operation. This yields a total cost of .

Why can we not simply use the value of for input size and say that the cost of the algorithm is ? Because, is typically much less than . For example, a picture might have 1000 1000 pixels and a range of 256 possible colors. So, is one million, which is much larger than . But, if is smaller, or larger (even if it is still less than ), then can become the larger quantity. Thus, neither variable should be ignored.

3.13. Space Bounds

Besides time, space is the other computing resource that is commonly of concern to programmers. Just as computers have become much faster over the years, they have also received greater allotments of memory. Even so, the amount of available disk space or main memory can be significant constraints for algorithm designers.

The analysis techniques used to measure space requirements are similar to those used to measure time requirements. However, while time requirements are normally measured for an algorithm that manipulates a particular data structure, space requirements are normally determined for the data structure itself. The concepts of asymptotic analysis for growth rates on input size apply completely to measuring space requirements.

Example 3.13.1

What are the space requirements for an array of integers? If each integer requires bytes, then the array requires bytes, which is .

Example 3.13.2

Imagine that we want to keep track of friendships between people. We can do this with an array of size . Each row of the array represents the friends of an individual, with the columns indicating who has that individual as a friend. For example, if person is a friend of person , then we place a mark in column of row in the array. Likewise, we should also place a mark in column of row if we assume that friendship works both ways. For people, the total size of the array is .

A data structure’s primary purpose is to store data in a way that allows efficient access to those data. To provide efficient access, it may be necessary to store additional information about where the data are within the data structure. For example, each node of a linked list must store a pointer to the next value on the list. All such information stored in addition to the actual data values is referred to as overhead. Ideally, overhead should be kept to a minimum while allowing maximum access. The need to maintain a balance between these opposing goals is what makes the study of data structures so interesting.

One important aspect of algorithm design is referred to as the space/time tradeoff principle. The space/time tradeoff principle says that one can often achieve a reduction in time if one is willing to sacrifice space or vice versa. Many programs can be modified to reduce storage requirements by “packing” or encoding information. “Unpacking” or decoding the information requires additional time. Thus, the resulting program uses less space but runs slower. Conversely, many programs can be modified to pre-store results or reorganize information to allow faster running time at the expense of greater storage requirements. Typically, such changes in time and space are both by a constant factor.

A classic example of a space/time tradeoff is the lookup table. A lookup table pre-stores the value of a function that would otherwise be computed each time it is needed. For example, 12! is the greatest value for the factorial function that can be stored in a 32-bit int variable. If you are writing a program that often computes factorials, it is likely to be much more time efficient to simply pre-compute and store the 12 values in a table. Whenever the program needs the value of it can simply check the lookup table. (If , the value is too large to store as an int variable anyway.) Compared to the time required to compute factorials, it may be well worth the small amount of additional space needed to store the lookup table.

Lookup tables can also store approximations for an expensive function such as sine or cosine. If you compute this function only for exact degrees or are willing to approximate the answer with the value for the nearest degree, then a lookup table storing the computation for exact degrees can be used instead of repeatedly computing the sine function. Note that initially building the lookup table requires a certain amount of time. Your application must use the lookup table often enough to make this initialization worthwhile.

Another example of the space/time tradeoff is typical of what a programmer might encounter when trying to optimize space. Here is a simple code fragment for sorting an array of integers. We assume that this is a special case where there are integers whose values are a permutation of the integers from 0 to . This is an example of a binsort. Binsort assigns each value to an array position corresponding to its value.

for (i=0; i<A.length; i++)
  B[A[i]] = A[i];

This is efficient and requires time. However, it also requires two arrays of size . Next is a code fragment that places the permutation in order but does so within the same array (thus it is an example of an “in place” sort).

for (i=0; i<A.length; i++)
  while (A[i] != i) // Swap element A[i] with A[A[i]]
    swap(A, i, A[i]);

Function swap(A, i, j) exchanges elements i and j in array A. It may not be obvious that the second code fragment actually sorts the array. To see that this does work, notice that each pass through the for loop will at least move the integer with value to its correct position in the array, and that during this iteration, the value of A[i] must be greater than or equal to . A total of at most swap operations take place, because an integer cannot be moved out of its correct position once it has been placed there, and each swap operation places at least one integer in its correct position. Thus, this code fragment has cost . However, it requires more time to run than the first code fragment. On my computer the second version takes nearly twice as long to run as the first, but it only requires half the space.

A second principle for the relationship between a program’s space and time requirements applies to programs that process information stored on disk. Strangely enough, the disk-based space/time tradeoff principle is almost the reverse of the space/time tradeoff principle for programs using main memory.

The disk-based space/time tradeoff principle states that the smaller you can make your disk storage requirements, the faster your program will run. This is because the time to read information from disk is enormous compared to computation time, so almost any amount of additional computation needed to unpack the data is going to be less than the disk-reading time saved by reducing the storage requirements. Naturally this principle does not hold true in all cases, but it is good to keep in mind when designing programs that process information stored on disk.

3.14. Code Tuning and Empirical Analysis

3.14.1. Code Tuning and Empirical Analysis

In practice, there is not such a big difference in running time between an algorithm with growth rate and another with growth rate . There is, however, an enormous difference in running time between algorithms with growth rates of and . As you shall see during the course of your study of common data structures and algorithms, there are many problems whose obvious solution requires time, but that also have a solution requiring time. Examples include sorting and searching, two of the most important computer problems.

While not nearly so important as changing an algorithm to reduce its growth rate, “code tuning” can also lead to dramatic improvements in running time. Code tuning is the art of hand-optimizing a program to run faster or require less storage. For many programs, code tuning can reduce running time or cut the storage requirements by a factor of two or more. Even speedups by a factor of five to ten are not uncommon. Occasionally, you can get an even bigger speedup by converting from a symbolic representation of the data to a numeric coding scheme on which you can do direct computation.

Here are some suggestions for ways to speed up your programs by code tuning. The most important thing to realize is that most statements in a program do not have much effect on the running time of that program. There are normally just a few key subroutines, possibly even key lines of code within the key subroutines, that account for most of the running time. There is little point to cutting in half the running time of a subroutine that accounts for only 1% of the total running time. Focus your attention on those parts of the program that have the most impact.

When tuning code, it is important to gather good timing statistics. Many compilers and operating systems include profilers and other special tools to help gather information on both time and space use. These are invaluable when trying to make a program more efficient, because they can tell you where to invest your effort.

A lot of code tuning is based on the principle of avoiding work rather than speeding up work. A common situation occurs when we can test for a condition that lets us skip some work. However, such a test is never completely free. Care must be taken that the cost of the test does not exceed the amount of work saved. While one test might be cheaper than the work potentially saved, the test must always be made and the work can be avoided only some fraction of the time.

Example 3.14.1

A common operation in computer graphics applications is to find which among a set of complex objects contains a given point in space. Many useful data structures and algorithms have been developed to deal with variations of this problem. Most such implementations involve the following tuning step. Directly testing whether a given complex object contains the point in question is relatively expensive. Instead, we can screen for whether the point is contained within a bounding box for the object. The bounding box is simply the smallest rectangle (usually defined to have sides perpendicular to the and axes) that contains the object. If the point is not in the bounding box, then it cannot be in the object. If the point is in the bounding box, only then would we conduct the full comparison of the object versus the point. Note that if the point is outside the bounding box, we saved time because the bounding box test is cheaper than the comparison of the full object versus the point. But if the point is inside the bounding box, then that test is redundant because we still have to compare the point against the object. Typically the amount of work avoided by making this test is greater than the cost of making the test on every object.

Be careful not to use tricks that make the program unreadable. Most code tuning is simply cleaning up a carelessly written program, not taking a clear program and adding tricks. In particular, you should develop an appreciation for the capabilities of modern compilers to make extremely good optimizations of expressions. “Optimization of expressions” here means a rearrangement of arithmetic or logical expressions to run more efficiently. Be careful not to damage the compiler’s ability to do such optimizations for you in an effort to optimize the expression yourself. Always check that your “optimizations” really do improve the program by running the program before and after the change on a suitable benchmark set of input. Many times I have been wrong about the positive effects of code tuning in my own programs. Most often I am wrong when I try to optimize an expression. It is hard to do better than the compiler.

The greatest time and space improvements come from a better data structure or algorithm. The most important rule of code tuning is:

First tune the algorithm, then tune the code.

3.14.1.1. Empirical Analysis

Asymptotic algorithm analysis is an analytic tool, whereby we model the key aspects of an algorithm to determine the growth rate of the algorithm as the input size grows. It has proved hugely practical, guiding developers to use more efficient algorithms. But it is really an estimation technique, and it has its limitations. These include the effects at small problem size, determining the finer distinctions between algorithms with the same growth rate, and the inherent difficulty of doing mathematical modeling for more complex problems.

An alternative to analytical approaches are empirical ones. The most obvious empirical approach is simply to run two competitors and see which performs better. In this way we might overcome the deficiencies of analytical approaches.

Be warned that comparative timing of programs is a difficult business, often subject to experimental errors arising from uncontrolled factors (system load, the language or compiler used, etc.). The most important concern is that you might be biased in favor of one of the programs. If you are biased, this is certain to be reflected in the timings. One look at competing software or hardware vendors’ advertisements should convince you of this. The most common pitfall when writing two programs to compare their performance is that one receives more code-tuning effort than the other, since code tuning can often reduce running time by a factor of five to ten. If the running times for two programs differ by a constant factor regardless of input size (i.e., their growth rates are the same), then differences in code tuning might account for any difference in running time. Be suspicious of empirical comparisons in this situation.

Another approach to analytical analysis is simulation. The idea of simulation is to model the problem with a computer program and then run it to get a result. In the context of algorithm analysis, simulation is distinct from empirical comparison of two competitors because the purpose of the simulation is to perform analysis that might otherwise be too difficult. A good example of this appears in the following figure.

This figure shows the cost for inserting or deleting a record from a hash table under two different assumptions for the policy used to find a free slot in the table. The axes is the cost in number of hash table slots evaluated, and the axes is the percentage of slots in the table that are full. The mathematical equations for these curves can be determined, but this is not so easy. A reasonable alternative is to write simple variations on hashing. By timing the cost of the program for various loading conditions, it is not difficult to construct a plot similar to this one. The purpose of this analysis was not to determine which approach to hashing is most efficient, so we are not doing empirical comparison of hashing alternatives. Instead, the purpose was to analyze the proper loading factor that would be used in an efficient hashing system to balance time cost versus hash table size (space cost).

3.15. Algorithm Analysis Summary Exercises

3.16. Algorithm Analysis Summary Exercises

Chapter 4: Linear Structures

4.1. Abstract Data Types

4.1.1. Abstract Data Types

This module presents terminology and definitions related to techniques for managing the tremendous complexity of computer programs. It also presents working definitions for the fundamental but somewhat slippery terms “data item“ and “data structure“. We begin with the basic elements on which data structures are built.

A type is a collection of values. For example, the Boolean type consists of the values true and false. The integers also form a type. An integer is a simple type because its values contain no subparts. A bank account record will typically contain several pieces of information such as name, address, account number, and account balance. Such a record is an example of an aggregate type or composite type. A data item is a piece of information or a record whose value is drawn from a type. A data item is said to be a member of a type.

A data type is a type together with a collection of operations to manipulate the type. For example, an integer variable is a member of the integer data type. Addition is an example of an operation on the integer data type.

A distinction should be made between the logical concept of a data type and its physical implementation in a computer program. For example, there are two traditional implementations for the list data type: the linked list and the array-based list. The list data type can therefore be implemented using a linked list or an array. But we don’t need to know how the list is implemented when we wish to use a list to help in a more complex design. For example, a list might be used to help implement a graph data structure.

As another example, the term “array” could refer either to a data type or an implementation. “Array” is commonly used in computer programming to mean a contiguous block of memory locations, where each memory location stores one fixed-length data item. By this meaning, an array is a physical data structure. However, array can also mean a logical data type composed of a (typically homogeneous) collection of data items, with each data item identified by an index number. It is possible to implement arrays in many different ways besides as a block of contiguous memory locations. The sparse matrix refers to a large, two-dimensional array that stores only a relatively few non-zero values. This is often implemented with a linked structure, or possibly using a hash table. But it could be implemented with an interface that uses traditional row and column indices, thus appearing to the user in the same way that it would if it had been implemented as a block of contiguous memory locations.

An abstract data type (ADT) is the specification of a data type within some language, independent of an implementation. The interface for the ADT is defined in terms of a type and a set of operations on that type. The behavior of each operation is determined by its inputs and outputs. An ADT does not specify how the data type is implemented. These implementation details are hidden from the user of the ADT and protected from outside access, a concept referred to as encapsulation.

A data structure is the implementation for an ADT. In an object-oriented language, an ADT and its implementation together make up a class. Each operation associated with the ADT is implemented by a member function or method. The variables that define the space required by a data item are referred to as data members. An object is an instance of a class, that is, something that is created and takes up storage during the execution of a computer program.

The term data structure often refers to data stored in a computer’s main memory. The related term file structure often refers to the organization of data on peripheral storage, such as a disk drive or CD.

Example 4.1.1

The mathematical concept of an integer, along with operations that manipulate integers, form a data type. The int variable type is a physical representation of the abstract integer. The int variable type, along with the operations that act on an int variable, form an ADT. Unfortunately, the int implementation is not completely true to the abstract integer, as there are limitations on the range of values an int variable can store. If these limitations prove unacceptable, then some other representation for the ADT “integer” must be devised, and a new implementation must be used for the associated operations.

Example 4.1.2

An ADT for a list of integers might specify the following operations:

1. Insert a new integer at a particular position in the list.
2. Return True if the list is empty.
3. Reinitialize the list.
4. Return the number of integers currently in the list.
5. Retrieve the integer at a particular position in the list.
6. Delete the integer at a particular position in the list.

From this description, the input and output of each operation should be clear, but the implementation for lists has not been specified.

One application that makes use of some ADT might use particular member functions of that ADT more than a second application, or the two applications might have different time requirements for the various operations. These differences in the requirements of applications are the reason why a given ADT might be supported by more than one implementation.

Example 4.1.3

Two popular implementations for large disk-based database applications are hashing and the B-tree. Both support efficient insertion and deletion of records, and both support exact-match queries. However, hashing is more efficient than the B-tree for exact-match queries. On the other hand, the B-tree can perform range queries efficiently, while hashing is hopelessly inefficient for range queries. Thus, if the database application limits searches to exact-match queries, hashing is preferred. On the other hand, if the application requires support for range queries, the B-tree is preferred. Despite these performance issues, both implementations solve versions of the same problem: updating and searching a large collection of records.

The concept of an ADT can help us to focus on key issues even in non-computing applications.

Example 4.1.4

When operating a car, the primary activities are steering, accelerating, and braking. On nearly all passenger cars, you steer by turning the steering wheel, accelerate by pushing the gas pedal, and brake by pushing the brake pedal. This design for cars can be viewed as an ADT with operations “steer”, “accelerate”, and “brake”. Two cars might implement these operations in radically different ways, say with different types of engine, or front- versus rear-wheel drive. Yet, most drivers can operate many different cars because the ADT presents a uniform method of operation that does not require the driver to understand the specifics of any particular engine or drive design. These differences are deliberately hidden.

The concept of an ADT is one instance of an important principle that must be understood by any successful computer scientist: managing complexity through abstraction. A central theme of computer science is complexity and techniques for handling it. Humans deal with complexity by assigning a label to an assembly of objects or concepts and then manipulating the label in place of the assembly. Cognitive psychologists call such a label a metaphor. A particular label might be related to other pieces of information or other labels. This collection can in turn be given a label, forming a hierarchy of concepts and labels. This hierarchy of labels allows us to focus on important issues while ignoring unnecessary details.

Example 4.1.5

We apply the label “hard drive” to a collection of hardware that manipulates data on a particular type of storage device, and we apply the label “CPU” to the hardware that controls execution of computer instructions. These and other labels are gathered together under the label “computer”. Because even the smallest home computers today have millions of components, some form of abstraction is necessary to comprehend how a computer operates.

Consider how you might go about the process of designing a complex computer program that implements and manipulates an ADT. The ADT is implemented in one part of the program by a particular data structure. While designing those parts of the program that use the ADT, you can think in terms of operations on the data type without concern for the data structure’s implementation. Without this ability to simplify your thinking about a complex program, you would have no hope of understanding or implementing it.

Example 4.1.6

Consider the design for a relatively simple database system stored on disk. Typically, records on disk in such a program are accessed through a buffer pool rather than directly. Variable length records might use a memory manager to find an appropriate location within the disk file to place the record. Multiple index structures will typically be used to support access to a collection of records using multiple search keys. Thus, we have a chain of classes, each with its own responsibilities and access privileges. A database query from a user is implemented by searching an index structure. This index requests access to the record by means of a request to the buffer pool. If a record is being inserted or deleted, such a request goes through the memory manager, which in turn interacts with the buffer pool to gain access to the disk file. A program such as this is far too complex for nearly any human programmer to keep all of the details in their head at once. The only way to design and implement such a program is through proper use of abstraction and metaphors. In object-oriented programming, such abstraction is handled using classes.

Data types have both a logical form and a physical form. The definition of the data type in terms of an ADT is its logical form. The implementation of the data type as a data structure is its physical form. Sometimes you might see the term concrete implementation, but the word concrete is redundant. The figure below illustrates this relationship between logical and physical forms for data types. When you implement an ADT, you are dealing with the physical form of the associated data type. When you use an ADT elsewhere in your program, you are concerned with the associated data type’s logical form. Some sections of this book focus on physical implementations for a given data structure. Other sections use the logical ADT for the data structure in the context of a higher-level task.

Figure 4.1.1: The relationship between data items, abstract data types, and data structures.

The ADT defines the logical form of the data type. The data structure implements the physical form of the data type. Users of an ADT are typically programmers working in the same language as the implementer of the ADT. Typically, these programmers want to use the ADT as a component in another application. The interface to an ADT is also commonly referred to as the Application Programmer Interface, or API, for the ADT. The interface becomes a form of communication between the two programmers.

Example 4.1.7

A particular programming environment might provide a library that includes a list class. The logical form of the list is defined by the public functions, their inputs, and their outputs that define the class. This might be all that you know about the list class implementation, and this should be all you need to know. Within the class, a variety of physical implementations for lists is possible.

4.2. Chapter Introduction: Lists

If your program needs to store a few things—numbers, payroll records, or job descriptions for example—the simplest and most effective approach might be to put them in a list. Only when you have to organize and search through a large number of things do more sophisticated data structures like search trees become necessary. Many applications don’t require any form of search, and they do not require that an ordering be placed on the objects being stored. Some applications require that actions be performed in a strict chronological order, processing objects in the order that they arrived, or perhaps processing objects in the reverse of the order that they arrived. For all these situations, a simple list structure is appropriate.

This chapter describes representations both for lists and for two important list-like structures called the stack and the queue. Along with presenting these fundamental data structures, the other goals of the chapter are to:

1. Give examples that show the separation of a logical representation in the form of an ADT from a physical implementation as a data structure.
2. Illustrate the use of asymptotic analysis in the context of simple operations that you might already be familiar with. In this way you can begin to see how asymptotic analysis works, without the complications that arise when analyzing more sophisticated algorithms and data structures.

We begin by defining an ADT for lists. Two implementations for the list ADT—the array-based list and the linked list—are covered in detail and their relative merits discussed. The chapter finishes with implementations for stacks and queues.

4.3. The List ADT

4.3.1. The List ADT

We all have an intuitive understanding of what we mean by a “list”. We want to turn this intuitive understanding into a concrete data structure with implementations for its operations. The most important concept related to lists is that of position. In other words, we perceive that there is a first element in the list, a second element, and so on. So, define a list to be a finite, ordered sequence of data items known as elements. This is close to the mathematical concept of a sequence.

“Ordered” in this definition means that each element has a position in the list. So the term “ordered” in this context does not mean that the list elements are sorted by value. (Of course, we can always choose to sort the elements on the list if we want; it’s just that keeping the elements sorted is not an inherent property of being a list.)

Each list element must have some data type. In the simple list implementations discussed in this chapter, all elements of the list are usually assumed to have the same data type, although there is no conceptual objection to lists whose elements have differing data types if the application requires it. The operations defined as part of the list ADT do not depend on the elemental data type. For example, the list ADT can be used for lists of integers, lists of characters, lists of payroll records, even lists of lists.

A list is said to be empty when it contains no elements. The number of elements currently stored is called the length of the list. The beginning of the list is called the head, the end of the list is called the tail.

We need some notation to show the contents of a list, so we will use the same angle bracket notation that is normally used to represent sequences. To be consistent with standard array indexing, the first position on the list is denoted as 0. Thus, if there are elements in the list, they are given positions 0 through as . The subscript indicates an element’s position within the list. Using this notation, the empty list would appear as .

4.3.1.1. Defining the ADT

What basic operations do we want our lists to support? Our common intuition about lists tells us that a list should be able to grow and shrink in size as we insert and remove elements. We should be able to insert and remove elements from anywhere in the list. We should be able to gain access to any element’s value, either to read it or to change it. We must be able to create and clear (or reinitialize) lists. It is also convenient to access the next or previous element from the “current” one.

Now we can define the ADT for a list object in terms of a set of operations on that object. We will use an interface to formally define the list ADT. List defines the member functions that any list implementation inheriting from it must support, along with their parameters and return types.

True to the notion of an ADT, an interface does not specify how operations are implemented. Two complete implementations are presented later in later modules, both of which use the same list ADT to define their operations. But they are considerably different in approaches and in their space/time tradeoffs.

The code below presents our list ADT. Any implementation for a container class such as a list should be able to support different data types for the elements. One way to do this in Java is to store data values of type Object. Languages that support generics (Java) or templates (C++) give more control over the element types.

The comments given with each member function describe what it is intended to do. However, an explanation of the basic design should help make this clearer. Given that we wish to support the concept of a sequence, with access to any position in the list, the need for many of the member functions such as insert and moveToPos is clear. The key design decision embodied in this ADT is support for the concept of a current position. For example, member moveToStart sets the current position to be the first element on the list, while methods next and prev move the current position to the next and previous elements, respectively. The intention is that any implementation for this ADT support the concept of a current position. The current position is where any action such as insertion or deletion will take place. An alternative design is to factor out position as a separate position object, sometimes referred to as an iterator.

// List class ADT. Generalize the element type using Java Generics.
public interface List<E> { // List class ADT
  // Remove all contents from the list, so it is once again empty
  public void clear();

  // Insert "it" at the current location
  // The client must ensure that the list's capacity is not exceeded
  public boolean insert(E it);

  // Append "it" at the end of the list
  // The client must ensure that the list's capacity is not exceeded
  public boolean append(E it);

  // Remove and return the current element
  public E remove();

  // Set the current position to the start of the list
  public void moveToStart();

  // Set the current position to the end of the list
  public void moveToEnd();

  // Move the current position one step left, no change if already at beginning
  public void prev();

  // Move the current position one step right, no change if already at end
  public void next();

  // Return the number of elements in the list
  public int length();

  // Return the position of the current element
  public int currPos();

  // Set the current position to "pos"
  public boolean moveToPos(int pos);

  // Return true if current position is at end of the list
  public boolean isAtEnd();

  // Return the current element
  public E getValue();

  // Tell if the list is empty or not
  public boolean isEmpty();
}

The List member functions allow you to build a list with elements in any desired order, and to access any desired position in the list. You might notice that the clear method is a “convenience” method, since it could be implemented by means of the other member functions in the same asymptotic time.

A list can be iterated through as follows:

for (L.moveToStart(); !L.isAtEnd(); L.next()) {
  it = L.getValue();
  doSomething(it);
}

In this example, each element of the list in turn is stored in it, and passed to the doSomething function. The loop terminates when the current position reaches the end of the list.

The list class declaration presented here is just one of many possible interpretations for lists. Our list interface provides most of the operations that one naturally expects to perform on lists and serves to illustrate the issues relevant to implementing the list data structure. As an example of using the list ADT, here is a function to return true if there is an occurrence of a given integer in the list, and false otherwise. The find method needs no knowledge about the specific list implementation, just the list ADT.

// Return true if k is in list L, false otherwise
static boolean find(List<Integer> L, int k) {
  for (L.moveToStart(); !L.isAtEnd(); L.next())
    if (k == L.getValue()) return true; // Found k
  return false;                         // k not found
}

In languages that support it, this implementation for find could be rewritten as a generic or template with respect to the element type. While making it more flexible, even generic types still are limited in their ability to handle different data types stored on the list. In particular, for the find function generic types would only work when the description for the object being searched for (k in the function) is of the same type as the objects themselves. They also have to be comparable when using the == operator. A more realistic situation is that we are searching for a record that contains a key field whose value matches k. Similar functions to find and return a composite type based on a key value can be created using the list implementation, but to do so requires some agreement between the list ADT and the find function on the concept of a key, and on how keys may be compared.

There are two standard approaches to implementing lists, the array-based list, and the linked list.

4.4. Array-Based List Implementation

4.4.1. Array-Based List Implementation

Here is an implementation for the array-based list, named AList. AList inherits from the List ADT,and so must implement all of the member functions of List.

// Array-based list implementation
class AList<E> implements List<E> {
  private E listArray[];                  // Array holding list elements
  private static final int DEFAULT_SIZE = 10; // Default size
  private int maxSize;                    // Maximum size of list
  private int listSize;                   // Current # of list items
  private int curr;                       // Position of current element

  // Constructors
  // Create a new list object with maximum size "size"
  @SuppressWarnings("unchecked") // Generic array allocation
  AList(int size) {
    maxSize = size;
    listSize = curr = 0;
    listArray = (E[])new Object[size];         // Create listArray
  }
  // Create a list with the default capacity
  AList() { this(DEFAULT_SIZE); }          // Just call the other constructor

  public void clear()                     // Reinitialize the list
    { listSize = curr = 0; }              // Simply reinitialize values

  // Insert "it" at current position
  public boolean insert(E it) {
    if (listSize >= maxSize) return false;
    for (int i=listSize; i>curr; i--)  // Shift elements up
      listArray[i] = listArray[i-1];   //   to make room
    listArray[curr] = it;
    listSize++;                        // Increment list size
    return true;
  }

  // Append "it" to list
  public boolean append(E it) {
    if (listSize >= maxSize) return false;
    listArray[listSize++] = it;
    return true;
  }

  // Remove and return the current element
  public E remove() {
    if ((curr<0) || (curr>=listSize))  // No current element
      return null;
    E it = listArray[curr];            // Copy the element
    for(int i=curr; i<listSize-1; i++) // Shift them down
      listArray[i] = listArray[i+1];
    listSize--;                        // Decrement size
    return it;
  }

  public void moveToStart() { curr = 0; }       // Set to front
  public void moveToEnd() { curr = listSize; }  // Set at end
  public void prev() { if (curr != 0) curr--; } // Move left
  public void next() { if (curr < listSize) curr++; } // Move right
  public int length() { return listSize; }      // Return list size
  public int currPos() { return curr; }         // Return current position

  // Set current list position to "pos"
  public boolean moveToPos(int pos) {
    if ((pos < 0) || (pos > listSize)) return false;
    curr = pos;
    return true;
  }

  // Return true if current position is at end of the list
  public boolean isAtEnd() { return curr == listSize; }

  // Return the current element
  public E getValue() {
    if ((curr < 0) || (curr >= listSize)) // No current element
      return null;
    return listArray[curr];
  }

  public String toString() {
  StringBuffer out = new StringBuffer((listSize + 1) * 4);

  out.append("< ");
  for (int i = 0; i < curr; i++) {
    out.append(listArray[i]);
    out.append(" ");
  }
  out.append("| ");
  for (int i = curr; i < listSize; i++) {
    out.append(listArray[i]);
    out.append(" ");
  }
  out.append(">");
  return out.toString();
  }

  //Tell if the list is empty or not
  public boolean isEmpty() {
    return listSize == 0;
  }
}

4.4.1.1. Insert

Because the array-based list implementation is defined to store list elements in contiguous cells of the array, the insert, append, and remove methods must maintain this property.

4.4.1.2. Insert Practice Exericse

4.4.2. Append and Remove

Removing an element from the head of the list is similar to insert in that all remaining elements must shift toward the head by one position to fill in the gap. If we want to remove the element at position , then elements must shift toward the head, as shown in the following slideshow.

In the average case, insertion or removal each requires moving half of the elements, which is .

4.4.2.1. Remove Practice Exericise

Aside from insert and remove, the only other operations that might require more than constant time are the constructor and clear. The other methods for Class AList simply access the current list element or move the current position. They all require time.

4.4.3. Array-based List Practice Questions

4.5. Linked Lists

4.5.1. Linked Lists

In this module we present one of the two traditional implementations for lists, usually called a linked list. The linked list uses dynamic memory allocation, that is, it allocates memory for new list elements as needed. The following diagram illustrates the linked list concept. Here there are three nodes that are “linked” together. Each node has two boxes. The box on the right holds a link to the next node in the list. Notice that the rightmost node has a diagonal slash through its link box, signifying that there is no link coming out of this box.

Because a list node is a distinct object (as opposed to simply a cell in an array), it is good practice to make a separate list node class. (We can also re-use the list node class to implement linked implementations for the stack and queue data structures. Here is an implementation for list nodes, called the Link class. Objects in the Link class contain an element field to store the element value, and a next field to store a pointer to the next node on the list. The list built from such nodes is called a singly linked list, or a one-way list, because each list node has a single pointer to the next node on the list.

class Link<E> {         // Singly linked list node class
  private E e;          // Value for this node
  private Link<E> n;    // Point to next node in list

  // Constructors
  Link(E it, Link<E> inn) { e = it; n = inn; }
  Link(Link<E> inn) { e = null; n = inn; }

  E element() { return e; }                        // Return the value
  E setElement(E it) { return e = it; }            // Set element value
  Link<E> next() { return n; }                     // Return next link
  Link<E> setNext(Link<E> inn) { return n = inn; } // Set next link
}

The Link class is quite simple. There are two forms for its constructor, one with an initial element value and one without. Member functions allow the link user to get or set the element and link fields.

4.5.1.1. Why This Has Problems

There are a number of problems with the representation just described. First, there are lots of special cases to code for. For example, when the list is empty we have no element for head, tail, and curr to point to. Implementing special cases for insert and remove increases code complexity, making it harder to understand, and thus increases the chance of introducing bugs.

4.5.1.2. A Better Solution

Fortunately, there is a fairly easy way to deal with all of the special cases, as well as the problem with deleting the last node. Many special cases can be eliminated by implementing linked lists with an additional header node as the first node of the list. This header node is a link node like any other, but its value is ignored and it is not considered to be an actual element of the list. The header node saves coding effort because we no longer need to consider special cases for empty lists or when the current position is at one end of the list. The cost of this simplification is the space for the header node. However, there are space savings due to smaller code size, because statements to handle the special cases are omitted. We get rid of the remaining special cases related to being at the end of the list by adding a “trailer” node that also never stores a value.

The following diagram shows initial conditions for a linked list with header and trailer nodes.

Here is what a list with some elements looks like with the header and trailer nodes added.

Adding the trailer node also solves our problem with deleting the last node on the list, as we will see when we take a closer look at the remove method’s implementation.

4.5.1.3. Linked List Implementation

Here is the implementation for the linked list class, named LList.

// Linked list implementation
class LList<E> implements List<E> {
  private Link<E> head;      // Pointer to list header
  private Link<E> tail;      // Pointer to last element
  private Link<E> curr;      // Access to current element
  private int listSize;      // Size of list

  // Constructors
  LList(int size) { this(); }     // Constructor -- Ignore size
  LList() { clear(); }

  // Remove all elements
  public void clear() {
    curr = tail = new Link<E>(null); // Create trailer
    head = new Link<E>(tail);        // Create header
    listSize = 0;
  }

  // Insert "it" at current position
  public boolean insert(E it) {
    curr.setNext(new Link<E>(curr.element(), curr.next()));
    curr.setElement(it);
    if (tail == curr) tail = curr.next();  // New tail
    listSize++;
    return true;
  }

  // Append "it" to list
  public boolean append(E it) {
    tail.setNext(new Link<E>(null));
    tail.setElement(it);
    tail = tail.next();
    listSize++;
    return true;
  }

  // Remove and return current element
  public E remove () {
    if (curr == tail) return null;          // Nothing to remove
    E it = curr.element();                  // Remember value
    curr.setElement(curr.next().element()); // Pull forward the next element
    if (curr.next() == tail) tail = curr;   // Removed last, move tail
    curr.setNext(curr.next().next());       // Point around unneeded link
    listSize--;                             // Decrement element count
    return it;                              // Return value
  }

  public void moveToStart() { curr = head.next(); } // Set curr at list start
  public void moveToEnd() { curr = tail; }     // Set curr at list end

  // Move curr one step left; no change if now at front
  public void prev() {
    if (head.next() == curr) return; // No previous element
    Link<E> temp = head;
    // March down list until we find the previous element
    while (temp.next() != curr) temp = temp.next();
    curr = temp;
  }

  // Move curr one step right; no change if now at end
  public void next() { if (curr != tail) curr = curr.next(); }

  public int length() { return listSize; } // Return list length

  // Return the position of the current element
  public int currPos() {
    Link<E> temp = head.next();
    int i;
    for (i=0; curr != temp; i++)
      temp = temp.next();
    return i;
  }

  // Move down list to "pos" position
  public boolean moveToPos(int pos) {
    if ((pos < 0) || (pos > listSize)) return false;
    curr = head.next();
    for(int i=0; i<pos; i++) curr = curr.next();
    return true;
  }

  // Return true if current position is at end of the list
  public boolean isAtEnd() { return curr == tail; }

  // Return current element value. Note that null gets returned if curr is at the tail
  public E getValue() {
    return curr.element();
  }

  public String toString() {
  Link<E> temp = head.next();
  StringBuffer out = new StringBuffer((listSize + 1) * 4);

  out.append("< ");
  for (int i = 0; i < currPos(); i++) {
    out.append(temp.element());
    out.append(" ");
    temp = temp.next();
  }
  out.append("| ");
  for (int i = currPos(); i < listSize; i++) {
    out.append(temp.element());
    out.append(" ");
    temp = temp.next();
  }
  out.append(">");
  return out.toString();
  }

  //Tell if the list is empty or not
  public boolean isEmpty() {
    return listSize == 0;
  }
}

Here are some special cases for linked list insertion: Inserting at the end, and inserting to an empty list.

4.5.2. Linked List Remove

Implementations for the remaining operations each require time.

4.6. Comparison of List Implementations

4.6.1. Space Comparison

Now that you have seen two substantially different implementations for lists, it is natural to ask which is better. In particular, if you must implement a list for some task, which implementation should you choose?

Given a collection of elements to store, they take up some amount of space whether they are simple integers or large objects with many fields. Any container data structure like a list then requires some additional space to organize the elements being stored. This additional space is called overhead.

Array-based lists have the disadvantage that their size must be predetermined before the array can be allocated. Array-based lists cannot grow beyond their predetermined size. Whenever the list contains only a few elements, a substantial amount of space might be tied up in a largely empty array. This empty space is the overhead required by the array-based list. Linked lists have the advantage that they only need space for the objects actually on the list. There is no limit to the number of elements on a linked list, as long as there is free store memory available. The amount of space required by a linked list is , while the space required by the array-based list implementation is , but can be greater.

Array-based lists have the advantage that there is no wasted space for an individual element. Linked lists require that an extra pointer for the next field be added to every list node. So the linked list has these next pointers as overhead. If the element size is small, then the overhead for links can be a significant fraction of the total storage. When the array for the array-based list is completely filled, there is no wasted space, and so no overhead. The array-based list will then be more space efficient, by a constant factor, than the linked implementation.

A simple formula can be used to determine whether the array-based list or the linked list implementation will be more space efficient in a particular situation. Call the number of elements currently in the list, the size of a pointer in storage units (typically four bytes), the size of a data element in storage units (this could be anything, from one bit for a Boolean variable on up to thousands of bytes or more for complex records), and the maximum number of list elements that can be stored in the array. The amount of space required for the array-based list is , regardless of the number of elements actually stored in the list at any given time. The amount of space required for the linked list is . The smaller of these expressions for a given value determines the more space-efficient implementation for elements. In general, the linked implementation requires less space than the array-based implementation when relatively few elements are in the list. Conversely, the array-based implementation becomes more space efficient when the array is close to full. Using the equation, we can solve for to determine the break-even point beyond which the array-based implementation is more space efficient in any particular situation. This occurs when

If , then the break-even point is at . This would happen if the element field is either a four-byte int value or a pointer, and the next field is a typical four-byte pointer. That is, the array-based implementation would be more efficient (if the link field and the element field are the same size) whenever the array is more than half full.

As a rule of thumb, linked lists are more space efficient when implementing lists whose number of elements varies widely or is unknown. Array-based lists are generally more space efficient when the user knows in advance approximately how large the list will become, and can be confident that the list will never grow beyond a certain limit.

4.6.2. Time Comparison

Array-based lists are faster for access by position. Positions can easily be adjusted forwards or backwards by the next and prev methods. These operations always take time. In contrast, singly linked lists have no explicit access to the previous element, and access by position requires that we march down the list from the front (or the current position) to the specified position. Both of these operations require time in the average and worst cases, if we assume that each position on the list is equally likely to be accessed on any call to prev or moveToPos.

Given a pointer to a suitable location in the list, the insert and remove methods for linked lists require only time. Array-based lists must shift the remainder of the list up or down within the array. This requires time in the average and worst cases. For many applications, the time to insert and delete elements dominates all other operations. For this reason, linked lists are often preferred to array-based lists.

When implementing the array-based list, an implementor could allow the size of the array to grow and shrink depending on the number of elements that are actually stored. This data structure is known as a dynamic array. For example, both the Java and C++/STL Vector classes implement a dynamic array, and JavaScript arrays are always dynamic. Dynamic arrays allow the programmer to get around the limitation on the traditional array that its size cannot be changed once the array has been created. This also means that space need not be allocated to the dynamic array until it is to be used. The disadvantage of this approach is that it takes time to deal with space adjustments on the array. Each time the array grows in size, its contents must be copied. A good implementation of the dynamic array will grow and shrink the array in such a way as to keep the overall cost for a series of insert/delete operations relatively inexpensive, even though an occasional insert/delete operation might be expensive. A simple rule of thumb is to double the size of the array when it becomes full, and to cut the array size in half when it becomes one quarter full. To analyze the overall cost of dynamic array operations over time, we need to use a technique known as amortized analysis.

4.6.2.1. Practice Questions

4.7. Doubly Linked Lists

4.7.1. Doubly Linked Lists

The singly linked list allows for direct access from a list node only to the next node in the list. A doubly linked list allows convenient access from a list node to the next node and also to the preceding node on the list. The doubly linked list node accomplishes this in the obvious way by storing two pointers: one to the node following it (as in the singly linked list), and a second pointer to the node preceding it.

Figure 4.7.1: A doubly linked list.

The most common reason to use a doubly linked list is because it is easier to implement than a singly linked list. While the code for the doubly linked implementation is a little longer than for the singly linked version, it tends to be a bit more “obvious” in its intention, and so easier to implement and debug. Whether a list implementation is doubly or singly linked should be hidden from the List class user.

Like our singly linked list implementation, the doubly linked list implementation makes use of a header node. We also add a tailer node to the end of the list. The tailer is similar to the header, in that it is a node that contains no value, and it always exists. When the doubly linked list is initialized, the header and tailer nodes are created. Data member head points to the header node, and tail points to the tailer node. The purpose of these nodes is to simplify the insert, append, and remove methods by eliminating all need for special-case code when the list is empty, or when we insert at the head or tail of the list.

In our implementation, curr will point to the current position (or to the trailer node if the current position is at the end of the list).

Here is the complete implementation for a Link class to be used with doubly linked lists. This code is a little longer than that for the singly linked list node implementation since the doubly linked list nodes have an extra data member.

class Link<E> {         // Doubly linked list node
  private E e;          // Value for this node
  private Link<E> n;    // Pointer to next node in list
  private Link<E> p;    // Pointer to previous node

  // Constructors
  Link(E it, Link<E> inp, Link<E> inn) { e = it;  p = inp; n = inn; }
  Link(Link<E> inp, Link<E> inn) { p = inp; n = inn; }

  // Get and set methods for the data members
  public E element() { return e; }                                // Return the value
  public E setElement(E it) { return e = it; }                    // Set element value
  public Link<E> next() { return n; }                             // Return next link
  public Link<E> setNext(Link<E> nextval) { return n = nextval; } // Set next link
  public Link<E> prev() { return p; }                             // Return prev link
  public Link<E> setPrev(Link<E> prevval) { return p = prevval; } // Set prev link
}

4.7.1.1. Insert

The following slideshows illustrate the insert and append doubly linked list methods. The class declaration and the remaining member functions for the doubly linked list class are nearly identical to the singly linked list version. While the code for these methods might be a little longer than their singly linked list counterparts (since there is an extra pointer in each node to deal with), they tend to be easier to understand.

4.7.1.2. Append

4.7.1.3. Remove

4.7.1.4. Prev

The only disadvantage of the doubly linked list as compared to the singly linked list is the additional space used. The doubly linked list requires two pointers per node, and so in the implementation presented it requires twice as much overhead as the singly linked list.

4.7.1.5. Mangling Pointers

There is a space-saving technique that can be employed to eliminate the additional space requirement, though it will complicate the implementation and be somewhat slower. Thus, this is an example of a space/time tradeoff. It is based on observing that, if we store the sum of two values, then we can get either value back by subtracting the other. That is, if we store in variable , then and . Of course, to recover one of the values out of the stored summation, the other value must be supplied. A pointer to the first node in the list, along with the value of one of its two link fields, will allow access to all of the remaining nodes of the list in order. This is because the pointer to the node must be the same as the value of the following node’s prev pointer, as well as the previous node’s next pointer. It is possible to move down the list breaking apart the summed link fields as though you were opening a zipper.

The principle behind this technique is worth remembering, as it has many applications. The following code fragment will swap the contents of two variables without using a temporary variable (at the cost of three arithmetic operations).

a = a + b;
b = a - b; // Now b contains original value of a
a = a - b; // Now a contains original value of b

A similar effect can be had by using the exclusive-or operator. This fact is widely used in computer graphics. A region of the computer screen can be highlighted by XORing the outline of a box around it. XORing the box outline a second time restores the original contents of the screen.

4.8. List Element Implementations

4.8.1. List Element Implementations

When designing any container class, there are a number of design choices to be made regarding the data elements.

What to do if something can appear multiple times on a list? One option is to use a reference to elements. Another is to store separate copies. In general, the larger the elements and the more that they are duplicated, the more likely that pointers to shared elements is the better approach.

4.8.1.1. Homogeneity

The next issue to consider is whether to enforce homogeneity in the list elements. That is, should lists be restricted so that all data elements stored are of the same object type? Or should it be possible to store different types?

If you want to enforce homogeneity, the most rigid way is to simply define the elements to be of a fixed type. But that does not help if you want one list to store integers while another stores strings. A much more flexible approach is to use Java generics or C++ templates. In this way, the compiler will enforce that a given list will only store a single data type, while still allowing different lists to have different data types. Another approach is to store an object of the appropriate type in the header node of the list (perhaps an object of the appropriate type is supplied as a parameter to the list constructor), and then check that all insert operations on that list use the same element type. This approach is useful in a language like JavaScript that does not use strong typing, but does allow a program to test the type of an object.

In some applications, the designer would like to allow a given list store elements with different types. In Java, declaring the element to be of type Object will stop the compiler from enforcing any type restrictions. In C++, a similar effect can be achieved by using void* pointers.

4.8.1.2. Element Deletion

Our last design issue is what to do to the list elements when the list itself is deleted? This is a serious concern in a language like C++ that does not support automatic garbage collection.

4.8.1.3. Practice Questions

4.9. Stacks

4.9.1. Stack Terminology and Implementation

The stack is a list-like structure in which elements may be inserted or removed from only one end. While this restriction makes stacks less flexible than lists, it also makes stacks both efficient (for those operations they can do) and easy to implement. Many applications require only the limited form of insert and remove operations that stacks provide. In such cases, it is more efficient to use the simpler stack data structure rather than the generic list. For example, the freelist is really a stack.

Despite their restrictions, stacks have many uses. Thus, a special vocabulary for stacks has developed. Accountants used stacks long before the invention of the computer. They called the stack a “LIFO“ list, which stands for “Last-In, First-Out.” Note that one implication of the LIFO policy is that stacks remove elements in reverse order of their arrival.

The accessible element of the stack is called the top element. Elements are not said to be inserted, they are pushed onto the stack. When removed, an element is said to be popped from the stack. Here is a simple stack ADT.

public interface Stack<E> { // Stack class ADT
  // Reinitialize the stack.
  public void clear();

  // Push "it" onto the top of the stack
  public boolean push(E it);

  // Remove and return the element at the top of the stack
  public E pop();

  // Return a copy of the top element
  public E topValue();

  // Return the number of elements in the stack
  public int length();

  // Tell if the stack is empty or not
  public boolean isEmpty();
}

As with lists, there are many variations on stack implementation. The two approaches presented here are the array-based stack and the linked stack, which are analogous to array-based and linked lists, respectively.

4.9.1.1. Array-Based Stacks

Here is a complete implementation for the array-based stack class.

class AStack<E> implements Stack<E> {
  private E stackArray[];         // Array holding stack
  private static final int DEFAULT_SIZE = 10;
  private int maxSize;            // Maximum size of stack
  private int top;                // First free position at top

  // Constructors
  @SuppressWarnings("unchecked") // Generic array allocation
  AStack(int size) {
    maxSize = size;
    top = 0;
    stackArray = (E[])new Object[size]; // Create stackArray
  }
  AStack() { this(DEFAULT_SIZE); }

  public void clear() { top = 0; }    // Reinitialize stack

// Push "it" onto stack
  public boolean push(E it) {
    if (top >= maxSize) return false;
    stackArray[top++] = it;
    return true;
  }

// Remove and return top element
  public E pop() {
    if (top == 0) return null;
    return stackArray[--top];
  }

  public E topValue() {          // Return top element
    if (top == 0) return null;
    return stackArray[top-1];
  }

  public int length() { return top; } // Return stack size

  public boolean isEmpty() { return top == 0; }  // Tell if the stack is empty
}

The array-based stack implementation is essentially a simplified version of the array-based list. The only important design decision to be made is which end of the array should represent the top of the stack.

4.9.2. Pop

4.10. Linked Stacks

4.10.1. Linked Stack Implementation

The linked stack implementation is quite simple. Elements are inserted and removed only from the head of the list. A header node is not used because no special-case code is required for lists of zero or one elements. Here is the complete linked stack implementation.

// Linked stack implementation
class LStack<E> implements Stack<E> {
  private Link<E> top;            // Pointer to first element
  private int size;               // Number of elements

  // Constructors
  LStack() { top = null; size = 0; }
  LStack(int size) { top = null; size = 0; }

  // Reinitialize stack
  public void clear() { top = null; size = 0; }

// Put "it" on stack
  public boolean push(E it) {
    top = new Link<E>(it, top);
    size++;
    return true;
  }

// Remove "it" from stack
  public E pop() {
    if (top == null) return null;
    E it = top.element();
    top = top.next();
    size--;
    return it;
  }

  public E topValue() {      // Return top value
    if (top == null) return null;
    return top.element();
  }

  // Return stack length
  public int length() { return size; }

  // Tell if the stack is empty
  public boolean isEmpty() { return size == 0; }
}

Here is a visual representation for the linked stack.

4.10.1.1. Linked Stack Push

4.10.2. Linked Stack Pop

4.10.2.1. Comparison of Array-Based and Linked Stacks

All operations for the array-based and linked stack implementations take constant time, so from a time efficiency perspective, neither has a significant advantage. Another basis for comparison is the total space required. The analysis is similar to that done for list implementations. The array-based stack must declare a fixed-size array initially, and some of that space is wasted whenever the stack is not full. The linked stack can shrink and grow but requires the overhead of a link field for every element.

When implementing multiple stacks, sometimes you can take advantage of the one-way growth of the array-based stack by using a single array to store two stacks. One stack grows inward from each end as illustrated by the figure below, hopefully leading to less wasted space. However, this only works well when the space requirements of the two stacks are inversely correlated. In other words, ideally when one stack grows, the other will shrink. This is particularly effective when elements are taken from one stack and given to the other. If instead both stacks grow at the same time, then the free space in the middle of the array will be exhausted quickly.

4.11. Implementing Recursion

WARNING! You should not read this section unless you are already comfortable with implementing recursive functions. One of the biggest hang-ups for students learning recursion is too much focus on the recursive “process”. The right way to think about recursion is to just think about the return value that the recursive call gives back. Thinking about how that answer is computed just gets in the way of understanding. There are good reasons to understand how recursion is implemented, but helping you to write recursive functions is not one of them.

Perhaps the most common computer application that uses stacks is not even visible to its users. This is the implementation of subroutine calls in most programming language runtime environments. A subroutine call is normally implemented by pushing necessary information about the subroutine (including the return address, parameters, and local variables) onto a stack. This information is called an activation record. Further subroutine calls add to the stack. Each return from a subroutine pops the top activation record off the stack. As an example, here is a recursive implementation for the factorial function.

// Recursively compute and return n!
long rfact(int n) {
  // fact(20) is the largest value that fits in a long
  if ((n < 0) || (n > 20)) return -1;
  if (n <= 1)  return 1;  // Base case: return base solution
  return n * rfact(n-1);   // Recursive call for n > 1
}

Here is an illustration for how the internal processing works.

values indicate the address of the program instruction to return to after completing the current function call. On each recursive function call to fact, both the return address and the current value of n must be saved. Each return from fact pops the top activation record off the stack.

Consider what happens when we call fact with the value 4. We use to indicate the address of the program instruction where the call to fact is made. Thus, the stack must first store the address , and the value 4 is passed to fact. Next, a recursive call to fact is made, this time with value 3. We will name the program address from which the call is made . The address , along with the current value for (which is 4), is saved on the stack. Function fact is invoked with input parameter 3.

In similar manner, another recursive call is made with input parameter 2, requiring that the address from which the call is made (say ) and the current value for (which is 3) are stored on the stack. A final recursive call with input parameter 1 is made, requiring that the stack store the calling address (say ) and current value (which is 2).

At this point, we have reached the base case for fact, and so the recursion begins to unwind. Each return from fact involves popping the stored value for from the stack, along with the return address from the function call. The return value for fact is multiplied by the restored value for , and the result is returned.

Because an activation record must be created and placed onto the stack for each subroutine call, making subroutine calls is a relatively expensive operation. While recursion is often used to make implementation easy and clear, sometimes you might want to eliminate the overhead imposed by the recursive function calls. In some cases, such as the factorial function above, recursion can easily be replaced by iteration.

Example 4.11.1

As a simple example of replacing recursion with a stack, consider the following non-recursive version of the factorial function.

// Return n!
long sfact(int n) {
  // fact(20) is the largest value that fits in a long
  if ((n < 0) || (n > 20)) return -1;
  // Make a stack just big enough
  Stack S = new AStack(n);
  while (n > 1) S.push(n--);
  long result = 1;
  while (S.length() > 0)
    result = result * (Integer)S.pop();
  return result;
}

Here, we simply push successively smaller values of onto the stack until the base case is reached, then repeatedly pop off the stored values and multiply them into the result.

An iterative form of the factorial function is both simpler and faster than the version shown in the example. But it is not always possible to replace recursion with iteration. Recursion, or some imitation of it, is necessary when implementing algorithms that require multiple branching such as in the Towers of Hanoi algorithm, or when traversing a binary tree. The Mergesort and Quicksort sorting algorithms also require recursion. Fortunately, it is always possible to imitate recursion with a stack. Let us now turn to a non-recursive version of the Towers of Hanoi function, which cannot be done iteratively.

Example 4.11.2

Here is a recursive implementation for Towers of Hanoi.

// Compute the moves to solve a Tower of Hanoi puzzle.
// Function move does (or prints) the actual move of a disk
// from one pole to another.
// n: The number of disks
// start: The start pole
// goal: The goal pole
// temp: The other pole
static void TOH(int n, Pole start, Pole goal, Pole temp) {
  if (n == 0) return;          // Base case
  TOH(n-1, start, temp, goal); // Recursive call: n-1 rings
  move(start, goal);            // Move bottom disk to goal
  TOH(n-1, temp, goal, start); // Recursive call: n-1 rings
}

TOH makes two recursive calls: one to move rings off the bottom ring, and another to move these rings back to the goal pole. We can eliminate the recursion by using a stack to store a representation of the three operations that TOH must perform: two recursive calls and a move operation. To do so, we must first come up with a representation of the various operations, implemented as a class whose objects will be stored on the stack.

class TOHobj {
  int op;
  int num;
  Pole start, goal, temp;

  // Recursive call operation
  TOHobj(int o, int n, Pole s, Pole g, Pole t)
  { op = o; num = n; start = s; goal = g; temp = t; }

  // MOVE operation
  TOHobj(int o, Pole s, Pole g)
  { op = o; start = s; goal = g; }
}

void TOHs(int n, Pole start, Pole goal, Pole temp) {
  // Make a stack just big enough
  Stack S = new AStack(2*n+1);
  S.push(new TOHobj(TOH, n, start, goal, temp));
  while (S.length() > 0) {
    TOHobj it = (TOHobj)S.pop();   // Get next task
    if (it.op == MOVE) // Do a move
      move(it.start, it.goal);
    else if (it.num > 0) { // Imitate TOH recursive solution (in reverse)
      S.push(new TOHobj(TOH, it.num-1, it.temp, it.goal, it.start));
      S.push(new TOHobj(MOVE, it.start, it.goal));  // A move to do
      S.push(new TOHobj(TOH, it.num-1, it.start, it.temp, it.goal));
    }
  }
}

We first enumerate the possible operations MOVE and TOH, to indicate calls to the move function and recursive calls to TOH, respectively. Class TOHobj stores five values: an operation value (indicating either a MOVE or a new TOH operation), the number of rings, and the three poles. Note that the move operation actually needs only to store information about two poles. Thus, there are two constructors: one to store the state when imitating a recursive call, and one to store the state for a move operation.

An array-based stack is used because we know that the stack will need to store exactly elements. The new version of TOH begins by placing on the stack a description of the initial problem for rings. The rest of the function is simply a while loop that pops the stack and executes the appropriate operation. In the case of a TOH operation (for ), we store on the stack representations for the three operations executed by the recursive version. However, these operations must be placed on the stack in reverse order, so that they will be popped off in the correct order.

Recursive algorithms lend themselves to efficient implementation with a stack when the amount of information needed to describe a sub-problem is small. For example, Quicksort can effectively use a stack to replace its recursion since only bounds information for the subarray to be processed needs to be saved.

4.12. Queues

4.12.1. Queue Terminology and Implementation

Like the stack, the queue is a list-like structure that provides restricted access to its elements. Queue elements may only be inserted at the back (called an enqueue operation) and removed from the front (called a dequeue operation). Queues operate like standing in line at a movie theater ticket counter. If nobody cheats, then newcomers go to the back of the line. The person at the front of the line is the next to be served. Thus, queues release their elements in order of arrival. In Britain, a line of people is called a “queue”, and getting into line to wait for service is called “queuing up”. Accountants have used queues since long before the existence of computers. They call a queue a “FIFO” list, which stands for “First-In, First-Out”. Here is a sample queue ADT. This section presents two implementations for queues: the array-based queue and the linked queue.

public interface Queue<E> { // Queue class ADT
  // Reinitialize queue
  public void clear();

  // Put element on rear
  public boolean enqueue(E it);

  // Remove and return element from front
  public E dequeue();

  // Return front element
  public E frontValue();

  // Return queue size
  public int length();

  //Tell if the queue is empty or not
  public boolean isEmpty();
}

4.12.1.1. Array-Based Queues

The array-based queue is somewhat tricky to implement effectively. A simple conversion of the array-based list implementation is not efficient.

4.12.1.2. The Circular Queue

If the value of front is fixed, then different values for rear are needed to distinguish among the states. However, there are only possible values for rear unless we invent a special case for, say, empty queues. This is an example of the Pigeonhole Principle. The Pigeonhole Principle states that, given pigeonholes and pigeons, when all of the pigeons go into the holes we can be sure that at least one hole contains more than one pigeon. In similar manner, we can be sure that two of the states are indistinguishable by the relative values of front and rear. We must seek some other way to distinguish full from empty queues.

One obvious solution is to keep an explicit count of the number of elements in the queue, or at least a Boolean variable that indicates whether the queue is empty or not. Another solution is to make the array be of size , and only allow elements to be stored. Which of these solutions to adopt is purely a matter of the implementor’s taste in such affairs. Our choice here is to use an array of size .

Here is an array-based queue implementation.

class AQueue<E> implements Queue<E> {
  private E queueArray[];      // Array holding queue elements
  private static final int DEFAULT_SIZE = 10;
  private int maxSize;         // Maximum size of queue
  private int front;           // Index of front element
  private int rear;            // Index of rear element

  // Constructors
  @SuppressWarnings("unchecked") // Generic array allocation
  AQueue(int size) {
    maxSize = size+1;          // One extra space is allocated
    rear = 0; front = 1;
    queueArray = (E[])new Object[maxSize];  // Create queueArray
  }
  AQueue() { this(DEFAULT_SIZE); }

  // Reinitialize
  public void clear() { rear = 0; front = 1; }

  // Put "it" in queue
  public boolean enqueue(E it) {
    if (((rear+2) % maxSize) == front) return false;  // Full
    rear = (rear+1) % maxSize; // Circular increment
    queueArray[rear] = it;
    return true;
  }

  // Remove and return front value
  public E dequeue() {
    if(length() == 0) return null;
    E it = queueArray[front];
    front = (front+1) % maxSize; // Circular increment
    return it;
  }

  // Return front value
  public E frontValue() {
    if (length() == 0) return null;
    return queueArray[front];
  }

  // Return queue size
  public int length() { return ((rear+maxSize) - front + 1) % maxSize; }

  //Tell if the queue is empty or not
  public boolean isEmpty() { return front - rear == 1; }
}

4.12.1.3. Array-based Queue Implementation

In this implementation, the front of the queue is defined to be toward the lower numbered positions in the array (in the counter-clockwise direction in the circular array), and the rear is defined to be toward the higher-numbered positions. Thus, enqueue increments the rear pointer (modulus maxSize), and dequeue increments the front pointer. Implementation of all member functions is straightforward.

4.12.2. Array-based Dequeue Practice

4.13. Linked Queues

4.13.1. Linked Queues

The linked queue implementation is a straightforward adaptation of the linked list. Here is the linked queue class declaration.

// Linked queue implementation
class LQueue<E> implements Queue<E> {
  private Link<E> front; // Pointer to front queue node
  private Link<E> rear;  // Pointer to rear queue node
  private int size;      // Number of elements in queue

  // Constructors
  LQueue() { init(); }
  LQueue(int size) { init(); } // Ignore size

  // Initialize queue
  void init() {
    front = rear = new Link<E>(null);
    size = 0;
  }

  // Put element on rear
  public boolean enqueue(E it) {
    rear.setNext(new Link<E>(it, null));
    rear = rear.next();
    size++;
    return true;
  }

  // Remove and return element from front
  public E dequeue() {
    if (size == 0) return null;
    E it = front.next().element(); // Store the value
    front.setNext(front.next().next()); // Advance front
    if (front.next() == null) rear = front; // Last element
    size--;
    return it; // Return element
  }

  // Return front element
  public E frontValue() {
    if (size == 0) return null;
    return front.next().element();
  }

  // Return queue size
  public int length() { return size; }

  //Tell if the queue is empty or not
  public boolean isEmpty() { return size == 0; }
}

4.13.2. Linked Dequeue

4.13.3. Comparison of Array-Based and Linked Queues

All member functions for both the array-based and linked queue implementations require constant time. The space comparison issues are the same as for the equivalent stack implementations. Unlike the array-based stack implementation, there is no convenient way to store two queues in the same array, unless items are always transferred directly from one queue to the other.

4.14. Linear Structure Summary Exercises

4.14.1. Practice Questions

Here are some general practice questions about various data structures in this chapter.

4.14.2. Chapter Review Questions

Here is a summary exercise with questions from everything in this chapter.

Chapter 5: Design

5.1. Alternative List ADT Designs

The list ADT specifies that a List comprises not only a collection of objects in linear order, but also “the current position”. While this is a simple way to present the main concepts embodied by a list, it complicates any algorithm that relies on having two or more distinct “current positions” in the same list, such as any algorithm that steps from both ends towards the middle.

An alternative design is to separate the “current position” as a separate object. In the following ADT, we will call this a ListIndex. This is a simple form of a concept that is sometimes called an iterator. The ListIndex interface abstracts the notion of a position in a list.

interface ListIndex {
  void prev();
  void next();
}

interface List {
  void clear();
  void insert(Object it, ListIndex where);
  void append(Object it);
  Object remove(ListIndex where);
  ListIndex getStart();
  ListIndex getEnd();
  ListIndex pointToPos(int where);
  int length();
  Object getValue(ListIndex where);
}

There is the issue in an implementation of how the two classes will communicate. For the array-based list, the ListIndex merely needs to store an integer for the position. For the linked list class, the ListIndex would store a pointer to a linked list node. This means that the List class needs to be able to set and get this pointer, but nobody outside should need to know about it. Some languages like Java and C++ have mechanisms that allow a specific class to have access to non-public members of another class. Oher languages like Processing have no such concept.

One general solution is to make the interface for ListIndex public, but make the implementation a private inner class of the List implementation. This approach is used in the following implmentation for the Array-based list.

// Array-based list implementation
class AList implements List {
  private class AListIndex implements ListIndex {
    int pos;

    AListIndex(int posit) { pos = posit; }
    void prev() { if (pos != 0) pos--; }
    void next() { if (pos < listSize) pos++; }
  }

  private static final int defaultSize = 10; // Default size
  private int maxSize;                    // Maximum size of list
  private int listSize;                   // Current # of list items
  private Object listArray[];             // Array holding list elements

  // Constructors
  // Create a new list object with maximum size "size"
  AList(int size) {
    maxSize = size;
    listSize = 0;
    listArray = new Object[size];         // Create listArray
  }
  // Create a list with the default capacity
  AList() { this(defaultSize); }          // Just call the other constructor

  void clear()                     // Reinitialize the list
    { listSize = 0; }              // Simply reinitialize values

  // Insert "it" at current position
  void insert(Object it, ListIndex where) {
    if (listSize >= maxSize) {
      println("List capacity exceeded, nothing inserted");
      return;
    }
    int pos = ((AListIndex)where).pos;
    for (int i=listSize; i>pos; i--)     // Shift elements up
      listArray[i] = listArray[i-1];      //   to make room
    listArray[pos] = it;
    listSize++;                           // Increment list size
  }

  // Append "it" to list
  void append(Object it) {
    if (listSize >= maxSize) {
      println("List capacity exceeded, nothing inserted");
      return;
    }
    listArray[listSize++] = it;
  }

  // Remove and return the current element
  Object remove(ListIndex where) {
    int pos = ((AListIndex)where).pos;
    if ((pos<0) || (pos>=listSize))     // No current element
      return null;
    Object it = listArray[pos];          // Copy the element
    for(int i=pos; i<listSize-1; i++)    // Shift them down
      listArray[i] = listArray[i+1];
    listSize--;                           // Decrement size
    return it;
  }

  // Return list size
  int length() { return listSize; }

  // Return a ListIndex to the beginning of the list
  ListIndex getStart() {
    return new AListIndex(0);
  }

  // Return a ListIndex past the end of the list
  ListIndex getEnd() {
    return new AListIndex(listSize);
  }

  ListIndex pointToPos(int pos) {
    return new AListIndex(pos);
  }

  // Return the current element
  Object getValue(ListIndex where) {
    int pos = ((AListIndex)where).pos;
    if ((pos < 0) || (pos >= listSize)) // No current element
      return null;
    return listArray[pos];
  }
}

5.2. Comparing Records

5.2.1. Comparing Records

If we want to sort some things, we have to be able to compare them, to decide which one is bigger. How do we compare two things? If all that we wanted to sort or search for was simple integer values, this would not be an interesting question. We can just use standard comparison operators like “<” or “>”. Even if we wanted to store strings, most programming languages give us built-in functions for comparing strings alphabetically. But we do not usually want to store just integers or strings in a data structure. Usually we want to store records, where a record is made up of multiple values, such as a name, an address, and a phone number. In that case, how can we “compare” records to decide which one is “smaller”? We cannot just use “<” to compare the records! Nearly always in this situation, we actually are interested in sorting the records based on the values of one particular field used to represent the record, which itself is something simple like an integer. This field is referred to as the key for the record.

Likewise, if we want to search for a given record in a database, how should we describe what we are looking for? A database record could simply be a number, or it could be quite complicated, such as a payroll record with many fields of varying types. We do not want to describe what we are looking for by detailing and matching the entire contents of the record. If we knew everything about the record already, we probably would not need to look for it. Instead, we typically define what record we want in terms of a key value. For example, if searching for payroll records, we might wish to search for the record that matches a particular ID number. In this example the ID number is the search key.

To implement sorting or searching, we require that keys be comparable. At a minimum, we must be able to take two keys and reliably determine whether they are equal or not. That is enough to enable a sequential search through a database of records and find one that matches a given key. However, we typically would like for the keys to define a total order, which means that we can always tell which of two keys is greater than the other. Using key types with total orderings gives the database implementor the opportunity to organize a collection of records in a way that makes searching more efficient. An example is storing the records in sorted order in an array, which permits a binary search. Fortunately, in practice most fields of most records consist of simple data types with natural total orders. For example, integers, floats, doubles, and character strings all are totally ordered.

But if we want to write a general purpose sorting or searching function, we need a general way to get the key for the record. We could insist that every record have a particular method called .key(). That seems like a good name for it!

Some languages like Java and C++ have special infrastructure for supporting this (such as the Comparable interface in Java, which has the .compareTo() method for defining the exact process by which two objects are compared). But many languages like Processing and JavaScript do not.

But what if the programmer had already used that method name for another purpose? An even bigger problem is, what if the programmer wants to sort the record now using one field as the key, and later using another field? Or search sometimes on one key, and at other times on another? The problem is that the “keyness” of a given field is not an inherent property within the record, but rather depends on the context. So, you cannot always count on being able to use your favorite method name (or even the comparable interface) to extract the desired key value.

Another, more general approach is to supply a function or class—called a comparator—whose job is to extract the key from the record. A comparator function can be passed in as a parameter, such as in a call to a sorting function. In this case, the comparator function would be invoked on two records whenever they need to be compared. In this way, different comparator functions can be passed in to handle different record types or different fields within a record. In Java (with generics) or C++ (with templates), a comparator class can be a parameter for another class definition. For example, a BST could take a comparator class as a generics parameter in Java. This comparator class would be responsible for dealing with the comparison of two records.

Unfortunately, while flexible and able to handle nearly all situations, there are a few situations for which it is not possible to write a key extraction method. In that case, a comparator will not work. [^1]

One good general-purpose solution is to explicitly store key-value pairs in the data structure. For example, if we want to sort a bunch of records, we can store them in an array where every array entry contains both a key value for the record and a pointer to the record itself. This might seem like a lot of extra space required, but remember that we can then store pointers to the records in another array with another field as the key for another purpose. The records themselves do not need to be duplicated. A simple class for representing key-value pairs is shown here.

// KVPair class definition
public class KVPair<K extends Comparable<K>, E> implements Comparable<KVPair<K, E>> {
  K theKey;
  E theVal;

  KVPair(K k, E v) {
    theKey = k;
    theVal = v;
  }

  // Compare KVPairs
  public int compareTo(KVPair<K,E> it) {
    return theKey.compareTo(it.key());
  }

  // Compare against a key
  public int compareTo(K it) {
    return theKey.compareTo(it);
  }

  public K key() {
    return theKey;
  }

  public E value() {
    return theVal;
  }

  public String toString() {
    String s = "(";
    if (theKey != null) { s += theKey.toString(); }
    else { s += "null"; }
    s += ", ";
    if (theVal != null) { s += theVal.toString(); }
    else { s += "null"; }
    s += ")";
    return s;
  }
}

The main places where we will need to be concerned with comparing records and extracting keys is for various dictionary implementations and sorting algorithms. To keep them clear and simple, visualizations for sorting algorithms will usually show them as operating on integer values stored in an array. But almost never do people really want to sort an array of integers. But to be useful, a real sorting algorithm typically has to deal with the fact that it is sorting a collection of records. A general-purpose sorting routine meant to operate on multiple record types would have to be written in a way to deal with the generic comparison problem. To illustrate, here is an example of Insertion Sort implemented to work on an array that stores records that support the Comparable interface. Note that since KVPair is implemented to implement the Comparable interface, an array of KVPair could be used by this sort function.

static <T extends Comparable<T>> void inssort(T[] A) {
  for (int i=1; i<A.length; i++) // Insert i'th record
    for (int j=i; (j>0) && (A[j].compareTo(A[j-1]) < 0); j--)
      swap(A, j, j-1);
}

Here are some review questions to test your knowledge from this module.

[^1]: One example of a situation where it is not possible to write a function that extracts a key from a record is when we have a collection of records that describe books in a library. One of the fields for such a record might be a list of subject keywords, where the typical record stores a few keywords. Our dictionary might be implemented as a list of records sorted by keyword. If a book contains three keywords, it would appear three times on the list, once for each associated keyword. However, given the record, there is no simple way to determine which keyword on the keyword list triggered this appearance of the record. Thus, we cannot write a function that extracts the key from such a record.

5.3. The Dictionary ADT

5.3.1. The Dictionary ADT

The most common objective of computer programs is to store and retrieve data. Much of this book is about efficient ways to organize collections of data records so that they can be stored and retrieved quickly. In this section we describe a simple interface for such a collection, called a dictionary. The dictionary ADT provides operations for storing records, finding records, and removing records from the collection. This ADT gives us a standard basis for comparing various data structures. Loosly speaking, we can say that any data structure that supports insert, search, and deletion is a “dictionary”.

Dictionaries depend on the concepts of a search key and comparable objects. To implement the dictionary’s search function, we will require that keys be totally ordered. Ordering fields that are naturally multi-dimensional, such as a point in two or three dimensions, present special opportunities if we wish to take advantage of their multidimensional nature. This problem is addressed by spatial data structures.

Here is code to define a simple abstract dictionary class.

/** The Dictionary abstract class. */
public interface Dictionary<K, E> {

  /** Reinitialize dictionary */
  public void clear();

  /** Insert a record
      @param k The key for the record being inserted.
      @param e The record being inserted. */
  public void insert(K key, E elem);

  /** Remove and return a record.
      @param k The key of the record to be removed.
      @return A maching record. If multiple records match
      "k", remove an arbitrary one. Return null if no record
      with key "k" exists. */
  public E remove(K key);

  /** Remove and return an arbitrary record from dictionary.
      @return the record removed, or null if none exists. */
  public E removeAny();

  /** @return A record matching "k" (null if none exists).
      If multiple records match, return an arbitrary one.
      @param k The key of the record to find */
  public E find(K key);

  /** @return The number of records in the dictionary. */
  public int size();
}

The methods insert and find are the heart of the class. Method insert takes a record and inserts it into the dictionary. Method find takes a key value and returns some record from the dictionary whose key matches the one provided. If there are multiple records in the dictionary with that key value, there is no requirement as to which one is returned.

Method clear simply re-initializes the dictionary. The remove method is similar to find, except that it also deletes the record returned from the dictionary. Once again, if there are multiple records in the dictionary that match the desired key, there is no requirement as to which one actually is removed and returned. Method size returns the number of elements in the dictionary.

The remaining Method is removeAny. This is similar to remove, except that it does not take a key value. Instead, it removes an arbitrary record from the dictionary, if one exists. The purpose of this method is to allow a user the ability to iterate over all elements in the dictionary (of course, the dictionary will become empty in the process). Without the removeAny method, dictionary users could not get at a record of the dictionary that they didn’t already know the key value for. With the removeAny method, the user can process all records in the dictionary as shown in the following code fragment.

while (dict.size() > 0) {
  Object it = dict.removeAny();
  doSomething(it);
}

There are other approaches that might seem more natural for iterating though a dictionary, such as using a “first” and a “next” function. But not all data structures that we want to use to implement a dictionary are able to do “first” efficiently. For example, a hash table implementation cannot efficiently locate the record in the table with the smallest key value. By using RemoveAny, we have a mechanism that provides generic access.

Given a database storing records of a particular type, we might want to search for records in multiple ways. For example, we might want to store payroll records in one dictionary that allows us to search by ID, and also store those same records in a second dictionary that allows us to search by name.

Here is an implementation for a payroll record.

/** A simple payroll entry with ID, name, address fields */
class Payroll {

  private Integer ID;
  private String name;
  private String address;

  /** Constructor */
  Payroll(int inID, String inname, String inaddr) {
    ID = inID;
    name = inname;
    address = inaddr;
  }

  /** Data member access functions */
  Integer getID() { return ID; }
  String getname() { return name; }
  String getaddr() { return address; }
}

Class Payroll has multiple fields, each of which might be used as a search key. Simply by varying the type for the key, and using the appropriate field in each record as the key value, we can define a dictionary whose search key is the ID field, another whose search key is the name field, and a third whose search key is the address field. Here is an example where Payroll objects are stored in two separate dictionaries, one using the ID field as the key and the other using the name field as the key.

// IDdict organizes Payroll records by ID
Dictionary IDdict = new UALdictionary();

// namedict organizes Payroll records by name
Dictionary namedict = new UALdictionary();

Payroll foo1 = new Payroll(5, "Joe", "Anytown");
Payroll foo2 = new Payroll(10, "John", "Mytown");

IDdict.insert(foo1.getID(), foo1);
IDdict.insert(foo2.getID(), foo2);
namedict.insert(foo1.getname(), foo1);
namedict.insert(foo2.getname(), foo2);

Payroll findfoo1 = (Payroll)IDdict.find(5);
Payroll findfoo2 = (Payroll)namedict.find("John");

One problem with the example as it is written is that the dictionary relies on the programmer to be reasonable about being consistent with the keys. These dictionaries are intended to have homogeneous elements. But nothing stops the programmer from inserting an integer key into the names dictionary, or searching with an integer search key. This problem can be handled by using C++ templates or Java generics.

The fundamental operation for a dictionary is finding a record that matches a given key. This raises the issue of how to extract the key from a record. We will usually assume that dictionary implementations store a key-value pair so as to be able to extract the key associated with a record for this particular dictionary.

The insert method of the dictionary class supports the key-value pair implementation because it takes two parameters, a record and its associated key for that dictionary.

Now that we have defined the dictionary ADT and settled on the design approach of storing key-value pairs for our dictionary entries, we are ready to consider ways to implement it. Two possibilities would be to use an array-based or linked list. Here is an implementation for the dictionary using an (unsorted) array-based list.

// Dictionary implemented by unsorted array-based list.
class UALdictionary implements Dictionary {
  private static final int defaultSize = 10; // Default size
  private AList list;                        // To store dictionary

  // Constructors
  UALdictionary() { this(defaultSize); }
  UALdictionary(int sz) { list = new AList(sz); }

  // Reinitialize
  void clear() { list.clear(); }

  // Insert an element: append to list
  void insert(Comparable k, Object e) {
    KVPair temp = new KVPair(k, e);
    list.append(temp);
  }

  // Use sequential search to find the element to remove
  Object remove(Comparable k) {
    Object temp = find(k);
    if (temp != null) list.remove();
    return temp;
  }

  // Remove the last element
  Object removeAny() {
    if (size() != 0) {
      list.moveToEnd();
      list.prev();
      KVPair e = (KVPair)list.remove();
      return e.value();
    }
    else return null;
  }

  // Find k using sequential search
  // Return the record with key value k
  Object find(Comparable k) {
    for(list.moveToStart(); list.currPos() < list.length();
        list.next()) {
      KVPair temp = (KVPair)list.getValue();
      if (k.compareTo(temp.key()))
        return temp.value();
    }
    return null; // "k" does not appear in dictionary
  }

  // Return list size
  int size() { return list.length(); }
}

Examining class UALdict (UAL stands for “unsorted array-based list”), we can easily see that insert is a constant-time operation, because it simply inserts the new record at the end of the list. However, find, and remove both require time in the average and worst cases, because we need to do a sequential search. Method remove in particular must touch every record in the list, because once the desired record is found, the remaining records must be shifted down in the list to fill the gap. Method removeAny removes the last record from the list, so this is a constant-time operation.

As an alternative, we could implement the dictionary using a linked list. The implementation would be quite similar to that for UALDictionary, and the cost of the functions should be the same asymptotically.

Another alternative would be to implement the dictionary with a sorted list. The advantage of this approach would be that we might be able to speed up the find operation by using a binary search. To do so, first we must define a variation on the List ADT to support sorted lists. A sorted list is somewhat different from an unsorted list in that it cannot permit the user to control where elements get inserted. Thus, the insert method must be quite different in a sorted list than in an unsorted list. Likewise, the user cannot be permitted to append elements onto the list. For these reasons, a sorted list cannot be implemented with straightforward inheritance from the List ADT.

The cost for find in a sorted list is for a list of length . This is a great improvement over the cost of find in an unsorted list. Unfortunately, the cost of insert changes from constant time in the unsorted list to time in the sorted list. Whether the sorted list implementation for the dictionary ADT is more or less efficient than the unsorted list implementation depends on the relative number of insert and find operations to be performed. If many more find operations than insert operations are used, then it might be worth using a sorted list to implement the dictionary. In both cases, remove requires time in the worst and average cases. Even if we used binary search to cut down on the time to find the record prior to removal, we would still need to shift down the remaining records in the list to fill the gap left by the remove operation.

Search trees are search structures that can perform all three key operations of insert, search, and delete in time.

Chapter 6: Binary Trees

6.1. Binary Trees Chapter Introduction

Tree structures enable efficient access and efficient update to large collections of data. Binary trees in particular are widely used and relatively easy to implement. But binary trees are useful for many things besides searching. Just a few examples of applications that trees can speed up include prioritizing jobs, describing mathematical expressions and the syntactic elements of computer programs, or organizing the information needed to drive data compression algorithms.

This chapter covers terminology used for discussing binary trees, tree traversals, approaches to implementing tree nodes, and various examples of binary trees.

6.2. Binary Trees

6.2.1. Definitions and Properties

A binary tree is made up of a finite set of elements called nodes. This set either is empty or consists of a node called the root together with two binary trees, called the left and right subtrees, which are disjoint from each other and from the root. (Disjoint means that they have no nodes in common.) The roots of these subtrees are children of the root. There is an edge from a node to each of its children, and a node is said to be the parent of its children.

If is a sequence of nodes in the tree such that is the parent of for , then this sequence is called a path from to . The length of the path is . If there is a path from node to node , then is an ancestor of , and is a descendant of . Thus, all nodes in the tree are descendants of the root of the tree, while the root is the ancestor of all nodes. The depth of a node in the tree is the length of the path from the root of the tree to . The height of a tree is the depth of the deepest node in the tree. All nodes of depth are at level in the tree. The root is the only node at level 0, and its depth is 0. A leaf node is any node that has two empty children. An internal node is any node that has at least one non-empty child.

Figure 6.2.1: A binary tree. Node is the root. Nodes and are ’s children. Nodes and together form a subtree. Node has two children: Its left child is the empty tree and its right child is . Nodes , , and are ancestors of . Nodes , , and make up level 2 of the tree; node is at level 0. The edges from to to to form a path of length 3. Nodes , , , and are leaves. Nodes , , , , and are internal nodes. The depth of is 3. The height of this tree is 3.

Figure 6.2.2: Two different binary trees. (a) A binary tree whose root has a non-empty left child. (b) A binary tree whose root has a non-empty right child. (c) The binary tree of (a) with the missing right child made explicit. (d) The binary tree of (b) with the missing left child made explicit.

Figure 6.2.1 illustrates the various terms used to identify parts of a binary tree. Figure 6.2.2 illustrates an important point regarding the structure of binary trees. Because all binary tree nodes have two children (one or both of which might be empty), the two binary trees of Figure 6.2.2 are not the same.

Two restricted forms of binary tree are sufficiently important to warrant special names. Each node in a full binary tree is either (1) an internal node with exactly two non-empty children or (2) a leaf. A complete binary tree has a restricted shape obtained by starting at the root and filling the tree by levels from left to right. In the complete binary tree of height , all levels except possibly level are completely full. The bottom level has its nodes filled in from the left side.

Figure 6.2.3: Examples of full and complete binary trees.

Figure 6.2.3 illustrates the differences between full and complete binary trees. [^1] There is no particular relationship between these two tree shapes; that is, the tree of Figure 6.2.3 (a) is full but not complete while the tree of Figure 6.2.3 (b) is complete but not full. The heap data structure is an example of a complete binary tree. The Huffman coding tree is an example of a full binary tree.

[^1]: While these definitions for full and complete binary tree are the ones most commonly used, they are not universal. Because the common meaning of the words “full” and “complete” are quite similar, there is little that you can do to distinguish between them other than to memorize the definitions. Here is a memory aid that you might find useful: “Complete” is a wider word than “full”, and complete binary trees tend to be wider than full binary trees because each level of a complete binary tree is as wide as possible.

6.2.2. Practice Questions

6.3. Binary Tree as a Recursive Data Structure

6.3.1. Binary Tree as a Recursive Data Structure

A recursive data structure is a data structure that is partially composed of smaller or simpler instances of the same data structure. For example, linked lists and binary trees can be viewed as recursive data structures. A list is a recursive data structure because a list can be defined as either (1) an empty list or (2) a node followed by a list. A binary tree is typically defined as (1) an empty tree or (2) a node pointing to two binary trees, one its left child and the other one its right child.

The recursive relationships used to define a structure provide a natural model for any recursive algorithm on the structure.

6.4. The Full Binary Tree Theorem

Some binary tree implementations store data only at the leaf nodes, using the internal nodes to provide structure to the tree. By definition, a leaf node does not need to store pointers to its (empty) children. More generally, binary tree implementations might require some amount of space for internal nodes, and a different amount for leaf nodes. Thus, to compute the space required by such implementations, it is useful to know the minimum and maximum fraction of the nodes that are leaves in a tree containing internal nodes.

Unfortunately, this fraction is not fixed. A binary tree of internal nodes might have only one leaf. This occurs when the internal nodes are arranged in a chain ending in a single leaf as shown in Figure 6.4.1. In this example, the number of leaves is low because each internal node has only one non-empty child. To find an upper bound on the number of leaves for a tree of internal nodes, first note that the upper bound will occur when each internal node has two non-empty children, that is, when the tree is full. However, this observation does not tell what shape of tree will yield the highest percentage of non-empty leaves. It turns out not to matter, because all full binary trees with internal nodes have the same number of leaves. This fact allows us to compute the space requirements for a full binary tree implementation whose leaves require a different amount of space from its internal nodes.

Figure 6.4.1: A tree containing many internal nodes and a single leaf.

Theorem 6.4.1

Full Binary Tree Theorem: The number of leaves in a non-empty full binary tree is one more than the number of internal nodes.

Proof: The proof is by mathematical induction on , the number of internal nodes. This is an example of the style of induction proof where we reduce from an arbitrary instance of size to an instance of size that meets the induction hypothesis.

1. Base Cases: The non-empty tree with zero internal nodes has one leaf node. A full binary tree with one internal node has two leaf nodes. Thus, the base cases for and conform to the theorem.
2. Induction Hypothesis: Assume that any full binary tree containing internal nodes has leaves.
3. Induction Step: Given tree with internal nodes, select an internal node whose children are both leaf nodes. Remove both of ’s children, making a leaf node. Call the new tree . has internal nodes. From the induction hypothesis, has leaves. Now, restore ’s two children. We once again have tree with internal nodes. How many leaves does have? Because has leaves, adding the two children yields . However, node counted as one of the leaves in and has now become an internal node. Thus, tree has leaf nodes and internal nodes.

By mathematical induction the theorem holds for all values of .

When analyzing the space requirements for a binary tree implementation, it is useful to know how many empty subtrees a tree contains. A simple extension of the Full Binary Tree Theorem tells us exactly how many empty subtrees there are in any binary tree, whether full or not. Here are two approaches to proving the following theorem, and each suggests a useful way of thinking about binary trees.

Theorem 6.4.2

The number of empty subtrees in a non-empty binary tree is one more than the number of nodes in the tree.

Proof 1: Take an arbitrary binary tree and replace every empty subtree with a leaf node. Call the new tree . All nodes originally in will be internal nodes in (because even the leaf nodes of have children in ). is a full binary tree, because every internal node of now must have two children in , and each leaf node in must have two children in (the leaves just added). The Full Binary Tree Theorem tells us that the number of leaves in a full binary tree is one more than the number of internal nodes. Thus, the number of new leaves that were added to create is one more than the number of nodes in . Each leaf node in corresponds to an empty subtree in . Thus, the number of empty subtrees in is one more than the number of nodes in .

Proof 2: By definition, every node in binary tree has two children, for a total of children in a tree of nodes. Every node except the root node has one parent, for a total of nodes with parents. In other words, there are non-empty children. Because the total number of children is , the remaining children must be empty.

6.5. Binary Tree Traversals

6.5.1. Binary Tree Traversals

Often we wish to process a binary tree by “visiting” each of its nodes, each time performing a specific action such as printing the contents of the node. Any process for visiting all of the nodes in some order is called a traversal. Any traversal that lists every node in the tree exactly once is called an enumeration of the tree’s nodes. Some applications do not require that the nodes be visited in any particular order as long as each node is visited precisely once. For other applications, nodes must be visited in an order that preserves some relationship.

6.5.1.1. Preorder Traversal

For example, we might wish to make sure that we visit any given node before we visit its children. This is called a preorder traversal.

Figure 6.5.1: A binary tree for traversal examples.

Example 6.5.1

The preorder enumeration for the tree of Figure 6.5.1 is A B D C E G F H I.

The first node printed is the root. Then all nodes of the left subtree are printed (in preorder) before any node of the right subtree.

6.5.1.2. Postorder Traversal

Alternatively, we might wish to visit each node only after we visit its children (and their subtrees). For example, this would be necessary if we wish to return all nodes in the tree to free store. We would like to delete the children of a node before deleting the node itself. But to do that requires that the children’s children be deleted first, and so on. This is called a postorder traversal.

Example 6.5.2

The postorder enumeration for the tree of Figure 6.5.1 is D B G E H I F C A.

6.5.1.3. Inorder Traversal

An inorder traversal first visits the left child (including its entire subtree), then visits the node, and finally visits the right child (including its entire subtree). The binary search tree makes use of this traversal to print all nodes in ascending order of value.

Example 6.5.3

The inorder enumeration for the tree of Figure 6.5.1 is B D A G E C H F I.

6.5.1.4. Implementation

Now we will discuss some implementations for the traversals, but we need to define a node ADT to work with. Just as a linked list is composed of a collection of link objects, a tree is composed of a collection of node objects. Here is an ADT for binary tree nodes, called BinNode. This class will be used by some of the binary tree structures presented later. Member functions are provided that set or return the element value, return a pointer to the left child, return a pointer to the right child, or indicate whether the node is a leaf.

interface BinNode<E> { // Binary tree node ADT
  // Get and set the element value
  public E value();
  public void setValue(E v);

  // return the children
  public BinNode<E> left();
  public BinNode<E> right();

  // return TRUE if a leaf node, FALSE otherwise
  public boolean isLeaf();
}

A traversal routine is naturally written as a recursive function. Its input parameter is a pointer to a node which we will call rt because each node can be viewed as the root of a some subtree. The initial call to the traversal function passes in a pointer to the root node of the tree. The traversal function visits rt and its children (if any) in the desired order. For example, a preorder traversal specifies that rt be visited before its children. This can easily be implemented as follows.

static <E> void preorder(BinNode<E> rt) {
  if (rt == null) return; // Empty subtree - do nothing
  visit(rt);              // Process root node
  preorder(rt.left());    // Process all nodes in left
  preorder(rt.right());   // Process all nodes in right
}

Function preorder first checks that the tree is not empty (if it is, then the traversal is done and preorder simply returns). Otherwise, preorder makes a call to visit, which processes the root node (i.e., prints the value or performs whatever computation as required by the application). Function preorder is then called recursively on the left subtree, which will visit all nodes in that subtree. Finally, preorder is called on the right subtree, visiting all nodes in the right subtree. Postorder and inorder traversals are similar. They simply change the order in which the node and its children are visited, as appropriate.

6.5.2. Postorder Traversal Practice

6.5.3. Inorder Traversal Practice

6.6. Implementing Tree Traversals

6.6.1. Implementing Tree Traversals

Recall that any recursive function requires the following:

1. The base case and its action.
2. The recursive case and its action.

In this module, we will talk about some details related to correctly and clearly implementing recursive tree traversals.

6.6.1.1. Base Case

In binary tree traversals, most often the base case is to check if we have an empty tree. A common mistake is to check the child pointers of the current node, and only make the recursive call for a non-null child.

Recall the basic preorder traversal function.

static <E> void preorder(BinNode<E> rt) {
  if (rt == null) return; // Empty subtree - do nothing
  visit(rt);              // Process root node
  preorder(rt.left());    // Process all nodes in left
  preorder(rt.right());   // Process all nodes in right
}

Here is an alternate design for the preorder traversal, in which the left and right pointers of the current node are checked so that the recursive call is made only on non-empty children.

// This is a bad idea
static <E> void preorder2(BinNode<E> rt) {
  visit(rt);
  if (rt.left() != null) preorder2(rt.left());
  if (rt.right() != null) preorder2(rt.right());
}

At first it might appear that preorder2 is more efficient than preorder, because it makes only half as many recursive calls (since it won’t try to call on a null pointer). On the other hand, preorder2 must access the left and right child pointers twice as often. The net result is that there is no performance improvement.

Perhaps the writer of preorder2 wants to protect against the case where the root is null. But preorder2 has an error. While preorder2 insures that no recursive calls will be made on empty subtrees, it will fail if the orignal call from outside passes in a null pointer. This would occur if the original tree is empty. Since an empty tree is a legitimate input to the initial call on the function, there is no safe way to avoid this case. So it is necessary that the first thing you do on a binary tree traversal is to check that the root is not null. If we try to fix preorder2 by adding this test, then making the tests on the children is completely redundant because the pointer will be checked again in the recursive call.

The design of preorder2 is inferior to that of preorder for a deeper reason as well. Looking at the children to see if they are null means that we are worrying too much about something that can be dealt with just as well by the children. This makes the function more complex, which can become a real problem for more complex tree structures. Even in the relatively simple preorder2 function, we had to write two tests for null rather than the one needed by preorder. This makes it more complicated than the original version. The key issue is that it is much easier to write a recursive function on a tree when we only think about the needs of the current node. Whenever we can, we want to let the children take care of themselves. In this case, we care that the current node is not null, and we care about how to invoke the recursion on the children, but we do not have to care about how or when that is done.

6.6.1.2. The Recursive Call

The secret to success when writing a recursive function is to not worry about how the recursive call works. Just accept that it will work correctly. One aspect of this principle is not to worry about checking your children when you don’t need to. You should only look at the values of your children if you need to know those values in order to compute some property of the current node. Child values should not be used to decide whether to call them recursviely. Make the call, and let their own base case handle it.

Example 6.6.1

Consider the problem of incrementing the value for each node in a binary tree. The following solution has an error, since it does redundant manipulation to left and the right children of each node.

static void ineff_BTinc(BinNode root) {
  if (root != null) {
    root.setValue((root.value()) + 1);
    if (root.left() != null) {
      root.left().setValue((root.left().value()) + 1);
      ineff_BTinc(root.left().left());
    }
    if (root.right() != null) {
      root.right().setValue((root.right().value()) + 1);
      ineff_BTinc(root.right().right());
    }
  }
}

The efficient solution should not explicitly set the children values that way. Changing the value of a node does not depend on the child values. So the function should simply increment the root value, and make recursive calls on the children.

In rare problems, you might need to explicitly check if the children are null or access the children values for each node. For example, you might need to check if all nodes in a tree satisfy the property that each node stores the sum of its left and right children. In this situation you must look at the values of the children to decide something about the current node. You do not look at the children to decide whether to make a recursive call.

6.7. Binary Tree Node Implementations

6.7.1. Binary Tree Node Implementations

In this module we examine various ways to implement binary tree nodes. By definition, all binary tree nodes have two children, though one or both children can be empty. Binary tree nodes typically contain a value field, with the type of the field depending on the application. The most common node implementation includes a value field and pointers to the two children.

Here is a simple implementation for the BinNode interface, which we will name BSTNode. Its element type is an Object. When we need to support search structures such as the Binary Search Tree, the node will typically store a key-value pair. Every BSTNode object also has two pointers, one to its left child and another to its right child.

// Binary tree node implementation: supports comparable objects
class BSTNode<E extends Comparable<? super E>> implements BinNode<E> {
  private E element;           // Element for this node
  private BSTNode<E> left;     // Pointer to left child
  private BSTNode<E> right;    // Pointer to right child

  // Constructors
  BSTNode() {left = right = null; }
  BSTNode(E val) { left = right = null; element = val; }
  BSTNode(E val, BSTNode<E> l, BSTNode<E> r)
    { left = l; right = r; element = val; }

  // Get and set the element value
  public E value() { return element; }
  public void setValue(E v) { element = v; }

  // Get and set the left child
  public BSTNode<E> left() { return left; }
  public void setLeft(BSTNode<E> p) { left = p; }

  // Get and set the right child
  public BSTNode<E> right() { return right; }
  public void setRight(BSTNode<E> p) { right = p; }

  // return TRUE if a leaf node, FALSE otherwise
  public boolean isLeaf() { return (left == null) && (right == null); }
}

Figure 6.7.1: Illustration of a typical pointer-based binary tree implementation, where each node stores two child pointers and a value.

Some programmers find it convenient to add a pointer to the node’s parent, allowing easy upward movement in the tree. Using a parent pointer is somewhat analogous to adding a link to the previous node in a doubly linked list. In practice, the parent pointer is almost always unnecessary and adds to the space overhead for the tree implementation. It is not just a problem that parent pointers take space. More importantly, many uses of the parent pointer are driven by improper understanding of recursion and so indicate poor programming. If you are inclined toward using a parent pointer, consider if there is a more efficient implementation possible.

An important decision in the design of a pointer-based node implementation is whether the same class definition will be used for leaves and internal nodes. Using the same class for both will simplify the implementation, but might be an inefficient use of space. Some applications require data values only for the leaves. Other applications require one type of value for the leaves and another for the internal nodes. Examples include the binary trie, the PR Quadtree, the Huffman coding tree, and the expression tree illustrated by Figure 6.7.2. By definition, only internal nodes have non-empty children. If we use the same node implementation for both internal and leaf nodes, then both must store the child pointers. But it seems wasteful to store child pointers in the leaf nodes. Thus, there are many reasons why it can save space to have separate implementations for internal and leaf nodes.

Figure 6.7.2: An expression tree for .

As an example of a tree that stores different information at the leaf and internal nodes, consider the expression tree illustrated by Figure 6.7.2. The expression tree represents an algebraic expression composed of binary operators such as addition, subtraction, multiplication, and division. Internal nodes store operators, while the leaves store operands. The tree of Figure 6.7.2 represents the expression . The storage requirements for a leaf in an expression tree are quite different from those of an internal node. Internal nodes store one of a small set of operators, so internal nodes could store a small code identifying the operator such as a single byte for the operator’s character symbol. In contrast, leaves store variable names or numbers, which is considerably larger in order to handle the wider range of possible values. At the same time, leaf nodes need not store child pointers.

Object-oriented languages allow us to differentiate leaf from internal nodes through the use of a class hierarchy. A base class provides a general definition for an object, and a subclass modifies a base class to add more detail. A base class can be declared for binary tree nodes in general, with subclasses defined for the internal and leaf nodes. The base class in the following code is named VarBinNode. It includes a virtual member function named isLeaf, which indicates the node type. Subclasses for the internal and leaf node types each implement isLeaf. Internal nodes store child pointers of the base class type; they do not distinguish their children’s actual subclass. Whenever a node is examined, its version of isLeaf indicates the node’s subclass.

// Base class for expression tree nodes
interface VarBinNode {
  boolean isLeaf(); // All subclasses must implement
}

/** Leaf node */
class VarLeafNode implements VarBinNode {
  private String operand;                 // Operand value

  VarLeafNode(String val) { operand = val; }
  boolean isLeaf() { return true; }
  String value() { return operand; }
}

/** Internal node */
class VarIntlNode implements VarBinNode {
  private VarBinNode left;                // Left child
  private VarBinNode right;               // Right child
  private Character operator;             // Operator value

  VarIntlNode(Character op, VarBinNode l, VarBinNode r)
    { operator = op; left = l; right = r; }
  boolean isLeaf() { return false; }
  VarBinNode leftchild() { return left; }
  VarBinNode rightchild() { return right; }
  Character value() { return operator; }
}

/** Preorder traversal */
static void traverse(VarBinNode rt) {
  if (rt == null) return;          // Nothing to visit
  if (rt.isLeaf())                 // Process leaf node
    Visit.VisitLeafNode(((VarLeafNode)rt).value());
  else {                           // Process internal node
    Visit.VisitInternalNode(((VarIntlNode)rt).value());
    traverse(((VarIntlNode)rt).leftchild());
    traverse(((VarIntlNode)rt).rightchild());
  }
}

The Expression Tree implementation includes two subclasses derived from class VarBinNode, named LeafNode and IntlNode. Class IntlNode can access its children through pointers of type VarBinNode. Function traverse illustrates the use of these classes. When traverse calls method isLeaf, the language’s runtime environment determines which subclass this particular instance of rt happens to be and calls that subclass’s version of isLeaf. Method isLeaf then provides the actual node type to its caller. The other member functions for the derived subclasses are accessed by type-casting the base class pointer as appropriate, as shown in function traverse.

6.8. Composite-based Expression Tree

6.8.1. Composite-based Expression Tree

There is another approach that we can take to represent separate leaf and internal nodes, also using a virtual base class and separate node classes for the two types. This is to implement nodes using the Composite design pattern. This approach is noticeably different from the procedural approach in that the node classes themselves implement the functionality of traverse. Here is the implementation. Base class VarBinNode declares a member function traverse that each subclass must implement. Each subclass then implements its own appropriate behavior for its role in a traversal. The whole traversal process is called by invoking traverse on the root node, which in turn invokes traverse on its children.

/** Base class: Composite */
interface VarBinNode {
  boolean isLeaf();
  void traverse();
}

/** Leaf node: Composite */
class VarLeafNode implements VarBinNode {
  private String operand;                 // Operand value

  VarLeafNode(String val) { operand = val; }
  boolean isLeaf() { return true; }
  String value() { return operand; }

  void traverse() {
    Visit.VisitLeafNode(operand);
  }
}

/** Internal node: Composite */
class VarIntlNode implements VarBinNode { // Internal node
  private VarBinNode left;                // Left child
  private VarBinNode right;               // Right child
  private Character operator;             // Operator value

  VarIntlNode(Character op,
                     VarBinNode l, VarBinNode r)
    { operator = op; left = l; right = r; }
  boolean isLeaf() { return false; }
  VarBinNode leftchild() { return left; }
  VarBinNode rightchild() { return right; }
  Character value() { return operator; }

  void traverse() {
    Visit.VisitInternalNode(operator);
    if (left != null) left.traverse();
    if (right != null) right.traverse();
  }
}

/** Preorder traversal */
static void traverse(VarBinNode rt) {
  if (rt != null) rt.traverse();
}

When comparing the composite implementation to the procedural approach, each has advantages and disadvantages. The non-composite approach does not require that the node classes know about the traverse function. With this approach, it is easy to add new methods to the tree class that do other traversals or other operations on nodes of the tree. However, we see that traverse in the non-composite approach does need to be familiar with each node subclass. Adding a new node subclass would therefore require modifications to the traverse function. In contrast, the composite approach requires that any new operation on the tree that requires a traversal also be implemented in the node subclasses. On the other hand, the composite approach avoids the need for the traverse function to know anything about the distinct abilities of the node subclasses. Those subclasses handle the responsibility of performing a traversal on themselves. A secondary benefit is that there is no need for traverse to explicitly enumerate all of the different node subclasses, directing appropriate action for each. With only two node classes this is a minor point. But if there were many such subclasses, this could become a bigger problem. A disadvantage is that the traversal operation must not be called on a NULL pointer, because there is no object to catch the call. This problem could be avoided by using a Flyweight to implement empty nodes.

Typically, the non-composite version would be preferred in this example if traverse is a member function of the tree class, and if the node subclasses are hidden from users of that tree class. On the other hand, if the nodes are objects that have meaning to users of the tree separate from their existence as nodes in the tree, then the composite version might be preferred because hiding the internal behavior of the nodes becomes more important.

Another advantage of the composite design is that implementing each node type’s functionality might be easier. This is because you can focus solely on the information passing and other behavior needed by this node type to do its job. This breaks down the complexity that many programmers feel overwhelmed by when dealing with complex information flows related to recursive processing.

6.9. Binary Tree Space Requirements

6.9.1. Binary Tree Space Requirements

This module presents techniques for calculating the amount of overhead required by a binary tree, based on its node implementation. Recall that overhead is the amount of space necessary to maintain the data structure. In other words, it is any space not used to store data records. The amount of overhead depends on several factors including which nodes store data values (all nodes, or just the leaves), whether the leaves store child pointers, and whether the tree is a full binary tree.

In a simple pointer-based implementation for binary tree nodes, every node has two pointers to its children (even when the children are NULL). This implementation requires total space amounting to for a tree of nodes. Here, stands for the amount of space required by a pointer, and stands for the amount of space required by a data value. The total overhead space will be for the entire tree. Thus, the overhead fraction will be . The actual value for this expression depends on the relative size of pointers versus data fields. If we arbitrarily assume that , then a binary tree has about two thirds of its total space taken up in overhead. Worse yet, the Full Binary Tree Theorem tells us that about half of the pointers are “wasted” NULL values that serve only to indicate tree structure, but which do not provide access to new data.

In many languages (such as Java or JavaScript), the most typical implementation is not to store any actual data in a node, but rather a pointer to the data record. In this case, each node will typically store three pointers, all of which are overhead, resulting in an overhead fraction of .

If only leaves store data values, then the fraction of total space devoted to overhead depends on whether the tree is full. If the tree is not full, then conceivably there might only be one leaf node at the end of a series of internal nodes. Thus, the overhead can be an arbitrarily high percentage for non-full binary trees. The overhead fraction drops as the tree becomes closer to full, being lowest when the tree is truly full. In this case, about one half of the nodes are internal.

Great savings can be had by eliminating the pointers from leaf nodes in full binary trees. Again assume the tree stores a pointer to the data field. Because about half of the nodes are leaves and half internal nodes, and because only internal nodes now have child pointers, the overhead fraction in this case will be approximately

If , the overhead drops to about one half of the total space. However, if only leaf nodes store useful information, the overhead fraction for this implementation is actually three quarters of the total space, because half of the “data” space is unused.

If a full binary tree needs to store data only at the leaf nodes, a better implementation would have the internal nodes store two pointers and no data field while the leaf nodes store only a pointer to the data field. This implementation requires

units of space. If , then the overhead is . It might seem counter-intuitive that the overhead ratio has gone up while the total amount of space has gone down. The reason is because we have changed our definition of “data” to refer only to what is stored in the leaf nodes, so while the overhead fraction is higher, it is from a total storage requirement that is lower.

There is one serious flaw with this analysis. When using separate implementations for internal and leaf nodes, there must be a way to distinguish between the node types. When separate node types are implemented via Java subclasses, the runtime environment stores information with each object allowing it to determine, for example, the correct subclass to use when the isLeaf virtual function is called. Thus, each node requires additional space. Only one bit is truly necessary to distinguish the two possibilities. In rare applications where space is a critical resource, implementors can often find a spare bit within the node’s value field in which to store the node type indicator. An alternative is to use a spare bit within a node pointer to indicate node type. For example, this is often possible when the compiler requires that structures and objects start on word boundaries, leaving the last bit of a pointer value always zero. Thus, this bit can be used to store the node-type flag and is reset to zero before the pointer is dereferenced. Another alternative when the leaf value field is smaller than a pointer is to replace the pointer to a leaf with that leaf’s value. When space is limited, such techniques can make the difference between success and failure. In any other situation, such “bit packing” tricks should be avoided because they are difficult to debug and understand at best, and are often machine dependent at worst.

6.10. Binary Search Trees

6.10.1. Binary Search Tree Definition

A binary search tree (BST) is a binary tree that conforms to the following condition, known as the binary search tree property. All nodes stored in the left subtree of a node whose key value is have key values less than or equal to . All nodes stored in the right subtree of a node whose key value is have key values greater than . Figure 6.10.1 shows two BSTs for a collection of values. One consequence of the binary search tree property is that if the BST nodes are printed using an inorder traversal, then the resulting enumeration will be in sorted order from lowest to highest.

Figure 6.10.1: Two Binary Search Trees for a collection of values. Tree (a) results if values are inserted in the order 37, 24, 42, 7, 2, 40, 42, 32, 120. Tree (b) results if the same values are inserted in the order 120, 42, 42, 7, 2, 32, 37, 24, 40.

Here is a class declaration for the BST. Recall that there are various ways to deal with keys and comparing records Three typical approaches are key-value pairs, a special comparison method such as using the Comparator class, and passing in a comparator function. Our BST implementation will require that records implement the Comparable interface.

// Binary Search Tree implementation
class BST<E extends Comparable<E>> {
  private BSTNode<E> root; // Root of the BST
  private int nodecount; // Number of nodes in the BST

  // constructor
  BST() { root = null; nodecount = 0; }

  // Reinitialize tree
  public void clear() { root = null; nodecount = 0; }

  // Insert a record into the tree.
  // Records can be anything, but they must be Comparable
  // e: The record to insert.
  public void insert(E e) {
    root = inserthelp(root, e);
    nodecount++;
  }

  // Remove a record from the tree
  // key: The key value of record to remove
  // Returns the record removed, null if there is none.
  public E remove(E key) {
    E temp = findhelp(root, key); // First find it
    if (temp != null) {
      root = removehelp(root, key); // Now remove it
      nodecount--;
    }
    return temp;
  }

  // Return the record with key value k, null if none exists
  // key: The key value to find
  public E find(E key) { return findhelp(root, key); }

  // Return the number of records in the dictionary
  public int size() { return nodecount; }

6.10.1.1. BST Search

The first operation that we will look at in detail will find the record that matches a given key. Notice that in the BST class, public member function find calls private member function findhelp. Method find takes the search key as an explicit parameter and its BST as an implicit parameter, and returns the record that matches the key. However, the find operation is most easily implemented as a recursive function whose parameters are the root of a subtree and the search key. Member findhelp has the desired form for this recursive subroutine and is implemented as follows.

6.10.2. BST Insert

Now we look at how to insert a new node into the BST.

Note that, except for the last node in the path, inserthelp will not actually change the child pointer for any of the nodes that are visited. In that sense, many of the assignments seem redundant. However, the cost of these additional assignments is worth paying to keep the insertion process simple. The alternative is to check if a given assignment is necessary, which is probably more expensive than the assignment!

We have to decide what to do when the node that we want to insert has a key value equal to the key of some node already in the tree. If during insert we find a node that duplicates the key value to be inserted, then we have two options. If the application does not allow nodes with equal keys, then this insertion should be treated as an error (or ignored). If duplicate keys are allowed, our convention will be to insert the duplicate in the left subtree.

The shape of a BST depends on the order in which elements are inserted. A new element is added to the BST as a new leaf node, potentially increasing the depth of the tree. Figure 6.10.1 illustrates two BSTs for a collection of values. It is possible for the BST containing nodes to be a chain of nodes with height . This would happen if, for example, all elements were inserted in sorted order. In general, it is preferable for a BST to be as shallow as possible. This keeps the average cost of a BST operation low.

6.10.3. BST Remove

Removing a node from a BST is a bit trickier than inserting a node, but it is not complicated if all of the possible cases are considered individually. Before tackling the general node removal process, we will first see how to remove from a given subtree the node with the largest key value. This routine will be used later by the general node removal function.

The return value of the deletemax method is the subtree of the current node with the maximum-valued node in the subtree removed. Similar to the inserthelp method, each node on the path back to the root has its right child pointer reassigned to the subtree resulting from its call to the deletemax method.

A useful companion method is getmax which returns a pointer to the node containing the maximum value in the subtree.

// Get the maximum valued element in a subtree
private BSTNode<E> getmax(BSTNode<E> rt) {
  if (rt.right() == null) return rt;
  return getmax(rt.right());
}

Now we are ready for the removehelp method. Removing a node with given key value from the BST requires that we first find and then remove it from the tree. So, the first part of the remove operation is a search to find . Once is found, there are several possibilities. If has no children, then ’s parent has its pointer set to NULL. If has one child, then ’s parent has its pointer set to ’s child (similar to deletemax). The problem comes if has two children. One simple approach, though expensive, is to set ’s parent to point to one of ’s subtrees, and then reinsert the remaining subtree’s nodes one at a time. A better alternative is to find a value in one of the subtrees that can replace the value in .

Thus, the question becomes: Which value can substitute for the one being removed? It cannot be any arbitrary value, because we must preserve the BST property without making major changes to the structure of the tree. Which value is most like the one being removed? The answer is the least key value greater than the one being removed, or else the greatest key value less than (or equal to) the one being removed. If either of these values replace the one being removed, then the BST property is maintained.

When duplicate node values do not appear in the tree, it makes no difference whether the replacement is the greatest value from the left subtree or the least value from the right subtree. If duplicates are stored in the left subtree, then we must select the replacement from the left subtree. [^1] To see why, call the least value in the right subtree . If multiple nodes in the right subtree have value , selecting as the replacement value for the root of the subtree will result in a tree with equal values to the right of the node now containing . Selecting the greatest value from the left subtree does not have a similar problem, because it does not violate the Binary Search Tree Property if equal values appear in the left subtree.

[^1]: Alternatively, if we prefer to store duplicate values in the right subtree, then we must replace a deleted node with the least value from its right subtree.

6.10.4. BST Analysis

The cost for findhelp and inserthelp is the depth of the node found or inserted. The cost for removehelp is the depth of the node being removed, or in the case when this node has two children, the depth of the node with smallest value in its right subtree. Thus, in the worst case, the cost for any one of these operations is the depth of the deepest node in the tree. This is why it is desirable to keep BSTs balanced, that is, with least possible height. If a binary tree is balanced, then the height for a tree of nodes is approximately . However, if the tree is completely unbalanced, for example in the shape of a linked list, then the height for a tree with nodes can be as great as . Thus, a balanced BST will in the average case have operations costing , while a badly unbalanced BST can have operations in the worst case costing . Consider the situation where we construct a BST of nodes by inserting records one at a time. If we are fortunate to have them arrive in an order that results in a balanced tree (a “random” order is likely to be good enough for this purpose), then each insertion will cost on average , for a total cost of . However, if the records are inserted in order of increasing value, then the resulting tree will be a chain of height . The cost of insertion in this case will be .

Traversing a BST costs regardless of the shape of the tree. Each node is visited exactly once, and each child pointer is followed exactly once.

Below is an example traversal, named printhelp. It performs an inorder traversal on the BST to print the node values in ascending order.

private void printhelp(BSTNode<E> rt) {
  if (rt == null) return;
  printhelp(rt.left());
  printVisit(rt.value());
  printhelp(rt.right());
}

While the BST is simple to implement and efficient when the tree is balanced, the possibility of its being unbalanced is a serious liability. There are techniques for organizing a BST to guarantee good performance. Two examples are the AVL tree and the splay tree. There also exist other types of search trees that are guaranteed to remain balanced, such as the 2-3 Tree.

6.11. Dictionary Implementation Using a BST

A simple implementation for the Dictionary ADT can be based on sorted or unsorted lists. When implementing the dictionary with an unsorted list, inserting a new record into the dictionary can be performed quickly by putting it at the end of the list. However, searching an unsorted list for a particular record requires time in the average case. For a large database, this is probably much too slow. Alternatively, the records can be stored in a sorted list. If the list is implemented using a linked list, then no speedup to the search operation will result from storing the records in sorted order. On the other hand, if we use a sorted array-based list to implement the dictionary, then binary search can be used to find a record in only time. However, insertion will now require time on average because, once the proper location for the new record in the sorted list has been found, many records might be shifted to make room for the new record.

Is there some way to organize a collection of records so that inserting records and searching for records can both be done quickly? We can do this with a binary search tree (BST). The advantage of using the BST is that all major operations (insert, search, and remove) are in the average case. Of course, if the tree is badly balanced, then the cost can be as bad as .

Here is an implementation for the Dictionary interface, using a BST to store the records.

// Dictionary implementation using BST
// This uses KVPair to manage the key/value pairs
class BSTDict implements Dictionary {
  private BST theBST; // The BST that stores the records

  // constructor
  BSTDict() { theBST = new BST(); }

  // Reinitialize dictionary
  public void clear() { theBST = new BST(); }

  // Insert a record
  // k: the key for the record being inserted.
  // e: the record being inserted.
  void insert(Comparable k, Object e) {
    theBST.insert(new KVPair(k, e));
  }

  // Remove and return a record.
  // k: the key of the record to be removed.
  // Return a maching record. If multiple records match "k", remove
  // an arbitrary one. Return null if no record with key "k" exists.
  Object remove(Comparable k) {
    Object temp = theBST.remove(k);
    if (temp == null) return temp;
    else return ((KVPair)temp).value();
  }

  // Remove and return an arbitrary record from dictionary.
  // Return the record removed, or null if none exists.
  Object removeAny() {
    if (theBST.size() == 0) return null;
    Object temp = theBST.remove(((KVPair)(theBST.root.element)).key());
    return ((KVPair)temp).value();
  }

  // Return a record matching "k" (null if none exists).
  // If multiple records match, return an arbitrary one.
  // k: the key of the record to find
  Object find(Comparable k) {
    Object temp = theBST.find(k);
    if (temp == null) return temp;
    else return ((KVPair)temp).value();
  }

  // Return the number of records in the dictionary.
  int size() {
    return theBST.size();
  }
}

6.12. Array Implementation for Complete Binary Trees

6.12.1. Array Implementation for Complete Binary Trees

From the full binary tree theorem, we know that a large fraction of the space in a typical binary tree node implementation is devoted to structural overhead, not to storing data. This module presents a simple, compact implementation for complete binary trees. Recall that complete binary trees have all levels except the bottom filled out completely, and the bottom level has all of its nodes filled in from left to right. Thus, a complete binary tree of nodes has only one possible shape. You might think that a complete binary tree is such an unusual occurrence that there is no reason to develop a special implementation for it. However, the complete binary tree has practical uses, the most important being the heap data structure. Heaps are often used to implement priority queues and for external sorting algorithms.

We begin by assigning numbers to the node positions in the complete binary tree, level by level, from left to right as shown in Figure 6.12.1. An array can store the tree’s data values efficiently, placing each data value in the array position corresponding to that node’s position within the tree. The table lists the array indices for the children, parent, and siblings of each node in Figure 6.12.1.

Figure 6.12.1: A complete binary tree of 12 nodes, numbered starting from 0.

Here is a table that lists, for each node position, the positions of the parent, sibling, and children of the node.

Looking at the table, you should see a pattern regarding the positions of a node’s relatives within the array. Simple formulas can be derived for calculating the array index for each relative of a node from ’s index. No explicit pointers are necessary to reach a node’s left or right child. This means there is no overhead to the array implementation if the array is selected to be of size for a tree of nodes.

The formulae for calculating the array indices of the various relatives of a node are as follows. The total number of nodes in the tree is . The index of the node in question is , which must fall in the range 0 to .

• Parent() if .
• Left child() if .
• Right child() if .
• Left sibling() if is even and .
• Right sibling() if is odd and .

6.13. Heaps and Priority Queues

6.13.1. Heaps and Priority Queues

There are many situations, both in real life and in computing applications, where we wish to choose the next “most important” from a collection of people, tasks, or objects. For example, doctors in a hospital emergency room often choose to see next the “most critical” patient rather than the one who arrived first. When scheduling programs for execution in a multitasking operating system, at any given moment there might be several programs (usually called jobs) ready to run. The next job selected is the one with the highest priority. Priority is indicated by a particular value associated with the job (and might change while the job remains in the wait list).

When a collection of objects is organized by importance or priority, we call this a priority queue. A normal queue data structure will not implement a priority queue efficiently because search for the element with highest priority will take time. A list, whether sorted or not, will also require time for either insertion or removal. A BST that organizes records by priority could be used, with the total of inserts and remove operations requiring time in the average case. However, there is always the possibility that the BST will become unbalanced, leading to bad performance. Instead, we would like to find a data structure that is guaranteed to have good performance for this special application.

This section presents the heap [^1] data structure. A heap is defined by two properties. First, it is a complete binary tree, so heaps are nearly always implemented using the array representation for complete binary trees. Second, the values stored in a heap are partially ordered. This means that there is a relationship between the value stored at any node and the values of its children. There are two variants of the heap, depending on the definition of this relationship.

[^1]: Note that the term “heap” is also sometimes used to refer to free store.

A max heap has the property that every node stores a value that is greater than or equal to the value of either of its children. Because the root has a value greater than or equal to its children, which in turn have values greater than or equal to their children, the root stores the maximum of all values in the tree.

A min heap has the property that every node stores a value that is less than or equal to that of its children. Because the root has a value less than or equal to its children, which in turn have values less than or equal to their children, the root stores the minimum of all values in the tree.

Note that there is no necessary relationship between the value of a node and that of its sibling in either the min heap or the max heap. For example, it is possible that the values for all nodes in the left subtree of the root are greater than the values for every node of the right subtree. We can contrast BSTs and heaps by the strength of their ordering relationships. A BST defines a total order on its nodes in that, given the positions for any two nodes in the tree, the one to the “left” (equivalently, the one appearing earlier in an inorder traversal) has a smaller key value than the one to the “right”. In contrast, a heap implements a partial order. Given their positions, we can determine the relative order for the key values of two nodes in the heap only if one is a descendant of the other.

Min heaps and max heaps both have their uses. For example, the Heapsort uses the max heap, while the Replacement Selection algorithm used for external sorting uses a min heap. The examples in the rest of this section will use a max heap.

Be sure not to confuse the logical representation of a heap with its physical implementation by means of the array-based complete binary tree. The two are not synonymous because the logical view of the heap is actually a tree structure, while the typical physical implementation uses an array.

Here is an implementation for max heaps. The class uses records that support the Comparable interface to provide flexibility.

// Max-heap implementation
class MaxHeap {
  private Comparable[] Heap; // Pointer to the heap array
  private int size;          // Maximum size of the heap
  private int n;             // Number of things now in heap

  // Constructor supporting preloading of heap contents
  MaxHeap(Comparable[] h, int num, int max)
  { Heap = h;  n = num;  size = max;  buildheap(); }

  // Return current size of the heap
  int heapsize() { return n; }

  // Return true if pos a leaf position, false otherwise
  boolean isLeaf(int pos)
  { return (pos >= n/2) && (pos < n); }

  // Return position for left child of pos
  int leftchild(int pos) {
    if (pos >= n/2) return -1;
    return 2*pos + 1;
  }

  // Return position for right child of pos
  int rightchild(int pos) {
    if (pos >= (n-1)/2) return -1;
    return 2*pos + 2;
  }

  // Return position for parent
  int parent(int pos) {
    if (pos <= 0) return -1;
    return (pos-1)/2;
  }

  // Insert val into heap
  void insert(int key) {
    if (n >= size) {
      println("Heap is full");
      return;
    }
    int curr = n++;
    Heap[curr] = key;  // Start at end of heap
    // Now sift up until curr's parent's key > curr's key
    while ((curr != 0) && (Heap[curr].compareTo(Heap[parent(curr)]) > 0)) {
      swap(Heap, curr, parent(curr));
      curr = parent(curr);
    }
  }

  // Heapify contents of Heap
  void buildheap()
    { for (int i=n/2-1; i>=0; i--) siftdown(i); }

  // Put element in its correct place
  void siftdown(int pos) {
    if ((pos < 0) || (pos >= n)) return; // Illegal position
    while (!isLeaf(pos)) {
      int j = leftchild(pos);
      if ((j<(n-1)) && (Heap[j].compareTo(Heap[j+1]) < 0))
        j++; // j is now index of child with greater value
      if (Heap[pos].compareTo(Heap[j]) >= 0) return;
      swap(Heap, pos, j);
      pos = j;  // Move down
    }
  }

  // Remove and return maximum value
  Comparable removemax() {
    if (n == 0) return -1;  // Removing from empty heap
    swap(Heap, 0, --n); // Swap maximum with last value
    if (n != 0)      // Not on last element
      siftdown(0);   // Put new heap root val in correct place
    return Heap[n];
  }

  // Remove and return element at specified position
  Comparable remove(int pos) {
    if ((pos < 0) || (pos >= n)) return -1; // Illegal heap position
    if (pos == (n-1)) n--; // Last element, no work to be done
    else {
      swap(Heap, pos, --n); // Swap with last value
      update(pos);
    }
  }

  // Modify the value at the given position
  void modify(int pos, Comparable newVal) {
    if ((pos < 0) || (pos >= n)) return; // Illegal heap position
    Heap[pos] = newVal();
    update(pos);
  }

  // The value at pos has been changed, restore the heap property
  void update(pos) {
    // If it is a big value, push it up
    while ((pos > 0) && (Heap[pos].compareTo(Heap[parent(pos)]) > 0)) {
      swap(Heap, pos, parent(pos));
      pos = parent(pos);
    }
    if (n != 0) siftdown(pos); // If it is little, push down
  }
}

This class definition makes two concessions to the fact that an array-based implementation is used. First, heap nodes are indicated by their logical position within the heap rather than by a pointer to the node. In practice, the logical heap position corresponds to the identically numbered physical position in the array. Second, the constructor takes as input a pointer to the array to be used. This approach provides the greatest flexibility for using the heap because all data values can be loaded into the array directly by the client. The advantage of this comes during the heap construction phase, as explained below. The constructor also takes an integer parameter indicating the initial size of the heap (based on the number of elements initially loaded into the array) and a second integer parameter indicating the maximum size allowed for the heap (the size of the array).

Method heapsize returns the current size of the heap. H.isLeaf(pos) returns TRUE if position pos is a leaf in heap H, and FALSE otherwise. Members leftchild, rightchild, and parent return the position (actually, the array index) for the left child, right child, and parent of the position passed, respectively.

One way to build a heap is to insert the elements one at a time. Method insert will insert a new element into the heap.

You might expect the heap insertion process to be similar to the insert function for a BST, starting at the root and working down through the heap. However, this approach is not likely to work because the heap must maintain the shape of a complete binary tree. Equivalently, if the heap takes up the first positions of its array prior to the call to insert, it must take up the first positions after. To accomplish this, insert first places at position of the array. Of course, is unlikely to be in the correct position. To move to the right place, it is compared to its parent’s value. If the value of is less than or equal to the value of its parent, then it is in the correct place and the insert routine is finished. If the value of is greater than that of its parent, then the two elements swap positions. From here, the process of comparing to its (current) parent continues until reaches its correct position.

Since the heap is a complete binary tree, its height is guaranteed to be the minimum possible. In particular, a heap containing nodes will have a height of . Intuitively, we can see that this must be true because each level that we add will slightly more than double the number of nodes in the tree (the th level has nodes, and the sum of the first levels is ). Starting at 1, we can double only times to reach a value of . To be precise, the height of a heap with nodes is .

Each call to insert takes time in the worst case, because the value being inserted can move at most the distance from the bottom of the tree to the top of the tree. Thus, to insert values into the heap, if we insert them one at a time, will take time in the worst case.

6.13.2. Building a Heap

If all values are available at the beginning of the building process, we can build the heap faster than just inserting the values into the heap one by one. Consider this example, with two possible ways to heapify an initial set of values in an array.

Figure 6.13.1: Two series of exchanges to build a max heap. (a) This heap is built by a series of nine exchanges in the order (4-2), (4-1), (2-1), (5-2), (5-4), (6-3), (6-5), (7-5), (7-6). (b) This heap is built by a series of four exchanges in the order (5-2), (7-3), (7-1), (6-1).

From this example, it is clear that the heap for any given set of numbers is not unique, and we see that some rearrangements of the input values require fewer exchanges than others to build the heap. So, how do we pick the best rearrangement?

One good algorithm stems from induction. Suppose that the left and right subtrees of the root are already heaps, and is the name of the element at the root. This situation is illustrated by this figure:

Figure 6.13.2: Final stage in the heap-building algorithm. Both subtrees of node are heaps. All that remains is to push down to its proper level in the heap.

In this case there are two possibilities.

1. has a value greater than or equal to its two children. In this case, construction is complete.
2. has a value less than one or both of its children.

should be exchanged with the child that has greater value. The result will be a heap, except that might still be less than one or both of its (new) children. In this case, we simply continue the process of “pushing down” until it reaches a level where it is greater than its children, or is a leaf node. This process is implemented by the private method siftdown.

This approach assumes that the subtrees are already heaps, suggesting that a complete algorithm can be obtained by visiting the nodes in some order such that the children of a node are visited before the node itself. One simple way to do this is simply to work from the high index of the array to the low index. Actually, the build process need not visit the leaf nodes (they can never move down because they are already at the bottom), so the building algorithm can start in the middle of the array, with the first internal node.

Here is a visualization of the heap build process.

Method buildHeap implements the building algorithm.

What is the cost of buildHeap? Clearly it is the sum of the costs for the calls to siftdown. Each siftdown operation can cost at most the number of levels it takes for the node being sifted to reach the bottom of the tree. In any complete tree, approximately half of the nodes are leaves and so cannot be moved downward at all. One quarter of the nodes are one level above the leaves, and so their elements can move down at most one level. At each step up the tree we get half the number of nodes as were at the previous level, and an additional height of one. The maximum sum of total distances that elements can go is therefore

The summation on the right is known to have a closed-form solution of approximately 2, so this algorithm takes time in the worst case. This is far better than building the heap one element at a time, which would cost in the worst case. It is also faster than the average-case time and worst-case time required to build the BST.

6.13.3. Removing from the heap or updating an object’s priority

Because the heap is levels deep, the cost of deleting the maximum element is in the average and worst cases.

For some applications, objects might get their priority modified. One solution in this case is to remove the object and reinsert it. To do this, the application needs to know the position of the object in the heap. Another option is to change the priority value of the object, and then update its position in the heap. Note that a remove operation implicitly has to do this anyway, since when the last element in the heap is swapped with the one being removed, that value might be either too small or too big for its new position. So we use a utility method called update in both the remove and modify methods to handle this process.

6.13.4. Priority Queues

The heap is a natural implementation for the priority queue discussed at the beginning of this section. Jobs can be added to the heap (using their priority value as the ordering key) when needed. Method removemax can be called whenever a new job is to be executed.

Some applications of priority queues require the ability to change the priority of an object already stored in the queue. This might require that the object’s position in the heap representation be updated. Unfortunately, a max heap is not efficient when searching for an arbitrary value; it is only good for finding the maximum value. However, if we already know the index for an object within the heap, it is a simple matter to update its priority (including changing its position to maintain the heap property) or remove it. The remove method takes as input the position of the node to be removed from the heap. A typical implementation for priority queues requiring updating of priorities will need to use an auxiliary data structure that supports efficient search for objects (such as a BST). Records in the auxiliary data structure will store the object’s heap index, so that the object’s priority can be updated. Priority queues can be helpful for solving graph problems such as single-source shortest paths and minimal-cost spanning tree.

For a story about Priority Queues and dragons, see Computational Fairy Tales: Stacks, Queues, Priority Queues, and the Prince’s Complaint Line.

6.14. Huffman Coding Trees

6.14.1. Huffman Coding Trees

One can often gain an improvement in space requirements in exchange for a penalty in running time. There are many situations where this is a desirable tradeoff. A typical example is storing files on disk. If the files are not actively used, the owner might wish to compress them to save space. Later, they can be uncompressed for use, which costs some time, but only once.

We often represent a set of items in a computer program by assigning a unique code to each item. For example, the standard ASCII coding scheme assigns a unique eight-bit value to each character. It takes a certain minimum number of bits to provide enough unique codes so that we have a different one for each character. For example, it takes or seven bits to provide the 128 unique codes needed to represent the 128 symbols of the ASCII character set. [^1]

The requirement for bits to represent unique code values assumes that all codes will be the same length, as are ASCII codes. These are called fixed-length codes. If all characters were used equally often, then a fixed-length coding scheme is the most space efficient method. However, you are probably aware that not all characters are used equally often in many applications. For example, the various letters in an English language document have greatly different frequencies of use.

Table 6.14.1 shows the relative frequencies of the letters of the alphabet. From this table we can see that the letter ‘E’ appears about 60 times more often than the letter ‘Z’. In normal ASCII, the words “DEED” and “MUCK” require the same amount of space (four bytes). It would seem that words such as “DEED”, which are composed of relatively common letters, should be storable in less space than words such as “MUCK”, which are composed of relatively uncommon letters.

Table 6.14.1

Relative frequencies for the 26 letters of the alphabet as they appear in a selected set of English documents. “Frequency” represents the expected frequency of occurrence per 1000 letters, ignoring case.

If some characters are used more frequently than others, is it possible to take advantage of this fact and somehow assign them shorter codes? The price could be that other characters require longer codes, but this might be worthwhile if such characters appear rarely enough. This concept is at the heart of file compression techniques in common use today. The next section presents one such approach to assigning variable-length codes, called Huffman coding. While it is not commonly used in its simplest form for file compression (there are better methods), Huffman coding gives the flavor of such coding schemes. One motivation for studying Huffman coding is because it provides our first opportunity to see a type of tree structure referred to as a search trie.

[^1]: To keep things simple, these examples for building Huffman trees uses a sorted list to keep the partial Huffman trees ordered by frequency. But a real implementation would use a heap to implement a priority queue keyed by the frequencies.

6.14.1.1. Building Huffman Coding Trees

Huffman coding assigns codes to characters such that the length of the code depends on the relative frequency or weight of the corresponding character. Thus, it is a variable-length code. If the estimated frequencies for letters match the actual frequency found in an encoded message, then the length of that message will typically be less than if a fixed-length code had been used. The Huffman code for each letter is derived from a full binary tree called the Huffman coding tree, or simply the Huffman tree. Each leaf of the Huffman tree corresponds to a letter, and we define the weight of the leaf node to be the weight (frequency) of its associated letter. The goal is to build a tree with the minimum external path weight. Define the weighted path length of a leaf to be its weight times its depth. The binary tree with minimum external path weight is the one with the minimum sum of weighted path lengths for the given set of leaves. A letter with high weight should have low depth, so that it will count the least against the total path length. As a result, another letter might be pushed deeper in the tree if it has less weight.

The process of building the Huffman tree for letters is quite simple. First, create a collection of initial Huffman trees, each of which is a single leaf node containing one of the letters. Put the partial trees onto a priority queue organized by weight (frequency). Next, remove the first two trees (the ones with lowest weight) from the priority queue. Join these two trees together to create a new tree whose root has the two trees as children, and whose weight is the sum of the weights of the two trees. Put this new tree back into the priority queue. This process is repeated until all of the partial Huffman trees have been combined into one.

Table 6.14.2

The relative frequencies for eight selected letters.

The following slideshow illustrates the Huffman tree construction process for the eight letters of Table 6.14.2. [^2]

Here is the implementation for Huffman tree nodes.

/** Huffman tree node implementation: Base class */
interface HuffBaseNode {
  boolean isLeaf();
  int weight();
}

/** Huffman tree node: Leaf class */
class HuffLeafNode implements HuffBaseNode {
  private char element;      // Element for this node
  private int weight;        // Weight for this node

  /** Constructor */
  HuffLeafNode(char el, int wt)
    { element = el; weight = wt; }

  /** @return The element value */
  char value() { return element; }

  /** @return The weight */
  int weight() { return weight; }

  /** Return true */
  boolean isLeaf() { return true; }
}

/** Huffman tree node: Internal class */
class HuffInternalNode implements HuffBaseNode {
  private int weight;
  private HuffBaseNode left;
  private HuffBaseNode right;

  /** Constructor */
  HuffInternalNode(HuffBaseNode l,
                          HuffBaseNode r, int wt)
    { left = l; right = r; weight = wt; }

  /** @return The left child */
  HuffBaseNode left() { return left; }

  /** @return The right child */
  HuffBaseNode right() { return right; }

  /** @return The weight */
  int weight() { return weight; }

  /** Return false */
  boolean isLeaf() { return false; }
}

This implementation is similar to a typical class hierarchy for implementing full binary trees. There is an abstract base class, named HuffNode, and two subclasses, named LeafNode and IntlNode. This implementation reflects the fact that leaf and internal nodes contain distinctly different information.

Here is the implementation for the Huffman Tree class.

/** A Huffman coding tree */
class HuffTree implements Comparable {
  private HuffBaseNode root;

  /** Constructors */
  HuffTree(char el, int wt)
    { root = new HuffLeafNode(el, wt); }
  HuffTree(HuffBaseNode l, HuffBaseNode r, int wt)
    { root = new HuffInternalNode(l, r, wt); }

  HuffBaseNode root() { return root; }
  int weight() // Weight of tree is weight of root
    { return root.weight(); }
  int compareTo(Object t) {
    HuffTree that = (HuffTree)t;
    if (root.weight() < that.weight()) return -1;
    else if (root.weight() == that.weight()) return 0;
    else return 1;
  }
}

Here is the implementation for the tree-building process.

static HuffTree buildTree() {
  HuffTree tmp1, tmp2, tmp3 = null;

  while (Hheap.heapsize() > 1) { // While two items left
    tmp1 = Hheap.removemin();
    tmp2 = Hheap.removemin();
    tmp3 = new HuffTree(tmp1.root(), tmp2.root(),
                             tmp1.weight() + tmp2.weight());
    Hheap.insert(tmp3);   // Return new tree to heap
  }
  return tmp3;            // Return the tree
}

buildHuff takes as input fl, the min-heap of partial Huffman trees, which initially are single leaf nodes as shown in Step 1 of the slideshow above. The body of function buildTree consists mainly of a for loop. On each iteration of the for loop, the first two partial trees are taken off the heap and placed in variables temp1 and temp2. A tree is created (temp3) such that the left and right subtrees are temp1 and temp2, respectively. Finally, temp3 is returned to fl.

[^2]: ASCII coding actually uses 8 bits per character. Seven bits are used to represent the 128 codes of the ASCII character set. The eigth bit as a parity bit, that can be used to check if there is a transmission error for the character.

Assigning and Using Huffman Codes

Once the Huffman tree has been constructed, it is an easy matter to assign codes to individual letters. Beginning at the root, we assign either a ‘0’ or a ‘1’ to each edge in the tree. ‘0’ is assigned to edges connecting a node with its left child, and ‘1’ to edges connecting a node with its right child. This process is illustrated by the following slideshow.

Now that we see how the edges associate with bits in the code, it is a simple matter to generate the codes for each letter (since each letter corresponds to a leaf node in the tree).

Now that we have a code for each letter, encoding a text message is done by replacing each letter of the message with its binary code. A lookup table can be used for this purpose.

6.14.1.2. Decoding

A set of codes is said to meet the prefix property if no code in the set is the prefix of another. The prefix property guarantees that there will be no ambiguity in how a bit string is decoded. In other words, once we reach the last bit of a code during the decoding process, we know which letter it is the code for. Huffman codes certainly have the prefix property because any prefix for a code would correspond to an internal node, while all codes correspond to leaf nodes.

When we decode a character using the Huffman coding tree, we follow a path through the tree dictated by the bits in the code string. Each ‘0’ bit indicates a left branch while each ‘1’ bit indicates a right branch. The following slideshow shows an example for how to decode a message by traversing the tree appropriately.

6.14.1.3. How efficient is Huffman coding?

In theory, Huffman coding is an optimal coding method whenever the true frequencies are known, and the frequency of a letter is independent of the context of that letter in the message. In practice, the frequencies of letters in an English text document do change depending on context. For example, while E is the most commonly used letter of the alphabet in English documents, T is more common as the first letter of a word. This is why most commercial compression utilities do not use Huffman coding as their primary coding method, but instead use techniques that take advantage of the context for the letters.

Another factor that affects the compression efficiency of Huffman coding is the relative frequencies of the letters. Some frequency patterns will save no space as compared to fixed-length codes; others can result in great compression. In general, Huffman coding does better when there is large variation in the frequencies of letters.

Example 6.14.1

In the particular case of the frequencies shown in Table 6.14.1, we can determine the expected savings from Huffman coding if the actual frequencies of a coded message match the expected frequencies. Because the sum of the frequencies is 306 and E has frequency 120, we expect it to appear 120 times in a message containing 306 letters. An actual message might or might not meet this expectation. Letters D, L, and U have code lengths of three, and together are expected to appear 121 times in 306 letters. Letter C has a code length of four, and is expected to appear 32 times in 306 letters. Letter M has a code length of five, and is expected to appear 24 times in 306 letters. Finally, letters K and Z have code lengths of six, and together are expected to appear only 9 times in 306 letters. The average expected cost per character is simply the sum of the cost for each character () times the probability of its occurring (), or This can be reorganized as , where is the (relative) frequency of letter and is the total for all letter frequencies. For this set of frequencies, the expected cost per letter is

A fixed-length code for these eight characters would require bits per letter as opposed to about 2.57 bits per letter for Huffman coding. Thus, Huffman coding is expected to save about 14% for this set of letters.

Huffman coding for all ASCII symbols should do better than this example. The letters of Table 6.14.1 are atypical in that there are too many common letters compared to the number of rare letters. Huffman coding for all 26 letters would yield an expected cost of 4.29 bits per letter. The equivalent fixed-length code would require about five bits. This is somewhat unfair to fixed-length coding because there is actually room for 32 codes in five bits, but only 26 letters. More generally, Huffman coding of a typical text file will save around 40% over ASCII coding if we charge ASCII coding at eight bits per character. Huffman coding for a binary file (such as a compiled executable) would have a very different set of distribution frequencies and so would have a different space savings. Most commercial compression programs use two or three coding schemes to adjust to different types of files.

In decoding example, “DEED” was coded in 8 bits, a saving of 33% over the twelve bits required from a fixed-length coding. However, “MUCK” would require 18 bits, more space than required by the corresponding fixed-length coding. The problem is that “MUCK” is composed of letters that are not expected to occur often. If the message does not match the expected frequencies of the letters, than the length of the encoding will not be as expected either.

6.15. Trees versus Tries

6.15.1. Trees versus Tries

We see that all letters with codes beginning with ‘0’ are stored in the left branch, while all letters with codes beginning with ‘1’ are stored in the right branch. Contrast this with storing records in a BST. There, all records with key value less than the root value are stored in the left branch, while all records with key values greater than the root are stored in the right branch.

Recall that the Huffman coding tree stored in the left branch all letters whose codes start with 0, and in the right branch all letters whose codes start with 1. We can use this same concept to store records in a search tree that is slightly different from the behavior of a BST. We can view all keys stored as appearing on a numberline. The BST splits the numberline based on the positions of key values as it receives them. In contrast, we could split key values based on their binary reprsentation similar to what the Huffman coding tree does. The following slideshows present this in more detail.

6.16. Proof of Optimality for Huffman Coding

6.16.1. Proof of Optimality for Huffman Coding

Huffman tree building is an example of a greedy algorithm. At each step, the algorithm makes a “greedy” decision to merge the two subtrees with least weight. This makes the algorithm simple, but does it give the desired result? This section concludes with a proof that the Huffman tree indeed gives the most efficient arrangement for the set of letters. The proof requires the following lemma.

Lemma: For any Huffman tree built by function buildHuff containing at least two letters, the two letters with least frequency are stored in sibling nodes whose depth is at least as deep as any other leaf nodes in the tree.

Proof: Call the two letters with least frequency and . They must be siblings because buildHuff selects them in the first step of the construction process. Assume that and are not the deepest nodes in the tree. In this case, the Huffman tree must either look as shown in Figure 6.16.1, or effectively symmetrical to this. For this situation to occur, the parent of and , labeled , must have greater weight than the node labeled . Otherwise, function buildHuff would have selected node in place of node as the child of node . However, this is impossible because and are the letters with least frequency.

Figure 6.16.1: An impossible Huffman tree, showing the situation where the two nodes with least weight, and , are not the deepest nodes in the tree. Triangles represent subtrees.

Here is the proof.

Theorem: Function buildHuff builds the Huffman tree with the minimum external path weight for the given set of letters.

Proof: The proof is by induction on , the number of letters.

• Base Case: For , the Huffman tree must have the minimum external path weight because there are only two possible trees, each with identical weighted path lengths for the two leaves.
• Induction Hypothesis: Assume that any tree created by buildHuff that contains leaves has minimum external path length.
• Induction Step: Given a Huffman tree built by buildHuff with leaves, , suppose that where to are the weights of the letters. Call the parent of the letters with frequencies and . From the lemma, we know that the leaf nodes containing the letters with frequencies and are as deep as any nodes in . If any other leaf nodes in the tree were deeper, we could reduce their weighted path length by swapping them with or . But the lemma tells us that no such deeper nodes exist. Call the Huffman tree that is identical to except that node is replaced with a leaf node whose weight is . By the induction hypothesis, has minimum external path length. Returning the children to restores tree , which must also have minimum external path length.

Thus by mathematical induction, function buildHuff creates the Huffman tree with minimum external path length.

6.17. Binary Tree Chapter Summary

Chapter 7: Balanced Trees

7.1. Balanced Trees

The Binary Search Tree has a serious deficiency for practical use as a search structure. That is the fact that it can easily become unbalanced, so that some nodes are deep in the tree. In fact, it is possible for a BST with nodes to have a depth of , making it no faster to search in the worst case than a linked list. If we could keep the tree balanced in some way, then search cost would only be , a huge improvement.

One solution to this problem is to adopt another search tree structure instead of using a BST at all. An example of such an alternative tree structure is the 2-3 Tree or the B-Tree. But another alternative would be to modify the BST access functions in some way to guarantee that the tree performs well. This is an appealing concept, and the concept works well for heaps, whose access functions maintain the heap in the shape of a complete binary tree. Unfortunately, the heap keeps its balanced shape at the cost of weaker restrictions on the relative values of a node and its children, making it a bad search structure. And requiring that the BST always be in the shape of a complete binary tree requires excessive modification to the tree during update, as we see in this example.

Figure 7.1.1: An attempt to re-balance a BST after insertion can be expensive. (a) A BST with six nodes in the shape of a complete binary tree. (b) A node with value 1 is inserted into the BST of (a). To maintain both the complete binary tree shape and the BST property, a major reorganization of the tree is required.

If we are willing to weaken the balance requirements, we can come up with alternative update routines that perform well both in terms of cost for the update and in balance for the resulting tree structure. The AVL tree works in this way, using insertion and deletion routines altered from those of the BST to ensure that, for every node, the depths of the left and right subtrees differ by at most one.

A different approach to improving the performance of the BST is to not require that the tree always be balanced, but rather to expend some effort toward making the BST more balanced every time it is accessed. This is a little like the idea of path compression used by the UNION/FIND algorithm. One example of such a compromise is called the splay tree.

The Red-Black Tree is also a binary tree, but it uses a different balancing mechanism.

7.2. The AVL Tree

The AVL tree (named for its inventors Adelson-Velskii and Landis) should be viewed as a BST with the following additional property: For every node, the heights of its left and right subtrees differ by at most 1. As long as the tree maintains this property, if the tree contains nodes, then it has a depth of at most . As a result, search for any node will cost , and if the updates can be done in time proportional to the depth of the node inserted or deleted, then updates will also cost , even in the worst case.

The key to making the AVL tree work is to alter the insert and delete routines so as to maintain the balance property. Of course, to be practical, we must be able to implement the revised update routines in time.

Figure 7.2.1: Example of an insert operation that violates the AVL tree balance property. Prior to the insert operation, all nodes of the tree are balanced (i.e., the depths of the left and right subtrees for every node differ by at most one). After inserting the node with value 5, the nodes with values 7 and 24 are no longer balanced.

Consider what happens when we insert a node with key value 5, as shown in Figure 7.2.1. The tree on the left meets the AVL tree balance requirements. After the insertion, two nodes no longer meet the requirements. Because the original tree met the balance requirement, nodes in the new tree can only be unbalanced by a difference of at most 2 in the subtrees. For the bottommost unbalanced node, call it , there are 4 cases:

1. The extra node is in the left child of the left child of .
2. The extra node is in the right child of the left child of .
3. The extra node is in the left child of the right child of .
4. The extra node is in the right child of the right child of .

Cases 1 and 4 are symmetrical, as are cases 2 and 3. Note also that the unbalanced nodes must be on the path from the root to the newly inserted node.

Our problem now is how to balance the tree in time. It turns out that we can do this using a series of local operations known as rotations. Cases 1 and 4 can be fixed using a single rotation, as shown in Figure 7.2.2. Cases 2 and 3 can be fixed using a double rotation, as shown in Figure 7.2.3.

Figure 7.2.2: A single rotation in an AVL tree. This operation occurs when the excess node (in subtree ) is in the left child of the left child of the unbalanced node labeled . By rearranging the nodes as shown, we preserve the BST property, as well as re-balance the tree to preserve the AVL tree balance property. The case where the excess node is in the right child of the right child of the unbalanced node is handled in the same way.

Figure 7.2.3: A double rotation in an AVL tree. This operation occurs when the excess node (in subtree ) is in the right child of the left child of the unbalanced node labeled . By rearranging the nodes as shown, we preserve the BST property, as well as re-balance the tree to preserve the AVL tree balance property. The case where the excess node is in the left child of the right child of is handled in the same way.

The AVL tree insert algorithm begins with a normal BST insert. Then as the recursion unwinds up the tree, we perform the appropriate rotation on any node that is found to be unbalanced. Deletion is similar; however, consideration for unbalanced nodes must begin at the level of the deletemin operation.

Example 7.2.1

In Figure 7.2.1 (b), the bottom-most unbalanced node has value 7. The excess node (with value 5) is in the right subtree of the left child of 7, so we have an example of Case 2. This requires a double rotation to fix. After the rotation, 5 becomes the left child of 24, 2 becomes the left child of 5, and 7 becomes the right child of 5.

7.3. The Splay Tree

Like the AVL tree, the splay tree is not actually a distinct data structure, but rather reimplements the BST insert, delete, and search methods to improve the performance of a BST. The goal of these revised methods is to provide guarantees on the time required by a series of operations, thereby avoiding the worst-case linear time behavior of standard BST operations. No single operation in the splay tree is guaranteed to be efficient. Instead, the splay tree access rules guarantee that a series of operations will take time for a tree of nodes whenever . Thus, a single insert or search operation could take time. However, such operations are guaranteed to require a total of time, for an average cost of per access operation. This is a desirable performance guarantee for any search-tree structure.

Unlike the AVL tree, the splay tree is not guaranteed to be height balanced. What is guaranteed is that the total cost of the entire series of accesses will be cheap. Ultimately, it is the cost of the series of operations that matters, not whether the tree is balanced. Maintaining balance is really done only for the sake of reaching this time efficiency goal.

The splay tree access functions operate in a manner reminiscent of the move-to-front rule for self-organizing lists, and of the path compression technique for managing a series of Union/Find operations. These access functions tend to make the tree more balanced, but an individual access will not necessarily result in a more balanced tree.

Whenever a node is accessed (e.g., when is inserted, deleted, or is the goal of a search), the splay tree performs a process called splaying. Splaying moves to the root of the BST. When is being deleted, splaying moves the parent of to the root. As in the AVL tree, a splay of node consists of a series of rotations. A rotation moves higher in the tree by adjusting its position with respect to its parent and grandparent. A side effect of the rotations is a tendency to balance the tree. There are three types of rotation.

A single rotation is performed only if is a child of the root node. The single rotation is illustrated by Figure 7.3.1. It basically switches with its parent in a way that retains the BST property. While Figure 7.3.1 is slightly different from Figure 7.2.2, in fact the splay tree single rotation is identical to the AVL tree single rotation.

Figure 7.3.1: Splay tree single rotation. This rotation takes place only when the node being splayed is a child of the root. Here, node is promoted to the root, rotating with node . Because the value of is less than the value of , must become ‘s right child. The positions of subtrees , , and are altered as appropriate to maintain the BST property, but the contents of these subtrees remains unchanged. (a) The original tree with as the parent. (b) The tree after a rotation takes place. Performing a single rotation a second time will return the tree to its original shape. Equivalently, if (b) is the initial configuration of the tree (i.e., is at the root and is its right child), then (a) shows the result of a single rotation to splay to the root.

Unlike the AVL tree, the splay tree requires two types of double rotation. Double rotations involve , its parent (call it ), and ‘s grandparent (call it ). The effect of a double rotation is to move up two levels in the tree.

The first double rotation is called a . It takes place when either of the following two conditions are met:

1. is the left child of , and is the right child of .
2. is the right child of , and is the left child of .

In other words, a zigzag rotation is used when , , and form a zigzag. The zigzag rotation is illustrated by Figure 7.3.2.

Figure 7.3.2: Splay tree zigzag rotation. (a) The original tree with , , and in zigzag formation. (b) The tree after the rotation takes place. The positions of subtrees , , , and are altered as appropriate to maintain the BST property.

The other double rotation is known as a zigzig rotation. A zigzig rotation takes place when either of the following two conditions are met:

1. is the left child of , which is in turn the left child of .
2. is the right child of , which is in turn the right child of .

Thus, a zigzig rotation takes place in those situations where a zigzag rotation is not appropriate. The zigzig rotation is illustrated by Figure 7.3.3. While Figure 7.3.3 appears somewhat different from Figure 7.2.3, in fact the zigzig rotation is identical to the AVL tree double rotation.

Figure 7.3.3: Splay tree zigzig rotation. (a) The original tree with , , and in zigzig formation. (b) The tree after the rotation takes place. The positions of subtrees , , , and are altered as appropriate to maintain the BST property.

Note that zigzag rotations tend to make the tree more balanced, because they bring subtrees and up one level while moving subtree down one level. The result is often a reduction of the tree’s height by one. Zigzig promotions and single rotations do not typically reduce the height of the tree; they merely bring the newly accessed record toward the root.

Splaying node involves a series of double rotations until reaches either the root or the child of the root. Then, if necessary, a single rotation makes the root. This process tends to re-balance the tree. Regardless of balance, splaying will make frequently accessed nodes stay near the top of the tree, resulting in reduced access cost. Proof that the splay tree meets the guarantee of is beyond the scope of our study.

Example 7.3.1

Consider a search for value 89 in the splay tree of Figure 7.3.4 (a). The splay tree’s search operation is identical to searching in a BST. However, once the value has been found, it is splayed to the root. Three rotations are required in this example. The first is a zigzig rotation, whose result is shown in Figure 7.3.4 (b). The second is a zigzag rotation, whose result is shown in Figure 7.3.4 (c). The final step is a single rotation resulting in the tree of Figure 7.3.4 (d). Notice that the splaying process has made the tree shallower.

Figure 7.3.4: Example of splaying after performing a search in a splay tree. After finding the node with key value 89, that node is splayed to the root by performing three rotations. (a) The original splay tree. (b) The result of performing a zigzig rotation on the node with key value 89 in the tree of (a). (c) The result of performing a zigzag rotation on the node with key value 89 in the tree of (b). (d) The result of performing a single rotation on the node with key value 89 in the tree of (c). If the search had been for 91, the search would have been unsuccessful with the node storing key value 89 being that last one visited. In that case, the same splay operations would take place.

Chapter 8: General Trees

8.2. Union/Find and the Parent Pointer Implementation

8.2.1. The Union/Find Problem

General trees are trees whose internal nodes have no fixed number of children. Compared to general trees, binary trees are relatively easy to implement because each internal node of a binary tree can just store two pointers to reach its (potential) children. In a general tree, we have to deal with the fact that a given node might have no children or few children or many children.

Even in a general tree, each node can have only one parent. If we didn’t need to go from a node to its children, but instead only needed to go from a node to its parent, then implementing a node would be easy. A simple way to represent such a general tree would be to store for each node only a pointer to that node’s parent. We will call this the parent pointer representation for general trees. Clearly this implementation is not general purpose, because it is inadequate for such important operations as finding the leftmost child or the right sibling for a node. Thus, it may seem to be a poor idea to implement a general tree in this way. However, the parent pointer implementation stores precisely the information required to answer the following, useful question: Given two nodes, are they in the same tree? To answer this question, we need only follow the series of parent pointers from each node to its respective root. If both nodes reach the same root, then they must be in the same tree. If the roots are different, then the two nodes are not in the same tree. The process of finding the ultimate root for a given node we will call FIND.

8.2.1.1. Parent Pointer Trees

The parent pointer representation is most often used to maintain a collection of disjoint sets. Two disjoint sets share no members in common (their intersection is empty). A collection of disjoint sets partitions some objects such that every object is in exactly one of the disjoint sets. There are two basic operations that we wish to support:

1. Determine if two objects are in the same set (the FIND operation), and
2. Merge two sets together.

Because two merged sets are united, the merging operation is called UNION and the whole process of determining if two objects are in the same set and then merging the sets goes by the name UNION/FIND.

To implement UNION/FIND, we represent each disjoint set with a separate general tree. Two objects are in the same disjoint set if they are in the same tree. Every node of the tree (except for the root) has precisely one parent. Thus, each node requires the same space to represent it. The collection of objects is typically stored in an array, where each element of the array corresponds to one object, and each element stores the object’s value (or a pointer to the object). The objects also correspond to nodes in the various disjoint trees (one tree for each disjoint set), so we also store the parent value with each object in the array. Those nodes that are the roots of their respective trees store an appropriate indicator. Note that this representation means that a single array is being used to implement a collection of trees. This makes it easy to merge trees together with UNION operations.

Here is an implementation for parent pointer trees and the UNION/FIND process.

// General Tree implementation for UNION/FIND
class ParPtrTree {
  private int[] array;     // Node array

  ParPtrTree(int size) {
    array = new int[size]; // Create node array
    for (int i=0; i<size; i++)
      array[i] = -1;       // Each node is its own root to start
  }

  // Merge two subtrees if they are different
  void UNION(int a, int b) {
    int root1 = FIND(a);     // Find root of node a
    int root2 = FIND(b);     // Find root of node b
    if (root1 != root2)          // Merge two trees
      array[root1] = root2;
  }

  // Return the root of curr's tree
  int FIND(int curr) {
    while (array[curr] != -1)
      curr = array[curr];
    return curr; // Now at root
  }
}

The ParPtrTree class has an array where each array position corresponds to one object in some collection. Each array element stores the array index for its parent. There are two main methods to implement. Method UNION merges two sets together, where each set corresponds to a tree. Method FIND is used to find the ultimate root for a node.

An application using the UNION/FIND operations should store a set of objects, where each object is assigned a unique index in the range 0 to . The indices refer to the corresponding parent pointers in the array. Class ParPtrTree creates and initializes the UNION/FIND array, and methods UNION and FIND take array indices as inputs.

Figure 8.2.1: The parent pointer array implementation. Each node corresponds to a position in the node array, which stores its value and a pointer to its parent. The parent pointers are represented by an array index corresponding to the position of the parent. The root of any tree stores a special value, such as -1. This is represented graphically in the figure by a slash in the “Parent’s Index” box. This figure shows two trees stored in the same parent pointer array, one rooted at (with a total of 9 nodes), and the other rooted at (with a total of 1 node).

8.2.1.2. Equivalence Classes

Consider the problem of assigning the members of a set to disjoint subsets called equivalence classes. Recall that an equivalence relation is reflexive, symmetric, and transitive. Thus, if objects and are equivalent, and objects and are equivalent, then we must be able to recognize that objects and are also equivalent. In this representation, since and are equivalent, they must be in the same tree. Likewise for and . We can recognize that and are equivalent because they must also be in the same tree.

There are many practical uses for disjoint sets and representing equivalences. For example, consider this graph of ten nodes labeled through .

Figure 8.2.2: A graph with two connected components. The tree of Figure 8.2.1 shows the corresponding tree structure resulting form processing the edges to determine the connected components.

Notice that for nodes through , there is some series of edges that connects any pair of these nodes, but node is disconnected from the rest of the nodes. Such a graph might be used to represent connections such as wires between components on a circuit board, or roads between cities. We can consider two nodes of the graph to be equivalent if there is a path between them. Thus, nodes , , and would be considered as equivalent, but is not equivalent to any other. A subset of equivalent (connected) edges in a graph is called a connected component. The goal is to quickly classify the objects into disjoint sets that correspond to the connected components.

Another use for UNION/FIND occurs in Kruskal’s algorithm for computing the minimal-cost spanning tree for a graph. That algorithm seeks to select the cheapest subset of the edges that still connects all of the nodes in the graph. It does so by processing all edges of the graph from shortest to longest, only adding an edge to the connecting subset if it does not connect two nodes that already have some series of edges connecting them.

The input to the UNION/FIND algorithm is typically a series of equivalence pairs. In the case of the connected components example, the equivalence pairs would simply be the set of edges in the graph. An equivalence pair might say that object is equivalent to object . If so, and are placed in the same subset. If a later equivalence relates and , then by implication is also equivalent to . Thus, an equivalence pair may cause two subsets to merge, each of which contains several objects.

Equivalence classes can be managed efficiently with the UNION/FIND algorithm. Initially, each object is at the root of its own tree. An equivalence pair is processed by checking to see if both objects of the pair are in the same tree by calling FIND on each of them. If their roots are the same, then no change need be made because the objects are already in the same equivalence class. Otherwise, the two equivalence classes should be merged by the UNION method.

The parent pointer representation places no limit on the number of nodes that can share a parent. To make equivalence processing as efficient as possible, the distance from each node to the root of its respective tree should be as small as possible. Thus, we would like to keep the height of the trees small when merging two equivalence classes together. Ideally, each tree would have all nodes pointing directly to the root. Achieving this goal all the time would require too much additional processing to be worth the effort, so we must settle for getting as close as possible.

8.2.1.3. Weighted Union

A low-cost approach to reducing the height is to be smart about how two trees are joined together. One simple technique, called the weighted union rule, joins the tree with fewer nodes to the tree with more nodes by making the smaller tree’s root point to the root of the bigger tree. This will limit the total depth of the tree to , because the depth of nodes only in the smaller tree will now increase by one, and the depth of the deepest node in the combined tree can only be at most one deeper than the deepest node before the trees were combined. The total number of nodes in the combined tree is therefore at least twice the number in the smaller subtree. Thus, the depth of any node can be increased at most times when equivalences are processed (since each addition to the depth must be accompanied by at least doubling the size of the tree).

Here is an implementation for the UNION method when using weighted union.

void UNION(int a, int b) {
  int root1 = FIND(a);     // Find root of node a
  int root2 = FIND(b);     // Find root of node b
  if (root1 != root2)          // Merge with weighted union
    if (weights[root2] > weights[root1]) {
      array[root1] = root2;
      weights[root2] += weights[root1];
    } else {
      array[root2] = root1;
      weights[root1] += weights[root2];
    }
}

The following slideshow illustrates a series of UNION operations with weighted union.

8.2.1.4. Path Compression

The weighted union rule helps to minimize the depth of the tree, but we can do better than this. Path compression is a method that tends to create extremely shallow trees. Path compression takes place while finding the root for a given node . Call this root . Path compression resets the parent of every node on the path from to to point directly to . This can be implemented by first finding . A second pass is then made along the path from to , assigning the parent field of each node encountered to . Alternatively, a recursive algorithm can be implemented as follows. This version of FIND not only returns the root of the current node, but also makes all ancestors of the current node point to the root.

// Return the root of curr's tree with path compression
int FIND(int curr) {
  if (array[curr] == -1) return curr; // At root
  array[curr] = FIND(array[curr]);
  return array[curr];
}

The following slide show illustrates path compression using the last step in the previous example.

Path compression keeps the cost of each FIND operation very close to constant.

To be more precise about what is meant by “very close to constant”, the cost of path compression for FIND operations on nodes (when combined with the weighted union rule for joining sets) is approximately . The notation means the number of times that the log of must be taken before . For example, is 4 because , and finally . Thus, grows very slowly, so the cost for a series of FIND operations is very close to .

Note that this does not mean that the tree resulting from processing equivalence pairs necessarily has depth . One can devise a series of equivalence operations that yields depth for the resulting tree. However, many of the equivalences in such a series will look only at the roots of the trees being merged, requiring little processing time. The total amount of processing time required for operations will be , yielding nearly constant time for each equivalence operation. This is an example of amortized analysis.

The expression is closely related to the inverse of Ackermann’s function. For more information about Ackermann’s function and the cost of path compression for UNION/FIND, see [Tarjan75]. The survey article by Galil & Italiano [GalilItaliano91] covers many aspects of the equivalence class problem.

8.3. Sequential Tree Representations

8.3.1. Sequential Tree Representations

Next we consider a fundamentally different approach to implementing trees. The goal is to store a series of node values with the minimum information needed to reconstruct the tree structure. This approach, known as a sequential tree representation, has the advantage of saving space because no pointers are stored. It has the disadvantage that accessing any node in the tree requires sequentially processing all nodes that appear before it in the node list. In other words, node access must start at the beginning of the node list, processing nodes sequentially in whatever order they are stored until the desired node is reached. Thus, one primary virtue of the other implementations discussed in this section is lost: efficient access (typically time) to arbitrary nodes in the tree. Sequential tree implementations are ideal for archiving trees on disk for later use because they save space, and the tree structure can be reconstructed as needed for later processing.

Sequential tree implementations can be used to serialize a tree structure. Serialization is the process of storing an object as a series of bytes, typically so that the data structure can be transmitted between computers. This capability is important when using data structures in a distributed processing environment.

A sequential tree implementation typically stores the node values as they would be enumerated by a preorder traversal, along with sufficient information to describe the tree’s shape. If the tree has restricted form, for example if it is a full binary tree, then less information about structure typically needs to be stored. A general tree, because it has the most flexible shape, tends to require the most additional shape information. There are many possible sequential tree implementation schemes. We will begin by describing methods appropriate to binary trees, then generalize to an implementation appropriate to a general tree structure.

Because every node of a binary tree is either a leaf or has two (possibly empty) children, we can take advantage of this fact to implicitly represent the tree’s structure. The most straightforward sequential tree implementation lists every node value as it would be enumerated by a preorder traversal. Unfortunately, the node values alone do not provide enough information to recover the shape of the tree. In particular, as we read the series of node values, we do not know when a leaf node has been reached. However, we can treat all non-empty nodes as internal nodes with two (possibly empty) children. Only NULL values will be interpreted as leaf nodes, and these can be listed explicitly. Such an augmented node list provides enough information to recover the tree structure.

Figure 8.3.1: Sample binary tree for sequential tree implementation examples.

8.3.2. Alternative Sequential Representation

To illustrate the difficulty involved in using the sequential tree representation for processing, consider searching for the right child of the root node. We must first move sequentially through the node list of the left subtree. Only at this point do we reach the value of the root’s right child. Clearly the sequential representation is space efficient, but not time efficient for descending through the tree along some arbitrary path.

Assume that each node value takes a constant amount of space. An example would be if the node value is a positive integer and null is indicated by the value zero. From the Full Binary Tree Theorem, we know that the size of the node list will be about twice the number of nodes (i.e., the overhead fraction is 1/2). The extra space is required by the null pointers. We should be able to store the node list more compactly. However, any sequential implementation must recognize when a leaf node has been reached, that is, a leaf node indicates the end of a subtree. One way to do this is to explicitly list with each node whether it is an internal node or a leaf. If a node is an internal node, then we know that its two children (which may be subtrees) immediately follow in the node list. If is a leaf node, then the next node in the list is the right child of some ancestor of , not the right child of . In particular, the next node will be the child of ‘s most recent ancestor that has not yet seen its right child. However, this assumes that each internal node does in fact have two children, in other words, that the tree is full. Empty children must be indicated in the node list explicitly. Assume that internal nodes are marked with a prime (‘) and that leaf nodes show no mark. Empty children of internal nodes are indicated by “/“, but the (empty) children of leaf nodes are not represented at all. Note that a full binary tree stores no null values with this implementation, and so requires less overhead.

Storing extra bits can be a considerable savings over storing null values. In the example above, each node was shown with a mark if it is internal, or no mark if it is a leaf. This requires that each node value has space to store the mark bit. This might be true if, for example, the node value were stored as a 4-byte integer but the range of the values sored was small enough so that not all bits are used. An example would be if all node values must be positive. Then the high-order (sign) bit of the integer value could be used as the mark bit.

8.3.3. Bit Vector Representation

Another approach is to store a separate bit vector to represent the status of each node. In this case, each node of the tree corresponds to one bit in the bit vector. A value of “1” could indicate an internal node, and “0” could indicate a leaf node.

8.3.4. General Tree Sequential Representation

Storing general trees by means of a sequential implementation requires that more explicit structural information be included with the node list. Not only must the general tree implementation indicate whether a node is leaf or internal, it must also indicate how many children the node has. Alternatively, the implementation can indicate when a node’s child list has come to an end. The next example dispenses with marks for internal or leaf nodes. Instead it includes a special mark (we will use the “)” symbol) to indicate the end of a child list. All leaf nodes are followed by a “)” symbol because they have no children. A leaf node that is also the last child for its parent would indicate this by two or more successive “)” symbols.

Note that this representation for serializing general trees cannot be used for binary trees. This is because a binary tree is not merely a restricted form of general tree with at most two children. Every binary tree node has a left and a right child, though either or both might be empty. So this representation cannot let us distinguish whether node in Figure 8.3.1 is the left or right child of node .

Chapter 9: Indexing

9.1. Indexing Chapter Introduction

Many large-scale computing applications are centered around data sets that are too large to fit into main memory. The classic example is a large database of records with multiple search keys, requiring the ability to insert, delete, and search for records. Hashing provides outstanding performance for such situations, but only in the limited case in which all searches are of the form “find the record with key value ”. Many applications require more general search capabilities. One example is a range query search for all records whose key lies within some range. Other queries might involve visiting all records in order of their key value, or finding the record with the greatest key value. Hash tables are not organized to support any of these queries efficiently.

This chapter introduces file structures used to organize a large collection of records stored on disk. Such file structures support efficient insertion, deletion, and search operations, for exact-match queries, range queries, and largest/smallest key value searches.

Before discussing such file structures, we must become familiar with some basic file-processing terminology. An entry-sequenced file stores records in the order that they were added to the file. Entry-sequenced files are the disk-based equivalent to an unsorted list and so do not support efficient search. The natural solution is to sort the records by order of the search key. However, a typical database, such as a collection of employee or customer records maintained by a business, might contain multiple search keys. To answer a question about a particular customer might require a search on the name of the customer. Businesses often wish to sort and output the records by zip code order for a bulk mailing. Government paperwork might require the ability to search by Social Security number. Thus, there might not be a single “correct” order in which to store the records.

Indexing is the process of associating a key with the location of a corresponding data record. An external sort typically uses the concept of a key sort, in which an index file is created whose records consist of key/pointer pairs. Here, each key is associated with a pointer to a complete record in the main database file. The index file could be sorted or organized using a tree structure, thereby imposing a logical order on the records without physically rearranging them. One database might have several associated index files, each supporting efficient access through a different key field.

Each record of a database normally has a unique identifier, called the primary key. For example, the primary key for a set of personnel records might be the Social Security number or ID number for the individual. Unfortunately, the ID number is generally an inconvenient value on which to perform a search because the searcher is unlikely to know it. Instead, the searcher might know the desired employee’s name. Alternatively, the searcher might be interested in finding all employees whose salary is in a certain range. If these are typical search requests to the database, then the name and salary fields deserve separate indices. However, key values in the name and salary indices are not likely to be unique.

A key field such as salary, where a particular key value might be duplicated in multiple records, is called a secondary key. Most searches are performed using a secondary key. The secondary key index (or more simply, secondary index) will associate a secondary key value with the primary key of each record having that secondary key value. At this point, the full database might be searched directly for the record with that primary key, or there might be a primary key index (or primary index) that relates each primary key value with a pointer to the actual record on disk. In the latter case, only the primary index provides the location of the actual record on disk, while the secondary indices refer to the primary index.

Indexing is an important technique for organizing large databases, and many indexing methods have been developed. Direct access through hashing is discussed in Chapter Hashing. A simple list sorted by key value can also serve as an index to the record file. Indexing disk files by sorted lists are discussed in the following section. Unfortunately, a sorted list does not perform well for insert and delete operations.

A third approach to indexing is the tree index. Trees are typically used to organize large databases that must support record insertion, deletion, and key range searches. ISAM was a a tentative step toward solving the problem of storing a large database that must support insertion and deletion of records. Its shortcomings help to illustrate the value of tree indexing techniques. Module TreeIndexing introduces the basic issues related to tree indexing. Module 2-3 tree introduces the 2-3 tree, a balanced tree structure that is a simple form of the B-tree. B-trees are the most widely used indexing method for large disk-based databases, and for implementing file systems.

9.2. Linear Indexing

9.2.1. Linear Indexing

A linear index is an index file organized as a sequence of key-value pairs where the keys are in sorted order and the pointers either (1) point to the position of the complete record on disk, (2) point to the position of the primary key in the primary index, or (3) are actually the value of the primary key. Depending on its size, a linear index might be stored in main memory or on disk. A linear index provides a number of advantages. It provides convenient access to variable-length database records, because each entry in the index file contains a fixed-length key field and a fixed-length pointer to the beginning of a (variable-length) record as shown in the following slideshow A linear index also allows for efficient search and random access to database records, because it is amenable to binary search.

If the database contains enough records, the linear index might be too large to store in main memory. This makes binary search of the index more expensive because many disk accesses would typically be required by the search process. One solution to this problem is to store a second-level linear index in main memory that indicates which disk block in the index file stores a desired key. For example, the linear index on disk might reside in a series of 1024-byte blocks. If each key/pointer pair in the linear index requires 8~bytes (a 4-byte key and a 4-byte pointer), then 128 key/pointer pairs are stored per block. The second-level index, stored in main memory, consists of a simple table storing the value of the key in the first position of each block in the linear index file. This arrangement is shown in the next slideshow. If the linear index requires 1024 disk blocks (1MB), the second-level index contains only 1024 entries, one per disk block.

To find which disk block contains a desired search key value, first search through the 1024-entry table to find the greatest value less than or equal to the search key. This directs the search to the proper block in the index file, which is then read into memory. At this point, a binary search within this block will produce a pointer to the actual record in the database. Because the second-level index is stored in main memory, accessing a record by this method requires two disk reads: one from the index file and one from the database file for the actual record.

A simple two-level linear index. The linear index is stored on disk. The smaller, second-level index is stored in main memory. Each element in the second-level index stores the first key value in the corresponding disk block of the index file. In this example, the first disk block of the linear index stores keys in the range 1 to 2001, and the second disk block stores keys in the range 2003 to 5688. Thus, the first entry of the second-level index is key value 1 (the first key in the first block of the linear index), while the second entry of the second-level index is key value 2003.

Every time a record is inserted to or deleted from the database, all associated secondary indices must be updated. Updates to a linear index are expensive, because the entire contents of the array might be shifted. Another problem is that multiple records with the same secondary key each duplicate that key value within the index. When the secondary key field has many duplicates, such as when it has a limited range (e.g., a field to indicate job category from among a small number of possible job categories), this duplication might waste considerable space.

One improvement on the simple sorted array is a two-dimensional array where each row corresponds to a secondary key value. A row contains the primary keys whose records have the indicated secondary key value. Figure 9.2.1 illustrates this approach. Now there is no duplication of secondary key values, possibly yielding a considerable space savings. The cost of insertion and deletion is reduced, because only one row of the table need be adjusted. Note that a new row is added to the array when a new secondary key value is added. This might lead to moving many records, but this will happen infrequently in applications suited to using this arrangement.

{width=40%}

Figure 9.2.1: A two-dimensional linear index. Each row lists the primary keys associated with a particular secondary key value. In this example, the secondary key is a name. The primary key is a unique four-character code.

A drawback to this approach is that the array must be of fixed size, which imposes an upper limit on the number of primary keys that might be associated with a particular secondary key. Furthermore, those secondary keys with fewer records than the width of the array will waste the remainder of their row. A better approach is to have a one-dimensional array of secondary key values, where each secondary key is associated with a linked list. This works well if the index is stored in main memory, but not so well when it is stored on disk because the linked list for a given key might be scattered across several disk blocks.

Consider a large database of employee records. If the primary key is the employee’s ID number and the secondary key is the employee’s name, then each record in the name index associates a name with one or more ID numbers. The ID number index in turn associates an ID number with a unique pointer to the full record on disk. The secondary key index in such an organization is also known as an inverted list or inverted file. It is inverted in that searches work backwards from the secondary key to the primary key to the actual data record. It is called a list because each secondary key value has (conceptually) a list of primary keys associated with it. Figure 9.2.2 illustrates this arrangement. Here, we have last names as the secondary key. The primary key is a four-character unique identifier.

{width=40%}

Figure 9.2.2: Illustration of an inverted list. Each secondary key value is stored in the secondary key list. Each secondary key value on the list has a pointer to a list of the primary keys whose associated records have that secondary key value.

Figure 9.2.3 shows a better approach to storing inverted lists. An array of secondary key values is shown as before. Associated with each secondary key is a pointer to an array of primary keys. The primary key array uses a linked-list implementation. This approach combines the storage for all of the secondary key lists into a single array, probably saving space. Each record in this array consists of a primary key value and a pointer to the next element on the list. It is easy to insert and delete secondary keys from this array, making this a good implementation for disk-based inverted files.

{width=40%}

Figure 9.2.3: An inverted list implemented as an array of secondary keys and combined lists of primary keys. Each record in the secondary key array contains a pointer to a record in the primary key array. The next field of the primary key array indicates the next record with that secondary key value.

9.3. ISAM

How do we handle large databases that require frequent update? The main problem with the linear index is that it is a single, large array that does not adjust well to updates because a single update can require changing the position of every key in the index. Inverted lists reduce this problem, but they are only suitable for secondary key indices with many fewer secondary key values than records. The linear index would perform well as a primary key index if it could somehow be broken into pieces such that individual updates affect only a part of the index. This concept will be pursued throughout the rest of this chapter, eventually culminating in the Tree the most widely used indexing method today. But first, we begin by studying ISAM, an early attempt to solve the problem of large databases requiring frequent update. Its weaknesses help to illustrate why the Tree works so well.

Before the invention of effective tree indexing schemes, a variety of disk-based indexing methods were in use. All were rather cumbersome, largely because no adequate method for handling updates was known. Typically, updates would cause the index to degrade in performance. ISAM is one example of such an index and was widely used by IBM prior to adoption of the B-tree.

{width=40%}

Figure 9.3.1: Illustration of the ISAM indexing system.

ISAM is based on a modified form of the linear index, as illustrated by Figure 9.3.1. Records are stored in sorted order by primary key. The disk file is divided among a number of cylinders on disk. Each cylinder holds a section of the list in sorted order. Initially, each cylinder is not filled to capacity, and the extra space is set aside in the cylinder overflow. In memory is a table listing the lowest key value stored in each cylinder of the file. Each cylinder contains a table listing the lowest key value for each block in that cylinder, called the cylinder index. When new records are inserted, they are placed in the correct cylinder’s overflow area (in effect, a cylinder acts as a bucket). If a cylinder’s overflow area fills completely, then a system-wide overflow area is used. Search proceeds by determining the proper cylinder from the system-wide table kept in main memory. The cylinder’s block table is brought in from disk and consulted to determine the correct block. If the record is found in that block, then the search is complete. Otherwise, the cylinder’s overflow area is searched. If that is full, and the record is not found, then the system-wide overflow is searched.

After initial construction of the database, so long as no new records are inserted or deleted, access is efficient because it requires only two disk fetches. The first disk fetch recovers the block table for the desired cylinder. The second disk fetch recovers the block that, under good conditions, contains the record. After many inserts, the overflow list becomes too long, resulting in significant search time as the cylinder overflow area fills up. Under extreme conditions, many searches might eventually lead to the system overflow area. The “solution” to this problem is to periodically reorganize the entire database. This means re-balancing the records among the cylinders, sorting the records within each cylinder, and updating both the system index table and the within-cylinder block table. Such reorganization was typical of database systems during the 1960s and would normally be done each night or weekly.

9.4. Tree-based Indexing

9.4.1. Tree-based Indexing

Linear indexing is efficient when the database is static, that is, when records are inserted and deleted rarely or never. ISAM is adequate for a limited number of updates, but not for frequent changes. Because it has essentially two levels of indexing, ISAM will also break down for a truly large database where the number of cylinders is too great for the top-level index to fit in main memory.

In their most general form, database applications have the following characteristics:

1. Large sets of records that are frequently updated.
2. Search is by one or a combination of several keys.
3. Key range queries or min/max queries are used.

For such databases, a better organization must be found. One approach would be to use the binary search tree (BST) to store primary and secondary key indices. BSTs can store duplicate key values, they provide efficient insertion and deletion as well as efficient search, and they can perform efficient range queries. When there is enough main memory, the BST is a viable option for implementing both primary and secondary key indices.

Unfortunately, the BST can become unbalanced. Even under relatively good conditions, the depth of leaf nodes can easily vary by a factor of two. This might not be a significant concern when the tree is stored in main memory because the time required is still for search and update. When the tree is stored on disk, however, the depth of nodes in the tree becomes crucial. Every time a BST node is visited, it is necessary to visit all nodes along the path from the root to . Each node on this path must be retrieved from disk. Each disk access returns a block of information. If a node is on the same block as its parent, then the cost to find that node is trivial once its parent is in main memory. Thus, it is desirable to keep subtrees together on the same block. Unfortunately, many times a node is not on the same block as its parent. Thus, each access to a BST node could potentially require that another block to be read from disk. Using a buffer pool to store multiple blocks in memory can mitigate disk access problems if BST accesses display good locality of reference. But a buffer pool cannot eliminate disk I/O entirely. The problem becomes greater if the BST is unbalanced, because nodes deep in the tree have the potential of causing many disk blocks to be read. Thus, there are two significant issues that must be addressed to have efficient search from a disk-based BST. The first is how to keep the tree balanced. The second is how to arrange the nodes on blocks so as to keep the number of blocks encountered on any path from the root to the leaves at a minimum.

We could select a scheme for balancing the BST and allocating BST nodes to blocks in a way that minimizes disk I/O, as illustrated by the first slideshow. However, maintaining such a scheme in the face of insertions and deletions is difficult. In particular, the tree should remain balanced when an update takes place, but doing so might require much reorganization. Each update should affect only a few blocks, or its cost will be too high.

As you can see from this slideshow, adopting a rule such as requiring the BST to be complete can cause a great deal of rearranging of data within the tree.

We can solve these problems by selecting another tree structure that automatically remains balanced after updates, and which is amenable to storing in blocks. There are a number of balanced tree data structures, and there are also techniques for keeping BSTs balanced. Examples are the AVL and splay trees. As an alternative, the 2-3 Tree has the property that its leaves are always at the same level. The main reason for discussing the 2-3 Tree here in preference to the other balanced search trees is that it naturally leads to the B-tree, which is by far the most widely used indexing method today.

9.5. 2-3 Trees

9.5.1. 2-3 Trees

This section presents a data structure called the 2-3 tree. The 2-3 tree is not a binary tree, but instead its shape obeys the following definition:

1. A node contains one or two keys.
2. Every internal node has either two children (if it contains one key) or three children (if it contains two keys). Hence the name.
3. All leaves are at the same level in the tree, so the tree is always height balanced.

In addition to these shape properties, the 2-3 tree has a search tree property analogous to that of a BST. For every node, the values of all descendants in the left subtree are less than the value of the first key, while values in the center subtree are greater than or equal to the value of the first key. If there is a right subtree (equivalently, if the node stores two keys), then the values of all descendants in the center subtree are less than the value of the second key, while values in the right subtree are greater than or equal to the value of the second key. To maintain these shape and search properties requires that special action be taken when nodes are inserted and deleted. The 2-3 tree has the advantage over the BST in that the 2-3 tree can be kept height balanced at relatively low cost. Here is an example 2-3 tree.

Figure 9.5.1: An example of a 2-3 tree.

Nodes are shown as rectangular boxes with two key fields. (These nodes actually would contain complete records or pointers to complete records, but the figures will show only the keys.) Internal nodes with only two children have an empty right key field. Leaf nodes might contain either one or two keys. Here is an implementation for the 2-3 tree node class.

// 2-3 tree node implementation
class TTNode<Key extends Comparable<? super Key>,E> {
  private E lval;        // The left record
  private Key lkey;        // The node's left key
  private E rval;        // The right record
  private Key rkey;        // The node's right key
  private TTNode<Key,E> left;   // Pointer to left child
  private TTNode<Key,E> center; // Pointer to middle child
  private TTNode<Key,E> right;  // Pointer to right child

  public TTNode() { center = left = right = null; }
  public TTNode(Key lk, E lv, Key rk, E rv,
                TTNode<Key,E> p1, TTNode<Key,E> p2,
                TTNode<Key,E> p3) {
    lkey = lk; rkey = rk;
    lval = lv; rval = rv;
    left = p1; center = p2; right = p3;
  }

  public boolean isLeaf() { return left == null; }
  public TTNode<Key,E> lchild() { return left; }
  public TTNode<Key,E> rchild() { return right; }
  public TTNode<Key,E> cchild() { return center; }
  public Key lkey() { return lkey; }  // Left key
  public E lval() { return lval; }  // Left value
  public Key rkey() { return rkey; }  // Right key
  public E rval() { return rval; }  // Right value
  public void setLeft(Key k, E e) { lkey = k; lval = e; }
  public void setRight(Key k, E e) { rkey = k; rval = e; }
  public void setLeftChild(TTNode<Key,E> it) { left = it; }
  public void setCenterChild(TTNode<Key,E> it)
    { center = it; }
  public void setRightChild(TTNode<Key,E> it)
    { right = it; }
}

Note that this sample declaration does not distinguish between leaf and internal nodes and so is space inefficient, because leaf nodes store three pointers each. We can use a class hierarcy to implement separate internal and leaf node types.

From the defining rules for 2-3 trees we can derive relationships between the number of nodes in the tree and the depth of the tree. A 2-3 tree of height has at least leaves, because if every internal node has two children it degenerates to the shape of a complete binary tree. A 2-3 tree of height has at most leaves, because each internal node can have at most three children.

Searching for a value in a 2-3 tree is similar to searching in a BST. Search begins at the root. If the root does not contain the search key , then the search progresses to the only subtree that can possibly contain . The value(s) stored in the root node determine which is the correct subtree. For example, if searching for the value 30 in the tree of Figure 9.5.1, we begin with the root node. Because 30 is between 18 and 33, it can only be in the middle subtree. Searching the middle child of the root node yields the desired record. If searching for 15, then the first step is again to search the root node. Because 15 is less than 18, the first (left) branch is taken. At the next level, we take the second branch to the leaf node containing 15. If the search key were 16, then upon encountering the leaf containing 15 we would find that the search key is not in the tree. Here is an implementation for the 2-3 tree search method.

private E findhelp(TTNode<Key,E> root, Key k) {
  if (root == null) return null;          // val not found
  if (k.compareTo(root.lkey()) == 0) return root.lval();
  if ((root.rkey() != null) && (k.compareTo(root.rkey())
       == 0))
    return root.rval();
  if (k.compareTo(root.lkey()) < 0)       // Search left
    return findhelp(root.lchild(), k);
  else if (root.rkey() == null)           // Search center
    return findhelp(root.cchild(), k);
  else if (k.compareTo(root.rkey()) < 0)  // Search center
    return findhelp(root.cchild(), k);
  else return findhelp(root.rchild(), k); // Search right
}

Insertion into a 2-3 tree is similar to insertion into a BST to the extent that the new record is placed in the appropriate leaf node. Unlike BST insertion, a new child is not created to hold the record being inserted, that is, the 2-3 tree does not grow downward. The first step is to find the leaf node that would contain the record if it were in the tree. If this leaf node contains only one value, then the new record can be added to that node with no further modification to the tree, as illustrated in the following visualization.

If we insert the new record into a leaf node that already contains two records, then more space must be created. Consider the two records of node and the record to be inserted without further concern for which two were already in and which is the new record. The first step is to split into two nodes. Thus, a new node—call it —must be created from free store. receives the record with the least of the three key values. receives the greatest of the three. The record with the middle of the three key value is passed up to the parent node along with a pointer to . This is called a promotion. The promoted key is then inserted into the parent. If the parent currently contains only one record (and thus has only two children), then the promoted record and the pointer to are simply added to the parent node. If the parent is full, then the split-and-promote process is repeated. Here is an example of a a simple promotion.

Here is an illustration for what happens when promotions require the root to split, adding a new level to the tree. Note that all leaf nodes continue to have equal depth.

Here is an implementation for the insertion process.

private TTNode<Key,E> inserthelp(TTNode<Key,E> rt, Key k, E e) {
  TTNode<Key,E> retval;
  if (rt == null) // Empty tree: create a leaf node for root
    return new TTNode<Key,E>(k, e, null, null, null, null, null);
  if (rt.isLeaf()) // At leaf node: insert here
    return rt.add(new TTNode<Key,E>(k, e, null, null, null, null, null));
  // Add to internal node
  if (k.compareTo(rt.lkey()) < 0) { // Insert left
    retval = inserthelp(rt.lchild(), k, e);
    if (retval == rt.lchild()) return rt;
    else return rt.add(retval);
  }
  else if((rt.rkey() == null) || (k.compareTo(rt.rkey()) < 0)) {
    retval = inserthelp(rt.cchild(), k, e);
    if (retval == rt.cchild()) return rt;
    else return rt.add(retval);
  }
  else { // Insert right
    retval = inserthelp(rt.rchild(), k, e);
    if (retval == rt.rchild()) return rt;
    else return rt.add(retval);
  }
}

// Add a new key/value pair to the node. There might be a subtree
// associated with the record being added. This information comes
// in the form of a 2-3 tree node with one key and a (possibly null)
// subtree through the center pointer field.
public TTNode<Key,E> add(TTNode<Key,E> it) {
  if (rkey == null) { // Only one key, add here
    if (lkey.compareTo(it.lkey()) < 0) {
      rkey = it.lkey(); rval = it.lval();
      center = it.lchild(); right = it.cchild();
    }
    else {
      rkey = lkey; rval = lval; right = center;
      lkey = it.lkey(); lval = it.lval();
      center = it.cchild();
    }
    return this;
  }
  else if (lkey.compareTo(it.lkey()) >= 0) { // Add left
    TTNode<Key,E> N1 = new TTNode<Key,E>(lkey, lval, null, null, it, this, null);
    it.setLeftChild(left);
    left = center; center = right; right = null;
    lkey = rkey; lval = rval; rkey = null; rval = null;
    return N1;
  }
  else if (rkey.compareTo(it.lkey()) >= 0) { // Add center
    it.setCenterChild(new TTNode<Key,E>(rkey, rval, null, null, it.cchild(), right, null));
    it.setLeftChild(this);
    rkey = null; rval = null; right = null;
    return it;
  }
  else { // Add right
    TTNode<Key,E> N1 = new TTNode<Key,E>(rkey, rval, null, null, this, it, null);
    it.setLeftChild(right);
    right = null; rkey = null; rval = null;
    return N1;
  }
}

Note that inserthelp takes three parameters. The first is a pointer to the root of the current subtree, named rt. The second is the key for the record to be inserted, and the third is the record itself. The return value for inserthelp is a pointer to a 2-3 tree node. If rt is unchanged, then a pointer to rt is returned. If rt is changed (due to the insertion causing the node to split), then a pointer to the new subtree root is returned, with the key value and record value in the leftmost fields, and a pointer to the (single) subtree in the center pointer field. This revised node will then be added to the parent as illustrated by the splitting visualization above.

When deleting a record from the 2-3 tree, there are three cases to consider. The simplest occurs when the record is to be removed from a leaf node containing two records. In this case, the record is simply removed, and no other nodes are affected. The second case occurs when the only record in a leaf node is to be removed. The third case occurs when a record is to be removed from an internal node. In both the second and the third cases, the deleted record is replaced with another that can take its place while maintaining the correct order, similar to removing a node from a BST. If the tree is sparse enough, there is no such record available that will allow all nodes to still maintain at least one record. In this situation, sibling nodes are merged together. The delete operation for the 2-3 tree is excessively complex and will not be described further. Instead, a complete discussion of deletion will be postponed until the next section, where it can be generalized for a particular variant of the B-tree.

The 2-3 tree insert and delete routines do not add new nodes at the bottom of the tree. Instead they cause leaf nodes to split or merge, possibly causing a ripple effect moving up the tree to the root. If necessary the root will split, causing a new root node to be created and making the tree one level deeper. On deletion, if the last two children of the root merge, then the root node is removed and the tree will lose a level. In either case, all leaf nodes are always at the same level. When all leaf nodes are at the same level, we say that a tree is height balanced. Because the 2-3 tree is height balanced, and every internal node has at least two children, we know that the maximum depth of the tree is . Thus, all 2-3 tree insert, find, and delete operations require time.

Click here for another visualization that will let you construct and interact with a 2-3 tree. Actually, this visualization is for a data structure that is more general than just a 2-3 tree. To see how a 2-3 would behave, be sure to use the “Max Degree = 3” setting. This visualization was written by David Galles of the University of San Francisco as part of his Data Structure Visualizations package.

9.6. B-Trees

9.6.1. B-Trees

This section presents the B-tree. B-trees are usually attributed to R. Bayer and E. McCreight who described the B-tree in a 1972 paper. By 1979, B-trees had replaced virtually all large-file access methods other than hashing. B-trees, or some variant of B-trees, are the standard file organization for applications requiring insertion, deletion, and key range searches. They are used to implement most modern file systems. B-trees address effectively all of the major problems encountered when implementing disk-based search trees:

1. The B-tree is shallow, in part because the tree is always height balanced (all leaf nodes are at the same level), and in part because the branching factor is quite high. So only a small number of disk blocks are accessed to reach a given record.
2. Update and search operations affect only those disk blocks on the path from the root to the leaf node containing the query record. The fewer the number of disk blocks affected, the less disk I/O is required.
3. B-trees keep related records (that is, records with similar key values) on the same disk block, which helps to minimize disk I/O on range searches.
4. B-trees guarantee that every node in the tree will be full at least to a certain minimum percentage. This improves space efficiency while reducing the typical number of disk fetches necessary during a search or update operation.

A B-tree of order is defined to have the following shape properties:

• The root is either a leaf or has at least two children.
• Each internal node, except for the root, has between and children.
• All leaves are at the same level in the tree, so the tree is always height balanced.

The B-tree is a generalization of the 2-3 tree. Put another way, a 2-3 tree is a B-tree of order three. Normally, the size of a node in the B-tree is chosen to fill a disk block. A B-tree node implementation typically allows 100 or more children. Thus, a B-tree node is equivalent to a disk block, and a “pointer” value stored in the tree is actually the number of the block containing the child node (usually interpreted as an offset from the beginning of the corresponding disk file). In a typical application, the B-tree’s access to the disk file will be managed using a buffer pool and a block-replacement scheme such as LRU.

Figure 9.6.1 shows a B-tree of order four. Each node contains up to three keys, and internal nodes have up to four children.

Figure 9.6.1: A B-tree of order four.

Search in a B-tree is a generalization of search in a 2-3 tree. It is an alternating two-step process, beginning with the root node of the B-tree.

1. Perform a binary search on the records in the current node. If a record with the search key is found, then return that record. If the current node is a leaf node and the key is not found, then report an unsuccessful search.
2. Otherwise, follow the proper branch and repeat the process.

For example, consider a search for the record with key value 47 in the tree of Figure 9.6.1. The root node is examined and the second (right) branch taken. After examining the node at level 1, the third branch is taken to the next level to arrive at the leaf node containing a record with key value 47.

B-tree insertion is a generalization of 2-3 tree insertion. The first step is to find the leaf node that should contain the key to be inserted, space permitting. If there is room in this node, then insert the key. If there is not, then split the node into two and promote the middle key to the parent. If the parent becomes full, then it is split in turn, and its middle key promoted.

Note that this insertion process is guaranteed to keep all nodes at least half full. For example, when we attempt to insert into a full internal node of a B-tree of order four, there will now be five children that must be dealt with. The node is split into two nodes containing two keys each, thus retaining the B-tree property. The middle of the five children is promoted to its parent.

9.6.1.1. B+ Trees

The previous section mentioned that B-trees are universally used to implement large-scale disk-based systems. Actually, the B-tree as described in the previous section is almost never implemented. What is most commonly implemented is a variant of the B-tree, called the tree. When greater efficiency is required, a more complicated variant known as the tree is used.

Consider again the linear index. When the collection of records will not change, a linear index provides an extremely efficient way to search. The problem is how to handle those pesky inserts and deletes. We could try to keep the core idea of storing a sorted array-based list, but make it more flexible by breaking the list into manageable chunks that are more easily updated. How might we do that? First, we need to decide how big the chunks should be. Since the data are on disk, it seems reasonable to store a chunk that is the size of a disk block, or a small multiple of the disk block size. If the next record to be inserted belongs to a chunk that hasn’t filled its block then we can just insert it there. The fact that this might cause other records in that chunk to move a little bit in the array is not important, since this does not cause any extra disk accesses so long as we move data within that chunk. But what if the chunk fills up the entire block that contains it? We could just split it in half. What if we want to delete a record? We could just take the deleted record out of the chunk, but we might not want a lot of near-empty chunks. So we could put adjacent chunks together if they have only a small amount of data between them. Or we could shuffle data between adjacent chunks that together contain more data. The big problem would be how to find the desired chunk when processing a record with a given key. Perhaps some sort of tree-like structure could be used to locate the appropriate chunk. These ideas are exactly what motivate the tree. The tree is essentially a mechanism for managing a sorted array-based list, where the list is broken into chunks.

The most significant difference between the tree and the BST or the standard B-tree is that the tree stores records only at the leaf nodes. Internal nodes store key values, but these are used solely as placeholders to guide the search. This means that internal nodes are significantly different in structure from leaf nodes. Internal nodes store keys to guide the search, associating each key with a pointer to a child tree node. Leaf nodes store actual records, or else keys and pointers to actual records in a separate disk file if the tree is being used purely as an index. Depending on the size of a record as compared to the size of a key, a leaf node in a tree of order might have enough room to store more or less than records. The requirement is simply that the leaf nodes store enough records to remain at least half full. The leaf nodes of a tree are normally linked together to form a doubly linked list. Thus, the entire collection of records can be traversed in sorted order by visiting all the leaf nodes on the linked list. Here is a Java-like pseudocode representation for the tree node interface. Leaf node and internal node subclasses would implement this interface.

/** Interface for B+ Tree nodes */
public interface BPNode<Key,E> {
  public boolean isLeaf();
  public int numrecs();
  public Key[] keys();
}

An important implementation detail to note is that while Figure 9.6.1 shows internal nodes containing three keys and four pointers, class BPNode is slightly different in that it stores key/pointer pairs. Figure 9.6.1 shows the tree as it is traditionally drawn. To simplify implementation in practice, nodes really do associate a key with each pointer. Each internal node should be assumed to hold in the leftmost position an additional key that is less than or equal to any possible key value in the node’s leftmost subtree. tree implementations typically store an additional dummy record in the leftmost leaf node whose key value is less than any legal key value.

trees are exceptionally good for range queries. Once the first record in the range has been found, the rest of the records with keys in the range can be accessed by sequential processing of the remaining records in the first node, and then continuing down the linked list of leaf nodes as far as necessary. Figure 9.6.2 illustrates the tree.

Figure 9.6.2: Example of a tree of order four. Internal nodes must store between two and four children. For this example, the record size is assumed to be such that leaf nodes store between three and five records.

Search in a tree is nearly identical to search in a regular B-tree, except that the search must always continue to the proper leaf node. Even if the search-key value is found in an internal node, this is only a placeholder and does not provide access to the actual record. To find a record with key value 33 in the tree of Figure 9.6.2, search begins at the root. The value 33 stored in the root merely serves as a placeholder, indicating that keys with values greater than or equal to 33 are found in the second subtree. From the second child of the root, the first branch is taken to reach the leaf node containing the actual record (or a pointer to the actual record) with key value 33. Here is a pseudocode sketch of the tree search algorithm.

private E findhelp(BPNode<Key,E> rt, Key k) {
  int currec = binaryle(rt.keys(), rt.numrecs(), k);
  if (rt.isLeaf())
    if ((((BPLeaf<Key,E>)rt).keys())[currec] == k)
      return ((BPLeaf<Key,E>)rt).recs(currec);
    else return null;
  else
    return findhelp(((BPInternal<Key,E>)rt).pointers(currec), k);
}

tree insertion is similar to B-tree insertion. First, the leaf that should contain the record is found. If is not full, then the new record is added, and no other tree nodes are affected. If is already full, split it in two (dividing the records evenly among the two nodes) and promote a copy of the least-valued key in the newly formed right node. As with the 2-3 tree, promotion might cause the parent to split in turn, perhaps eventually leading to splitting the root and causing the tree to gain a new level. tree insertion keeps all leaf nodes at equal depth. Figure 9.6.3 illustrates the insertion process through several examples.

Figure 9.6.3: Examples of tree insertion. (a) B- tree containing five records. (b) The result of inserting a record with key value 50 into the tree of (a). The leaf node splits, causing creation of the first internal node. (c) The tree of (b) after further insertions. (d) The result of inserting a record with key value 30 into the tree of (c). The second leaf node splits, which causes the internal node to split in turn, creating a new root.

Here is a a Java-like pseudocode sketch of the tree insert algorithm.

private BPNode<Key,E> inserthelp(BPNode<Key,E> rt,
                                 Key k, E e) {
  BPNode<Key,E> retval;
  if (rt.isLeaf()) // At leaf node: insert here
    return ((BPLeaf<Key,E>)rt).add(k, e);
  // Add to internal node
  int currec = binaryle(rt.keys(), rt.numrecs(), k);
  BPNode<Key,E> temp = inserthelp(
         ((BPInternal<Key,E>)root).pointers(currec), k, e);
  if (temp != ((BPInternal<Key,E>)rt).pointers(currec))
    return ((BPInternal<Key,E>)rt).
               add((BPInternal<Key,E>)temp);
  else
    return rt;
}

Here is an exercise to see if you get the basic idea of tree insertion.

To delete record from the tree, first locate the leaf that contains . If is more than half full, then we need only remove , leaving still at least half full. This is demonstrated by Figure 9.6.4.

Figure 9.6.4: Simple deletion from a tree. The record with key value 18 is removed from the tree of Figure 9.6.2. Note that even though 18 is also a placeholder used to direct search in the parent node, that value need not be removed from internal nodes even if no record in the tree has key value 18. Thus, the leftmost node at level one in this example retains the key with value 18 after the record with key value 18 has been removed from the second leaf node.

If deleting a record reduces the number of records in the node below the minimum threshold (called an underflow), then we must do something to keep the node sufficiently full. The first choice is to look at the node’s adjacent siblings to determine if they have a spare record that can be used to fill the gap. If so, then enough records are transferred from the sibling so that both nodes have about the same number of records. This is done so as to delay as long as possible the next time when a delete causes this node to underflow again. This process might require that the parent node has its placeholder key value revised to reflect the true first key value in each node. Figure 9.6.5 illustrates the process.

Figure 9.6.5: Deletion from the tree of Figure 9.6.2 via borrowing from a sibling. The key with value 12 is deleted from the leftmost leaf, causing the record with key value 18 to shift to the leftmost leaf to take its place. Note that the parent must be updated to properly indicate the key range within the subtrees. In this example, the parent node has its leftmost key value changed to 19.

If neither sibling can lend a record to the under-full node (call it ), then must give its records to a sibling and be removed from the tree. There is certainly room to do this, because the sibling is at most half full (remember that it had no records to contribute to the current node), and has become less than half full because it is under-flowing. This merge process combines two subtrees of the parent, which might cause it to underflow in turn. If the last two children of the root merge together, then the tree loses a level. Figure 9.6.6 illustrates the node-merge deletion process.

Figure 9.6.6: Deleting the record with key value 33 from the tree of Figure 9.6.2 via collapsing siblings. (a) The two leftmost leaf nodes merge together to form a single leaf. Unfortunately, the parent node now has only one child. (b) Because the left subtree has a spare leaf node, that node is passed to the right subtree. The placeholder values of the root and the right internal node are updated to reflect the changes. Value 23 moves to the root, and old root value 33 moves to the rightmost internal node.

Here is a Java-like pseudocode for the tree delete algorithm.

/** Delete a record with the given key value, and
    return true if the root underflows */
private boolean removehelp(BPNode<Key,E> rt, Key k) {
  int currec = binaryle(rt.keys(), rt.numrecs(), k);
  if (rt.isLeaf())
    if (((BPLeaf<Key,E>)rt).keys()[currec] == k)
      return ((BPLeaf<Key,E>)rt).delete(currec);
    else return false;
  else // Process internal node
    if (removehelp(((BPInternal<Key,E>)rt).pointers(currec),
        k))
      // Child will merge if necessary
      return ((BPInternal<Key,E>)rt).underflow(currec);
    else return false;
}

The tree requires that all nodes be at least half full (except for the root). Thus, the storage utilization must be at least 50%. This is satisfactory for many implementations, but note that keeping nodes fuller will result both in less space required (because there is less empty space in the disk file) and in more efficient processing (fewer blocks on average will be read into memory because the amount of information in each block is greater). Because B-trees have become so popular, many algorithm designers have tried to improve B-tree performance. One method for doing so is to use the tree variant known as the $\mathrm{B}^\mathrm{B}^\mathrm{B}^+\mathrm{B}^*$ tree gives some records to its neighboring sibling, if possible. If the sibling is also full, then these two nodes split into three. Similarly, when a node underflows, it is combined with its two siblings, and the total reduced to two nodes. Thus, the nodes are always at least two thirds full. [^1]

Click here for a visualization that will let you construct and interact with a tree. This visualization was written by David Galles of the University of San Francisco as part of his Data Structure Visualizations package.

[^1]: This concept can be extended further if higher space utilization is required. However, the update routines become much more complicated. I once worked on a project where we implemented 3-for-4 node split and merge routines. This gave better performance than the 2-for-3 node split and merge routines of the tree. However, the spitting and merging routines were so complicated that even their author could no longer understand them once they were completed!

9.6.1.2. B-Tree Analysis

The asymptotic cost of search, insertion, and deletion of records from B-trees, trees, and trees is where is the total number of records in the tree. However, the base of the log is the (average) branching factor of the tree. Typical database applications use extremely high branching factors, perhaps 100 or more. Thus, in practice the B-tree and its variants are extremely shallow.

As an illustration, consider a tree of order 100 and leaf nodes that contain up to 100 records. A B- tree with height one (that is, just a single leaf node) can have at most 100 records. A tree with height two (a root internal node whose children are leaves) must have at least 100 records (2 leaves with 50 records each). It has at most 10,000 records (100 leaves with 100 records each). A tree with height three must have at least 5000 records (two second-level nodes with 50 children containing 50 records each) and at most one million records (100 second-level nodes with 100 full children each). A tree with height four must have at least 250,000 records and at most 100 million records. Thus, it would require an extremely large database to generate a tree of more than height four.

The tree split and insert rules guarantee that every node (except perhaps the root) is at least half full. So they are on average about 3/4 full. But the internal nodes are purely overhead, since the keys stored there are used only by the tree to direct search, rather than store actual data. Does this overhead amount to a significant use of space? No, because once again the high fan-out rate of the tree structure means that the vast majority of nodes are leaf nodes. A K-ary tree has approximately of its nodes as internal nodes. This means that while half of a full binary tree’s nodes are internal nodes, in a tree of order 100 probably only about of its nodes are internal nodes. This means that the overhead associated with internal nodes is very low.

We can reduce the number of disk fetches required for the B-tree even more by using the following methods. First, the upper levels of the tree can be stored in main memory at all times. Because the tree branches so quickly, the top two levels (levels 0 and 1) require relatively little space. If the B-tree is only height four, then at most two disk fetches (internal nodes at level two and leaves at level three) are required to reach the pointer to any given record.

A buffer pool could be used to manage nodes of the B-tree. Several nodes of the tree would typically be in main memory at one time. The most straightforward approach is to use a standard method such as LRU to do node replacement. However, sometimes it might be desirable to “lock” certain nodes such as the root into the buffer pool. In general, if the buffer pool is even of modest size (say at least twice the depth of the tree), no special techniques for node replacement will be required because the upper-level nodes will naturally be accessed frequently.

9.7. Indexing Summary Exercises

Here are some review questions.

Chapter 10: Hashing

10.1. Introduction

10.1.1. Introduction

Hashing is a method for storing and retrieving records from a database. It lets you insert, delete, and search for records based on a search key value. When properly implemented, these operations can be performed in constant time. In fact, a properly tuned hash system typically looks at only one or two records for each search, insert, or delete operation. This is far better than the average cost required to do a binary search on a sorted array of records, or the average cost required to do an operation on a binary search tree. However, even though hashing is based on a very simple idea, it is surprisingly difficult to implement properly. Designers need to pay careful attention to all of the details involved with implementing a hash system.

A hash system stores records in an array called a hash table, which we will call HT. Hashing works by performing a computation on a search key K in a way that is intended to identify the position in HT that contains the record with key K. The function that does this calculation is called the hash function, and will be denoted by the letter h. Since hashing schemes place records in the table in whatever order satisfies the needs of the address calculation, records are not ordered by value. A position in the hash table is also known as a slot. The number of slots in hash table HT will be denoted by the variable with slots numbered from 0 to .

The goal for a hashing system is to arrange things such that, for any key value K and some hash function , is a slot in the table such that , and we have the key of the record stored at HT[i] equal to K.

Hashing is not good for applications where multiple records with the same key value are permitted. Hashing is not a good method for answering range searches. In other words, we cannot easily find all records (if any) whose key values fall within a certain range. Nor can we easily find the record with the minimum or maximum key value, or visit the records in key order. Hashing is most appropriate for answering the question, ‘What record, if any, has key value K?’ For applications where all search is done by exact-match queries, hashing is the search method of choice because it is extremely efficient when implemented correctly. As this tutorial shows, however, there are many approaches to hashing and it is easy to devise an inefficient implementation. Hashing is suitable for both in-memory and disk-based searching and is one of the two most widely used methods for organizing large databases stored on disk (the other is the B-tree).

As a simple (though unrealistic) example of hashing, consider storing records, each with a unique key value in the range 0 to . A record with key k can be stored in HT[k], and so the hash function is . To find the record with key value k, look in HT[k].

In most applications, there are many more values in the key range than there are slots in the hash table. For a more realistic example, suppose the key can take any value in the range 0 to 65,535 (i.e., the key is a two-byte unsigned integer), and that we expect to store approximately 1000 records at any given time. It is impractical in this situation to use a hash table with 65,536 slots, because then the vast majority of the slots would be left empty. Instead, we must devise a hash function that allows us to store the records in a much smaller table. Because the key range is larger than the size of the table, at least some of the slots must be mapped to from multiple key values. Given a hash function h and two keys and , if where is a slot in the table, then we say that and have a collision at slot under hash function h.

Finding a record with key value K in a database organized by hashing follows a two-step procedure:

1. Compute the table location .
2. Starting with slot , locate the record containing key K using (if necessary) a collision resolution policy .

10.2. Hash Function Principles

10.2.1. Hash Function Principles

Hashing generally takes records whose key values come from a large range and stores those records in a table with a relatively small number of slots. Collisions occur when two records hash to the same slot in the table. If we are careful—or lucky—when selecting a hash function, then the actual number of collisions will be few. Unfortunately, even under the best of circumstances, collisions are nearly unavoidable. To illustrate, consider a classroom full of students. What is the probability that some pair of students shares the same birthday (i.e., the same day of the year, not necessarily the same year)? If there are 23 students, then the odds are about even that two will share a birthday. This is despite the fact that there are 365 days in which students can have birthdays (ignoring leap years). On most days, no student in the class has a birthday. With more students, the probability of a shared birthday increases. The mapping of students to days based on their birthday is similar to assigning records to slots in a table (of size 365) using the birthday as a hash function. Note that this observation tells us nothing about which students share a birthday, or on which days of the year shared birthdays fall.

Try it for yourself. You can use the calculator to see the probability of a collision. The default values are set to show the number of people in a room such that the chance of a duplicate is just over 50%. But you can set any table size and any number of records to determine the probability of a collision under those conditions.

Use the calculator to answer the following questions.

To be practical, a database organized by hashing must store records in a hash table that is not so large that it wastes space. To balance time and space efficiency, this means that the hash table should be around half full. Because collisions are extremely likely to occur under these conditions (by chance, any record inserted into a table that is half full should have a collision half of the time), does this mean that we need not worry about how well a hash function does at avoiding collisions? Absolutely not. The difference between using a good hash function and a bad hash function makes a big difference in practice in the number of records that must be examined when searching or inserting to the table. Technically, any function that maps all possible key values to a slot in the hash table is a hash function. In the extreme case, even a function that maps all records to the same slot in the array is a hash function, but it does nothing to help us find records during a search operation.

We would like to pick a hash function that maps keys to slots in a way that makes each slot in the hash table have equal probablility of being filled for the actual set keys being used. Unfortunately, we normally have no control over the distribution of key values for the actual records in a given database or collection. So how well any particular hash function does depends on the actual distribution of the keys used within the allowable key range. In some cases, incoming data are well distributed across their key range. For example, if the input is a set of random numbers selected uniformly from the key range, any hash function that assigns the key range so that each slot in the hash table receives an equal share of the range will likely also distribute the input records uniformly within the table. However, in many applications the incoming records are highly clustered or otherwise poorly distributed. When input records are not well distributed throughout the key range it can be difficult to devise a hash function that does a good job of distributing the records throughout the table, especially if the input distribution is not known in advance.

There are many reasons why data values might be poorly distributed.

1. Natural frequency distributions tend to follow a common pattern where a few of the entities occur frequently while most entities occur relatively rarely. For example, consider the populations of the 100 largest cities in the United States. If you plot these populations on a numberline, most of them will be clustered toward the low side, with a few outliers on the high side. This is an example of a Zipf distribution. Viewed the other way, the home town for a given person is far more likely to be a particular large city than a particular small town.
2. Collected data are likely to be skewed in some way. Field samples might be rounded to, say, the nearest 5 (i.e., all numbers end in 5 or 0).
3. If the input is a collection of common English words, the beginning letter will be poorly distributed.

Note that for items 2 and 3 on this list, either high- or low-order bits of the key are poorly distributed.

When designing hash functions, we are generally faced with one of two situations:

1. We know nothing about the distribution of the incoming keys. In this case, we wish to select a hash function that evenly distributes the key range across the hash table, while avoiding obvious opportunities for clustering such as hash functions that are sensitive to the high- or low-order bits of the key value.
2. We know something about the distribution of the incoming keys. In this case, we should use a distribution-dependent hash function that avoids assigning clusters of related key values to the same hash table slot. For example, if hashing English words, we should not hash on the value of the first character because this is likely to be unevenly distributed.

In the next module, you will see several examples of hash functions that illustrate these points.

10.3. Sample Hash Functions

10.3.1. Sample Hash Functions

10.3.1.1. Simple Mod Function

Consider the following hash function used to hash integers to a table of sixteen slots:

int h(int x) {
  return x % 16;
}

Here “%” is the symbol for the mod function.

Recall that the values 0 to 15 can be represented with four bits (i.e., 0000 to 1111). The value returned by this hash function depends solely on the least significant four bits of the key. Because these bits are likely to be poorly distributed (as an example, a high percentage of the keys might be even numbers, which means that the low order bit is zero), the result will also be poorly distributed. This example shows that the size of the table can have a big effect on the performance of a hash system because the table size is typically used as the modulus to ensure that the hash function produces a number in the range 0 to .

10.3.1.2. Binning

Say we are given keys in the range 0 to 999, and have a hash table of size 10. In this case, a possible hash function might simply divide the key value by 100. Thus, all keys in the range 0 to 99 would hash to slot 0, keys 100 to 199 would hash to slot 1, and so on. In other words, this hash function “bins” the first 100 keys to the first slot, the next 100 keys to the second slot, and so on.

Binning in this way has the problem that it will cluster together keys if the distribution does not divide evenly on the high-order bits. In the above example, if more records have keys in the range 900-999 (first digit 9) than have keys in the range 100-199 (first digit 1), more records will hash to slot 9 than to slot 1. Likewise, if we pick too big a value for the key range and the actual key values are all relatively small, then most records will hash to slot 0. A similar, analogous problem arises if we were instead hashing strings based on the first letter in the string.

In general with binning we store the record with key value at array position for some value (using integer division). A problem with Binning is that we have to know the key range so that we can figure out what value to use for . Let’s assume that the keys are all in the range 0 to 999. Then we want to divide key values by 100 so that the result is in the range 0 to 9. There is no particular limit on the key range that binning could handle, so long as we know the maximum possible value in advance so that we can figure out what to divide the key value by. Alternatively, we could also take the result of any binning computation and then mod by the table size to be safe. So if we have keys that are bigger than 999 when dividing by 100, we can still make sure that the result is in the range 0 to 9 with a mod by 10 step at the end.

Binning looks at the opposite part of the key value from the mod function. The mod function, for a power of two, looks at the low-order bits, while binning looks at the high-order bits. Or if you want to think in base 10 instead of base 2, modding by 10 or 100 looks at the low-order digits, while binning into an array of size 10 or 100 looks at the high-order digits.

As another example, consider hashing a collection of keys whose values follow a normal distribution, as illustrated by Figure 10.3.1. Keys near the mean of the normal distribution are far more likely to occur than keys near the tails of the distribution. For a given slot, think of where the keys come from within the distribution. Binning would be taking thick slices out of the distribution and assign those slices to hash table slots. If we use a hash table of size 8, we would divide the key range into 8 equal-width slices and assign each slice to a slot in the table. Since a normal distribution is more likely to generate keys from the middle slice, the middle slot of the table is most likely to be used. In contrast, if we use the mod function, then we are assigning to any given slot in the table a series of thin slices in steps of 8. In the normal distribution, some of these slices associated with any given slot are near the tails, and some are near the center. Thus, each table slot is equally likely (roughly) to get a key value.

Figure 10.3.1: A comparison of binning vs. modulus as a hash function.

10.3.1.3. The Mid-Square Method

A good hash function to use with integer key values is the mid-square method. The mid-square method squares the key value, and then takes out the middle bits of the result, giving a value in the range 0 to . This works well because most or all bits of the key value contribute to the result. For example, consider records whose keys are 4-digit numbers in base 10, as shown in Figure 10.3.2. The goal is to hash these key values to a table of size 100 (i.e., a range of 0 to 99). This range is equivalent to two digits in base 10. That is, . If the input is the number 4567, squaring yields an 8-digit number, 20857489. The middle two digits of this result are 57. All digits of the original key value (equivalently, all bits when the number is viewed in binary) contribute to the middle two digits of the squared value. Thus, the result is not dominated by the distribution of the bottom digit or the top digit of the original key value. Of course, if the key values all tend to be small numbers, then their squares will only affect the low-order digits of the hash value.

{width=10%}

Figure 10.3.2: An example of the mid-square method. This image shows the traditional gradeschool long multiplication process. The value being squared is 4567. The result of squaring is 20857489. At the bottom, of the image, the value 4567 is show again, with each digit at the bottom of a “V”. The associated “V” is showing the digits from the result that are being affected by each digit of the input. That is, “4” affects the output digits 2, 0, 8, 5, an 7. But it has no affect on the last 3 digits. The key point is that the middle two digits of the result (5 and 7) are affected by every digit of the input.

Here is a little calculator for you to see how this works. Start with ‘4567’ as an example.

10.3.2. A Simple Hash Function for Strings

Now we will examine some hash functions suitable for storing strings of characters. We start with a simple summation function:

int sascii(String x, int M) {
  char ch[];
  ch = x.toCharArray();
  int xlength = x.length();

  int i, sum;
  for (sum=0, i=0; i < x.length(); i++)
    sum += ch[i];
  return sum % M;
}

This function sums the ASCII values of the letters in a string. If the hash table size is small compared to the resulting summations, then this hash function should do a good job of distributing strings evenly among the hash table slots, because it gives equal weight to all characters in the string. This is an example of the folding method to designing a hash function. Note that the order of the characters in the string has no effect on the result. A similar method for integers would add the digits of the key value, assuming that there are enough digits to

1. keep any one or two digits with bad distribution from skewing the results of the process and
2. generate a sum much larger than .

As with many other hash functions, the final step is to apply the modulus operator to the result, using table size to generate a value within the table range. If the sum is not sufficiently large, then the modulus operator will yield a poor distribution. For example, because the ASCII value for ‘A’ is 65 and ‘Z’ is 90, sum will always be in the range 650 to 900 for a string of ten upper case letters. For a hash table of size 100 or less, a reasonable distribution results. For a hash table of size 1000, the distribution is terrible because only slots 650 to 900 can possibly be the home slot for some key value, and the values are not evenly distributed even within those slots.

Now you can try it out with this calculator.

10.3.3. String Folding

Here is a much better hash function for strings.

// Use folding on a string, summed 4 bytes at a time
int sfold(String s, int M) {
  long sum = 0, mul = 1;
  for (int i = 0; i < s.length(); i++) {
    mul = (i % 4 == 0) ? 1 : mul * 256;
    sum += s.charAt(i) * mul;
  }
  return (int)(Math.abs(sum) % M);
}

This function takes a string as input. It processes the string four bytes at a time, and interprets each of the four-byte chunks as a single long integer value. The integer values for the four-byte chunks are added together. In the end, the resulting sum is converted to the range 0 to using the modulus operator.

For example, if the string “aaaabbbb” is passed to sfold, then the first four bytes (“aaaa”) will be interpreted as the integer value 1,633,771,873, and the next four bytes (“bbbb”) will be interpreted as the integer value 1,650,614,882. Their sum is 3,284,386,755 (when treated as an unsigned integer). If the table size is 101 then the modulus function will cause this key to hash to slot 75 in the table.

Now you can try it out with this calculator.

For any sufficiently long string, the sum for the integer quantities will typically cause a 32-bit integer to overflow (thus losing some of the high-order bits) because the resulting values are so large. But this causes no problems when the goal is to compute a hash function.

The reason that hashing by summing the integer representation of four letters at a time is superior to summing one letter at a time is because the resulting values being summed have a bigger range. This still only works well for strings long enough (say at least 7-12 letters), but the original method would not work well for short strings either. There is nothing special about using four characters at a time. Other choices could be made. Another alternative would be to fold two characters at a time.

10.3.4. Hash Function Practice

Now here is an exercise to let you practice these various hash functions. You should use the calculators above for the more complicated hash functions.

10.3.5. Hash Function Review Questions

Here are some review questions.

10.4. Open Hashing

10.4.1. Open Hashing

While the goal of a hash function is to minimize collisions, some collisions are unavoidable in practice. Thus, hashing implementations must include some form of collision resolution policy. Collision resolution techniques can be broken into two classes: open hashing (also called separate chaining) and closed hashing (also called open addressing). (Yes, it is confusing when “open hashing” means the opposite of “open addressing”, but unfortunately, that is the way it is.) The difference between the two has to do with whether collisions are stored outside the table (open hashing), or whether collisions result in storing one of the records at another slot in the table (closed hashing).

The simplest form of open hashing defines each slot in the hash table to be the head of a linked list. All records that hash to a particular slot are placed on that slot’s linked list. The following figure illustrates a hash table where each slot points to a linked list to hold the records associated with that slot. The hash function used is the simple mod function.

Records within a slot’s list can be ordered in several ways: by insertion order, by key value order, or by frequency-of-access order. Ordering the list by key value provides an advantage in the case of an unsuccessful search, because we know to stop searching the list once we encounter a key that is greater than the one being searched for. If records on the list are unordered or ordered by frequency, then an unsuccessful search will need to visit every record on the list.

Given a table of size storing records, the hash function will (ideally) spread the records evenly among the positions in the table, yielding on average records for each list. Assuming that the table has more slots than there are records to be stored, we can hope that few slots will contain more than one record. In the case where a list is empty or has only one record, a search requires only one access to the list. Thus, the average cost for hashing should be ). However, if clustering causes many records to hash to only a few of the slots, then the cost to access a record will be much higher because many elements on the linked list must be searched.

Open hashing is most appropriate when the hash table is kept in main memory, with the lists implemented by a standard in-memory linked list. Storing an open hash table on disk in an efficient way is difficult, because members of a given linked list might be stored on different disk blocks. This would result in multiple disk accesses when searching for a particular key value, which defeats the purpose of using hashing.

There are similarities between open hashing and Binsort. One way to view open hashing is that each record is simply placed in a bin. While multiple records may hash to the same bin, this initial binning should still greatly reduce the number of records accessed by a search operation. In a similar fashion, a simple Binsort reduces the number of records in each bin to a small number that can be sorted in some other way.

10.5. Bucket Hashing

10.5.1. Bucket Hashing

Closed hashing stores all records directly in the hash table. Each record with key value has a home position that is , the slot computed by the hash function. If is to be inserted and another record already occupies ’s home position, then will be stored at some other slot in the table. It is the business of the collision resolution policy to determine which slot that will be. Naturally, the same policy must be followed during search as during insertion, so that any record not found in its home position can be recovered by repeating the collision resolution process.

One implementation for closed hashing groups hash table slots into buckets. The slots of the hash table are divided into buckets, with each bucket consisting of slots. The hash function assigns each record to the first slot within one of the buckets. If this slot is already occupied, then the bucket slots are searched sequentially until an open slot is found. If a bucket is entirely full, then the record is stored in an overflow bucket of infinite capacity at the end of the table. All buckets share the same overflow bucket. A good implementation will use a hash function that distributes the records evenly among the buckets so that as few records as possible go into the overflow bucket.

When searching for a record, the first step is to hash the key to determine which bucket should contain the record. The records in this bucket are then searched. If the desired key value is not found and the bucket still has free slots, then the search is complete. If the bucket is full, then it is possible that the desired record is stored in the overflow bucket. In this case, the overflow bucket must be searched until the record is found or all records in the overflow bucket have been checked. If many records are in the overflow bucket, this will be an expensive process.

Now you can try it yourself.

10.5.2. An Alternate Approach

A simple variation on bucket hashing is to hash a key value to some slot in the hash table as though bucketing were not being used. If the home position is full, then we search through the rest of the bucket to find an empty slot. If all slots in this bucket are full, then the record is assigned to the overflow bucket. The advantage of this approach is that initial collisions are reduced, because any slot can be a home position rather than just the first slot in the bucket.

Bucket methods are good for implementing hash tables stored on disk, because the bucket size can be set to the size of a disk block. Whenever search or insertion occurs, the entire bucket is read into memory. Because the entire bucket is then in memory, processing an insert or search operation requires only one disk access, unless the bucket is full. If the bucket is full, then the overflow bucket must be retrieved from disk as well. Naturally, overflow should be kept small to minimize unnecessary disk accesses.

10.6. Collision Resolution

10.6.1. Collision Resolution

We now turn to the most commonly used form of hashing: closed hashing with no bucketing, and a collision resolution policy that can potentially use any slot in the hash table.

During insertion, the goal of collision resolution is to find a free slot in the hash table when the home position for the record is already occupied. We can view any collision resolution method as generating a sequence of hash table slots that can potentially hold the record. The first slot in the sequence will be the home position for the key. If the home position is occupied, then the collision resolution policy goes to the next slot in the sequence. If this is occupied as well, then another slot must be found, and so on. This sequence of slots is known as the probe sequence, and it is generated by some probe function that we will call p. Insertion works as follows:

// Insert e into hash table HT
void hashInsert(const Key& k, const Elem& e) {
  int home;                     // Home position for e
  int pos = home = h(k);        // Init probe sequence
  for (int i=1; EMPTYKEY != (HT[pos]).key(); i++) {
    pos = (home + p(k, i)) % M; // probe
    if (k == HT[pos].key()) {
      println("Duplicates not allowed");
      return;
    }
  }
  HT[pos] = e;
}

Method hashInsert first checks to see if the home slot for the key is empty. If the home slot is occupied, then we use the probe function to locate a free slot in the table. Function p has two parameters, the key and a count of where in the probe sequence we wish to be. That is, to get the first position in the probe sequence after the home slot for key , we call . For the next slot in the probe sequence, call . Note that the probe function returns an offset from the original home position, rather than a slot in the hash table. Thus, the for loop in hashInsert is computing positions in the table at each iteration by adding the value returned from the probe function to the home position. The th call to p returns the th offset to be used.

Searching in a hash table follows the same probe sequence that was followed when inserting records. In this way, a record not in its home position can be recovered. An implementation for the search procedure is as follows.:

// Search for the record with Key K
bool hashSearch(const Key& K, Elem& e) const {
  int home;              // Home position for K
  int pos = home = h(K); // Initial position is the home slot
  for (int i = 1;
       (K != (HT[pos]).key()) && (EMPTYKEY != (HT[pos]).key());
       i++)
    pos = (home + p(K, i)) % M; // Next on probe sequence
  if (K == (HT[pos]).key()) {   // Found it
    e = HT[pos];
    return true;
  }
  else return false;            // K not in hash table
}

Both the insert and the search routines assume that at least one slot on the probe sequence of every key will be empty. Otherwise they will continue in an infinite loop on unsuccessful searches. Thus, the hash system should keep a count of the number of records stored, and refuse to insert into a table that has only one free slot.

The simplest approach to collsion resolution is simply to move down the table from the home slot until a free slot is found. This is known as linear probing. The probe function for simple linear probing is . That is, the th offset on the probe sequence is just , meaning that the th step is simply to move down slots in the table. Once the bottom of the table is reached, the probe sequence wraps around to the beginning of the table (since the last step is to mod the result to the table size). Linear probing has the virtue that all slots in the table will be candidates for inserting a new record before the probe sequence returns to the home position.

Can you see any reason why this might not be the best approach to collision resolution?

10.6.1.1. The Problem with Linear Probing

While linear probing is probably the first idea that comes to mind when considering collision resolution policies, it is not the only one possible. Probe function p allows us many options for how to do collision resolution. In fact, linear probing is one of the worst collision resolution methods. The main problem is illustrated by the next slideshow.

Again, the ideal behavior for a collision resolution mechanism is that each empty slot in the table will have equal probability of receiving the next record inserted (assuming that every slot in the table has equal probability of being hashed to initially). This tendency of linear probing to cluster items together is known as primary clustering. Small clusters tend to merge into big clusters, making the problem worse. The objection to primary clustering is that it leads to long probe sequences.

10.7. Improved Collision Resolution

10.7.1. Linear Probing by Steps

How can we avoid primary clustering? One possible improvement might be to use linear probing, but to skip slots by some constant other than 1. This would make the probe function , and so the th slot in the probe sequence will be . In this way, records with adjacent home positions will not follow the same probe sequence.

One quality of a good probe sequence is that it will cycle through all slots in the hash table before returning to the home position. Clearly linear probing (which “skips” slots by one each time) does this. Unfortunately, not all values for will make this happen. For example, if and the table contains an even number of slots, then any key whose home position is in an even slot will have a probe sequence that cycles through only the even slots. Likewise, the probe sequence for a key whose home position is in an odd slot will cycle through the odd slots. Thus, this combination of table size and linear probing constant effectively divides the records into two sets stored in two disjoint sections of the hash table. So long as both sections of the table contain the same number of records, this is not really important. However, just from chance it is likely that one section will become fuller than the other, leading to more collisions and poorer performance for those records. The other section would have fewer records, and thus better performance. But the overall system performance will be degraded, as the additional cost to the side that is more full outweighs the improved performance of the less-full side.

Constant must be relatively prime to to generate a linear probing sequence that visits all slots in the table (that is, and must share no factors). For a hash table of size , if is any one of 1, 3, 7, or 9, then the probe sequence will visit all slots for any key. When , any value for between 1 and 10 generates a probe sequence that visits all slots for every key.

Now you can practice linear probing by different step sizes.

10.7.2. Pseudo-Random Probing

Consider the situation where and we wish to insert a record with key such that . The probe sequence for is 3, 5, 7, 9, and so on. If another key has home position at slot 5, then its probe sequence will be 5, 7, 9, and so on. The probe sequences of and are linked together in a manner that contributes to clustering. In other words, linear probing with a value of does not solve the problem of primary clustering. We would like to find a probe function that does not link keys together in this way. We would prefer that the probe sequence for after the first step on the sequence should not be identical to the probe sequence of . Instead, their probe sequences should diverge.

The ideal probe function would select the next position on the probe sequence at random from among the unvisited slots; that is, the probe sequence should be a random permutation of the hash table positions. Unfortunately, we cannot actually select the next position in the probe sequence at random, because we would not be able to duplicate this same probe sequence when searching for the key. However, we can do something similar called pseudo-random probing. In pseudo-random probing, the th slot in the probe sequence is where is the th value in a random permutation of the numbers from 1 to . All inserts and searches must use the same sequence of random numbers. The probe function would be where Permutation is an array of length that stores a value of 0 in position Permutation[0], and stores a random permutation of the values from 1 to in slots 1 to .

Here is a practice exercise for pseudo-random probing.

Pseudo-random probing exhibits another desirable feature in a hash function.

10.7.3. Quadratic Probing

Another probe function that eliminates primary clustering is called quadratic probing. Here the probe function is some quadratic function for some choice of constants , , and .

The simplest variation is (i.e., , , and ). Then the th value in the probe sequence would be .

Now you can practice quadratic probing.

There is one problem with quadratic probing: Its probe sequence typically will not visit all slots in the hash table.

For many hash table sizes, this probe function will cycle through a relatively small number of slots. If all slots on that cycle happen to be full, this means that the record cannot be inserted at all! A more realistic example is a table with 105 slots. The probe sequence starting from any given slot will only visit 23 other slots in the table. If all 24 of these slots should happen to be full, even if other slots in the table are empty, then the record cannot be inserted because the probe sequence will continually hit only those same 24 slots.

Fortunately, it is possible to get good results from quadratic probing at low cost. The right combination of probe function and table size will visit many slots in the table. In particular, if the hash table size is a prime number and the probe function is , then at least half the slots in the table will be visited. Thus, if the table is less than half full, we can be certain that a free slot will be found. Alternatively, if the hash table size is a power of two and the probe function is , then every slot in the table will be visited by the probe function.

10.7.4. Double Hashing

Both pseudo-random probing and quadratic probing eliminate primary clustering, which is the name given to the the situation when keys share substantial segments of a probe sequence. If two keys hash to the same home position, however, then they will always follow the same probe sequence for every collision resolution method that we have seen so far. The probe sequences generated by pseudo-random and quadratic probing (for example) are entirely a function of the home position, not the original key value. This is because function p ignores its input parameter for these collision resolution methods. If the hash function generates a cluster at a particular home position, then the cluster remains under pseudo-random and quadratic probing. This problem is called secondary clustering.

To avoid secondary clustering, we need to have the probe sequence make use of the original key value in its decision-making process. A simple technique for doing this is to return to linear probing by a constant step size for the probe function, but to have that constant be determined by a second hash function, . Thus, the probe sequence would be of the form . This method is called double hashing.

There are important restrictions on . Most importantly, the value returned by must never be zero (or ) because that will immediately lead to an infinite loop as the probe sequence makes no progress. However, a good implementation of double hashing should also ensure that all of the probe sequence constants are relatively prime to the table size . For example, if the hash table size were 100 and the step size for linear probing (as generated by function ) were 50, then there would be only one slot on the probe sequence. If instead the hash table size is 101 (a prime number), than any step size less than 101 will visit every slot in the table.

This can be achieved easily. One way is to select to be a prime number, and have return a value in the range . We can do this by using this secondary hash function: . An alternative is to set for some value and have return an odd value between 1 and . We can get that result with this secondary hash function: . [^1]

Now you can try it.

[^1]: The secondary hash function might seem rather mysterious, so let’s break this down. This is being used in the context of two facts: (1) We want the function to return an odd value that is less than the hash table size, and (2) we are using a hash table of size , which means that taking the mod of size is using the bottom bits of the key value. OK, since is multiplying something by 2 and adding 1, we guarentee that it is an odd number. Now, must be in the range 1 and (if you need to, play around with this on paper to convince yourself that this is true). This is exactly what we want. The last piece of the puzzle is the first part . That is not strictly necessary. But remember that since the table size is , this is the same as shifting the key value right by bits. In other words, we are not using the bottom bits to decide on the second hash function value, which is especially a good thing if we used the bottom bits to decide on the first hash function value! In other words, we really do not want the value of the step sized used by the linear probing to be fixed to the slot in the hash table that we chose. So we are using the next bits of the key value instead. Note that this would only be a good idea if we have keys in a large enough key range, that is, we want plenty of use of those second bits in the key range. This will be true if the max key value uses at least bits, meaning that the max key value should be at least the square of the hash table size. This is not a problem for typical hashing applications.

10.8. Analysis of Closed Hashing

10.8.1. Analysis of Closed Hashing

How efficient is hashing? We can measure hashing performance in terms of the number of record accesses required when performing an operation. The primary operations of concern are insertion, deletion, and search. It is useful to distinguish between successful and unsuccessful searches. Before a record can be deleted, it must be found. Thus, the number of accesses required to delete a record is equivalent to the number required to successfully search for it. To insert a record, an empty slot along the record’s probe sequence must be found. This is equivalent to an unsuccessful search for the record (recall that a successful search for the record during insertion should generate an error because two records with the same key are not allowed to be stored in the table).

When the hash table is empty, the first record inserted will always find its home position free. Thus, it will require only one record access to find a free slot. If all records are stored in their home positions, then successful searches will also require only one record access. As the table begins to fill up, the probability that a record can be inserted into its home position decreases. If a record hashes to an occupied slot, then the collision resolution policy must locate another slot in which to store it. Finding records not stored in their home position also requires additional record accesses as the record is searched for along its probe sequence. As the table fills up, more and more records are likely to be located ever further from their home positions.

From this discussion, we see that the expected cost of hashing is a function of how full the table is. Define the load factor for the table as , where is the number of records currently in the table.

An estimate of the expected cost for an insertion (or an unsuccessful search) can be derived analytically as a function of in the case where we assume that the probe sequence follows a random permutation of the slots in the hash table. Assuming that every slot in the table has equal probability of being the home slot for the next record, the probability of finding the home position occupied is . The probability of finding both the home position occupied and the next slot on the probe sequence occupied is . The probability of collisions is . If and are large, then this is approximately . The expected number of probes is one plus the sum over of the probability of collisions, which is approximately

The cost for a successful search (or a deletion) has the same cost as originally inserting that record. However, the expected value for the insertion cost depends on the value of not at the time of deletion, but rather at the time of the original insertion. We can derive an estimate of this cost (essentially an average over all the insertion costs) by integrating from 0 to the current value of , yielding a result of

It is important to realize that these equations represent the expected cost for operations when using the unrealistic assumption that the probe sequence is based on a random permutation of the slots in the hash table. We thereby avoid all the expense that results from a less-than-perfect collision resolution policy. Thus, these costs are lower-bound estimates in the average case. The true average cost under linear probing is for insertions or unsuccessful searches and for deletions or successful searches.

Figure 10.8.1: A plot showing the growth rate of the cost for insertion and deletion into a hash table as the load factor increases.

Figure 10.8.1 shows how the expected number of record accesses grows as grows. The horizontal axis is the value for , the vertical axis is the expected number of accesses to the hash table. Solid lines show the cost for “random” probing (a theoretical lower bound on the cost), while dashed lines show the cost for linear probing (a relatively poor collision resolution strategy). The two leftmost lines show the cost for insertion (equivalently, unsuccessful search); the two rightmost lines show the cost for deletion (equivalently, successful search).

From the figure, you should see that the cost for hashing when the table is not too full is typically close to one record access. This is extraordinarily efficient, much better than binary search which requires record accesses. As increases, so does the expected cost. For small values of , the expected cost is low. It remains below two until the hash table is about half full. When the table is nearly empty, adding a new record to the table does not increase the cost of future search operations by much. However, the additional search cost caused by each additional insertion increases rapidly once the table becomes half full. Based on this analysis, the rule of thumb is to design a hashing system so that the hash table never gets above about half full, because beyond that point performance will degrade rapidly. This requires that the implementor have some idea of how many records are likely to be in the table at maximum loading, and select the table size accordingly. The goal should be to make the table small enough so that it does not waste a lot of space on the one hand, while making it big enough to keep performance good on the other.

10.9. Deletion

10.9.1. Deletion

When deleting records from a hash table, there are two important considerations.

1. Deleting a record must not hinder later searches. In other words, the search process must still pass through the newly emptied slot to reach records whose probe sequence passed through this slot. Thus, the delete process cannot simply mark the slot as empty, because this will isolate records further down the probe sequence.
2. We do not want to make positions in the hash table unusable because of deletion. The freed slot should be available to a future insertion.

Both of these problems can be resolved by placing a special mark in place of the deleted record, called a tombstone. The tombstone indicates that a record once occupied the slot but does so no longer. If a tombstone is encountered when searching along a probe sequence, the search procedure continues with the search. When a tombstone is encountered during insertion, that slot can be used to store the new record. However, to avoid inserting duplicate keys, it will still be necessary for the search procedure to follow the probe sequence until a truly empty position has been found, simply to verify that a duplicate is not in the table. However, the new record would actually be inserted into the slot of the first tombstone encountered.

The use of tombstones allows searches to work correctly and allows reuse of deleted slots. However, after a series of intermixed insertion and deletion operations, some slots will contain tombstones. This will tend to lengthen the average distance from a record’s home position to the record itself, beyond where it could be if the tombstones did not exist. A typical database application will first load a collection of records into the hash table and then progress to a phase of intermixed insertions and deletions. After the table is loaded with the initial collection of records, the first few deletions will lengthen the average probe sequence distance for records (it will add tombstones). Over time, the average distance will reach an equilibrium point because insertions will tend to decrease the average distance by filling in tombstone slots. For example, after initially loading records into the database, the average path distance might be 1.2 (i.e., an average of 0.2 accesses per search beyond the home position will be required). After a series of insertions and deletions, this average distance might increase to 1.6 due to tombstones. This seems like a small increase, but it is three times longer on average beyond the home position than before deletions.

Two possible solutions to this problem are

1. Do a local reorganization upon deletion to try to shorten the average path length. For example, after deleting a key, continue to follow the probe sequence of that key and swap records further down the probe sequence into the slot of the recently deleted record (being careful not to remove any key from its probe sequence). This will not work for all collision resolution policies.
2. Periodically rehash the table by reinserting all records into a new hash table. Not only will this remove the tombstones, but it also provides an opportunity to place the most frequently accessed records into their home positions.

Now here are some practice questions.

Congratulations! You have reached the end of the hashing tutorial. In summary, a properly tuned hashing system will return records with an average cost of less than two record accesses. This makes it the most effective way known to store a database of records to support exact-match queries. Unfortunately, hashing is not effective when implementing range queries, or answering questions like “Which record in the collection has the smallest key value?”

10.10. Hashing Chapter Summary Exercises

10.10.1. Hashing Review

Here is a complete set of review questions, taken from all of the questions in the modules of this chapter. If anything goes wrong with one of the questions, or if you think that you are in a series of repeating questions, then just reload the page.

Chapter 11: Graphs

11.1. Graphs Chapter Introduction

11.1.1. Graph Terminology and Implementation

Graphs provide the ultimate in data structure flexibility. A graph consists of a set of nodes, and a set of edges where an edge connects two nodes. Trees and lists can be viewed as special cases of graphs.

Graphs are used to model both real-world systems and abstract problems, and are the data structure of choice in many applications. Here is a small sampling of the types of problems that graphs are routinely used for.

1. Modeling connectivity in computer and communications networks.
2. Representing an abstract map as a set of locations with distances between locations. This can be used to compute shortest routes between locations such as in a GPS routefinder.
3. Modeling flow capacities in transportation networks to find which links create the bottlenecks.
4. Finding a path from a starting condition to a goal condition. This is a common way to model problems in artificial intelligence applications and computerized game players.
5. Modeling computer algorithms, to show transitions from one program state to another.
6. Finding an acceptable order for finishing subtasks in a complex activity, such as constructing large buildings.
7. Modeling relationships such as family trees, business or military organizations, and scientific taxonomies.

The rest of this module covers some basic graph terminology. The following modules will describe fundamental representations for graphs, provide a reference implementation, and cover core graph algorithms including traversal, topological sort, shortest paths algorithms, and algorithms to find the minimal-cost spanning tree. Besides being useful and interesting in their own right, these algorithms illustrate the use of many other data structures presented throughout the course.

A graph consists of a set of vertices and a set of edges , such that each edge in is a connection between a pair of vertices in . [^1] The number of vertices is written , and the number of edges is written . can range from zero to a maximum of .

[^1]: Some graph applications require that a given pair of vertices can have multiple or parallel edges connecting them, or that a vertex can have an edge to itself. However, the applications discussed here do not require either of these special cases. To simplify our graph API, we will assume that there are no dupicate edges, and no edges that connect a node to itself.

A graph whose edges are not directed is called an undirected graph, as shown in part (a) of the following figure. A graph with edges directed from one vertex to another (as in (b)) is called a directed graph or digraph. A graph with labels associated with its vertices (as in (c)) is called a labeled graph. Associated with each edge may be a cost or weight. A graph whose edges have weights (as in (c)) is said to be a weighted graph.

Figure 11.1.1: Some types of graphs.

An edge connecting Vertices and is written . Such an edge is said to be incident with Vertices and . The two vertices are said to be adjacent. If the edge is directed from to , then we say that is adjacent to , and is adjacent from . The degree of a vertex is the number of edges it is incident with. For example, Vertex below has a degree of three.

In a directed graph, the out degree for a vertex is the number of neighbors adjacent from it (or the number of edges going out from it), while the in degree is the number of neighbors adjacent to it (or the number of edges coming in to it). In (c) above, the in degree of Vertex 1 is two, and its out degree is one.

A sequence of vertices forms a path of length if there exist edges from to for . A path is a simple path if all vertices on the path are distinct. The length of a path is the number of edges it contains. A cycle is a path of length three or more that connects some vertex to itself. A cycle is a simple cycle if the path is simple, except for the first and last vertices being the same.

An undirected graph is a connected graph if there is at least one path from any vertex to any other. The maximally connected subgraphs of an undirected graph are called connected components. For example, this figure shows an undirected graph with three connected components.

A graph with relatively few edges is called a sparse graph, while a graph with many edges is called a dense graph. A graph containing all possible edges is said to be a complete graph. A subgraph is formed from graph by selecting a subset of ’s vertices and a subset of ‘s edges such that for every edge , both vertices of are in . Any subgraph of where all vertices in the graph connect to all other vertices in the subgraph is called a clique.

A graph without cycles is called an acyclic graph. Thus, a directed graph without cycles is called a directed acyclic graph or DAG.

A free tree is a connected, undirected graph with no simple cycles. An equivalent definition is that a free tree is connected and has edges.

11.1.1.1. Graph Representations

There are two commonly used methods for representing graphs. The adjacency matrix for a graph is a array. We typically label the vertices from through . Row of the adjacency matrix contains entries for Vertex . Column in row is marked if there is an edge from to and is not marked otherwise. The space requirements for the adjacency matrix are .

The second common representation for graphs is the adjacency list. The adjacency list is an array of linked lists. The array is items long, with position storing a pointer to the linked list of edges for Vertex . This linked list represents the edges by the vertices that are adjacent to Vertex .

Here is an example of the two representations on a directed graph. The entry for Vertex 0 stores 1 and 4 because there are two edges in the graph leaving Vertex 0, with one going to Vertex 1 and one going to Vertex 4. The list for Vertex 2 stores an entry for Vertex 4 because there is an edge from Vertex 2 to Vertex 4, but no entry for Vertex 3 because this edge comes into Vertex 2 rather than going out.

Figure 11.1.7: Representing a directed graph.

Both the adjacency matrix and the adjacency list can be used to store directed or undirected graphs. Each edge of an undirected graph connecting Vertices and is represented by two directed edges: one from to and one from to . Here is an example of the two representations on an undirected graph. We see that there are twice as many edge entries in both the adjacency matrix and the adjacency list. For example, for the undirected graph, the list for Vertex 2 stores an entry for both Vertex 3 and Vertex 4.

Figure 11.1.8: Representing an undirected graph.

The storage requirements for the adjacency list depend on both the number of edges and the number of vertices in the graph. There must be an array entry for each vertex (even if the vertex is not adjacent to any other vertex and thus has no elements on its linked list), and each edge must appear on one of the lists. Thus, the cost is .

Sometimes we want to store weights or distances with each each edge, such as in Figure 11.1.1 (c). This is easy with the adjacency matrix, where we will just store values for the weights in the matrix. In Figures 11.1.7 and 11.1.8 we store a value of “1” at each position just to show that the edge exists. That could have been done using a single bit, but since bit manipulation is typically complicated in most programming languages, an implementation might store a byte or an integer at each matrix position. For a weighted graph, we would need to store at each position in the matrix enough space to represent the weight, which might typically be an integer.

The adjacency list needs to explicitly store a weight with each edge. In the adjacency list shown below, each linked list node is shown storing two values. The first is the index for the neighbor at the end of the associated edge. The second is the value for the weight. As with the adjacency matrix, this value requires space to represent, typically an integer.

Which graph representation is more space efficient depends on the number of edges in the graph. The adjacency list stores information only for those edges that actually appear in the graph, while the adjacency matrix requires space for each potential edge, whether it exists or not. However, the adjacency matrix requires no overhead for pointers, which can be a substantial cost, especially if the only information stored for an edge is one bit to indicate its existence. As the graph becomes denser, the adjacency matrix becomes relatively more space efficient. Sparse graphs are likely to have their adjacency list representation be more space efficient.

Example 11.1.1

Assume that a vertex index requires two bytes, a pointer requires four bytes, and an edge weight requires two bytes. Then, each link node in the adjacency list needs bytes. The adjacency matrix for the directed graph above requires bytes while the adjacency list requires bytes. For the undirected version of the graph above, the adjacency matrix requires the same space as before, while the adjacency list requires bytes (because there are now 12 edges represented instead of 6).

The adjacency matrix often requires a higher asymptotic cost for an algorithm than would result if the adjacency list were used. The reason is that it is common for a graph algorithm to visit each neighbor of each vertex. Using the adjacency list, only the actual edges connecting a vertex to its neighbors are examined. However, the adjacency matrix must look at each of its potential edges, yielding a total cost of time when the algorithm might otherwise require only time. This is a considerable disadvantage when the graph is sparse, but not when the graph is closer to full.

11.1.2. Graph Terminology Questions

11.2. Graph Implementations

We next turn to the problem of implementing a general-purpose graph class. There are two traditional approaches to representing graphs: The adjacency matrix and the adjacency list. In this module we will show actual implementations for each approach. We will begin with an interface defining an ADT for graphs that a given implementation must meet.

interface Graph { // Graph class ADT
  // Initialize the graph with some number of vertices
  void init(int n);

  // Return the number of vertices
  int nodeCount();

  // Return the current number of edges
  int edgeCount();

  // Get the value of node with index v
  Object getValue(int v);

  // Set the value of node with index v
  void setValue(int v, Object val);

  // Adds a new edge from node v to node w with weight wgt
  void addEdge(int v, int w, int wgt);

  // Get the weight value for an edge
  int weight(int v, int w);

  // Removes the edge from the graph.
  void removeEdge(int v, int w);

  // Returns true iff the graph has the edge
  boolean hasEdge(int v, int w);

  // Returns an array containing the indicies of the neighbors of v
  int[] neighbors(int v);
}

This ADT assumes that the number of vertices is fixed when the graph is created, but that edges can be added and removed. The init method sets (or resets) the number of nodes in the graph, and creates necessary space for the adjacency matrix or adjacency list.

Vertices are defined by an integer index value. In other words, there is a Vertex 0, Vertex 1, and so on through Vertex . We can assume that the graph’s client application stores any additional information of interest about a given vertex elsewhere, such as a name or application-dependent value. Note that in a language like Java or C++, this ADT would not be implemented using a language feature like a generic or template, because it is the Graph class users’ responsibility to maintain information related to the vertices themselves. The Graph class need have no knowledge of the type or content of the information associated with a vertex, only the index number for that vertex.

Interface Graph has methods to return the number of vertices and edges (methods n and e, respectively). Function weight returns the weight of a given edge, with that edge identified by its two incident vertices. For example, calling weight(0, 4) on the graph of Figure 11.1.1 (c) would return 4. If no such edge exists, the weight is defined to be 0. So calling weight(0, 2) on the graph of Figure 11.1.1 (c) would return 0.

Functions addEdge and removeEdge add an edge (setting its weight) and removes an edge from the graph, respectively. Again, an edge is identified by its two incident vertices. addEdge does not permit the user to set the weight to be 0, because this value is used to indicate a non-existent edge, nor are negative edge weights permitted. Functions getValue and setValue get and set, respectively, a requested value for Vertex . In our example applications the most frequent use of these methods will be to indicate whether a given node has previously been visited in the process of the algorithm

Nearly every graph algorithm presented in this chapter will require visits to all neighbors of a given vertex. The neighbors method returns an array containing the indices for the neighboring vertices, in ascending order. The following lines appear in many graph algorithms.

int[] nList = G.neighbors(v);
for (int i=0; i< nList.length; i++)
  if (G.getValue(nList[i]) != VISITED)
    DoSomething();

First, an array is generated that contains the indices of the nodes that can be directly reached from node v. The for loop then iterates through this neighbor array to execute some function on each.

It is reasonably straightforward to implement our graph ADT using either the adjacency list or adjacency matrix. The sample implementations presented here do not address the issue of how the graph is actually created. The user of these implementations must add functionality for this purpose, perhaps reading the graph description from a file. The graph can be built up by using the addEdge function provided by the ADT.

Here is an implementation for the adjacency matrix.

class GraphM implements Graph {
  private int[][] matrix;
  private Object[] nodeValues;
  private int numEdge;

  // No real constructor needed
  GraphM() { }

  // Initialize the graph with n vertices
  void init(int n) {
    matrix = new int[n][n];
    nodeValues = new Object[n];
    numEdge = 0;
  }

  // Return the number of vertices
  int nodeCount() { return nodeValues.length; }

  // Return the current number of edges
  int edgeCount() { return numEdge; }

  // Get the value of node with index v
  Object getValue(int v) { return nodeValues[v]; }

  // Set the value of node with index v
  void setValue(int v, Object val) { nodeValues[v] = val; }

  // Adds a new edge from node v to node w
  // Returns the new edge
  void addEdge(int v, int w, int wgt) {
    if (wgt == 0) return; // Can't store weight of 0
    matrix[v][w] = wgt;
    numEdge++;
  }

  // Get the weight value for an edge
  int weight(int v, int w) { return matrix[v][w]; }

  // Removes the edge from the graph.
  void removeEdge(int v, int w) {
    matrix[v][w] = 0;
    numEdge--;
  }

  // Returns true iff the graph has the edge
  boolean hasEdge(int v, int w) { return matrix[v][w] != 0; }

  // Returns an array containing the indicies of the neighbors of v
  int[] neighbors(int v) {
    int i;
    int count = 0;
    int[] temp;

    for (i=0; i<nodeValues.length; i++)
      if (matrix[v][i] != 0) count++;
    temp = new int[count];
    for (i=0, count=0; i<nodeValues.length; i++)
      if (matrix[v][i] != 0) temp[count++] = i;
    return temp;
  }
}

Array nodeValues stores the information manipulated by the setValue and getValue functions. The edge matrix is implemented as an integer array of size for a graph of vertices. Position in the matrix stores the weight for edge if it exists. A weight of zero for edge is used to indicate that no edge connects Vertices and .

Given a vertex , the neighbors method scans through row v of the matix to locate the positions of the various neighbors. If no edge is incident on , then returned neighbor array will have length 0. Functions addEdge and removeEdge adjust the appropriate value in the array. Function weight returns the value stored in the appropriate position in the array.

Here is an implementation of the adjacency list representation for graphs. Its main data structure is an array of linked lists, one linked list for each vertex. These linked lists store objects of type Edge, which merely stores the index for the vertex pointed to by the edge, along with the weight of the edge.

class GraphL implements Graph {

  private class Edge { // Doubly linked list node
    int vertex, weight;
    Edge prev, next;

    Edge(int v, int w, Edge p, Edge n) {
      vertex = v;
      weight = w;
      prev = p;
      next = n;
    }
  }

  private Edge[] nodeArray;
  private Object[] nodeValues;
  private int numEdge;

  // No real constructor needed
  GraphL() {}

  // Initialize the graph with n vertices
  void init(int n) {
    nodeArray = new Edge[n];
    // List headers;
    for (int i=0; i<n; i++) nodeArray[i] = new Edge(-1, -1, null, null);
    nodeValues = new Object[n];
    numEdge = 0;
  }

  // Return the number of vertices
  int nodeCount() { return nodeArray.length; }

  // Return the current number of edges
  int edgeCount() { return numEdge; }

  // Get the value of node with index v
  Object getValue(int v) { return nodeValues[v]; }

  // Set the value of node with index v
  void setValue(int v, Object val) { nodeValues[v] = val; }

  // Return the link in v's neighbor list that preceeds the
  // one with w (or where it would be)
  private Edge find (int v, int w) {
    Edge curr = nodeArray[v];
    while ((curr.next != null) && (curr.next.vertex < w))
      curr = curr.next;
    return curr;
  }

  // Adds a new edge from node v to node w with weight wgt
  void addEdge(int v, int w, int wgt) {
    if (wgt == 0) return; // Can't store weight of 0
    Edge curr = find(v, w);
    if ((curr.next != null) && (curr.next.vertex == w))
      curr.next.weight = wgt;
    else {
      curr.next = new Edge(w, wgt, curr, curr.next);
      if (curr.next.next != null) curr.next.next.prev = curr.next;
    }
    numEdge++;
  }

  // Get the weight value for an edge
  int weight(int v, int w) {
    Edge curr = find(v, w);
    if ((curr.next == null) || (curr.next.vertex != w)) return 0;
    else return curr.next.weight;
  }

  // Removes the edge from the graph.
  void removeEdge(int v, int w) {
    Edge curr = find(v, w);
    if ((curr.next == null) || curr.next.vertex != w) return;
    else {
      curr.next = curr.next.next;
      if (curr.next != null) curr.next.prev = curr;
    }
    numEdge--;
  }

  // Returns true iff the graph has the edge
  boolean hasEdge(int v, int w) { return weight(v, w) != 0; }

  // Returns an array containing the indicies of the neighbors of v
  int[] neighbors(int v) {
    int cnt = 0;
    Edge curr;
    for (curr = nodeArray[v].next; curr != null; curr = curr.next)
      cnt++;
    int[] temp = new int[cnt];
    cnt = 0;
    for (curr = nodeArray[v].next; curr != null; curr = curr.next)
      temp[cnt++] = curr.vertex;
    return temp;
  }
}

Implementation for GraphL member functions is straightforward in principle, with the key functions being addEdge, removeEdge, and weight. They simply start at the beginning of the adjacency list and move along it until the desired vertex has been found. Private method find is a utility for finding the last edge preceding the one that holds vertex if that exists.

11.3. Graph Traversals

11.3.1. Graph Traversals

Many graph applications need to visit the vertices of a graph in some specific order based on the graph’s topology. This is known as a graph traversal and is similar in concept to a tree traversal. Recall that tree traversals visit every node exactly once, in some specified order such as preorder, inorder, or postorder. Multiple tree traversals exist because various applications require the nodes to be visited in a particular order. For example, to print a BST’s nodes in ascending order requires an inorder traversal as opposed to some other traversal. Standard graph traversal orders also exist. Each is appropriate for solving certain problems. For example, many problems in artificial intelligence programming are modeled using graphs. The problem domain might consist of a large collection of states, with connections between various pairs of states. Solving this sort of problem requires getting from a specified start state to a specified goal state by moving between states only through the connections. Typically, the start and goal states are not directly connected. To solve this problem, the vertices of the graph must be searched in some organized manner.

Graph traversal algorithms typically begin with a start vertex and attempt to visit the remaining vertices from there. Graph traversals must deal with a number of troublesome cases. First, it might not be possible to reach all vertices from the start vertex. This occurs when the graph is not connected. Second, the graph might contain cycles, and we must make sure that cycles do not cause the algorithm to go into an infinite loop.

Graph traversal algorithms can solve both of these problems by flagging vertices as VISITED when appropriate. At the beginning of the algorithm, no vertex is flagged as VISITED. The flag for a vertex is set when the vertex is first visited during the traversal. If a flagged vertex is encountered during traversal, it is not visited a second time. This keeps the program from going into an infinite loop when it encounters a cycle.

Once the traversal algorithm completes, we can check to see if all vertices have been processed by checking whether they have the VISITED flag set. If not all vertices are flagged, we can continue the traversal from another unvisited vertex. Note that this process works regardless of whether the graph is directed or undirected. To ensure visiting all vertices, graphTraverse could be called as follows on a graph :

void graphTraverse(Graph G) {
  int v;
  for (v=0; v<G.nodeCount(); v++)
    G.setValue(v, null); // Initialize
  for (v=0; v<G.nodeCount(); v++)
    if (G.getValue(v) != VISITED)
      doTraversal(G, v);
}

Function doTraversal might be implemented by using one of the graph traversals described next.

11.3.1.1. Depth-First Search

Our first method for organized graph traversal is called depth-first search (DFS). Whenever a vertex is visited during the search, DFS will recursively visit all of ‘s unvisited neighbors. Equivalently, DFS will add all edges leading out of to a stack. The next vertex to be visited is determined by popping the stack and following that edge. The effect is to follow one branch through the graph to its conclusion, then it will back up and follow another branch, and so on. The DFS process can be used to define a depth-first search tree. This tree is composed of the edges that were followed to any new (unvisited) vertex during the traversal, and leaves out the edges that lead to already visited vertices. DFS can be applied to directed or undirected graphs.

This visualization shows a graph and the result of performing a DFS on it, resulting in a depth-first search tree.

Here is an implementation for the DFS algorithm.

void DFS(Graph G, int v) {
  PreVisit(G, v);
  G.setValue(v, VISITED);
  int[] nList = G.neighbors(v);
  for (int i=0; i< nList.length; i++)
    if (G.getValue(nList[i]) != VISITED)
      DFS(G, nList[i]);
  PostVisit(G, v);
}

This implementation contains calls to functions PreVisit and PostVisit. These functions specify what activity should take place during the search. Just as a preorder tree traversal requires action before the subtrees are visited, some graph traversals require that a vertex be processed before ones further along in the DFS. Alternatively, some applications require activity after the remaining vertices are processed; hence the call to function PostVisit. This would be a natural opportunity to make use of the visitor design pattern.

The following visualization shows a random graph each time that you start it, so that you can see the behavior on different examples. It can show you DFS run on a directed graph or an undirected graph. Be sure to look at an example for each type of graph.

DFS processes each edge once in a directed graph. In an undirected graph, DFS processes each edge from both directions. Each vertex must be visited, but only once, so the total cost is .

Here is an exercise for you to practice DFS.

11.3.2. Breadth-First Search

Our second graph traversal algorithm is known as a breadth-first search (BFS). BFS examines all vertices connected to the start vertex before visiting vertices further away. BFS is implemented similarly to DFS, except that a queue replaces the recursion stack. Note that if the graph is a tree and the start vertex is at the root, BFS is equivalent to visiting vertices level by level from top to bottom.

This visualization shows a graph and the result of performing a BFS on it, resulting in a breadth-first search tree.

Here is an implementation for BFS.

void BFS(Graph G, int v) {
  LQueue Q = new LQueue(G.nodeCount());
  Q.enqueue(v);
  G.setValue(v, VISITED);
  while (Q.length() > 0) { // Process each vertex on Q
    v = (Integer)Q.dequeue();
    PreVisit(G, v);
    int[] nList = G.neighbors(v);
    for (int i=0; i< nList.length; i++)
      if (G.getValue(nList[i]) != VISITED) { // Put neighbors on Q
        G.setValue(nList[i], VISITED);
        Q.enqueue(nList[i]);
      }
    PostVisit(G, v);
  }
}

The following visualization shows a random graph each time that you start it, so that you can see the behavior on different examples. It can show you BFS run on a directed graph or an undirected graph. Be sure to look at an example for each type of graph.

Here is an exercise for you to practice BFS.

11.4. Topological Sort

11.4.1. Topological Sort

Assume that we need to schedule a series of tasks, such as classes or construction jobs, where we cannot start one task until after its prerequisites are completed. We wish to organize the tasks into a linear order that allows us to complete them one at a time without violating any prerequisites. We can model the problem using a DAG. The graph is directed because one task is a prerequisite of another – the vertices have a directed relationship. It is acyclic because a cycle would indicate a conflicting series of prerequisites that could not be completed without violating at least one prerequisite. The process of laying out the vertices of a DAG in a linear order to meet the prerequisite rules is called a topological sort.

Figure 11.4.1: An example graph for topological sort. Seven tasks have dependencies as shown by the directed graph.

Figure 11.4.1 illustrates the problem. An acceptable topological sort for this example is J1, J2, J3, J4, J5, J6, J7. However, other orders are also acceptable, such as J1, J3, J2, J6, J4, J5, J7.

11.4.1.1. Depth-first solution

A topological sort may be found by performing a DFS on the graph. When a vertex is visited, no action is taken (i.e., function PreVisit does nothing). When the recursion pops back to that vertex, function PostVisit prints the vertex. This yields a topological sort in reverse order. It does not matter where the sort starts, as long as all vertices are visited in the end. Here is implementation for the DFS-based algorithm.

void topsortDFS(Graph G) {
  int v;
  for (v=0; v<G.nodeCount(); v++)
    G.setValue(v, null); // Initialize
  for (v=0; v<G.nodeCount(); v++)
    if (G.getValue(v) != VISITED)
      tophelp(G, v);
}

void tophelp(Graph G, int v) {
  G.setValue(v, VISITED);
  int[] nList = G.neighbors(v);
  for (int i=0; i< nList.length; i++)
    if (G.getValue(nList[i]) != VISITED)
      tophelp(G, nList[i]);
  printout(v);
}

Using this algorithm starting at J1 and visiting adjacent neighbors in alphabetic order, vertices of the graph in Figure 11.4.1 are printed out in the order J7, J5, J4, J6, J2, J3, J1. Reversing this yields the topological sort J1, J3, J2, J6, J4, J5, J7.

Here is another example.

11.4.1.2. Queue-based Solution

We can implement topological sort using a queue instead of recursion, as follows.

First visit all edges, counting the number of edges that lead to each vertex (i.e., count the number of prerequisites for each vertex). All vertices with no prerequisites are placed on the queue. We then begin processing the queue. When Vertex is taken off of the queue, it is printed, and all neighbors of (that is, all vertices that have as a prerequisite) have their counts decremented by one. Place on the queue any neighbor whose count becomes zero. If the queue becomes empty without printing all of the vertices, then the graph contains a cycle (i.e., there is no possible ordering for the tasks that does not violate some prerequisite). The printed order for the vertices of the graph in Applying the queue version of topological sort to the graph of Figure 11.4.1 produces J1, J2, J3, J6, J4, J5, J7. Here is an implementation for the algorithm.

Here is the code to implement the queue-based topological sort:

void topsortBFS(Graph G) {          // Topological sort: Queue
  Queue Q = new LQueue(G.nodeCount());
  int[] Count = new int[G.nodeCount()];
  int[] nList;
  int v;
  for (v=0; v<G.nodeCount(); v++) Count[v] = 0; // Initialize
  for (v=0; v<G.nodeCount(); v++) { // Process every edge
    nList = G.neighbors(v);
    for (int i=0; i< nList.length; i++)
      Count[nList[i]]++;            // Add to v's prereq count
  }
  for (v=0; v<G.nodeCount(); v++)   // Initialize Queue
    if (Count[v] == 0)              // V has no prerequisites
      Q.enqueue(v);
  while (Q.length() > 0) {          // Process the vertices
    v = (Integer)Q.dequeue();
    printout(v);                    // PreVisit for Vertex V
    nList = G.neighbors(v);
    for (int i=0; i< nList.length; i++) {
      Count[nList[i]]--;            // One less prerequisite
      if (Count[nList[i]] == 0)     // This vertex is now free
        Q.enqueue(nList[i]);
    }
  }
}

11.5. Shortest-Paths Problems

11.5.1. Shortest-Paths Problems

On a road map, a road connecting two towns is typically labeled with its distance. We can model a road network as a directed graph whose edges are labeled with real numbers. These numbers represent the distance (or other cost metric, such as travel time) between two vertices. These labels may be called weights, costs, or distances, depending on the application. Given such a graph, a typical problem is to find the total length of the shortest path between two specified vertices. This is not a trivial problem, because the shortest path may not be along the edge (if any) connecting two vertices, but rather may be along a path involving one or more intermediate vertices.

For example, in Figure 11.5.1, the cost of the path from to to is 15. The cost of the edge directly from to is 20. The cost of the path from to to to is 10. Thus, the shortest path from to is 10 (rather than along the edge connecting to ). We use the notation to indicate that the shortest distance from to is 10. In Figure 11.5.1, there is no path from to , so we set . We define to be the weight of edge , that is, the weight of the direct connection from to . Because there is no edge from to , . Note that because the graph of Figure 11.5.1 is directed. We assume that all weights are positive.

Figure 11.5.1: Example graph for shortest-path definitions.

11.5.1.1. Single-Source Shortest Paths

We will now present an algorithm to solve the single-source shortest paths problem. Given Vertex in Graph , find a shortest path from to every other vertex in . We might want only the shortest path between two vertices, and . However in the worst case, finding the shortest path from to requires us to find the shortest paths from to every other vertex as well. So there is no better algorithm (in the worst case) for finding the shortest path to a single vertex than to find shortest paths to all vertices. The algorithm described here will only compute the distance to every such vertex, rather than recording the actual path. Recording the path requires only simple modifications to the algorithm.

Computer networks provide an application for the single-source shortest-paths problem. The goal is to find the cheapest way for one computer to broadcast a message to all other computers on the network. The network can be modeled by a graph with edge weights indicating time or cost to send a message to a neighboring computer.

For unweighted graphs (or whenever all edges have the same cost), the single-source shortest paths can be found using a simple breadth-first search. When weights are added, BFS will not give the correct answer.

One approach to solving this problem when the edges have differing weights might be to process the vertices in a fixed order. Label the vertices to , with . When processing Vertex , we take the edge connecting and . When processing , we consider the shortest distance from to and compare that to the shortest distance from to to . When processing Vertex , we consider the shortest path for Vertices through that have already been processed. Unfortunately, the true shortest path to might go through Vertex for . Such a path will not be considered by this algorithm. However, the problem would not occur if we process the vertices in order of distance from . Assume that we have processed in order of distance from to the first vertices that are closest to ; call this set of vertices . We are now about to process the th closest vertex; call it .

A shortest path from to must have its next-to-last vertex in . Thus,

In other words, the shortest path from to is the minimum over all paths that go from to , then have an edge from to , where is some vertex in .

This solution is usually referred to as Dijkstra’s algorithm. It works by maintaining a distance estimate for all vertices in . The elements of are initialized to the value INFINITE. Vertices are processed in order of distance from . Whenever a vertex is processed, is updated for every neighbor of . Here is an implementation for Dijkstra’s algorithm. At the end, array D will contain the shortest distance values.

// Compute shortest path distances from s, store them in D
void Dijkstra(Graph G, int s, int[] D) {
  for (int i=0; i<G.nodeCount(); i++)    // Initialize
    D[i] = INFINITY;
  D[s] = 0;
  for (int i=0; i<G.nodeCount(); i++) {  // Process the vertices
    int v = minVertex(G, D);     // Find next-closest vertex
    G.setValue(v, VISITED);
    if (D[v] == INFINITY) return; // Unreachable
    int[] nList = G.neighbors(v);
    for (int j=0; j<nList.length; j++) {
      int w = nList[j];
      if (D[w] > (D[v] + G.weight(v, w)))
        D[w] = D[v] + G.weight(v, w);
    }
  }
}

There are two reasonable solutions to the key issue of finding the unvisited vertex with minimum distance value during each pass through the main for loop. The first method is simply to scan through the list of vertices searching for the minimum value, as follows:

// Find the unvisited vertex with the smalled distance
int minVertex(Graph G, int[] D) {
  int v = 0;  // Initialize v to any unvisited vertex;
  for (int i=0; i<G.nodeCount(); i++)
    if (G.getValue(i) != VISITED) { v = i; break; }
  for (int i=0; i<G.nodeCount(); i++)  // Now find smallest value
    if ((G.getValue(i) != VISITED) && (D[i] < D[v]))
      v = i;
  return v;
}

Because this scan is done times, and because each edge requires a constant-time update to D, the total cost for this approach is , because is in .

An alternative approach is to store unprocessed vertices in a min-heap ordered by their distance from the processed vertices. The next-closest vertex can be found in the heap in time. Every time we modify , we could reorder in the heap by deleting and reinserting it. This is an example of a priority queue with priority update. To implement true priority updating, we would need to store with each vertex its position within the heap so that we can remove its old distances whenever it is updated by processing new edges. A simpler approach is to add the new (always smaller) distance value for a given vertex as a new record in the heap. The smallest value for a given vertex currently in the heap will be found first, and greater distance values found later will be ignored because the vertex will already be marked as VISITED. The only disadvantage to repeatedly inserting distance values in this way is that it will raise the number of elements in the heap from to in the worst case. But in practice this only adds a slight increase to the depth of the heap. The time complexity is , because for each edge that we process we must reorder the heap. We use the KVPair class to store key-value pairs in the heap, with the edge weight as the key and the target vertex as the value. here is the implementation for Dijkstra’s algorithm using a heap.

// Dijkstra's shortest-paths: priority queue version
void DijkstraPQ(Graph G, int s, int[] D) {
  int v;                                 // The current vertex
  KVPair[] E = new KVPair[G.edgeCount()];        // Heap for edges
  E[0] = new KVPair(0, s);               // Initial vertex
  MinHeap H = new MinHeap(E, 1, G.edgeCount());
  for (int i=0; i<G.nodeCount(); i++)            // Initialize distance
    D[i] = INFINITY;
  D[s] = 0;
  for (int i=0; i<G.nodeCount(); i++) {          // For each vertex
    do { KVPair temp = H.removemin();
         if (temp == null) return;       // Unreachable nodes exist
         v = (Integer)temp.value(); } // Get position
      while (G.getValue(v) == VISITED);
    G.setValue(v, VISITED);
    if (D[v] == INFINITY) return;        // Unreachable
    int[] nList = G.neighbors(v);
    for (int j=0; j<nList.length; j++) {
      int w = nList[j];
      if (D[w] > (D[v] + G.weight(v, w))) { // Update D
        D[w] = D[v] + G.weight(v, w);
        H.insert(D[w], w);
      }
    }
  }
}

Using MinVertex to scan the vertex list for the minimum value is more efficient when the graph is dense, that is, when approaches . Using a heap is more efficient when the graph is sparse because its cost is . However, when the graph is dense, this cost can become as great as .

Now you can practice using Dijkstra’s algorithm.

11.6. Minimal Cost Spanning Trees

11.6.1. Minimal Cost Spanning Trees

The minimal-cost spanning tree (MCST) problem takes as input a connected, undirected graph , where each edge has a distance or weight measure attached. The MCST is the graph containing the vertices of along with the subset of ‘s edges that (1) has minimum total cost as measured by summing the values for all of the edges in the subset, and (2) keeps the vertices connected. Applications where a solution to this problem is useful include soldering the shortest set of wires needed to connect a set of terminals on a circuit board, and connecting a set of cities by telephone lines in such a way as to require the least amount of cable.

The MCST contains no cycles. If a proposed MCST did have a cycle, a cheaper MCST could be had by removing any one of the edges in the cycle. Thus, the MCST is a free tree with edges. The name “minimum-cost spanning tree” comes from the fact that the required set of edges forms a tree, it spans the vertices (i.e., it connects them together), and it has minimum cost. Figure 11.6.1 shows the MCST for an example graph.

Figure 11.6.1: A graph and its MCST. All edges appear in the original graph. Those edges drawn with heavy lines indicate the subset making up the MCST. Note that edge could be replaced with edge to form a different MCST with equal cost.

11.6.1.1. Prim’s Algorithm

The first of our two algorithms for finding MCSTs is commonly referred to as Prim’s algorithm. Prim’s algorithm is very simple. Start with any Vertex in the graph, setting the MCST to be initially. Pick the least-cost edge connected to . This edge connects to another vertex; call this . Add Vertex and Edge to the MCST. Next, pick the least-cost edge coming from either or to any other vertex in the graph. Add this edge and the new vertex it reaches to the MCST. This process continues, at each step expanding the MCST by selecting the least-cost edge from a vertex currently in the MCST to a vertex not currently in the MCST.

Prim’s algorithm is quite similar to Dijkstra’s algorithm for finding the single-source shortest paths. The primary difference is that we are seeking not the next closest vertex to the start vertex, but rather the next closest vertex to any vertex currently in the MCST. Thus we replace the lines:

if (D[w] > (D[v] + G.weight(v, w)))
  D[w] = D[v] + G.weight(v, w);

in Djikstra’s algorithm with the lines:

if (D[w] > G.weight(v, w))
  D[w] = G.weight(v, w);

in Prim’s algorithm.

The following code shows an implementation for Prim’s algorithm that searches the distance matrix for the next closest vertex.

// Compute shortest distances to the MCST, store them in D.
// V[i] will hold the index for the vertex that is i's parent in the MCST
void Prim(Graph G, int s, int[] D, int[] V) {
  for (int i=0; i<G.nodeCount(); i++)    // Initialize
    D[i] = INFINITY;
  D[s] = 0;
  for (int i=0; i<G.nodeCount(); i++) {  // Process the vertices
    int v = minVertex(G, D);     // Find next-closest vertex
    G.setValue(v, VISITED);
    if (D[v] == INFINITY) return; // Unreachable
    if (v != s) AddEdgetoMST(V[v], v);
    int[] nList = G.neighbors(v);
    for (int j=0; j<nList.length; j++) {
      int w = nList[j];
      if (D[w] > G.weight(v, w)) {
        D[w] = G.weight(v, w);
        V[w] = v;
      }
    }
  }
}

For each vertex , when is processed by Prim’s algorithm, an edge going to is added to the MCST that we are building. Array V[I] stores the previously visited vertex that is closest to Vertex I. This information lets us know which edge goes into the MCST when Vertex is processed. The implementation above also contains calls to AddEdgetoMST to indicate which edges are actually added to the MCST.

11.6.1.2. Prim’s Algorithm Alternative Implementation

Alternatively, we can implement Prim’s algorithm using a priority queue to find the next closest vertex, as shown next. As with the priority queue version of Dijkstra’s algorithm, the heap stores DijkElem objects.

// Prims MCST algorithm: priority queue version
void PrimPQ(Graph G, int s, int[] D, int[] V) {
  int v;                                 // The current vertex
  KVPair[] E = new KVPair[G.edgeCount()];        // Heap for edges
  E[0] = new KVPair(0, s);               // Initial vertex
  MinHeap H = new MinHeap(E, 1, G.edgeCount());
  for (int i=0; i<G.nodeCount(); i++)            // Initialize distance
    D[i] = INFINITY;
  D[s] = 0;
  for (int i=0; i<G.nodeCount(); i++) {          // For each vertex
    do { KVPair temp = H.removemin();
         if (temp == null) return;       // Unreachable nodes exist
         v = (Integer)temp.value(); } // Get position
      while (G.getValue(v) == VISITED);
    G.setValue(v, VISITED);
    if (D[v] == INFINITY) return;  // Unreachable
    if (v != s) AddEdgetoMST(V[v], v); // Add edge to MST
    int[] nList = G.neighbors(v);
    for (int j=0; j<nList.length; j++) {
      int w = nList[j];
      if (D[w] > G.weight(v, w)) { // Update D
        D[w] = G.weight(v, w);
        V[w] = v;                  // Where it came from
        H.insert(D[w], w);
      }
    }
  }
}

Prim’s algorithm is an example of a greedy algorithm. At each step in the for loop, we select the least-cost edge that connects some marked vertex to some unmarked vertex. The algorithm does not otherwise check that the MCST really should include this least-cost edge. This leads to an important question: Does Prim’s algorithm work correctly? Clearly it generates a spanning tree (because each pass through the for loop adds one edge and one unmarked vertex to the spanning tree until all vertices have been added), but does this tree have minimum cost?

Theorem: Prim’s algorithm produces a minimum-cost spanning tree.

Proof: We will use a proof by contradiction. Let be a graph for which Prim’s algorithm does not generate an MCST. Define an ordering on the vertices according to the order in which they were added by Prim’s algorithm to the MCST: . Let edge connect for some and . Let be the lowest numbered (first) edge added by Prim’s algorithm such that the set of edges selected so far cannot be extended to form an MCST for . In other words, is the first edge where Prim’s algorithm “went wrong.” Let be the “true” MCST. Call the vertex connected by edge , that is, .

Because is a tree, there exists some path in connecting and . There must be some edge in this path connecting vertices and , with and . Because is not part of , adding edge to forms a cycle. Edge must be of lower cost than edge , because Prim’s algorithm did not generate an MCST. This situation is illustrated in Figure 11.6.2. However, Prim’s algorithm would have selected the least-cost edge available. It would have selected , not . Thus, it is a contradiction that Prim’s algorithm would have selected the wrong edge, and thus, Prim’s algorithm must be correct. BOX HERE

Figure 11.6.2: Prim’s MCST algorithm proof. The left oval contains that portion of the graph where Prim’s MCST and the “true” MCST agree. The right oval contains the rest of the graph. The two portions of the graph are connected by (at least) edges (selected by Prim’s algorithm to be in the MCST) and (the “correct” edge to be placed in the MCST). Note that the path from to cannot include any marked vertex , because to do so would form a cycle.

11.7. Kruskal’s Algorithm

11.7.1. Kruskal’s Algorithm

Our next MCST algorithm is commonly referred to as Kruskal’s algorithm. Kruskal’s algorithm is also a simple, greedy algorithm. First partition the set of vertices into disjoint sets, each consisting of one vertex. Then process the edges in order of weight. An edge is added to the MCST, and two disjoint sets combined, if the edge connects two vertices in different disjoint sets. This process is repeated until only one disjoint set remains.

The edges can be processed in order of weight by using a min-heap. This is generally faster than sorting the edges first, because in practice we need only visit a small fraction of the edges before completing the MCST. This is an example of finding only a few smallest elements in a list.

The only tricky part to this algorithm is determining if two vertices belong to the same equivalence class. Fortunately, the ideal algorithm is available for the purpose — the UNION/FIND. Here is an implementation for Kruskal’s algorithm. Class KruskalElem is used to store the edges on the min-heap.

// Kruskal's MST algorithm
void Kruskal(Graph G) {
  ParPtrTree A = new ParPtrTree(G.nodeCount()); // Equivalence array
  KVPair[] E = new KVPair[G.edgeCount()];       // Minheap array
  int edgecnt = 0; // Count of edges

  for (int i=0; i<G.nodeCount(); i++) {         // Put edges in the array
    int[] nList = G.neighbors(i);
    for (int w=0; w<nList.length; w++)
      E[edgecnt++] = new KVPair(G.weight(i, nList[w]), new int[]{i,nList[w]});
  }
  MinHeap H = new MinHeap(E, edgecnt, edgecnt);
  int numMST = G.nodeCount();                   // Initially n disjoint classes
  for (int i=0; numMST>1; i++) {        // Combine equivalence classes
    KVPair temp = H.removemin();        // Next cheapest edge
    if (temp == null) return;           // Must have disconnected vertices
    int v = ((int[])temp.value())[0];
    int u = ((int[])temp.value())[1];
    if (A.differ(v, u)) {               // If in different classes
      A.UNION(v, u);                    // Combine equiv classes
      AddEdgetoMST(v, u);               // Add this edge to MST
      numMST--;                         // One less MST
    }
  }
}

Kruskal’s algorithm is dominated by the time required to process the edges. The differ and UNION functions are nearly constant in time if path compression and weighted union is used. Thus, the total cost of the algorithm is in the worst case, when nearly all edges must be processed before all the edges of the spanning tree are found and the algorithm can stop. More often the edges of the spanning tree are the shorter ones,and only about edges must be processed. If so, the cost is often close to in the average case.

11.8. All-Pairs Shortest Paths

We next consider the problem of finding the shortest distance between all pairs of vertices in the graph, called the all-pairs shortest paths problem. To be precise, for every , calculate .

One solution is to run Dijkstra’s algorithm for finding the shortest path times, each time computing the shortest path from a different start vertex. If is sparse (that is, ) then this is a good solution, because the total cost will be for the version of Dijkstra’s algorithm based on priority queues. For a dense graph, the priority queue version of Dijkstra’s algorithm yields a cost of , but the version using MinVertex yields a cost of .

Another solution that limits processing time to regardless of the number of edges is known as Floyd’s algorithm. It is an example of dynamic programming. The chief problem with solving this problem is organizing the search process so that we do not repeatedly solve the same subproblems. We will do this organization through the use of the -path. Define a k-path from vertex to vertex to be any path whose intermediate vertices (aside from and ) all have indices less than . A 0-path is defined to be a direct edge from to . Figure 11.8.1 illustrates the concept of -paths.

{width=40%}

Figure 11.8.1: An example of -paths in Floyd’s algorithm. Path 1, 3 is a 0-path by definition. Path 3, 0, 2 is not a 0-path, but it is a 1-path (as well as a 2-path, a 3-path, and a 4-path) because the largest intermediate vertex is 0. Path 1, 3, 2 is a 4-path, but not a 3-path because the intermediate vertex is 3. All paths in this graph are 4-paths.

Define to be the length of the shortest -path from vertex to vertex . Assume that we already know the shortest -path from to . The shortest -path either goes through vertex or it does not. If it does go through , then the best path is the best -path from to followed by the best -path from to . Otherwise, we should keep the best -path seen before. Floyd’s algorithm simply checks all of the possibilities in a triple loop. Here is the implementation for Floyd’s algorithm. At the end of the algorithm, array D stores the all-pairs shortest distances.

/** Compute all-pairs shortest paths */
static void Floyd(Graph G, int[][] D) {
  for (int i=0; i<G.n(); i++) // Initialize D with weights
    for (int j=0; j<G.n(); j++)
      if (G.weight(i, j) != 0) D[i][j] = G.weight(i, j);
  for (int k=0; k<G.n(); k++) // Compute all k paths
    for (int i=0; i<G.n(); i++)
      for (int j=0; j<G.n(); j++)
        if ((D[i][k] != Integer.MAX_VALUE) &&
            (D[k][j] != Integer.MAX_VALUE) &&
            (D[i][j] > (D[i][k] + D[k][j])))
          D[i][j] = D[i][k] + D[k][j];
}

Clearly this algorithm requires running time, and it is the best choice for dense graphs because it is (relatively) fast and easy to implement.

Chapter 12: Sorting

12.1. Chapter Introduction: Sorting

We sort many things in our everyday lives: A handful of cards when playing Bridge; bills and other piles of paper; jars of spices; and so on. And we have many intuitive strategies that we can use to do the sorting, depending on how many objects we have to sort and how hard they are to move around. Sorting is also one of the most frequently performed computing tasks. We might sort the records in a database so that we can search the collection efficiently. We might sort customer records by zip code so that when we print an advertisement we can then mail them more cheaply. We might use sorting to help an algorithm to solve some other problem. For example, Kruskal’s algorithm to find a minimal-cost spanning tree must sort the edges of a graph by their lengths before it can process them.

Because sorting is so important, naturally it has been studied intensively and many algorithms have been devised. Some of these algorithms are straightforward adaptations of schemes we use in everyday life. For example, a natural way to sort your cards in a bridge hand is to go from left to right, and place each card in turn in its correct position relative to the other cards that you have already sorted. This is the idea behind Insertion Sort. Other sorting algorithms are totally alien to how humans do things, having been invented to sort thousands or even millions of records stored on the computer. For example, no normal person would use Quicksort to order a pile of bills by date, even though Quicksort is the standard sorting algorithm of choice for most software libraries. After years of study, there are still unsolved problems related to sorting. New algorithms are still being developed and refined for special-purpose applications.

Along with introducing this central problem in computer science, studying sorting algorithms helps us to understand issues in algorithm design and analysis. For example, the sorting algorithms in this chapter show multiple approaches to using divide and conquer. In particular, there are multiple ways to do the dividing. Mergesort divides a list in half. Quicksort divides a list into big values and small values. Radix Sort divides the problem by working on one digit of the key at a time. Sorting algorithms can also illustrate a wide variety of algorithm analysis techniques. Quicksort illustrates that it is possible for an algorithm to have an average case whose growth rate is significantly smaller than its worst case. It is possible to speed up one sorting algorithm (such as Shellsort or Quicksort) by taking advantage of the best case behavior of another algorithm (Insertion Sort). Special case behavior by some sorting algorithms makes them a good solution for special niche applications (Heapsort). Sorting provides an example of an important technique for analyzing the lower bound for a problem. External Sorting refers to the process of sorting large files stored on disk.

This chapter covers several standard algorithms appropriate for sorting a collection of records that fit into the computer’s main memory. It begins with a discussion of three simple, but relatively slow, algorithms that require time in the average and worst cases to sort records. Several algorithms with considerably better performance are then presented, some with worst-case running time. The final sorting method presented requires only worst-case time under special conditions (but it cannot run that fast in the general case). The chapter concludes with a proof that sorting in general requires time in the worst case.

12.2. Sorting Terminology and Notation

12.2.1. Sorting Terminology and Notation

Given a set of records , , …, with associated key values , , …, , the Sorting Problem is to arrange the records into any order such that records , , …, have keys obeying the property . In other words, the sorting problem is to arrange a set of records so that the values of their key fields are in non-decreasing order.

As defined, the Sorting Problem allows input with two or more records that have the same key value. Certain applications require that input not contain duplicate key values. Typically, sorting algorithms can handle duplicate key values unless noted otherwise.

When duplicate key values are allowed, there might be an implicit ordering to the duplicates, typically based on their order of occurrence within the input. It might be desirable to maintain this initial ordering among duplicates. A sorting algorithm is said to be stable if it does not change the relative ordering of records with identical key values. Many, but not all, of the sorting algorithms presented in this chapter are stable, or can be made stable with minor changes.

When comparing two sorting algorithms, the simplest approach would be to program both and measure their running times. This is an example of empirical comparison. However, doing fair empirical comparisons can be tricky because the running time for many sorting algorithms depends on specifics of the input values. The number of records, the size of the keys and the records, the allowable range of the key values, and the amount by which the input records are “out of order” can all greatly affect the relative running times for sorting algorithms.

When analyzing sorting algorithms, it is traditional to measure the cost by counting the number of comparisons made between keys. This measure is usually closely related to the actual running time for the algorithm and has the advantage of being machine and data-type independent. However, in some cases records might be so large that their physical movement might take a significant fraction of the total running time. If so, it might be appropriate to measure the cost by counting the number of swap operations performed by the algorithm. In most applications we can assume that all records and keys are of fixed length, and that a single comparison or a single swap operation requires a constant amount of time regardless of which keys are involved. However, some special situations “change the rules” for comparing sorting algorithms. For example, an application with records or keys having widely varying length (such as sorting a sequence of variable length strings) cannot expect all comparisons to cost roughly the same. Not only do such situations require special measures for analysis, they also will usually benefit from a special-purpose sorting technique.

Other applications require that a small number of records be sorted, but that the sort be performed frequently. An example would be an application that repeatedly sorts groups of five numbers. In such cases, the constants in the runtime equations that usually get ignored in asymptotic analysis now become crucial. Note that recursive sorting algorithms end up sorting lots of small lists as well.

Finally, some situations require that a sorting algorithm use as little memory as possible. We will call attention to sorting algorithms that require significant extra memory beyond the input array.

12.3. Insertion Sort

12.3.1. Insertion Sort

What would you do if you have a stack of phone bills from the past two years and you want to order by date? A fairly natural way to handle this is to look at the first two bills and put them in order. Then take the third bill and put it into the right position with respect to the first two, and so on. As you take each bill, you would add it to the sorted pile that you have already made. This simple approach is the inspiration for our first sorting algorithm, called Insertion Sort.

Insertion Sort iterates through a list of records. For each iteration, the current record is inserted in turn at the correct position within a sorted list composed of those records already processed. Here is an implementation. The input is an array named A that stores records.

static <T extends Comparable<T>> void inssort(T[] A) {
  for (int i=1; i<A.length; i++) // Insert i'th record
    for (int j=i; (j>0) && (A[j].compareTo(A[j-1]) < 0); j--)
      swap(A, j, j-1);
}

(Note that to make the explanation for these sorting algorithms as simple as possible, our visualizations will show the array as though it stored simple integers rather than more complex records. But you should realize that in practice, there is rarely any point to sorting an array of simple integers. Nearly always we want to sort more complex records that each have a key value. In such cases we must have a way to associate a key value with a record. The sorting algorithms will simply assume that the records are comparable.)

Here we see the first few iterations of Insertion Sort.

This continues on with each record in turn. Call the current record . Insertion Sort will move it to the left so long as its value is less than that of the record immediately preceding it. As soon as a key value less than or equal to is encountered, inssort is done with that record because all records to its left in the array must have smaller keys.

12.3.2. Insertion Sort Analysis

While the best case is significantly faster than the average and worst cases, the average and worst cases are usually more reliable indicators of the “typical” running time. However, there are situations where we can expect the input to be in sorted or nearly sorted order. One example is when an already sorted list is slightly disordered by a small number of additions to the list; restoring sorted order using Insertion Sort might be a good idea if we know that the disordering is slight. And even when the input is not perfectly sorted, Insertion Sort’s cost goes up in proportion to the number of inversions. So a “nearly sorted” list will always be cheap to sort with Insertion Sort. Examples of algorithms that take advantage of Insertion Sort’s near-best-case running time are Shellsort and Quicksort.

Counting comparisons or swaps yields similar results. Each time through the inner for loop yields both a comparison and a swap, except the last (i.e., the comparison that fails the inner for loop’s test), which has no swap. Thus, the number of swaps for the entire sort operation is less than the number of comparisons. This is 0 in the best case, and in the average and worst cases.

Later we will see algorithms whose growth rate is much better than . Thus for larger arrays, Insertion Sort will not be so good a performer as other algorithms. So Insertion Sort is not the best sorting algorithm to use in most situations. But there are special situations where it is ideal. We already know that Insertion Sort works great when the input is sorted or nearly so. Another good time to use Insertion Sort is when the array is very small, since Insertion Sort is so simple. The algorithms that have better asymptotic growth rates tend to be more complicated, which leads to larger constant factors in their running time. That means they typically need fewer comparisons for larger arrays, but they cost more per comparison. This observation might not seem that helpful, since even an algorithm with high cost per comparison will be fast on small input sizes. But there are times when we might need to do many, many sorts on very small arrays. You should spend some time right now trying to think of a situation where you will need to sort many small arrays. Actually, it happens a lot.

See Computational Fairy Tales: Why Tailors Use Insertion Sort for a discussion on how the relative costs of search and insert can affect what is the best sort algorithm to use.

12.4. Bubble Sort

12.4.1. Bubble Sort

Our next sorting algorithm is called Bubble Sort. Bubble Sort is often taught to novice programmers in introductory computer science courses. This is unfortunate, because Bubble Sort has no redeeming features whatsoever. It is rather slow, even compared to the other sorts that are commonly known. It is not particularly intutitive – nobody is going to come naturally to Bubble Sort as a way to sort their Bridge hand or their pile of bills like they might with Insertion Sort or Selection Sort. However, Bubble Sort can viewed as a close relative of Selection Sort.

Like Insertion Sort, Bubble Sort consists of a simple double for loop. The inner for loop moves through the record array from left to right, comparing adjacent keys. If a record’s key value is greater than the key of its right neighbor, then the two records are swapped. Once the record with the largest key value is encountered, this process will cause it to “bubble” up to the right of the array (which is where Bubble Sort gets its name). The second pass through the array repeats this process. However, because we know that the record with the largest value already reached the right of the array on the first pass, there is no need to compare the rightmost two records on the second pass. Likewise, each succeeding pass through the array compares adjacent records, looking at one less record toward the end than did the preceding pass. Here is an implementation.

static <T extends Comparable<T>> void bubblesort(T[] A) {
  for (int i=0; i<A.length-1; i++) // Insert i'th record
    for (int j=1; j<A.length-i; j++)
      if (A[j-1].compareTo(A[j]) > 0)
        swap(A, j-1, j);
}

Now we continue with the second pass. However, since the largest record has “bubbled” to the very right, we will not need to look at it again.

Bubble Sort continues in this way until the entire array is sorted.

The following visualization shows the complete Bubble Sort. You can input your own data if you like.

Now try for yourself to see if you understand how Bubble Sort works.

12.4.2. Bubble Sort Analysis

The following visualization illustrates the running time analysis of Bubble Sort.

Thus, Bubble Sort’s running time is roughly the same in the best, average, and worst cases.

The number of swaps required depends on how often a record’s value is less than that of the record immediately preceding it in the array. We can expect this to occur for about half the comparisons in the average case, leading to for the expected number of swaps. The actual number of swaps performed by Bubble Sort will be identical to that performed by Insertion Sort.

Here are some review questions to check your understanding of Bubble Sort.

12.5. Selection Sort

12.5.1. Selection Sort

Consider again the problem of sorting a pile of phone bills for the past year. Another intuitive approach might be to look through the pile until you find the bill for January, and pull that out. Then look through the remaining pile until you find the bill for February, and add that behind January. Proceed through the ever-shrinking pile of bills to select the next one in order until you are done. This is the inspiration for our last sort, called Selection Sort. The ’th pass of Selection Sort “selects” the ’th smallest key in the array, placing that record at the start of the array. In other words, Selection Sort first finds the smallest key in an unsorted list, then the next smallest, and so on. Its unique feature is that there are few record swaps. To find the next-smallest key value requires searching through the entire unsorted portion of the array, but only one swap is required to put the record into place. Thus, the total number of swaps required will be (we get the last record in place “for free”).

Here is an implementation for Selection Sort.

static <T extends Comparable<T>> void selsort(T[] A) {
  for (int i=0; i<A.length-1; i++) {         // Select i'th smallest record
    int smallindex = i;                      // Current smallest index
    for (int j=i+1; j<A.length; j++)         // Find the min value
      if (A[j].compareTo(A[smallindex]) < 0) // Found something smaller
        smallindex = j;                      // Remember smaller index
    swap(A, i, bigindex);                    // Put it into place
  }
}

Consider the example of the following array.

Now we continue with the second pass. However, since the smallest record is already at the beginning, we will not need to look at it again.

Selection Sort continues in this way until the entire array is sorted.

The following visualization puts it all together.

Now try for yourself to see if you understand how Selection Sort works.

12.5.2. Selection Sort Analysis

Any algorithm can be written in slightly different ways. For example, we could have written Selection Sort to find the largest record, the next largest, and so on. Such a version of selection sort would essentially be a Bubble Sort, except that rather than repeatedly swapping adjacent values to get the next largest record into place, we instead remember the position of the record to be selected and do one swap at the end.

This visualization analyzes the number of comparisons and swaps required by Selection Sort.

There is another approach to keeping the cost of swapping records low, and it can be used by any sorting algorithm even when the records are large. This is to have each element of the array store a pointer to a record rather than store the record itself. In this implementation, a swap operation need only exchange the pointer values. The large records do not need to move. This technique is illustrated by the following visualization. Additional space is needed to store the pointers, but the return is a faster swap operation.

Here are some review questions to check how well you understand Selection Sort.

12.6. The Cost of Exchange Sorting

12.6.1. The Cost of Exchange Sorting

Here is a summary for the cost of Insertion Sort, Bubble Sort, and Selection Sort in terms of their required number of comparisons and swaps in the best, average, and worst cases. The running time for each of these sorts is in the average and worst cases.

The remaining sorting algorithms presented in this tutorial are significantly better than these three under typical conditions. But before continuing on, it is instructive to investigate what makes these three sorts so slow. The crucial bottleneck is that only adjacent records are compared. Thus, comparisons and moves (for Insertion and Bubble Sort) are by single steps. Swapping adjacent records is called an exchange. Thus, these sorts are sometimes referred to as an exchange sort. The cost of any exchange sort can be at best the total number of steps that the records in the array must move to reach their “correct” location. Recall that this is at least the number of inversions for the record, where an inversion occurs when a record with key value greater than the current record’s key value appears before it.

12.6.2. Analysis

12.7. Optimizing Sort Algorithms with Code Tuning

12.7.1. Code Tuning for Simple Sorting Algorithms

Since sorting is such an important application, it is natural for programmers to want to optimize their sorting code to run faster. Of course all quadratic sorts (Insertion Sort, Bubble Sort and Selection Sort) are relatively slow. Each has (as the name “quadratic suggests) worst case running time. The best way to speed them up is to find a better sorting algorithm. Nonetheless, there have been many suggestions given over the years about how to speed up one or another of these particular algorithms. There are useful lessons to be learned about code tuning by seeing which of these ideas actually turn out to give better performance. It is also interesting to see the relative performance of the three algorithms, as well as how various programming languages compare.

We start by trying to speed up Insertion Sort. Recall that Insertion Sort repeatedly moves an element toward the beginning of the sorted part of the list until it encounters a key with lesser value. In the original code, this is done with a series of swap operations. There is a better alternative than continuously swapping the record to the left until a smaller value is found. This is to move the current record to a temporary variable, and then shift all of the records with greater value one step to the right. Since swap requires three assignments per element and shifting requires only one assignment per element, we can hope that this will yield a big improvement. Of course, the amount of improvement that we actually get will depend on how much movement there is among the records. If the list is already nearly sorted, then there will be few swaps anyway. Here is an implementation for Insertion Sort using this optimization.

// Instead of swapping, "shift" the values down the array
static void inssortshift(int[] A) {
  for (int i=1; i<A.length; i++) { // Insert i'th record
    int j;
    int temp = A[i];
    for (j=i; (j>0) && (temp < A[j-1]); j--)
      A[j] = A[j-1];
    A[j] = temp;
  }
}

Now, you can test whether you understand how this works.

Table 12.7.1

Empirical comparison of proposed optimizations to quadratic sort implementations. Each sorting algorithm is run on a random integer array with 10,000 items. Times are in milliseconds. The arrays being sorted use the Comparable interface in languages that support this.

Table 12.7.1 shows the relative costs for a number of optimizations in four programming languages: Java, JavaScipt, Processing, and Python.

The programming language that you use can have a big influence on the runtime for a program. Perhaps the greatest distinction is whether your language is compiled or not. Java, C++, and Processing are normally compiled, while JavaScript and Python are normally interpreted. This can make a huge difference in whether a given code change will actually speed the program up or not. In the case of the “shift” vs “swap” choice, shifting always turns out to be a big improvement. This is more true for the interpreted languages JavaScript and Python than for Java and Processing, but still an improvement either way. But the biggest effect that we see is that Python takes over 100 times as long to execute the same program as Java.

Some languages have peculiarities that it pays to be aware of. It turns out that there is a big difference in JavaScript between using i < n or i != n to test termination of a loop.

Turning to Bubble Sort, the first thing we should notice from this table is that it is far slower on random input than Insertion Sort. Let’s consider a possible improvement that is sometimes suggested for Bubble Sort. That is to check during each iteration of the outer loop to see if any swaps took place during that iteration, and quit if not (since we know the list is ordered at this point). We can improve on this idea even more by recognizing that if the last swap done affects the values at positions and , no swaps could happen to values at positions greater than . Thus, we never need to check higher-positioned values again, which could save many iterations even if there are a few swaps lower down. Here is code to implement this approach.

static <T extends Comparable<T>> void bubblecheckswap(T[] A) {
  int n = A.length - 1;
  while (n > 0) {
    int newn = 0;
    for (int i = 0; i < n; i++) {
      /* if this pair is out of order */
      if (A[i].compareTo(A[i+1]) > 0) {
        swap(A, i, i+1);
        newn = i;
      }
    }
    n = newn;
  }
}

The problem with this idea is that a considerable amount of effort (relatively speaking) is required to track the position for the last swap within the inner loop. This tracking process has a cost, and that cost is worthwhile only if the amount of work it saves is greater than the amout of work that it causes. Unfortunately, as the table shows, in the average case it just is not worth the time. Modifying the code simply by removing the tracking steps (and so not getting either the cost of tracking or the benefit of avoiding some of the key comparisons) is faster in the average case. Of course, whether this is always true will depend on how much it costs to extract the record keys and compare them, which depends on the details of the record type and the sort implementation. In our test implementation we are sorting integer values and so the cost to compare records is lower than it would be if we had to get a field out of a more complex object.

It is also true that tracking the last swap position can substantially improve the best case cost. In fact, tracking the last swap position makes the best case cost of Bubble Sort to be only . But going out of one’s way to artificially improve the best case has dubious value if doing so imposes additional cost on nearly all other inputs. Note that we could nominally convert any sorting algorithm to have a best-case cost of by simply adding code at the beginning that checks if the list is already sorted. It should be obvious that this is a waste of time, even though it has the (small) possibility of winning big. Unlike Insertion Sort whose best case cost is naturally and whose time increases in proportion to how “out of order” the list is, the number of iterations avoided by swap checking in Bubble Sort is sensitive to the detailed placements of the out-of-order records. In fact, if we took a sorted list and moved the smallest value to the end, then there would be no benefit from swap checking whatsoever.

Finally, let’s consider Selection Sort. The table shows foremost that Selection Sort can be viewed as a far better optimization to Bubble Sort than tracking the last swap position. That is, tracking the position of the largest element and performing one swap to put it into place is a far better optimization to Bubble Sort than tracking the position of the last swap seen. The table also shows that Selection Sort is faster in the average case than Insertion Sort when implemented in Python. Evidently, the cost to swap is high for Python.

Our original Selection Sort implementation is written to make a call to swap even if the current record is already in its correct location. For example, if the record with the largest value is already in the rightmost array position, then selsort will still call swap with the two position parameters being the same. The net effect is that the work done by swap will not change anything in the array, and this is a waste of time. Thus, the total number of swaps done by Selection Sort is always in the best, average and worst cases. It might seem like a good idea to test if the positions are the same before calling swap, especially since Selection Sort’s claim to fame is its low number of swaps. Actually, we can’t expect this to ever make much difference since we are talking about actions within total steps, an inconsequential fraction. The other consideration is whether this is could typically be expected to save time even when just considering the time needed to do the swaps. Doing the check to see if a swap is necessary also takes some time. It is only worthwhile to test if the time required by the test is more than made up for by the work saved when the unnecessary swap was avoided. For randomly ordered input, it is probably more expensive to test this condition before every swap than to just do the swap. If the input records are already sorted, then all of the swaps are unnecessary and it would be (trivially) faster to test. But in the average case, few swaps will be saved this way and the “optimization” might actually slow down the program (but only slightly).

For all of these sorting algorithms, the swap function call might be a key part of the cost since it is called so many times. A simple way to speed things up is to replace this function call with the code that the function would perform. Depending on the language, compiler, and operating system, one might expect to save between 5 and 10 percent of the total time by doing so.

Another important consideration is the type of data object being used. For Processing and Java, we use a simple Integer wrapper object that supports the Comparable interface. This means that some dereferencing of the key value from an object is required, which is a typical expectation in a realistic application of a sorting function. However, if we were to sort a simple array of int values, the cost for all sorting algorithms will be less than half that shown. If we use a the more complicated KVPair objects, the costs will more than double over those shown in the table.

12.8. Shellsort

12.8.1. Shellsort

Shellsort was named for its inventor, D.L. Shell, who first published it in 1959. It is also sometimes called the diminishing increment sort. When properly implemented, Shellsort will give substantially better performance than any of the sorts like Insertion Sort or Selection Sort. But it is also a bit more complicated than those simple sorts. Unlike Insertion Sort and Selection Sort, there is no real-life intuition to inspire Shellsort – nobody will use Shellsort to sort their Bridge hand or organize their bills. The key idea behind Shellsort is to exploit the best-case performance of Insertion Sort. Recall that when a list is sorted or nearly sorted, Insertion Sort runs in linear time. So Shellsort’s strategy is to quickly make the list “mostly sorted”, so that a final Insertion Sort can finish the job.

Shellsort does what most good sorts do: Break the input into pieces, sort the pieces, then recombine them. But Shellsort does this in an unusual way, breaking its input into “virtual” sublists that are often not contiguous. Each such sublist is sorted using an Insertion Sort. Another group of sublists is then chosen and sorted, and so on.

Shellsort works by performing its Insertion Sorts on carefully selected sublists, first on small sublists and then on increasingly large sublists. So at each stage, any Insertion Sort is either working on a small list (and so is fast) or is working on a nearly sorted list (and again is fast).

Shellsort breaks the list into disjoint sublists, where a sublist is defined by an “increment”, . Each record in a given sublist is positions apart. For example, if the increment were 4, then each record in the sublist would be 4 positions apart.

One possible implementation for Shellsort is to use increments that are all powers of two. We start by picking as the largest power of two less than . This will generate sublists of 2 records each. If there were 16 records in the array indexed from 0 to 15, there would initially be 8 sublists of 2 records each, with each record in the sublist being 8 positions apart. The first sublist would be the records in positions 0 and 8. The second is in positions 1 and 9, and so on.

Actually, the increment size does not need to start at exactly . In the following example, we will use an array of 12 records (since 16 records makes the example a bit long). We will still begin with an increment size of 8. As you click through the following slideshow, you will see each of the sublists of length 2. If we reach a point where the remaining sublists have only one record (as will be the case for each of the sublists beginning with records 4 through 7), then we can skip processing them.

Shellsort will sort each of these sublists of length 2 using Insertion Sort. As you click through the next slideshow, you will first see the current sublist highlighted in yellow. Then a pair of records to be compared will be shown in blue. They are swapped if necessary to put them in sort order. (Of course, since these first sublists are each of length 2 when the two items are being compared you won’t see anything yellow anymore!)

At the end of the first pass, the resulting array is “a little better sorted”.

The second pass of Shellsort looks at fewer, bigger sublists. In our example, the second pass will have an increment of size 4, resulting in sublists. Since the array in our example has records, we have 4 sublists that each have records. Thus, the second pass would have as its first sublist the 3 records in positions 0, 4, and 8. The second sublist would have records in positions 1, 5, and 9, and so on.

As you click through the slides, you will see the sublists for increment size 4.

Each sublist of 3 records would also be sorted using an Insertion Sort, as shown next.

At the end of processing sublists with increment 4, the array is “even more sorted”.

The third pass will be made on sublists with increment 2. The effect is that we process 2 lists, one consisting of the odd positions and the other consisting of the even positions. As usual, we sort the sublists using Insertion Sort.

At this point, we are getting close to sorted.

Shellsort’s final pass will always use an increment of 1, which means a “regular” Insertion Sort of all records. But the list is far closer to sorted than it was at the start, so this final call to Insertion Sort runs far faster than if we had run Insertion Sort on the original array.

Finally, the array is sorted.

Here is a code implementation for Shellsort.

Now, test your understanding of the sublist concept.

12.8.2. Putting It Together

There is a lot of flexibility to picking the increment series. It does not need to start with the greatest power of less than and cut in half each time. In fact that is not even a good choice for the increment series. We will come back to this later. For now, just realize that so long as each increment is smaller than the last, and the last increment is 1, Shellsort will work.

At this point try running Shellsort on an array of your chosen size, with either random values or values that you select. You can also set the increment series. Use this visualization to make sure that you understand how Shellsort works.

Next, let’s review what makes for a legal increment series.

12.8.3. Shellsort Practice Exercise

Now test yourself to see how well you understand Shellsort. Can you reproduce its behavior?

12.8.4. Optimizing Shellsort

Some choices for the series of increments will make Shellsort run more efficiently than others. In particular, the choice of increments described above turns out to be relatively inefficient. You should notice for example that all records in a given 8 increment sublist are also part of some 4 increment sublist, which are all in turn records of the same 2 increment sublist. So there is no “crossover” between sublists as the increments reduce. A better choice is the following series based on “ ”: (…, 121, 40, 13, 4, 1). Another approach is to make sure that the various increments are relatively prime. The series (…, 11, 7, 3, 1) would be an example. In this case, there is a lot of “crossover” between the lists at the various increment sizes.

Now you are ready to try out some different increment series to see how they affect the cost of Shellsort.

A theoretical analysis of Shellsort is difficult, so we must accept without proof that the average-case performance of Shellsort (for a reasonable increment series) is . Thus, Shellsort is substantially better than Insertion Sort, or any of the other sorts presented earlier. In fact, Shellsort is not so much worse than the asymptotically better sorts to be presented later, whenever is of medium size (though it tends to be a little slower than these other algorithms if they are well implemented). Shellsort illustrates how we can sometimes exploit the special properties of an algorithm (in this case Insertion Sort) even if in general that algorithm is unacceptably slow.

Here are some review questions to check that you understand Shellsort.

If you want to know more about Shellsort, you can find a lot of details about its analysis along with ideas on how to pick a good increment series in [KnuthV3].

12.9. Mergesort Concepts

12.9.1. Mergesort Concepts

A natural approach to problem solving is divide and conquer. To use divide and conquer when sorting, we might consider breaking the list to be sorted into pieces, process the pieces, and then put them back together somehow. A simple way to do this would be to split the list in half, sort the halves, and then merge the sorted halves together. This is the idea behind Mergesort.

Mergesort is one of the simplest sorting algorithms conceptually, and has good performance both in the asymptotic sense and in empirical running time. Unfortunately, even though it is based on a simple concept, it is relatively difficult to implement in practice. Here is a pseudocode sketch of Mergesort:

List mergesort(List inlist) {
  if (inlist.length() <= 1) return inlist;;
  List L1 = half of the items from inlist;
  List L2 = other half of the items from inlist;
  return merge(mergesort(L1), mergesort(L2));
}

Here is a visualization that illustrates how Mergesort works.

The hardest step to understand about Mergesort is the merge function. The merge function starts by examining the first record of each sublist and picks the smaller value as the smallest record overall. This smaller value is removed from its sublist and placed into the output list. Merging continues in this way, comparing the front records of the sublists and continually appending the smaller to the output list until no more input records remain.

Here is pseudocode for merge on lists:

List merge(List L1, List L2) {
  List answer = new List();
  while (L1 != NULL || L2 != NULL) {
    if (L1 == NULL) { // Done L1
      answer.append(L2);
      L2 = NULL;
    }
    else if (L2 == NULL) { // Done L2
      answer.append(L1);
      L1 = NULL;
    }
    else if (L1.value() <= L2.value()) {
      answer.append(L1.value());
      L1 = L1.next();
    }
    else {
      answer.append(L2.value());
      L2 = L2.next();
    }
  }
  return answer;
}

Here is a visualization for the merge operation.

Here is a mergesort warmup exercise to practice merging.

12.9.2. Mergesort Practice Exercise

Now here is a full proficiency exercise to put it all together.

This visualization provides a running time analysis for Merge Sort.

12.10. Implementing Mergesort

12.10.1. Implementing Mergesort

Implementing Mergesort presents a number of technical difficulties. The first decision is how to represent the lists. Mergesort lends itself well to sorting a singly linked list because merging does not require random access to the list elements. Thus, Mergesort is the method of choice when the input is in the form of a linked list. Implementing merge for linked lists is straightforward, because we need only remove items from the front of the input lists and append items to the output list. Breaking the input list into two equal halves presents some difficulty. Ideally we would just break the lists into front and back halves. However, even if we know the length of the list in advance, it would still be necessary to traverse halfway down the linked list to reach the beginning of the second half. A simpler method, which does not rely on knowing the length of the list in advance, assigns elements of the input list alternating between the two sublists. The first element is assigned to the first sublist, the second element to the second sublist, the third to first sublist, the fourth to the second sublist, and so on. This requires one complete pass through the input list to build the sublists.

When the input to Mergesort is an array, splitting input into two subarrays is easy if we know the array bounds. Merging is also easy if we merge the subarrays into a second array. Note that this approach requires twice the amount of space as any of the sorting methods presented so far, which is a serious disadvantage for Mergesort. It is possible to merge the subarrays without using a second array, but this is extremely difficult to do efficiently and is not really practical. Merging the two subarrays into a second array, while simple to implement, presents another difficulty. The merge process ends with the sorted list in the auxiliary array. Consider how the recursive nature of Mergesort breaks the original array into subarrays. Mergesort is recursively called until subarrays of size 1 have been created, requiring levels of recursion. These subarrays are merged into subarrays of size 2, which are in turn merged into subarrays of size 4, and so on. We need to avoid having each merge operation require a new array. With some difficulty, an algorithm can be devised that alternates between two arrays. A much simpler approach is to copy the sorted sublists to the auxiliary array first, and then merge them back to the original array.

Here is a complete implementation for mergesort following this approach. The input records are in array A. Array temp is used as a place to temporarily copy records during the merge process. Parameters left and right define the left and right indices, respectively, for the subarray being sorted. The initial call to mergesort would be mergesort(array, temparray, 0, n-1).

Here is a visualization for the merge step.

An optimized Mergesort implementation is shown below. It reverses the order of the second subarray during the initial copy. Now the current positions of the two subarrays work inwards from the ends, allowing the end of each subarray to act as a sentinel for the other. Unlike the previous implementation, no test is needed to check for when one of the two subarrays becomes empty. This version also has a second optimization: It uses Insertion Sort to sort small subarrays whenever the size of the array is smaller than a value defined by THRESHOLD.

Here is a visualization for the optimized merge step.

12.11. Quicksort

12.11.1. Introduction

While Mergesort uses the most obvious form of divide and conquer (split the list in half then sort the halves), this is not the only way that we can break down the sorting problem. We saw that doing the merge step for Mergesort when using an array implementation is not so easy. So perhaps a different divide and conquer strategy might turn out to be more efficient?

Quicksort is aptly named because, when properly implemented, it is the fastest known general-purpose in-memory sorting algorithm in the average case. It does not require the extra array needed by Mergesort, so it is space efficient as well. Quicksort is widely used, and is typically the algorithm implemented in a library sort routine such as the UNIX qsort function. Interestingly, Quicksort is hampered by exceedingly poor worst-case performance, thus making it inappropriate for certain applications.

Before we get to Quicksort, consider for a moment the practicality of using a Binary Search Tree for sorting. You could insert all of the values to be sorted into the BST one by one, then traverse the completed tree using an inorder traversal. The output would form a sorted list. This approach has a number of drawbacks, including the extra space required by BST pointers and the amount of time required to insert nodes into the tree. However, this method introduces some interesting ideas. First, the root of the BST (i.e., the first node inserted) splits the list into two sublists: The left subtree contains those values in the list less than the root value while the right subtree contains those values in the list greater than or equal to the root value. Thus, the BST implicitly implements a “divide and conquer” approach to sorting the left and right subtrees. Quicksort implements this same concept in a much more efficient way.

Quicksort first selects a value called the pivot. (This is conceptually like the root node’s value in the BST.) Assume that the input array contains records with key values less than the pivot. The records are then rearranged in such a way that the values less than the pivot are placed in the first, or leftmost, positions in the array, and the values greater than or equal to the pivot are placed in the last, or rightmost, positions. This is called a partition of the array. The values placed in a given partition need not (and typically will not) be sorted with respect to each other. All that is required is that all values end up in the correct partition. The pivot value itself is placed in position . Quicksort then proceeds to sort the resulting subarrays now on either side of the pivot, one of size and the other of size . How are these values sorted? Because Quicksort is such a good algorithm, using Quicksort on the subarrays would be appropriate.

Unlike some of the sorts that we have seen earlier in this chapter, Quicksort might not seem very “natural” in that it is not an approach that a person is likely to use to sort real objects. But it should not be too surprising that a really efficient sort for huge numbers of abstract objects on a computer would be rather different from our experiences with sorting a relatively few physical objects.

Here is an implementation for Quicksort. Parameters i and j define the left and right indices, respectively, for the subarray being sorted. The initial call to quicksort would be quicksort(array, 0, n-1).

Function partition will move records to the appropriate partition and then return k, the first position in the right partition. Note that the pivot value is initially placed at the end of the array (position j). Thus, partition must not affect the value of array position j. After partitioning, the pivot value is placed in position k, which is its correct position in the final, sorted array. By doing so, we guarantee that at least one value (the pivot) will not be processed in the recursive calls to qsort. Even if a bad pivot is selected, yielding a completely empty partition to one side of the pivot, the larger partition will contain at most records.

Selecting a pivot can be done in many ways. The simplest is to use the first key. However, if the input is sorted or reverse sorted, this will produce a poor partitioning with all values to one side of the pivot. It is better to pick a value at random, thereby reducing the chance of a bad input order affecting the sort. Unfortunately, using a random number generator is relatively expensive, and we can do nearly as well by selecting the middle position in the array. Here is a simple findpivot function.

12.11.2. Partition

We now turn to function partition. If we knew in advance how many keys are less than the pivot, partition could simply copy records with key values less than the pivot to the low end of the array, and records with larger keys to the high end. Because we do not know in advance how many keys are less than the pivot, we use a clever algorithm that moves indices inwards from the ends of the subarray, swapping values as necessary until the two indices meet. Here is an implementation for the partition step.

Note the check that right >= left in the second inner while loop. This ensures that right does not run off the low end of the partition in the case where the pivot is the least value in that partition. Function partition returns the first index of the right partition (the place where left ends at) so that the subarray bound for the recursive calls to qsort can be determined.

And here is a visualization illustrating the running time analysis of the partition function

12.11.3. Putting It Together

Here is a visualization for the entire Quicksort algorithm. This visualization shows you how the logical decomposition caused by the partitioning process works. In the visualization, the separate sub-partitions are separated out to match the recursion tree. In reality, there is only a single array involved (as you will see in the proficiency exercise that follows the visualization).

Here is a complete proficiency exercise to see how well you understand Quicksort.

12.11.4. Quicksort Analysis

This visualization explains the worst-case running time of Quick Sort

This is terrible, no better than Bubble Sort. When will this worst case occur? Only when each pivot yields a bad partitioning of the array. If the pivot values are selected at random, then this is extremely unlikely to happen. When selecting the middle position of the current subarray, it is still unlikely to happen. It does not take many good partitionings for Quicksort to work fairly well.

This visualization explains the best-case running time of Quick Sort

Quicksort’s average-case behavior falls somewhere between the extremes of worst and best case. Average-case analysis considers the cost for all possible arrangements of input, summing the costs and dividing by the number of cases. We make one reasonable simplifying assumption: At each partition step, the pivot is equally likely to end in any position in the (sorted) array. In other words, the pivot is equally likely to break an array into partitions of sizes 0 and , or 1 and , and so on.

Given this assumption, the average-case cost is computed from the following equation:

This visualization will help you to understand how this recurrence relation was formed.

This is an unusual situation that the average case cost and the worst case cost have asymptotically different growth rates. Consider what “average case” actually means. We compute an average cost for inputs of size by summing up for every possible input of size the product of the running time cost of that input times the probability that that input will occur. To simplify things, we assumed that every permutation is equally likely to occur. Thus, finding the average means summing up the cost for every permutation and dividing by the number of permuations (which is ). We know that some of these inputs cost . But the sum of all the permutation costs has to be . Given the extremely high cost of the worst inputs, there must be very few of them. In fact, there cannot be a constant fraction of the inputs with cost . If even, say, 1% of the inputs have cost , this would lead to an average cost of . Thus, as grows, the fraction of inputs with high cost must be going toward a limit of zero. We can conclude that Quicksort will run fast if we can avoid those very few bad input permutations. This is why picking a good pivot is so important.

The running time for Quicksort can be improved (by a constant factor), and much study has gone into optimizing this algorithm. Since Quicksort’s worst case behavior arises when the pivot does a poor job of splitting the array into equal size subarrays, improving findpivot seems like a good place to start. If we are willing to do more work searching for a better pivot, the effects of a bad pivot can be decreased or even eliminated. Hopefully this will save more time than was added by the additional work needed to find the pivot. One widely-used choice is to use the “median of three” algorithm, which uses as a pivot the middle of three randomly selected values. Using a random number generator to choose the positions is relatively expensive, so a common compromise is to look at the first, middle, and last positions of the current subarray. However, our simple findpivot function that takes the middle value as its pivot has the virtue of making it highly unlikely to get a bad input by chance, and it is quite cheap to implement. This is in sharp contrast to selecting the first or last record as the pivot, which would yield bad performance for many permutations that are nearly sorted or nearly reverse sorted.

A significant improvement can be gained by recognizing that Quicksort is relatively slow when is small. This might not seem to be relevant if most of the time we sort large arrays, nor should it matter how long Quicksort takes in the rare instance when a small array is sorted because it will be fast anyway. But you should notice that Quicksort itself sorts many, many small arrays! This happens as a natural by-product of the divide and conquer approach.

A simple improvement might then be to replace Quicksort with a faster sort for small numbers, say Insertion Sort or Selection Sort. However, there is an even better—and still simpler—optimization. When Quicksort partitions are below a certain size, do nothing! The values within that partition will be out of order. However, we do know that all values in the array to the left of the partition are smaller than all values in the partition. All values in the array to the right of the partition are greater than all values in the partition. Thus, even if Quicksort only gets the values to “nearly” the right locations, the array will be close to sorted. This is an ideal situation in which to take advantage of the best-case performance of Insertion Sort. The final step is a single call to Insertion Sort to process the entire array, putting the records into final sorted order. Empirical testing shows that the subarrays should be left unordered whenever they get down to nine or fewer records.

The last speedup to be considered reduces the cost of making recursive calls. Quicksort is inherently recursive, because each Quicksort operation must sort two sublists. Thus, there is no simple way to turn Quicksort into an iterative algorithm. However, Quicksort can be implemented using a stack to imitate recursion, as the amount of information that must be stored is small. We need not store copies of a subarray, only the subarray bounds. Furthermore, the stack depth can be kept small if care is taken on the order in which Quicksort’s recursive calls are executed. We can also place the code for findpivot and partition inline to eliminate the remaining function calls. Note however that by not processing sublists of size nine or less as suggested above, about three quarters of the function calls will already have been eliminated. Thus, eliminating the remaining function calls will yield only a modest speedup.

12.12. Heapsort

12.12.1. Heapsort

Our discussion of Quicksort began by considering the practicality of using a BST for sorting. The BST requires more space than the other sorting methods and will be slower than Quicksort or Mergesort due to the relative expense of inserting values into the tree. There is also the possibility that the BST might be unbalanced, leading to a worst-case running time. Subtree balance in the BST is closely related to Quicksort’s partition step. Quicksort’s pivot serves roughly the same purpose as the BST root value in that the left partition (subtree) stores values less than the pivot (root) value, while the right partition (subtree) stores values greater than or equal to the pivot (root).

A good sorting algorithm can be devised based on a tree structure more suited to the purpose. In particular, we would like the tree to be balanced, space efficient, and fast. The algorithm should take advantage of the fact that sorting is a special-purpose application in that all of the values to be stored are available at the start. This means that we do not necessarily need to insert one value at a time into the tree structure.

Heapsort is based on the heap data structure. Heapsort has all of the advantages just listed. The complete binary tree is balanced, its array representation is space efficient, and we can load all values into the tree at once, taking advantage of the efficient buildheap function. The asymptotic performance of Heapsort when all of the records have unique key values is in the best, average, and worst cases. It is not as fast as Quicksort in the average case (by a constant factor), but Heapsort has special properties that will make it particularly useful for external sorting algorithms, used when sorting data sets too large to fit in main memory.

A complete implementation is as follows.

Here is a warmup practice exercise for Heapsort.

12.12.2. Heapsort Proficiency Practice

Now test yourself to see how well you understand Heapsort. Can you reproduce its behavior?

12.12.3. Heapsort Analysis

This visualization presents the running time analysis of Heap Sort

While typically slower than Quicksort by a constant factor (because unloading the heap using removemax is somewhat slower than Quicksort’s series of partitions), Heapsort has one special advantage over the other sorts studied so far. Building the heap is relatively cheap, requiring time. Removing the maximum-valued record from the heap requires time. Thus, if we wish to find the records with the largest key values in an array, we can do so in time . If is small, this is a substantial improvement over the time required to find the largest-valued records using one of the other sorting methods described earlier (many of which would require sorting all of the array first). One situation where we are able to take advantage of this concept is in the implementation of Kruskal’s algorithm for minimal-cost spanning trees. That algorithm requires that edges be visited in ascending order (so, use a min-heap), but this process stops as soon as the MST is complete. Thus, only a relatively small fraction of the edges need be sorted.

12.13. Binsort

12.13.1. Binsort

Imagine that for the past year, as you paid your various bills, you then simply piled all the paperwork into a corner somewhere. Now the year has ended and you have decided that it is time to sort all of these papers by what the bill was for (phone, electricity, rent, etc.) and date. A pretty natural approach is to make some space on the floor and, as you go through the pile of papers, put the phone bills into one pile, the electric bills into another pile, and so on. Once this initial assignment of bills to piles is done (in one pass), you can then sort each pile by date relatively quickly, because each pile is fairly small. This is the basic idea behind a Binsort.

Let’s start with an especially easy situation. Consider the following code fragment to sort a permutation of the numbers 0 through .

for (i=0; i<A.length; i++)
  B[A[i]] = A[i];

Here the key value is used to determine the position for a record in the final sorted array. This is the most basic example of a Binsort, where key values are used to assign records to bins. This algorithm is extremely efficient, always taking time regardless of the initial ordering of the keys. This is far better than the performance of any sorting algorithm that we have seen so far. The problem is that this algorithm has limited use because it works only for a permutation of the numbers from 0 to .

We can extend this simple version of the Binsort algorithm to be more useful. Because Binsort must perform direct computation on the key value (as opposed to just asking which of two records comes first as our previous sorting algorithms did), we will assume that the records use an integer key type.

The simplest extension is to allow for duplicate values among the keys. This can be done by turning array slots into arbitrary-length bins by turning array B into an array of linked lists. In this way, all records with key value can be placed in bin B[i]. A second extension allows for a key range greater than . For example, a set of records might have keys in the range 1 to . The only requirement is that each possible key value have a corresponding bin in B. We assume that we know that the range of possible keys is between 0 and MaxKeyValue. Here is the extended Binsort algorithm.

void binsort(Integer[] A) {
  List[] B = new LinkedList[MaxKeyValue+1];
  Object item;
  for (int i=0; i<=MaxKeyValue; i++)
    B[i] = new LinkedList();
  for (int i=0; i<A.length; i++) B[A[i]].append(new Integer(A[i]));
  int pos = 0;
  for (int i=0; i<=MaxKeyValue; i++)
    for (B[i].moveToStart(); (item = B[i].getValue()) != null; B[i].next())
      A[pos++] = (Integer)item;
}

This version of Binsort can sort any collection of records whose key values fall in the range from 0 to MaxKeyValue.

The total work required is simply that needed to place each record into the appropriate bin and then take all of the records out of the bins. Thus, we need to process each record twice, for work.

Does that cost analysis really make sense? Actually, that last statement is wrong, because it neglects a crucial observation. Taking all of the records out of the bins requires Binsort to look at every bin to see if it contains a record. Thus, the algorithm must process MaxKeyValue bins, regardless of how many of them actually hold records. If MaxKeyValue is small compared to , then this is not a great expense. Suppose that MaxKeyValue . In this case, the total amount of work done will be . This results in a poor sorting algorithm. And the algorithm becomes even worse as the disparity between and MaxKeyValue increases. In addition, a large key range requires an unacceptably large array B. Thus, even the extended Binsort is useful only for a limited key range.

A further generalization to Binsort would yield a bucket sort. Here, each bin (now called a bucket) is associated with not just one key, but rather a range of key values. A bucket sort assigns records to buckets and then relies on some other sorting technique to sort the records within each bucket. The hope is that the relatively inexpensive bucketing process will put only a small number of records into each bucket, and that a “cleanup sort” to each bucket will then be relatively cheap. This is similar in spirit to the Radix Sort, which extends the concept of the Binsort in a practical way.

12.14. Radix Sort

12.14.1. Radix Sort

The major problem with Binsort is that it does not work so well for a large key range. Fortunately, there is a way to keep the number of bins small and the related processing relatively cheap while still using the idea of binning records that have similar key values. Consider a sequence of records with keys in the range 0 to 99. If we have ten bins available, we can first assign records to bins by taking their key value modulo 10. Thus, every key will be assigned to the bin matching its rightmost decimal digit. We can then take these records from the bins in order, and reassign them to the bins on the basis of their leftmost (10’s place) digit. We will define values in the range 0 to 9 to have a leftmost digit of 0. In other words, assign the ’th record from array A to a bin using the formula A[i]/10. If we now gather the values from the bins in order, the result is a sorted list. We can see this process in the following visualization.

In this example, we have bins and key values in the range 0 to . The total computation is , because we look at each record and each bin a constant number of times. This is a great improvement over the simple Binsort where the number of bins must be as large as the key range. Note that the example uses so as to make the bin computations easy to visualize: Records were placed into bins based on the value of first the rightmost and then the leftmost decimal digits. Any number of bins would have worked if we interpret the key values in terms of the corresponding base. This is an example of a Radix Sort, so called because the bin computations are based on the radix or the base of the key values. This sorting algorithm can be extended to any number of keys in any key range. We simply assign records to bins based on the keys’ digit values working from the rightmost digit to the leftmost. If there are digits, then this requires that we assign keys to bins times.

Here is a practice exercise for placing keys into bins.

12.14.2. Array-based Radix Sort

As with Mergesort, an efficient implementation of Radix Sort is somewhat difficult to achieve. In particular, we would prefer to sort an array of values and avoid processing linked lists. If we knew how many values would be in each bin, then an auxiliary array of size can be used to define these lengths and guide us to were each one starts in the output array. For example, if during the first pass the 0 bin will receive three records and the 1 bin will receive five records, then we could simply reserve the first three array positions for the 0 bin and the next five array positions for the 1 bin. Exactly this approach is taken by the following implementation. At the end of each pass, the records are copied back to the original array.

The first inner for loop initializes array count. The second loop counts the number of records to be assigned to each bin. The third loop sets the values in count to their proper indices within array B. Note that the index stored in count[j] is the last index for bin j; bins are filled from high index to low index. The fourth loop assigns the records to the bins (within array B). The final loop simply copies the records back to array A to be ready for the next pass. Variable rtoi stores for use in bin computation on the ’th iteration.

12.14.2.1. Radix Sort Analysis

Is it really a reasonable assumption to treat as a constant? Or is there some relationship between and ? If the key range is limited and duplicate key values are common, there might be no relationship between and . To make this distinction more clear, use to denote the number of distinct key values used by the records. Thus, . Because it takes a minimum of base digits to represent distinct key values, we know that .

Now, consider the situation in which no keys are duplicated. If there are unique keys then . It would require distinct values to represent them. So now it takes a minimum of base digits to represent the distinct key values. This means that . Because it requires at least digits to distinguish between the distinct keys (within a constant factor—meaning, the number of digits is ), is in . This means that Radix Sort requires time to process distinct key values.

Of course the key range could be much bigger bits is merely the best case possible for distinct values. Thus, the estimate for could be overly optimistic. The bottom line of this analysis is that, for the general case of distinct key values, Radix Sort is at best a sorting algorithm.

Radix Sort’s running time can be much improved (by a constant factor) if we make base be as large as possible. This is simplest if we think about integer key values. Set for some . In other words, the value of is related to the number of bits of the key processed on each pass. Each time the number of bits is doubled, the number of passes is cut in half. When processing an integer key value, setting allows the key to be processed one byte at a time. Processing a 32-bit integer key requires only four passes. It is not unreasonable on most computers to use , resulting in only two passes for a 32-bit key. Of course, this requires a count array of size 64K. Performance will be good only if the number of records is about 64K or greater. In other words, the number of records must be large compared to the key size for Radix Sort to be efficient. In many sorting applications, Radix Sort can be tuned in this way to give better performance.

Radix Sort depends on the ability to make a fixed number of multiway choices based on a digit value, as well as random access to the bins. Thus, Radix Sort might be difficult to implement for certain key types. For example, if the keys are real numbers or arbitrary length strings, then some care will be necessary in implementation. In particular, Radix Sort will need to be careful about deciding when the “last digit” has been found to distinguish among real numbers, or the last character in variable length strings. Implementing the concept of Radix Sort with the alphabet trie data structure is most appropriate for these situations.

12.15. An Empirical Comparison of Sorting Algorithms

12.15.1. An Empirical Comparison of Sorting Algorithms

Which sorting algorithm is fastest? Asymptotic complexity analysis lets us distinguish between and algorithms, but it does not help distinguish between algorithms with the same asymptotic complexity. Nor does asymptotic analysis say anything about which algorithm is best for sorting small lists. For answers to these questions, we can turn to empirical testing.

Table 12.15.1

Empirical comparison of sorting algorithms run on a 3.4 GHz Intel Pentium 4 CPU running Linux. All times shown are milliseconds.

Table 12.15.1 shows timing results for actual implementations of the sorting algorithms presented in this chapter. The algorithms compared include Insertion Sort, Bubble Sort, Selection Sort, Shellsort, Quicksort, Mergesort, Heapsort, Radix Sort.

Shellsort compares times for both the basic version and a version with increments based on division by three. Mergesort compares both the basic array-based implementation and an optimized version (which includes calls to Insertion Sort for lists of length below nine). For Quicksort, two versions are compared: the basic implementation and an optimized version that does not partition sublists below length nine (with Insertion Sort performed at the end). The first Heapsort version uses a standard class definition with methods to implement access functions like “parent”. The second version removes all the method definitions and operates directly on the array using inlined code for all access functions.

Except for the rightmost columns, the input to each algorithm is a random array of integers. This affects the timing for some of the sorting algorithms. For example, Selection Sort is not being used to best advantage because the record size is small, so it does not get the best possible showing. The Radix Sort implementation certainly takes advantage of this key range in that it does not look at more digits than necessary. On the other hand, it was not optimized to use bit shifting instead of division, even though the bases used would permit this.

The various sorting algorithms are shown for lists of sizes 10, 100, 1000, 10,000, 100,000, and 1,000,000. The final two columns of each table show the performance for the algorithms on inputs of size 10,000 where the numbers are in ascending (sorted) and descending (reverse sorted) order, respectively. These columns demonstrate best-case performance for some algorithms and worst-case performance for others. They also show that for some algorithms, the order of input has little effect.

These figures show a number of interesting results. As expected, the sorts are quite poor performers for large arrays. Insertion Sort is by far the best of this group, unless the array is already reverse sorted. Shellsort is clearly superior to any of these sorts for lists of even 100 records. Optimized Quicksort is clearly the best overall algorithm for all but lists of 10 records. Even for small arrays, optimized Quicksort performs well because it does one partition step before calling Insertion Sort. Compared to the other sorts, unoptimized Heapsort is quite slow due to the overhead of the class structure. When all of this is stripped away and the algorithm is implemented to manipulate an array directly, it is still somewhat slower than mergesort. In general, optimizing the various algorithms makes a noticeable improvement for larger array sizes.

Overall, Radix Sort is a surprisingly poor performer. If the code had been tuned to use bit shifting of the key value, it would likely improve substantially; but this would seriously limit the range of record types that the sort could support.

Here are a few multiple choice questions that ask you to compare the sorting algorithms that we learned about in this chapter.

12.16. Lower Bounds for Sorting

12.16.1. Lower Bounds for Sorting

By now you have seen many analyses for algorithms. These analyses generally define the upper and lower bounds for algorithms in their worst and average cases. For many of the algorithms presented so far, analysis has been easy. This module considers a more difficult task: An analysis for the cost of a problem as opposed to an algorithm. The upper bound for a problem can be defined as the asymptotic cost of the fastest known algorithm. The lower bound defines the best possible cost for any algorithm that solves the problem, including algorithms not yet invented. Once the upper and lower bounds for the problem meet, we know that no future algorithm can possibly be (asymptotically) more efficient.

A simple estimate for a problem’s lower bound can be obtained by measuring the size of the input that must be read and the output that must be written. Certainly no algorithm can be more efficient than the problem’s I/O time. From this we see that the sorting problem cannot be solved by any algorithm in less than time because it takes at least steps to read and write the values to be sorted. Alternatively, any sorting algorithm must at least look at every input value to recognize whether the input values are in sorted order. So, based on our current knowledge of sorting algorithms and the size of the input, we know that the problem of sorting is bounded by and .

Computer scientists have spent much time devising efficient general-purpose sorting algorithms, but no one has ever found one that is faster than in the worst or average cases. Should we keep searching for a faster sorting algorithm? Or can we prove that there is no faster sorting algorithm by finding a tighter lower bound?

This section presents one of the most important and most useful proofs in computer science: No sorting algorithm based on key comparisons can possibly be faster than in the worst case. This proof is important for three reasons. First, knowing that widely used sorting algorithms are asymptotically optimal is reassuring. In particular, it means that you need not bang your head against the wall searching for an sorting algorithm. (Or at least not one that is in any way based on key comparisons. But it is hard to imagine how to sort without any comparisons. Even Radix Sort is does comparisons, though in quite a different way.) Second, this proof is one of the few non-trivial lower-bounds proofs that we have for any problem; that is, this proof provides one of the relatively few instances where our lower bound is tighter than simply measuring the size of the input and output. As such, it provides a useful model for proving lower bounds on other problems. Finally, knowing a lower bound for sorting gives us a lower bound in turn for other problems whose solution could be made to work as the basis for a sorting algorithm. The process of deriving asymptotic bounds for one problem from the asymptotic bounds of another is called a reduction.

Except for the Radix Sort and Binsort, all of the sorting algorithms we have studied make decisions based on the direct comparison of two key values. For example, Insertion Sort sequentially compares the value to be inserted into the sorted list until a comparison against the next value in the list fails. In contrast, Radix Sort has no direct comparison of key values. All decisions are based on the value of specific digits in the key value, so it is possible to take approaches to sorting that do not involve direct key comparisons. Of course, Radix Sort in the end does not provide a more efficient sorting algorithm than comparison-based sorting. Thus, empirical evidence suggests that comparison-based sorting is a good approach.

(Actually, the truth is stronger than this statement implies. In reality, Radix Sort relies on comparisons as well and so can be modeled by the technique used in this section. The result is an bound in the general case even for algorithms that look like Radix Sort.)

The proof that any comparison sort requires comparisons in the worst case is structured as follows. First, comparison-based decisions can be modeled as the branches in a tree. This means that any sorting algorithm based on comparisons between records can be viewed as a binary tree whose nodes correspond to the comparisons, and whose branches correspond to the possible outcomes. Next, the minimum number of leaves in the resulting tree is shown to be the factorial of . Finally, the minimum depth of a tree with leaves is shown to be in .

Before presenting the proof of an lower bound for sorting, we first must define the concept of a decision tree. A decision tree is a binary tree that can model the processing for any algorithm that makes binary decisions. Each (binary) decision is represented by a branch in the tree. For the purpose of modeling sorting algorithms, we count all comparisons of key values as decisions. If two keys are compared and the first is less than the second, then this is modeled as a left branch in the decision tree. In the case where the first value is greater than the second, the algorithm takes the right branch.

Here is a Visualization that illustrates decision trees and the sorting lower bound proof.

Any sorting algorithm requiring comparisons in the worst case requires running time in the worst case. Because any sorting algorithm requires running time, the problem of sorting also requires time. We already know of sorting algorithms with running time, so we can conclude that the problem of sorting requires time. As a corollary, we know that no comparison-based sorting algorithm can improve on existing time sorting algorithms by more than a constant factor.

Here are some review questions to check that you understand this proof.

12.17. Sorting Summary Exercises

Here is a complete set of review questions, taken from all of the questions in the modules of this chapter.

Chapter 13: Searching

13.1. Chapter Introduction: Search

Organizing and retrieving information is at the heart of most computer applications, and searching is surely the most frequently performed of all computing tasks. Search can be viewed abstractly as a process to determine if an element with a particular value is a member of a particular set. The more common view of searching is an attempt to find the record within a collection of records that has a particular key value, or those records in a collection whose key values meet some criterion such as falling within a range of values.

We can define searching formally as follows. Suppose that we have a collection L of records of the form

where is information associated with key from record for . Given a particular key value , the search problem is to locate a record in L such that (if one exists). Searching is a systematic method for locating the record (or records) with key value .

A successful search is one in which a record with key is found. An unsuccessful search is one in which no record with is found (and no such record exists).

An exact-match query is a search for the record whose key value matches a specified key value. A range query is a search for all records whose key value falls within a specified range of key values.

We can categorize search algorithms into three general approaches:

1. Sequential and list methods.
2. Direct access by key value (hashing).
3. Tree indexing methods.

Any of these approaches are potentially suitable for implementing the Dictionary ADT. However, each has different performance characteristics that make it the method of choice in particular circumstances.

The current chapter considers methods for searching data stored in lists. List in this context means any list implementation including a linked list or an array. Most of these methods are appropriate for sequences (i.e., duplicate key values are allowed), although there are special techniques applicable to sets. The techniques from the first three sections of this chapter are most appropriate for searching a collection of records stored in RAM. Chapter Hashing introduces hashing, a technique for organizing data in an array such that the location of each record within the array is a function of its key value. Hashing is appropriate when records are stored either in RAM or on disk.

Chapter Indexing discusses tree-based methods for organizing information on disk, including a commonly used file structure called the B-tree. Nearly all programs that must organize large collections of records stored on disk use some variant of either hashing or the B-tree. Hashing is practical for only certain access applications (exact-match queries) and is generally appropriate only when duplicate key values are not allowed. B-trees are the method of choice for dynamic disk-based applications anytime hashing is not appropriate.

13.2. Searching in an Array

13.2.1. Searching in an Array

13.2.1.1. Sequential Search

If you want to find the position in an unsorted array of integers that stores a particular value, you cannot really do better than simply looking through the array from the beginning and move toward the end until you find what you are looking for. This algorithm is called sequential search. If you do find it, we call this a successful search. If the value is not in the array, eventually you will reach the end. We will call this an unsuccessful search. Here is a simple implementation for sequential search.

// Return the position of an element in array A with value K.
// If K is not in A, return A.length.
static int sequential(int[] A, int K) {
  for (int i=0; i<A.length; i++) // For each element
    if (A[i] == K)               // if we found it
       return i;                 //   return this position
  return A.length;               // Otherwise, return the array length
}

It is natural to ask how long a program or algorithm will take to run. But we do not really care exactly how long a particular program will run on a particular computer. We just want some sort of estimate that will let us compare one approach to solving a problem with another. This is the basic idea of algorithm analysis. In the case of sequential search, it is easy to see that if the value is in position of the array, then sequential search will look at values to find it. If the value is not in the array at all, then we must look at values if the array holds values. This would be called the worst case for sequential search. Since the amount of work is proportional to , we say that the worst case for sequential search has linear cost. For this reason, the sequential search algorithm is sometimes called linear search.

13.2.1.2. Binary Search

Sequential search is the best that we can do when trying to find a value in an unsorted array. [^1] But if the array is sorted in increasing order by value, then we can do much better. We use a process called binary search.

Binary search begins by examining the value in the middle position of the array; call this position and the corresponding value . If , then processing can stop immediately. This is unlikely to be the case, however. Fortunately, knowing the middle value provides useful information that can help guide the search process. In particular, if , then you know that the value cannot appear in the array at any position greater than . Thus, you can eliminate future search in the upper half of the array. Conversely, if , then you know that you can ignore all positions in the array less than . Either way, half of the positions are eliminated from further consideration. Binary search next looks at the middle position in that part of the array where value may exist. The value at this position again allows us to eliminate half of the remaining positions from consideration. This process repeats until either the desired value is found, or there are no positions remaining in the array that might contain the value . Here is an illustration of the binary search method.

With the right math techniques, it is not too hard to show that the cost of binary search on an array of values is at most . This is because we are repeatedly splitting the size of the subarray that we must look at in half. We stop (in the worst case) when we reach a subarray of size 1. And we can only cut the value of in half times before we reach 1. [^2]

[^1]: It seems to be really “obvious” that sequential search is the best that you can do on an unsorted array. But writing a convincing proof that no algorithm could ever be discovered that is better is surprisingly difficult. This is an example of a lower bounds proof to find the cost for the best possible algorithm to solve the problem of search in an unsorted array.

[^2]: It is possible to prove that binary search is the most efficient algorithm possible in the worst case when searching in a sorted array. This is even more difficult than proving that sequential search is the most efficient algorithm possible on an unsorted array.

13.3. Analyzing Search in Unsorted Lists

13.3.1. Analyzing Search in Unsorted Lists

You already know the simplest form of search: the sequential search algorithm. Sequential search on an unsorted list requires time in the worst case.

How many comparisons does linear search do on average? A major consideration is whether is in list L at all. We can simplify our analysis by ignoring everything about the input except the position of if it is found in L. Thus, we have distinct possible events: That is in one of positions 0 to in L (each position having its own probability), or that it is not in at all. We can express the probability that is not in L as

where is the probability of event .

Let be the probability that is in position of L (indexed from 0 to . For any position in the list, we must look at records to reach it. So we say that the cost when is in position is . When is not in L, sequential search will require comparisons. Let be the probability that is not in L. Then the average cost will be

What happens to the equation if we assume all the ‘s are equal (except )?

Depending on the value of , .

13.3.1.1. Lower Bounds Proofs

Given an (unsorted) list L of elements and a search key , we seek to identify one element in L which has key value , if any exists. For the rest of this discussion, we will assume that the key values for the elements in L are unique, that the set of all possible keys is totally ordered (that is, the operations , , and are defined for all pairs of key values), and that comparison is our only way to find the relative ordering of two keys. Our goal is to solve the problem using the minimum number of comparisons.

Given this definition for searching, we can easily come up with the standard sequential search algorithm, and we can also see that the lower bound for this problem is “obviously” comparisons. (Keep in mind that the key might not actually appear in the list.) However, lower bounds proofs are a bit slippery, and it is instructive to see how they can go wrong.

Theorem 13.3.1

The lower bound for the problem of searching in an unsorted list is comparisons.

Here is our first attempt at proving the theorem.

Proof 1

We will try a proof by contradiction. Assume an algorithm exists that requires only (or less) comparisons of with elements of L. Because there are elements of L, must have avoided comparing with L []. We can feed the algorithm an input with in position . Such an input is legal in our model, so the algorithm is incorrect.

Is this proof correct? Hopefully it is reasonably obvious to you that not all algorithms must search through the list in a specific order, so not all algorithms have to look at position L [] last.

OK, so we can try to dress up the proof by making the process a bit more flexible.

Proof 2

We will try a proof by contradiction. Assume an algorithm exists that requires only (or less) comparisons of with elements of L. Because there are elements of L, must have avoided comparing with L [] for some value . We can feed the algorithm an input with in position . Such an input is legal in our model, so the algorithm is incorrect.

Is this proof correct? Still, no. First of all, any given algorithm need not necessarily consistently skip any given position in its searches. For example, it is not necessary that all algorithms search the list from left to right. It is not even necessary that all algorithms search the same positions first each time through the list. Perhaps it picks them at random.

Again, we can try to dress up the proof as follows.

Proof 3

On any given run of the algorithm, if elements are compared against , then some element position (call it position ) gets skipped. It is possible that is in position at that time, and will not be found. Therefore, comparisons are required.

Unfortunately, there is another error that needs to be fixed. It is not true that all algorithms for solving the problem must work by comparing elements of L against . An algorithm might make useful progress by comparing elements of L against each other. For example, if we compare two elements of L, then compare the greater against and find that this element is less than , we know that the other element is also less than . It seems intuitively obvious that such comparisons won’t actually lead to a faster algorithm, but how do we know for sure? We somehow need to generalize the proof to account for this approach.

We will now present a useful abstraction for expressing the state of knowledge for the value relationships among a set of objects. A total order defines relationships within a collection of objects such that for every pair of objects, one is greater than the other. A partially ordered set or poset is a set on which only a partial order is defined. That is, there can be pairs of elements for which we cannot decide which is “greater”. For our purpose here, the partial order is the state of our current knowledge about the objects, such that zero or more of the order relations between pairs of elements are known. We can represent this knowledge by drawing directed acyclic graphs (DAGs) showing the known relationships, as illustrated by the following slideshow.

Proof 4

Initially, we know nothing about the relative order of the elements in L, or their relationship to . So initially, we can view the elements in L as being in separate partial orders. Any comparison between two elements in L can affect the structure of the partial orders.

Now, every comparison between elements in L can at best combine two of the partial orders together. Any comparison between and an element, say , in L can at best eliminate the partial order that contains . Thus, if we spend comparisons comparing elements in L we have at least partial orders. Every such partial order needs at least one comparison against to make sure that is not somewhere in that partial order. Thus, any algorithm must make at least comparisons in the worst case.

13.4. Search in Sorted Arrays

13.4.1. Analysis

For large collections of records that are searched repeatedly, sequential search is unacceptably slow. One way to reduce search time is to preprocess the records by sorting them. Given a sorted array, an obvious improvement over simple linear search is to test if the current element in L is greater than . If it is, then we know that cannot appear later in the array, and we can quit the search early. But this still does not improve the worst-case cost of the algorithm.

13.4.1.1. Jump Search

We can also observe that if we look first at position 1 in sorted array L and find that K is bigger, then we rule out position 0 as well as position 1. Because more is often better, what if we look at position 2 in L and find that is bigger yet? This rules out positions 0, 1, and 2 with one comparison. What if we carry this to the extreme and look first at the last position in L and find that is bigger? Then we know in one comparison that is not in L. This is useful to know, but what is wrong with the conclusion that we should always start by looking at the last position? The problem is that, while we learn a lot sometimes (in one comparison we might learn that is not in the list), usually we learn only a little bit (that the last element is not ).

The question then becomes: What is the right amount to jump? This leads us to an algorithm known as Jump Search. For some value , we check every ‘th element in L, that is, we check elements , , and so on. So long as is greater than the values we are checking, we continue on. But when we reach a value in L greater than , we do a linear search on the piece of length that we know brackets if it is in the list.

If we define such that , then the total cost of this algorithm is at most 3-way comparisons. (They are 3-way because at each comparison of with some we need to know if is less than, equal to, or greater than .) Therefore, the cost to run the algorithm on items with a jump of size is

What is the best value that we can pick for ? We want to minimize the cost:

Take the derivative and solve for to find the minimum, which is . In this case, the worst case cost will be roughly .

This example invokes a basic principle of algorithm design. We want to balance the work done while selecting a sublist with the work done while searching a sublist. In general, it is a good strategy to make subproblems of equal effort. This is an example of a divide and conquer algorithm.

What if we extend this idea to three levels? We would first make jumps of some size to find a sublist of size whose end values bracket value . We would then work through this sublist by making jumps of some smaller size, say . Finally, once we find a bracketed sublist of size , we would do sequential search to complete the process.

This probably sounds convoluted to do two levels of jumping to be followed by a sequential search. While it might make sense to do a two-level algorithm (that is, jump search jumps to find a sublist and then does sequential search on the sublist), it almost never seems to make sense to do a three-level algorithm. Instead, when we go beyond two levels, we nearly always generalize by using recursion. This leads us to the most commonly used search algorithm for sorted arrays, the binary search.

13.4.1.2. Binary Search

You are probably pretty familiar with Binary Search already. So that we have a concrete example to discuss, here is an implementation.

Of couse you know that Binary Search is far better than Sequential Why would that be? Because we have additional information to work with that we do not have when the list is unsorted. You probably already “know” that the standard binary search algorithm has a worst case cost of . Let’s do the math to make sure that it really is in , and see how to handle the nasty details of modeling the exact behavior of a recursive algorithm. After that, we can deal with proving that Binary Search is indeed optimal (at least in the worst case) for solving the problem of search in a sorted list.

If we are willing to be casual about our analysis, we can reason that we look at one element (for a cost of one), and then repeat the process on half of the array. This would give us a recurrence that looks like . But if we want to be more precise, then we need to think carefully about what is going on in the worst case. First, we should notice that we are doing a little more than cutting the array in half. We never look again at a particular position that we test. For example, if the input size is nine, then we actually look at position 4 (since when rounded down), and we then either continue to consider four positions to the left (positions 0 to 3) or four positions to the right (positions 5 to 8). But what if there are ten element? Then we actually look at position 5 (since ). We will then either need to continue dealing with five positions to the left (positions 0 to 4), or four positions to the right. Which means that in the worst case, we are looking at a little less than half when the array size is odd, or exactly half when the array size is even. To capture this, we can use the floor function, to get an exact worst case model as follows:

$$
\begin{aligned}f(n) = \left{

\right.\end{aligned}
$$

Since , and since is assumed to be non-decreasing (since adding more elements won’t decrease the work) we can estimate the upper bound with the simplification .

This recurrence is fairly easy to solve via expansion:

Then, collapse to

Now, we can prove that this is correct with induction.

By the IH, .

How do we calculate the average cost for Binary Search? This requires some modeling, because we need to know things about the probabilities of the various inputs. We will estimate given these assumptions:

1. is in L.
2. is equally likely to be in any position.
3. for some non-negative integer .

What is the cost?

• There is one chance to hit in one probe.
• There are two chances to hit in two probes.
• There are chances to hit in probes.
• .

What is the resulting equation?

Note that .

To solve the summation:

Note that in the above series of equations, we change variables: .

Now what? Subtract from the original!

Note that

So,

Now we come back to solving the original equation. Since we have a closed-form solution for the summation in hand, we can restate the equation with the appropriate variable substitutions.

So the average cost is only about one or two comparisons less than the worst cost.

If we want to relax the assumption that , we get this as the exact cost:

$$
\begin{aligned}f(n) = \left{

\right.\end{aligned}
$$

Identify each of the components of this equation as follows:

• Left side:
• has no additional cost, with chance
• Right side:

13.4.1.3. Lower Bounds Proof

So, time for Binary Search seems pretty good. Can we do better than this? We can prove that this is the best possible algorithm in the worst case for searching in a sorted list by using a proof similar to that used to show the lower bound on sorting.

We use the decision tree to model our algorithm. Unlike when searching an unsorted list, comparisons between elements of L tell us nothing new about their relative order (since L is already sorted), so we consider only comparisons between and an element in L. At the root of the decision tree, our knowledge rules out no positions in L, so all are potential candidates. As we take branches in the decision tree based on the result of comparing to an element in L, we gradually rule out potential candidates. Eventually we reach a leaf node in the tree representing the single position in L that can contain . There must be at least nodes in the tree because we have distinct positions that can be in (any position in L, plus not in L at all). Some path in the tree must be at least levels deep, and the deepest node in the tree represents the worst case for that algorithm. Thus, any algorithm on a sorted array requires at least comparisons in the worst case.

We can modify this proof to find the average cost lower bound. Again, we model algorithms using decision trees. Except now we are interested not in the depth of the deepest node (the worst case) and therefore the tree with the least-deepest node. Instead, we are interested in knowing what the minimum possible is for the “average depth” of the leaf nodes. Define the total path length as the sum of the levels for each node. The cost of an outcome is the level of the corresponding node plus 1. The average cost of the algorithm is the average cost of the outcomes (total path length / ). What is the tree with the least average depth? This is equivalent to the tree that corresponds to binary search. Thus, binary search is optimal in the average case.

While binary search is indeed an optimal algorithm for a sorted list in the worst and average cases when searching a sorted array, there are a number of circumstances that might lead us to select another algorithm instead. One possibility is that we know something about the distribution of the data in the array. If each position in L is equally likely to hold (equivalently, the data are well distributed along the full key range), then an interpolation search is in the average case. If the data are not sorted, then using binary search requires us to pay the cost of sorting the list in advance, which is only worthwhile if many (at least searches will be performed on the list. Binary search also requires that the list (even if sorted) be implemented using an array or some other structure that supports random access to all elements with equal cost. Finally, if we know all search requests in advance, we might prefer to sort the list by frequency and do linear search in extreme search distributions, or use a self-organizing list.

13.4.1.4. Interpolation and Quadratic Binary Search

If we know nothing about the distribution of key values, then we have just proved that binary search is the best algorithm available for searching a sorted array. However, sometimes we do know something about the expected key distribution. Consider the typical behavior of a person looking up a word in a large dictionary. Most people certainly do not use sequential search! Typically, people use a modified form of binary search, at least until they get close to the word that they are looking for. The search generally does not start at the middle of the dictionary. People looking for a word starting with ‘S’ generally assume that entries beginning with ‘S’ start about three quarters of the way through the dictionary. Thus, they will first open the dictionary about three quarters of the way through and then make a decision based on what is found as to where to look next. In other words, people typically use some knowledge about the expected distribution of key values to “compute” where to look next. This form of “computed” binary search is called a dictionary search or interpolation search. In a dictionary search, we search L at a position that is appropriate to the value of as follows.

This equation is computing the position of as a fraction of the distance between the smallest and largest key values. This will next be translated into that position which is the same fraction of the way through the array, and this position is checked first. As with binary search, the value of the key found eliminates all records either above or below that position. The actual value of the key found can then be used to compute a new position within the remaining range of the array. The next check is made based on the new computation. This proceeds until either the desired record is found, or the array is narrowed until no records are left.

A variation on dictionary search is known as (QBS), and we will analyze this in detail because its analysis is easier than that of the general dictionary search. QBS will first compute (p) and then examine . If then QBS will sequentially probe to the left by steps of size , that is, we step through

until we reach a value less than or equal to . Similarly for we will step to the right by until we reach a value in L that is greater than . We are now within positions of . Assume (for now) that it takes a constant number of comparisons to bracket within a sublist of size . We then take this sublist and repeat the process recursively. That is, at the next level we compute an interpolation to start somewhere in the subarray. We then step to the left or right (as appropriate) by steps of size .

What is the cost for QBS? Note that , and we will be repeatedly taking square roots of the current sublist size until we find the item that we are looking for. Because and we can cut in half only times, the cost is if the number of probes on jump search is constant.

Say that the number of comparisons needed is , in which case the cost is (since we have to do comparisons). If is the probability of needing exactly probes, then

$$
\begin{aligned}\sum_{i=1}^{\sqrt{n}} i \mathbf{P}(\text{need exactly probes})\cr
= 1 \mathbf{P}_1 + 2 \mathbf{P}_2 + 3 \mathbf{P}3 + \cdots +
\sqrt{n} \mathbf{P}{\sqrt{n}}\end{aligned}
$$

We now show that this is the same as

$$
\begin{aligned}&= 1 + (1-\mathbf{P}_1) + (1-\mathbf{P}_1-\mathbf{P}2) +
\cdots + \mathbf{P}{\sqrt{n}}\cr
&= (\mathbf{P}1 + … + \mathbf{P}{\sqrt{n}}) +
(\mathbf{P}2 + … + \mathbf{P}{\sqrt{n}}) +\cr
&& \qquad (\mathbf{P}3 + … + \mathbf{P}{\sqrt{n}}) + \cdots\cr
&= 1 \mathbf{P}_1 + 2 \mathbf{P}_2 + 3 \mathbf{P}3 + \cdots +
\sqrt{n} \mathbf{P}{\sqrt{n}}\end{aligned}
$$

We require at least two probes to set the bounds, so the cost is

We now make take advantage of a useful fact known as Chebyshev’s Inequality. Chebyshev’s inequality states that , or , is

because for any probability . This assumes uniformly distributed data. Thus, the expected number of probes is

Is QBS better than binary search? Theoretically yes, because grows slower than . However, we have a situation here which illustrates the limits to the model of asymptotic complexity in some practical situations. Yes, does grow faster than . In fact, it is exponentially faster! But even so, for practical input sizes, the absolute cost difference is fairly small. Thus, the constant factors might play a role. First we compare to .

It is not always practical to reduce an algorithm’s growth rate. There is a “practicality window” for every problem, in that we have a practical limit to how big an input we wish to solve for. If our problem size never grows too big, it might not matter if we can reduce the cost by an extra log factor, because the constant factors in the two algorithms might differ by more than the log of the log of the input size.

For our two algorithms, let us look further and check the actual number of comparisons used. For binary search, we need about total comparisons. Quadratic binary search requires about comparisons. If we incorporate this observation into our table, we get a different picture about the relative differences.

But we still are not done. This is only a count of raw comparisons. Binary search is inherently much simpler than QBS, because binary search only needs to calculate the midpoint position of the array before each comparison, while quadratic binary search must calculate an interpolation point which is more expensive. So the constant factors for QBS are even higher.

Not only are the constant factors worse on average, but QBS is far more dependent than binary search on good data distribution to perform well. For example, imagine that you are searching a telephone directory for the name “Young”. Normally you would look near the back of the book. If you found a name beginning with ‘Z’, you might look just a little ways toward the front. If the next name you find also begins with ‘Z’ you would look a little further toward the front. If this particular telephone directory were unusual in that half of the entries begin with ‘Z’, then you would need to move toward the front many times, each time eliminating relatively few records from the search. In the extreme, the performance of interpolation search might not be much better than sequential search if the distribution of key values is badly calculated.

While it turns out that QBS is not a practical algorithm, this is not a typical situation. Fortunately, algorithm growth rates are usually well behaved, so that asymptotic algorithm analysis nearly always gives us a practical indication for which of two algorithms is better.

13.5. Self-Organizing Lists

13.5.1. Introduction

While ordering of lists is most commonly done by key value, this is not the only viable option. Another approach to organizing lists to speed search is to order the records by expected frequency of access. While the benefits might not be as great as when sorted by key value, the cost to organize (at least approximately) by frequency of access can be much cheaper, and thus can speed up sequential search in some situations.

Assume that we know, for each key , the probability that the record with key will be requested. Assume also that list is ordered so that the most frequently requested record is first, then the next most frequently requested record, and so on. Search in the list will be done sequentially, beginning with the first position. Over the course of many searches, the expected number of comparisons required for one search is

$$
\overline{C}n = 1 p_0 + 2 p_1 + … + n p{n-1}.
$$

In other words, the cost to access the record in is 1 (because one key value is looked at), and the probability of this occurring is . The cost to access the record in is 2 (because we must look at the first and the second records’ key values), with probability , and so on. For records, assuming that all searches are for records that actually exist, the probabilities through must sum to one.

Certain probability distributions give easily computed results.

Example 13.5.1

Calculate the expected cost to search a list when each record has equal chance of being accessed (the classic sequential search through an unsorted list). Setting yields

$$
\overline{C}n = \sum{i=1}^n i/n = (n+1)/2.
$$

This result matches our expectation that half the records will be accessed on average by normal sequential search. If the records truly have equal access probabilities, then ordering records by frequency yields no benefit. In the more general case, we must consider the probability (labeled ) that the search key does not match that for any record in the array. In that case, the general formula gives us

Thus, , depending on the value of .

A geometric probability distribution can yield quite different results.

Example 13.5.2

Calculate the expected cost for searching a list ordered by frequency when the probabilities are defined as

$$
\begin{aligned}p_i = \left{ \right.\end{aligned}
$$

Then,

$$
\overline{C}n \approx \sum{i=0}^{n-1} (i+1)/2^{i+1} =
\sum_{i=1}^n (i/2^i) \approx 2.
$$

For this example, the expected number of accesses is a constant. This is because the probability for accessing the first record is high (one half), the second is much lower (one quarter) but still much higher than for the third record, and so on. This shows that for some probability distributions, ordering the list by frequency can yield an efficient search technique.

In many search applications, real access patterns follow a rule of thumb called the 80/20 rule. The 80/20 rule says that 80% of the record accesses are to 20% of the records. The values of 80 and 20 are only estimates; every data access pattern has its own values. However, behavior of this nature occurs surprisingly often in practice (which explains the success of caching techniques widely used by web browsers for speeding access to web pages, and the use of a buffer pool to speed access to data stored in slower memory such as a disk drive). When the 80/20 rule applies, we can expect considerable improvements to search performance from a list ordered by frequency of access over standard sequential search in an unordered list.

Example 13.5.3

The 80/20 rule is an example of a Zipf distribution. Naturally occurring distributions often follow a Zipf distribution. Examples include the observed frequency for the use of words in a natural language such as English, and the size of the population for cities (i.e., view the relative proportions for the populations as equivalent to the “frequency of use”). Zipf distributions are related to the Harmonic Series. Define the Zipf frequency for item in the distribution for records as . The expected cost for the series whose members follow this Zipf distribution will be

$$
\overline{C}n = \sum{i=1}^n i/i {\cal H}_n = n/{\cal H}_n \approx
n/\log_e n.
$$

When a frequency distribution follows the 80/20 rule, the average search looks at about 10-15% of the records in a list ordered by frequency.

This is potentially a useful observation that typical “real-life” distributions of record accesses, if the records were ordered by frequency, would require that we visit on average only 10-15% of the list when doing sequential search. This means that if we had an application that used sequential search, and we wanted to make it go a bit faster (by a constant amount), we could do so without a major rewrite to the system to implement something like a search tree. But that is only true if there is an easy way to (at least approximately) order the records by frequency.

In most applications, we have no means of knowing in advance the frequencies of access for the data records. To complicate matters further, certain records might be accessed frequently for a brief period of time, and then rarely thereafter. Thus, the probability of access for records might change over time (in most database systems, this is to be expected). Self-organizing lists seek to solve both of these problems.

Self-organizing lists modify the order of records within the list based on the actual pattern of record access. Self-organizing lists use a heuristic for deciding how to reorder the list. These heuristics are similar to the rules for managing buffer pools. In fact, a buffer pool is a form of self-organizing list. Ordering the buffer pool by expected frequency of access is a good strategy, because typically we must search the contents of the buffers to determine if the desired information is already in main memory. When ordered by frequency of access, the buffer at the end of the list will be the one most appropriate for reuse when a new page of information must be read.

13.5.1.1. Frequency Count

There are three traditional heuristics for managing self-organizing lists.

The most obvious way to keep a list ordered by frequency would be to store a count of accesses to each record and always maintain records in this order. This method will be referred to as frequency count or just “count”. Count is similar to the least frequently used buffer replacement strategy. Whenever a record is accessed, it might move toward the front of the list if its number of accesses becomes greater than a record preceding it. Thus, count will store the records in the order of frequency that has actually occurred so far. Besides requiring space for the access counts, count does not react well to changing frequency of access over time. Once a record has been accessed a large number of times under the frequency count system, it will remain near the front of the list regardless of further access history.

13.5.2. Move to Front

Bring a record to the front of the list when it is found, pushing all the other records back one position. This is analogous to the least recently used buffer replacement strategy and is called move-to-front. This heuristic is easy to implement if the records are stored using a linked list. When records are stored in an array, bringing a record forward from near the end of the array will result in a large number of records (slightly) changing position. Move-to-front’s cost is bounded in the sense that it requires at most twice the number of accesses required by the optimal static ordering for records when at least searches are performed. In other words, if we had known the series of (at least ) searches in advance and had stored the records in order of frequency so as to minimize the total cost for these accesses, this cost would be at least half the cost required by the move-to-front heuristic. (This can be proved using amortized analysis.) Finally, move-to-front responds well to local changes in frequency of access, in that if a record is frequently accessed for a brief period of time it will be near the front of the list during that period of access. Move-to-front does poorly when the records are processed in sequential order, especially if that sequential order is then repeated multiple times.

13.5.3. Transpose

Swap any record found with the record immediately preceding it in the list. This heuristic is called transpose. Transpose is good for list implementations based on either linked lists or arrays. Frequently used records will, over time, move to the front of the list. Records that were once frequently accessed but are no longer used will slowly drift toward the back. Thus, it appears to have good properties with respect to changing frequency of access. Unfortunately, there are some pathological sequences of access that can make transpose perform poorly. Consider the case where the last record of the list (call it ) is accessed. This record is then swapped with the next-to-last record (call it ), making the last record. If is now accessed, it swaps with . A repeated series of accesses alternating between and will continually search to the end of the list, because neither record will ever make progress toward the front. However, such pathological cases are unusual in practice. A variation on transpose would be to move the accessed record forward in the list by some fixed number of steps.

13.5.3.1. An Example

While self-organizing lists do not generally perform as well as search trees or a sorted list, both of which require search time, there are many situations in which self-organizing lists prove a valuable tool. Obviously they have an advantage over sorted lists in that they need not be sorted. This means that the cost to insert a new record is low, which could more than make up for the higher search cost when insertions are frequent. Self-organizing lists are simpler to implement than search trees and are likely to be more efficient for small lists. Nor do they require additional space. Finally, in the case of an application where sequential search is “almost” fast enough, changing an unsorted list to a self-organizing list might speed the application enough at a minor cost in additional code.

As an example of applying self-organizing lists, consider an algorithm for compressing and transmitting messages. [^1] The list is self-organized by the move-to-front rule. Transmission is in the form of words and numbers, by the following rules:

1. If the word has been seen before, transmit the current position of the word in the list. Move the word to the front of the list.
2. If the word is seen for the first time, transmit the word. Place the word at the front of the list.

Both the sender and the receiver keep track of the position of words in the list in the same way (using the move-to-front rule), so they agree on the meaning of the numbers that encode repeated occurrences of words. Consider the following example message to be transmitted (for simplicity, ignore case in letters).

The car on the left hit the car I left

The first three words have not been seen before, so they must be sent as full words. The fourth word is the second appearance of “the” which at this point is the third word in the list. Thus, we only need to transmit the position value “3”. The next two words have not yet been seen, so must be sent as full words. The seventh word is the third appearance of “the”, which coincidentally is again in the third position. The eighth word is the second appearance of “car”, which is now in the fifth position of the list. “I” is a new word, and the last word “left” is now in the fifth position. Thus the entire transmission would be

The car on 3 left hit 3 5 I 5

This approach to compression is similar in spirit to Ziv-Lempel coding, which is a class of coding algorithms commonly used in file compression utilities. Ziv-Lempel coding replaces repeated occurrences of strings with a pointer to the location in the file of the first occurrence of the string. The codes are stored in a self-organizing list in order to speed up the time required to search for a string that has previously been seen.

[^1]: The compression algorithm and the example used both come from the following paper: J.L. Bentley, D.D. Sleator, R.E. Tarjan, and V.K. Wei, “A Locally Adaptive Data Compression Scheme”, Communications of the ACM 29, 4(April 1986), 320-330.

Chapter 14: Additional Topics

14.1. Dynamic Programming

14.1.1. Dynamic Programming

Dynamic programming is an algorithm design technique that can improve the efficiency of any inherently recursive algorithm that repeatedly re-solves the same subproblems. Using dynamic programming requires two steps:

1. You find a recursive solution to a problem where subproblems are redundantly solved many times.
2. Optimize the recursive algorithm to eliminate re-solving subproblems. The resulting algorithm may be recursive or iterative. The iterative form is commonly referred to by the term dynamic programming.

We will see first how to remove redundancy with a simple, non-optimization problem. We then go to an optimization problem, which will be efficiently solved by dynamic programming.

14.1.1.1. Computing Fibonacci Numbers

Consider the recursive function for computing the ’th Fibonacci number.

/** Recursively generate and return the n'th Fibonacci
    number */
static long fibr(int n) {
  // fibr(91) is the largest value that fits in a long
  if ((n <= 0) || (n > 91)) return -1;
  if ((n == 1) || (n == 2)) return 1;     // Base case
  return fibr(n-1) + fibr(n-2);      // Recursive call
}

The cost of this algorithm (in terms of function calls) is the size of the ’th Fibonacci number itself, which our analysis of Module summation showed to be exponential (approximately ). Why is this so expensive? Primarily because two recursive calls are made by the function, and the work that they do is largely redundant. That is, each of the two calls is recomputing most of the series, as is each sub-call, and so on. Thus, the smaller values of the function are being recomputed a huge number of times. If we could eliminate this redundancy, the cost would be greatly reduced. The approach that we will use can also improve any algorithm that spends most of its time recomputing common subproblems.

The upper half of the following figure shows the recursion tree obtained for n=8, and it has 67 nodes. However, the lower half of the figure shows that the number of unique subproblems is only n+1=9. The latter graphical representation is called a dependency graph, and was obtained from the recursion tree by joining different occurrences of the same recursive call, preserving their corresponding arcs.

Note that the dependency graph was laid out on in a one dimensional table of size 9, corresponding to the unique subproblems invoked by the algorithm. This table can simply store the value of each subproblem. In this way, redundant calls can be avoided because the value of a subproblem which was previously computed can be read from its corresponding cell in the table without the need to recompute it again.

The table can be used to derive two alternative, but efficient, algorithms. One way to accomplish this goal is to keep a table of values, and first check the table to see if the computation can be avoided. This technique is called memoization. Here is a straightforward example of doing so. Note that it mirrors the original version of the Fibonacci recursive algorithm.

int fibrt(int n) {
  // Assume Values has at least n slots, and all
  // slots are initialized to 0
  if ((n <= 0) || (n > 91)) return -1;
  if (n <= 2) return 1;             // Base case
  if (Values[n] == 0)
    Values[n] = fibrt(n-1) + fibrt(n-2);
  return Values[n];
}

This version of the algorithm will not compute a value more than once, so its cost should be linear.

A second technique is called tabulation. The dependency graph must be analyzed to infer an alternative computation order for the subproblems. The only restriction is that a subproblem can only be computed when the subproblems it depends on have been computed. In addition, the value of each subproblem must be stored in the table. In the case of computing a value in the Fibonacci series, we reverse the order to calculate the series from the starting point, and implement this by a simple loop. Unfortunately, since it does not have any similarity to the original recursive algorithm, there is no mechanical way to get from the orginal recursive form to the dynamic programming form.

An additional optimization can be made. Of course, we didn’t actually need to use a table storing all of the values, since future computations do not need access to all prior subproblems. Instead, we could build the value by working from 0 and 1 up to rather than backwards from down to 0 and 1. Going up from the bottom we only need to store the previous two values of the function, as is done by our iterative version.

/** Iteratively generate and return the n'th Fibonacci
    number */
static long fibi(int n) {
  // fibr(91) is the largest value that fits in a long
  if ((n <= 0) || (n > 91)) return -1;
  long curr, prev, past;
  if ((n == 1) || (n == 2)) return 1;
  curr = prev = 1;     // curr holds current Fib value
  for (int i=3; i<=n; i++) { // Compute next value
    past = prev;             // past holds fibi(i-2)
    prev = curr;             // prev holds fibi(i-1)
    curr = past + prev;      // curr now holds fibi(i)
  }
  return curr;
}

Recomputing of subproblems comes up in many algorithms. It is not so common that we can store only a few prior results as we did for fibi. Thus, there are many times where storing a complete table of subresults will be useful.

The approach shown above to designing an algorithm that works by storing a table of results for subproblems is called dynamic programming when it is applied to optimization algorithms. The name is somewhat arcane, because it doesn’t bear much obvious similarity to the process that is taking place when storing subproblems in a table. However, it comes originally from the field of dynamic control systems, which got its start before what we think of as computer programming. The act of storing precomputed values in a table for later reuse is referred to as “programming” in that field. Dynamic programming algorithms are usually implemented with the tabulation technique described above. Thus, fibi better represents the most common form of dynamic programming than does fibrt, even though it doesn’t use the complete table.

14.1.1.2. The Knapsack Problem

We will next consider a problem that appears with many variations in a variety of commercial settings. Many businesses need to package items with the greatest efficiency. One way to describe this basic idea is in terms of packing items into a knapsack, and so we will refer to this as the Knapsack Problem. We will first define a particular formulation of the knapsack problem, and then we will discuss an algorithm to solve it based on dynamic programming. There are many other versions for the problem

Assume that we have a knapsack with a certain amount of space that we will define using integer value . We also have items each with a certain size such that that item has integer size . The problem is to find a subset of the items whose sizes exactly sum to , if one exists. For example, if our knapsack has capacity and the two items are of size and , then no such subset exists. But if we add a third item of size , then we can fill the knapsack exactly with the second and third items. We can define the problem more formally as: Find such that

Example 14.1.1

Assume that we are given a knapsack of size and 10 items of sizes 4, 9, 15, 19, 27, 44, 54, 68, 73, 101. Can we find a subset of the items that exactly fills the knapsack? You should take a few minutes and try to do this before reading on and looking at the answer.

One solution to the problem is: 19, 27, 44, 73.

Example 14.1.2

Having solved the previous example for knapsack of size 163, how hard is it now to solve for a knapsack of size 164? Try it.

Unfortunately, knowing the answer for 163 is of almost no use at all when solving for 164. One solution is: 9, 54, 101.

If you tried solving these examples, you probably found yourself doing a lot of trial-and-error and a lot of backtracking. To come up with an algorithm, we want an organized way to go through the possible subsets. Is there a way to make the problem smaller, so that we can apply divide and conquer? We essentially have two parts to the input: The knapsack size and the items. It probably will not do us much good to try and break the knapsack into pieces and solve the sub-pieces (since we already saw that knowing the answer for a knapsack of size 163 did nothing to help us solve the problem for a knapsack of size 164).

So, what can we say about solving the problem with or without the ’th item? This seems to lead to a way to break down the problem. If the ’th item is not needed for a solution (that is, if we can solve the problem with the first items) then we can also solve the problem when the ’th item is available (we just ignore it). On the other hand, if we do include the ’th item as a member of the solution subset, then we now would need to solve the problem with the first items and a knapsack of size (since the ’th item is taking up space in the knapsack).

To organize this process, we can define the problem in terms of two parameters: the knapsack size and the number of items . Denote a given instance of the problem as . Now we can say that has a solution if and only if there exists a solution for either or . That is, we can solve only if we can solve one of the sub problems where we use or do not use the th item. Of course, the ordering of the items is arbitrary. We just need to give them some order to keep things straight.

Continuing this idea, to solve any subproblem of size , we need only to solve two subproblems of size . And so on, until we are down to only one item that either fills the knapsack or not. This naturally leads to a cost expressed by the recurrence relation . That can be pretty expensive!

But… we should quickly realize that there are only subproblems to solve! Clearly, there is the possibility that many subproblems are being solved repeatedly. This is a natural opportunity to apply dynamic programming. If we draw the recursion tree of this naive recursive algorithm and derive its corresponding dependency graph, we notice that all the recursive calls can be laid out on an array of size to contain the solutions for all subproblems .

As mentioned above, there are two approaches to actually solving the problem. One is memoization, that is, to start with our problem of size and make recursive calls to solve the subproblems, each time checking the array to see if a subproblem has been solved, and filling in the corresponding cell in the array whenever we get a new subproblem solution. The other is tabulation. Conceiveably we could adopt one of several computation orders, although the most “natural” is to start filling the array for row 1 (which indicates a successful solution only for a knapsack of size ). We then fill in the succeeding rows from to , left to right, as follows.

if has a solution,

then has a solution

else if has a solution

then has a solution

else has no solution.

In other words, a new slot in the array gets its solution by looking at most at two slots in the preceding row. Since filling each slot in the array takes constant time, the total cost of the algorithm is .

Example 14.1.3

Solve the Knapsack Problem for and five items with sizes 9, 2, 7, 4, 1. We do this by building the following array.

Key:

-: No solution for .

O: Solution(s) for with omitted.

I: Solution(s) for with included.

I/O: Solutions for with included AND omitted.

For example, stores value I/O. It contains O because has a solution. It contains I because has a solution. Since is marked with an I, it has a solution. We can determine what that solution actually is by recognizing that it includes the 5th item (of size 1), which then leads us to look at the solution for . This in turn has a solution that omits the 4th item, leading us to . At this point, we can either use the third item or not. We can find a solution by taking one branch. We can find all solutions by following all branches when there is a choice.

Note that the table is first filled with the values of the different subproblems, and later we inferred the sequence of decisions that allows computing an optimal solution from the values stored in the table. This last phase of the algorithm precludes the possibility of actually reducing the size of the table. Otherwise, the table for the knapsack problem could have been reduced to a one dimensional array.

14.2. The Sparse Matrix

Sometimes we need to represent a large, two-dimensional matrix where many of the elements have a value of zero. A difficult situation arises when the vast majority of values stored in an matrix are zero, but there is no restriction on which positions are zero and which are non-zero. This is known as a sparse matrix.

One approach to representing a sparse matrix is to concatenate (or otherwise combine) the row and column coordinates into a single value and use this as a key in a hash table. Thus, if we want to know the value of a particular position in the matrix, we search the hash table for the appropriate key. If a value for this position is not found, it is assumed to be zero. This is an ideal approach when all queries to the matrix are in terms of access by specified position. However, if we wish to find the first non-zero element in a given row, or the next non-zero element below the current one in a given column, or recover all of the non-zero values in a given column, then the hash table requires us to check sequentially through the entire table.

Another approach is to implement the matrix as an orthogonal list. Consider the following sparse matrix:

The corresponding orthogonal array is shown in the Figure. Here we have a list of row headers, each of which contains a pointer to a list of matrix records. A second list of column headers also contains pointers to matrix records. Each non-zero matrix element stores pointers to its non-zero neighbors in the row, both following and preceding it. Each non-zero element also stores pointers to its non-zero neighbors following and preceding it in the column. Thus, each non-zero element stores its own value, its position within the matrix, and four pointers. Non-zero elements are found by traversing a row or column list. Note that the first non-zero element in a given row could be in any column; likewise, the neighboring non-zero element in any row or column list could be at any (higher) row or column in the array.

{width=40%}

Figure 14.2.1: A representative orthogonal list sparse matrix representation. Depending on the needs of the application, a given cell might store references as part of a singly linked or doubly linked list, and the cell might store row/column positions for the cell or might store references back to the row/column headers.

What exactly should be stored in the various (non-zero) cells of the sparse matrix depends on the application. In some cases, knowing the row/column locations for individual cells is important. For example, if we want to normal math operations on matricies, such as add two matricies that are stored using the sparse matrix representation, then it is an important part of each cell comparison to know were exactly we are in the array any natural traversal of the arrays. Thus, each non-zero element would store its row and column position explicitly. To find if a particular position in the matrix contains a non-zero element, we traverse the appropriate row or column list. For example, when looking for the element at Row 7 and Column 1, we can traverse the list either for Row 7 or for Column 1. When traversing a row or column list, if we come to an element with the correct position, then its value is non-zero. If we encounter an element with a higher position, then the element we are looking for is not in the sparse matrix. In this case, the element’s value is zero. For example, when traversing the list for Row 7 in the matrix of the figure, we first reach the element at Row 7 and Column 1. If this is what we are looking for, then the search can stop. If we are looking for the element at Row 7 and Column 2, then the search proceeds along the Row 7 list to next reach the element at Column 3. At this point we know that no element at Row 7 and Column 2 is stored in the sparse matrix.

In some applications, a given row or column represents a vector of information about some object. For example, consider if we want to store a database about reviewer ratings of movies. If there are a lot of movies and a lot of reviewers in the database, then no reviewer will have reviewed a signficant fraction of the movies, and no movie will have been reviewed by a significant fraction of reviewers. So a sparse matrix representation might be ideal, where each column stores the ratings information for a given reviewer, and each row stores the ratings information for a given movie. This allows operations like finding all reviews by a given reviewer However, which column a given movie is in is arbitrary. In this case, each (non-zero) cell of the sparse matrix might need to store a reference to its row and column headers (which might provide further information about the record, such as movie and review information), but the cells probably do not need to store meaningless row/column numbers.

Insertion and deletion can be performed by working in a similar way to insert or delete elements within the appropriate row and column lists.

Each non-zero element stored in the sparse matrix representation takes much more space than an element stored in a simple matrix. When is the sparse matrix more space efficient than the standard representation? To calculate this, we need to determine how much space the standard matrix requires, and how much the sparse matrix requires. The size of the sparse matrix depends on the number of non-zero elements (we will refer to this value as NNZ), while the size of the standard matrix representation does not vary. We need to know the (relative) sizes of a pointer and a data value. For simplicity, our calculation will ignore the space taken up by the row and column header (which is not much affected by the number of elements in the sparse array).

As an example, assume that a data value, a row or column index, and a pointer each require four bytes. An matrix requires bytes. The sparse matrix requires 28~bytes per non-zero element (four pointers, two array indices, and one data value). If we set to be the percentage of non-zero elements, we can solve for the value of below which the sparse matrix representation is more space efficient. Using the equation and solving for , we find that the sparse matrix using this implementation is more space efficient when , that is, when less than about 14% of the elements are non-zero. Different values for the relative sizes of data values, pointers, or matrix indices can lead to a different break-even point for the two implementations.

The time required to process a sparse matrix should ideally depend on NNZ. When searching for an element, the cost is the number of elements preceding the desired element on its row or column list. The cost for operations such as adding two matrices should be in the worst case when the one matrix stores non-zero elements and the other stores non-zero elements.

Another representation for sparse matrices is sometimes called the Yale representation. Matlab uses a similar representation, with a primary difference being that the Matlab representation uses column-major order. (Scientific packages tend to prefer column-oriented representations for matrices since this the dominant access need for the operations to be performed.) The Matlab representation stores the sparse matrix using three lists. The first is simply all of the non-zero element values, in column-major order. The second list stores the start position within the first list for each column. The third list stores the row positions for each of the corresponding non-zero values. In the Yale representation, the matrix of the figure above would appear as:

If the matrix has columns, then the total space required will be proportional to . This is good in terms of space. It allows fairly quick access to any column, and allows for easy processing of the non-zero values along a column. However, it does not do a good job of providing access to the values along a row, and is terrible when values need to be added or removed from the representation. Fortunately, when doing computations such as adding or multiplying two sparse matrices, the processing of the input matrices and construction of the output matrix can be done reasonably efficiently.

14.3. Finding the Maximum Value

14.3.1. Finding the Maximum Value

How can we find the th largest value in a sorted list? Obviously we just go to the th position. But what if we have an unsorted list? Can we do better than to sort it? If we are looking for the minimum or maximum value, certainly we can do better than sorting the list. Is this true for the second biggest value? For the median value? In later sections we will examine those questions. For this section, we will continue our examination of lower bounds proofs by reconsidering the simple problem of finding the maximum value in an unsorted list.

Here is a simple algorithm for finding the largest value.

// Return position of largest value in integer array A
static int largest(int[] A) {
  int currlarge = 0;             // Position of largest element seen
  for (int i=1; i<A.length; i++) // For each element
    if (A[currlarge] < A[i])     //   if A[i] is larger
       currlarge = i;            //     remember its position
  return currlarge;              // Return largest position
}

Obviously this algorithm requires comparisons. Is this optimal? It should be intuitively obvious that it is, but let us try to prove it. (Before reading further you might try writing down your own proof.)

Proof 1

The winner must compare against all other elements, so there must be comparisons.

This proof is clearly wrong, because the winner does not need to explicitly compare against all other elements to be recognized. For example, a standard single-elimination playoff sports tournament requires only comparisons, and the winner does not play every opponent. So let’s try again.

Proof 2

Only the winner does not lose. There are losers. A single comparison generates (at most) one (new) loser. Therefore, there must be comparisons.

This proof is sound. However, it will be useful later to abstract this by introducing the concept of posets. We can view the maximum-finding problem as starting with a poset where there are no known relationships, so every member of the collection is in its own separate DAG of one element.

Proof 2a

To find the largest value, we start with a poset of DAGs each with a single element, and we must build a poset having all elements in one DAG such that there is one maximum value (and by implication, losers). We wish to connect the elements of the poset into a single DAG with the minimum number of links. This requires at least links. A comparison provides at most one new link. Thus, a minimum of comparisons must be made.

What is the average cost of largest? Because it always does the same number of comparisons, clearly it must cost comparisons. We can also consider the number of assignments that largest must do. Function largest might do an assignment on any iteration of the for loop.

Because this event does happen, or does not happen, if we are given no information about distribution we could guess that an assignment is made after each comparison with a probability of one half. But this is clearly wrong. In fact, largest does an assignment on the th iteration if and only if A [] is the biggest of the the first elements. Assuming all permutations are equally likely, the probability of this being true is . Thus, the average number of assignments done is

which is the Harmonic Series .

More exactly, is close to .

How “reliable” is this average? That is, how much will a given run of the program deviate from the mean cost? According to Cebysev’s Inequality, an observation will fall within two standard deviations of the mean at least 75% of the time. For Largest, the variance is

The standard deviation is thus about . So, 75% of the observations are between and . Is this a narrow spread or a wide spread? Compared to the mean value, this spread is pretty wide, meaning that the number of assignments varies widely from run to run of the program.

14.4. Adversarial Lower Bounds Proofs

Our next problem will be finding the second largest in a collection of objects. Consider what happens in a standard single-elimination tournament. Even if we assume that the “best” team wins in every game, is the second best the one that loses in the finals? Not necessarily. We might expect that the second best must lose to the best, but they might meet at any time.

Let us go through our standard “algorithm for finding algorithms” by first proposing an algorithm, then a lower bound, and seeing if they match. Unlike our analysis for most problems, this time we are going to count the exact number of comparisons involved and attempt to minimize this count. A simple algorithm for finding the second largest is to first find the maximum (in comparisons), discard it, and then find the maximum of the remaining elements (in comparisons) for a total cost of comparisons. Is this optimal? That seems doubtful, but let us now proceed to the step of attempting to prove a lower bound.

Theorem 14.4.1

The lower bound for finding the second largest value is .

Proof

Any element that loses to anything other than the maximum cannot be second. So, the only candidates for second place are those that lost to the maximum. Function largest might compare the maximum element to others. Thus, we might need additional comparisons to find the second largest.

This proof is wrong. It exhibits the necessary fallacy: “Our algorithm does something, therefore all algorithms solving the problem must do the same.”

This leaves us with our best lower bounds argument at the moment being that finding the second largest must cost at least as much as finding the largest, or . Let us take another try at finding a better algorithm by adopting a strategy of divide and conquer. What if we break the list into halves, and run largest on each half? We can then compare the two winners (we have now used a total of comparisons), and remove the winner from its half. Another call to largest on the winner’s half yields its second best. A final comparison against the winner of the other half gives us the true second place winner. The total cost is . Is this optimal? What if we break the list into four pieces? The best would be . What if we break the list into eight pieces? Then the cost would be about . Notice that as we break the list into more parts, comparisons among the winners of the parts becomes a larger concern.

Looking at this another way, the only candidates for second place are losers to the eventual winner, and our goal is to have as few of these as possible. So we need to keep track of the set of elements that have lost in direct comparison to the (eventual) winner. We also observe that we learn the most from a comparison when both competitors are known to be larger than the same number of other values. So we would like to arrange our comparisons to be against “equally strong” competitors. We can do all of this with a defit{binomial tree}. A binomial tree of height has nodes. Either it is a single node (if ), or else it is two height binomial trees with one tree’s root becoming a child of the other. Let’s see how a binomial tree with eight nodes would be constructed.

The resulting algorithm is simple in principle: Build the binomial tree for all elements, and then compare the children of the root to find second place. We could store the binomial tree as an explicit tree structure, and easily build it in time linear on the number of comparisons as each comparison requires one link be added. Because the shape of a binomial tree is heavily constrained, we can also store the binomial tree implicitly in an array, much as we do for a heap. Assume that two trees, each with nodes, are in the array. The first tree is in positions 1 to 2^k. The second tree is in positions to . The root of each subtree is in the final array position for that subtree.

To join two trees, we simply compare the roots of the subtrees. If necessary, swap the subtrees so that tree with the the larger root element becomes the second subtree. This trades space (we only need space for the data values, no node pointers) for time (in the worst case, all of the data swapping might cost , though this does not affect the number of comparisons required). Note that for some applications, this is an important observation that the array’s data swapping requires no comparisons. If a comparison is simply a check between two integers, then of course moving half the values within the array is too expensive. But if a comparison requires that a competition be held between two sports teams, then the cost of a little bit (or even a lot) of book keeping becomes irrelevent.

Because the binomial tree’s root has children, and building the tree requires comparisons, the number of comparisons required by this algorithm is . This is clearly better than our previous algorithm. Is it optimal?

We now go back to trying to improve the lower bounds proof. To do this, we introduce the concept of an adversary. The adversary’s job is to make an algorithm’s cost as high as possible. Imagine that the adversary keeps a list of all possible inputs. We view the algorithm as asking the adversary for information about the algorithm’s input. The adversary may never lie, in that its answer must be consistent with the previous answers. But it is permitted to “rearrange” the input as it sees fit in order to drive the total cost for the algorithm as high as possible. In particular, when the algorithm asks a question, the adversary must answer in a way that is consistent with at least one remaining input. The adversary then crosses out all remaining inputs inconsistent with that answer. Keep in mind that there is not really an entity within the computer program that is the adversary, and we don’t actually modify the program. The adversary operates merely as an analysis device, to help us reason about the program.

As an example of the adversary concept, consider the standard game of Hangman. Player A picks a word and tells player B how many letters the word has. Player B guesses various letters. If B guesses a letter in the word, then A will indicate which position(s) in the word have the letter. Player B is permitted to make only so many guesses of letters not in the word before losing.

In the Hangman game example, the adversary is imagined to hold a dictionary of words of some selected length. Each time the player guesses a letter, the adversary consults the dictionary and decides if more words will be eliminated by accepting the letter (and indicating which positions it holds) or saying that its not in the word. The adversary can make any decision it chooses, so long as at least one word in the dictionary is consistent with all of the decisions. In this way, the adversary can hope to make the player guess as many letters as possible.

Before explaining how the adversary plays a role in our lower bounds proof, first observe that at least values must lose at least once. This requires at least compares. In addition, at least values must lose to the second largest value. That is, direct losers to the winner must be compared. There must be at least comparisons. The question is: How low can we make ?

Call the strength of element A[i] the number of elements that A[i] is (known to be) bigger than. If A[i] has strength , and A[j] has strength , then the winner has strength . The algorithm gets to know the (current) strengths for each element, and it gets to pick which two elements are compared next. The adversary gets to decide who wins any given comparison. What strategy by the adversary would cause the algorithm to learn the least from any given comparison? It should minimize the rate at which any element improves it strength. It can do this by making the element with the greater strength win at every comparison. This is a “fair” use of an adversary in that it represents the results of providing a worst-case input for that given algorithm.

To minimize the effects of worst-case behavior, the algorithm’s best strategy is to maximize the minimum improvement in strength by balancing the strengths of any two competitors. From the algorithm’s point of view, the best outcome is that an element doubles in strength. This happens whenever , where and are the strengths of the two elements being compared. All strengths begin at zero, so the winner must make at least comparisons when . Thus, there must be at least comparisons. So our algorithm is optimal.

14.5. State Space Lower Bounds Proofs

We now consider the problem of finding both the minimum and the maximum from an (unsorted) list of values. This might be useful if we want to know the range of a collection of values to be plotted, for the purpose of drawing the plot’s scales. Of course we could find them independently in comparisons. A slight modification is to find the maximum in comparisons, remove it from the list, and then find the minimum in further comparisons for a total of comparisons. Can we do better than this?

Before continuing, think a moment about how this problem of finding the minimum and the maximum compares to the problem of the last section, that of finding the second biggest value (and by implication, the maximum). Which of these two problems do you think is harder? It is probably not at all obvious to you that one problem is harder or easier than the other. There is intuition that argues for either case. On the one hand intuition might argue that the process of finding the maximum should tell you something about the second biggest value, more than that process should tell you about the minimum value. On the other hand, any given comparison tells you something about which of two can be a candidate for maximum value, and which can be a candidate for minimum value, thus making progress in both directions.

We will start by considering a simple divide-and-conquer approach to finding the minimum and maximum. Split the list into two parts and find the minimum and maximum elements in each part. Then compare the two minimums and maximums to each other with a further two comparisons to get the final result. The algorithm is as follows:

// Return the minimum and maximum values in A between positions l and r
void MinMax(int A[], int l, int r, int Out[]) {
  if (l == r) {        // n=1
    Out[0] = A[r];
    Out[1] = A[r];
  }
  else if (l+1 == r) { // n=2
    Out[0] = Math.min(A[l], A[r]);
    Out[1] = Math.max(A[l], A[r]);
  }
  else {               // n>2
    int[] Out1 = new int[2];
    int[] Out2 = new int[2];
    int mid = (l + r)/2;
    MinMax(A, l, mid, Out1);
    MinMax(A, mid+1, r, Out2);
    Out[0] = Math.min(Out1[0], Out2[0]);
    Out[1] = Math.max(Out1[1], Out2[1]);
  }
}

The cost of this algorithm can be modeled by the following recurrence.

$$
\begin{aligned}\mathbf{T}(n) = \left{
\right.\end{aligned}
$$

This is a rather interesting recurrence, and its solution ranges between (when or ) and (when ). We can infer from this behavior that how we divide the list affects the performance of the algorithm. For example, what if we have six items in the list? If we break the list into two sublists of three elements, the cost would be 8. If we break the list into a sublist of size two and another of size four, then the cost would only be 7.

With divide and conquer, the best algorithm is the one that minimizes the work, not necessarily the one that balances the input sizes. One lesson to learn from this example is that it can be important to pay attention to what happens for small sizes of , because any division of the list will eventually produce many small lists.

We can model all possible divide-and-conquer strategies for this problem with the following recurrence.

$$
\begin{aligned}\mathbf{T}(n) = \left{
\right.\end{aligned}
$$

That is, we want to find a way to break up the list that will minimize the total work. If we examine various ways of breaking up small lists, we will eventually recognize that breaking the list into a sublist of size 2 and a sublist of size (n-2) will always produce results as good as any other division. This strategy yields the following recurrence.

$$
\begin{aligned}\mathbf{T}(n) = \left{
\right.\end{aligned}
$$

This recurrence (and the corresponding algorithm) yields comparisons. Is this optimal? We now introduce yet another tool to our collection of lower bounds proof techniques: The state space proof.

We will model our algorithm by defining a state that the algorithm must be in at any given instant. We can then define the start state, the end state, and the transitions between states that any algorithm can support. From this, we will reason about the minimum number of states that the algorithm must go through to get from the start to the end, to reach a state space lower bound.

At any given instant, we can track the following four categories of elements:

• Untested: Elements that have not been tested.
• Winners: Elements that have won at least once, and never lost.
• Losers: Elements that have lost at least once, and never won.
• Middle: Elements that have both won and lost at least once.

We define the current state to be a vector of four values, for untested, winners, losers, and middles, respectively. For a set of elements, the initial state of the algorithm is and the end state is . Thus, every run for any algorithm must go from state to state . We also observe that once an element is identified to be a middle, it can then be ignored because it can neither be the minimum nor the maximum.

Given that there are four types of elements, there are 10 types of comparison. Comparing with a middle cannot be more efficient than other comparisons, so we should ignore those, leaving six comparisons of interest. We can enumerate the effects of each comparison type as follows. If we are in state and we have a comparison, then the state changes are as follows.

Now, let us consider what an adversary will do for the various comparisons. The adversary will make sure that each comparison does the least possible amount of work in taking the algorithm toward the goal state. For example, comparing a winner to a loser is of no value because the worst case result is always to learn nothing new (the winner remains a winner and the loser remains a loser). Thus, only the following five transitions are of interest:

Only the last two transition types increase the number of middles, so there must be of these. The number of untested elements must go to 0, and the first transition is the most efficient way to do this. Thus, of these are required. Our conclusion is that the minimum possible number of transitions (comparisons) is . Thus, our algorithm is optimal.

14.6. Finding the th Best Element

We now tackle the problem of finding the th best element in a list. One solution is to sort the list and simply look in the th position. However, this process provides considerably more information than we need to solve the problem. The minimum amount of information that we actually need to know can be visualized as shown in Figure 14.6.1. That is, all we need to know is the items less than our desired value, and the items greater. We do not care about the relative order within the upper and lower groups. So can we find the required information faster than by first sorting? Looking at the lower bound, can we tighten that beyond the trivial lower bound of comparisons? We will focus on the specific question of finding the median element (i.e., the element with rank ), because the resulting algorithm can easily be modified to find the th largest value for any .

{width=40%}

Figure 14.6.1: The poset that represents the minimum information necessary to determine the th element in a list. We need to know which element has values less and values more, but we do not need to know the relationships among the elements with values less or greater than the th element.

Looking at the Quicksort algorithm might give us some insight into solving the median problem. Recall that Quicksort works by selecting a pivot value, partitioning the array into those elements less than the pivot and those greater than the pivot, and moving the pivot to its proper location in the array. If the pivot is in position , then we are done. If not, we can solve the subproblem recursively by only considering one of the sublists. That is, if the pivot ends up in position , then we simply solve by finding the th best element in the left partition. If the pivot is at position , then we wish to find the th element in the right partition.

What is the worst case cost of this algorithm? As with Quicksort, we get bad performance if the pivot is the first or last element in the array. This would lead to possibly performance. However, if the pivot were to always cut the array in half, then our cost would be modeled by the recurrence or cost.

Finding the average cost requires us to use a recurrence with full history, similar to the one we used to model the cost of Quicksort. If we do this, we will find that is in in the average case.

Is it possible to modify our algorithm to get worst-case linear time? To do this, we need to pick a pivot that is guaranteed to discard a fixed fraction of the elements. We cannot just choose a pivot at random, because doing so will not meet this guarantee. The ideal situation would be if we could pick the median value for the pivot each time. But that is essentially the same problem that we are trying to solve to begin with.

Notice, however, that if we choose any constant , and then if we pick the median from a sample of size , then we can guarantee that we will discard at least elements. Actually, we can do better than this by selecting small subsets of a constant size (so we can find the median of each in constant time), and then taking the median of these medians. Figure 14.6.2 illustrates this idea.

Figure 14.6.2: A method for finding a pivot for partitioning a list that guarantees at least a fixed fraction of the list will be in each partition. We divide the list into groups of five elements, and find the median for each group. We then recursively find the median of these medians. The median of five elements is guaranteed to have at least two in each partition. The median of three medians from a collection of 15 elements is guaranteed to have at least five elements in each partition.

This observation leads directly to the following algorithm.

• Choose the medians for groups of five elements from the list. Choosing the median of five items can be done in constant time.
• Recursively, select , the median of the medians-of-fives.
• Partition the list into those elements larger and smaller than .

While selecting the median in this way is guaranteed to eliminate a fraction of the elements (leaving at most elements left), we still need to be sure that our recursion yields a linear-time algorithm. We model the algorithm by the following recurrence.

The term comes from computing the median of the medians-of-fives, the term comes from the cost to calculate the median-of-fives (exactly six comparisons for each group of five element), and the term comes from the recursive call of the remaining (up to) 70% of the elements that might be left.

We will prove that this recurrence is linear using the process of constructive induction. We assume that it is linear for some constant , and then show that for all greater than some bound.

This is true for and . This provides a base case that allows us to use induction to prove that .

In reality, this algorithm is not practical because its constant factor costs are so high. So much work is being done to guarantee linear time performance that it is more efficient on average to rely on chance to select the pivot, perhaps by picking it at random or picking the middle value out of the current subarray.

14.7. Optimal Sorting

What if we would like to find the sorting algorithm with the absolute fewest possible comparisons? It might well be that the result will not be practical for a general-purpose use. But consider this analogy to sports tournaments. In sports, a “comparison” between two teams or individuals means doing a competition between the two. This is fairly expensive (at least compared to some minor book keeping in a computer), and it might be worth trading a fair amount of book keeping to cut down on the number of games that need to be played. What if we want to figure out how to hold a tournament that will give us the exact ordering for all teams in the fewest number of total games? Of course, we are assuming that the results of each game will be “accurate” in that we assume not only that the outcome of A playing B would always be the same (at least over the time period of the tournament), but that transitivity in the results also holds. In practice these are unrealistic assumptions, but such assumptions are implicitly part of many tournament organizations. Like most tournament organizers, we can simply accept these assumptions and come up with an algorithm for playing the games that gives us some rank ordering based on the results we obtain.

Recall Insertion Sort, where we put element into a sorted sublist of the first elements. What if we modify the standard Insertion Sort algorithm to use binary search to locate where the th element goes in the sorted sublist? This algorithm is called binary insert sort. As a general-purpose sorting algorithm, this is not practical because we then have to (on average) move about elements to make room for the newly inserted element in the sorted sublist. But if we count only comparisons, binary insert sort is pretty good. And we can use some ideas from binary insert sort to get closer to an algorithm that uses the absolute minimum number of comparisons needed to sort.

Consider what happens when we run binary insert sort on five elements. How many comparisons do we need to do? We can insert the second element with one comparison, the third with two comparisons, and the fourth with 2 comparisons. When we insert the fifth element into the sorted list of four elements, we need to do three comparisons in the worst case. Notice exactly what happens when we attempt to do this insertion. We compare the fifth element against the second. If the fifth is bigger, we have to compare it against the third, and if it is bigger we have to compare it against the fourth. In general, when is binary search most efficient? When we have elements in the list. It is least efficient when we have elements in the list. So, we can do a bit better if we arrange our insertions to avoid inserting an element into a list of size if possible.

Figure 14.7.1 illustrates a different organization for the comparisons that we might do. First we compare the first and second element, and the third and fourth elements. The two winners are then compared, yielding a binomial tree. We can view this as a (sorted) chain of three elements, with element hanging off from the root. If we then insert element into the sorted chain of three elements, we will end up with one of the two posets shown on the right side of Figure 14.7.1, at a cost of 2 comparisons. We can then merge into the chain, for a cost of two comparisons (because we already know that it is smaller then either one or two elements, we are actually merging it into a list of two or three elements). Thus, the total number of comparisons needed to sort the five elements is at most seven instead of eight.

Figure 14.7.1: Organizing comparisons for sorting five elements. First we order two pairs of elements, and then compare the two winners to form a binomial tree of four elements. The original loser to the root is labeled , and the remaining three elements form a sorted chain. We then insert element into the sorted chain. Finally, we put into the resulting chain to yield a final sorted list.

If we have ten elements to sort, we can first make five pairs of elements (using five compares) and then sort the five winners using the algorithm just described (using seven more compares). Now all we need to do is to deal with the original losers. We can generalize this process for any number of elements as:

• Pair up all the nodes with comparisons.
• Recursively sort the winners.
• Fold in the losers.

We use binary insert to place the losers. However, we are free to choose the best ordering for inserting, keeping in mind the fact that binary search has the same cost for through items. For example, binary search requires three comparisons in the worst case for lists of size 4, 5, 6, or 7. So we pick the order of inserts to optimize the binary searches, which means picking an order that avoids growing a sublist size such that it crosses the boundary on list size to require an additional comparison. This sort is called merge insert sort, and also known as the Ford and Johnson sort.

For ten elements, given the poset shown in Figure 14.7.2 we fold in the last four elements (labeled 1 to 4) in the order Element 3, Element 4, Element 1, and finally Element 2. Element 3 will be inserted into a list of size three, costing two comparisons. Depending on where Element 3 then ends up in the list, Element 4 will now be inserted into a list of size 2 or 3, costing two comparisons in either case. Depending on where Elements 3 and 4 are in the list, Element 1 will now be inserted into a list of size 5, 6, or 7, all of which requires three comparisons to place in sort order. Finally, Element 2 will be inserted into a list of size 5, 6, or 7.

{width=40%}

Figure 14.7.2: Merge insert sort for ten elements. First five pairs of elements are compared. The five winners are then sorted. This leaves the elements labeled 1-4 to be sorted into the chain made by the remaining six elements.

Merge insert sort is pretty good, but is it optimal? We know from the sorting lower bound proof that no sorting algorithm can be faster than . To be precise, the information theoretic lower bound for sorting can be proved to be . That is, we can prove a lower bound of exactly comparisons. Merge insert sort gives us a number of comparisons equal to this information theoretic lower bound for all values up to . At , merge insert sort requires 30 comparisons while the information theoretic lower bound is only 29 comparisons. However, for such a small number of elements, it is possible to do an exhaustive study of every possible arrangement of comparisons. It turns out that there is in fact no possible arrangement of comparisons that makes the lower bound less than 30 comparisons when . Thus, the information theoretic lower bound is an underestimate in this case, because 30 really is the best that can be done.

Call the optimal worst cost for elements . We know that because we could sort elements and use binary insert for the last one. For all and , where is the best time to merge two sorted lists. For , it turns out that we can do better by splitting the list into pieces of size 5 and 42, and then merging. Thus, merge sort is not quite optimal. But it is extremely good, and nearly optimal for smallish numbers of elements.

Chapter 15: Appendix

15.1. Glossary

2-3 tree

A specialized form of the B-tree where each internal node has either 2 children or 3 children. Key values are ordered to maintain the binary search tree property. The 2-3 tree is always height balanced, and its insert, search, and remove operations all have cost.

80/20 rule

Given a typical application where there is a collection of records and a series of search operations for records, the 80/20 rule is an empirical observation that 80% of the record accessess typically go to 20% of the records. The exact values varies between data collections, and is related to the concept of locality of reference.

abstract data type

Abbreviated ADT. The specification of a data type within some language, independent of an implementation. The interface for the ADT is defined in terms of a type and a set of operations on that type. The behavior of each operation is determined by its inputs and outputs. An ADT does not specify how the data type is implemented. These implementation details are hidden from the user of the ADT and protected from outside access, a concept referred to as encapsulation.

accept

When a finite automata executes on a string and terminates in an accepting state, it is said to accept the string. The finite automata is said to accept the language that consists of all strings for which the finite automata completes execution in an accepting state.

accepting state

Part of the definition of a finite automata is to designate some states as accepting states. If the finite automata executes on an input string and completes the computation in an accepting state, then the machine is said to accept the string.

activation record

The entity that is stored on the runtime stack during program execution. It stores any active local variable and the return address from which a new subroutine is being called, so that this information can be recovered when the subroutine terminates.

acyclic graph

In graph terminology, a graph that contains no cycles.

address

A location in memory.

adjacency list

An implementation for a graph that uses an (array-based) list to represent the vertices of the graph, and each vertex is in turn represented by a (linked) list of the vertices that are neighbors.

adjacency matrix

An implementation for a graph that uses a 2-dimensional array where each row and each column corresponds to a vertex in the graph. A given row and column in the matrix corresponds to an edge from the vertex corresponding to the row to the vertex corresponding to the column.

adjacent

Two nodes of a tree or two vertices of a graph are said to be adjacent if they have an edge connecting them. If the edge is directed from to , then we say that is adjacent to , and is adjacent from .

ADT

Abbreviation for abstract data type.

adversary

A fictional construct introduced for use in an adversary argument.

adversary argument

A type of lower bounds proof for a problem where a (fictional) “adversary” is assumed to control access to an algorithm’s input, and which yields information about that input in such a way that will drive the cost for any proposed algorithm to solve the problem as high as possible. So long as the adversary never gives an answer that conflicts with any previous answer, it is permitted to do whatever necessary to make the algorithm require as much cost as possible.

aggregate type

A data type whose members have subparts. For example, a typical database record. Another term for this is composite type.

algorithm

A method or a process followed to solve a problem.

algorithm analysis

A less formal version of the term asymptotic algorithm analysis, generally used as a synonym for asymptotic analysis.

alias

Another name for something. In programming, this usually refers to two references that refer to the same object.

all-pairs shortest paths problem

Given a graph with weights or distances on the edges, find the shortest paths between every pair of vertices in the graph. One approach to solving this problem is Floyd’s algorithm, which uses the dynamic programming algorithmic technique.

allocated

allocation

Reserving memory for an object in the Heap memory.

alphabet

The characters or symbols that strings in a given language may be composed of.

alphabet trie

A trie data structure for storing variable-length strings. Level of the tree corresponds to the letter in position of the string. The root will have potential branches on each intial letter of string. Thus, all strings starting with “a” will be stored in the “a” branch of the tree. At the second level, such strings will be separated by branching on the second letter.

amortized analysis

An algorithm analysis techique that looks at the total cost for a series of operations and amortizes this total cost over the full series. This is as opposed to considering every individual operation to independently have the worst case cost, which might lead to an overestimate for the total cost of the series.

amortized cost

The total cost for a series of operations to be used in an amortized analysis.

ancestor

In a tree, for a given node , any node on a path from up to the root is an ancestor of .

antisymmetric

In set notation, relation is antisymmetric if whenever and , then , for all .

approximation algorithm

An algorthm for an optimization problem that finds a good, but not necessarily cheapest, solution.

arm

In the context of an I/O head, this attaches the sensor on the I/O head to the boom.

array

A data type that is used to store elements in consecutive memory locations and refers to them by an index.

array-based list

An implementation for the list ADT that uses an array to store the list elements. Typical implementations fix the array size at creation of the list, and the overhead is the number of array positions that are presently unused.

array-based queue

Analogous to an array-based list, this uses an array to store the elements when implementing the queue ADT.

array-based stack

Analogous to an array-based list, this uses an array to store the elements when implementing the stack ADT.

ASCII character coding

American Standard Code for Information Interchange. A commonly used method for encoding characters using a binary code. Standard ASCII uses an 8-bit code to represent upper and lower case letters, digits, some punctuation, and some number of non-printing characters (such as carrage return). Now largely replaced by UTF-8 encoding.

assembly code

A form of intermediate code created by a compiler that is easy to convert into the final form that the computer can execute. An assembly language is typically a direct mapping of one or a few instructions that the CPU can execute into a mnemonic form that is relatively easy for a human to read.

asymptotic algorithm analysis

A more formal term for asymptotic analysis.

asymptotic analysis

A method for estimating the efficiency of an algorithm or computer program by identifying its growth rate. Asymptotic analysis also gives a way to define the inherent difficulty of a problem. We frequently use the term algorithm analysis to mean the same thing.

attribute

In object-oriented programming, a synonym for data members.

automata

Synonym for finite state machine.

automatic variable

A synonym for local variable. When program flow enters and leaves the variable’s scope, automatic variables will be allocated and de-allocated automatically.

average case

In algorithm analysis, the average of the costs for all problem instances of a given input size . If not all problem instances have equal probability of occurring, then average case must be calculated using a weighted average.

average seek time

Expected (average) time to perform a seek operation on a disk drive, assuming that the seek is between two randomly selected tracks. This is one of two metrics commonly provided by disk drive vendors for disk drive performance, with the other being track-to-track seek time.

AVL Tree

A variant implementation for the BST, which differs from the standard BST in that it uses modified insert and remove methods in order to keep the tree balanced. Similar to a Splay Tree in that it uses the concept of rotations in the insert and remove operations.

B-tree

A variant on the B-tree. The $\mathrm{B}^\mathrm{B}^+\mathrm{B}^$ tree gives some records to its neighboring sibling, if possible. If the sibling is also full, then these two nodes split into three. Similarly, when a node underflows, it is combined with its two siblings, and the total reduced to two nodes. Thus, the nodes are always at least two thirds full.

B-tree

The most commonly implemented form of B-tree. A B-tree does not store data at the internal nodes, but instead only stores search key values as direction finders for the purpose of searching through the tree. Only the leaf nodes store a reference to the actual data records.

B-tree

A method for indexing a large collection of records. A B-tree is a balanced tree that typically has high branching factor (commonly as much as 100 children per internal node), causing the tree to be very shallow. When stored on disk, the node size is selected to be same as the desired unit of I/O (so some multiple of the disk sector size). This makes it easy to gain access to the record associated with a given search key stored in the tree with few disk accesses. The most commonly implemented variant of the B-tree is the B-tree.

backing storage

In the context of a caching system or buffer pool, backing storage is the relatively large but slower source of data that needs to be cached. For example, in a virtual memory, the disk drive would be the backing storage. In the context of a web browser, the Internet might be considered the backing storage.

backtracking

A heuristic for brute-force search of a solution space. It is essentially a depth-first search of the solution space. This can be improved using a branch-and-bounds algorithm.

bag

In set notation, a bag is a collection of elements with no order (like a set), but which allows for duplicate-valued elements (unlike a set).

balanced tree

A tree where the subtrees meet some criteria for being balanced. Two possibilities are that the tree is height balanced, or that the tree has a roughly equal number of nodes in each subtree.

base

Synonym for radix.

base case

In recursion or proof by induction, the base case is the termination condition. This is a simple input or value that can be solved (or proved in the case of induction) without resorting to a recursive call (or the induction hypothesis).

base class

In object-oriented programming, a class from which another class inherits. The class that inherits is called a subclass.

base type

The data type for the elements in a set. For example, the set might consist of the integer values 3, 5, and 7. In this example, the base type is integers.

basic operation

Examples of basic operations include inserting a data item into the data structure, deleting a data item from the data structure, and finding a specified data item.

best case

In algorithm analysis, the problem instance from among all problem instances for a given input size that has least cost. Note that the best case is not when is small, since we are referring to the best from a class of inputs (i.e, we want the best of those inputs of size ).

best fit

In a memory manager, best fit is a heuristic for deciding which free block to use when allocating memory from a memory pool. Best fit will always allocate from the smallest free block that is large enough to service the memory request. The rationale is that this will be the method that best preserves large blocks needed for unusually large requests. The disadvantage is that it tends to cause external fragmentation in the form of small, unuseable memory blocks.

BFS

Abbreviation for breadth-first search.

big-Oh notation

In algorithm analysis, a shorthand notation for describing the upper bound for an algorithm or problem.

binary insert sort

A variation on insertion sort where the position of the value being inserted is located by binary search, and then put into place. In normal usage this is not an improvement on standard insertion sort because of the expense of moving many items in the array. But it is directly useful if the cost of comparison is high compared to that of moving an element, or is theoretically useful if we only care to count the cost of comparisons.

binary search

A standard recursive algorithm for finding the record with a given search key value within a sorted list. It runs in time. At each step, look at the middle of the current sublist, and throw away the half of the records whose keys are either too small or too large.

binary search tree

A binary tree that imposes the following constraint on its node values: The search key value for any node must be greater than the (key) values for all nodes in the left subtree of , and less than the key values for all nodes in the right subtree of . Some convention must be adopted if multiple nodes with the same key value are permitted, typically these are required to be in the right subtree.

binary search tree property

The defining relationship between the key values for nodes in a BST. All nodes stored in the left subtree of a node whose key value is have key values less than or equal to . All nodes stored in the right subtree of a node whose key value is have key values greater than .

binary tree

A finite set of nodes which is either empty, or else has a root node together two binary trees, called the left and right subtrees, which are disjoint from each other and from the root.

binary trie

A binary tree whose structure is that of a trie. Generally this is an implementation for a search tree. This means that the search key values are thought of a binary digits, with the digit in the position corresponding to this a node’s level in the tree indicating a left branch if it is “0”, or a right branch if it is “1”. Examples include the Huffman coding tree and the Bintree.

binning

In hashing, binning is a type of hash function. Say we are given keys in the range 0 to 999, and have a hash table of size 10. In this case, a possible hash function might simply divide the key value by 100. Thus, all keys in the range 0 to 99 would hash to slot 0, keys 100 to 199 would hash to slot 1, and so on. In other words, this hash function “bins” the first 100 keys to the first slot, the next 100 keys to the second slot, and so on. This approach tends to make the hash function dependent on the distribution of the high-order bits of the keys.

Binsort

A sort that works by taking each record and placing it into a bin based on its value. The bins are then gathered up in order to sort the list. It is generally not practical in this form, but it is the conceptual underpinning of the radix sort.

bintree

A spatial data structure in the form of binary trie, typically used to store point data in two or more dimensions. Similar to a PR quadtree except that at each level, it splits one dimension in half. Since many leaf nodes of the PR quadtree will contain no data points, implementation often makes use of the flyweight design pattern.

bitmap

bit vector

An array that stores a single bit at each position. Typically these bits represent Boolean variables associated with a collection of objects, such that the th bit is the Boolean value for the th object.

block

A unit of storage, usually referring to storage on a disk drive or other peripheral storage device. A block is the basic unit of I/O for that device.

Boolean expression

A Boolean expression is comprised of Boolean variables combined using the operators AND (), OR (), and NOT (to negate Boolean variable we write ).

Boolean variable

A variable that takes on one of the two values True and False.

boom

In the context of an I/O head, is the central structure to which all of the I/O heads are attached. Thus, the all move together during a seek operation.

bounding box

A box (usually aligned to the coordinate axes of the reference system) that contains a (potentially complex) object. In graphics and computational geometry, complex objects might be associated with a bounding box for use by algorithms that search for objects in a particular location. The idea is that if the bounding box is not within the area of interest, then neither is the object. Checking the bounding box is cheaper than checking the object, but it does require some time. So if enough objects are not outside the area of interest, this approach will not save time. But if most objects are outside of the area of interest, then checking bounding boxes first can save a lot of time.

branch-and-bounds algorithm

A variation on backtracking that applies to optimization problems. We traverse the solution tree as with backtracking. Proceeding deeper in the solution tree generally requires additional cost. We remember the best-cost solution found so far. If the cost of the current branch in the tree exceeds the best tour cost found so far, then we know to stop pursuing this branch of the tree. At this point we can immediately back up and take another branch.

breadth-first search

A graph traversal algorithm. As the name implies, all immediate neighbors for a node are visited before any more-distant nodes are visited. BFS is driven by a queue. A start vertex is placed on the queue. Then, until the queue is empty, a node is taken off the queue, visited, and and then any unvisited neighbors are placed onto the queue.

break-even point

The point at which two costs become even when measured as the function of some variable. In particular, used to compare the space requirements of two implementations. For example, when comparing the space requirements of an array-based list implementation versus a linked list implementation, the key issue is how full the list is compared to its capacity limit (for the array-based list). The point where the two representations would have the same space cost is the break-even point. As the list becomes more full beyond this point, the array-based list implementation becomes more space efficent, while as the list becomes less full below this point, the linked list implementation becomes more space efficient.

BST

Abbreviation for binary search tree.

bubble sort

A simple sort that requires time in best, average, and worst cases. Even an optimized version will normally run slower than insertion sort, so it has little to recommend it.

bucket

In bucket hashing, a bucket is a sequence of slots in the hash table that are grouped together.

bucket hashing

A method of hashing where multiple slots of the hash table are grouped together to form a bucket. The hash function then either hashes to some bucket, or else it hashes to a home slot in the normal way, but this home slot is part of some bucket. Collision resolution is handled first by attempting to find a free position within the same bucket as the home slot. If the bucket if full, then the record is placed in an overflow bucket.

bucket sort

A variation on the Binsort, where each bin is associated with a range of key values. This will require some method of sorting the records placed into each bin.

buddy method

In a memory manager, an alternative to using a free block list and a sequential fit method to seach for a suitable free block to service a memory request. Instead, the memory pool is broken down as needed into smaller chunks by splitting it in half repeatedly until the smallest power of 2 that is as big or bigger than the size of the memory request is reached. The name comes from the fact that the binary representation for the start of the block positions only differ by one bit for adjacent blocks of the same size. These are referred to as “buddies” and will be merged together if both are free.

buffer

A block of memory, most often in primary storage. The size of a buffer is typically one or a multiple of the basic unit of I/O that is read or written on each access to secondary storage such as a disk drive.

buffer passing

An approach to implementing the ADT for a buffer pool, where a pointer to a buffer is passed between the client and the buffer pool. This is in contrast to a message passing approach, it is most likely to be used for long messages or when the message size is always the same as the buffer size, such as when implementing a B-tree.

buffer pool

A collection of one or more buffers. The buffer pool is an example of a cache. It is stored in primary storage, and holds data that is expected to be used in the near future. When a data value is requested, the buffer pool is searched first. If the value is found in the buffer pool, then secondary storage need not be accessed. If the value is not found in the buffer pool, then it must be fetched from secondary storage. A number of traditional heuristics have been developed for deciding which data to flush from the buffer pool when new data must be stored, such as least recently used.

buffering

A synonym for caching. More specifically, it refers to an arrangement where all accesses to data (such as on a peripheral storage device) must be done in multiples of some minimum unit of storage. On a disk drive, this basic or smallest unit of I/O is a sector. It is called “buffering” because the block of data returned by such an access is stored in a buffer.

caching

The concept of keeping selected data in main memory. The goal is to have in main memory the data values that are most likely to be used in the near future. An example of a caching technique is the use of a buffer pool.

call stack

Known also as execution stack. A stack that stores the function call sequence and the return address for each function.

Cartesian product

For sets, this is another name for the set product.

ceiling

Written , for real value the ceiling is the least integer .

child

In a tree, the set of nodes directly pointed to by a node are the children of .

circular first fit

In a memory manager, circular first fit is a heuristic for deciding which free block to use when allocating memory from a memory pool. Circular first fit is a minor modification on first fit memory allocation, where the last free block allocated from is remembered, and search for the next suitable free block picks up from there. Like first fit, it has the advantage that it is typically not necessary to look at all free blocks on the free block list to find a suitable free block. And it has the advantage over first fit that it spreads out memory allocations evenly across the free block list. This might help to minimize external fragmentation.

circular list

A list ADT implementation variant where the last element of the list provides access to the first element of the list.

class

In the object-oriented programming paradigm an ADT and its implementation together make up a class. An instantiation of a class within a program is termed an object.

class hierarchy

In object-oriented programming, a set of classes and their interrelationships. One of the classes is the base class, and the others are subclasses that inherit either directly or indirectly from the base class.

clause

In a Boolean expression, a clause is one or more literals OR’ed together.

client

The user of a service. For example, the object or part of the program that calls a memory manager class is the client of that memory manager. Likewise the class or code that calls a buffer pool.

clique

In graph terminology, a clique is a subgraph, defined as any subset of the graph’s vertices such that every vertex in has an edge to every other vertex in . The size of the clique is the number of vertices in the clique.

closed

A set is closed over a (binary) operation if, whenever the operation is applied to two members of the set, the result is a member of the set.

closed hash system

A hash system where all records are stored in slots of the hash table. This is in contrast to an open hash system.

closed-form solution

An algebraic equation with the same value as a summation or recurrence relation. The process of replacing the summation or recurrence with its closed-form solution is known as solving the summation or recurrence.

cluster

In file processing, a collection of physically adjacent sectors that define the smallest allowed allocation unit of space to a disk file. The idea of requiring space to be allocated in multiples of sectors is that this will reduce the number of extents required to store the file, which reduces the expected number of seek operations reuquired to process a series of disk accesses to the file. The disadvantage of large cluster size is that it increases internal fragmentation since any space not actually used by the file in the last cluster is wasted.

code generation

A phase in a compiler that transforms intermediate code into the final executable form of the code. More generally, this can refer to the process of turning a parse tree (that determines the correctness of the structure of the program) into actual instructions that the computer can execute.

code optimization

A phase in a compiler that makes changes in the code (typically assembly code) with the goal of replacing it with a version of the code that will run faster while performing the same computation.

cohesion

In object-oriented programming, a term that refers to the degree to which a class has a single well-defined role or responsibility.

Collatz sequence

For a given integer value , the sequence of numbers that derives from performing the following computatin on :

while (n > 1)
  if (ODD(n))
    n = 3 * n + 1;
  else
    n = n / 2;

This is famous because, while it terminates for any value of that you try, it has never been proven to be a fact that this always terminates.

collision

In a hash system, this refers to the case where two search keys are mapped by the hash function to the same slot in the hash table. This can happen on insertion or search when another record has already been hashed to that slot. In this case, a closed hash system will require a process known as collision resolution to find the location of the desired record.

collision resolution

The outcome of a collision resolution policy.

collision resolution policy

In hashing, the process of resolving a collision. Specifically in a closed hash system, this is the process of finding the proper position in a hash table that contains the desired record if the hash function did not return the correct position for that record due to a collision with another record.

comparable

The concept that two objects can be compared to determine if they are equal or not, or to determine which one is greater than the other. In set notation, elements and of a set are comparable under a given relation if either or . To be reliably compared for a greater/lesser relationship, the values being compared must belong to a total order. In programming, the property of a data type such that two elements of the type can be compared to determine if they the same (a weaker version), or which of the two is larger (a stronger version). Comparable is also the name of an interface in Java that asserts a comparable relationship between objects with a class, and .compareTo() is the Comparable interface method that implements the actual comparison between two objects of the class.

comparator

A function given as a parameter to a method of a library (or alternatively, a parameter for a C++ template or a Java generic). The comparator function concept provides a generic way encapulates the process of performing a comparison between two objects of a specific type. For example, if we want to write a generic sorting routine, that can handle any record type, we can require that the user of the sorting routine pass in a comparator function to define how records in the collection are to be compared.

comparison

The act of comparing two keys or records. For many data types, a comparison has constant time cost. The number of comparisons required is often used as a measure of cost for sorting and searching algorithms.

compile-time polymorphism

A form of polymorphism known as Overloading. Overloaded methods have the same names, but different signatures as a method available elsewhere in the class. Compare to run-time polymorphism.

compiler

A computer program that reads computer programs and converts them into a form that can be directly excecuted by some form of computer. The major phases in a compiler include lexical analysis, syntax analysis, intermediate code generation, code optimization, and code generation. More broadly, a compiler can be viewed as parsing the program to verify that it is syntactically correct, and then doing code generation to convert the hig-level program into something that the computer can execute.

complete binary tree

A binary tree where the nodes are filled in row by row, with the bottom row filled in left to right. Due to this requirement, there is only one tree of nodes for any value of . Since storing the records in an array in row order leads to a simple mapping from a node’s position in the array to its parent, siblings, and children, the array representation is most commonly used to implement the complete binary tree. The heap data structure is a complete binary tree with partial ordering constraints on the node values.

complete graph

A graph where every vertex connects to every other vertex.

complex number

In mathematics, an imaginary number, that is, a number with a real component and an imaginary component.

Composite design pattern

Given a class hierarchy representing a set of objects, and a container for a collection of objects, the composite design pattern addresses the relationship between the object hierarchy and a bunch of behaviors on the objects. In the composite design, each object is required to implement the collection of behaviors. This is in contrast to the procedural approach where a behavior (such as a tree traversal) is implemented as a method on the object collection (such as a tree). Procedural tree traversal requires that the tree have a method that understands what to do when it encounters any of the object types (internal or leaf nodes) that the tree might contain. The composite approach would have the tree call the “traversal” method on its root node, which then knows how to perform the “traversal” behavior. This might in turn require invoking the traversal method of other objects (in this case, the children of the root).

composite type

A type whose members have subparts. For example, a typical database record. Another term for this is aggregate type.

composition

Relationships between classes based on usage rather than inheritance, i.e. a HAS-A relationship. For example, some code in class ‘A’ has a reference to some other class ‘B’.

computability

A branch of computer science that deals with the theory of solving problems through computation. More specificially, it deals with the limits to what problems (functions) are computable. An example of a famous problem that cannot in principle be solved by a computer is the halting problem.

computation

In a finite automata, a computation is a sequence of configurations for some length . In general, it is a series of operations that the machine performs.

computational complexity theory

A branch of the theory of computation in theoretical computer science and mathematics that focuses on classifying computational problems according to their inherent difficulty, and relating those classes to each other. An example is the study of NP-Complete problems.

configuration

For a finite automata, a complete specification for the current condition of the machine on some input string. This includes the current state that the machine is in, and the current condition of the string, including which character is about to be processed.

Conjunctive Normal Form

CNF

A Boolean expression written as a series of clauses that are AND’ed together.

connected component

In an undirected graph, a subset of the nodes such that each node in the subset can be reached from any other node in that subset.

connected graph

An undirected graph is a connected graph if there is at least one path from any vertex to any other.

constant running time

The cost of a function whose running time is not related to its input size. In Theta notation, this is traditionally written as .

constructive induction

A process for finding the closed form for a recurrence relation, that involves substituting in a guess for the closed form to replace the recursive part(s) of the recurrence. Depending on the goal (typically either to show that the hypothesized growth rate is right, or to find the precise constants), one then manipulates the resulting non-recursive equation.

container

container class

A data structure that stores a collection of records. Typical examples are arrays, search trees, and hash tables.

context-free grammar

A grammar comprised only of productions of the form where is a non-terminal and is a series of one or more terminals and non-terminals. That is, the given non-terminal can be replaced at any time.

context-free language

The set of languages that can be defined by context-sensitive grammars.

context-sensitive grammar

A grammar comprised only of productions of the form where is a non-terminal and and are each a series of one or more terminals and non-terminals. That is, the given non-terminal can be replaced only when it is within the proper context.

cost

The amount of resources that the solution consumes.

cost model

In algorithm analysis, a definition for the cost of each basic operation performed by the algorithm, along with a definition for the size of the input. Having these definitions allows us to calculate the cost to run the algorithm on a given input, and from there determine the growth rate of the algorithm. A cost model would be considered “good” if it yields predictions that conform to our understanding of reality.

countably infinite

countable

A set is countably infinite if it contains a finite number of elements, or (for a set with an infinite number of elements) if there exists a one-to-one mapping from the set to the set of integers.

CPU

Acronym for Central Processing Unit, the primary processing device for a computer.

current position

A property of some list ADTs, where there is maintained a “current position” state that can be referred to later.

cycle

In graph terminology, a cycle is a path of length three or more that connects some vertex to itself.

cylinder

A disk drive normally consists of a stack of platters. While this might not be so true today, traditionally all of the I/O heads moved together during a seek operation. Thus, when a given I/O head is positioned over a particular track on a platter, the other I/O heads are also positioned over the corresponding track on their platters. That collection of tracks is called a cylinder. A given cylinder represents all of the data that can be read from all of the platters without doing another seek operation.

cylinder index

In the ISAM system, a simple linear index that stores the lowest key value stored in each cylinder.

cylinder overflow

In the ISAM system, this is space reserved for storing any records that can not fit in their respective cylinder.

DAG

Abbreviation for directed acyclic graph.

data field

In object-oriented programming, a synonym for data member.

data item

A piece of information or a record whose value is drawn from a type.

data member

The variables that together define the space required by a data item are referred to as data members. Some of the commonly used synonyms include data field, attribute, and instance variable.

data structure

The implementation for an ADT.

data type

A type together with a collection of operations to manipulate the type.

deallocated

deallocation

Free the memory allocated to an unused object.

decision problem

A problem whose output is either “YES” or “NO”.

decision tree

A theoretical construct for modeling the behavior of algorithms. Each point at which the algorithm makes a decision (such as an if statement) is modeled by a branch in the tree that represents the algorithms behavior. Decision trees can be used in lower bounds proofs, such as the proof that sorting requires comparisons in the worst case.

deep copy

Copying the actual content of a pointee.

degree

In graph terminology, the degree for a vertex is its number of neighbors. In a directed graph, the in degree is the number of edges directed into the vertex, and the out degree is the number of edges directed out of the vertex. In tree terminology, the degree for a node is its number of children.

delegation mental model for recursion

A way of thinking about the process of recursion. The recursive function “delegates” most of the work when it makes the recursive call. The advantage of the delegation mental model for recursion is that you don’t need to think about how the delegated task is performed. It just gets done.

dense graph

A graph where the actual number of edges is a large fraction of the possible number of edges. Generally, this is interpreted to mean that the degree for any vertex in the graph is relatively high.

depth

The depth of a node in a tree is the length of the path from the root of the tree to .

depth-first search

A graph traversal algorithm. Whenever a is visited during the traversal, DFS will recursively visit all of ‘s unvisited neighbors.

depth-first search tree

A tree that can be defined by the operation of a depth-first search (DFS) on a graph. This tree would consist of the nodes of the graph and a subset of the edges of the graph that was followed during the DFS.

dequeue

A specialized term used to indicate removing an element from a queue.

dereference

Accessing the value of the pointee for some reference variable. Commonly, this happens in a language like Java when using the “dot” operator to access some field of an object.

derivation

In formal languages, the process of executing a series of production rules from a grammar. A typical example of a derivation would be the series of productions executed to go from the start symbol to a given string.

descendant

In a tree, the set of all nodes that have a node as an ancestor are the descendants of . In other words, all of the nodes that can be reached from by progressing downwards in tree. Another way to say it is: The children of , their children, and so on.

deserialization

The process of returning a serialized representation for a data structure back to its original in-memory form.

design pattern

An abstraction for describing the design of programs, that is, the interactions of objects and classes. Experienced software designers learn and reuse patterns for combining software components, and design patterns allow this design knowledge to be passed on to new programmers more quickly.

deterministic

Any finite automata in which, for every pair of state and symbol, there is only a single transition. This means that whenever the machine is in a given state and sees a given symbol, only a single thing can happen. This is in contrast to a non-deterministic finite automata, which has at least one state with multiple transitions on at least one symbol.

deterministic algorithm

An algorithm that does not involve any element of randomness, and so its behavior on a given input will always be the same. This is in contrast to a randomized algorithm.

Deterministic Finite Automata

Deterministic Finite Acceptor

DFA

An automata or abstract machine that can process an input string (shown on a tape) from left to right. There is a control unit (with states), behavior defined for what to do when in a given state and with a given symbol on the current square of the tape. All that we can “do” is change state before going to the next letter to the right.

DFS

Abbreviation for depth-first search.

diagonalization argument

A proof technique for proving that a set is uncountably infinite. The approach is to show that, no matter what order the elements of the set are put in, a new element of the set can be constructed that is not in that ordering. This is done by changing the th value or position of the element to be different from that of the th element in the proposed ordering.

dictionary

An abstract data type or interface for a data structure or software subsystem that supports insertion, search, and deletion of records.

dictionary search

A close relative of an interpolation search. In a classical (paper) dictionary of words in a natural language, there are markings for where in the dictionary the words with a given letter start. So in typical usage of such a dictionary, words are found by opening the dictionary to some appropriate place within the pages that contain words starting with that letter.

digraph

Abbreviation for directed graph.

Dijkstra’s algorithm

An algorithm to solve the single-source shortest paths problem in a graph. This is a greedy algorithm. It is nearly identical to Prim’s algorithm for finding a minimal-cost spanning tree, with the only difference being the calculation done to update the best-known distance.

diminishing increment sort

Another name for Shellsort.

direct access

A storage device, such as a disk drive, that has some ability to move to a desired data location more-or-less directly. This is in contrast to a sequential access storage device such as a tape drive.

direct proof

In general, a direct proof is just a “logical explanation”. A direct proof is sometimes referred to as an argument by deduction. This is simply an argument in terms of logic. Often written in English with words such as “if … then”, it could also be written with logic notation such as .

directed acyclic graph

A graph with no cycles. Abbreviated as DAG. Note that a DAG is not necessarily a tree since a given node might have multiple parents.

directed edge

An edge that goes from vertex to another. In contrast, an undirected edge simply links to vertices without a direction.

directed graph

A graph whose edges each are directed from one of its defining vertices to the other.

dirty bit

Within a buffer pool, a piece of information associated with each buffer that indicates whether the contents of the buffer have changed since being read in from backing storage. When the buffer is flushed from the buffer pool, the buffer’s contents must be written to the backing storage if the dirty bit is set (that is, if the contents have changed). This means that a relatively expensive write operation is required. In contrast, if the dirty bit is not set, then it is unnecessary to write the contents to backing storage, thus saving time over not keeping track of whether the contents have changed or not.

Discrete Fourier Transform

DFT

Let be a vector that stores the coefficients for a polynomial being evaluated. We can then do the calculations to evaluate the polynomial at the th $roots of unity A_{z}F_{z}$ is called the Discrete Fourier Transform (or DFT) for the polynomial.

discriminator

A part of a multi-dimensional search key. Certain tree data structures such as the bintree and the kd tree operate by making branching decisions at nodes of the tree based on a single attribute of the multi-dimensional key, with the attribute determined by the level of the node in the tree. For example, in 2 dimensions, nodes at the odd levels in the tree might branch based on the value of a coordinate, while at the even levels the tree would branch based on the value of the coordinate. Thus, the coordinate is the discriminator for the odd levels, while the coordinate is the discriminator for the even levels.

disjoint

Two parts of a data structure or two collections with no objects in common are disjoint. This term is often used in conjunction with a data structure that has nodes (such as a tree). Also used in the context of sets, where two subsets are disjoint if they share no elements.

disjoint sets

A collection of sets, any pair of which share no elements in common. A collection of disjoint sets partitions some objects such that every object is in exactly one of the disjoint sets.

disk access

The act of reading data from a disk drive (or other form of peripheral storage). The number of times data must be read from (or written to) a disk is often a good measure of cost for an algorithm that involves disk I/O, since this is usually the dominant cost.

disk controller

The control mechanism for a disk drive. Responsible for the action of reading or writing a sector of data.

disk drive

An example of peripheral storage or secondary storage. Data access times are typically measured in thousandths of a second (milliseconds), which is roughly a million times slower than access times for RAM, which is an example of a primary storage device. Reads from and writes to a disk drive are always done in terms of some minimum size, which is typically called a block. The block size is 512 bytes on most disk drives. Disk drives and RAM are typical parts of a computer’s memory hierarchy.

disk I/O

Refers to the act of reading data from or writing data to a disk drive. All disk reads and writes are done in units of a sector or block.

disk-based space/time tradeoff

In contrast to the standard space/time tradeoff, this principle states that the smaller you can make your disk storage requirements, the faster your program will run. This is because the time to read information from disk is enormous compared to computation time, so almost any amount of additional computation needed to unpack the data is going to be less than the disk-reading time saved by reducing the storage requirements.

distance

In graph representations, a synonym for weight.

divide and conquer

A technique for designing algorithms where a solution is found by breaking the problem into smaller (similar) subproblems, solving the subproblems, then combining the subproblem solutions to form the solution to the original problem. This process is often implemented using recursion.

divide-and-conquer recurrences

A common form of recurrence relation that have the form

where , , , and are constants. In general, this recurrence describes a problem of size divided into subproblems of size , while is the amount of work necessary to combine the partial solutions.

divide-and-guess

A technique for finding a closed-form solution to a summation or recurrence relation.

domain

The set of possible inputs to a function.

double buffering

The idea of using multiple buffers to allow the CPU to operate in parallel with a peripheral storage device. Once the first buffer’s worth of data has been read in, the CPU can process this while the next block of data is being read from the peripheral storage. For this idea to work, the next block of data to be processed must be known or predicted with reasonable accuracy.

double hashing

A collision resolution method. A second hash function is used to generate a value on the key. That value is then used by this key as the step size in linear probing by steps. Since different keys use different step sizes (as generated by the second hash function), this process avoids the clustering caused by standard linear probing by steps.

double rotation

A type of rebalancing operation used by the Splay Tree and AVL Tree.

doubly linked list

A linked list implementation variant where each list node contains access pointers to both the previous element and the next element on the list.

DSA

Abbreviation for Data Structures and Algorithms.

dynamic

Something that is changes (in contrast to static). In computer programming, dynamic normally refers to something that happens at run time. For example, run-time analysis is analysis of the program’s behavior, as opposed to its (static) text or structure Dynamic binding or dynamic memory allocation occurs at run time.

dynamic allocation

The act of creating an object from free store. In C++, Java, and JavaScript, this is done using the new operator.

dynamic array

Arrays, once allocated, are of fixed size. A dynamic array puts an interface around the array so as to appear to allow the array to grow and shrink in size as necessary. Typically this is done by allocating a new copy, copying the contents of the old array, and then returning the old array to free store. If done correctly, the amortized cost for dynamically resizing the array can be made constant. In some programming languages such as Java, the term vector is used as a synonym for dynamic array.

dynamic memory allocation

A programming technique where linked objects in a data structure are created from free store as needed. When no longer needed, the object is either returned to free store or left as garbage, depending on the programming language.

dynamic programming

An approach to designing algorithms that works by storing a table of results for subproblems. A typical cause for excessive cost in recursive algorithms is that different branches of the recursion might solve the same subproblem. Dynamic programming uses a table to store information about which subproblems have already been solved, and uses the stored information to immediately give the answer for any repeated attempts to solve that subproblem.

edge

The connection that links two nodes in a tree, linked list, or graph.

edit distance

Given strings and , the edit distance is a measure for the number of editing steps required to convert into .

efficient

A solution is said to be efficient if it solves the problem within the required resource constraints. A solution is sometimes said to be efficient if it requires fewer resources than known alternatives, regardless of whether it meets any particular requirements.

element

One value or member in a set.

empirical comparison

An approach to comparing to things by actually seeing how they perform. Most typically, we are referring to the comparison of two programs by running each on a suite of test data and measuring the actual running times. Empirical comparison is subject to many possible complications, including unfair selection of test data, and inaccuracies in the time measurements due to variations in the computing environment between various executions of the programs.

empty

For a container class, the state of containing no elements.

encapsulation

In programming, the concept of hiding implementation details from the user of an ADT, and protecting data members of an object from outside access.

enqueue

A specialized term used to indicate inserting an element onto a queue.

entry-sequenced file

A file that stores records in the order that they were added to the file.

enumeration

The process by which a traversal lists every object in the container exactly once. Thus, a traversal that prints the nodes is said to enumerate the nodes. An enumeration can also refer to the actual listing that is produced by the traversal (as well as the process that created that listing).

equidistribution property

In random number theory, this means that a given series of random numbers cannot be described more briefly than simply listing it out.

equivalence class

An equivalence relation can be used to partition a set into equivalence classes.

equivalence relation

Relation is an equivalence relation on set if it is reflexive, symmetric, and transitive.

estimation

As a technical skill, this is the process of generating a rough estimate in order to evaluate the feasibility of a proposed solution. This is sometimes known as “back of the napkin” or “back of the envelope” calculation. The estimation process can be formalized as (1) determine the major parameters that affect the problem, (2) derive an equation that relates the parameters to the problem, then (3) select values for the parameters and apply the equation to yield an estimated solution.

evaluation

The act of finding the value for a polynomial at a given point.

exact-match query

Records are accessed by unique identifier.

exceptions

Exceptions are techniques used to predict possible runtime errors and handle them properly.

exchange

A swap of adjacent records in an array.

exchange sort

A sort that relies solely on exchanges (swaps of adjacent records) to reorder the list. Insertion Sort and Bubble Sort are examples of exchange sorts. All exchange sorts require time in the worst case.

expanding the recurrence

A technique for solving a recurrence relation. The idea is to replace the recursive part of the recurrence with a copy of recurrence.

exponential growth rate

A growth rate function where (the input size) appears in the exponent. For example, .

expression tree

A tree structure meant to represent a mathematical expression. Internal nodes of the expression tree are operators in the expression, with the subtrees being the sub-expressions that are its operand. All leaf nodes are operands.

extent

A physically contiguous block of sectors on a disk drive that are all part of a given disk file. The fewer extents needed to store the data for a disk file, generally the fewer seek operations that will be required to process a series of disk access operations on that file.

external fragmentation

A condition that arises when a series of memory requests result in lots of small free blocks, no one of which is useful for servicing typical requests.

external sort

A sorting algorithm that is applied to data stored in peripheral storage such as on a disk drive. This is in contrast to an internal sort that works on data stored in main memory.

factorial

The factorial function is defined as for .

failure policy

In a memory manager, a failure policy is the response that takes place when there is no way to satisfy a memory request from the current free blocks in the memory pool. Possibilities include rejecting the request, expanding the memory pool, collecting garbage, and reorganizing the memory pool (to collect together free space).

family of languages

Given some class or type of finite automata (for example, the deterministic finite automata), the set of languages accepted by that class of finite automata is called a family. For example, the regular languages is a family defined by the DFAs.

FIFO

Abbreviation for “first-in, first-out”. This is the access paradigm for a queue, and an old terminology for the queue is “FIFO list”.

file allocation table

A legacy file system architecture orginially developed for DOS and then used in Windows. It is still in use in many small-scale peripheral devices such as USB memory sticks and digital camera memory.

file manager

A part of the operating system responsible for taking requests for data from a logical file and mapping those requests to the physical location of the data on disk.

file processing

The domain with Computer Science that deals with processing data stored on a disk drive (in a file), or more broadly, dealing with data stored on any peripheral storage device. Two fundamental properties make dealing with data on a peripheral device different from dealing with data in main memory: (1) Reading/writing data on a peripheral storage device is far slower than reading/writing data to main memory (for example, a typical disk drive is about a million times slower than RAM). (2) All I/O to a peripheral device is typically in terms of a block of data (for example, nearly all disk drives do all I/O in terms of blocks of 512 bytes).

file structure

The organization of data on peripheral storage, such as a disk drive or DVD drive.

final state

A required element of any acceptor. When computation on a string ends in a final state, then the machine accepts the string. Otherwise the machine rejects the string.

FIND

One half of the UNION/FIND algorithm for managing disjoint sets. It is the process of moving upwards in a tree to find the tree’s root.

Finite State Acceptor

A simple type of finite state automata, an acceptor’s only ability is to accept or reject a string. So, a finite state acceptor does not have the ability to modify the input tape. If computation on the string ends in a final state, then the the string is accepted, otherwise it is rejected.

Finite State Machine

FSM

Finite State Automata

FSA

Finite Automata

Any abstract state machine, generally represented as a graph where the nodes are the states, and the edges represent transitions between nodes that take place when the machine is in that node (state) and sees an appropriate input. See, as an example, Deterministic Finite Automata.

first fit

In a memory manager, first fit is a heuristic for deciding which free block to use when allocating memory from a memory pool. First fit will always allocate the first free block on the free block list that is large enough to service the memory request. The advantage of this approach is that it is typically not necessary to look at all free blocks on the free block list to find a suitable free block. The disadvantage is that it is not “intelligently” selecting what might be a better choice of free block.

fixed-length coding

Given a collection of objects, a fixed-length coding scheme assigns a code to each object in the collection using codes that are all of the same length. Standard ASCII and Unicode representations for characters are both examples of fixed-length coding schemes. This is in contrast to variable-length coding.

floor

Written , for real value the floor is the greatest integer .

Floyd’s algorithm

An algorithm to solve the all-pairs shortest paths problem. It uses the dynamic programming algorithmic technique, and runs in time. As with any dynamic programming algorithm, the key issue is to avoid duplicating work by using proper bookkeeping on the algorithm’s progress through the solution space. The basic idea is to first find all the direct edge costs, then improving those costs by allowing paths through vertex 0, then the cheapest paths involving paths going through vertices 0 and 1, and so on.

flush

The act of removing data from a cache, most typically because other data considered of higher future value must replace it in the cache. If the data being flushed has been modified since it was first read in from secondary storage (and the changes are meant to be saved), then it must be written back to that secondary storage.

flush

The the context of a buffer pool, the process of removing the contents stored in a buffer when that buffer is required in order to store new data. If the buffer’s contents have been changed since having been read in from backing storage (this fact would normally be tracked by using a dirty bit), then they must be copied back to the backing storage before the buffer can be reused.

flyweight

A design pattern that is meant to solve the following problem: You have an application with many objects. Some of these objects are identical in the information that they contain, and the role that they play. But they must be reached from various places, and conceptually they really are distinct objects. Because there is so much duplication of the same information, we want to reduce memory cost by sharing that space. For example, in document layout, the letter “C” might be represented by an object that describes that character’s strokes and bounding box. However, we do not want to create a separate “C” object everywhere in the document that a “C” appears. The solution is to allocate a single copy of the shared representation for “C” objects. Then, every place in the document that needs a “C” in a given font, size, and typeface will reference this single copy. The various instances of references to a specific form of “C” are called flyweights. Flyweights can also be used to implement the empty leaf nodes of the bintree and PR quadtree.

folding method

In hashing, an approach to implementing a hash function. Most typically used when the key is a string, the folding method breaks the string into pieces (perhaps each letter is a piece, or a small series of letters is a piece), converts the letter(s) to an integer value (typically by using its underlying encoding value), and summing up the pieces.

Ford and Johnson sort

A sorting algorithm that is close to the theoretical minimum number of key comparisons necessary to sort. Generally not considered practical in practice due to the fact that it is not efficient in terms of the number of records that need to be moved. It consists of first sorting pairs of nodes into winners and losers (of the pairs comparisons), then (recursively) sorting the winners of the pairs, and then finally carefully selecting the order in which the losers are added to the chain of sorted items.

forest

A collection of one or more trees.

free block

A block of unused space in a memory pool.

free block list

In a memory manager, the list that stores the necessary information about the current free blocks. Generally, this is done with some sort of linked list, where each node of the linked list indicates the start position and length of the free block in the memory pool.

free store

Space available to a program during runtime to be used for dynamic allocation of objects. The free store is distinct from the runtime stack. The free store is sometimes referred to as the heap, which can be confusing because heap more often refers to a specific data structure. Most programming languages provide functions to allocate (and maybe to deallocate) objects from the free store, such as new in C++ and Java.

free tree

A connected, undirected graph with no simple cycles. An equivalent definition is that a free tree is connected and has edges.

freelist

A simple and faster alternative to using free store when the objects being dynamically allocated are all of the same size (and thus are interchangeable). Typically implemented as a linked stack, released objects are put on the front of the freelist. When a request is made to allocate an object, the freelist is checked first and it provides the object if possible. If the freelist is empty, then a new object is allocated from free store.

frequency count

A heuristic used to maintain a self-organizing list. Under this heuristic, a count is maintained for every record. When a record access is made, its count is increased. If this makes its count greater than that of another record in the list, it moves up toward the front of the list accordingly so as to keep the list sorted by frequency. Analogous to the least frequently used heuristic for maintaining a buffer pool.

full binary tree theorem

This theorem states that the number of leaves in a non-empty full binary tree is one more than the number of internal nodes. Equivalently, then number of null pointers in a standard pointer-based implementation for binary tree nodes is one more than the number of nodes in the binary tree.

full tree

A binary tree is full if every node is either a leaf node or else it is an internal node with two non-empty children.

function

In mathematics, a matching between inputs (the domain) and outputs (the range). In programming, a subroutine that takes input parameters and uses them to compute and return a value. In this case, it is usually considered bad practice for a function to change any global variables (doing so is called a side effect).

garbage

In memory management, any memory that was previously (dynamically) allocated by the program during runtime, but which is no longer accessible since all pointers to the memory have been deleted or overwritten. In some languages, garbage can be recovered by garbage collection. In languages such as C and C++ that do not support garbage collection, so creating garbage is considered a memory leak.

garbage collection

Languages with garbage collection such Java, JavaScript, Lisp, and Scheme will periodically reclaim garbage and return it to free store.

general tree

A tree in which any given node can have any number of children. This is in contrast to, for example, a binary tree where each node has a fixed number of children (some of which might be null). General tree nodes tend to be harder to implement for this reason.

grammar

A formal definition for what strings make up a language, in terms of a set of production rules.

graph

A graph consists of a set of vertices and a set of edges , such that each edge in is a connection between a pair of vertices in .

greedy algorithm

An algorithm that makes locally optimal choices at each step.

growth rate

In algorithm analysis, the rate at which the cost of the algorithm grows as the size of its input grows.

guess-and-test

A technique used when trying to determine the closed-form solution for a summation or recurrence relation. Given a hypothesis for the closed-form solution, if it is correct, then it is often relatively easy to prove that using induction.

guided traversal

A tree traversal that does not need to visit every node in the tree. An example would be a range query in a BST.

halt state

In a finite automata, a designated state which causes the machine to immediately halt when it is entered.

halted configuration

A halted configuration occurs in a Turing machine when the machine transitions into the halt state.

halting problem

The halting problem is to answer this question: Given a computer program and an input , will program halt when executed on input ? This problem has been proved impossible to solve in the general case. Thus, it is an example of an unsolveable problem.

handle

When using a memory manager to store data, the client will pass data to be stored (the message) to the memory manager, and the memory manager will return to the client a handle. The handle encodes the necessary information that the memory manager can later use to recover and return the message to the client. This is typically the location and length of the message within the memory pool.

hanging configuration

A hanging configuration occurs in a Turing machine when the I/O head moves to the left from the left-most square of the tape, or when the machine goes into an infinite loop.

hard algorithm

“Hard” is traditionally defined in relation to running time, and a “hard” algorithm is defined to be an algorithm with exponential running time.

hard problem

“Hard” is traditionally defined in relation to running time, and a “hard” problem is defined to be one whose best known algorithm requires exponential running time.

harmonic series

The sum of reciprocals from 1 to is called the Harmonic Series, and is written . This sum has a value between and .

hash function

In a hash system, the function that converts a key value to a position in the hash table. The hope is that this position in the hash table contains the record that matches the key value.

hash system

The implementation for search based on hash lookup in a hash table. The search key is processed by a hash function, which returns a position in a hash table, which hopefully is the correct position in which to find the record corresponding to the search key.

hash table

The data structure (usually an array) that stores data records for lookup using hashing.

hashing

A search method that uses a hash function to convert a search key value into a position within a hash table. In a properly implemented hash system, that position in the table will have high probability of containing the record that matches the key value. Sometimes, the hash function will return an position that does not store the desired key, due to a process called collision. In that case, the desired record is found through a process known as collision resolution.

head

The beginning of a list.

header node

Commonly used in implementations for a linked list or related structure, this node preceeds the first element of the list. Its purpose is to simplify the code implementation by reducing the number of special cases that must be programmed for.

heap

This term has two different meanings. Uncommonly, it is a synonym for free store. Most often it is used to refer to a particular data structure. This data structure is a complete binary tree with the requirement that every node has a value greater than its children (called a max heap), or else the requirement that every node has a value less than its children (called a min heap). Since it is a complete binary tree, a heap is nearly always implemented using an array rather than an explicit tree structure. To add a new value to a heap, or to remove the extreme value (the max value in a max-heap or min value in a min-heap) and update the heap, takes time in the worst case. However, if given all of the values in an unordered array, the values can be re-arranged to form a heap in only time. Due to its space and time efficiency, the heap is a popular choice for implementing a priority queue.

heapsort

A sorting algorithm that costs time in the best, average, and worst cases. It tends to be slower than Mergesort and Quicksort. It works by building a max heap, and then repeatedly removing the item with maximum key value (moving it to the end of the heap) until all elements have been removed (and replaced at their proper location in the array).

height

The height of a tree is one more than the depth of the deepest node in the tree.

height balanced

The condition the depths of each subtree in a tree are roughly the same.

heuristic

A way to solve a problem that is not guarenteed to be optimal. While it might not be guarenteed to be optimal, it is generally expected (by the agent employing the heuristic) to provide a reasonably efficient solution.

heuristic algorithm

A type of approximation algorithm, that uses a heuristic to find a good, but not necessarily cheapest, solution to an optimization problem.

home position

In hashing, a synonym for home slot.

home slot

In hashing, this is the slot in the hash table determined for a given key by the hash function.

homogeneity

In a container class, this is the property that all objects stored in the ncontainer are of the same class. For example, if you have a list intended to store Payroll records, is it possible for the programmer to insert an integer onto the list instead?

Huffman codes

The codes given to a collection of letters (or other symbols) through the process of Huffman coding. Huffman coding uses a Huffman coding tree to generate the codes. The codes can be of variable length, such that the letters which are expected to appear most frequently are shorter. Huffman coding is optimal whenever the true frequencies are known, and the frequency of a letter is independent of the context of that letter in the message.

Huffman coding tree

A Huffman coding tree is a full binary tree that is used to represent letters (or other symbols) efficiently. Each letter is associated with a node in the tree, and is then given a Huffman code based on the position of the associated node. A Huffman coding tree is an example of a binary trie.

Huffman tree

Shorter form of the term Huffman coding tree.

I/O head

On a disk drive (or similar device), the part of the machinery that actually reads data from the disk.

image-space decomposition

A from of key-space decomposition where the key space splitting points is predetermined (typically by splitting in half). For example, a Huffman coding tree splits the letters being coded into those with codes that start with 0 on the left side, and those with codes that start with 1 on the right side. This regular decomposition of the key space is the basis for a trie data structure. An image-space decomposition is in opposition to an object-space decomposition.

in degree

In graph terminology, the in degree for a vertex is the number of edges directed into the vertex.

incident

In graph terminology, an edge connecting two vertices is said to be incident with those vertices. The two vertices are said to be adjacent.

index file

A file whose records consist of key-value pairs where the pointers are referencing the complete records stored in another file.

indexing

The process of associating a search key with the location of a corresponding data record. The two defining points to the concept of an index is the association of a key with a record, and the fact that the index does not actually store the record itself but rather it stores a reference to the record. In this way, a collection of records can be supported by multiple indices, typically a separate index for each key field in the record.

induction hypothesis

The key assumption used in a proof by induction, that the theorem to be proved holds for smaller instances of the theorem. The induction hypothesis is equivalent to the recursive call in a recursive function.

induction step

Part of a proof by induction. In its simplest form, this is a proof of the implication that if the theorem holds for , then it holds for . As an alternative, see strong induction.

induction variable

The variable used to parameterize the theorem being proved by induction. For example, if we seek to prove that the sum of the integers from 1 to is , then is the induction variable. An induction variable must be an integer.

information theoretic lower bound

A lower bound on the amount of resources needed to solve a problem based on the number of bits of information needed to uniquely specify the answer. Sometimes referred to as a “Shannon theoretic lower bound” due to Shannon’s work on information theory and entropy. An example is that sorting has a lower bound of because there are possible orderings for values. This observation alone does not make the lower bound tight, because it is possible that no algorithm could actually reach the information theory lower limit.

inherit

In object-oriented programming, the process by which a subclass gains data members and methods from a base class.

initial state

A synonym for start state.

inode

Short for “index node”. In UNIX-style file systems, specific disk sectors that hold indexing information to define the layout of the file system.

inorder traversal

In a binary tree, a traversal that first recursively visits the left child, then visits the root, an then recursively visits the right child. In a binary search tree, this traversal will enumerate the nodes in sorted order.

Insertion Sort

A sorting algorithm with average and worst case cost, and best case cost. This best case cost makes it useful when we have reason to expect the input to be nearly sorted.

instance variable

In object-oriented programming, a synonym for data member.

integer function

Any function whose input is an integer and whose output is an integer. It can be proved by diagonalization that the set of integer functions is uncountably infinite.

inter-sector gap

On a disk drive, a physical gap in the data that occurs between the sectors. This allows the I/O head detect the end of the sector.

interface

An interface is a class-like structure that only contains method signatures and fields. An interface does not contain an implementation of the methods or any data members.

intermediate code

A step in a typical compiler is to transform the original high-level language into a form on which it is easier to do other stages of the process. For example, some compilers will transform the original high-level source code into assembly code on which it can do code optimization, before translating it into its final executable form.

intermediate code generation

A phase in a compiler, that walks through a parse tree to produce simple assembly code.

internal fragmentation

A condition that occurs when more than bytes are allocated to service a memory request for bytes, wasting free storage. This is often done to simplify memory management.

internal node

In a tree, any node that has at least one non-empty child is an internal node.

internal sort

A sorting algorithm that is applied to data stored in main memory. This is in contrast to an external sort that is meant to work on data stored in peripheral storage such as on a disk drive.

interpolation

The act of finding the coefficients of a polynomial, given the values at some points. A polynomal of degree requires points to interpolate the coefficients.

interpolation search

Given a sorted array, and knowing the first and last key values stored in some subarray known to contain search key , interpolation search will compute the expected location of in the subarray as a fraction of the distance between the known key values. So it will next check that computed location, thus narrowing the search for the next iteration. Given reasonable key value distribution, the average case for interpolation search will be , or better than the expected cost of binary search. Nonetheless, binary search is expected to be faster in nearly all practical situations due to the small difference between the two costs, combined with the higher constant factors required to implement interpolation search as compared to binary search.

interpreter

In contrast to a compiler that translates a high-level program into something that can be repeatedly executed to perform a computation, an interpreter directly performs computation on the high-level langauge. This tends to make the computation much slower than if it were performed on the directly executable version produced by a compiler.

inversion

A measure of how disordered a series of values is. For each element in the series, count one inversion for each element to left of that is greater than the value of (and so must ultimately be moved to the right of during a sorting process).

inverted file

Synonym for inverted list when the inverted list is stored in a disk file.

inverted list

An index which links secondary keys to either the associated primary key or the actual record in the database.

irreflexive

In set notation, binary relation on set is irreflexive if is never in the relation for any .

ISAM

Indexed Sequential Access Method: an obsolete method for indexing data for (at the time) fast retrieval. More generally, the term is used also to generically refer to an index that supports both sequential and keyed access to data records. Today, that would nearly always be implemented using a B-Tree.

iterator

In a container such as a List, a separate class that indicates position within the container, with support for traversing through all elements in the container.

job

Common name for processes or tasks to be run by an operating system. They typically need to be processed in order of importance, and so are kept organized by a priority queue. Another common use for this term is for a collection of tasks to be ordered by a topological sort.

jump search

An algorithm for searching a sorted list, that falls between sequential search and binary search in both computational cost and conceptual complexity. The idea is to keep jumping by some fixed number of positions until a value is found that is bigger than search key , then do a sequential search over the subarray that is now known to contain the search key. The optimal number of steps to jump will be for an array of size , and the worst case cost will be .

K-ary tree

A type of full tree where every internal node has exactly children.

k-path

In Floyd’s algorithm, a k-path is a path between two vertices and that can only go through vertices with an index value less than or equal to .

kd tree

A spatial data structure that uses a binary tree to store a collection of data records based on their (point) location in space. It uses the concept of a discriminator at each level to decide which single component of the multi-dimensional search key to branch on at that level. It uses a key-space decomposition, meaning that all data records in the left subtree of a node have a value on the corresponding discriminator that is less than that of the node, while all data records in the right subtree have a greater value. The bintree is the image-space decomposition analog of the kd tree.

key

A field or part of a larger record used to represent that record for the purpose of searching or comparing. Another term for search key.

key sort

Any sorting operation applied to a collection of key-value pairs where the value in this case is a reference to a complete record (that is, a pointer to the record in memory or a position for a record on disk). This is in contrast to a sorting operation that works directly on a collection of records. The intention is that the collection of key-value pairs is far smaller than the collection of records themselves. As such, this might allow for an internal sort when sorting the records directly would require an external sort. The collection of key-value pairs can also act as an index.

key space

The range of values that a key value may take on.

key-space decomposition

The idea that the range for a search key will be split into pieces. There are two general approaches to this: object-space decomposition and image-space decomposition.

key-value pair

A standard solution for solving the problem of how to relate a key value to a record (or how to find the key for a given record) within the context of a particular index. The idea is to simply store as records in the index pairs of keys and records. Specifically, the index will typically store a copy of the key along with a reference to the record. The other standard solution to this problem is to pass a comparator function to the index.

knapsack problem

While there are many variations of this problem, here is a typical version: Given knapsack of a fixed size, and a collection of objects of various sizes, is there a subset of the objects that exactly fits into the knapsack? This problem is known to be NP-complete, but can be solved for problem instances in practical time relatively quickly using dynamic programming. Thus, it is considered to have pseudo-polynomial cost. An optimization problem version is to find the subset that can fit with the greatest amount of items, either in terms of their total size, or in terms of the sum of values associated with each item.

Kruskal’s algorithm

An algorithm for computing the MCST of a graph. During processing, it makes use of the UNION/FIND process to efficiently determine of two vertices are within the same subgraph.

labeled graph

A graph with labels associated with the nodes.

language

A set of strings.

Las Vegas algorithms

A form of randomized algorithm. We always find the maximum value, and “usually” we find it fast. Such algorithms have a guaranteed result, but do not guarantee fast running time.

leaf node

In a binary tree, leaf node is any node that has two empty children. (Note that a binary tree is defined so that every node has two children, and that is why the leaf node has to have two empty children, rather than no children.) In a general tree, any node is a leaf node if it has no children.

least frequently used

Abbreviated LFU, it is a heuristic that can be used to decide which buffer in a buffer pool to flush when data in the buffer pool must be replaced by new data being read into a cache. However, least recently used is more popular than LFU. Analogous to the frequency count heuristic for maintaining a self-organizing list.

least recently used

Abbreviated LRU, it is a popular heuristic to use for deciding which buffer in a buffer pool to flush when data in the buffer pool must be replaced by new data being read into a cache. Analogous to the move-to-front heuristic for maintaining a self-organizing list.

left recursive

In automata theory, a production is left recursive if it is of the form , where is the set of non-terminals and is the set of terminals in the grammar.

length

In a list, the number of elements. In a string, the number of characters.

level

In a tree, all nodes of depth are at level in the tree. The root is the only node at level 0, and its depth is 0.

lexical analysis

A phase of a compiler or interpreter responsible for reading in characters of the program or language and grouping them into tokens.

lexical scoping

Within programming languages, the convention of allowing access to a variable only within the block of code in which the variable is defined. A synonym for static scoping.

LFU

Abbreviation for least frequently used.

lifetime

For a variable, lifetime is the amount of time it will exist before it is destroyed.

LIFO

Abbreviation for “Last-In, First-Out”. This is the access paradigm for a stack, and an old terminolgy for the stack is “LIFO list”.

linear congruential method

In random number theory, a process for computing the next number in a pseudo-random sequence. Starting from a seed, the next term in the series is calculated from term by the equation

where and are constants. These constants must be well chosen for the resulting series of numbers to have desireable properties as a random number sequence.

linear growth rate

For input size , a growth rate of (for any positive constant). In other words, the cost of the associated function is linear on the input size.

linear index

A form of indexing that stores key-value pairs in a sorted array. Typically this is used for an index to a large collection of records stored on disk, where the linear index itself might be on disk or in main memory. It allows for efficient search (including for range queries), but it is not good for inserting and deleting entries in the array. Therefore, it is an ideal indexing structure when the system needs to do range queries but the collection of records never changes once the linear index has been created.

linear order

Another term for total order.

linear probing

In hashing, this is the simplest collision resolution method. Term of the probe sequence is simply , meaning that collision resolution works by moving sequentially through the hash table from the home slot. While simple, it is also inefficient, since it quickly leads to certain free slots in the hash table having higher probability of being selected during insertion or search.

linear probing by steps

In hashing, this collision resolution method is a variation on simple linear probing. Some constant is defined such that term of the probe sequence is . This means that collision resolution works by moving sequentially through the hash table from the home slot in steps of size . While not much improvement on linear probing, it forms the basis of another collision resolution method called double hashing, where each key uses a value for defined by a second hash function.

linear search

Another name for sequential search.

link node

A widely used supporting object that forms the basic building block for a linked list and similar data structures. A link node contains one or more fields that store data, and a pointer or reference to another link node.

linked list

An implementation for the list ADT that uses dynamic allocation of link nodes to store the list elements. Common variants are the singly linked list, doubly linked list and circular list. The overhead required is the pointers in each link node.

linked stack

Analogous to a linked list, this uses dynamic allocation of nodes to store the elements when implementing the stack ADT.

list

A finite, ordered sequence of data items known as elements. This is close to the mathematical concept of a sequence. Note that “ordered” in this definition means that the list elements have position. It does not refer to the relationship between key values for the list elements (that is, “ordered” does not mean “sorted”).

literal

In a Boolean expression, a literal is a Boolean variable or its negation. In the context of compilers, it is any constant value. Similar to a terminal.

load factor

In hashing this is the fraction of the hash table slots that contain a record. Hash systems usually try to keep the load factor below 50%.

local storage

local storage.

local variable

local variables

A variable declared within a function or method. It exists only from the time when the function is called to when the function exits. When a function is suspended (due to calling another function), the function’s local variables are stored in an activation record on the runtime stack.

locality of reference

The concept that accesses within a collection of records is not evenly distributed. This can express itself as some small fraction of the records receiving the bulk of the accesses (80/20 rule). Alternatively, it can express itself as an increased probability that the next or future accesses will come close to the most recent access. This is the fundamental property for success of caching.

logarithm

The logarithm of base for value is the power to which is raised to get .

logical file

In file processing, the programmer’s view of a random access file stored on disk as a contiguous series of bytes, with those bytes possibly combining to form data records. This is in contrast to the physical file.

logical form

The definition for a data type in terms of an ADT. Contrast to the physical form for the data type.

lookup table

A table of pre-calculated values, used to speed up processing time when the values are going to be viewed many times. The costs to this approach are the space required for the table and the time required to compute the table. This is an example of a space/time tradeoff.

lower bound

In algorithm analysis, a growth rate that is always less than or equal to the that of the algorithm in question. In practice, this is the fastest-growing function that we know grows no faster than all but a constant number of inputs. It could be a gross under-estimate of the truth. Since the lower bound for the algorithm can be very different for different situations (such as the best case or worst case), we typically have to specify which situation we are referring to.

lower bounds proof

A proof regarding the lower bound, with this term most typically referring to the lower bound for any possible algorithm to solve a given problem. Many problems have a simple lower bound based on the concept that the minimum amount of processing is related to looking at all of the problem’s input. However, some problems have a higher lower bound than that. For example, the lower bound for the problem of sorting () is greater than the input size to sorting (). Proving such “non-trivial” lower bounds for problems is notoriously difficult.

LRU

Abbreviation for least recently used.

main memory

A synonym for primary storage. In a computer, typically this will be RAM.

map

A data structure that relates a key to a record.

mapping

A function that maps every element of a given set to a unique element of another set; a correspondence.

mark array

It is typical in graph algorithms that there is a need to track which nodes have been visited at some point in the algorithm. An array of bits or values called the mark array is often maintained for this purpose.

mark/sweep algorithm

An algorithm for garbage collection. All accessible variables, and any space that is reachable by a chain of pointers from any accessible variable, is “marked”. Then a sequential sweep of all memory in the pool is made. Any unmarked memory locations are assumed to not be needed by the program and can be considered as free to be reused.

master theorem

A theorem that makes it easy to solve divide-and-conquer recurrences.

matching

In graph theory, a pairing (or match) of various nodes in a graph.

matching problem

Any problem that involves finding a matching in a graph with some desired property. For example, a well-known NP-complete problem is to find a maximum match for an undirected graph.

max heap

A heap where every node has a key value greater than its children. As a consequence, the node with maximum key value is at the root.

maximal match

In a graph, any matching that leaves no pair of unmatched vertices that are connected. A maximal matching is not necessarily a maximum match. In other words, there might be a larger matching than the maximal matching that was found.

maximum lower bound

The lower bound for the problem of finding the maximum value in an unsorted list is .

maximum match

In a graph, the largest possible matching.

MCST

MST

Abbreviation for minimal-cost spanning tree.

measure of cost

When comparing two things, such as two algorithms, some event or unit must be used as the basic unit of comparison. It might be number of milliseconds needed or machine instructions expended by a program, but it is usually desirable to have a way to do comparison between two algorithms without writing a program. Thus, some other measure of cost might be used as a basis for comparison between the algorithms. For example, when comparing two sorting algorthms it is traditional to use as a measure of cost the number of comparisons made between the key values of record pairs.

member

In set notation, this is a synonym for element. In abstract design, a data item is a member of a type. In an object-oriented language, data members are data fields in an object.

member function

Each operation associated with the ADT is implemented by a member function or method.

memory allocation

In a memory manager, the act of honoring a request for memory.

memory deallocation

In a memory manager, the act of freeing a block of memory, which should create or add to a free block.

memory hierarchy

The concept that a computer system stores data in a range of storage types that range from fast but expensive (primary storage) to slow but cheap (secondary storage). When there is too much data to store in primary storage, the goal is to have the data that is needed soon or most often in the primary storage as much as possible, by using caching techniques.

memory leak

In programming, the act of creating garbage. In languages such as C and C++ that do not support garbage collection, repeated memory leaks will evenually cause the program to terminate.

memory manager

Functionality for managing a memory pool. Typically, the memory pool is viewed as an array of bytes by the memory manager. The client of the memory manager will request a collection of (adjacent) bytes of some size, and release the bytes for reuse when the space is no longer needed. The memory manager should not know anything about the interpretation of the data that is being stored by the client into the memory pool. Depending on the precise implementation, the client might pass in the data to be stored, in which case the memory manager will deal with the actual copy of the data into the memory pool. The memory manager will return to the client a handle that can later be used by the client to retrieve the data.

memory pool

Memory (usually in RAM but possibly on disk or peripheral storage device) that is logically viewed as an array of memory positions. A memory pool is usually managed by a memory manager.

memory request

In a memory manager, a request from some client to the memory manager to reserve a block of memory and store some bytes there.

merge insert sort

A synonym for the Ford and Johnson sort.

Mergesort

A sorting algorithm that requires in the best, average, and worst cases. Conceptually it is simple: Split the list in half, sort the halves, then merge them together. It is a bit complicated to implement efficiently on an array.

message

In a memory manager implementation (particularly a memory manager implemented with a message passing style of interface), the message is the data that the client of the memory manager wishes to have stored in the memory pool. The memory manager will reply to the client by returning a handle that defines the location and size of the message as stored in the memory pool. The client can later recover the message by passing the handle back to the memory manager.

message passing

A common approach to implementing the ADT for a memory manager or buffer pool, where the contents of a message to be stored is explicitly passed between the client and the memory manager. This is in contrast to a buffer passing approach.

metaphor

Humans deal with complexity by assigning a label to an assembly of objects or concepts and then manipulating the label in place of the assembly. Cognitive psychologists call such a label a metaphor.

method

In the object-oriented programming paradigm, a method is an operation on a class. A synonym for member function.

mid-square method

In hashing, an approach to implementing a hash function. The key value is squared, and some number of bits from the middle of the resulting value are extracted as the hash code. Some care must be taken to extract bits that tend to actually be in the middle of the resulting value, which requires some understanding of the typical key values. When done correctly, this has the advantage of having the hash code be affected by all bits of the key

min heap

A heap where every node has a key value less than its children. As a consequence, the node with minimum key value is at the root.

minimal-cost spanning tree

Abbreviated as MCST, or sometimes as MST. Derived from a weighted graph, the MCST is the subset of the graph’s edges that maintains the connectivitiy of the graph while having lowest total cost (as defined by the sum of the weights of the edges in the MCST). The result is referred to as a tree because it would never have a cycle (since an edge could be removed from the cycle and still preserve connectivity). Two algorithms to solve this problem are Prim’s algorithm and Kruskal’s algorithm.

minimum external path weight

Given a collection of objects, each associated with a leaf node in a tree, the binary tree with minimum external path weight is the one with the minimum sum of weighted path lengths for the given set of leaves. This concept is used to create a Huffman coding tree, where a letter with high weight should have low depth, so that it will count the least against the total path length. As a result, another letter might be pushed deeper in the tree if it has less weight.

mod

Abbreviation for the modulus function.

model

A simplification of reality that preserves only the essential elements. With a model, we can more easily focus on and reason about these essentials. In algorithm analysis, we are especially concerned with the cost model for measuring the cost of an algorithm.

modulus

The modulus function returns the remainder of an integer division. Sometimes written in mathematical expressions, the syntax in many programming languages is n % m.

Monte Carlo algorithms

A form of randomized algorithm. We find the maximum value fast, or we don’t get an answer at all (but fast). While such algorithms have good running time, their result is not guaranteed.

move-to-front

A heuristic used to maintain a self-organizing list. Under this heuristic, whenever a record is accessed it is moved to the front of the list. Analogous to the least recently used heuristic for maintaining a buffer pool.

multi-dimensional search key

A search key containing multiple parts, that works in conjunction with a multi-dimensional search structure. Most typically, a spatial search key representing a position in multi-dimensional (2 or 3 dimensions) space. But a multi-dimensional key could be used to organize data within non-spatial dimensions, such as temperature and time.

multi-dimensional search structure

A data structure used to support efficient search on a multi-dimensional search key. The main concept here is that a multi-dimensional search structure works more efficiently by considering the multiple parts of the search key as a whole, rather than making independent searches on each one-dimensional component of the key. A primary example is a spatial data structure that can efficiently represent and search for records in multi-dimensional space.

multilist

A list that may contain sublists. This term is sometimes used as a synonym to the term bag.

natural numbers

Zero and the positive integers.

necessary fallacy

A common mistake in a lower bounds proof for a problem, where the proof makes an inappropriate assumption that any algorithm must operate in some manner (typically in the way that some known algorithm behaves).

neighbor

In a graph, a node is said to be a neighbor of node if there is an edge from to .

node

The objects that make up a linked structure such as a linked list or binary tree. Typically, nodes are allocated using dynamic memory allocation. In graph terminology, the nodes are more commonly called vertices.

non-deterministic

In a finite automata, at least one state has multiple transitions on at least one symbol. This means that it is not deterministic about what transition to take in that situation. A non-deterministic machine is said to accept a string if it completes execution on the string in an accepting state under at least one choice of non-deterministic transitions. Generally, non-determinism can be simulated with a deterministic machine by alternating between the execution that would take place under each of the branching choices.

non-deterministic algorithm

An algorithm that may operate using a non-deterministic choice operation.

non-deterministic choice

An operation that captures the concept of nondeterminism. A nondeterministic choice can be viewed as either “correctly guessing” between a set of choices, or implementing each of the choices in parallel. In the parallel view, the nondeterminism was successful if at least one of the choices leads to a correct answer.

non-deterministic polynomial time algorithm

An algorithm that runs in polynomial time, and which may (or might not) use non-deterministic choice.

non-strict partial order

In set notation, a relation that is reflexive, antisymmetric, and transitive.

non-terminal

In contrast to a terminal, a non-terminal is an abstract state in a production rule. Begining with the start symbol, all non-terminals must be converted into terminals in order to complete a derivation.

NP

An abbreviation for non-deterministic polynomial.

NP-Complete

A class of problems that are related to each other in this way: If ever one such problem is proved to be solvable in polynomial time, or proved to require exponential time, then all other NP-Complete problems will cost likewise. Since so many real-world problems have been proved to be NP-Complete, it would be extremely useful to determine if they have polynomial or exponential cost. But so far, nobody has been able to determine the truth of the situation. A more technical definition is that a problem is NP-Complete if it is in NP and is NP-hard.

NP-Completeness proof

A type of reduction used to demonstrate that a particular problem is NP-complete. Specifically, an NP-Completeness proof must first show that the problem is in class NP, and then show (by using a reduction to another NP-Complete problem) that the problem is NP-hard.

NP-hard

A problem that is “as hard as” any other problem in NP. That is, Problem X is NP-hard if any algorithm in NP can be reduced to X in polynomial time.

nth roots of unity

All of the points along the unit circle in the complex plane that represent multiples of the primitive nth root of unity.

object

An instance of a class, that is, something that is created and takes up storage during the execution of a computer program. In the object-oriented programming paradigm, objects are the basic units of operation. Objects have state in the form of data members, and they know how to perform certain actions (methods).

object-oriented programming paradigm

An approach to problem-solving where all computations are carried out using objects.

object-space decomposition

A from of key-space decomposition where the key space is determined by the actual values of keys that are found. For example, a BST stores a key value in its root, and all other values in the tree with lesser value are in the left subtree. Thus, the root value has split (or decomposed) the key space for that key based on its value into left and right parts. An object-space decomposition is in opposition to an image-space decomposition.

octree

The three-dimensional equivalent of the quadtree would be a tree with or eight branches.

Omega notation

In algorithm analysis, notation is used to describe a lower bound. Roughly (but not completely) analogous to big-Oh notation used to define an upper bound.

one-way list

A synonym for a singly linked list.

open addressing

A synonym for closed hashing.

open hash system

A hash system where multiple records might be associated with the same slot of a hash table. Typically this is done using a linked list to store the records. This is in contrast to a closed hash system.

operating system

The control program for a computer. Its purpose is to control hardware, manage resources, and present a standard interface to these to other software components.

optimal static ordering

A theoretical construct defining the best static (non-changing) order in which to place a collection of records so as to minimize the number of records visited by a series of sequential searches. It is a useful concept for the purpose of defining a theoretical optimum against which to compare the performance for a self-organizing list heuristic.

optimization problem

Any problem where there are a (typically large) collection of potential solutions, and the goal is to find the best solution. An example is the Traveling Salesman Problem, where visiting cities in some order has a cost, and the goal is to visit in the cheapest order.

out degree

In graph terminology, the out degree for a vertex is the number of edges directed out of the vertex.

overflow

The condition where the amount of data stored in an entity has exceeded its capacity. For example, a node in a B-tree can store a certain number of records. If a record is attempted to be inserted into a node that is full, then something has to be done to handle this case.

overflow bucket

In bucket hashing, this is the bucket into which a record is placed if the bucket containing the record’s home slot is full. The overflow bucket is logically considered to have infinite capacity, though in practice search and insert will become relatively expensive if many records are stored in the overflow bucket.

overhead

All information stored by a data structure aside from the actual data. For example, the pointer fields in a linked list or BST, or the unused positions in an array-based list.

page

A term often used to refer to the contents of a single buffer within a buffer pool or other virtual memory. This corresponds to a single block or sector of data from backing storage, which is the fundamental unit of I/O.

parameter

The values making up an input to a function.

parent

In a tree, the node that directly links to a node is the parent of . is the child of .

parent pointer representation

For trees, a node implementation where each node stores only a pointer to its parent, rather than to its children. This makes it easy to go up the tree toward the root, but not down the tree toward the leaves. This is most appropriate for solving the UNION/FIND problem.

parity

The concept of matching even-ness or odd-ness, the basic idea behind using a parity bit for error detection.

parity bit

A common method for checking if transmission of a sequence of bits has been performed correctly. The idea is to count the number of 1 bits in the sequence, and set the parity bit to 1 if this number is odd, and 0 if it is even. Then, the transmitted sequence of bits can be checked to see if its parity matches the value of the parity bit. This will catch certain types of errors, in particular if the value for a single bit has been reversed. This was used, for example, in early versions of ASCII character coding.

parse tree

A tree that represents the syntactic structure of an input string, making it easy to compare against a grammar to see if it is syntactically correct.

parser

A part of a compiler that takes as input the program text (or more typically, the tokens from the scanner), and verifies that the program is syntactically correct. Typically it will build a parse tree as part of the process.

partial order

In set notation, a binary relation is called a partial order if it is antisymmetric and transitive. If the relation is also reflexive, then it is a non-strict partial order. Alternatively, if the relation is also irreflexive, then it is a strict partial order.

partially ordered set

The set on which a partial order is defined is called a partially ordered set.

partition

In Quicksort, the process of splitting a list into two sublists, such that one sublist has values less than the pivot value, and the other with values greater than the pivot. This process takes time on a sublist of length .

pass by reference

A reference to the variable is passed to the called function. So, any modifications will affect the original variable.

pass by value

A copy of a variable is passed to the called function. So, any modifications will not affect the original variable.

path

In tree or graph terminology, a sequence of vertices forms a path of length if there exist edges from to for .

path compression

When implementing the UNION/FIND algorithm, path compression is a local optimization step that can be performed during the FIND step. Once the root of the tree for the current object has been found, the path to the root can be traced a second time, with all objects in the tree made to point directly to the root. This reduces the depth of the tree from typically to nearly constant.

peripheral storage

Any storage device that is not part of the core processing of the computer (that is, RAM). A typical example is a disk drive.

permutation

A permutation of a sequence is the elements of arranged in some order.

persistent

In the context of computer memory, this refers to a memory that does not lose its stored information when the power is turned off.

physical file

The collection of sectors that comprise a file on a disk drive. This is in contrast to the logical file.

physical form

The implementation of a data type as a data structure. Contrast to the physical form for the data type.

Pigeonhole Principle

A commonly used lemma in Mathematics. A typical variant states: When objects are stored in locations, at least one of the locations must store two or more of the objects.

pivot

In Quicksort, the value that is used to split the list into sublists, one with lesser values than the pivot, the other with greater values than the pivot.

platter

In a disk drive, one of a series of flat disks that comprise the storage space for the drive. Typically, each surface (top and bottom) of each platter stores data, and each surface has its own I/O head.

point quadtree

A spatial data structure for storing point data. It is similar to a PR quadtree in that it (in two dimensions) splits the world into four parts. However, it splits using an object-space decomposition. That is, quadrant containing the point is split into four parts at the point. It is similar to the kd tree which splits alternately in each dimension, except that it splits in all dimensions at once.

point-region quadtree

Formal name for what is commonly referred to as a PR quadtree.

pointee

The term pointee refers to anything that is pointed to by a pointer or reference.

pointer

A variable whose value is the address of another variable; a link.

pointer-based implementation for binary tree nodes

A common way to implement binary tree nodes. Each node stores a data value (or a reference to a data value), and pointers to the left and right children. If either or both of the children does not exist, then a null pointer is stored.

polymorphism

An object-oriented programming term meaning one name, many forms. It describes the ability of software to change its behavior dynamically. Two basic forms exist: run-time polymorphism and compile-time polymorphism.

pop

A specialized term used to indicate removing an element from a stack.

poset

Another name for a partially ordered set.

position

The defining property of the list ADT, this is the concept that list elements are in a position. Many list ADTs support access by position.

postorder traversal

In a binary tree, a traversal that first recursively visits the left child, then recursively visits the right child, and then visits the root.

potential

A concept related to amortized analysis. Potential is the total or currently available amount of work that can be done.

powerset

For a set , the power set is the set of all possible subsets for .

PR quadtree

A type of quadtree that stores point data in two dimensions. The root of the PR quadtree represents some square region of 2d space. If that space stores more than one data point, then the region is decomposed into four equal subquadrants, each represented recursively by a subtree of the PR quadtree. Since many leaf nodes of the PR quadtree will contain no data points, implementation often makes use of the Flyweight design pattern. Related to the bintree.

prefix property

Given a collection of strings, the collection has the prefix property if no string in the collection is a prefix for another string in the collection. The significance is that, given a long string composed of members of the collection, it can be uniquely decomposed into the constituent members. An example of such a collection of strings with the prefix property is a set of Huffman codes.

preorder traversal

In a binary tree, a traversal that first visits the root, then recursively visits the left child, then recursively visits the right child.

Prim’s algorithm

A greedy algorithm for computing the MCST of a graph. It is nearly identical to Dijkstra’s algorithm for solving the single-source shortest paths problem, with the only difference being the calculation done to update the best-known distance.

primary clustering

In hashing, the tendency in certain collision resolution methods to create clustering in sections of the hash table. The classic example is linear probing. This tends to happen when a group of keys follow the same probe sequence during collision resolution.

primary index

Synonym for primary key index.

primary key

A unique identifier for a record.

primary key index

Relates each primary key value with a pointer to the actual record on disk.

primary storage

The faster but more expensive memory in a computer, most often RAM in modern computers. This is in contrast to secondary storage, which together with primary storage devices make up the computer’s memory hierarchy.

primitive data type

In Java, one of a particular group of simple types that are not implemented as objects. An example is an int.

primitive element

In set notation, this is a single element that is a member of the base type for the set. This is as opposed to an element of the set being another set.

primitive nth root of unity

The th root of 1. Normally a complex number. An intuitive way to view this is one th of the unit circle in the complex plain.

priority

A quantity assigned to each of a collection of jobs or tasks that indicate importance for order of processing. For example, in an operating system, there could be a collection of processes (jobs) ready to run. The operating system must select the next task to execute, based on their priorities.

priority queue

An ADT whose primary operations of insert of records, and deletion of the greatest (or, in an alternative implementation, the least) valued record. Most often implemented using the heap data structure. The name comes from a common application where the records being stored represent tasks, with the ordering values based on the priorities of the tasks.

probabilistic algorithm

A form of randomized algorithm that might yield an incorrect result, or that might fail to produce a result.

probabilistic data structure

Any data structure that uses probabilistic algorithms to perform its operations. A good example is the skip list.

probe function

In hashing, the function used by a collision resolution method to calculate where to look next in the hash table.

probe sequence

In hashing, the series of slots visited by the probe function during collision resolution.

problem

A task to be performed. It is best thought of as a function or a mapping of inputs to outputs.

problem instance

A specific selection of values for the parameters to a problem. In other words, a specific set of inputs to a problem. A given problem instance has a size under some cost model.

problem lower bound

In algorithm analysis, the tightest lower bound that we can prove over all algorithms for that problem. This is often much harder to determine than the problem upper bound. Since the lower bound for the algorithm can be very different for different situations (such as the best case or worst case), we typically have to specify which situation we are referring to.

problem upper bound

In algorithm analysis, the upper bound for the best algorithm that we know for the problem. Since the upper bound for the algorithm can be very different for different situations (such as the best case or worst case), we typically have to specify which situation we are referring to.

procedural

Typically referring to the procedural programming paradigm, in contrast to the object-oriented programming paradigm.

procedural programming paradigm

Procedural programming uses a list of instructions (and procedure calls) that define a series of computational steps to be carried out. This is in contrast to the object-oriented programming paradigm.

production

production rule

A grammar is comprised of production rules. The production rules consist of terminals and non-terminals, with one of the non-terminals being the start symbol. Each production rule replaces one or more non-terminals (perhaps with associated terminals) with one or more terminals and non-terminals. Depending on the restrictions placed on the form of the rules, there are classes of languages that can be represented by specific types of grammars. A derivation is a series of productions that results in a string (that is, all non-terminals), and this derivation can be represented as a parse tree.

program

An instance, or concrete representation, of an algorithm in some programming language.

promotion

In the context of certain balanced tree structures such as the 2-3 tree, a promotion takes place when an insertion causes the node to overflow. In the case of the 2-3 tree, the key with the middlemost value is sent to be stored in the parent.

proof

The establishment of the truth of anything, a demonstration.

proof by contradiction

A mathematical proof technique that proves a theorem by first assuming that the theorem is false, and then uses a chain of reasoning to reach a logical contradiction. Since when the theorem is false a logical contradiction arises, the conclusion is that the theorem must be true.

proof by induction

A mathematical proof technique similar to recursion. It is used to prove a parameterized theorem , that is, a theorem where there is a induction variable involved (such as the sum of the numbers from 1 to ). One first proves that the theorem holds true for a base case, then one proves the implication that whenever is true then is also true. Another variation is strong induction.

proving the contrapositive

We can prove that by proving .

pseudo polynomial

In complexity analysis, refers to the time requirements of an algorithm for an NP-Complete problem that still runs acceptably fast for practical application. An example is the standard dynamic programming algorithm for the knapsack problem.

pseudo random

In random number theory this means that, given all past terms in the series, no future term of the series can be accurately predicted in polynomial time.

pseudo-random probing

In hashing, this is a collision resolution method that stores a random permutation of the values 1 through the size of the hash table. Term of the probe sequence is simply the value of position in the permuation.

push

A specialized term used to indicate inserting an element onto a stack.

pushdown automata

PDA

A type of Finite State Automata that adds a stack memory to the basic Deterministic Finite Automata machine. This extends the set of languages that can be recognize to the context-free languages.

quadratic growth rate

A growth rate function of the form where is the input size and is a constant.

quadratic probing

In hashing, this is a collision resolution method that computes term of the probe sequence using some quadratic equation for suitable constants . The simplest form is simply to use as term of the probe sequence.

quadtree

A full tree where each internal node has four children. Most typically used to store two dimensional spatial data. Related to the bintree. The difference is that the quadtree splits all dimensions simultaneously, while the bintree splits one dimension at each level. Thus, to extend the quadtree concept to more dimensions requires a rapid increase in the number of splits (for example, 8 in three dimensions).

queue

A list-like structure in which elements are inserted only at one end, and removed only from the other one end.

Quicksort

A sort that is in the best and average cases, though in the worst case. However, a reasonable implmentation will make the worst case occur under exceedingly rare circumstances. Due to its tight inner loop, it tends to run better than any other known sort in general cases. Thus, it is a popular sort to use in code libraries. It works by divide and conquer, by selecting a pivot value, splitting the list into parts that are either less than or greater than the pivot, and then sorting the two parts.

radix

Synonym for base. The number of digits in a number representation. For example, we typically represent numbers in base (or radix) 10. Hexidecimal is base (or radix) 16.

radix sort

A sorting algorithm that works by processing records with digit keys in passes, where each pass sorts the records according to the current digit. At the end of the process, the records will be sorted. This can be efficient if the number of digits is small compared to the number of records. However, if the records all have unique key values, than at least digits are required, leading to an sorting algorithm that tends to be much slower than other sorting algorithms like Quicksort or mergesort.

RAM

Abbreviation for Random Access Memory.

random access

In file processing terminology, a disk access to a random position within the file. More generally, the ability to access an arbitrary record in the file.

random access memory

Abbreviated RAM, this is the principle example of primary storage in a modern computer. Data access times are typically measured in billionths of a second (microseconds), which is roughly a million times faster than data access from a disk drive. RAM is where data are held for immediate processing, since access times are so much faster than for secondary storage. RAM is a typical part of a computer’s memory hierarchy.

random permutation

One of the possible permutations for a set of element is selected in such a way that each permutation has equal probability of being selected.

randomized algorithm

An algorithm that involves some form of randomness to control its behavior. The ultimate goal of a randomized algorithm is to improve performance over a deterministic algorithm to solve the same problem. There are a number of variations on this theme. A “Las Vegas algorithm” returns a correct result, but the amount of time required might or might not improve over a deterministic algorithm. A “Monte Carlo algorithm” is a form of probabilistic algorithm that is not guarenteed to return a correct result, but will return a result relatively quickly.

range

The set of possible outputs for a function.

range query

Records are returned if their relevant key value falls with a specified range.

read/write head

Synonym for I/O head.

rebalancing operation

An operation performed on balanced search trees, such as the AVL Tree or Splay Tree, for the purpose of keeping the tree height balanced.

record

A collection of information, typically implemented as an object in an object-oriented programming language. Many data structures are organized containers for a collection of records.

recurrence relation

A recurrence relation (or less formally, recurrence) defines a function by means of an expression that includes one or more (smaller) instances of itself. A classic example is the recursive definition for the factorial function, .

recurrence with full history

A special form of recurrence relation that includes a summation with a copy of the recurrence inside. The recurrence that represents the average case cost for Quicksort is an example. This internal summation can typically be removed with simple techniques to simplify solving the recurrence.

recursion

The process of using recursive calls. An algorithm is recursive if it calls itself to do part of its work. See recursion.

recursive call

Within a recursive function, it is a call that the function makes to itself.

recursive data structure

A data structure that is partially composed of smaller or simpler instances of the same data structure. For example, linked lists and binary trees can be viewed as recursive data structures.

recursive function

A function that includes a recursive call.

recursively enumerable

A language is recursively enumerable if there exists a Turing machine such that .

Red-Black Tree

A balanced variation on a BST.

reduction

In algorithm analysis, the process of deriving asymptotic bounds for one problem from the asymptotic bounds of another. In particular, if problem A can be used to solve problem B, and problem A is proved to be in , then problem B must also be in . Reductions are often used to show that certain problems are at least as expensive as sorting, or that certain problems are NP-Complete.

reference

A value that enables a program to directly access some particular data item. An example might be a byte position within a file where the record is stored, or a pointer to a record in memory. (Note that Java makes a distinction between a reference and the concept of a pointer, since it does not define a reference to necessarily be a byte position in memory.)

reference count algorithm

An algorithm for garbage collection. Whenever a reference is made from a variable to some memory location, a counter associated with that memory location is incremented. Whenever the reference is changed or deleted, the reference count is decremented. If this count goes to zero, then the memory is considered free for reuse. This approach can fail if there is a cycle in the chain of references.

reference parameter

A parameter that has been passed by reference. Such a parameter can be modified inside the function or method.

reflexive

In set notation, binary relation on set is reflexive if for all .

Region Quadtree

A spatial data structure for storing 2D pixel data. The idea is that the root of the tree represents the entire image, and it is recursively divided into four equal subquadrants if not all pixels associated with the current node have the same value. This is structurally equivalent to a PR quadtree, only the decomposition rule is changed.

regular language

A language is a regular language if and only if there exists a Deterministic Finite Automata such that .

relation

In set notation, a relation over set is a set of ordered pairs from .

replacement selection

A variant of heapsort most often used as one phase of an external sort. Given a collection of records stored in an array, and a stream of additional records too large to fit into working memory, replacement selection will unload the heap by sending records to an output stream, and seek to bring new records into the heap from the input stream in preference to shrinking the heap size whenever possible.

reserved block

In a memory manager, this refers to space in the memory pool that has been allocated to store data received from the client. This is in contrast to the free blocks that represent space in the memory pool that is not allocated to storing client data.

resource constraints

Examples of resource constraints include the total space available to store the data (possibly divided into separate main memory and disk space constraints) and the time allowed to perform each subtask.

root

In a tree, the topmost node of the tree. All other nodes in the tree are descendants of the root.

rotation

In the AVL Tree and Splay Tree, a rotation is a local operation performed on a node, its children, and its grandchildren that can result in reordering their relationship. The goal of performing a rotation is to make the tree more balanced.

rotational delay

When processing a disk access, the time that it takes for the first byte of the desired data to move to under the I/O head. On average, this will take one half of a disk rotation, and so constitutes a substantial portion of the time required for the disk access.

rotational latency

A synonym for rotational delay.

run

A series of sorted records. Most often this refers to a (sorted) subset of records that are being sorted by means of an external sort.

run file

A temporary file that is created during the operation of an external sort, the run file contains a collection of runs. A common structure for an external sort is to first create a series of runs (stored in a run file), followed by merging the runs together.

run-time polymorphism

A form of polymorphism known as Overriding. Overridden methods are those which implement a new method with the same signature as a method inherited from its base class. Compare to compile-time polymorphism.

runtime environment

The environment in which a program (of a particular programming language) executes. The runtime environment handles such activities as managing the runtime stack, the free store, and the garbage collector, and it conducts the execution of the program.

runtime stack

The place where an activation record is stored when a subroutine is called during a program’s runtime.

scanner

The part of a compiler that is responsible for doing lexical analysis.

scope

The parts of a program that can see and access a variable.

search key

A field or part of a record that is used to represent the record when searching. For example, in a database of customer records, we might want to search by name. In this case the name field is used as the search key.

search lower bound

The problem of searching in an array has provable lower bounds for specific variations of the problem. For an unsorted array, it is comparisons in the worst case, typically proved using an adversary argument. For a sorted array, it is in the worst case, typically proved using an argument similar to the sorting lower bound proof. However, it is possible to search a sorted array in the average case in time.

search problem

Given a particular key value , the search problem is to locate a record in some collection of records L such that (if one exists). Searching is a systematic method for locating the record (or records) with key value .

search tree

A tree data structure that makes search by key value more efficient. A type of container, it is common to implement an index using a search tree. A good search tree implementation will guarentee that insertion, deletion, and search operations are all .

search trie

Any search tree that is a trie.

searching

Given a search key and some collection of records L, searching is a systematic method for locating the record (or records) in L with key value .

secondary clustering

In hashing, the tendency in certain collision resolution methods to create clustering in sections of the hash table. In primary clustering, this is caused by a cluster of keys that don’t necessarily hash to the same slot but which following significant portions of the same probe sequence during collision resolution. Secondary clustering results from the keys hashing to the same slot of the table (and so a collision resolution method that is not affected by the key value must use the same probe sequence for all such keys). This problem can be resolved by double hashing since its probe sequence is determined in part by a second hash function.

secondary index

Synonym for secondary key index.

secondary key

A key field in a record such as salary, where a particular key value might be duplicated in multiple records. A secondary key is more likely to be used by a user as a search key than is the record’s primary key.

secondary key index

Associates a secondary key value with the primary key of each record having that secondary key value.

secondary storage

Refers to slower but cheaper means of storing data. Typical examples include a disk drive, a USB memory stick, or a solid state drive.

sector

A unit of space on a disk drive that is the amount of data that will be read or written at one time by the disk drive hardware. This is typically 512 bytes.

sector header

On a disk drive, a piece of information at the start of a sector that allows the I/O head to recognize the identity (or equivalently, the address) of the current sector.

seed

In random number theory, the starting value for a random number series. Typically used with any linear congruential method.

seek

On a disk drive, the act of moving the I/O head from one track to another. This is usually considered the most expensive step during a disk access.

selection sort

While this sort requires time in the best, average, and worst cases, it requires only swap operations. Thus, it does relatively well in applications where swaps are expensive. It can be viewed as an optimization on bubble sort, where a swap is deferred until the end of each iteration.

self-organizing list

A list that, over a series of search operations, will make use of some heuristic to re-order its elements in an effort to improve search times. Generally speaking, search is done sequentially from the beginning, but the self-organizing heuristic will attempt to put the records that are most likely to be searched for at or near the front of the list. While typically not as efficient as binary search on a sorted list, self-organizing lists do not require that the list be sorted (and so do not pay the cost of doing the sorting operation).

self-organizing list heuristic

A heuristic to use for the purpose of maintaining a self-organizing list. Commonly used heuristics include move-to-front and transpose.

separate chaining

In hashing, a synonym for open hashing

sequence

In set notation, a collection of elements with an order, and which may contain duplicate-valued elements. A sequence is also sometimes called a tuple or a vector.

sequential access

In file processing terminology, the requirement that all records in a file are accessed in sequential order. Alternatively, a storage device that can only access data sequentially, such as a tape drive.

sequential fit

In a memory manager, the process of searching the memory pool for a free block large enough to service a memory request, possibly reserving the remaining space as a free block. Examples are first fit, circular first fit, best fit, and worst fit.

sequential search

The simplest search algorithm: In an array, simply look at the array elements in the order that they appear.

sequential tree representation

A representation that stores a series of node values with the minimum information needed to reconstruct the tree structure. This is a technique for serializing a tree.

serialization

The process of taking a data structure in memory and representing it as a sequence of bytes. This is sometimes done in order to transmit the data structure across a network or store the data structure in a stream, such as on disk. Deserialization reconstructs the original data structure from the serialized representation.

set

A collection of distinguishable members or elements.

set product

Written , the set product is a set of ordered pairs such that ordered pair is in the product whenever and . For example, when and , .

shallow copy

Copying the reference or pointer value without copying the actual content.

Shellsort

A sort that relies on the best-case cost of insertion sort to improve over worst case cost.

shifting method

A technique for finding a closed-form solution to a summation or recurrence relation.

shortest path

Given a graph with distances or weights on the edges, the shortest path between two nodes is the path with least total distance or weight. Examples of the shortest paths problems are the single-source shortest paths problem and the all-pairs shortest paths problem.

sibling

In a tree, a sibling of node is any other node with the same parent as .

signature

In a programming language, the signature for a function is its return type and its list of parameters and their types.

signature file

In document processing, a signature file is a type of bitmap used to indicate which documents in a collection contain a given keyword, such that there is a bitmap for each keyword.

simple cycle

In graph terminology, a cycle is simple if its corresponding path is simple, except that the first and last vertices of the cycle are the same.

simple path

In graph terminology, a path is simple if all vertices on the path are distinct.

simple type

A data type whose values contain no subparts. An example is the integers.

simulating recursion

If a programming language does not support recursion, or if you want to implement the effects of recursion more efficiently, you can use a stack to maintain the collection of subproblems that would be waiting for completion during the recursive process. Using a loop, whenever a recursive call would have been made, simply add the necessary program state to the stack. When a return would have been made from the recursive call, pop the previous program state off of the stack.

single rotation

A type of rebalancing operation used by the Splay Tree and AVL Tree.

single-source shortest paths problem

Given a graph with weights or distances on the edges, and a designated start vertex , find the shortest path from to every other vertex in the graph. One algorithm to solve this problem is Dijkstra’s algorithm.

singly linked list

A linked list implementation variant where each list node contains access an pointer only to the next element in the list.

skip list

A form of linked list that adds additional links to improve the cost of fundamental operations like insert, delete, and search. It is a probabilistic data structure since it adds the additional links using a probabilistic algorithm. It can implement a dictionary more efficiently than a BST, and is roughly as difficult to implement.

slot

In hashing, a position in a hash table.

snowplow argument

An analogy used to give intuition for why replacement selection will generate runs that are on average twice the size of working memory. Records coming from the input stream have key values that might be of any size, whose size is related to the position of a falling snowflake. The replacement selection process is analogous to a snowplow that moves around a circular track picking up snow. In steady state, given a certain amount of snow equivalent to working memory size , an amount of snow (incoming records from the input stream) is expected to fall ahead of the plow as the size of the working memory during one cycle of the plow (analogously, one run of the replacement selection algorithm). Thus, the snowplow is expected in one pass (one run of replacement selection) to pick up snow.

software engineering

The study and application of engineering to the design, development, and maintenance of software.

software reuse

In software engineering, the concept of reusing a piece of software. In particular, using an existing piece of software (such as a function or library) when creating new software.

solution space

The possible solutions to a problem. This typically refers to an optimization problem, where some solutions are more desireable than others.

solution tree

An ordering imposed on the set of solutions within a solution space in the form of a tree, typically derived from the order that some algorithm would visit the solutions.

sorted list

A list where the records stored in the list are arranged so that their key values are in ascending order. If the list uses an array-based list implementation, then it can use binary search for a cost of . But both insertion and deletion will be require time.

sorting lower bound

The lower bound for the problem of sorting is . This is traditionally proved using a decision tree model for sorting algorithms, and recognizing that the minimum depth of the decision tree for any sorting algorithm is since there are permutations of the input records to distinguish between during the sorting process.

sorting problem

Given a set of records , , …, with key values , , …, , the sorting problem is to arrange the records into any order such that records , , …, have keys obeying the property . In other words, the sorting problem is to arrange a set of records so that the values of their key fields are in non-decreasing order.

space/time tradeoff

Many programs can be designed to either speed processing at the cost of additional storage, or reduce storage at the cost of additional processing time.

sparse graph

A graph where the actual number of edges is much less than the possible number of edges. Generally, this is interpreted to mean that the degree for any vertex in the graph is relatively low.

sparse matrix

A matrix whose values are mostly zero. There are a number of data structures that have been developed to store sparse matrices, with the goal of reducing the amount of space required to represent it as compared to simply using a regular matrix representation that stores a value for every matrix position.

spatial

Referring to a position in space.

spatial application

An application what has spatial aspects. In particular, an application that stores records that need to be searched by location.

spatial attribute

An attribute of a record that has a position in space, such as the coordinate. This is typically in two or more dimensions.

spatial data

Any object or record that has a position (in space).

spatial data structure

A data structure designed to support efficient processing when a spatial attribute is used as the key. In particular, a data structure that supports efficient search by location, or finds all records within a given region in two or more dimensions. Examples of spatial data structures to store point data include the bintree, the PR quadtree and the kd tree.

spindle

The center of a disk drive that holds the platters in place.

Splay Tree

A variant implementation for the BST, which differs from the standard BST in that it uses modified insert and remove methods in order to keep the tree balanced. Similar to an AVL Tree in that it uses the concept of rotations in the insert and remove operations. While a Splay Tree does not guarentee that the tree is balanced, it does guarentee that a series of operations on the tree will have a total cost of cost, meaning that any given operation can be viewed as having amortized cost of .

splaying

The act of performing an rebalancing operation on a Splay Tree.

stable

A sorting algorithm is said to be stable if it does not change the relative ordering of records with identical key values.

stack

A list-like structure in which elements may be inserted or removed from only one end.

stack frame

Frame of data that pushed into and poped from call stack

stack variable

Another name for a local variable.

stale pointer

Within the context of a buffer pool or memory manager, this means a reference to a buffer or memory location that is no longer valid. For example, a program might make a memory request to a buffer pool, and be given a reference to the buffer holding the requested data. Over time, due to inactivity, the contents of this buffer might be flushed. If the program holding the buffer reference then tries to access the contents of that buffer again, then the data contents will have changed. The possibility for this to occur depends on the design of the interface to the buffer pool system. Some designs make this impossible to occur. Other designs make it possible in an attempt to deliver greater performance.

start state

In a finite automata, the designated state in which the machine will always begin a computation.

start symbol

In a grammar, the designated non-terminal that is the intial point for deriving a string in the langauge.

state

The condition that something is in at some point in time. In computing, this typically means the collective values of any existing variables at some point in time. In an automata, a state is an abstract condition, possibly with associated information, that is primarily defined in terms of the conditions that the automata may transition from its present state to another state.

State Machine

Synonym for finite automata.

static

Something that is not changing (in contrast to dynamic). In computer programming, static normally refers to something that happens at compile time. For example, static analysis is analysis of the program’s text or structure, as opposed to its run-time behavior. Static binding or static memory allocation occurs at compile time.

static scoping

A synonym for lexical scoping.

Strassen’s algorithm

A recursive algorithm for matrix multiplication. When multiplying two matrices, this algorithm runs faster than the time required by the standard matrix multiplication algorithm. Specifically, Strassen’s algorithm requires time time. This is achieved by refactoring the sub-matrix multiplication and addition operations so as to need only 7 sub-matrix multiplications instead of 8, at a cost of additional sub-matrix addition operations. Thus, while the asymptotic cost is lower, the constant factor in the growth rate equation is higher. This makes Strassen’s algorithm inefficient in practice unless the arrays being multiplied are rather large. Variations on Strassen’s algorithm exist that reduce the number of sub-matrix multiplications even futher at a cost of even more sub-matrix additions.

strategy

An approach to accomplish a task, often encapsulated as an algorithm. Also the name for a design pattern that separates the algorithm for performing a task from the control for applying that task to each member of a collection. A good example is a generic sorting function that takes a collection of records (such as an array) and a “strategy” in the form of an algorithm that knows how to extract the key from a record in the array. Only subtly different from the visitor design pattern, where the difference is primarily one of intent rather than syntax. The strategy design pattern is focused on encapsulating an activity that is part of a larger process, so that different ways of performing that activity can be substituted. The visitor design pattern is focused on encapsulating an activity that will be performed on all members of a collection so that completely different activities can be substituted within a generic method that accesses all of the collection members.

stream

The process of delivering content in a serialized form.

strict partial order

In set notation, a relation that is irreflexive, antisymmetric, and transitive.

strong induction

An alternative formulation for the induction step in a proof by induction. The induction step for strong induction is: If Thrm holds for all , then Thrm holds for .

subclass

In object-oriented programming, any class within a class hierarchy that inherits from some other class.

subgraph

A subgraph is formed from graph by selecting a subset of ’s vertices and a subset of ’s edges such that for every edge , both vertices of are in .

subset

In set theory, a set is a subset of a set , or equivalently is a superset of , if all elements of are also elements of .

subtract-and-guess

A technique for finding a closed-form solution to a summation or recurrence relation.

subtree

A subtree is a subset of the nodes of a binary tree that includes some node of the tree as the subtree root along with all the descendants of .

successful search

When searching for a key value in a collection of records, we might find it. If so, we call this a successful search. The alternative is an unsuccessful search.

summation

The sum of costs for some function applied to a range of parameter values. Often written using Sigma notation. For example, the sum of the integers from 1 to can be written as .

superset

In set theory, a set is a subset of a set , or equivalently is a superset of , if all elements of are also elements of .

symbol table

As part of a compiler, the symbol table stores all of the identifiers in the program, along with any necessary information needed about the identifier to allow the compiler to do its job.

symmetric

In set notation, relation is symmetric if whenever , then , for all .

symmetric matrix

A square matrix that is equal to its transpose. Equivalently, for a matrix , for all , .

syntax analysis

A phase of compilation that accepts tokens, checks if program is syntactically correct, and then generates a parse tree.

tail

The end of a list.

terminal

A specific character or string that appears in a production rule. In contrast to a non-terminal, which represents an abstract state in the production. Similar to a literal, but this is the term more typically used in the context of a compiler.

Theta notation

In algorithm analysis, notation is used to indicate that the upper bound and lower bound for an algorithm or problem match.

token

The basic logical units of a program, as deterimined by lexical analysis. These are things like arithmetic operators, language keywords, variable or function names, or numbers.

tombstone

In hashing, a tombstone is used to mark a slot in the hash table where a record has been deleted. Its purpose is to allow the collision resolution process to probe through that slot (so that records further down the probe sequence are not unreachable after deleting the record), while also allowing the slot to be reused by a future insert operation.

topological sort

The process of laying out the vertices of a DAG in a linear order such that no vertex in the order is preceded by a vertex that can be reached by a (directed) path from . Usually the (directed) edges in the graph define a prerequisite system, and the goal of the topological sort is to list the vertices in an order such that no prerequisites are violated.

total order

A binary relation on a set where every pair of distinct elements in the set are comparable (that is, one can determine which of the pair is greater than the other).

total path length

In a tree, the sum of the levels for each node.

Towers of Hanoi problem

A standard example of a recursive algorithm. The problem starts with a stack of disks (each with unique size) stacked decreasing order on the left pole, and two additional poles. The problem is to move the disks to the right pole, with the constraints that only one disk can be moved at a time and a disk may never be on top of a smaller disk. For disks, this problem requires moves. The standard solution is to move disks to the middle pole, move the bottom disk to the right pole, and then move the disks on the middle pole to the right pole.

track

On a disk drive, a concentric circle representing all of the sectors that can be viewed by the I/O head as the disk rotates. The significance is that, for a given placement of the I/O head, the sectors on the track can be read without performing a (relatively expensive) seek operation.

track-to-track seek time

Expected (average) time to perform a seek operation from a random track to an adjacent track. Thus, this can be viewed as the minimum possible seek time for the disk drive. This is one of two metrics commonly provided by disk drive vendors for disk drive performance, with the other being average seek time.

trailer node

Commonly used in implementations for a linked list or related structure, this node follows the last element of the list. Its purpose is to simplify the code implementation by reducing the number of special cases that must be programmed for.

transitive

In set notation, relation is transitive if whenever and , then , for all .

transpose

In the context of linear algebra, the transpose of a matrix is another matrix created by writing the rows of as the columns of . In the context of a self-organizing list, transpose is a heuristic used to maintain the list. Under this heuristic, whenever a record is accessed it is moved one position closer to the front of the list.

trap state

In a FSA, any state that has all transitions cycle back to itself. Such a state might be final.

traversal

Any process for visiting all of the objects in a collection (such as a tree or graph) in some order.

tree

A tree is a finite set of one or more nodes such that there is one designated node , called the root of . If the set is not empty, these nodes are partitioned into disjoint sets , $\mathbf{T}1\mathbf{T}{n-1}$, each of which is a tree, and whose roots , respectively, are children of .

tree traversal

A traversal performed on a tree. Traditional tree traversals include preorder and postorder traversals for both binary and general trees, and inorder traversal that is most appropriate for a BST.

trie

A form of search tree where an internal node represents a split in the key space at a predetermined location, rather than split based on the actual key values seen. For example, a simple binary search trie for key values in the range 0 to 1023 would store all records with key values less than 512 on the left side of the tree, and all records with key values equal to or greater than 512 on the right side of the tree. A trie is always a full tree. Folklore has it that the term comes from “retrieval”, and should be pronounced as “try” (in contrast to “tree”, to distinguish the differences in the space decomposition method of a search tree versus a search trie). The term “trie” is also sometimes used as a synonym for the alphabet trie.

truth table

In symbolic logic, a table that contains as rows all possible combinations of the boolean variables, with a column that shows the outcome (true or false) for the expression when given that row’s truth assignment for the boolean variables.

tuple

In set notation, another term for a sequence.

Turing machine

A type of finite automata that, while simple to define completely, is capable of performing any computation that can be performed by any known computer.

Turing-acceptable

A language is if there is some Turing machine that accepts it. That is, the machine will halt in an accepting configuration if the string is in the language, and go into a hanging configuration if the string is not in the language.

Turing-computable function

Any function for which there exists a Turing machine that can perform the necessary work to compute the function.

Turing-decidable

A language is Turing-decideable if there exists a Turing machine that can clearly indicate for every string whether that string is in the language or not. Every Turing-decidable language is also Turing-acceptable, because the Turing machine that can decide if the string is in the language can be modified to go into a hanging configuration if the string is not in the language.

two-coloring

An assignment from two colors to regions in an image such that no two regions sharing a side have the same color.

type

A collection of values.

unary notation

A way to represent natural numbers, where the value of zero is represented by the empty string, and the value is represented by a series of marks.

uncountably infinite

uncountable

An infinite set is uncountably infinite if there does not exist any mapping from it to the set of integers. This is often proved using a diagonalization argument. The real numbers is an example of an uncountably infinite set.

underflow

The condition where the amount of data stored in an entity has dropped below some minimum threshold. For example, a node in a B-tree is required to be at least half full. If a record deletion causes the node to be less than half full, then it is in a condition of underflow, and something has to be done to correct this.

undirected edge

An edge that connects two vertices with no direction between them. Many graph representations will represent such an edge with two directed edges.

undirected graph

A graph whose edges do not have a direction.

uninitialized

Uninitialized variable means it has no initial value.

UNION

One half of the UNION/FIND algorithm for managing disjoint sets. It is the process of merging two trees that are represented using the parent pointer representation by making the root for one of the trees set its parent pointer to the root of the other tree.

UNION/FIND

A process for mainining a collection of disjoint sets. The FIND operation determines which disjoint set a given object resides in, and the UNION operation combines two disjoint sets when it is determined that they are members of the same equivalence class under some equivalence relation.

unit production

A unit production is a production in a grammar of the form , where the set of non-terminals for the grammar. Any grammar with unit productions can be rewritten to remove them.

unsolveable problem

A problem that can proved impossible to solve on a computer. The classic example is the halting problem.

unsorted list

A list where the records stored in the list can appear in any order (as opposed to a sorted list). An unsorted list can support efficient () insertion time (since you can put the record anywhere convenient), but requires time for both search and and deletion.

unsuccessful search

When searching for a key value in a collection of records, we might not find it. If so, we call this an unsuccessful search. Usually we require that this means that no record in the collection actually has that key value (though a probabilistic algorithm for search might not require this to be true). The alternative to an unsuccessful search is a successful search.

unvisited

In graph algorithms, this refers to a node that has not been processed at the current point in the algorithm. This information is typically maintained by using a mark array.

upper bound

In algorithm analysis, a growth rate that is always greater than or equal to the that of the algorithm in question. In practice, this is the slowest-growing function that we know grows at least as fast as all but a constant number of inputs. It could be a gross over-estimate of the truth. Since the upper bound for the algorithm can be very different for different situations (such as the best case or worst case), we typically have to specify which situation we are referring to.

value parameter

A parameter that has been passed by value. Changing such a parameter inside the function or method will not affect the value of the calling parameter.

variable-length coding

Given a collection of objects, a variable-length coding scheme assigns a code to each object in the collection using codes that can be of different lengths. Typically this is done in a way such that the objects that are most likely to be used have the shortest codes, with the goal of minimizing the total space needed to represent a sequence of objects, such as when representing the characters in a document. Huffman coding is an example of a variable-length coding scheme. This is in contrast to fixed-length coding.

vector

In set notation, another term for a sequence. As a data structure, the term vector usually used as a snyonym for a dynamic array.

vertex

Another name for a node in a graph.

virtual memory

A memory management technique for making relatively fast but small memory appear larger to the program. The large “virtual” data space is actually stored on a relatively slow but large backing storage device, and portions of the data are copied into the smaller, faster memory as needed by use of a buffer pool. A common example is to use RAM to manage access to a large virtual space that is actually stored on a disk drive. The programmer can implement a program as though the entire data content were stored in RAM, even if that is larger than the physical RAM available making it easier to implement.

visit

During the process of a traversal on a graph or tree the action that takes place on each node.

visited

In graph algorithms, this refers to a node that has previously been processed at the current point in the algorithm. This information is typically maintained by using a mark array.

visitor

A design pattern where a traversal process is given a function (known as the visitor) that is applied to every object in the collection being traversed. For example, a generic tree or graph traversal might be designed such that it takes a function parameter, where that function is applied to each node.

volatile

In the context of computer memory, this refers to a memory that loses all stored information when the power is turned off.

weight

A cost or distance most often associated with an edge in a graph.

weighted graph

A graph whose edges each have an associated weight or cost.

weighted path length

Given a tree, and given a weight for each leaf in the tree, the weighted path length for a leaf is its weight times its depth.

weighted union rule

When merging two disjoint sets using the UNION/FIND algorithm, the weighted union rule is used to determine which subtree’s root points to the other. The root of the subtree with fewer nodes will be set to point to the root of the subtree with more nodes. In this way, the average depth of nodes in the resulting tree will be less than if the assignment had been made in the other direction.

working memory

The portion of main memory available to an algorithm for its use. Typically refers to main memory made available to an algorithm that is operating on large amounts of data stored in peripheral storage, the working memory represents space that can hold some subset of the total data being processed.

worst case

In algorithm analysis, the problem instance from among all problem instances for a given input size that has the greatest cost. Note that the worst case is not when is big, since we are referring to the wrost from a class of inputs (i.e, we want the worst of those inputs of size ).

worst fit

In a memory manager, worst fit is a heuristic for deciding which free block to use when allocating memory from a memory pool. Worst fit will always allocate from the largest free block. The rationale is that this will be the method least likely to cause external fragmentation in the form of small, unuseable memory blocks. The disadvantage is that it tends to eliminate the availability of large freeblocks needed for unusually large requests.

zigzig

A type of rebalancing operation used by splay trees.

Zipf distribution

A data distribution that follows Zipf’s law, an emprical observation that many types of data studied in the physical and social sciences follow a power law probability distribution. That is, the frequency of any record in the data collection is inversely proportional to its rank when the collection is sorted by frequency. Thus, the most frequently appearing record has a frequency much higher than the next most frequently appearing record, which in turn has a frequency much higher than the third (but with ratio slightly lower than that for the first two records) and so on. The 80/20 rule is a casual characterization of a Zipf distribution. Adherence to a Zipf distribution is important to the successful operation of a cache or self-organizing list.

zone

In memory management, the concept that different parts of the memory pool are handled in different ways. For example, some of the memory might be handled by a simple freelist, while other portions of the memory pool might be handled by a sequential fit memory manager. On a disk drive the concept of a zone relates to the fact that there are limits to the maximum data density, combined with the fact that the need for the same angular distance to be used for a sector in each track means that tracks further from the center of the disk will become progressively less dense. A zone in this case is a series of adjacent tracks whose data density is set by the maximum density of the innermost track of that zone. The next zone can then reset the data density for its innermost track, thereby gaining more total storage space while preserving angular distance for each sector.

15.2. Bibliography

[Ahern05]

Dennis Ahern et al., CMMI Distilled: a practical introduction to integrated process improvement, 2005. ISBN: 0-321-18613-3.

[Bacon]

Francis Bacon, Novum Organum, Google eBook, Clarendon Press, 1878.

[Beck99]

Kent Beck. Extreme Programming Explained: Embrace Change. 1999.

[Bloch]

Joshua Bloch, Effective Java, Second Edition, Addison-Wesley, 2008.

[Boehm03]

Barry Boehm and Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, 2003. ISBN: 0-321-18612-5.

[Booch]

Grady Booch, Object-Oriented Design With Applications, Benjamin/Cummings, Menlo Park, California, 1991.

[Brooks95]

Frederick P. Brooks, The Mythical Man-Month: Essays on Software Engineering, Second Edition, Addison-Wesley, 1995.

[Cockburn04]

Alistair Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams, 2004. ISBN: 0-201-69947-8

[GalilItaliano91]

Zvi Galil and Giuseppe F. Italiano, “Data Structures and Algorithms for Disjoint Set Union Problems”, Computing Surveys 23, 3(September 1991), 319-344.

[Gauss65]

Carl F. Gauss, Arthur A. Clarke (translator) Disquisitiones Arithmeticae, Yale University Press, 1965.

[KnuthV3]

Donald E. Knuth, The Art of Computer Programming Volume 3: Sorting and Searching, Second Edition, Addison-Wesley, Reading, MA, 1998.

[Lafore]

Robert Lafore, Data Structures & Algorithms in Java, Second Edition, Sams Publishing, 2003.

[Sierra]

Kathy Sierra and Bert Bates, OCA/OCP Java 7 SE Programmer I & II Study Guide (Exams 1Z0-803 & 1Z0-804), McGraw-Hill Education, 2015.

[Tarjan75]

Robert E. Tarjan, “On the efficiency of a good but not linear set merging algorithm”, Journal of the ACM 22, 2(April 1975), 215-225.

15.3. Spotlight: Carl Friedrich Gauss

He lives everywhere in mathematics.
— E.T. Bell, Men of Mathematics

Portrait of Carl Gauss [^1]

Figure 15.3.1: Oil painting of mathematician and philosopher Carl Friedrich Gauss by G. Biermann (1824-1908)

Carl Friedrich Gauss is considered by many the greatest mathematician who ever lived. He was born in Brunswick, Germany on April 30, 1777. Gauss was a child prodigy, who was reported as able to perform long computations in his head. At age 10 he studied algebra and analysis. He made his first fundamental discoveries while still a teenager. Among these was the least squares method for handling statistical data and a proof that a regular 17-sided polygon can be constructed with only a straightedge and a compass. This was the first result of its kind since discoveries by the Greeks 2,000 years earlier.

He completed his monumental book on number theory, Disquisitiones Arithmeticae in 1798 at the age of 21 [Gauss65]. It summarized previous work in a systematic way and introducing many fundamental ideas of his own.

In 1801, the same year Disquisitiones Arithmeticae was published, the asteroid Ceres was observed by astronomers. Unfortunately, they could only make observations across 3 degrees of the sky before it was obscured by the sun. Several months later, when Ceres should have reappeared, Piazzi could not locate it: the mathematical tools of the time were not able to extrapolate a position from such a scant amount of data—three degrees represent less than 1% of the total orbit. In what seemed a superhuman feat at the time, Gauss used the available data to calculate the orbit of Ceres. As part of his work, he showed that experimental data varies within a bell-shaped curve, now called the Gaussian distribution. This achievement established his reputation as a genius before the age of 25.

Gauss also developed tables of logarithms now known as Gaussian logarithms.

Gaussian logarithms are designed to facilitate finding the common logarithm of a sum or difference of two numbers whose common logarithms are known. The object of a table of Gaussian logarithms, sometimes known as Addition and Subtraction Logarithms, is to give by single entry when and are known.

When Gauss died, many unpublished notes and manuscripts were found in his desk. When his complete Collected Works were finally published later, it had taken a group of scientists nearly seventy years to review and edit his writings.

Today Gauss’s name occurs in many places in mathematics and science:

• The normal probability distribution as also called the Gaussian curve or distribution
• Gauss’s Laws for Gravity and Electrostatics
• The hypergeometric series, a.k.a the Gaussian series
• Gaussian equations in spherical trigonometry
• Gaussian curvature in differential geometry
• Gaussian optics and Gaussian beams describing electromagnetic radiation

Gauss died in Göttingen, at the age of 78 on February 23, 1855. In Brunswick, there is a statue of him. Its base is, appropriately, a 17 pointed star.

[^1]: Gottlieb Biermann, Portrait of Carl Friedrich Gauss By Gottlieb Biermann A. Wittmann (photo) [Public domain], via Wikimedia Commons

15.4. Spotlight: Francis Bacon

Portrait of Francis Bacon [^1]

Figure 15.4.1: Sr. Francis Bacon Lord Keeper, and afterwards Lord Chancellor of England, 1617

Sir Francis Bacon (1561 – 1626), was an English philosopher, statesman, scientist, jurist, orator, essayist, and author. He is considered one of the fathers of modern science and of the scientific method.

His scientific method was put forward in his book Nova Organum (New Method), and was designed to replace the methods put forward in Aristotle’s original works on logic from around 350 BCE. This New Method strongly influenced the development of the scientific method in modern science [Bacon].

His method distinguishes itself from the approach set down by Aristotle and his disciples, which had been followed by scientists for nearly 2,000 years. The Aristotelian approach favored exploring scientific problems through the application of logic, discussion, and rhetoric. In contrast, Bacon proposed an approach based on inductive reasoning supported by evidence.

Bacon also listed what he called the ‘idols of the mind’. He described four types of false images—things which obstructed the path of correct scientific reasoning.

Idols of the Tribe:

The tendency to perceive more order and regularity in systems than truly exists, and is due to people following their preconceived ideas about things.

Idols of the Cave:

Personal weaknesses in reasoning due to particular personalities, likes and dislikes.

Idols of the Marketplace:

Confusions in the use of language and taking some words in science to have a different meaning than their common usage.

Idols of the Theatre:

Following of academic dogma and not asking questions about the world.

[^1]: Frans Pourbus the younger, Portrait of Francis Bacon [Public domain], via Wikimedia Commons. File URL: https://upload.wikimedia.org/wikipedia/commons/a/a7/Pourbus_Francis_Bacon.jpg