7.

1
7 Lower Bounds for
7 Lower Bounds for
Sorting
Sorting
-
-
by
by
-
-
Comparisons
Comparisons
We have learned several algorithms that can sort n elements in
O(n lg n) time. Mergesort and heapsort achieve this upper bound
in the worst case; while quicksort achieves it on average.
These algorithms share an interesting property: the sorted
order is based only on comparisons between the input elements.
Such sorting algorithms are called comparison sorts.
We shall prove that any comparison sort must make (n lg n)
comparisons in the worst case to sort n elements.
Thus, mergesort and heapsort are asymptotically optimal.
There are some sorting algorithms (for example, counting sort,
radix sort, and bucket sort) that run in O(n) time. These
algorithms do not use comparisons to determine the sorted
order. The (n lg n) lower bound does not apply to them.
7.2
Comparisons
Comparisons
In a comparison sort, given two elements a
i
and a
j
, we perform
one of the tests: a
i
< a
j
, a
i
a
j
, a
i
a
j
, or a
i
> a
j
to determine their
correct order.
We do not inspect the values of the elements or gain order
information about them in any other way.
Note that the comparisons a
i
< a
j
, a
i
a
j
, a
i
a
j
, and a
i
> a
j
are all
equivalent in the sense that they give identical information
about the relative order of a
i
and a
j
. Therefore, we can assume
that all comparisons have the form a
i
a
j
.
Comparison sorts can be viewed in terms of decision trees.
A decision tree is a full binary tree that represents the
comparisons between elements that are performed by a
particular sorting algorithm operating on an input of a given size.
For example, see the decision tree for insertion sort operating
on 3 elements, on the whiteboard.
7.3
Decision
Decision
-
-
Tree Model
Tree Model
Each internal node is annotated by i:j which
represents a comparison a
i
a
j
for some i and j in the
range 1 i, j n, where n is the number of elements in
the input sequence.
Each leaf is annotated by a permutation
(1), (2), ..., (n).
The execution of the sorting algorithm corresponds
to tracing a path from the root of the tree to a leaf.
At each internal node, a comparison a
i
a
j
is made.
The left subtree indicates subsequent comparisons if
a
i
a
j
is true, and the right subtree indicates
subsequent comparisons if a
i
> a
j
is true.
When we come to a leaf, the sorting algorithm has
established the correct ordering.
7.4
A necessary condition for a comparison sort to be
correct is that each of the n! permutation on n
elements must appear as one of the leaves of the
decision tree, and that each of these leaves must be
reachable from the root by a path corresponding to
an actual execution of the comparison sort.
The worst-case number of comparisons for a given
comparison sort algorithm equals the height of its
decision tree.
A lower bound on the heights of all decision trees in
which each permutation appears as a reachable leaf
is a lower bound on the running time of any
comparison sort algorithm.
7.5
Theorem:
Any comparison sort algorithm requires (n lg n) compa-
risons in the worst case, where n is the number of
elements.
Proof:
Consider a decision tree of height h with k reachable leaves
corresponding to a comparison sort on n elements. Because each of
n! permutations of the input appears as a leaf, we have k >= n!.
Furthermore, since a binary tree of height h has no more than 2^h
leaves, we have 2^h >= k >= n!. Taking logarithms, we obtain
h >= lg(n!)
= (n lg n).
7.6
Counting Sort
Counting Sort
Assumption: each of the n input elements is an integer in the
range 0 to k.
When k = O(n), counting sort runs in (n) time.
The basic idea is to determine, for each input element x, the
number of elements less than x. This information can be used to
place element x directly into its position in the output array.
For example, if there are 8 elements less than x, then x belongs
in output position 9.
In the following pseudocode for counting sort, we assume that
the input is an array A[1 . . n]. Two other arrays are required:
the array B[1 . . n] holds the sorted output, and the array
C[0 . . k] provides temporary working storage.
7.7
COUNTING-SORT( A, B, k ) // n = length[A]
1. for i 0 to k
2. do C[i] 0
3. for j 1 to n
4. do C[ A[j] ] C[ A[j] ] + 1
5. //C[i] now contains the number of elements equal to i.
6. for i 1 to k
7. do C[i] C[i] + C[i - 1]
8. //C[i] now contains the number of elements less than or equal to i.
9. for j n downto 1 //Place A[j] in its correct sorted position.
10. do B[ C[ A[j] ] ] A[j]
11. C[ A[j] ] C[ A[j] ] - 1 //Taking care of equal elements.
See an example on the whiteboard: 2, 5, 3, 0, 2, 3, 0, 3
7.8
Running time of counting
Running time of counting
-
-
sort
sort
(k + n) Total
(n) lines 9 - 11
(k) lines 6 - 7
(n) lines 3 - 4
(k) lines 1 - 2
Counting sort is usually used when k = O(n), in which case the
running time is (n).
Counting sort is not a comparison sort. It can beats the lower
bound of (n lg n).
Counting sort is stable.
7.9
Radix Sort
Radix Sort
839 657 839 355
720 457 329 720
657 355 657 436
457 839 457 839
436 436 436 657
355 329 355 457
329 720 720 329
The sorting of seven 3-digit numbers.
7.10
Radix sort solves the problem of sorting n d-digit numbers, by
sorting on the least significant digit first, then sorting on the
second-least significant digit, and so on.
The process continues until the numbers have been sorted on all d
digits.
Only d passes are required to sort.
It is essential that the sort on each digit be stable. For example,
we can use the counting sort.
RADIX-SORT(A, d)
//Each element in the n-element array A has d digits.
//Digit 1 is the lowest-order digit,
//and digit d is the highest-order digit.
1. for i 1 to d
2. do use a stable sort to sort array A on digit i
7.11
Lemma:
Given n d-digit numbers in which each digit can take on up to k
possible values, RADIX-SORT correctly sorts these numbers in
(d(n+k)) time.
Proof:
Correctness: by induction on the number of passes. (Exercise
9.3-3)
Time analysis: there are d passes and each pass takes (n+k)
time.

k is the radix of the numbers.

When d is constant and k = O(n), radix sort runs in (n) time,
that is, in linear time.
7.12
Lemma:
Given n b-bit numbers and any positive integer r b, RADIX-SORT
correctly sorts these numbers in ((b/r)(n + 2^r)) time.
Proof:
For a value r b, we can view each key as having d = b/r digits of
r bits each. Each digit is an integer in the range 0 to 2^r - 1, so
that we can use counting sort with k = 2^r - 1. (For example, we can
view a 32-bit word as having 4 digits of 8 bits each, so that b = 32,
r = 8, d = b/r = 4, and k = 2^r - 1 = 255.) Each pass of counting sort
takes time (n+k) = (n + 2^r) and there are d passes, therefore
the total running time is (d(n + 2^r)) = ((b/r)(n + 2^r)).
7.13
For given values of n and b, we want to choose the
value of r, with r b, that minimizes the expression
(b/r)(n + 2^r).
If b < lg n, choose r = b.
This will yield a running time of ((b/b)(n + 2^b)) =
(n).
If b lg n, choose r = lg n.
This will yield a running time of (bn/lg n).
7.14
Radix sort can be used to sort records of information
that are keyed by multiple fields.
For example,
we might want to sort dates by three fields: year,
month, and day. We could sort the information three
times with a stable sort: first on day, next on month,
and finally on year.
Note that radix sort which uses counting sort as the
intermediate stable sort does not sort in place. Thus,
when primary memory is at a premium, an in-place
algorithm such as quicksort may be preferable.