Models of Computation and Time Complexity

August 3, 2007

An algorithm is a sequence of computational steps which solves a well-dened computational problem. An algorithm receives input of the problem and produces as output the solution of the problem. For example, a sorting algorithm receives a set of numbers and outputs them in sorted order. An algorithm can be described in various ways. For example, an algorithm can be described in any of the programming languages. However, there is no reason to prefer one language over the other in describing the algorithm. Moreover, describing the algorithm in a specic programming language we might have to worry about strict syntactic rules of the language which might distract the reader from the main understanding of the algorithm. Due to all these reasons, it is better to describe an algorithm in English in a way which is very close to the any of these programming languages. This representation of the algorithm will be called its Pseudocode. In this course, we shall always describe an algorithm by writing its pseudocode. (In this chapter we shall describe two algorithms in pseudocode). A step in this pseudocode will usually correspond to an instruction in a high level programming languages. Designing efcient algorithms for various computational problems is one of the fundamental aim of compute science. The term efcient needs to be understood very clearly. We want to design the algorithms which can be executed in least amount of time when implemented on the computers. However, the real time is not a right measure of efciency due to the following reason. For a xed input, an algorithm which took days of running time on computer twenty years back would take a few minutes today and may be few seconds after few years. It therefore makes sense that the number of operations is more suitable measure of efciency of the algorithm. The aim is therefore to design an algorithm which takes least number of operations. Note that an operation in an algorithm would correspond to an instruction in high level language, and maps to a few instructions at assembly language. The number of operations performed by an algorithm to solve a given problem will depend upon the size of the input of the given instance of the problem. There is a difference between size and value of an input. Here one must understand that the size/length of the input is the number of bits required to store the input. We always assume that the input is given in binary format. For example an integer 1013 will need 10 bits in binary representation. In order to calculate time complexity of an algorithm, it is important to specify the underlying model of computation. There are many models of computation and they are equivalent in the sense that a problem which can be solved on one can also be solved by another. However, it is the cost for the basic computational steps where these models differ. In some model, a basic computational step (for example addition/multiplication) can be accomplished in one step whereas in other model it may take many steps depending upon the length of the operands for example. We shall rst describe two models of computation. 1. Logarithmic cost model : the cost of performing arithmetic/logical operation depends upon the size of the operand. For example, adding two numbers of n bits will take cn number of steps and multiplying two numbers using the simple add-and-shift method will take c n2 steps from some constant c > 0. 2. Uniform cost model : the cost of performing arithmetic/logical operation does not depend upon the size of the operand. So in this model, the time complexity of an algorithm is the count of the number of operations performed.

Now we shall describe the model of computation which is the most common model of computation for analyzing time complexity. This model is also very close to the real computers. RAM model : There are the following two features of this model. We shall soon provide justication for these features and show that RAM model is indeed close to the real computers. 1. The entire input resides in the RAM (random access memory). Each element of the input can be accessed in constant time from the RAM. 2. Let the input consists of n items, where each item could either be a basic data types or a combination of a constant number of these basic data types. If operands of an operation are of length c log n bits for a constant c, then the number of steps required for an operation is just constant. For example, sorting n numbers of log n bits each by selection sort will have time complexity bn2 for some constant b. Note : Throughout this course, whenever we have to analyze the running time of an algorithm in RAM model on an input of n items, we shall assume that each data item is of size c log n for some constant c. How realistic is the RAM model ? For most of the computational problems in real life, the input is stored in the internal memory (RAM), and of course the time required to access any data item from RAM is same irrespective of its location in the internal memory. Hence the rst feature of the RAM model is realistic. Let us now discuss the second feature. Before that, let us rst see how does a computer executes a program. A program in high level language is translated by compiler into machine language. In this translation, one instruction in high level language is translated to a constant number of machine language instructions. The CPU reads program which is a sequence of machine language instructions. A typical instruction involves performing some arithmetic/logical operation on its operands. The address of these operands is a binary number and is mentioned as a part of the instruction. The CPU executes this instruction by fetching the operands from the memory and then executing the corresponding operation. The result can further be written into some location specied in the instruction. In order to assign unique address to each location in the memory, it follows easily that if M is the size of the memory in bytes/words, then log2 M bits are necessary for specifying address of a data item stored in memory. In practice, it has always been the case that the length of the operands for arithmetic/logical operation supported by the architecture is at least as much as the number of bits to address a data element from memory. Consider for example, the most recent computer architecture. It has 2-4 GB of RAM, and it supports operation on 64 bits operands. For example, it can add two 64-bit numbers in one step. If input of an algorithm is n then based on the rst feature, it follows that memory size is large enough to accommodate the entire input, that is, n < 4 109 . Now log n = 32, and the architecture even supports operation with operands having length 64 (which is twice the value of log n). Thus the second feature of the RAM model, that an operation on operands with log n bits length takes single step is well supported by real computer as well. In short, the following guidelines should be followed when analyzing time complexity of an algorithm in RAM model. 1. In RAM model, when we analyze an algorithm on input of n basic data types, size of each item is assumed to be log n bits unless mentioned otherwise. And so, each operation can be viewed as an atomic operation (executed in single step) and time complexity analysis requires counting the number of operations. So in this sense, RAM model is uniform cost model. 2. To analyze the problems where the operand size is much more than log n bits, it is better to switch to logarithmic cost model for its time complexity. 3. Numerical computational problem are problems which involve arithmetic operations on numbers. In such applications, it is useful to get a bound on the maximum length of the operands which get generated during the algorithm. Sometimes operands generated during the algorithm may be of much 2

larger size than the input items(numbers). Gaussian elimination is such a numerical problem and its trivial implementation may generate numbers which may require n bits even when the initial matrix has entries with size log n at most. In any such situation, it is not suitable to use RAM model for the time complexity analysis of the algorithm. For algorithms for numerical problems, it is always desirable that to get an estimate of the maximum size of the numbers which may be generated during the algorithm, and if the maximum size does not exceed log n, then it is safe to just count the number of operation for time complexity of the algorithm in the RAM model.

Examples
Let us consider a few examples in order to better understand the two models we described above. We shall design three algorithms for one problem, and the problem is computing nth Fibonacci number. Fibonacci numbers are dened as follows. F (0) = 0, F (1) = 1, and F (i) = F (i 1) + f (i 2)for i > 1

Exercise 0.1 Show that F (i) > abi for some positive constants a, b with b > 1. It follows from the above exercise that the number of bits required for representing nth Fibonacci number is at least cn for some constant c. Algorithm 1 int recursiveb(i) { if i == 0 return 0 else if i == 1 return 1 else return (recursiveb(i-1) + recursiveb(i-2)) } Let T (n) be the number of steps taken to compute F (n) according to above algorithm. It follows that computing F (n) requires computing F (n 1) and F (n 2) recursively and separately, followed by constant number of steps. Hence T (n) > T (n 1) + T (n 2) + c. Comparing this inequality with the formula of Fibonacci number and using the exercise given above, it follows that T (n) is exponential in n. More specically it can be shown that the time complexity is exponential in terms of value of the number n in both the models. Algorithm 2 (iterative algorithm) int iterativeb(i) { if i==0 return 0 else if i ==1 return 1 else { b00, b11; 0; While( i) do { temp b0 + b1; b0 b1; b1 temp; +1 } return b1; } } 3

The above algorithm performs cn steps to compute nth Fibonacci number. So the time complexity in uniform cost model is linear in n and the time complexity in logarithmic cost model is quadratic in n (since addition of two numbers of k bits length takes k steps). Algorithm 3 is based on the following equation. F (i) F (i 1) Hence it follows that F (n) F (n 1) = 1 1 1 0 = 1 1 1 0 F (i 1) F (i 2)
n1

1 0

Although the above equation does not give us any improvement in running time, we can use the idea of repeated squaring to get better running time as follows. Let x be a number and we wish to compute xk . k1 Observe that xk = (xk/2 )2 if k is even and xk = x (x 2 )2 if k is odd. In this manner, it follows that we can compute xk using 2 log n multiplication operations. This approach can be used for computing the (n 1)th power of the 2 2 matrix. So in uniform cost model, the number of steps required will be c log n for some constant c. However, in the logarithmic cost model, the number of steps required to compute nth Fibonacci number will still be c n2 form some constant c . Do it as an exercise. You might have to use Exercise 0.1 and the fact that the number of steps required to multiply two k -bit numbers in logarithmic cost model by add-shift method is quadratic in k . Note : In some future class, we shall present an algorithm which can multiply two k -bit numbers in k log2 3 steps. This algorithm will lead to algorithm in logarithmic cost model which takes nlog2 3 steps to compute F (n).