UNIT 6

LIMITATIONS OF ALGORITHM POWER

Algorithms have the power to solve the problems thrown at them. 1.But there are some problems that do not have any algorithms. 2. There are some problems for which there are algorithms but their complexity is not of the polynomial order (ex. Exponential complexity algorithms like tower of Hanoi). 3. There are some problems that can be solved in polynomial order but there is a lower bound on such algorithms (i.e whatever you try the complexity of the algorithm you write for a problem cannot go lower than this lower bound)

11.1 LOWER-BOUND ALGORITHMS

Every problem that you try to solve generally has a lower bound on the complexity that you can achieve in your algorithm i.e you cannot write an algorithm whose complexity can be less than lower bound specified. There are many types (classes) of lower bounds Trivial Lower Bound Looking at the problem statement you can specify any algorithm that you write will have a complexity more than a specific lower bound. Generally whatever algorithm you write has to process all the inputs you provide and write all the outputs that are expected. So lower bound would be equal to at least the value of input and output. Ex 1: If there are n items then an algorithm that finds the permutation of all n items will have a complexity of n!. So complexity of your algorithm (n!) i.e lets say I have 1 mobile phone,1 psp,1 ipod,1 ipad. If I want to find out number of ways of arranging these items on a desk, the algorithm has to at least compute the 4*3*2*1=24 possibilities (your algorithm might have more complexities more instructions) but even if none of the over heads are present in your algorithm , it has to list 24 possibilities. So complexity of your algorithm will be atleast 24 i.e 4! i.e n! n n-1 Ex 2: Let P(x)=anx +an-1x +.......a0.if you want to write an algorithm to evaluate the polynomial when say x=10,n=8 and the coefficients are a= 2 1 4 8 3 5 2 6 so your polynomial will be : 2*10 +1*10 +4*10 +8*10 +3*10 +5*10 +2*10+6 whatever algorithm you write to solve this polynomial, you have to at least go through the array of coefficients (a0 to an). So minimum complexity is (n). Ex 3: Matrix multiplication 2 2 Each matrix has n*n=n elements so whatever algorithm you write has to at least read 2 matrices of n 2 2 elements and output a resultant matrix of n elements. The complexity of your algorithm will be atleast n so 2 lower bound on any matrix multiplication is (n ) Ex 4:Travelling Salesman Problem Some lower bounds are meaningless/impossible to achieve, one such example is travelling salesman problem. In a travelling salesman problem we have a 2-dimensional array of distance Between cities and the output is a n+1 path traversed by salesman. So the lower bound is at least reading the 2 distance between the cities so lower bound is (n ). However point to note is even if some brilliant 2 mathematician comes up with a solution you still cannot go below (n )
8 7 6 5 4 2

Information-theoretic Arguments
Trivial lower bound arguments try to generate lower bound based on the number of input or output. There are some approachs which generate lower bound based on the amount of information the algorithm time to guess the has to produce. For example the number guessing game we saw method-2 takes number. So number of decisions (yes/no) decide the lower bound in this case. Such an approach is called

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
Information Theoretical Approach that generates information theoretical lower bounds. The structure generated by the algorithm are called decision-trees.

Adversary Argument
Lets assume we are playing the number game and the opponent is a cunning guy. Though he says he has chosen a number ,he has not chosen any specific number. As we keep asking him questions he tries to maximize his chances and might change the number he had chosen to trick you to ask more questions. Let s play the game with both the methods we listen in the Huffman code METHOD 1: (non-optimal method) Lets assume adversary had chosen 8 Q1: have you chosen 8? A:No (he bluffs and he changes his number from 8 to 7) Q2:Have you chosen 7? A: No(he bluffs and he changes his number from 7 to say 6) Q3: Have your chosen 6? A: no (he bluffs and he changes his number from 6 to 5) . . . th Finally we corner him with 8 question Q8:have you chosen 1? A: Yes (he is an honest cheater so he will accept finally) So we had to ask him 8 questions. METHOD 2: (optimal method) In this method even if he keeps changing his number we corner him in 3 questions. So lower bound for this algorithm is Such a method of deriving lower bound is called adversary. is the lower bound because, no Argument method: actually method 2 is the most optimal method and trials. matter what you try cannot come up with an algorithm which guesses any number less than

PROBLEM REDUCTION
Lower Bound for a problem can be found using problem reduction method. Ex :- If P is a problem whose lower bound is not known. But Q is another problem whose lower is known. If we can express problem Q in terms of P and the algorithm for P solves the problem Q then we can safely say lower bound of Q applies to P. Example for problem reduction P:Euclidian minimum spanning tree problem Given n points in the Cartesian plane, construct a tree of minimum total length whose vertices are the Given points. (Basically construct a spanning tree) WE WANT TO FIND A LOWER BOUND OF THIS PROBLEM Q: Element uniqueness problem Given n elements x1,x2,x3,.......,xn. Check if the elements are unique. Now these elements x1,x2,x3,.......,xn are Converted to 2D elements as (x1,0),(x2,0),(x3,0)......,(xn,0) Create a minimum spanning tree using these vertices. If any spanning tree gas an edge of 0 length then it Means the number are not unique. Ex 1: if the set was S={1,2,3,4,6,2} Then we convert this set to 2d elements as:(1,0),(2,0),(3,0),(4,0),(6,0),(2,0) Create spanning trees. While creating spanning trees you will notice an edge from (2,0) to (2,0) which is of zero length. From this we can conclude that the set S={1,2,3,4,6,2} is not a unique set. So basically we solved uniqueness problem (Q) using spanning trees (P) So, since we can solve one problem using other, the complexity of Q be applied to P. Since complexity of q is nlogn. So complexity of minimum spanning is also nlogn

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6 3

11.2 DECISION TREES

Lets take an example of comparing 3 numbers to find the minimum of the 3 numbers. Ex:-9,4,2 Solution:- We would generally do the following 1. Is 9<4 No. So between 9 and 4, 4 is smaller 2. Is 4<2 No. So between 4 and 2, 2 is smaller If we are to draw/represent the above conclusion in the form of tree we would represent yes

9<4

No

Yes

4<2

No

2
So 2 is smallest Suppose the order of numbers was Ex:2,4,9 Solution:-The tree would have been Yes

2<4

No

Yes

2<9

No

So suppose we write the numbers as a,b,c, then we can write a decision tree as below

a<b

a<c

b<c

In whatever order the numbers are input (9,4,2 or 2,4,9 or 4,2,9 or 4,9,2 etc) the comparison flow will go through one of the branches. So the above tree takes care of all possibilities to find minimum number between 3 numbers. Some points to observe from the above example

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
1. Number of leaves is at least equal to the possible outcomes (ex: In the above example either a could be smallest or b could be smallest or c could be smallest. So 3 possibilities. So number of leaves should be at least 3) 2. All leaves represent the outcomes 3. All intermediate nodes represent the comparison operations/decision nodes.

AIM:Aim of this topic is to find how many comparisons need to be done before a result is obtained (in worst case scenario) If you see the above decision tree, if for example a is the smallest element then we do 2 comparisons (a<b and a<c) before arriving at this conclusion. This number 2 (i.e 2 comparisons) also indicates one more thing: the height of the tree You will see that height of the tree is also 2. So we can rephrase the below statement: ->Number of comparisons done to find minimal elements out of 3 elements is 2. We can rephrase the statement as: ->the height of the tree to find minimal elements out of 3 elements is 2. We can further refine the statement and say: ->the height of the tree to obtain one of the 3 leaf nodes is 2. In general we can state the same as: ->If h is the height of the tree and there are l leaf nodes then ..............(*) h (this is actually property of a binary tree) Now what we are trying to say is: If there are l possible outcomes (either a is smallest or b is smallest or c is smallest) Then the number of comparisons to be done before reaching to one outcome would at least be h= Lets apply the above logic to the comparison of 3 numbers: th There are 3 leaves/possible outcomes (4 leaf c is repeated so we can ignore) = =2 So So height of tree in worst case is at least 2 (i.e we need to do at least 2 comparisons before we decide the last element)

DECISION TREES FOR SORTING ALGORITHM

Selection sort:Selection sort tries to search for the minimum element and put it to the left side nd It then searches for 2 smallest element and puts it next to the smallest element Ex: 9,4,2 First check 9<4 no Next check 4<2 So 2 is smallest. Move 2 to leftmost i.e. 2,4,9 ...............(1) nd rd next check 2 and 3 element 4<9 yes so 4 is second smallest element. i.e 2,4,9 is the list in increasing order if we try to draw a decision tree for this it would look as below: elements 9,4,2

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
942 yes

5
no 942 No replace 2 in leftmost position. 2 is the smallest element out of 9 will go to 2s place

9<4

4<2

no

yes

4<9

2,4,9

In general we can write the decision tree for selection sort as below: abc

Yes Yes no

a<b
yes

no

a<c
Yes no no yes no

b<c

b<c

yes

b<a

a<c

b<a

a<b<c

a<c<b

c<a<b

b<a<c

b<c<a

c<b<a

Salient points: *There are 6 leaf nodes Each leaf node gives one possible ordering of 3 numbers. *There are 3 numbers so 6 possible outcomes=3! So if there are n numbers then there are n! outcomes/leaf nodes *Height of the tree is 3 i.e to obtain an outcome (leaf node) you have to take 3 decisions Ex: if a<b and a<c and b<c only then the order of numbers is abc and it took 3 decision to come to the solution. *We have seen previously that height of tree is h log l l->number of leaf nodes So if we have l nodes/outcomes/leaf nodes then in the worst case we have to do at least h comparisons and this h value can be calculated using just l value as log l We have seen that l=n! ie number of leaf node=6=3!=n! (number of elements that we were comparing=3) Hence h

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
comparisons. So in worst case you will do at least LHS: in case of selection sort example in worst case we do 3 comparisons before reaching the leaf. h=3 .....................(1) =3 ...............(2) RHS: log 3!=log 6= We see that we do at least 3 comparisons ............(3) Cworst(n) From Stirlings formula log2 (2n(n/e)n) 1/2 n n =log2 (2n) n (1/e) n -n 1/2 1/2 =log2 n e n (2) Expanding log n -n 1/2 1/2 =log2 n +log2 e +log2 n + log2 (2) =nlog2 n nlog2e + log2 n + log2 (2) n log2n .................(4) (You can cross check by substituting different values for n) Substituting (4) in (3) we get Cworst(n) nlog2 n What this means is whatever algorithm you write for sorting, sorting algorithm(your algorithm) will have a complexity of nlog2n. Basically you cannot write a sorting algorithm which has a complexity less than nlog2n (ANY SORTING ALGORITHM)

DECISION TREES FOR SEARCHING A SORTED ARRAY

We will consider binary search. Lets assume we have array A[]={1,4,7,9} Lets assume I want to search 10. The comparison are as below mid= = =1 A[mid]= A[1] num=10 Is num> A[1]? Yes .......... 1 comparison Move to second half ie {7,9} low=2 high=3 mid= = = =2 A[mid]=A[2]=7 Is num >A[mid] ? yes .......... 2 comparison 10>7 Move to right half of 7 ie {9} low=3 high=3 mid= = =3

A[mid]=A[3]=9 Is num > A[mid] ? ......... 3 comparison 10>9 It took 3 comparisons to decide the number does not exist in the list We can write a decision tree for it as follows:

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6 7
<

4
<

>

>

<

> 10 failure

>9

Number of leaves is decided based on the input elements. Height of the tree depends on the number of elements. NOTE: If n is the number of elements, For sorting: Number of leaf nodes is n! Height of the tree will be log2n For binary search: Number of leaf nodes is n+1 Height of the tree = log2(n+1) In general we can write the decision tree for the elements as:

<

A[1] A[0]
<

>

A[2]

>

<A[0]

(A[0],A[1])

(A[1],A[2] )

<

A[3]

> 10 failure

(A[2],A[3])

<A[3]

From the tree we observe the following: *If you try to search an element (ex:h) which is present in the array {1,4,7,9} you will hit a non-leaf node. *Leaf nodes represent failure scenarios ie when you are searching for an element not present in the list In this the worst case is when you search for element between {A[2],A[3]} or >A[3] i.e if you are searching for say 8 or 13 etc you will have to traverse 3 decisions (height of tree) before you can reach a conclusion whether the element is present or not.

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6 8

Now there are 4 elements (n) {1,4,7,9}.We see that the tree has 5 leaves (n+1) ] Using our * equation ie h [ Height (ie Cworst(n) log2 (leaves) ie Cworst (n) SUMMARIZING DECISION TREE TOPIC So what exactly have we done in this topic? *We have exploited the property of binary tree that if h-> height, l->leaves of trees then h *We remodelled all our problems such that all leaves indicate the solutions so number of leaves gives all possible solutions. *We remodelled all our non-leaf nodes as comparison So height of the tree gives the maximum comparisons done which also means it is the worst case for any algorithm because you are doing maximum comparisons. So height of tree gives the Cworst(n) and number of leaves gives all possible solutions logn(leaves) gives height. Which means if I know the number of leaves/ possible solutions. I need to do just log n(leaves) which gives the complexity of the worst case scenario. NOTE: (BINARY SEARCH) For some cases you do 2 comparisons. For some cases you do 3 comparisons. So for the algorithm as whole, we do 3 comparisons in the worst case. For 4 elements, we need at least 3 comparisons in worst case scenario. If A={1,4,7,9} Tree has 5 leaves (l=5) [(n+1) outcomes] =3 h So at least 3 comparisons has to be done for worst case which becomes the algorithms lower bound.

11.3 P,NP,NP-Complete
P-Polynomial algorithm A problem is said to be a P problem if we can write algorithms which can solve the problem in polynomial time. Ex: Searching a number in the list of number. The above problem can be solved by algorithms like linear search in polynomial time. Ex: Sorting a set of numbers in increasing order The above problem can be solved in polynomial time using algorithms like merge sort. 2 3 All these algorithms have complexities in polynomial time (n,n ,nlogn,n )

NP-Nondeterministic Polynomial Algorithm

A problem is said to be NP problem if: Some array gives a possible solution if you can determine if it is a possible solution in polynomial time. However if someone asks you to write an algorithm to obtain the solution in polynomial time, we may not be able to write one. Ex: Subset problem I have a set of numbers say A={1,2,3,4,5,6,7,8,9} and I need to find a subset of the above numbers that add upto say 19. Solution: For the above problem if someone says the solution is {6,4,9} We can easily verify if the above numbers add upto 19 or not. But if someone asks to come up with the subset of numbers using an algorithm that solves in polynomial time, we may not be able to do so.

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
Ex: Travelling Salesman Problem Given n cities connected to each other. Find a path between the cities such that the total distance travelled is less than say P Solution: If someone gives a path we can verify in polynomial time if the path satisfies the requirement P But if someone asks us to write an algorithm that comes up with solution in polynomial time then we may not able to do so.

NP-Complete:
An algorithm solves a problem in a polynomial time if its worst case efficiency belongs to O(p(n)) where p(n) is the polynomial of problems input size. Problems that can be solved in polynomial time are tractable and those that cannot be solved in polynomial time are intractable. Decision problems are problems with yes/no answers. Class P is a class of decision problems that can be solved in polynomial time by deterministic algorithms. This class of problems is called polynomial. Some problems are not decision problems in their natural form but can be reduced to decision problems. Ex: Instead of asking about minimum number of colours need to colour a graph so that no two adjacent vertices have the same colour, we can ask if there exists such a colouring of a graphs vertices with more than m colours for m=1,2,.......... Not every decision problem can be solved in halting time( eg: halting problem) Many of the decision problems can be computationally difficult to solve but checking if a proposed solution solves the problem is computationally easy. A non-deterministic algorithm is a two stage procedure that takes as input an instance of a decision problem and does the following: 1.Non-deterministic(guessing stage):An arbitrary string S is generated that can be thought of as a candidate solution for the instance I (though it need not be a solution). 2. Deterministic (verification stage): A deterministic algorithm takes both S and I as input and output yes if S represents a solution to I. If S is not a solution then it returns no or doesnt halt at all. A non-deterministic algorithm is said to be non deterministic polynomial if time efficiency of its verification stage is polynomial. Class NP is the class of problems that can be solved by non-deterministic polynomial. P is subset of NP. A decision problem D1 is said to be polynomially reducible to a decision problem D2 if there is a function t that transforms instances of D1 to instances of D2 such that: 1.t maps all yes instances of D1 to yes instances of D2 and all no instances of D1 to no instances of D2. 2.t is computable by a polynomial time algorithm. A decision problem D is said to be NP-Complete if: 1.It belongs to class NP. 2.Every problem in NP is polynomially reducible to D. Showing that a decision problem is NP-Complete can be done in two steps: 1. Show that the problem is in NP i.e a randomly generated string can be checked in polynomial time to determine whether or not it represents a solution to the problem 2. Show that every problem in NP is reducible to the problem in question in polynomial time. This can be done by making use of the transitivity of polynomial reduction. Hence we need to show that a known NP-complete problem can be transformed to the problem in question in polynomial time.(ie we are showing that a NPcomplete problem (to which all other NP problems can be reduced) itself can be reduced to the problem in hand)

POLYNOMIAL- TIME REDUCIBILITY AND NP-HARDNESS

We say that a language M defining some decision problem, is NP-hard if every other language L in NP is polynomial-time reducible to M. In more mathematical rotation, M is NP-hard if, for every L NP, L........>M. If a language M is NP-hard and it is also in the class NP itself, then M is NP-Complete. Thus an NP-Complete

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore

UNIT 6
problem is in a very formal sense, one of the hardest problems in NP, as far as polynomial-time computability is concerned. For if anyone ever shows that an NP-complete problem L is solvable in polynomial time then that immediately implies that every other problem in the entire class NP, is solvable in polynomial time. For, in this case, we could accept any other NP language by reducing it to L and then running the algorithm for L. In other words if anyone finds a deterministic polynomial-time algorithm for even one NP-complete problem then P=NP

10

A non-deterministic algorithm is a 2 stage procedure that takes as input an instance I of a decision problem and does the following: 1.Non-deterministic (guessing)stage: an arbitrary string S is generated that can be thought of as a candidate solution for the instance I (though it need not be a solution) 2.Deterministic (verification stage): a deterministic algorithm takes both S and I as input and outputs yes if S represents a solution to I. If S is not a solution then it returns no or doesnt halt at all. A non-deterministic algorithm is said to be non-deterministic polynomial if time efficiency of its verification stage is polynomial. Class NP is the class of problems that can be solved by non-deterministic polynomial. P is subset of NP.(since we ignore the string S generated by the non-deterministic stage and use the algorithm that solves it in the deterministic stage) If P were equal to NP then all problems would have polynomial time algorithms to solve them. It is still being researched as to whether P=NP.

Niharika Kumar, Asst Prof

Dept of ISE,RNSIT Blore