Programming Methodologies

A problem can be viewed as a sequence of decisions to be made.

A Brute Force algorithm solves a problem by exhaustively enumerating all the possibilities.
Example: brute force string matching

A Greedy algorithm always make the (seemingly) best choice in each decision. Once a decision has been made, that decision is never reconsidered.
Example: hill climbing; Huffman encoding; Dijkstra's shortest path

A Backtracking algorithm systematically considers all possible outcomes for each decision, just like the brute force algorithms. But backtracking algorithms can detect when an exhaustive search is unnecessary. Therefore, backtracking algorithms can perform much better than brute force ones when solving the same problem.
Example: eight queen; depth first traversal

A Divide And Conquer algorithm divides a problem into several (usually smaller) sub-problems, solves each sub-problem independently, then combines the solutions to obtain the solution to the original problem. Divide and conquer algorithms are often implemented using recursion. And generally the sub-problems are non-overlapping.
Example: quick sort; merge sort

A Dynamic Programming algorithm solves a series of sub-problems to solve the original problem. The series of sub-problems is devised carefully in such a way that each subsequent solution is obtained by combining the solutions to one or more of the sub-problems that have already been solved. All intermediate solutions are kept in a table in order to prevent unnecessary duplication of efforts.
Example: matrix chain product, all pairs shortest path (by Warshall, Floyd and Ingerman)

• Dynamic programming usually is used to solve optimization problems.
• The optimization problems usually have the following properties if dynamic programming techniques can be applied:
  • Simple subproblems: there has to be a simple way of repeatedly and recursively breaking the global optimization problem into subproblems.
  • Subproblem optimality: an optimal solution to the global problem must be a composition of optimal subproblem solutions.
  • Subproblem overlap: optimal solutions to unrelated subproblems can contain subproblems in common. This property keeps the total number of distinct subproblems small (usually is polynomial in the input size).
• The development of a dynamic programming algorithm can be broken into a sequence of the following steps:
  • Characterize the structure of an otpimal solution;
  • Recursively define the value of an optimal solution;
  • Compute the value of an optimal solution in a bottom-up fashion;
  • Construct an optimal solution from computed information.

Dynamic Programming examples:

• Optimal Chain Mattrix Multiplication
  Multiplication of mattrices is associative: (A₁A₂)A₃ = A₁(A₂A₃). But if A₁ is p by q, A₂ is q by r, and A₃ is r by s, then the number of multiplications needed for (A₁A₂)A₃ is p*r*q+p*s*r, while the number of multiplications needed for A₁(A₂A₃) is p*s*q+q*s*r. When p=5, q=4, r=6 and s=2, the multiplication operations needed for (A₁A₂)A₃ is 180, while the multiplication operations needed for A₁(A₂A₃) is only 88.
  Given A₁A₂ ... A_n, how can we place the parenthesis optimally to minimize the number of multiplication operations needed?
  1. The problem can be split to a number of distinct subproblems, each of which is to determine the optimal parenthesization of some expression of A_iA_i+1...A_j. The notation of M[i,j] can be used to denote the minimum number of multiplication needed to compute this expression, and S[i,j] to indicate the top most separator to divide the expression to two sub-expressions.
  2. It is possible to characterize an optimal solution to a particular problem in terms of optimal solutions to its subproblems, i.e., the problem has the subproblem optimality property.
  3. For A_i...A_j, its full parenthesization must have the form of (A_i...A_k)(A_k+1...A_j) for some k such that i <= k <= j. And for whichever k is the right one, (A_i...A_k) and (A_k+1...A_j) must also be solved optimally.
  4. dynamic programming algorithm for chain matrix multiplication:

     input: n - there are n matrices, and n+1 dimention numbers
            p[0..n] where matrix A_i is p[i-1] by p[i]
     output: M[i][j] - the minimum number of multiplication needed
             for expression A_i...A_j;
             S[i][j] - the place to put the top most two parenthesis
     // note that the index, except the one for p,
     //      starts from 1 for simplicity reasons
     Algorithm ChainMatrixMultiplication(int p[], int n)
     for i = 1 to n
         M[i][i] = 0;
         S[i][i] = -1;
     for L = 1 to n-1
         for i = 1 to n-L
             j = i + L;
             S[i][j] = i;
             M[i][j] = M[i][i]+M[i+1][j]+p[i-1]*p[i]*p[j]
             for k = i+1 to j-1
                 if M[i][j] > (M[i][k]+M[k+1][j]+p[i-1]*p[k]*p[j])
                     M[i][j] = M[i][k]+M[k+1][j]+p[i-1]*p[k]*p[j];
                     S[i][j] = k;
     

A Randomized algorithm has an element of randomness involved in solving a problem.
Example: skip list