In the present study, the software of Portable, Extensible Toolkit for Scientific (PETSc) Computation associated with a parallel multilevel iterative solver is embedded in our LES program to enhance the capability of computational power for the three dimensional turbulent flows. The three-dimensional Poiseuille, Couette and lid-driven cavity are conducted for the examination of numerical accuracy at laminar flows. For the moderate Reynolds number, the turbulent Poiseuille flow inside the square duct is simulated for the verification of turbulent model. The results show that the both velocity profiles and turbulence intensities are in line with the benchmark. Moreover, the downstream flow of primary eddy would be enhanced by the fast moving wall. However, as Reynolds number increases higher than 1,000, the location of primary eddy is saturated. Also, the secondary eddies increasing with Reynolds number is confirmed by the present study.

Abstract i
Contents ii
List of Figures iv
List of Tables v
1 Introduction 1
1.1 Introduction . . . . . . . . 1
1.2 Literature Survey . . . . . . 2
1.2.1 Lid Driven Cavity . . . . . 2
1.2.2 Parallel solver . . . . . . 3
1.3 Objective . . . . . . . . . . 4
2 Mathematical Models 5
2.1 Introduction . . . . . . . . 5
2.2 Governing Equations for Large Eddy Simulation . . . . . . . . . . . . 6
2.2.1 The filtering operation . .. 6
2.2.2 The filtered Navier-Stokes equations . . . . . . . . . . . . 7
2.3 Sub-grid Scale Modeling . .. . 7
2.3.1 Dynamic Smagorinsky Model . 8
2.4 Closure . . . . . . . . . . . 11
3 Numerical Solution 12
3.1 Introduction. . . . . . . . . 12
3.2 Grid Generation . . . . . . . 13
3.3 Discretization of the Transport Equation . . . . . . . . . . . . 14
3.3.1 Spatial discretization .. . 15
3.3.2 Temporal discretization . . 17
3.4 The Pressure Poisson Equation . 18
3.4.1 Multilevel Schwarz preconditioned CG iterative Poisson solver 20
3.5 Boundary Condition .. . . . . . 22
3.6 Closure . . . . . . . . . . . . 23
4 Numerical Results 27
4.1 Validation of parallel code .. . . . . . . . . . . . . . . 27
4.1.1 Laminar Poiseuille flow .. . . . . . . . . . . . . . . 27
4.1.2 Laminar Couette flow .. . . . 28
4.2 Algorithm Tuning and Parallel Efficiency for Poisson Solver . . . . . 28
4.2.1 Multilevel Schwarz preconditioned CG iterative Poisson solver 29
4.2.2 Parallel scalability studies . . . . . . . . . . . . . . 33
4.3 Laminar Lid-Driven Cavity flow . . . . . . . . . . . .. . . . 33
4.4 Turbulent Poiseuille flow . . . . . . . . . . . .. . . . 34
5 Conclusions 44

[1] Hsin-Wei Hsu and Feng-Nan Hwang Zih-Hao Wei, and Sheng-Hong Lai and
Chao-An Lin, A parallel multilevel preconditioned iterative pressure Poisson
solver for the large-eddy simulation of turbulent flow inside a duct. Computers
and Fluids,45 (2011) 138-146.
[2] U.Ghia,K.N. Ghia, C.T. Shinl, High-Re solutions for incompressible flow using
the Navier-Stokes equations and a multigrid method, J. Comput. Phys. 48
(1982) 387-411.
[3] R. Schreiberm H. B. Keller, Driven cavity flows by efficient numerical
techniques, J. Comput. Phys. 49 (1983) 310-333.
[4] O. R. Burggraf, Analytical and numerical studies of the structure of steady
separated flows, J. Fluid Mech. 24 (1966) 113-151.
[5] F. Pan, A. Acrivos, Steady flows in a rectangular cavities, J. Fluid Mech. 28
(1967) 643-655.
[6] O. Botella and R. Peyret, Benchmark spectral results on the lid-driven cavity
flow, Comput. Fluids 27 (1998) 421-433.
[7] A. Yu. Gelfgat, Implement of arbitrary inner product in the global Galerkin
method for incompressible Navior-Stokes equations, J. Comput. Phys. 211
(2006) 513-530.
[8] Reima Iwatsu, Katsuya Ishii, Numerical simulation of three dimensional flow
structure in a driven cavity, Fluid Dyn. Res. 5 (1989) 173-189.
[9] Li Q. Tang, TiWu Cheng and Tate T. H. Tsang , Transient solutions for threedimensional
lid-driven cavity flows by a least-squares finite element method,
International Journal for numerical methods in fluids. 21 (1995) 413-432.
[10] T. W. H. Sheu, S. F. Tsai, Flow topology in a steady three-dimensional liddriven
cavity, Compt. Fluids 31 (2002) 911-934.
[11] J. L. Guermond, C. Migeon, G. Pineau and L. Quartapelle, Start-up ows in a
three-dimensional rectangular driven cavity of aspect ratio 1:1:2 at Re = 1000,
J. Fluid Mech. 450 (2002) .
[12] Yuri Feldman and Alexander Yu. Gelfgat, Oscillatory instability of a threedimensional
lid-driven ow in a cube, Physics of fluids 22 (2010) 093602.
[13] C.H. Bruneau, and M. Saad, The 2D lid-driven cavity problem revisited,
Comput. fluids 35 (2006) 326.
[14] A. Fortin, M.Jardak, J. J. Gervais, and R. Pierre, Localization of
Hopfbifurcations in fluid flow problems, Int. J. Number. Methods Fluids 24
(1997) 1185.
[15] F. Auterim, N.Parolini, and L. Quartapelle , Numerical investigation on the
stability of singular driven cavity flow, J Comput. Phys 183 (2002) 1-25.
[16] G. D. Davis, G. D. Mallinson, An evalution of upwind and central difference
approximations by a study of recirculating flow, Comput. Fluids 28 (1967)
643-655.
[17] S. Balay and K. Buschelman and W. D. Gropp and D. Kaushik and
M. G. Knepley and L. C. Mclnnes, PETSc Web page, (2010) < http :
//www.mcs.anl.gov/petsc >.
Bibliography 47
[18] C. G. Speziale. On nonlinear k − ϵ and k − l models of turbulence. J. Fluid
Mech., 178 (1987) 459-475.
[19] T. H. Chiu and L. K. Yeh and C. A. Lin, Explicit algebraic stress modelling
of homogeneous and inhomogeneous flows.Int. J. Numer. Methods Fluids 49
(2005) 817-835.
[20] T. J. Craft and B. E. Launder and K. Suga, Development and application of
a cubic eddy-viscosity model of turbulence, Int. J. Heat Fluid Flow. 17 (1996)
108-115.
[21] B. E. Launder and S. P. Li, On the elimination of wall-topography parameters
from second moment closure, Phys. Fluids. 6 (1994) 990-1006.
[22] J. M. Tsao and C. A. Lin, Reynolds stress modelling of jet and swirl interaction
inside a gas turbine combustor,Int. J. Numer. Methods Fluids. 29 (1999) 451-
464.
[23] M. Germano and U. Piomelli and P. Moin and W. Cabot, A dynamic subgridscale
eddy viscosity model, Phys. Fluids. 3 (1991) 1760-1765.
[24] J. Smagorinsky, General circulation experiments with the primitive equations.
I. The basic experiment, Mon. Weather Rev. 91 (1963) 99-164,3.
[25] D. K. Lilly, A proposed modification of the Germano subgrid-scale closure
method, 4 (1992) 633-635.
[26] U. Piomelli and J. Liu, Large eddy simulation of rotating channel flows using
a localized dynamic model, Phys. Fluids 7 (1995) 839-848.
[27] J. Kim and P. Moin., Application of a fractional-step method to incompressible
navior-stokes equations, J. Comput Phys., 177 (1987) 133-166.
Bibliography 48
[28] Y. Saad, Iterative methods for sparse linear system. Second ed., Second
ed.,Philadelphia: SIAM (2004).
[29] B. Smith and P. Bjørstad and W. Gropp, Parallel multilevel methods for elliptic
partial differential equations, Domain decomposition, Domain decomposition,
(1996).
[30] J. Kim and P. Moin., Application of a fractional-step method to incompressible
navior-stokes equations, J. Comput Phys. 177 (1987) 133-166.
[31] A. George, Nested dissection of a regular finite element mesh,SIAM J. Numer.
Anal. 10 (1973) 345-363.
[32] E. Ng, On one-way dissection schemes,University of Waterloo (1979).
[33] E. Cuthill, Several strategies for reducing the bandwidth of matrices. in Sparse
Matrices and Their Applications, Plenum, New York,edited by D. J. Rose and
R. A. Willoughby, (2004) 157-169.
[34] A. George and J.W.H. Liu, The evolution of the minimum degree ordering
algorithm,SIAM Review. 32 (1989) 1-19.
[35] I.S. Duff and G.A. Meurant, The effect of ordering on preconditioned conjugate
gradients. BIT. 29 (1989) 635-657.
[36] S. Albensoeder, H.C. Kuhlmann, Accurate three-dimensional lid-driven cavity
flow, Journal of Computational Physics, 206 (2005) 536-558.
[37] R. Schreiber, H.B. Keller,Driven cavity flows by efficient numerical techniques.
(1983) 310-333.
[38] A. J. Chorin, Numerical solution of the navier-stokes equations, Math Compute
22 (1968) 745-762.
Bibliography 49
[39] M. Deville, T.-H. Le, Y. Morchoisne, Numerical simulation of 3-D
incompressible unsteady vscous laminar flows, Note on Numerical Fluid
Mechanics, Vieweg, Braunschweig 36 (1992).
[40] Howard C. Elman, Victoria E. Howle, John N. Shadid, Ray S. Tuminaro, A
parallel block multi-level preconditioner for the 3D incompressible NavierStokes
equations, J. Computaional Phys. 187 (2003) 504-523.
[41] S. Gavrilakis, Numerical simulation of low Reynolds number turbulent flow
through a straight square duct, J. Fluid Mech. 244 (1992) 101-129.
[42] J. Kim and P. Moin., Turbulent statistics in fully developed channel flow at
low Reynolds number, J. Fluid Mech. 177 (1987) 133-166.
[43] K. Iwamoto, Y. Suzuki and N. Kasagi, Fully Developed 2-D Channel Flow at
Re = 150 , http : //thtlab.jp/DNS/dnsdatabase.html(2002)
[44] J. R. Koseff and R. L. Street, The Lid-Driven Cavity Flow: A Synthesis of
Qualitative and Quantitative Observations , ASME 106 (1984) 390-398.
[45] K. Goda, A multistep technique with implicit difference schemes for calculating
two- or three-dimensional cavity flows, J. Comput. Phys. 30 (1979) 76-95.
[46] Y. Zang and R. L. Street and J. R. Koseff, A dynamic mixed subgrid-scale
model and its application to turbulent recirculating flows, Phys. Fluids, 5
(1993) 3186-3196
[47] K. M. Singh and J. J. J. R. Williams, Application of the additive Schwarz
method to large scale Poisson problem., Commun Numer Methods Eng. 20
(2004) 193-205.
[48] S. A. Nadeem and P. K. Jimack, Parallel implementation of an optimal twolevel
additive Schwarz preconditioner for the 3-D finite element soltion of
Bibliography 50
elliptic partial differential equations, Int, J. Numer Mech. Fluids, 40 (2002)
1571-1579.
[49] L. C. Dutto, The effect of ordering on preconditioned GMRES algorithm for
solving the compressible Navier-Stokes equations,Int. J. Numer Methods Eng.
36 (1993) 457-497.
[50] A. George and W.P. Tang and Y.D. Wu, Multi-level one-way dissection
factorization,SIAM J. Matrix Anal. appl. 22 (2000) 752-771.
[51] M. Benzi, Preconditioning techniques for large linear systems: a survey,J.
Comput. Phys. 182 (2002) 418-477.
[52] R. Bridson and W.P. Tang, A structural diagnosis of some IC orderings,SIAM
J. Sci. Comput. 22 (2000) 1527-1532.
[53] M. Benzi and D.B. Szyld and A.V. Duin, Orderings for incomplete factorization
preconditioning of nonsymmetric problems,SIAM J.