Computers and Fluids 183 (2019) 148–159

Contents lists available at ScienceDirect

Computers and Fluids

journal homepage: www.elsevier.com/locate/compfluid

A fast algorithm for Direct Numerical Simulation of turbulent

convection with immersed boundaries
D. Angeli a,b, E. Stalio c,∗
a
DISMI - Dipartimento di Scienze e Metodi per l’Ingegneria, Università di Modena e Reggio Emilia, via Amendola 2, Reggio Emilia 42122, Italy
b
Centro Interdipartimentale EN&TECH, Università di Modena e Reggio Emilia, via Amendola 2, Reggio Emilia 42122, Italy
c
DIEF - Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, via Vivarelli 10, Modena 41125, Italy

a r t i c l e i n f o a b s t r a c t

Article history: A parallel algorithm is presented for the Direct Numerical Simulation of convection flows in open or
Received 31 December 2017 partially confined periodic domains, containing immersed cylindrical bodies of arbitrary cross-section.
Revised 22 January 2019
The governing equations are discretized by means of the Finite Volume method on Cartesian grids.
Accepted 1 March 2019
The method presented includes a triperiodic Poisson solver employed irrespective of the actual
Available online 2 March 2019
boundary shape and a second order accuracy for the computational domain, including the near wall re-
Keywords: gions, when walls are defined as immersed boundaries.
Direct Numerical Simulation The numerical solution of the set of linear equations resulting from discretization is carried out by
Free convection means of efficient and highly parallel direct solvers. Verification and validation of the numerical proce-
Immersed Boundary Method dure is reported in the paper, for laminar and turbulent pipe flow, and for the case of flow around an
array of heated cylindrical rods arranged in a triangular lattice. The formal accuracy of the method is
demonstrated in laminar flow conditions, and DNS results in turbulent conditions are compared to avail-
able literature data, thus confirming the favorable qualities of the method.
© 2019 Elsevier Ltd. All rights reserved.

1. Introduction
While the continuous increase in available computing resources
as well as novel mathematical, algorithmic and coding techniques
has allowed DNSs to be performed at very high Reynolds numbers
[1] or with extremely refined grids [2], DNSs are often limited to
flows in very simple domains like parallel channels, pipes or flat
plates, mainly due to the limitations of some of the most widely
used discretization methods for the DNS (spectral methods) and to
the numerical errors that often affect the resolution of the discrete
equations on unstructured grids. The search for parallel and accu-
rate methods for the DNS of flows in complex geometries is still
a relevant topic of investigation in the field of turbulent flow and
heat transfer.
Approaches based on the representation of arbitrary domains
in the frame of a Finite Difference or Finite Volume Cartesian
grid, commonly known as Immersed Boundary Methods, are good
candidates given the easy mesh generation procedure and the
straightforward Cartesian parallelization. Stemming from the origi-
nal development of the Immersed Boundary Methodby Peskin [3],
a variety of different methodologies have been developed to sim-
∗
Corresponding author.
E-mail address: enrico.stalio@unimore.it (E. Stalio).
ulate flow over bodies that, in general, do not conform to the
computational grid. These include methods such as the immersed
interface method [4], the Cartesian grid method [5] and the
cut-cell method [6]. Most of these variants are listed and briefly
described in the review by Mittal and Iaccarino [7]. The main
drawback of many of these approaches is in the accurate represen-
tation of boundary conditions close to irregular boundaries, where
the increase in local truncation error can affect the global accuracy
of spatial discretization.
In more recent years the problem of the accuracy of spatial
schemes and of the representation of the boundaries has been con-
sidered and in good part resolved for finite difference [8–11] and
finite element [12,13] methods, also at high order of accuracy, but
to the authors’ knowledge it is difficult to find immersed boundary
methods which, in a Finite Volume context are capable of preserv-
ing the second order accuracy also close to the boundaries.
In this work, a novel methodology is presented for the Di-
rect Numerical Simulation of fully-developed, buoyancy-induced
flow and heat transfer in open or partially confined periodic do-
mains containing prismatic solid bodies of arbitrary cross-section
but aligned with one grid direction. Fluid domains can therefore be
built from single or many non overlapping two dimensional shapes
by orthogonal extrusion. Examples of suitable domain shapes are:
fully developed or axially homogeneous straight channels of ar-
bitrary cross section (pipes, ducts of triangular cross section,...)

https://doi.org/10.1016/j.compfluid.2019.03.002
0045-7930/© 2019 Elsevier Ltd. All rights reserved.
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159 149

including multiply connected sections (flow about tube bundles 2.2. Momentum and energy equations
where pipe sections are arbitrary) and, by application of wall
boundary conditions on all boundaries, three-dimensional closed A formulation of momentum equations, excluding sources, is
cavities containing internal heaters.  
∂ ui ∂ ui ∂p ∂ 2 ui
The continuity, Navier-Stokes and energy equations are dis- ρo + uj =− +μ (2)
cretized by means of the Finite Volume method on Cartesian grids. ∂t ∂xj ∂ xi ∂x j∂x j
Grids are uniform along the homogeneous direction x1 and can
Temperature is solved for in the energy equation
be non-uniform along x2 and x3 . The representation of arbitrar-  
ily irregular cylindrical boundaries on Cartesian grids is an ex- ∂T ∂T ∂ 2T
ρ oc p + ui =λ + σ, (3)
tension of the method developed by Barozzi et al. [14] for pure ∂t ∂ xi ∂ xi ∂ xi
diffusion problems. Setting boundary conditions on irregular solid
boundaries involves the definition of a 5-point computational sten- where σ represents a generic source term, expressed in W/m3 .
cil where boundary segments intersect the stencil arms. The dis- For statistically steady flows, thermally and hydrodynamically
cretization method locally uses Finite Differences in order to accu- fully developed in the x1 direction, governing Eqs. (1)–(3) can be
rately enforce conservation laws in the Finite Volume framework arranged to allow for the representation of all variables, including
of this approach. The variables on the modified stencil are mapped temperature and pressure through streamwise periodic quantities,
on the global grid, by means of a linear operator determined by see for example Ref. [15] where also buoyancy effects are consid-
geometrical features and boundary conditions. The overall accuracy ered. This is in view of the computational efficiency of the numer-
of the method is virtually preserved, as well as the computational ical code.
efficiency of the Cartesian approach. For the statistically steady case under scrutiny which is fully
Validation is done on three different test cases: the formal ac- developed along direction x1 , a time and superficial average is de-
curacy of the method is demonstrated against the analytical solu- fined for the generic scalar field f and indicated by angular brackets
tion of a laminar pipe flow. Accuracy at the boundary is further as-  
tb
sessed by comparing laminar results of a triangular array of heated 1
< f > ( x1 ) ≡ f (x, t ) d dt (4)
rods against previous studies. Finally, a Direct Numerical Simula-  (tb − ta ) ta 
tion is performed for the case of a turbulent pipe flow at Reτ =
180, confirming the favorable qualities of the present method and where  is the surface of a cross-flow plane (perpendicular to x1 ,
its adequacy for turbulent, fully-developed flow simulation in non- the direction of full development) while [ta , tb ] is a sufficiently
rectangular domains. large time interval for < f > to be independent of time. By aver-
A computational fluid dynamics method is presented includ- aging Eq. (2) for i = 1 and taking the x1 derivative of the result it
ing an Immersed Boundary algorithm which is meant for a Finite is found that the time average of the pressure field must be linear
Volume framework and whose second order accuracy is demon- in x1
strated up to the walls. This algorithm does not implement any d< p >
iterative procedure for adjusting a forcing term, instead it is based =γ (5)
dx1
upon the solution to small linear system of equations at the im-
mersed boundaries, see Section 3.2.1. Its parallel implementation for some constant quantity γ which is negative for flows in the
is based upon a domain decomposition method along the homoge- positive x1 direction. A linear function p(x1 ) can be defined with
neous direction x1 and is very efficient, see Appendix A. The idea respect to a fixed point x01 along the x1 -axis
is to decouple the solution to the Poisson problem along x1 and 
p(x1 ) = < p >(x01 ) + γ
 x1 − x01 (6)
to adopt a solution strategy for the viscous and the thermal dif-
fusion terms which are implicit only in the orthogonal plane. As which allows to split the pressure into a linear part 
p, and a fluc-
this is the plane where the mesh is often stretched for appropri- tuation p :
ately solving boundary layers, stability limits imposed on timestep
p(x1 ) + p (x, t ),
p( x , t ) =  (7)
by the discrete form of the diffusion term are typically not very
restrictive. where < p >(x1 ) = 0.
The form of the momentum equations to be solved is then
2. Mathematical formulation  
∂ ui ∂ ui ∂ p ∂ 2 ui
ρo + uj =− +μ − γ δi1 (8)
The methodology implements a discrete form of the in- ∂t ∂xj ∂ xi ∂x j∂x j
compressible Navier-Stokes and energy equations for hydro-
In the form of Eq. (8), periodic boundary conditions may be en-
dynamically and thermally fully-developed flows. All the fluid
forced along the streamwise direction on all variables, including
properties are assumed constant, the rate of heating due to vis-
the fluctuating pressure p .
cous dissipation is neglected. Fully developed flow along the x1 -
The momentum equation is written in terms of dimensionless
direction is specifically considered for the formulation of govern-
quantities as
ing equations, but the method presented here does not set specific
requirements for boundary conditions in directions orthogonal to ∂ u∗i ∂ u∗ ∂ p∗ 1 ∂ 2 u∗i ω∗
+ u∗j ∗i = − ∗ + + ∗ δi1 , (9)
the mean flow. ∂t ∗ ∂xj ∂ xi Reτ ∂ x j ∂ x j
∗ ∗ 
2.1. Continuity equation where the last term equals the wet perimeter divided by the cross
sectional area and it represents the ensemble average of the pres-
Under the above hypotheses, the continuity equation reads: sure drop along the streamwise direction. In Eq. (9), the friction
Reynolds number Reτ is defined as:
∂ ui
=0 (1)
∂ xi Reτ =
uτ Lref
, (10)
ν
where ui indicates the ith component of the velocity field and the

repeated index notation is implied. where the velocity scale is uref = uτ = τw /ρ o.

150 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159

2.3. Normalized form of the energy equation

Uniform heat flux boundary conditions are enforced at the solid

boundaries. Alternatively, temperature can be specified at the walls
solving a slightly different set of equations, see the method in
Ref. [16]. As in the hypothesis of assigned heat flux, the fully de-
veloped time-averaged temperature profile in the axial direction is
linear, the temperature slope in x1 may be evaluated from an en-
ergy balance
q ω
a= (11)
m˙ c p
In Eq. (11), q indicates the imposed wall heat flux over perime-
ter ω of surface ; the sign convention associates a positive heat
flux q > 0 with the heating of the fluid. The mass flow rate m˙ is
defined in the standard way

m˙ ≡ ρ ou1 d (12)

The cross flow area  as well as its perimeter ω are supposed in-
dependent of the streamwise coordinate x1 , this is also required by
the current implementation of the method presented.
Fig. 1. Computational grid G and set of intersection points B.
Given the linear behavior of temperature along x1 , periodicity
is enforced on the fluid excess temperature θ , defined as
where q is a general source term and might depend upon position
θ (x, t ) ≡ [T (x, t ) − < T >(x1 )], (13)
and time, is used in the discussion for the presentation of the nu-
where < θ > is independent of the streamwise coordinate x1 . From merical method. Specific algorithms for the solution of the Navier-
Eqs. (13) and (11) Stokes equations are shortly presented in Section 3.2.4.
∂T ∂θ ∂ 2T ∂ 2θ
=a+ ; = (14) 3. Discrete form of the equations over non-rectangular
∂ x1 ∂ x1 ∂ x21 ∂ x21 domains
the energy Eq. (3) becomes
  The general methodology for obtaining an algebraic set of
∂θ ∂θ ∂ 2θ
ρ cp
o
+ ui =λ − ρ o c p au1 (15) equations for each Cartesian Control Volume in a Finite Vol-
∂t ∂ xi ∂ xi ∂ xi ume framework is well known [18] and not repeated here. When
where the generic source term σ in Eq. (3) is set to zero while implemented over Cartesian grids, the Finite Volume method is
term ρ o cp au1 at the right-hand side of Eq. (15) follows from the simple, fast and rather accurate; it can be extended for the consid-
normalization of temperature [17]. It represents thermal advection eration of more complex domains via an Immersed Boundaries ap-
along the streamwise direction x1 due to averaged temperature proach. The methodology developed here represents an extension
variation along x1 . The first component of the local velocity vec- to convective heat transfer of a methodology developed elsewhere
tor u1 depends upon position and time. Using the same reference [14] for thermal diffusion problems.
quantities as for the momentum equation and
3.1. Stencils, intersection points and Control Volumes
q Lref
Tref = (16)
λ The Immersed Boundary Methodis described in two dimensions
for temperature, the energy equation is written in its dimension- for clarity, the three-dimensional code used in the results section
less form as is obtained by the mere extension to the third, homogeneous di-
∂θ ∗ ∗ ∂θ
∗
1 ∂ 2θ ∗ 1 ω∗ u∗1 rection of the two-dimensional theory reported here. Despite the
+ u = − (17)
∂t∗ i
∂ xi
∗ Reτ Pr ∂ xi ∂ xi
∗ ∗ Reτ Pr ∗ u∗m presentation is two-dimensional, volume averaged quantities and
surface fluxes rather than surface averages and line fluxes are dealt
And the following is the system of differential equations with in the following.
∂ u∗i A Cartesian grid of points distributed on a rectangle which con-
=0 (18)
∂ x∗i tains the physical domain is defined and named G. The grid is
in general non-uniformly spaced. Two families of orthogonal grid-
∂ u∗i ∗
∂ u∗i ∂ p∗ 1 ∂ 2 u∗i ω∗ lines derive from G. The set of intersection points which result from
+ u = − + + (19)
∂t∗ j
∂xj∗ ∂ xi
∗ Reτ ∂ x j ∂ x j
∗ ∗ ∗ the superimposition of the grid lines to the geometry of an arbi-
trarily shaped physical domain is named B, see Fig. 1.
∂θ ∗ ∂θ ∗ 1 ∂ 2θ ∗ 1 ω∗ u∗1 Two types of stencil and Control Volume are defined, these are
+ u∗i = − (20)
∂t ∗ ∂ xi
∗ Reτ Pr ∂ xi ∂ xi
∗ ∗ Reτ Pr ∗ u∗m represented in Fig. 2(a) and (b). Stencils of type g are based on
grid G and are not influenced by the shape of the domain while
which, complemented with an appropriate set of boundary condi-
stencils of type b conform to the domain geometry. When a sten-
tions, is numerically solved in this paper for the Direct Numerical
cil b falls close to the domain boundary, it includes points of in-
Simulation of fully-developed turbulent flow and heat transfer in
tersection while excluding outer points. Control Volumes built on
direction x1 . Asterisks in (18)–(20) are dropped in the following
g-stencils are indicated by  while Control Volumes whose size
text to ease notation. A model differential equation of the form:
and aspect ratio are based on stencil b are signaled by an overbar
∂ϕ ∂ϕ ∂ 2ϕ and named  ¯ . Far from the boundaries stencils b and g and Con-
+ ui = +q (21)
∂t ∂ xi ∂ xi ∂ xi trol Volumes  ¯ and  coincide. Control Volumes always keep a
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159 151

Fig. 2. Local stencils g (a) and d (b).

Fig. 3. Resizing (a) and geometry of the discretization Control Volume(b).

rectangular shape, and metrics of the cells falling near the bound-
ary are conveniently rescaled according to the relative position      
∂ϕ
of cell center and boundary, see Fig. 3(a) and (b), therefore the d
¯ + (uϕ ) · n dᾱ = (∇ϕ · n ) dᾱ + q d
¯
union of all Control Volumes of type  ¯ which are contained in 
¯ ∂t ᾱ ᾱ 
¯

the physical domain do not cover the whole domain volume, see (22)
Fig. 2(b).
in case ϕ represents a constant density times the jth component
Utilization of a (for example) lexicographic ordering of the set
of the velocity field, then Eq. (22) forms the momentum balance
of Cartesian Control Volumes defined about stencils of type g will
equation in the jth direction and becomes non-linear.
lead to the development of an efficient algorithm, easily amenable
The advective and diffusive terms of the generic transport
to a parallel implementation. As stencils g do not carry information
Eq. (22) in two dimensions can be approximated as follows
on the position of the domain boundary, it would be impossible in

this case to set the correct boundary conditions. The methodology 
4
ᾱ j 
presented introduces a linear mapping between stencils of type g (∇ϕ · n ) dᾱ   ϕˆ j − ϕˆ 0 ≡  s · ϕˆ (23)
ᾱ δx j
and b so that the discrete form of equations on Control Volumes j=1
¯ is in terms of quantities defined on the grid G and stencils g. In  
4
this way, the computational advantages of Cartesian grids is pre- (uϕ ) · n dᾱ  ŭ j ϕ̆ j ᾱ j ≡ h · ϕ̆ (24)
served and boundary conditions are set correctly also on irregular ᾱ j=1
boundaries.
where vector ϕ ˆ ≡ (ϕˆ 0 . . . ϕˆ 4 ) is the second-order approximation to
the volume average of ϕ over  ¯ and the four adjacent Control Vol-
3.2. Discrete form of the equations on Control Volumes umes while vector ϕ̆ ≡ (ϕ̆1 . . . ϕ̆4 ) contains the second-order ap-
proximation to the superficial average of ϕ over sides of  ¯ . Vol-
The transport equation for the continuous scalar function ϕ in- ume and surface averages over  and α built around stencils of
tegrated on the generic Control Volume  ¯ of external surface ᾱ is type g will be indicated by ϕ and ϕ ˜.
152 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159

In Eqs. (23) and (24) vector s ≡ (s0 . . . s4 ) contains metric coeffi- With reference to Fig. 4(a) and (b), the index k identifies the
cients. Vector h ≡ (h1 . . . h4 ) contains flow rates across Control Vol- node on the opposite side of k with respect to the center of the
ume boundaries at a specific time. Consistently with the second- stencil, node 0.
order spatial approximation of the method, the midpoint rule for The derivative of function ϕ in k along direction ηk can be ap-
surfaces is used in the advective and diffusive terms, and the mid- proximated using nodes 0, k, k as follows. Taylor-expansion of ϕ
point rule for volumes is used in the unsteady and the source along η and about the point j where j = 0, k, k is
term. Linear differentiation and interpolation are used to recover    
∂ϕ 1 ∂ 2ϕ
the algebraic counterparts of the diffusive and advective terms. The ϕ j  ϕk + ηj + η2j j = 0, k, k (31)
semi-discrete equation ∂η k
2 ∂η2 k

∂ ϕˆ 0 ¯ then the normal derivative of the variable ϕ at the boundary can

 + h · ϕ̆ =  s · ϕˆ + q¯ 
¯ (25) be expressed as a linear combination of the nodal values of the
∂t
variable itself in the points 0, k, k of the stencil; since the stencil
approximates Eq. (22) to the second order accuracy in space. In
is of type b a hat is added to the variable:
Eq. (25) q¯ is the volume-averaged source term.  
The algebraic set of equation which is consistent to the second- ∂ϕ 
 pk, j ϕˆ j (32)
order accuracy with Eq. (22) is obtained by applying a temporal ∂η k j=0,k,k
discretization to Eq. (25). When the combination of Crank-Nicolson
with second order Adams-Bashforth for the diffusive and the ad- where:
  
vective terms is used, the following discretized equation results 0= pk, j , 1= pk, j η j , 0= pk, j η2j (33)

3 1 j=0,k,k j=0,k,k j=0,k,k
(n ) (n ) (n−1 )
m · ϕ
ˆ = t − h(n ) · ϕ̆ + h(n−1) · ϕ̆
2 2 The following 3 × 3 linear system is then obtained for the generic

(n ) intersection node k:
+ s · ϕ
ˆ + q¯ 
¯ (26)     
1 1 1 pk,0 0
where η0 0 ηk  pk,k = 1 (34)
t
η02 0 ηk2 pk,k 0
(n )
m · ϕ
ˆ ≡ ϕˆ 0(n ) 
¯ −  s · ϕˆ (n) (27) as geometry is hypothesized to be fixed in time, coefficients
2
pk,j which depend on geometry only are calculated in the pre-
3.2.1. Linear map for the computational domain boundaries processing phase of the numerical procedure. Once coefficients pk,j
As mentioned above to preserve a structured grid approach also are known, Eq. (32) can be substituted into (30) to obtain:
for irregularly bounded domains, a linear map is introduced so that 
unknowns on stencils of type b can be expressed through values ak ϕˆ k + bk pk, j ϕˆ j = ck (35)
j=0,k,k
defined on stencils g, plus a non-homogeneous term accounting for
the boundary conditions. Hence, in vector notation: Eq. (35) is written in terms of an approximation to function ϕ at
the boundary, which is denoted by ϕˆ k together with ϕˆ 0 and ϕˆ k
ϕˆ = E ϕ + g (28)
while all coefficients ak , bk , pk,j are known. Hence a general equa-
In general, coefficients of matrix E and vector g depend on in- tion can be written between vector ϕ ˆ and vector ϕ which is valid
tersections of the computational stencil g with the domain bound- for stencils far from boundaries and also enforces the boundary
aries as well as the specific conditions on the boundary. For fixed conditions (30) in intersection points
boundaries and stationary boundary conditions, temporal incre-

4
ments of the unknown are expressed by ϕ ˆ = E ϕ; for stencils (ai δi, j + bi pi, j )ϕˆ j = ci + di ϕi i = 0, . . . , 4, (36)
in the interior of the domain Eq. (28) reduces to ϕ
ˆ = ϕ. j=0
Eqs. (26) and (27) can be set in terms of variables defined on
where for the ith inner point (i = k) coefficients take the following
stencils g as follows
 value ai = 1, bi = 0, ci = 0, di = 1, and Eq. (36) reduces to ϕˆ i = ϕi .
 3 (n ) 1 (n−1 ) For intersection points dk is set to zero dk = 0, in addition coeffi-
m · E ϕ (n ) = t (n ) − h(n ) · ϕ̆ + h(n−1) · ϕ̆
2 2 cients pk, j = 0 for j = 0, k, k . Eq. (36) holds true also for stencils
 having multiple intersection points. Eq. (36) can be reformulated
+  s · E ϕ (n ) + g + q¯ 
¯ (29)
in matrix notation as

The computation of surface averages ϕ̆ in Eq. (29) is presented Aϕ

ˆ = Dϕ + c (37)
in Section 3.2.2. The procedure for determining E and g is de- For example, in the case of Fig. 4(a) where node 2 of the stencil
scribed in the following. As the method uses point-wise values at is on the boundary, matrices A, D, c take the form:
the boundary of the computational domain, rather than fluxes and ⎛ 1 0 0 0 0
⎞
volume averages, the Finite Volume method for solving the set of
governing equations is paired with Finite Differences in the region ⎜ 0 1 0 0 0 ⎟
A = ⎜b2 p20 b1 p21 a2 + b2 p22 b2 p23 b2 p24 ⎟ (38)
of the domain boundary. Validation presented in Section 4 shows ⎝ ⎠
that such mixed approach preserves spatial accuracy at the bound- 0 0 0 1 0
aries, provided that the cell centers of the Cartesian mesh do not 0 0 0 0 1
fall excessively close to the boundary. ⎛1 ⎞
Robin-type boundary condition for the unknown ϕ at the inter-
section node k is ⎜ 1 ⎟
D=⎜ 0 ⎟ (39)
∂ϕk ⎝ ⎠
ak ϕk + bk = ck , (30) 1
∂ηk 1
where ηk denotes the direction normal to the boundary in k and
the variable ϕ k here indicates the local value of ϕ in point k. cT = ( 0, 0, c2 , 0, 0 ) (40)
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
Fig. 4. Local discretization stencils for intersections with k = 2 (a) and k = 3 (b).
For stencils having a single, different k or multiple intersection
nodes A, D, c change accordingly.
It is now possible to formulate the expressions for E and g in
(28):
E = A−1 D, g = A−1 c
As already mentioned, coefficients in E and g depend on the do-
main shape and boundary conditions (30) and are calculated once
at the start of the numerical procedure.
The Finite Difference approach close to the boundary is paired
to the Finite Volume method of the domain interior. It is clear
from the discussion above that resorting to the finite difference
allows for accurately setting boundary conditions when the do-
main boundary is not parallel to the Cartesian mesh grid-lines. In
Section 4.1 this approach is demonstrated to preserve the second
order accuracy of the method over the whole flow domain for both
uniform and non uniform mesh. The second order accuracy of the
whole method is ensured by the midpoint rule for surfaces and
volumes. Implementation of similar methods at higher orders of
spatial accuracy would require a scrupulous distinction between
point-wise and space averaged values and the use of specific oper-
ations of integration and deconvolution, see Ref. [19]
An issue remains to be addressed for the implementation of
the discrete Eq. (29), this is the determination of vectors h and
ϕ̆, which form a second-order approximation to advective fluxes.
3.2.2. Advective fluxes
Implementation of advective fluxes require that Eq. (24) which
is reported also here for clarity
 
4 from the only information available, i.e. the values at the intersec-
(uϕ ) · n dᾱ  ŭ j ϕ̆ j ᾱ j
ᾱ j=1
is represented numerically within a second order approximation
on both Cartesian and distorted stencils (type-b stencils).
The generic surface average ϕ̆ j is expressed as a function of the
volume averaged ϕ value over the central Control Volume ϕ 0 , and
the value of the jth neighbor ϕ j by the following second-order ap-
proximation
x j ϕ 0 + x 0 ϕ j
ϕ̆ j =
x j + x 0
When the stencil is crossed by the domain boundary and j = k,
since the ratio between length of the kth stencil arm and distance
of the centroid from the kth Control Volume side is invariant with
respect to the stencil deformation, the interpolation weights of dis-
torted and Cartesian stencils are the same (see Fig. 3(b)).
153
The determination of the normal velocity at the k-face, ŭk , is
very similar but requires an additional interpolation, because of
the staggering of variables (see Section 3.2.4).
(41) 3.2.3. Volumetric flow rate across deformed Control Volumes
Depending on the relative distance between the centroid of the
Control Volume and the intersection point k, three possible con-
figurations need to be considered, see Fig. 5. In case (a), the node
containing the required value of velocity lies within the domain.
Although this node is no more located on the Control Volume
boundary, due to the deformation induced by the presence of the
boundary itself, it can nevertheless be used to determine the value
of the normal velocity component ŭk . Such an estimation can be
performed by linear extrapolation from the value of the node itself
and the value at the intersection point k (which equals zero at a
solid boundary).
In case (b), the node containing the required velocity value lies
outside the domain and is therefore excluded from the calcula-
tions. In this situation, a linear interpolation must be performed
between the value at the symmetric node of the velocity grid, with
respect to the centroid, and the value at the boundary. Such a pro-
cedure is valid, however, only when at least one of the two nodes
associated with the stencil direction is contained within the do-
main of integration.
Otherwise, a situation similar to that represented by case (c)
arises: the normal flux on the Control Volume side can not be de-
termined from the nodal values of the velocity grid, since no “ac-
tive” node falls in the vicinity of the Control Volume itself. The
calculation of the normal velocity at the interface is then operated
tion nodes. In the case of a closed domain, this corresponds to im-
pose zero velocity on the Control Volume side.
3.2.4. Specific algorithms for the Navier-Stokes equations
Staggering. The numerical technique presented here makes use of
staggered grids. In three dimensions this involves four Cartesian
lattices where the union of Control Volumes bounded by grid-
lines of the “main grid” perfectly covers the computational domain,
aside from immersed boundaries. The three “staggered grids” are
(42) instead displaced by half spacing along x, y and z, respectively.
Over each of the four grids, volume averaged values and surface
averages over the six faces of each Control Volume are defined for
all variables. In a numerical code whose spatial accuracy is sec-
ond order, the volume averaged values can be identified also with
point-wise values defined in the volume center, and the same is
valid for surface averages.
154 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
Fig. 5. Procedure for the estimate of the normal velocity component near the boundary for the energy equation.
If by one side the use of a staggered mesh makes implementa-
tion and bookkeeping more complex, and it involves many inter-
polations, on the other side the use of different grids for the so-
lution of the energy equation and of each component of momen-
tum allows for a straightforward evaluation of the momentum and
energy fluxes across Control Volume faces. But most importantly,
it avoids the occurrence of the so-called “checker-boarding” effect
[20] on the variable distributions, which typically arises when a
single mesh is used.
Pressure-Velocity decoupling. It is well known that in the imple-
mentation of numerical algorithms for incompressible flows, the
continuity equation can be interpreted as a constraint for the com-
ponents of the velocity vector and procedures are employed for de-
coupling the calculation of the pressure field from velocity. In the
numerical code presented here, a Projection Method, introduced by
Chorin [21] and later developed by Gresho et al. [22], is imple-
mented.
Like all fractional step methods, also Projection Methods involve
the numerical solution to a Poisson equation per time step, this
implies an additional computational cost. The details of the algo-
rithms selected for the fast, parallel solution to the Poisson equa-
tion, are shortly presented in the Appendix A.2.
4. Validation and computational performance of the numerical
procedure
4.1. Laminar pipe flow
A challenging task for Immersed Boundary Methods on a Carte-
sian mesh is to represent curved boundaries at the prescribed for-
mal order of accuracy. In this regard, the laminar pipe flow is a
convenient test case, as it allows for a direct comparison against
the analytical solution.
A series of tests are carried out at Reτ = 1. In laminar condi-
tions the exact solution for the dimensionless streamwise velocity
component reads as follows:
Reτ
u (r ) = (1 − r 2 ) (43)
2
Both uniform and non-uniform grids are tested, the non-uniform
grids have the following hyperbolic tangent distribution:
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159 155

Fig. 7. Error distribution on the streamwise velocity component for a 32 × 32 and

Fig. 6. Assessment of second order accuracy for laminar pipe flow. Squares: error a 64 × 64, uniform mesh. In the 64 × 64 case, error maxima are observed in four
maxima on the streamwise velocity field (L∞ ), circles: error on the centerline ve- locations close to the solid wall. This effect is avoided by using a suitably stretched
locity; solid line: reference ax2 line. mesh, see Fig. 6.

  i−1 1 
tanh δ nx − 2 is used, both the centerline velocity and L∞ error decrease with
xi = R δ , (44) x2 , demonstrating that the present technique ensures second-
tanh 2
order spatial accuracy, provided that the mesh is well-suited for
where δ = 2 in the present case. the domain shape, i.e. free of local singularities which occur when
Accuracy results are reported in Fig. 6. For the uniform mesh, the intersections between the boundary and grid-lines lie too close
the centerline velocity approaches the analytical value display- to the cell centers. For simple cases like the turbulent pipe flow
ing second-order accuracy. Local error maxima are independent of described in the following, validation in laminar regime is also a
the mesh size, but depend only on the relative position of the quick, reliable test to check the grid quality before running a full
boundary and grid lines (see Fig. 7). When a nonuniform mesh DNS.

Fig. 8. Triangular rod bundles in laminar conditions: maps of streamwise velocity ux for different P/D ratios and for Reτ = 1, Pr = 1.
156 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
Fig. 9. Triangular rod bundles in laminar conditions: friction factor and Nusselt
number for different P/D ratios and for Reτ = 1, Pr = 1.
4.2. Flow about triangular bundles in laminar conditions
To further assess the accuracy of the present immersed bound-
ary technique, the computational procedure is validated in the case
of flow and heat transfer in a uniformly heated, infinite triangular
lattice of rod bundles in laminar conditions, for Reτ = 1 and Pr = 1.
The ratio between pitch and rod diameter P/D is varied in the test.
Fluid flow and heat transfer are fully developed along x1 and peri-
odic conditions are enforced for both the cross-flow directions, and
no-slip conditions and uniform heat flux are imposed at all the im-
Fig. 10. Pipe flow validation results for Reτ = 180 and Pr = 0.71.
mersed walls. In dimensionless terms, this translates to conditions
ui = 0, i = 1, . . . , 3 (45)
and
∂θ
= −1 (46)
∂η
at the solid walls.
Fig. 8 reports maps of the streamwise velocity component,
while Fig. 9 compares friction factor and Nusselt number values
against correlations reported by Rehme [23]. An excellent agree-
ment is observed between the integral values calculated and exist-
ing correlations.
4.3. Turbulent pipe flow
The suitability of the present technique for the application to
the DNS of turbulent flows is tested in the turbulent pipe flow
case. Reference data for the fluid flow are taken from Eggels
et al. [24], while DNS data for the thermal fields are retrieved by
Piller [25].
The Direct Numerical Simulation (DNS) performed represents
fully-developed forced convection conditions at Reτ = 180, corre-
sponding to a bulk Reynolds number Reb ≈ 2650; the Prandtl num-
ber is set to Pr = 0.71. The fluid is uniformly heated at the wall.
Boundary conditions for the velocity and temperature fields have
the same dimensionless form as given in Eqs. (45) and (46). The
domain length in streamwise direction is Lx = 10 R, where R is the
pipe radius. The computational grid includes 2563 cells, with uni-
form spacing. This corresponds, in wall units, to an average dis-
tance between solid wall and first computed value y+ w ≈ 0.3, a
x+ = 7, (r+ θ )max = 2. The timestep selected is t + = 0.018. A
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
Fig. 11. Pipe flow DNS at Reτ = 180: instantaneous contours of the streamwise velocity component on a surface at y+ = 10, and isosurfaces of Q.
constant dimensionless time step of t = 10−4 is used. Such pa-
rameters comply with current practice in DNS.
Fig. 10 shows flow and thermal field statistics computed on 100
realizations collected over 50 dimensionless time units. The agree-
ment with reference data is deemed as very satisfactory. In order
to provide further evidence of the capability of the method to per-
form resolved DNS of turbulent flow along curved boundaries, in-
stantaneous contours of the streamwise velocity component on a
surface at a distance y+ = 10 from the wall are displayed in Fig. 11,
together with isosurfaces of the Q-criterion (as defined by Hunt
et al. [26]). Typical features of turbulent wall bounded flows such
as low- and high-speed streaks can be observed, along with a mul-
tiplicity of near-wall structures.
4.4. Scalability and computational efficiency
The computational method presented here is implemented in
parallel through a domain decomposition technique along the ho-
mogeneous direction x1 . The parallel implementation is detailed in
the Appendix A. Fig. 12 reports results of a scalability test per-
formed on a small 4-node, 64-core cluster (Intel Xeon®E5-2680v2
2.8Ghz) equipped with an Infiniband network switch, operated by
the High Performance Computing (HPC) center of the University of
Modena and Reggio Emilia. Two different grids are employed for
Fig. 12. Speedup S and number of computational cores N for the current imple-
mentation of the proposed technique.
157
the test, including 643 and 2563 Control Volumes, respectively. For
the mesh size and number of cores considered, the speedup is sub-
linear but monotonically increasing.
The computational performance of the procedure is also as-
sessed in comparison to a conventional unstructured Finite Vol-
ume solver. The general purpose, open-source OpenFOAM toolbox
[27] is selected for the comparison. OpenFOAM is a second-order
implementation of the PISO algorithm [28] on arbitrarily-shaped
polyhedral meshes. One quarter of the cross-section of the 2563
Cartesian grid on which the present method is applied is shown
in Fig. 13(a), along with a similar visualization of the unstructured,
hexahedral grid used with OpenFOAM, see Fig. 13(b). Both grids
were prepared in order to respect typical sizing requirements for
turbulent pipe DNS at Reτ = 180, a summary of the main dimen-
sional features of the two grids is reported in Table 1.
The performance of the two methods is measured in terms of
the cpu time per DoF for the execution of 10 0 0 time steps on
the above-mentioned HPC infrastructure. The number of degrees
of freedom (DoF) of a DNS is defined as the number of grid points
times the number of time steps performed. In order to reason-
ably compare the performances of the present, fully direct method
with the OpenFOAM solver, which is iterative, residual tolerances
at each time step have been set so as to guarantee a cumulative
mass conservation error below 10−12 .
For the proposed methodology, a value of 7.2 μs/DoF is mea-
sured. This is perfectly comparable to the most up-to-date codes
being developed in recent years, see for example Table 4 in Ref.
[29] where a cost of 17.6 μs is reported as the cpu time per DoF
measured for a 8.9 billion point Cartesian mesh. Table 1 shows that
despite the overhead represented by the unused cells lying outside
the cylindrical boundary (about 25% of the total number of vol-
umes, see Fig. 13(a)), the present method is more than five times
faster than the conventional unstructured solver considered.
Table 1
Computational performance of the proposed method, as compared
with a conventional unstructured Finite Volume solver, on the tur-
bulent pipe flow case with Lx = 10R at Reτ = 180. The number of
cells reported in parentheses for the present method denotes the
actual number of elements lying in the fluid region.
N. of cells ×10−6 y+
w,mean x+ μs/DoF
present 16.8 (12.7) 0.3 7 7.2
OpenFOAM® 13.6 0.5 7 40.2
158 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
Fig. 13. Details of the grids employed for testing performances on the turbulent pipe flow case at Reτ = 180: (a) Cartesian grid with immersed boundaries (present method);
(b) Boundary-fitted grid (OpenFOAM).
5. Concluding remarks
An original technique was presented, for the Direct Numerical
Simulation of fully-developed turbulent convection flows in open
or partially confined periodic domains, containing immersed cylin-
drical bodies of general cross-section. The methodology is based
on an underlying second-order Finite Volume discretization of the
governing equations on a staggered, Cartesian grid arrangement.
The representation of immersed boundaries is carried out by defin-
ing locally modified computational stencils (and locally resized
cells) in the vicinity of the intersection points between the grid
lines and the boundary itself. A linear mapping between the vari-
ables defined on these stencils and those pertaining to the main
grid allows for the preservation of the numerical advantages of the
Cartesian approach, while retaining second order accuracy. The dis-
crete transport equations are then solved in parallel, by means of
efficient direct solvers. The method also includes a triperiodic Pois-
son solver for pressure-velocity decoupling, which is employed ir-
respective of the actual boundary shape.
As far as verification and validation are concerned, laminar pipe
flow results carried out with both uniform and non-uniform grids
confirmed the formal accuracy of the method. Further comparison
with the established literature, for the case of laminar flow and
heat transfer around a triangular array of heated rods, indicated
that the boundary reconstruction procedure predicts integral quan-
tities such as wall fluxes and stresses with excellent agreement.
A DNS of a turbulent, non-isothermal pipe flow at Reτ = 180 was
performed, and results were compared with analogous DNS data
available in the literature, showing a very satisfactory agreement,
and confirming the suitability of the present technique for the di-
rect simulation of fully-developed turbulent flow and heat transfer
in cylindrical domains of arbitrarily complex shape.
Finally, scalability and performance tests indicate that the com-
putational cost for degree of freedom associated with the proposed
procedure is indeed small also in comparison with a conventional
unstructured Finite Volume solver, and its parallel efficiency is
overall satisfactory. This suggests that the present technique repre-
sents as a good compromise between computational efficiency and
accuracy, which becomes particularly convenient when medium-
sized DNSs of flows in non-trivial geometries are aimed at.
Acknowledgments
The research leading to these results has received funding from
the Euratom research and training programme under grant agree-
ments No 249337 (THINS) and No 654935 (SESAME). We acknowl-
edge the CINECA award under the ISCRA initiative for the availabil-
ity of High Performance Computing resources and support.
Appendix A. Parallel implementation of the three-dimensional
solver
The three dimensional Immersed Boundaries algorithm is im-
plemented in Fortran and uses MPI for running in parallel. The par-
allel implementation employs a one-dimensional domain decom-
position method where the computational domain is partitioned
in np chunks along the homogeneous direction x, where np is the
number of processes.
A.1. Transport equations
In the solution procedure, the momentum and energy equations
are made explicit in the streamwise direction x; a one-dimensional
domain decomposition is easily implemented along the explicit,
decoupled direction x.
A.2. Poisson solver
The numerical procedure for the solution of the Navier-Stokes
Eq. (9) coupled to mass and energy conservation equations imple-
ments a projection method [30]. Like all fractional-step methods,
it involves the numerical solution to a Poisson equation, which in
general requires a relevant fraction of the computational effort per
time advancement step.
In the most classical theories [22] and implementations, bound-
ary conditions of the Poisson problem depend upon boundary con-
ditions enforced on the velocity field. In recent years, several im-
mersed boundary codes [31] implement triperiodic boundary con-
ditions for the pseudo-pressure, irrespective of the shape and the
conditions set at the boundary of the computational domain. Al-
tough lacking a theoretical framework, this solution is very conve-
nient from both the implementation and the computational point
of views and most importantly, it does not spoil the accuracy of
the solution [32]. Also the numerical algorithm described here uses
this strategy.
A.2.1. Domain decomposition in x √
In the following text the imaginary number is indicated: −1 =
i, while the symbol i indicates an index.
The Poisson problem is written in Cartesian coordinates
∂ 2φ ∂ 2φ ∂ 2φ
+ + = c (x, y, z ) (47)
∂ x2 ∂ y2 ∂ z2
The continuous function c(x, y, z) can be expressed as a Fourier
series
c (x, y, z ) = ci jk eiπ ix eiπ jy eiπ kz (48)
i j k
 D. Angeli and E. Stalio / Computers and Fluids 183 (2019) 148–159
where ci,j,k are the Fourier coefficients of c(x, y, z) and the Poisson
Eq. (47) becomes
   2005;37:239–61.
− φi jk π 2 i2 + j2 + k2 eiπ ix eiπ jy eiπ kz = ci jk eiπ ix eiπ jy eiπ kz
i, j,k i, j,k
(49)
given the uniqueness of the Fourier expansion
ci jk
φi jk = −  (50)
π 2 i2 + j 2 + k2
Eq. (50) is used in the computational code implemented in order to
decouple the three dimensional Poisson equation in one direction
(or more directions) for domain decomposition purposes.
∂ 2 φi ∂ 2φ
(y, z ) + 2 i (y, z ) = ci (y, z ) + φi (y, z )π 2 i2 (51)
∂y 2 ∂z
The algorithm for decoupling a three-dimensional Poisson problem
in nx two-dimensional problems can be sketched as follows
• Compute the forward Fourier transform along direction x of the
right hand side ci,j,k for all indices j, k
• Solve plane by plane a two dimensional Poisson problem where
the right hand is the DFT along x of ci,j,k and the solution matrix
is obtained from subtraction of the quantity (iπ /lx )2 from the
diagonal of the two-dimensional solution matrix
• The solution to the three dimensional Poisson problem is ob-
tained by the backward Fourier transform along direction x of
the plane-by plane solution.
A.2.2. Solution to the Poisson problem in each plane
As described in Section A.2.1, the domain decomposition along
the homogeneous direction x decouples the three dimensional
Poisson problem in nx two dimensional problems. Each of these
is solved by a Matrix Decomposition technique which is derived
from Ref. [33]. The Matrix Decomposition technique employed per-
forms the diagonalization of suitable sub-matrices of the global
matrix that couple the problem in two directions. After the Poisson
equation is further decoupled the Poisson problem is finally de-
composed in a set of nx × ny one dimensional cyclic Poisson prob-
lems. These are discretized with standard Finite Volume techniques
within the second order accuracy and the cyclic, tridiagonal lin-
ear problems derived are conveniently solved by the Thomas algo-
rithm.
References
[1] Lee M, Moser RD. Direct numerical simulation of turbulent channel flow up to
Reτ ≈ 5200. J Fluid Mech 2015;774:395–415.
[2] Yokokawa M, Itakura K, Uno A, Ishihara T, Kaneda Y. 16.4Tflopss direct nu-
merical simulation of turbulence by a fourier spectral method on the earth
simulator. In: Proceedings of the ACM/IEEE SC Conference; 2002. p. 1–17.
[3] Peskin C. The immersed boundary method. Acta Numer 2002;11:479–517.
[4] Leveque RJ, Li Z. Immersed interface method for elliptic equations with
discontinuous coefficients and singular sources. SIAM J Numer Anal
1994;31(4):1019–44.
[5] Patankar SV. A numerical method for conduction in composite materials, flow
in irregular geometries and conjugate heat transfer. In: Proceedings of the 6th
International Heat Transfer Conference, 3; 1978. p. 297.
159
[6] Leveque R. High resolution finite volume methods on arbitrary grids via wave
propagation. J Comput Phys 1988;78(1):36–63.
[7] Mittal R, Iaccarino G. Immersed boundary methods. Annu Rev Fluid Mech
[8] Bonfigli G. High-order finite-difference implementation of the im-
mersed-boundary technique for incompressible flows. Comput Fluids
2011;46(1):2–11.
[9] Gautier R, Biau D, Lamballais E. A reference solution of the flow over a circular
cylinder at Re=40. Comput Fluids 2013;75:103–11.
[10] Bernardini M, Modesti D, Pirozzoli S. On the suitability of the immersed
boundary method for the simulation of high-reynolds-number separated tur-
bulent flows. Comput Fluids 2016;130:84–93.
[11] Gronskis A, Artana G. A simple and efficient direct forcing immersed bound-
ary method combined with a high order compact scheme for simulating flows
with moving rigid boundaries. Comput Fluids 2016;124:86–104.
[12] Frisani A, Hassan Y. On the immersed boundary method: finite element versus
finite volume approach. Comput Fluids 2015;121:51–67.
[13] Krause D, Kummer F. An incompressible immersed boundary solver for mov-
ing body flows using a cut cell discontinuous Galerkin method. Comput Fluids
2017;153:118–29.
[14] Barozzi GS, Bussi C, Corticelli MA. A fast cartesian scheme for unsteady heat
diffusion on irregular domains. Numer Heat Trans B 2004;46:56–77.
[15] Piller M, Polidoro S, Stalio E. Multiplicity of solutions for laminar, fully-devel-
oped natural convection in inclined, parallel-plate channels. Int J Heat Mass
Trans 2014;79:1014–26.
[16] Stalio E, Piller M. Direct numerical simulation of heat transfer in converging–
diverging wavy channels. J Heat Trans 2007;129(7):769–77.
[17] Piller M, Stalio E. Numerical investigation of natural convection in inclined
parallel-plate channels partly filled with metal foams. Int J Heat Mass Trans
2012;55(23–24):6506–13.
[18] Versteeg HK, Malalasekera HK. An introduction to computational fluid dynam-
ics. Harlow: Longman Scientific and Technical; 1995.
[19] Piller M, Stalio E. Compact finite volume schemes on boundary-fitted grids. J
Comput Phys 2008;227(9):4736–62.
[20] Patankar SV. Numerical heat transfer and fluid flow. 1st ed. McGraw-Hill; 1980.
[21] Chorin AJ. Numerical solution of the Navier-Stokes equations. Math Comput
1968;22:754–62.
[22] Gresho PM, Sani R. On pressure boundary conditions for the Navier-Stokes
equations. Int J Numer Methods Fluids 1987;7:1111–44.
[23] Rehme K. Convective heat transfer over rod bundles. Handbook of Single-Phase
Convective Heat Transfer. Kakac S, Shah RK, Aung W, editors. John Wiley and
Sons; 1987.
[24] Eggels J, Westerweel J, Nieuwstadt F. Fully developed turbulent pipe flow: a
comparison between direct numerical simulation and experiment. J Fluid Mech
1994;268:175–209.
[25] Piller M. Direct numerical simulation of turbulent forced convection in a pipe.
Int J Numer Meth Fluids 2005;49.
[26] Hunt JCR, Wray AA, Moin P. Eddies, stream, and convergence zones in turbu-
lent flows. Center For Turbulence Research; Report CTR-S88; 1988.
[27] Weller H, Tabor G, Jasak H, Fureby C. A tensorial approach to computa-
tional continuum mechanics using object-oriented techniques. Comput Phys
1998;12(6):620–31.
[28] Issa R. Solution of the implicitly discretised fluid flow equations by operator-s-
plitting. J Comput Phys 1986;62(1):40–65.
[29] Borrell G, Sillero J, Jiménez J. A code for direct numerical simulation of turbu-
lent boundary layers at high reynolds numbers in BG/p supercomputers. Com-
put Fluids 2013;80(1):37–43.
[30] Gresho PM. On the theory of semi-implicit projection methods for incompress-
ible flow and its implementation via a finite element method that also intro-
duces a nearly consistent mass matrix. part 1: theory. Int J Numer Methods
Fluids 1990;11:587–620.
[31] Laizet S, Li N. Incompact3d: a powerful tool to tackle turbulence prob-
lems with up to o(105 ) computational cores. Int J Numer Methods Fluids
2011;67(11):1735–57.
[32] Lamballais E, Silvestrini J, Laizet S. Direct numerical simulation of a separa-
tion bubble on a rounded finite-width leading edge. Int J Heat Fluid Flow
2008;29(3):612–25.
[33] Babu V, Korpela S. On the direct solution of Poisson’s equation on a non-uni-
form grid. J Comput Phys 1993;104(1):93–8.