3D Graphics on iPhone

Douglass Turner
douglass.turner@gmail.com
@dugla #iphonemeetup3dtalkacton
A talk in three acts ...
• Intro to Computer Graphics

• Computer Graphics in OpenGL

• OpenGL and CocoaTouch Playing Nice

Together
Intro to Computer Graphics

(a trip down the graphics pipeline)

Build and Tesselate Model
Copyright Steve Jubinville
Pose Model
Copyright Steve Jubinville
Aim Camera
Design Appearance. Light Scene.
Copyright © 2006 Holger Schömann
Design Appearance. Light Scene.
Copyright © 2006 Florian Wild
Design Appearance. Light Scene.
Copyright © 2006 Jack Qiao
Render Scene
Cull & clip model triangles to viewing frustrum.
 Rasterize
From models to pixels. Scanline interpolation.

• Depth Comparison
• Apply Appearance - Vertex level. Pixel level.
• Composite Elements
Present Image
Update Model State
Rinse, repeat.
Computer Graphics in OpenGL
OpenGL Cocoa
OpenGL is old school
You are flipping levers on a state machine.
 float foo[]

glPushMatrix() and glPopMatrix()

Data Structures are for wimps.

OpenGL is a dragster. Not a Prius. Just buckle up ...
... and enjoy the ride, baby!
 iPhone 3GS by the Numbers
(via AnandTech http://bit.ly/1pQaJO)

~7M triangles / second

~230,000 realtime triangles (30fps)
iPhone display: 320 * 480 = 153,600 pixels
230,000 / 153,600 = ~1.5 realtime-triangles / pixel

;-)
OpenGL websites are a bit different
than Cocoa websites ...
Isn’t that sweet ...
Um... WTF?
Can’t you just feel the love?
Dude. I think my eyes are bleeding.
No worries. safari.oreilly.com is your friend
Hierarchical Modeling. Transformation Stacks. Coordinate Frames.

Pose Model
Pose Model
You are already familiar with hierarchical modeling and coordinate frames.
 [self.parentView addSubview:childView];

or ...
for (UIView *v in self.containerView.subviews) {

CGRect bounds = [v convertRect:v.bounds toView:scrollView];

if (CGRectIntersectsRect(bounds, scrollView.bounds)) {

NSLog(@"View %d is visible", v.tag);
} else {

NSLog(@"View %d is hidden", v.tag);
}

} // for (self.containerView.subviews)

You are already familiar with hierarchical modeling and coordinate frames.
- (void)setTransform:(CGAffineTransform)newValue {

// Scale along x-axis only
CGAffineTransform constrainedTransform =
CGAffineTransformScale(CGAffineTransformIdentity, newValue.a, 1.0);

newValue.a = constrainedTransform.a;
newValue.b = constrainedTransform.b;
newValue.c = constrainedTransform.c;
newValue.d = constrainedTransform.d;

[super setTransform:newValue];

}

You are already already familiar with transformations.

 HelloTeapot Demo

Source: http://github.com/turner/HelloTeapot
HelloTeapot Code Walkthrough
http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 // Position Camera
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix();

// Position Model
glPushMatrix();

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 Pose Model
// Position Camera
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix();

// Position Model
glPushMatrix();

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 Light Scene
// Position Camera
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix();

// Position Model
glPushMatrix();

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 // Position Camera
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix(); Aim Camera ...

// Position Model
glPushMatrix();

Dude, where is the camera ...?

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
camera teapot

wTc wTt

world
 cT t
camera teapot

glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(_openGLCameraInverseTransform);
 cT t
camera teapot

-1
(wTc) wTt

world

cT t = cT w * wTt
cT t = cT w * wTt
 // Position Camera
glMatrixMode(GL_MODELVIEW);

cT t = cT w * wTt
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix();

// Position Model
glPushMatrix();

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += new_teapot_indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
- (void)placeCameraAtLocation:(M3DVector3f)location
target:(M3DVector3f)target
up:(M3DVector3f)up;

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 Render
float teapot_vertices [] = {
0.0663, 0.1178, 0.0,
0.0672, 0.1152, 0.0,
0.0639, 0.1178, 0.0178043,
...
};

float teapot_normals[] = {
-0.987635, -0.156768, 0,
-0.902861, -0.429933, 0,
-0.953562, -0.156989, -0.257047,
...
};

short indicies[] = {
// howmany vertices in vertex strip
26,
// vertex strip indices
1122, 1243, 1272, 1242, ... ,1283, 1199,
...
};

http://github.com/turner/HelloTeapot/blob/master/teapot.h
 Render

glEnableClientState(GL_VERTEX_ARRAY);
glVertexPointer(3 ,GL_FLOAT, 0, teapot_vertices);

glEnableClientState(GL_NORMAL_ARRAY);
glNormalPointer(GL_FLOAT, 0, teapot_normals);
glEnable(GL_NORMALIZE);

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
 Render
// Position Camera
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLoadMatrixf(_openGLCameraInverseTransform);

// Position Light
glPushMatrix();
glMultMatrixf(_cameraTransform);

glEnable(GL_LIGHT3);
glLightfv(GL_LIGHT3, GL_DIFFUSE, spotLight);

glPopMatrix();

// Position Model
glPushMatrix();

JLMMatrix3DSetZRotationUsingDegrees(rotation, -45.0f);
JLMMatrix3DSetScaling(scale, sx, sy, sz);
JLMMatrix3DMultiply(rotation, scale, concatenation);
glMultMatrixf(concatenation);

// Render
for(int i = 0; i < num_teapot_indices; i += indicies[i] + 1) {

glDrawElements(GL_TRIANGLE_STRIP, indices[i], GL_UNSIGNED_SHORT, &indices[i+1]);

} // for (num_teapot_indices)

glPopMatrix();

http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
http://github.com/turner/HelloTeapot/blob/master/Classes/GLViewController.[hm]
invert( camera_transform )

red_light
green_light
blue_light

push()
camera_transform
white_spotlight
pop()

push()
teapot_transform
draw()
pop()
You’ll want to understand this stuff if you intend
to do AR* apps like all the cool kids.

*Augmented Reality - See http://layar.com

Design Appearance
Mipmapped textures are your friend.

They will become your workhorse for

visual complexity.
Texture Everything
Copyright © Douglass Turner

glGenTextures(1, &_name);
glBindTexture(GL_TEXTURE_2D, _name);

glTexEnvf(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

glTexImage2D(GL_TEXTURE_2D, 0, glColor, _w, _h, 0, glColor, glFormat, data);

glGenerateMipmapOES(GL_TEXTURE_2D);

Method
initWithTextureFile:mipmap:

Class
github.com/turner/HelloTexture/blob/master/Classes/TEITexture.[h,m]
 HelloTexture Demo

Source: http://github.com/turner/HelloTexture
 Two Words

Texture. Atlas.
t

Texture Space
t

Texture Space
HelloParticleSystem Demo

texture atlas hacks

OpenGL ES 2.0 == GLSL
Shaders Take it to Another Level Copyright © Douglass Turner
Shaders Take it to Another Level Copyright © Douglass Turner
 ZBrush Modeling Daniel Lieske
Shaders Take it to Another Level
 ZBrush Modeling Daniel Lieske
Shaders Take it to Another Level Material Design Ralf Stumpf
 ZBrush Modeling Daniel Lieske
Shaders Take it to Another Level Material Design Ralf Stumpf
Shaders Take it to Another Level Copyright: Stephen Molyneaux
GLSL is cool.

Sadly, OpenGL ES 2.0 (iPhone 3Gs) does not

support hardware anti-aliasing.

Stick with mipmapped textures until you really

know what you are doing.

Or don’t care about aliasing artifacts ...

1 sample per pixel 4 jittered/filtered samples per pixel
Remember the iPhone Imperitive

Your app will be beautiful.

OpenGL and Cocoa Touch Playing Nice Together
 MVC

OpenGL is M. Keep V dumb. Lots of chatter

between M and V.

• GLGravity | http://bit.ly/2dS8YI
• Jeff LaMarche OpenGL ES Template | http://bit.ly/2IvZtV
Touch Sequence & Accelerometer
HelloParticleSystem Demo

Particle system. Touch. Acceleration

Touch Sequence
Initialize model state
touchesBegan:withEvent:

Evolve model state

touchesMoved:withEvent:

Clean up model state

touchesEnded:withEvent:

touchesCancelled:withEvent:
Accelerometer
- (void)accelerometer:(UIAccelerometer*)accelerometer
didAccelerate:(UIAcceleration*)acceleration {

// Compute "G". Use low-pass filter to attenuate instantaneous acceleration.
self.accelerationValueX =
acceleration.x * kFilteringFactor + self.accelerationValueX * (1.0 - kFilteringFactor);
self.accelerationValueY =
acceleration.y * kFilteringFactor + self.accelerationValueY * (1.0 - kFilteringFactor);
self.accelerationValueZ =
acceleration.z * kFilteringFactor + self.accelerationValueZ * (1.0 - kFilteringFactor);

// ParticleSystem particles live in flatland. Use x and y compoments of "G".
[ParticleSystem setGravity:CGPointMake(self.accelerationValueX, self.accelerationValueY)];
}

UIAccelerometerDelegate
+ (void)setGravity:(CGPoint)gravityVector {

// Normalize 2D “G” vector. Only care about direction.
float length = sqrtf(gravityVector.x * gravityVector.x + gravityVector.y * gravityVector.y);
ParticleSystemGravity.x = gravityVector.x / length;
ParticleSystemGravity.y = gravityVector.y / length;

// Flip y-component of gravity vector to be consistent with screen space coordinate
// system used in ParticleSystem.
ParticleSystemGravity.y = -(ParticleSystemGravity.y);

}

ParticleSystem - Class State


// dv = dv/dt * dt
float dv_x = (ParticleSystemGravity.x * gravityScaleFactor * dt);
float dv_y = (ParticleSystemGravity.y * gravityScaleFactor * dt);
particle.velocity =
CGPointMake(particle.velocity.x + dv_x, particle.velocity.y + dv_y);

// dx = dx/dt * dt
float dx = particle.velocity.x * dt;
float dy = particle.velocity.y * dt;

// x = x_old * dx
particle.location =
CGPointMake(particle.location.x + dx, particle.location.y + dy);

ParticleSystem - Instance State

CoreAnimation:
[myView setNeedsDisplay] & [myView drawRect]

OpenGL:

- (void)startAnimation {
animationTimer = [NSTimer scheduledTimerWithTimeInterval:animationInterval
target:self
selector:@selector(drawView)
userInfo:nil
repeats:YES];
}
 iPhone is an Expression Platform

You Must See:

Stanford cs 193p Lecture 17 Video | On iTunes: http://bit.ly/PUrGj
Creating New Expressive Social Mediums on the iPhone
Ge Wang | CCO & CTO | Smule
iPhone is a general purpose sensing platform

• Touch
• Sound
• Proximity
• Cloud
• Camera
• INU (almost)
• Geography
iPhone encourages plausible, hyper-real,
extensions of human experience.

Be interesting. Be brilliant.
Resources
• @dugla - search on #iphonemeetup3dtalkacton
• http://theelasticimage.posterous.com
• This presentation on SlideShare: http://bit.ly/4wC4Mu
 Douglass Turner
@dugla #iphonemeetup3dtalkacton
douglass.turner@gmail.com

Copyright © Douglass Turner