Nomadik Application Processor

Andrea Gallo Giancarlo Asnaghi

ST is #1 world-wide leader in Digital TV and Consumer Audio

MP3 Portable Digital Satellite Radio Player Digital Car Radio

Set Top Box DVD Player

MMDSP+ inside more than 200 million produced chips

January 14, 2009

ST leader in mobile phone chips

January 14, 2009

Nomadik

Nomadik is based on this heritage providing:

Unrivalled multimedia performances Very low power consumption Scalable performances

January 14, 2009

Best Application Processor 2004

Lowest power consumption Scalable performance Video/Audio quality Cost-effective Nominees: Intel XScale PXA260, NeoMagic MiMagic 6, Nvidia MQ-9000, STMicroelectronics Nomadik STn8800, Texas Instruments OMAP 1611
January 14, 2009

Nomadik
Nomadik is a family of Application Processors
Distributed processing architecture ARM9 + multiple Smart Accelerators Support of a wide range of OS and applications Seamless integration in the OS through standard API drivers and MM framework

January 14, 2009

roadmap

...

January 14, 2009

Some Nomadik products on the market...

January 14, 2009

STn8815 block diagram

January 14, 2009

Nomadik : a true real time multiprocessor platform

SDRAM
Unlimited Space Limited Bandwidth

SRAM (Level 2 Cache for Video)

General Purpose System DMA

ARM926 (L1 + L2)

Master OS Peripherals

Memory Controller

multi-layer AHB bus

RTOS Multi-thread

RTOS (Scheduler FSM) MMDSP+ Video

NAND Flash
Unlimited Space No Bandwidth Mass storage

MMDSP+ Audio
133 MHz, 24-bit

66 MHz, 16-bit

HW Acceleration

Audio Peripherals
January 14, 2009

Video Peripherals

Context & task relation: Logical view

ARM boundary (WinCE & Linux OS type)
GUI, User Applications Multiple Address Space Application engines & servers

Virtual world

User side System side (kernel) Single Address Space

System calls

IO process IO

Drivers & Kernel components

HW/IO process Flat Address Space IO

Physical world SAA boundary

SVA boundary

January 14, 2009

Picture Post Processing

January 14, 2009

Examples of Video Resize filters targeting Mobile TV

Resized: zoom out (preview) Regular: display 1:1 scale Wide: zoom to full height Panoramic crop and zoom to full width Stretch zoom to full width and height

January 14, 2009

Rotate

January 14, 2009

Hardware/Software partitioning
MPEG4 software stack
Sequence ARM SW Group of picture Picture Group of Macro-Block HW/FW Macro-Block Block
Firmware

Handled by Software on the host CPU. High flexibility, low computation OS friendly

Handled by Hardware. No flexibility, high computation

January 14, 2009

MM integration in an open OS
Application (Player,) MM Framework
Audio Source Filter Codec Filter/Codec Video Filter/Codec Audio Sink Video Sink Virtual Media pipeline User side Kernel side PCM if SAA HW Driver (control path) SVA HW Driver (control path)

(plug-ins based)

Integration/Adaptation layer

SAA Audio Codec Video Codec Post-

SVA

Display FrameBuffer

Processing

January 14, 2009

Concurrent execution
t1 t2 t3 t4 t5

ARM

Read & Parse

Read & Parse Audio Decode

Read & Parse Audio Decode Audio Playback

Read & Parse Audio Decode Audio Playback Video Decode Video Display

Read & Parse Audio Decode Audio Playback Video Decode Video Display

SAA

SVA

Video Decode

Video Decode Video Display

January 14, 2009

Power Management support

RUN mode :

~200mA

Full clock speed 64 393MHz

Full clock speed

Doze mode :
ARM in WFI mode PLL1 & 2 OFF, clocks = 32 kHz 32 kHz All peripheral clocks are gated

Standby mode :
ARM in WFI mode (Wait For Interrupt) PLL1 & 2 are running 64-393MHz

All peripheral clocks are gated

Wake up: 3ms

Sleep mode :
ARM in WFI mode PLL1 & 2 OFF, clocks are stopped except the non switchable part @ 32 kHz All peripheral clocks are gated

~mA

Power Save mode: Power Save mode

ARM in WFI mode PLL1 OFF, PLL2 Running 19MHz

Deep -Sleep mode:

- Main part of the logic is no more powered - The non switchable part is clocked @32 kHz

~ A

CPU & Bus clocked at XTAL frequency All peripheral clocks unused are gated

January 14, 2009

SW architecture Linux

Screenshots and pictures

WVGA 800x480 @ 24bpp Philips TPO display

20% CPU

3% CPU

January 14, 2009

Average CPU load

VGA 30fps 2Mbps Xine FFMPEG 10-20% GSTLaunch MPEG4 27% GSTLaunch H264 37% Playbin OpenGL-ES panel carousel demo ARM load 3%
Lauterbach ETM trace H264 VGA 30fps 2Mbps AAC

Lauterbach ETM trace MPEG4 VGA 30fps 2Mbps MP3

+20pp

Khronos compliance ongoing

January 14, 2009

System overview for STn8815

cmp K+ architecture Application Framew ork + Application Framework + Azingo + Google Android + Access ALP + Trolltech Qtopia + OpenMoko + Poky

Full open source Linux 2.6.20 kernel OpenedHand PokyLinux for system test In line with GMAE, Limo and Maemo
Multimedia HCL + SAA HCL + SGA HCL + SVA HCL

Middleware Communication + Connection Manager + Bluetooth + Wi-Fi + Ethernet + Telephony Connectiv ity + SD/MMC + USB Hi-Speed MEMS + Gesture Library + MEMS Driver Security + Cryptoengine driver + IPSec + OpenSSL + Security Graphic User Interface + Matchbox + GTK+ Camera + Camera ISP + ISP Driver OpenGL-ES 1.1 + Clutter + EGL + OpenGL Driver + OpenGL-ES 1.1 + GLES Library GStreamer + GStreamer + OpenMAX-IL + FFMPEG + SAA Block Library Pow er Management + PM framework + Governor + Battery Manager + CPUFreq + Linux DPM + APM + LiMo PM

Kernel and driver level Multimedia Interfaces + ALSA + V4L2 + X.11 + xvideo + EXA + DirectFB File systems + JFFS2 + YAFFS + CRAMFS + RAMFS + MTD Kernel and Driv ers + DMA + I2C + Linux kernel 2.6.20 + MSP + SPI + UART Multimedia Driv ers + SAA Driver + SGA Lib and Driver + SVA Driver

Power Management

January 14, 2009

A/V features at a glance

Video
MPEG4 / H.264 decoder up to VGA 30fps 2Mbps MPEG4 encoder up to VGA 30fps H.263 codec JPEG codec MP3, AAC / eAAC+, WMA G.711, G.723, G.726, G.729 AMRnb / AMRwb Full duplex noise and acoustic echo cancelling Effects, e.g. equaliser, reverberation, spatial sound

Audio

January 14, 2009

Multimedia interfaces
cmp Multimedia Interfaces

Standard Linux interfaces

ALSA

X.11

DirectFB

V4L2

xvideo flow

EXA use use use

use

use

Nomadik drivers

Kernel and Drivers: :MSP

Multimedia Drivers: :SVA Driver

Multimedia Drivers: :SGA Lib and Driver

January 14, 2009

Graphic and video subsystem

2D Graphics
X.11 EXA for copy, solid fill, compositing Good acceleration up to 3x on big pixmaps Penalties for smaller primitives ARM + L2C is much faster ARM for EGL and parsing SVA+FPU for TnL SGA for rendering Good fit for smart GUI, not for 3D games Xvideo extension with hw acceleration in SVA Resize, rotate, colour conversion and blitting

3D Graphics

Video rendering

January 14, 2009

Multimedia with FFMPEG

cmp Karakum PMP application Xine Linphone Web Browser Minimo Matchbox use flow flow flow flow flow FFMPEG audio codecs flow flow BlockLib V4L2 API flow flow ALSA SAA Driver flow SVA driver use SGA Driver use SGA user lib use FFMPEG video codecs EXA flow flow flow use Xvideo use Xserver use use use GTK+

flow

flow

flow

flow

AudioCodec

frame buffer

January 14, 2009

GStreamer and OpenMAX layers

cmp Static View APPLICATION

Gstreamer Based Player App

instantiate PCM_Data_Buffer flow

instantiate

FRAMEWORK

instantiate

instantiate

GSTREAMER:: GstAlsaSink

GSTREAMER_OMX: :GstOmxMp3Dec

GSTREAMER_OMX: :GstOmxMpeg4Dec

flow

GSTREAMER:: GstXVImageSink

OMX::OMX_Core

INTEGRATION

OMX:: OMX_AudioDecComponent

OMX:: OMX_VideoDecComponent

XVideo

ST_FFmpeg::ffmpeg libavcodec SAA User Lib::Blocklib

KERNEL

Linux DDrivers::ALSA driver

Linux DDrivers::SAA driver

Linux DDrivers::SVA driver

NOMADIK HW

Nomadik HW::Audio Codec (ex. 5095)

Nomadik HW:: SAA

Nomadik HW:: SVA

Nomadik HW:: Frame Buffer

January 14, 2009

Google Android on Nomadik

APPLICATIONS
Home Contacts Phone Browser ...

APPLICATION FRAMEWORK
Activity Manager Package Manager Window Manager Telephony Manager Content Providers Resource Manager View System Location Manager Notification Manager XMPP Service

LIBRARIES
Surface Manager OpenGL|ES SGL Media Framework FreeType SSL SQLite WebKit libc

ANDROID RUNTIME
Core libraries

Dalvik Virtual Machine

LINUX KERNEL
Display Driver USB Driver Camera Driver Keypad Driver Bluetooth Driver WiFi Driver Modem Driver Crypto Engine Driver Flash Memory Driver A/V Driver Binder (IPC) Driver Power Management

January 14, 2009

Multiple clipping video rendering

Minimise bandwidth usage when not needed!!!

R1

R2

R3

January 14, 2009

Thread Model
Play bin thread
gst player

Input file gst media parser

gst XVImageSink

gst-omx mpeg4 dec plugin

gst-omx mp3 dec plugin

gst ALSA sink

XVLib

omx mpeg4 dec component

omx mp3 dec component

ALSA lib

omx mpeg4 msg thread

OMX mpeg4 dec thread

OMX mp3 dec thread

omx mp3 msg thread

omx mpeg4 dec msg handler

omx mpeg4 dec buf manager

omx mp3 dec buf manager

omx mp3 dec msg handler

ffmpeg mpeg4 dec

ffmpeg mp3 dec

SAA block lib

Linux SVA driver

Linux SAA driver

January 14, 2009

Power Management
CPUfreq to scale voltage and frequency CPUIdle to switch amongst running, idle, sleep and deep sleep Custom API to control clock gating to all internal IPs Feasibility on-going with DPM

PowerTop

flow /sysfs

/sysfs

/sysfs

Kernel Space

CpuIdle use

cpufreq

Constraint Manager

Drivers and Bus Drivers

Driver Interface

Dynamic Tick (Wakeup event)

Governor

January 14, 2009

DVFS Dynamic Voltage Frequency Scaling

stm Operating Points PH1
100.8/100.8 MHz 201.6/100.8 MHz

1.0V

1.2V

264.0/132.0 MHz

1.2V
326.4/163.2 MHz Boot

1.2V

Initial

393.6/131.2 MHz 489.6/163.2 MHz

1.26V

1.45V

(lab only)

January 14, 2009

PM Dump Tool
PM monitor to optimise operating modes and ULP, estimate consumption
(Under development)
UART SDMMC MCDE TVOUT AUDIO SAA SVA BT WIFI LAN CPU 0 CPU 1
35510ms 35530ms 35550ms 35570ms 35590ms 35610ms 35630ms

I (mA) V Clock

100 1.2 393

95 1.2 393

20 0.9 64

50 1 166

10 0.8 19

100 1.2 393

95 1.2 393

20 0.9 64

50 1 166

10 0.8 19

100 1.2 393

95 1.2 393

20 0.9 64

January 14, 2009

Thank you!