Java Media Framework

 The Java Media Framework (JMF) is an application 
programming interface (API) for incorporating time­
based media into Java applications and applets
 The JMF 1.0 API (the Java Media Player API) enabled
programmers to develop Java programs that presented time-
based media
 JMF 2.0 API extends the framework to provide support for
 capturing and storing media data,
 controlling the type of processing that is performed
during playback,
 performing custom processing on media data
streams
 JMF 2.0 defines a plug-in API to allow developers to
customize and extend JMF functionality
JMF Media Processing Model
Media Streams
 Often contain multiple channels
 Tracks
 Example
 MPEG-1 file usually 2 tracks
 Audio track
 Video track

 Demultiplexing
 Multiplexing
Example
 Process an mpeg-1 a/v media stream
 Transcode video track to H.263
 Transcode audio track to GSM

 Steps
 Demultiplex to obtain tracks
 Decompress video track
 Compress using H.263
 Decompress audio track
 Compress using GSM
 Multiplex two tracks
 Save to file
JMF Design Goals
 Enable input, processing and output of time-based
media
 Provides common cross platform API for
accessing underlying media frameworks
 Extensible
 support additional content types and formats,
optimize handling of supported format, create new
presentation mechanisms
Supported Content Types
 Supported types
 Not always both decode and encode
 Differences between platform independent and dependent
versions
 Audio
 WAV, GSM, MIDI, etc
 Image
 JPEG, etc
 Video
 H.261, H.263, MPEG-1, Quicktime, AVI, etc
Recording, processing, and
presenting time-based media
High-level Architecture
Some JMF Base Interfaces
 Clock
 Controller
 Control
Time Model
 The Clock interface
 Defines basic timing and synchronization operations
 Contains a Timebase
 Based on the system clock
 time-base time
 Simply provides current time
 Clock marks time for a particular media stream
 media time
 Current position within a media stream
Time Model
Clock
 Playback rate
 How fast the Clock is running in relation to its
TimeBase
 Examples:
 rate of 1.0 represents normal running time
 rate of 2.0 means presentation will run at twice the
normal rate
 Clock Transform
 Media-time = media start-time+ Rate(time-base time –
time-base start-time)
Example
 Example: Have a 20 sec MPEG video stream
 MediaStartTime= 10 secs,
 TimeBaseTime= 3 secs
 TimeBaseStartTime= 0 secs, TimeBaseTime–
TimeBaseStartTime= 3 secs
 Media-time= media start-time+ Rate(time-base time
– time-base start-time)
 So if Rate = 1.0, MediaTime= ??
 Alternatively, if rate = -2.0, MediaTime= ??
Achieving Synchronization
 Example
 Want to force a video renderer to sync to the
timebase of an audio renderer
 X = audio_renderer.getTimeBase()
 Video_renderer.setTimeBase(X)
 Both objects would now use the same source of
time.
Controller Interface
 Controller Interface
 Defines basic state and control mechanism for an
object that controls, presents or captures time-
based media
 Two types of Controller: Players and Processors
(considered later…)
Controllers
Controller lifecycle
Controller Events
 JMF objects can post a MediaEvent
 Events posted by a Controller:
 TransitionEvents
 Posted when a controller changes state
 Change notification events
 e.g. RateChangeEvent
 ControllerClosedEvents
 Posted when Controller shuts down
 Corresponding listener interface for each type of JMF
object that can post MediaEvents
JMF Event Model
Controls
 Mechanism for setting and querying attributes of
an object
 Certain objects expose Controls
 e.g. often used by PlugIns to provide access to their
Control objects
 Examples
 FrameRateControl
 GainControl
 Can associate listener for when volume changes
Key objects in JMF
 Managers
 DataSources
 Players
 Processors
 DataSinks
General Managers
 Intermediary objects
 Enables new implementations of key interfaces
 4 main types
 Manager
 PlugInManager

 PackageManager

 CaptureDeviceManager
The Manager object
 Manager object used for instantiating:
 DataSources,
 used to deliver time-based multimedia data
 Players,
 used to control and render multimedia data
 Processors,
 used to process data and output the processed data
 DataSinks,
 takes a DataSource as input and renders the output to a specified
destination
The Manager Object
Data Model in JMF
 Data Sources
 Media players use DataSources to manage the
transfer of media-content
 DataSource encapsulates location of media and the
protocol used to deliver the media
 Typically Identified by a:
 URL, MediaLocator
Capture
 Capture devices represented as DataSources
 e.g. microphone, video capture board, etc…
 Devices can deliver multiple data streams
 e.g. audio and video from a camera
 e.g. multiple audio tracks from a recording session
 You may then wish a single DataSource to contain
multiple SourceStreams
 Manager.createMergingDataSource(SourceStreams)
Push and Pull Data Sources
 Data sources can be categorized according to how
data transfer is initiated
 Pull Data-Source Client initiates the data transfer
 e.g. HTTP and FILE
 Push Data-Source Server initiates the data transfer
 e.g. broadcast and multicast media
Players
 Processes an input stream and renders it at a
precise time
 Does not provide any control over the processing
that it performs or how it renders the media data
Players
 Player extends the Controller interface.
 Has a lifecycle
 Sends media events
 Player as a MediaHandler
 player = Manager.createPlayer(myDataSource); 
player = Manager.createPlayer(myMediaLocator); 
player = Manager.createPlayer(myUrl);
Players
UI Components
 Players provides access to UI Components
 Player (or Processor) can provide two UI
components
 Visual component
 Control-panel component

 Can retrieve these components using methods:

 getVisualComponent()
 getControlPanelComponent()
Player States Continued
 Players post transitional events as they move
between states
 ControllerListener
 Is the Player in an appropriate state?
 Only certain methods make sense in certain states
 e.g. calling getTimeBase method on an unrealized
player gives an error
Processors
 Can also be used to present media data
 Specialized type of Player that provides control
over processing performed on the input media
stream
Processing
Processor Stages
Additional Processor States
 Two additional stand-by states:
 Configuring
 Configured – can use TrackControls
Processing Controls
 For a given track, can control processing
operations performed by the Processor by using
the TrackControl for that track.
TrackControl[] = processor.getTrackControls()
 Can explicitly select:
Effect, Codec and Renderer plug-ins to use
TrackControl[1].setCodecChain( array_of_codecs )
Configuring the Processor
 Consider using a processor to transcode an
(audio+video) QuickTime movie – changing
mpeg video track to h.263…
p = Manager.createProcessor(dataSource)
p.configure()
p.setContentDescriptor.QUICKTIME
tcs[] = p.getTrackControls()
 Returns an array, e.g. 2 track controls…
Configuring the Processor
Format f0 = new VideoFormat(VideoFormat.h263,
new Dimension(width, height),
Format.NOT_SPECIFIED,
Format.byteArray,
(float)frameRate);
l Format f1 = new AudioFormat(AudioFormat.mpeg,
8000
8,
2);
tcs[0].setFormat(f0)
tcs[1].setFormat(f1)
p.realize()
p.start()
Processor Summary
 A Processor does not have to output data as a
DataSource, such a processor (i.e. one that renders
the data) is effectively a configurable player.
Data Storage and Transmission
 DataSink
 Used to read data from a DataSource and render the
media to an output destination
 Typical actions…
 Write data to a file, across a network etc
Using the DataSink
MediaLocator dest = new
MediaLocator(file://newfile.wav);
dsink = Manager.createDataSink(ds, dest);
dsink.addDataSinkListener(this);
dsink.open();
p.start();
dsink.start();
 Wait for EndOfStream event
 Close DataSink and remove Listener
 dsink.close()
Example
 Applet Movie Player
Applet Movie Player
 Simple Java Applet that demonstrates how to
create a simple media player with a media event
listener. It will play the media clip right away and
continuously loop.
<!­­ Sample HTML
<applet code=TVApplet width=587 height=510>
<param name=file value=“playme.mov">
</applet> ­­>
Basic Steps
 Initialisation…
 Retrieve applet’s FILE parameter
 Use this to locate media file and build URL
 Create Player using the Manager object
 Register applet as a ControllerListener
Steps 1 & 2: Resolving a URL
for the media stream
// The applet tag should contain the path to the
// source media file, relative to the html page.
if ((mediaFile = getParameter("FILE")) == null)
Fatal("Invalid media file parameter");
try {
// Create a url from the file name and the url
// to the document containing this applet.
if ((url = new URL(codeBase, mediaFile)) == null)
Fatal("Can't build URL for " + mediaFile);
Step 3: Using Manager to
Create a Player
// Create an instance of a player for this media
try {
player = Manager.createPlayer(url);
}
catch (NoPlayerException e) {
System.out.println(e);
Fatal("Could not create player for " + url);
}
Step 4: Register applet as a
ControllerListener
// Add ourselves as listener for player's events
player.addControllerListener(this);
} catch (MalformedURLException e) {
Fatal("Invalid media file URL!");
} catch (IOException e) {
Fatal("IO exception creating player for " +
url);
Controlling the Player…
 Starting the Player
public void start() {
if (player != null)
player.realize();
 Stopping the Player
public void stop() {
 if (player != null) {
   player.stop();
   player.deallocate();
 }
}
Responding to media events

 Need to Implement ControllerListener

 When the Player is realized
 Posts a RealizeCompleteEvent
 Get the Visual component
if (( visualComponent =
player.getVisualComponent())!= null) {
cPanel.add(visualComponent);
 Get the Control Panel component
 When the media has reached the end…
 Posts an EndOfMediaEvent
 Rewind and start over

player.setMediaTime(new Time(0));
Extensibility
 Can extend JMF functionality in two ways:
 Using Plug-ins
 Effectively implementing custom processing components that can
be interchanged with standard components used by a Processor
 By direct implementation
 i.e implementing directly the Controller, Player , Processor,
DataSource, or DataSink interfaces
 e.g. implementing a player to utilise a h/w MPEG decoder
 e.g. integrating existing media engines
Interface Plugin
 The base interface for JMF plug-ins.
 A PlugIn is a media processing unit that accepts
data in a particular format and processes or
presents the data.
 Registered through the PlugInManager.
 Methods…
 Open(), Close(), getName(), reset()
 Sub-interfaces…
 Codec, effect, demultiplexer , multiplexer, etc.
Codecs
 One input and one output
 Methods…
 setInputFormat()
 setOutputFormat()
 getSupportedInputFormats()
 getSupportedOutputFormats()
 process()
 input buffer, output buffer
Example 2
 Accessing individual frames
FrameAccess
 Problem
 How to access individual decoded video frames
from a Processor while processing the media. This
could be used for scanning the decoded data;
computing statistics for each video frame, etc.
FrameAccess
 Solution…
 use a ‘pass-through’ plugin codec as a callback when
individual frames are being processed.
 Steps:
 1) Build the pass-through codec.
 2) Create a processor from the input file. This processor
will be used as a player to playback the media.
 3) Get the TrackControls from the processor.
 4) Set your codec on the video track:

TrackControl.setCodecChain(your_codec[])
Basic code
// Get Video track as a track control
TrackControlvideoTrack = null;
for (int i = 0; i < tc.length; i++) {
  if (tc[i].getFormat() instanceof VideoFormat) {
    videoTrack = tc[i]; break;
  }
}
// Instantiate & set frame access codec to data flow 
path.
Codec codec[] = { new PreAccessCodec(),
new PostAccessCodec() };
PreAccessCodec
public class PreAccessCodec implements Codec {
  void accessFrame(Buffer frame) {
    long t = (long)(frame.getTimeStamp()/10000000f);
    System.err.println("Pre: frame #: " +
    frame.getSequenceNumber () +
    ", time: " + ((float)t)/100f +
    ", len: " + frame.getLength());
  }
…………………
 Other methods, e.g. getSupportedInputFormats etc.
 The Codec’s Process method
public int process(Buffer in, Buffer out) {
  // This is the "Callback" to access individual frames.
  accessFrame(in);
  // Swap the data between the input & output.
  Object data = in.getData ();
  in.setData(out.getData ());
  out.setData(data );
  // Copy the input attributes to the output
  out.setFormat(in.getFormat());
  out.setLength(in.getLength());
  out.setOffset(in.getOffset());