Project Proposals

• Emotion-Controlled Digest Generation for Multimedia

Contents
• Emotion-Controlled Variable Multimedia Transmission
• Virtual Meeting Place
• Agent Technology for Entertainment/Consumer Services
• An Agent Digest Distribution System for Audio and Video
Contents
 Emotion-Controlled
Digest Generation
for Multimedia Contents
by
A. Barletta
 Multimedia Digest
• Give an fast preview of multimedia content
• Video, audio, books, pictures, etc
• Automatic algorithm
• Who is consuming the digest ?
• Are the information contained in a movie
sufficient for generating a digest ?
• What about the Human factor ?
Emotion-controlled Digest
Generation (I)

Digest
 Emotion-controlled Digest
Generation (II)
Emotions: Attention, Stress, Suspense, etc
Emotion Engine
Bio Sensors:
skin,
blood pressure,
retinal activity, etc

Digest
Generator
audio Model
video Stream:
Using the input
-Video
from the emotion
-Audio
Engine we select a
-Text
subset of the entire
Movie, song, book
 Emotion-controlled Digest
Generation (III) – networked -

Close Loop
Emotion Engine Control

Real Time
Digest
“Media” Server
Content Analysis
 Emotion Engine (I)
• Already available several solutions in terms of emotion detection
• Several input methods:
• Galvanic skin Response
• Electrocardiogram
• Elecromyogram
• BVP (Blood, Volume, Pressure) – Photoplethysmyograph -
• Respiration – Hall effect respiration sensor -
• Temperature (using sensors or infrared cameras)
• General Somatic Machine (GSM)
• Facial Action Coding System
• Posture, Gesture, Voice, Face
• Pupillary dilation
• Stomach motility
• Penis size
• Cortisol level in saliva
• Mature models applied to education, Human Machine Interaction, Automatic
Multimedia Selection, CE interface, etc
• No current usage in the digest generation
Emotion Engine (II)
 Digest Generator Model
• It collects emotions and synchronized with content
• It uses the data collected for filtering composite media (video,
audio, text) and for creating a digest
• The multimedia content is subdivided in time quanta and at
each of these quanta is associated the collected human
feedback
• It models the brain activity during the consuming of
information: we try to synthesize this activity
• With approximated emotion detection and a good empirical
model we can simulated the human mind and select only the
more important information that needs to be stored
 Possible Application
• Producing general digest for preview and
eCommerce
• Producing real time digest for specific users
(networking close loop case) for late complete
vision
• Fast consuming of multimedia contents (no digest)
• “Video thumbnails”
• Emotion based CE feedback (gaming)
 References
• http://affect.media.mit.edu/AC_research/sensing.html
• http://www.almaden.ibm.com/cs/BlueEyes/index.html
[BlueEyes-enabled television ]
• web.media.mit.edu/~carsonr/pdf/hcii01.pdf []
Emotion-Controlled Variable
Multimedia Transmission

by A. Barletta
 Multimedia
• Fix stream adapted to the available
bandwidth
• Who is the consumer ? A machine or a
human ?
• Human have not a constant consuming rate
• Variable bandwidth multimedia distribution
dependent on the user attention
Emotion-Controlled Multimedia
Distribution

Close Loop
Emotion Engine Control

Variable
Bandwidth
Usage “Media” Server
 Variable Quality/Bandwidth
• The Quality Of Service of the streamed
multimedia content can be changed in a wide
range
• Human information assumption is not constant: it
depends on the degree of attention
• We can adapt the quality of the content to the
status of the user
• Examples:
• Switching the video stream when the user is not
actively focusing on the screen (audio is maintained)
 Possible Applications
• Limited bandwidth devices (mobile phone)
• Video stream
 Virtual Meeting Place

How to meet your friends on

Amazon.com!
(by A. Barletta)
 Internet Experience
• Web site are virtual meeting place
• No support for communicating with visiting
people
• Meeting at the library ? Why not at Amazon ?
• Meeting at the News Kiosk ? Why not at
BBC.com ?
• Looking for a VAIO ? Ask people visiting
Sony.com !
Today Internet Experience
Internet: A Virtual meeting Place
Anybody knows S. Lew ?

Did you receive the

book you ordered ?

I did, but I didn’t like

it !
 Existing Solutions
• http://www.groove.net/
• Business oriented
• virtual shared spaces
• Business application already available
 Agent Technology for
Entertainment Services

A system for multimedia commerce

by A. Barletta, B. Moser
 The problem
• How to send digital contents ?
• How to sell digital content ?
• How to exchange digital content ?
• Which encoding mechanism ?
• A Agent framework can solve this problem
 Computing Paradigms
• Client/Server
• HTTP Server, FTP, etc
• Remote execution
• postscript
• Mobile code
• Applet
• Agents
• Aglets
Client/Server Paradigm

Agent Paradigm
 Agent Paradigm
• Reduce terminal-server interactions
• Embedded and Proactive Autonomous Behaviors
• [Data + Code] is moved to each node of the network
• Several “prototype” solutions for e-Business and network
management
• Bartering, Searching, Monitoring, etc
 AgEnt : Agent + Entertainment

• An Agent Distribution System for Audio

and Video Contents
An Agent Digest Distribution System
for Audio and Video Contents
• Autonomous software agents deliver audio and
video digest
• Expiration of contents
• Close system (security, proprietary encoder)
• Flexible: the terminals can be updated automatically
• The full chain is under control of the distributor
• Content formatting, distribution, decoding
• The user control only the selection of the contents
(Simple interface: select, try, buy, play)
 System View
Music Agent
- it contains code and the
digest data

Personal Agents Box (PAB)

- Under controlled of Sony
- provide support for Agent Distribution System (ADS):
managing/loading/ - it produces Active Agents
activating/destroy Agents - it can use a proprietary Sony
module protocol for distributing (user
- upgradable decoders transparent)
- manage security issues - it uses a Sony encoder
Music/Video Agent:
- responsible of rendering the content and looking for synchronized with the decoders
the right decoder in the PABs
-Encryption logic (public key, private key)
- maintain expiration/control (without calling back the
media server)
- in case of purchasing it connects via a secure
connection to the server and retrieve/load/display
contents
- User has not access to the contents itself
- It manages group of friends to exchange
 Agent Distribution System
(ADS)
Use case (I)
1

Personal Agents Box (PAB) 3

Digest
Song

4
 Use case (II)
• Antonio has got a VAIO system with a Personal Agents Box (PAB) application
provided by Sony
• Antonio navigates in a Sony Audio Web sites and he looks for new songs
• The Sony Web site offers an Agent Distribution System based on agents
• Antonio chooses a collection of MusicAgents he likes [2]
• Antonio decides to buy the song: after payment transaction, the content is
downloaded inside the agent, and played. Antonio unplug the VAIO from the
network and enjoy the music; … after an expiration time the agent disables the
playback and destroy the content; only the digest is available [3].
• Antonio decide to send this Agent to a friend, Boris (only the agent with the
digest is sent) [4]
• Boris open the agent, listen in the digest and decide to play the song: the agent
after a payment transaction, download the content from the Sony Web site and
plays the song
 Secure Agent Distribution
• Each Personal Agent Box has a private and
public key
• The Agent Distribution System (ADS)
receives the public key of each PAB
• A group of friends can exchange the Public
keys
• Transmission of Agent is secured using the
public key of the receiving PAB
 Secure Content Distribution
• Case 1:
• Each MusicAgent has one key
• Request of content with key
• Server encode with Agent key
• Agent decode with its own key
• Case 2
• Server has a private key
• MusicAgent has a public key of the server
• An asymmetric encryption is done
 References
• http://www.trl.ibm.com/aglets/index_e.htm
• Aglets Home Page: IBM framework for building agents
• No product available on the market from IBM