EyesWeb XMI 5.3.

0 SIEMPRE Library

December 22, 2012

Part I Description

The EyesWeb SIEMPRE Library includes a collection of software modules and patches which have been specically designed and developed to support the requirements from the EU ICT Siempre project. In particular, the SIEMPRE Library integrates the EyesWeb platform with modules which are required for a distributed multimodal data recording, playback, analysis. The SMPTE decoder block received an audio signal in inputs and decodes the SMPTE timecode contained in the signal. See SMPTE time code and the references therin for a description of the SMPTE encoding schema. The SMPTE encoder block generates an audio signal with an encoded time code. The Wave File Writer block writes timeseries in the binary format chosen for the SIEMPRE recordings. The chosen binary format uses a small subset of the Broadcast Wave Format specications. The DeckLink input block add support for the DeckLink family of framegrabbers, which are used in the Casa Paganini setup (see Acquisition setup at Casa Paganini) for synchronized audio/video input from High Denition (HD) videocameras. The Qualisys SMPTE decoder block, which decodes the SMPTE timecode as packed in 32 bits by the Qualisys Track Manager software. This block, toghether with the OSC support already available in EyesWeb, allows integration between EyesWeb and the Qualisys system. This provides the possibility to analyze, in real-time, the data tracked by the Qualisys Motion Tracking system. For the aim of the SIEMPRE project, the integration is particularly useful to be able to synchronize with the Qualisys Software to visualize the recorded data. The Qualisys SMPTE encoder block, which encodes an SMPTE timecode to a packed 32 bits integer number with the same coding schema as the Qualisys Track Manager software. The Timecode generator block, which generates a timecode locked to an audio input signal; it may be useful for data playback. The Qualisys RT Protocol Decoder block, which get the complete data packet, as sent by the Qualisys Track Manager software, and decodes all of the data in it. It let you to get all the data in a compact way; e.g., all the tracked markers are received in an unique labelled set, instead of being received by separate OSC blocks. The Qualisys TSV Reader block, which reads the data exported by the Qualisys Track Manager software in the TSV (Tab Separated Values) format. In the following chapters, some examples are described, which show the overall infrastructure used for many of the recordings which happens in the EU ICT Siempre project and the related EyesWeb patches.

Chapter 1 Example 1: String Quartet recording

The aim is to record audio, video, motion capture, and biometric-sensors data of performances of a String Quartet. The recording setup includes a Qualisys Motion Tracking system, one or more HD videocameras, ambient microphones, instrument microphones, wearable biometric sensors. The recording happens on distinct PCs, and the synchronization of the data on the computers is guaranteed by timestamping each sample with a reference SMPTE clock which is received by all computers (see Figure 1.1). In the depicted example, three computers are used, however, it is straighforward to integrate new computer and devices in the recording infrastructure. The three computers which are visible in the setup perform dierent operations: The black computer in the middle (A) records the Motion Tracking data. The same computer also operates as the master clock generator, i.e., it generates the SMPTE signal which is propoagated to the other computers. The lower-left computer (B) records audio/video data coming from the HD camera (JVC GY-HD-251 in this case). The lower-right computer (C) records audio data from the instruments. EyesWeb and the SIEMPRE Library are used for several purposes in this setup. In computer (A) a patch to generate the SMPTE signal on multiple audio channels is used. The patch si visible in Figure 1.2. In computer (B) a patch to save audio/video data from the HD camera is used. The patch si visible in Figure 1.3. In computer (C) a patch to save audio data from the microphones on the instruments is used. The patch si visible in Figure 1.4.

Figure 1.1: The acquisition setup installed at Casa Paganini, InfoMus Lab, Genova, Italy.

Figure 1.2: The smpte.eywx patch generates an Smpte signal.

Figure 1.3: A patch to save the HD audio/video signal.

Figure 1.4: A patch to save the multichannel audio signal from the microphones on the instruments.

Chapter 2 Qualisys integration

The SIEMPRE project is based on the use of the Qualisys Motion Tracking system to analyze the movements of the performers. EyesWeb has been extended in order to integrate with Qualisys and receive the tracking data in real-time. Figure 2.1 shows an example of an EyesWeb patch receiving motion tracking data in real-time. The Z coordinate of a marker placed on the top of a bow is received by the EyesWeb patch and plotted (see the graph in the lower part of the patch). Moreover, the patch also receives the SMPTE timecode from Qualisys; this timecode allows for synchronization of the processed data. Another interesting integration example is given in Figure 2.2. In this example, the Smpte audio signal is generated by EyesWeb basing on the value received, via OSC, from the Qualisys RTM software. This mechanism may be used, for instance, to listen to the recorded audio tracks in sync with the playback of the recorded motion-tracking data. As a matter of facts, many audio editors1 allow for using the Smpte value as the reference to which all the audio data is synchronized. When the audio data is recorded, it is timestamped with the current Smpte value; when the same Smpte value is received in input, the corresponding audio data is played. Thus, by regenerating the Smpte signal, EyesWeb can control the playback of the recorded data in synch with the playback of the motion tracking data in the Qualisys RTM software. To use this patch, open a QTM le2 and process it with the command Run real time processing on le..., which enables the Qualisys real time protocol (i.e., data streaming via OSC). Then, run the patch; you should see the same Smpte value in both the status bar of the Qualisys software and in the EyesWeb display in the top of the patch (the one with a black background and red digits). The maximum allowed dierence between these values can be tuned by means of the slider. However, note that imposing a small threshold may cause the generated smpte signal to be resynched to the value streamed by Qualisys too many times. This can cause degradation of the quality of the played audio signal as the generated Smpte signal contains discontinuities. The example in Figure 2.3 shows how the data received by the QTM software can be visualized in EyesWeb by means of the 3d display. In this patch, the data is received by the OSC block using the Raw Mode; i.e., te data is not actually decoded and each received packet is sent, as is, to the Qualisys RT Protocol Decoder block. This block provides, as output, two labelled sets, one for the idenditied markers and another one for the unidentied markers. The identied markers are sent to the BoneLines3DMaker block,
1 2

The setup at Casa Paganini uses Adobe Audition as the audio editor Of course, the le must have been recorded with Smpte enabled

Figure 2.1: The qualisys rt.eywx patch shows the integration between EyesWeb and the Qualisys RTM software.

Figure 2.2: The qualisys controlled smpte.eywx patch regenerates an audio Smpte timecode based on the Smpte valued received by Qualisys via OSC.

which adds bones among markers to form some skeleton structures. These structures are nally added to an OpenGl 3d world and visualized to the user. Note the dierence with the patch shown in Figure 2.1: in that patch an OSC receiver is used to get a single coordinate (the Z) of a single marker. That aproach makes sense when you need to show a small number of markers and you know their labels in advance (as the OSC receiver needs the name of the marker as the OSC tag; however, when the number of marker increases, as it is in the String Quartet case, that approach is unfeasible due to the large number of blocks which would be required. To overcame this problem, the Qualisys Visualization patch uses the RT Protocol decoder block: this block receives a raw packet from the OSC block, and decodes all the data in it (the packet is composed of many OSC bundles). The data is sent to the output pins; in particular two output pins provide the labelled set of identied and unidentied markers. Thus, the data can be easily managed with standard EyesWeb blocks working with geometric labelled sets, or it can be visualized to the user as it happens in this patch.

Figure 2.3: The qualisys visualization.eywx patch shows the visualization, in EyesWeb, of the 3D mocap data.

Chapter 3 Mocap data playback and analysis

Once the mocap data is recorded with the Qualisys software (or with any other mocap system), the data can be playback and analyzed in EyesWeb. This chaptershows some example patches, from simple ones which simply playback the mocap data, to more complex ones, where data from dierent sources is playback in synch in EyesWeb. Note that in this chapter we are not going to use the Qualisys software anymore: we suppose the data has been recorder, may be labelled, and exported in some readile format. Most examples will use the TSV format (Tab Separated Values) or the C3D format. Thus, you do not need to have the Qualisys software to test these patches; you can use the example les provided with EyesWeb. This rst simple example, visible in Figure 3.1, demonstrates the playback of a TSV (tab Separated Values) le as exported by Qualisys. The TSV player can be controlled by feeding it with an external clock signal. In this case the clock is extracted from an audio signal. The example in Figure 3.2 shows a simple processing of the data coming from a Qualisys TSV le. In this case the data of each single performer is extracted, and, for each subset of markers, we compute the bounding parallelepiped and draw it. Moreover, we extract the three markers of the head, compute their baricenter, and then we connect the obtained four head baricenters through bones. The resulting structure is visualized in the 3d display, together with the performers skeletons and bounding parallelepipeds.

10

Figure 3.1: The qualisys tsv player.eywx patch shows the visualization, in EyesWeb, of the 3D mocap data exported by Qualisys in TSV format.

11

Figure 3.2: The qualisys tsv processing.eywx patch shows a simple processing of data coming from a Qualisys TSV le.

12

Part II Reference

13

Chapter 4 SIEMPRE Catalog

4.1
4.1.1

Blocks
Qualisys6DTSVReader

bitmap

class name catalog name catalog id class id

Qualisys6DTSVReader SIEMPRE SIEMPRE Qualisys6DTSVReader

Read TSV le from Qualisys. The le must contains at least markers name. In the TSV export settings window check: - Include TSV header - Export time data for every frame Export column headers - Include TSV header Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys.

14

OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys. OutputFrameNumber id output frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* Extracted datas frame number. SMPTE id output smpte type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* Extracted datas SMPTE. Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsv File name

15

ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling domain [ 0, 4 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. It works only if ActivationMode is set to Polling mode. Loop: activates circular loop mode. Stop: not implemented yet. Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated. UseSMPTE id use SMPTE type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block use SMPTE or time column to read the le.

16

4.1.2

QualisysC3DReader

bitmap

class name catalog name catalog id class id

QualisysC3DReader SIEMPRE SIEMPRE QualisysC3DReader

Read C3D le format. The le follows C3D specications(www.C3D.org) Actually the block doesnt read analod data. Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys. OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys.

17

Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=C3D les (*.C3D)*.C3D File name ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling domain [ 0, 4 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. Loop: activates circular loop mode. Stop: not implemented yet.

18

Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated.

19

4.1.3

QualisysRTProtocolDecoder

bitmap

class name catalog name catalog id class id authors

QualisysRTProtocolDecoder SIEMPRE SIEMPRE qualisys rtprotocol decoder Paolo Coletta

Decodes the Raw Data received from the Qualisys QTM software via the OSC RealTime protocol. Details The OSC Realtime protocol of QTM can also be decoded by using standard EyesWeb OSC blocks. However, receiving data about all the markers is not strainghforward, as the number unidentied QTM trajectories is time-varying. This block gets all such data as once, building labelled-sets as output Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs RawData id type type id input Base, RawData base, raw data required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The Raw Data received from the Qualisys QTM software via the RT protocol

Outputs Timestamp id timestamp output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The time elapsed since the ecordng was started

20

SMTE timecode id smpte output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The value of SMPTE as received by QTM Frame number id frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* The frame number QTM Unidentied markers id unidentied markers type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The set of markers which were not recognized by QTM as belonging to an AIM model Identied markers id identied markers type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The set of markers which were recognized by QTM as belonging to an AIM model Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 )

21

The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.

22

4.1.4

QualisysSmpteDecoder

bitmap

class name catalog name catalog id class id authors

QualisysSmpteDecoder SIEMPRE SIEMPRE qualisys smpte decoder Paolo Coletta

Decodes an SMPTE timecode from an integer number. The SMPTE is assumed to be encoded with the same format used in the Qualisys RT protocol (see details). Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: Minutes Bit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Encoded SMPTE timecode id input type Kernel, Int datatype (Kernel Catalog). type id kernel, int required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The encoded SMPTE timecode as provided by the Qualisys RT protocolo. Usually obtained by means of the OSC block Outputs Decoded timecode id output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The decoded timecode

23

Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.

24

4.1.5

QualisysSmpteEncoder

bitmap

class name catalog name catalog id class id authors

QualisysSmpteEncoder SIEMPRE SIEMPRE qualisys smpte encoder Paolo Coletta

Encodes an SMPTE timecode to an integer number, to make it suitable, for instance, to be sent via OSC. The encoding protocol is the one used by the Qualisys RT communication protocol (see details). Details SMPTE time code little endian format: Bit 0 ? 4: Hours Bit 5 ? 10: Minutes Bit 11 ? 16: Seconds Bit 17 ? 21: Frame Bit 22 ? 30: Sub frame Bit 31: Valid bit Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Timecode id type type id input Kernel, Time datatype (Kernel Catalog). kernel, time required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* Timecode to be encoded to a packed SMPTE integer

Outputs Encoded timecode id output type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* The encoded timecode

25

Parameters TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Sampling rate id subframe sampling rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) The sampling rate of the data in the source system. Since the SMPTE is expressed also using frames and subframes, the sampling rate of the date is required to correctly interpret the frame and subframe elds.

26

4.1.6

QualisysTSVReader

bitmap

class name catalog name catalog id class id

QualisysTSVReader SIEMPRE SIEMPRE QualisysTSVReader

Read TSV le from Qualisys. The le must contains at least markers name. In the TSV export settings window check: - Include TSV header - Export time data for every frame Export column headers - Include TSV header Outputs OutputDoubleLabelledSet id output labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing points from qualisys. OutputUnidentiedDoubleLabelledSet id output unidentify labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double inplace id *no* inherited id *no* The labelled set of 3d points double, containing unidentied points from qualisys. OutputFrameNumber id output frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int inplace id *no* inherited id *no* Extracted datas frame number.

27

SMPTE id output smpte type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* Extracted datas SMPTE. Parameters FileName id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=false, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsv File name ActivationMode id read mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Bang Line layout Time Polling Frame domain [ 0, 5 ) Activation mode: Bang: external clock connected to the input, is used to activate the block. On line number: external integer value connected to the input (which is interpreted as the line number of the le) is used to activate the block. On time: external time value connected to the input (which is interpreted as one of the time columns of the le) is used to activate the block. Polling: an internal signal is used to activate the block. If the le has at least one time column the signal is generated according to the oset of consecutive times. If there is no time, Eyesweb automatically activates the block. On frame number: external integer value connected to the input (which is interpreted as the frame number of the le) is used to activate the block.

28

MediaTimeMode id media time type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block set time column of the le as media time. If not, interpolated time is used to set the value. If the input is not a time , this parameter is ignored and the block take the column value in the le ad default. LoopMode id eof mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Loop Stop layout Pause NoOutput domain [ 0, 4 ) Loop mode. It works only if ActivationMode is set to Polling mode. Loop: activates circular loop mode. Stop: not implemented yet. Pause: generates the current value of le. If the input value of line or time is out of bound it generates rst or last value of le. NoOutput: output is not generated. UseSMPTE id use SMPTE type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool It species if the block use SMPTE or time column to read the le.

29

4.1.7

QualisysTSVWriter

bitmap

class name catalog name catalog id class id

QualisysTSVWriter SIEMPRE SIEMPRE QualisysTSVWriter

Writes to le a geometric labelled set of 3d points. Points coordinates are written as tab-separated values, using the same format of the Qualisys software. Notes By default, the block writes the data in a Qualisys compatible format. To this aim, an header is written in the le which is needed by Qualisys to correctly read the data; the header is needed even if you plan to read the generated le through the EyesWeb QualisysTsvreader block. However, the user can decide to omit such header; this makes the resulting le not compatible with the Qualisys software, but easier to be imported in third parties software (e.g., Calc, Excel, Matlab). In this case, since the column names are not written: it is up to the user to remember the meaning of each column Labels mode parameter: the column number of the generated le must uniquely identify a marker (label), otherwise reading back the le will be impossible. However, the labels available in the input dataset might change during time (e.g., missing markers when capturing, new markers available, etc), thus, the problem arises to decide which labels (and hence which columns) will be written to le. The user can select among six possible solutions, through the Labels mode parameter: Custom: the user can specify the number of labels and, for each label, its value. Note that the set of user-specied labels is furtherly processed by the block to get rid of empty or duplicate labels Inputs InputGeometricLabelledSet3DDouble id input labelled set type Base, BaseGeometricLabelledSet3Double type id base, base geometric labelled set 3d double required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no*

30

Input Geometric Labelled Set of 3D double points. This must be an homogeneous list of 3d geometric points. Parameters Filename id le name type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=true, layout OverwritePrompt=true, Filter=TSV les (*.tsv)*.tsvAll les (*.*)*.* The name of the output le Input type id input type type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Double layout Int domain [ 0, 2 ) Species if the input geometric labelled set is double or int. Write time column id write time type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species if the block writes the time column on the output le. The value of the time column is extracted from the media time of the input data Write smpte column id write smpte type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species if the block writes the smpte column on the output le. If set to true, an additional input will be available; the value of the smpte is received through this input Write frame number column id write frame number type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species whether to block writes the frame number column on the output le

31

First frame number id rst frame number type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) Species the value of the rst frame number. By default, frames start at 1, but a dierent value can be specied. Write header id write header type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Species whether the block writes the le header or not. See the block notes for details NumOfCameras id num of camera type Kernel, Int datatype (Kernel Catalog). type id kernel, int Number of cameras. Its in the Qualisys TSV header. Frequency id frequency type Kernel, Int datatype (Kernel Catalog). type id kernel, int Frequency. Its in the Qualisys TSV header. Number of analog channels id num of analog type Kernel, Int datatype (Kernel Catalog). type id kernel, int Number of analog channels. This value is not used actually by EyesWeb when reading the TSV, however, it is used by the Qualisys software and must match the number of columns in the corresponding * a.tsv le. Analog frequency id analog freq type Kernel, Int datatype (Kernel Catalog). type id kernel, int Analog frequency. This value is not actually used by EyesWeb when reading the TSV, however, it is used by the Qualisys software and refers to the sampling frequency of the analog data which is available in the corresponding * a.tsv le. Description id description type Kernel, String datatype (Kernel Catalog). type id kernel, string 32

A free-text description to be wrote in the TSV header TimeStamp id time stamp type Kernel, String datatype (Kernel Catalog). type id kernel, string A string representing the timestamp (date and time) of the recording session (e.g., Friday, December 21st, 2012 Labels mode id labels mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom First non-empty frame First frame layout Kinect Microsoft SDK Kinect OpenNI File domain [ 0, 6 ) Species how the block decides which labels are written to le. See the block notes for details

33

4.1.8

SmpteDecoder

bitmap

class name catalog name catalog id class id authors

SmpteDecoder SIEMPRE SIEMPRE smpte decoder Paolo Coletta

Decodes an SMPTE timecode from the audio signal.

Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), with minor changes to build under Win32. The derived source code is available at https://svn.infomus.org/publ Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio Stream id type type id input Base, PCMAudioBuer base, pcm audio buer required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio stream which should contain the SMPTE timecode track

Outputs Timecode id output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The decoded timecode

34

Parameters TimeCode Track id time code track type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 0, +innity ) The zero-based index of the timecode track. If it is greater than the number of available channels, the last channel will be used. Output correction mode id output correction mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: None layout Output oset Buer start domain [ 0, 3 ) In general the SMPTE timecode might be not aligned with the audio buer. This parameter species how to cope with this issue. None: the error caused by this misalignemnt is neglected Output oset: an addition output is added to the block. The output species where the SMPTE timecode started with respecte to rst sample of the audio buer. The oset might be negative Buer start: the output timeoeis adjusted such that it can be interpreted as the timecode assigned to the rst sample of the buer Enable Output Locked id enable output locked type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool Enable/disable an output providing info about whether the SMPTE signale has beed locked TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 )

35

The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom

36

4.1.9

SmpteEncoder

bitmap

class name catalog name catalog id class id authors

SmpteEncoder SIEMPRE SIEMPRE smpte encoder Paolo Coletta

Encodes an SMPTE timecode to an audio signal.

Details Based on the LTC SMPTE library (http://ltcsmpte.sourceforge.net/), with minor changes to build under Win32. The derived source code is available at https://svn.infomus.org/publ Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio clock id type type id input Base, Audio clock base, audio clock required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio clock

Outputs Output id output type Base, PCMAudioBuer type id base, pcm audio buer inplace id *no* inherited id *no* The encoded timecode

37

Parameters TimeCode id time code type Kernel, Time datatype (Kernel Catalog). type id kernel, time The initial timeocde. Reset id reset type Kernel, Trigger datatype (Kernel Catalog). type id kernel, trigger Reset the encoder to the initial timecode value TimeCode Framerate id frame rate type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Custom ATSC24/lm (24fps) layout NSTC (29.97fps) PAL (25fps) ATSC30 (30fps) domain [ 0, 5 ) The framerate of the SMPTE timecode, i.e., how many timecode words are available in a second. If Custom is selected a custom value can be specied in the Custom Framerate parameter, otherwise one standard value can be selected Custom Framerate id custom frame rate type Kernel, Double datatype (Kernel Catalog). type id kernel, double The custom framerate of the SMPTE timecode; this parameter is used only when TimeCode Framerate is set to Custom Enable timecode output id enable timecode output type Kernel, Bool datatype (Kernel Catalog). type id kernel, bool If this parameter is true, an additional output is added to the block which provides the interpolated timecode of the begining of the audiobuer (i.e., the timecode virtually assigned to sample 0 of the buer)

38

4.1.10

TimeCodeGenerator

bitmap

class name catalog name catalog id class id authors

TimeCodeGenerator SIEMPRE SIEMPRE timecode generator Paolo Coletta

Generatae a timecode starting locked to the input audio clock signal. Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs Audio clock id type type id input Base, Audio clock base, audio clock required for initialization required required for execution read only/read write read only referred as inplace *no* referred as inherited *no* The input audio clock

Outputs Timecode output id timecode output type Kernel, Time datatype (Kernel Catalog). type id kernel, time inplace id *no* inherited id *no* The timecode virtually assigned to the sample 0 of the output buer Parameters TimeCode id time code type Kernel, Time datatype (Kernel Catalog). type id kernel, time

39

The initial timeocde. Reset id reset type Kernel, Trigger datatype (Kernel Catalog). type id kernel, trigger Reset the encoder to the initial timecode value

40

4.1.11

WaveFileWriter

bitmap

class name catalog name catalog id class id authors

WaveFileWriter SIEMPRE SIEMPRE wave le writer Paolo Coletta

Write the input time series as a wav le. Notes With the support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Inputs TimeSeries id type type id input Kernel, Generic datatype kernel, generic datatype required for initialization required required for execution read only/read write read write referred as inplace *no* referred as inherited *no* Time series to be saved to le

Required interfaces Kernel, StaticTimeSeries Parameters Filename id param lename type Kernel, String datatype (Kernel Catalog). type id kernel, string Filename, MustExist=true, SaveMode=true, layout OverwritePrompt=true, Filter=Wave les (*wav)*.wavAll les (*.*)*.*All les (*.*)*.* Name of the wav le. Use an empty name to stop the recording

41

Format id type type id

param format Kernel, Int datatype (Kernel Catalog). kernel, int Combo Box: 32bit oat (oat) 64bit oat (Double) layout Normalized 32bit oat (oat) Normalized 64bit oat (Double) domain [ 0, 4 ) Denes the format used to write the sample into the le : oat each sample is writed as 32bit oating value; double each sample is writed as 32bit oating value; Title id param title type Kernel, String datatype (Kernel Catalog). type id kernel, string Title tag. It will be mapped to the INAM Exif tag Datetime id param datetime type Kernel, String datatype (Kernel Catalog). type id kernel, string Datetime tag. It will be mapped to the ICRD Exif tag Comment id param comment type Kernel, String datatype (Kernel Catalog). type id kernel, string Comment tag. Free text that will be mapped to the ICMT Exif tag Timecode id param timecode type Kernel, String datatype (Kernel Catalog). type id kernel, string Timecode tag. It will be mapped to the ISMP Exif tag. AddMetaData id add metadata info type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: No metadata layout Using InfoChunk Using iXml domain [ 0, 3 )

42

Denes the format used to write metadata into the le.

43

4.1.12

DeckLinkInput

bitmap

class name catalog name catalog id class id authors

DeckLinkInput DeckLink decklink decklink input Paolo Coletta

Acquire synchronized audio/video from a DeckLink card. Notes With the partial support of the EU ICT Project 250026 - SIEMPRE (Social Interaction and Entrainment using Music PeRformance Experimentation), 2010-2012 Outputs Output id type type id inplace id inherited id video output Base, Image base, image *no* *no*

AudioOutput id audio output type Base, PCMAudioBuer type id base, pcm audio buer inplace id *no* inherited id *no* Parameters Device Index id device index type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 0, +innity ) The zero-base index of the Decklink card, in the case that more than one is installed in the system

44

InputMode id input mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: Audio/Video layout Video only domain [ 0, 2 ) Species whether to capture audio, or video, or both Video Mode id video mode type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: NTSC 720x486 30/1.001fps interlaced NTSC 720x486 30/1.001fps interlaced (3:2 pulldown) PAL 720x576 25fps interlaced NTSC 720x486 30/1.001fps progressive PAL 720x576 25fps progressive HD 1920x1080 24/1.001fps interlaced HD 1920x1080 24fps interlaced HD 1920x1080 25fps progressive HD 1920x1080 30/1.001fps progressive HD 1920x1080 30fps progressive HD 1920x1080 25fps interlaced layout HD 1920x1080 30/1.001fps interlaced HD 1920x1080 30/1.001fps interlaced HD 1920x1080 50fps progressive HD 1920x1080 60/1.001fps progressive HD 1920x1080 60fps progressive HD 1280x720 50fps progressive HD 1280x720 60/1.001fps progressive HD 1280x720 24/1.001fps progressive 2K 2048x1556 60fps interlaced 2K 2048x1556 60fps interlaced 2K 2048x1556 60fps interlaced domain [ 0, 22 ) Select the resolution and framerate of the video stream

45

Pixel Format id pixel format type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: UYVY 4:2:2 packed layout BGRA 4:4:4 raw domain [ 0, 2 ) Select the pixel format of the video stream Num Audio Channels id num channels type Kernel, Int datatype (Kernel Catalog). type id kernel, int domain [ 1, +innity ) Species the number of audio channels to capture Audio Sample Format id sample format type Kernel, Int datatype (Kernel Catalog). type id kernel, int Combo Box: 16bits signed layout 32bits signed domain [ 0, 2 ) Species the format of each sample of audio

46

Part III Appendices

47

Appendix A SIEMPRE Binary Format

A.1 SIEMPRE Binary le format

Monodimensional timeseries will be saved or converted to time-aligned Broadcast Wave Format compatible les (i.e., extended WAVE les). The SIEMPRE project will exploit subset of the Broadcast Wave Format standard. In particular it will x the number of channels to 1 (monodimensional data), the format to WAVE FORMAT IEEE FLOAT (0x03), and the bits per sample to 32. For a general description of Broadcast Wave Format and Wave les you may refer to the Microsoft documentation or start from the following links: Audio File Format Specifications SonicSpot

A.1.1

Format description

The WAVE (sub-)format used for SIEMPRE has the following structure: Field ChunkID ChunkSize WaveID Length 4 4 4 Content RIFF Size, in bytes, of the RIFF chunk. Should be equal to the length, in bytes, of the le minus 8. WAVE Some chunks. Each chunk is identied by a fourCC (four characters identier) followed by the size, in bytes, of the chunk (the size does not include the FourCC and the size itself). Two mandatory chunks are the fmt chunk and the data chunk, which contains the characteristics of the le (sample rate, num channels, sample size, etc.) and the sample data respectively.

Wave chunks

...

48

The fmt chunk and the data chunks have the following structure:

Fmt Chunk Field ChunkID ChunkSize FormatTag NumChannels

Length 4 4 2 2

NumSamplesPerSeq 4 AvgBytesPerSec 4

BlockAlignment

BitsPerSample

Content fmt For the SIEMPRE case the value is 16, a generic reader should be prepared to face dierent sizes too. For the SIEMPRE case the value is WAVE FORMAT IEEE FLOAT, i.e., 0x03. For the SIEMPRE case the value is xed to 1, i.e., monodimensional channels Sampling rate (e.g., somthing around 250.0 for kinematical sensors) Data rate, in bytes, of the le. Can be computed as NumSamplesPerSeq * BlockAlignment Size of a block (a sample for each channel). Can be computed as NumChannels * (BitsPerSample / 8). For the SIEMPRE case NumChannels is xed to 1, BitsPerSample is xed to 32, thus BlockAlignement is xed to 4. Number of bits for each sample. For the SIEMPRE case this is xed to 32 (single precision oating point numbers)

Data Chunk Field ChunkID ChunkSize Data Padding bytes

Length 4 4 ... 0 or 1

Content data For the SIEMPRE case the value is 4 (size of sample) multiplied by the number of samples This is the actal data: sequence of oat numbers in the SIEMPRE case In the SIEMPRE case this is not needed as data is aligned. In the general case this is only needed if data is not aligned to a 16bits boundary

49

A.1.2

Application compatibility

The les produced by the Wave File Writer block have been tested to load in the following applications: Players VLC media player Windows Media Player Quick Time player Music Software Audacity Sony Sound Force Sony Vegas Pro Computing environments Matlab: does not load additional info about the le. This is a general limitation for the WAVE format support in Matlab.

50