Professional Documents
Culture Documents
DOI 10.1007/s00450-014-0258-8
Abstract While structured software development processes are nowadays common-place, 3D applications are often
still developed in an ad-hoc and code-centric manner. This
state of affairs is somewhat surprising, as there are many
benefits to a more structured 3D development process. In
particular, model-based techniques with support for iterative
development could help to ensure the mutual consistency
of the code bases produced by the heterogeneous developer groups, i.e. 3D content developers and programmers.
However, a cultural gap exists between 3D content developers and programmers who use completely different tools
and terminologies. Common tools for model-driven development, for instance based on UML, are inadequate for the
often creatively oriented 3D content developers. This paper
presents a model-driven approach for the structured development of 3D applications in interdisciplinary teams based
on round-trip engineering. Abstract models of the application are specified in SSIML, a DSL tailored for the development of 3D applications. In a forward phase, consistent
skeletons of 3D scene models and program code are generated from the SSIML model. In a reverse phase, refinements of both the 3D scene model and program code are
abstracted and synchronized to result in an updated SSIML
model. This process is repeated in the subsequent iterations.
Several examples of 3D application development on diverse
B. Jung M. Lenk (B)
Virtual Reality and Multimedia, Institute of Computer Science,
Technical University Bergakademie Freiberg, Freiberg, Germany
e-mail: lenk@informatik.tu-freiberg.de
B. Jung
e-mail: jung@informatik.tu-freiberg.de
A. Vitzthum
Berufsakademie Sachsen, University of Cooperative Education,
Dresden, Germany
e-mail: arnd.vitzthum@ba-dresden.de
platforms including Web, immersive virtual reality, and augmented reality on mobile devices demonstrate the versatility
of our approach.
Keywords Model driven development Round-trip
engineering 3D applications Virtual reality Augmented
reality Model transformation Model differences Model
merging
1 Introduction
Interactive 3D applications including virtual reality (VR) and
augmented reality (AR) play a central role in many domains,
such as product visualization, entertainment, scientific visualization, training and education. So far, however, the use
of model-driven development (MDD) approaches and visual
modeling languages such as UML is far less common than
in the development of other kinds of software. Arguably, this
lack of acceptance of MDD approaches can be attributed to
the specifics of the 3D application development process.
First, 3D development is an interdisciplinary process.
Essentially, two groups of developers are involved who use
completely different tools and terminologies: 3D content
developers and programmers. Misunderstandings between
the developer groups can lead to an inconsistent system
implementation [25]. For example, if the 3D content developer does not follow the conventions for the naming of
3D objects, the programmer cannot address these objects
properly via program code. While MDD approaches may
be instrumental in avoiding such misunderstandings, general
purpose tools, for instance based on UML, may be appropriate for programmers, but are certainly inadequate for the
often creatively oriented 3D developers. More promising
123
286
123
B. Jung et al.
of the application logic. Cross-references (or interrelationships) between the 3D scene description and the application components specify the events that may be triggered by
user interaction with a 3D object and need to be handled by
the application logic. Note that as an iterative development
process is assumed, it suffices that the initial SSIML model
specifies only a rough first version of the application that can
be refined in later iterations [24].
During the forward engineering phase of the development process, the initial SSIML model is transformed to
JavaScript, C++ and X3D code skeletons. The 3D content
developers task is to model the individual 3D objects and
their composition in a single X3D scene while programmers
elaborate the auto-generated JavaScript and C++ code. In the
proposed development process, 3D designers and programmers may work concurrently, where both are using different
tools appropriate to their task. For example 3D designers
could model the 3D objects using tools like 3DS Max or
Blender, while the programmers use an IDE or a simple code
editor.
However, as software development is a non-trivial process,
the implementation of 3D and program code may be inconsistent with the current SSIML model in various ways: the
implementation could be incomplete or incorrectly reflect the
model, for example by using different names for 3D objects,
application classes or their attributes. Moreover, the implementation may also add parts to the application that may reasonably be reflected in future versions of the SSIML model.
In order to consolidate the various artifacts, the implementation code is then reverse engineered and synchronized with
the current SSIML model. For each inconsistency between
the SSIML model and the current state of the implementation it has to be decided whether the implementation is incorrect or incomplete or whether the implementation improves
the overall application in some way and should therefore
be reflected in the next version of the SSIML model. For
instance, on the one hand, say, the 3D designer may not care
too much about attribute names as only the visual appearance
of the 3D objects is usually important for his task. Here, the
mismatch should be considered as incorrectness of the implementation. On the other hand, a programmer may feel that a
class name in the SSIML model is inconsistent with the conventions of her programming language and uses a different
class name instead. Here, the changed attribute name may be
adopted in the next iteration of the SSIML model. To synchronize the various code bases and the SSIML model, we
propose two distinct approaches (Sect. 5.2), which provide
different degrees of interactive control for the responsible
software designer.
Once the implementation is synchronized with the SSIML
model, possibly resulting in updates to the model, the software developer may now further expand the SSIML model.
This completes the first cycle of the round-trip engineer-
287
123
288
B. Jung et al.
are used to trigger events within the 3D scene to notify application components within the interrelationship model. For
instance, when a SSIML object node is connected to a touch
sensor, this sensor attribute can notify an associated application class about an event, like a mouse click.
The car configuration application can be modeled using
SSIML as depicted in Fig. 3. Scene model elements which
will mainly result in the X3D code are shown in the upper
area. The scene root node has a light and viewpoint attribute
and contains the group node carGroup. This group node
contains a chassis object which in turn contains four
wheel objects and the windowpanes object. Elements of
the SSIML interrelationship model (displayed in the lower
area) will later be translated to application code. The application class MaterialConfigurator is associated with the
respective material attributes of the chassis and windowpanes
in order to change their color and transparency values. The
touch sensor attribute windowTouchSensor is connected
to the windowpanes object and triggers a CLICKED event
to the application class when the user clicks on the windowpanes. The second application class PartsConfigurator can
access the chassis object in order to replace its content, concretely to replace the four wheels by a different kind of wheel.
123
289
Listing 1 X3D code skeleton that has been generated from the SSIML model (Fig. 3). Inline nodes, such as the chassis object, are used to include
further 3D geometries. Some IDs and node attributes have been removed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
function MaterialConfigurator ( ) {
var c h a s s i s C o l o r = d o c u m e n t . g e t E l e m e n t B y I d ( c h a s s i s _ _ c h a s s i s C o l o r );
var w i n d o w T r a n s p a r e n c y = d o c u m e n t . g e t E l e m e n t B y I d ( w i n d o w p a n e s _ _ w i n d o w T r a n s p a r e n c y );
this . w i n d o w p a n e s _ C L I C K E D = f u n c t i o n ( obj ) {
};
// ! I n s e r t f u r t h e r a p p l i c a t i o n code below !//
// ! I n s e r t f u r t h e r a p p l i c a t i o n code above !//
}
function PartsConfigurator ( ) {
var c h a s s i s = r e t u r n d o c u m e n t . g e t E l e m e n t B y I d ( c h a s s i s );
...
}
f u n c t i o n init ( ) {
var m a t e r i a l C o n f i g u r a t o r = new M a t e r i a l C o n f i g u r a t o r ( );
var p a r t s C o n f i g u r a t o r = new P a r t s C o n f i g u r a t o r ( );
d o c u m e n t . g e t E l e m e n t B y I d ( w i n d o w p a n e s ). a d d E v e n t L i s t e n e r ( " c l i c k " ,
m a t e r i a l C o n f i g u r a t o r . w i n d o w p a n e s _ C L I C K E D );
...
}
Listing 2 Generated JavaScript code skeleton with functions and member variables to access the 3D scene. Comments, where IDs are stored, have
been removed for better readability
123
290
example is mapped to an HTML click event. Further application logic, for instance to set the transparency value of the
material, will be manually programmed by elaborating the
generated code stubs, possibly by also defining additional
helper functions. Like generated X3D elements, generated
JavaScript elements are also assigned IDs for tracking. The
mapping from SSIML elements to corresponding JavaScript
elements is more complex than the mapping to X3D nodes,
since multiple SSIML elements may contribute to the same
JavaScript function. For example, the event handler registration within the init() function is comprised of the concerned
object (namely windowpanes), the attached sensor and the
relationships which connect the related objects (Listing 2).
B. Jung et al.
The second platform supported by our round-trip environment is an ultra high-resolution CAVE consisting of 25 projectors (24 full HD plus 1 SXGA+) that offers a much more
immersive experience than the web platform. The user can
interactively configure the real sized car by using GUI elements on an iPad and by using a flystick for 3D interaction. The 3D scene (Sect. 3.3) is converted from X3D to the
VRML format.1 Functionalities to load, access and display
the 3D scene or to realize the event handling mechanism
are implemented with respect to corresponding libraries of a
proprietary CAVE framework. The structure of the generated
C++ application is similar to the JavaScript code (Sect. 3.4).
SSIML application classes result in respective C++ classes.
Relevant member variables and method stubs, such as the
windows_touch event handler are automatically generated.
The programmer can rely on the generated interface to access
the 3D scene in a correct manner.
During the edit phase, 3D content developers and programmers elaborate the generated code skeletons. However, edits
may occur that do not conform to the SSIML model. In
order to detect these inconsistencies and to derive an updated
SSIML model, the different code bases need to be synchronized in the reverse phase of the RTE process.
For our car configurator example we assume the following
three model-relevant changes to the code, which represent the
basic operations update, insert and delete: First, 3D content
developers rename the 3D object windowpanes to windows. Since JavaScript and C++ programmers rely on the
original naming, the windowpanes object is no longer accessible from the program code, which results in an inconsistent
In this section we give an example demonstrating the multiplatform development of a 3D application as an augmented
reality version on Android devices and a version for the web.
Augmented reality (AR), in contrast to pure virtual reality,
interactively enriches the real world with virtual 3D objects
[4]. AR applications can be found in many domains, such
as medicine, advertising, (scientific) visualizations, product configuration, gaming, safety or training simulations.
Todays pervasive and powerful mobile devices, such as
smartphones and tablet computers, with built-in cameras
have become popular platforms for AR applications. We use
123
CAVE developers may still programmatically add a high-detail background, but the background object will not be part of the platformindependent SSIML model.
291
http://www.qualcomm.com/solutions/augmented-reality, (2014).
https://github.com/MasDennis/Rajawali, (2014).
123
292
B. Jung et al.
assigning concrete marker IDs to them. Likewise, SSIML/AR models can be used to generate web or CAVE code
by simply ignoring the marker information.
Figure 7 shows the final AR application next to the web
version of the robot scenario,6 both generated from the same
SSIML model (Fig. 6). Round-trip development, as described
in Sect. 3.6, is also supported for the AR platform.
5 Implementation overview
In this section we first briefly describe the implementation of model-to-code and code-to-model transformations in
Sect. 5.1. A detailed description of the involved models and
transformations is given in [19]. Section 5.2 describes two
approaches for model and code synchronization including
conflict handling. Section 5.3 gives an overview of adapters
we use to embed our generated code structures in existing
runtimes and frameworks.
All functionalities to trigger transformations between
models and code as well as the synchronization between
them are accessible from within our Eclipse-based Roundtrip environment (Fig. 8).
Fig. 5 a 3D application in Web Browser; b CAVE-version of the application
123
http://elrond.informatik.tu-freiberg.de/roundtrip3d/RobotScene/
robot.html, (2014).
Fig. 6 The SSIML model for the AR robot application. Scene model
elements are located on the outer area while elements from the SSIML
interrelationship model are positioned on the dark gray, inner area. Both
293
123
294
B. Jung et al.
Fig. 9 Multi-platform RTE implementation with involved models and transformations. This workflow also depicts the semi-automatic synchronization process, without further user interaction, on the intermediate stage
123
(Fig. 10, outside the SSIML language) cannot cause inconsistencies or conflicts and therefore can be developed independently from each other.9
In iterative round-trip development, synchronization can
also be used as a verification method: when no inconsistencies or merge conflicts are detected during synchronization, a
consistent version of the 3D application has been developed.
Synchronization comprises four phases: comparison, conformance checking, merging and reconciliation [15]. In the
following, we propose two approaches for model synchro9
295
3D code
SSIML
(possible
specification to
implementation
inconsistencies)
Interrelationships
(possible
merge conflicts)
application code
123
296
B. Jung et al.
123
6 Discussion
Many existing RTE tools, such as UML LAB or Together,
offer support for software development with focus on simul-
297
123
298
123
B. Jung et al.
defined functions (Sect. 5.3). Formatting capabilities are provided by programming tools and our model editor.
6.1 Case study
We used our round-trip environment so far to develop several proof of concept applications. Furthermore, our tool has
been validated in an extensive student project. The goal was
to create a series of web-based 3D application presenting the
workings of several historic and for pedagogical reasons simplified pipe organs, using our round-trip environment. The
project comprised two phases, each lasting six months.
In the first project phase, we introduced the SSIML model
editor and the JavaScript and X3D code generators, to train
the students. In this phase, seven students participated with
expertise in the following areas: two students had advanced
JavaScript programming skills, three students were experienced in 3D modeling, and two students had basic training
in programming and software development. The latter two
students were assigned the role of software designers and
created the SSIML models while the other students were
used according to their core expertise in programming and,
respectively, 3D modeling. As result of the first phase, two
prototypical web applications were developed.10
For the second project phase, the students task was to
develop a complex schematical 3D organ to exemplify the
functionality of an organ in the web (Fig. 14). This should
be achieved by exploded drawings supported by animations,
which guide the user through various levels of detail, from a
rough overall view of the organ to a detailed representation
of each component. The application should be developed in
an iterative development process, using our round-trip tool.
Five students participated in the second project phase: one
student with extensive JavaScript programming skills, three
students with extensive 3D modeling skills and one student
with basic 3D modeling skills. Four of the five students were
familiar with SSIML modeling but only one student acted as
software designer.
Within six months, students completed two round-trips
and thus developed two versions of the 3D application.11
The original SSIML model specification included only one
viewpoint, which should be modified from within JavaScript
code in order to focus on the selected 3D object from a suitable perspective. However, defining viewpoint parameters is
a more appropriate task for 3D modelers, who use 3D modeling tools and have a distinct spatial awareness. Thus, 3D modelers added and defined these viewpoints. Additional changes
to the code concerned renaming of misspelled terms and the
10
11
http://vr.tu-freiberg.de/organ-visualization/organ-one, (2014).
299
Table 1 Number of model elements during forward and reverse engineering, for the car configuration example and a web-based visualization
of an organ
SSIML
IM
ASTX3D
ASTJS
elocJS
CarConfig.
Forward
27
27
13
25
13
Reverse
55
56
24
25
57
Organ Vis.
Forward
221
221
137
850
342
Reverse
333
340
144
847
752
The number of effective lines of code (eloc) is listed for JavaScript (JS)
7 Conclusion
deletion of the original viewpoint. The size of the final application is about 240 MB.12
Due to generated X3D and JavaScript code structures,
3D modeling and application programming could be performed concurrently. Furthermore, the JavaScript developer
was even able to finish his task before 3D developers finished 3D modeling. Therefore, implementation time could
significantly be reduced.
As a further result, we could show that our approach
also scales well with large model sizes and applications.
Table 1 gives an overview of the number of elements contained in each model during one round-trip iteration. The
SSIML model of the complex organ example proved as still
manageable. Transformations between models and text and
model merging are performed within several seconds.13 The
number of effective lines of code (eloc) for JavaScript contains no comments and at least three characters per line. For
X3D, the number of AST elements equals the number of
XML tags and is a more meaningful descriptor than code
statistics.
Furthermore, the students feedback on the semi-automatic merge process was crucial to initiate and inform the devel12
13
http://vr.tu-freiberg.de/organ-visualization, (2014).
123
300
References
1. Alanen M, Porres I (2003) Difference and union of models. In:
Stevens P, Whittle J, Booch G (eds) UML 2003the unified
modeling language, vol 2863. Lecture notes in computer science.
Springer, Berlin, Heidelberg, pp 217
2. Antkiewicz M (2007) Round-trip engineering using frameworkspecific modeling languages. In: Companion to the 22nd ACM
SIGPLAN conference on Object-oriented programming systems
and applications companion, ACM, pp 927928
3. Amann U (2003) Automatic roundtrip engineering. Electr Notes
Theor Comput Sci 82(5), proceedings of the Fifth Workshop on
quantitative aspects of programming languages (QAPL 2007)
4. Azuma RT et al (1997) A survey of augmented reality. Presence
6(4):355385
5. Behr J, Eschler P, Jung Y, Zllner M (2009) X3DOM: a dom-based
html5/x3d integration model. In: Spencer SN, Fellner DW, Behr J,
Walczak K (eds) Web3D. ACM, pp 127135
6. Brun C, Pierantonio A (2008) Model differences in the eclipse modelling framework. UPGRADE Eur J Inf Prof IX(2):2934. http://
www.cepis.org/upgrade/files/2008-II-pierantonio.pdf
7. Bruneliere H, Cabot J, Jouault F, Madiot F (2010) Modisco: a
generic and extensible framework for model driven reverse engineering. In: Proceedings of the IEEE/ACM international conference on automated software engineering. ACM, pp 173174
8. Dachselt R, Hinz M, Meissner K (2002) Contigra: an XML-based
architecture for component-oriented 3D applications. In: Proceedings of the seventh international conference on 3D Web technology.
ACM, Web3D 02, pp 155163. doi:10.1145/504502.504527
9. Efftinge S, Vlter M (2006) oAW xText: a framework for textual
DSLs. In: Eclipsecon Summit Europe 2006. http://www.eclipse.
org/Xtext
10. Figueroa P, Green M, Hoover HJ (2002) InTml: a description language for VR applications. Web3D 02: proceedings of the seventh
international conference on 3D Web technology. ACM, New York,
pp 5358
11. Johannes J, Samlaus R, Seifert M (2009) Round-trip support for
invasive software composition systems. In: Bergel A, Fabry J (eds)
Software composition. Lecture notes in computer science, vol
5634. Springer, pp 90106. http://dblp.uni-trier.de/db/conf/soco/
sc2009.html#JohannesSS09
12. Jones J (2003) Abstract syntax tree implementation idioms. In:
Pattern languages of program design proceedings of the 10th Conference on pattern languages of programs (PLoP2003)
13. Kehrer T, Kelter U, Pietsch P, Schmidt M (2012) Adaptability of
model comparison tools. In: Proceedings of the 27th IEEE/ACM
International Conference on automated software engineering.
ACM, New York, NY, USA, ASE 2012, pp 306309. doi:10.1145/
2351676.2351731. http://doi.acm.org/10.1145/2351676.2351731
14. Koegel M, Helming J (2010) EMFStore: a model repository for
EMF models. In: Proceedings of the 32Nd ACM/IEEE International Conference on software engineering, vol 2. ACM, New York,
ICSE 10, pp 307308. doi:10.1145/1810295.1810364. http://doi.
acm.org/10.1145/1810295.1810364
15. Kolovos D, Paige R, Polack F (2006) Merging models with the
Epsilon merging language (EML). In: Nierstrasz O, Whittle J, Harel
D, Reggio G (eds) Model driven engineering languages and systems, vol 4199. Springer, Berlin, Heidelberg, chap 16, pp 215229.
doi:10.1007/11880240_16
16. Kolovos DS, Paige RF, Polack F (2008) The epsilon transformation language. In: Vallecillo A, Gray J, Pierantonio A (eds)
Theory and practice of model transformations, First International
Conference, ICMT 2008, Z++rich, Switzerland, July 12, 2008,
Proceedings, vol 5063. Springer, LNCS, pp 4660. doi:10.1007/
978-3-540-69927-9_4
123
B. Jung et al.
17. Kolovos DS, Di Ruscio D, Pierantonio A, Paige RF, (2009) Different models for model matching: an analysis of approaches to
support model differencing. In: Comparison and versioning of software models, (2009) CVSM09. ICSE Workshop on, IEEE, pp 16
18. Ledermann F, Schmalstieg D (2005) April a high-level framework
for creating augmented reality presentations. In: Proceedings of the
2005 IEEE Conference 2005 on virtual reality. IEEE, VR 05, pp
187194
19. Lenk M, Schlegel C, Vitzthum A, Jung B (2012a) Roundtrip engineering for 3D applications: models and transformations. Preprint 6/2012, Faculty of Mathematics and Informatics, TU Bergakademie Freiberg. http://www.mathe.tu-freiberg.de/
forschung/preprints
20. Lenk M, Vitzthum A, Jung B (2012b) Non-simultaneous round-trip
engineering for 3D applications. In: Proceedings of the 2012 International Conference on software engineering research and practice,
SERP
21. Steinberg D, Budinsky F, Paternostro M, Merks E (2009) EMF:
eclipse modeling framework, 2nd edn. Addison-Wesley, Boston
22. Van Paesschen E, De Meuter W, DHondt M (2005) Selfsync: a
dynamic round-trip engineering environment. In: Companion to
the 20th annual ACM SIGPLAN conference on object-oriented
programming, systems, languages, and applications, ACM, New
York, NY, USA, OOPSLA 05, pp 146147. doi:10.1145/1094855.
1094906
23. Vitzthum A, Hussmann H (2006) Modeling augmented reality user
interfaces with SSIML/AR. J Multimed 1(3):1322
24. Vitzthum A, Jung B (2010) Iterative model driven VR and AR
development with round trip engineering. In: Proceedings of
SEARIS Workshop at the IEEE virtual reality 2010 Conference,
Shaker
25. Vitzthum A, Pleu A (2005) SSIML: designing structure and application integration of 3D scenes. In: Proceedings of the tenth international conference on 3D Web technology, ACM, New York, NY,
USA, Web3D 05, pp 917. doi:10.1145/1050491.1050493
301
123