LTE CONFORMANCE TESTING: EXPERIENCES IN A LONG TERM PROJECT

TTCN-3 User Conference 2011 Bled, Slovenia

Presented by MCC TF160: Wolfgang Seka
Authors: Wolfgang Seka, Hellen Griffiths, Shicheng Hu
wolfgang.seka@etsi.org , hellen.griffiths@etsi.org , shicheng.hu@etsi.org
ETSI 2011. All rights reserved

LTE Conformance Testing:

Experiences in a long term project

The Project
Technical Issues
Common Aspects of Long Term Projects
Conclusions

ETSI 2011. All rights reserved

THE PROJECT
3

ETSI 2011. All rights reserved

MCC TF160 - General

Task Force Mobile Competence Centre: Project Group at ETSI

Pool of TTCN expertise used by 3GPP

3GPP: 3rd Generation Partnership Project (http://www.3gpp.org)

Telecommunication Standardisation Bodies
TSG RAN: Radio Access Network
WG RAN5: Mobile terminal conformance testing

Conformance Tests

Specification (Prose): RAN5

Implementation (TTCN): MCC TF160
Verification and Validation: Test Industry

MCC TF160: Signalling Conformance Tests for 3GPP (RAN5: Testing)

Task: Develop Conformance Test Suites for UE world-wide certification

since 2000
Conformance Tests for UMTS Signalling (TTCN-2)
since 2006
2007..2008
2008..now

ETSI 2011. All rights reserved

Conformance Tests for IMS (TTCN-3)

Pre-evaluation of TTCN-3 for LTE Signalling
3GPP LTE/SAE UE Conformance Test

MCC TF160 LTE/SAE Project

Size:
Duration:
Deliveries
Test cases:

17 experts all over the world

since 2008 and at least until 2015 (LTE-advanced will follow)
2011: 8 deliveries planned
264 approved
391 fully implemented
468 in total
80..100 planned for IMS

Code size:
nearly 200 Modules
nearly 190 000 Lines of code (TTCN-3)
Type Definitions
25 TTCN-3 modules, 3 ASN.1 modules, 6 XML modules
TTCN-3: > 17 500 lines of code
ASN.1: > 38 000 lines of code
XML:
< 1 000 lines of code

Tools:
6 different compilers (all available at ETSI)
quality checks (naming conventions, template restrictions etc.)
code generation (top-level test case definitions, parameters, etc.)
code analysis (structure, approved objects)
TTCN-3 code is officially published and widely used
5

ETSI 2011. All rights reserved

Component Structure - 1
MTC
Start of PTCs, monitoring of done and killed
RAT independent interfaces (e.g. AT-/MMI-commands to control the UE)
In general no pass/fail verdicts
PTCs for each RAT (radio access technology)
LTE, UMTS, GSM/GPRS, CDMA2000
May be connected to any other RAT PTC
Only RAT specific interfaces
Assignment of pass/fail verdicts
PTCs for other protocols
IP data (e.g. DHCP, ICMPv6), IMS and protocol layer underneath LTE PTC (NAS Emulator)
To keep system interface simple and deal with parallel behaviour
In general no test characteristic ( no pass/fail verdicts)
Ports and interfaces
Connections: only one-to-one

( no send to, receive from; no use of address data type)

No duplicated interfaces: e.g. only one interface for AT-/MMI-commands hosted by the MTC
Message based communication only

ETSI 2011. All rights reserved

7 7

UTRAN
PTC

Coordination

C2K_Ut

CDMA2000
PTC

Upper Tester
(AT , MMI)

G_Ut

DRB

SYSIND

SYS

NASCTRL

SRB

GERAN
PTC

CTRL

TC_SRB

E_Ut

C2K

GERAN
PTC

UT

(LTE)

GERAN

EUTRA PTC

CDMA2000
PTC

UTRAN

Component Structure - 2

MTC

Configuration

U_Ut

Signalling

NAS Emulator

E_DRB

E_SYSIND

E_SYS

SYS_SRB

Ut

E_SRB

Ut

System Interface
ETSI 2011. All rights reserved

UTRAN
PTC

User Data

Test System
Test Control,
Logging

Host PC

SUT

System Simulator HW
RF

complex configuration
delay of messages
no matter what test
purpose is

Test Executable

UE

system

Codec

System/
Platform
Adaptor

specific

AT/MMI

e.g. requirements regarding real-time behaviour for System Simulator and TTCN-3 code
8

ETSI 2011. All rights reserved

TECHNICAL ISSUES
9

ETSI 2011. All rights reserved

Real Time Issues - 1

Requirements

Time resolution at the air interface: 1ms

Due to significant delay between air interface and test case executable:

Timing needs to be monitored/controlled at the air interface (not at the TRI)

All test equipment shall have similar behaviour at any time
Test results shall not depend on
how fast or slow a UE or system simulator is
specific combinations of UEs and system simulators

System Restrictions

Delay by codec and matching: up to some milliseconds;

typically < 5ms

Host system: typically Windows or Linux
no real-time OS
Architecture: Host-PC + System Simulator

Estimation

delay between test executable and air interface:

1ms < delay < 80ms

10

ETSI 2011. All rights reserved

Real Time Issues - 2

Impact on System Interface
LTE timing information
is part of all LTE primitives
is provided by the System Simulator for all received messages
is used to schedule sending of messages in the system simulator
can be retrieved from the system simulator

Impact on writing TTCN-3

Scheduled and non-scheduled sending shall not be mixed up
When there is no reaction (e.g. from the UE) it is hard to know in the TTCN code when the
message has been sent out
Scheduling vs. TTCN timers
TTCN timers are not as accurate as scheduling
Rules are required when to use TTCN timers or scheduled sending

Restrictions and Issues

No common system time: uncorrelated timing between different RATs
Legacy primitive definitions for UMTS and GSM/GPRS: less support of timing information
Short duration only: For LTE, wrap-around after 10s
No bell mechanism (yet):

to schedule a wake-up call in the system simulator to send back a trigger after a given time

11

ETSI 2011. All rights reserved

Race Conditions
Race conditions occur when

messages appear at the same time at different ports

messages appear in any order at one port

Impact on TTCN implementation and Design considerations

TTCN implementation

interleave (in general not more than two messages)

check operation
evaluation of timing information
Clever port definitions (few ports rather than many)
Component structure
(e.g. signalling and data at different PTCs)

12

ETSI 2011. All rights reserved

COMMON ASPECTS OF LONG TERM PROJECTS

13

ETSI 2011. All rights reserved

Extendibility
Extensions of TTCN-3 code

Modification of common objects shall not cause changes in the whole test suite

reduced maintenance effort

Impact on writing TTCN-3
foresighted TTCN implementation
introduction of additional optional parameters
Example:
template MyType c_MyType :=
{
field1 := 1,
field2 := 2
};

template MyType c_MyType(integer p_Int := 1) :=

{
field1 := p_Int,
field2 := 2
};

Note: Optional Parameters vs. Modified Templates

Optional parameters need to be added to modified templates too

modification of templates and addition of optional parameters may serve the
same purpose
possible conflicts, potential contradiction of both approaches

14

ETSI 2011. All rights reserved

Backward Compatibility - 1
Backward Compatibility at the System Interface

Modifications are caused by

Release upgrade for signalling messages may have impact on configuration of

the system simulator
New requirements (due to new test cases)
Corrections

Backward Compatibility Aspects

Old TTCN-3 code on new system simulator implementation

New TTCN-3 code on old system simulator implementation

Impact

system adaptor and codec

interface definition

top-level records or unions (even when not needed in the first place)
definitions of place holders

15

ETSI 2011. All rights reserved

Backward Compatibility - 2
Examples: Backward Compatibility at System Interface
type union MyUnion_Type := {
R8_Type R8

};

type union MyUnion_Type := {

R8_Type R8,
R9_Type R9
};

New branch: branches are mutual exclusive; requires new templates

type record MyRecord_Type := {

R8_Type R8Only
};

type record MyRecord_Type := {

R8_Type R8Only,
R9Extension_Type R9Ext optional
};

Extension: templates may be enhanced by optional parameter

16

ETSI 2011. All rights reserved

Tools
Large, long-term project

code reviews and manual checks are not sufficient anymore

Requirements

Quality checks
Approved objects: objects used by approved test cases shall not be changed

without change request

Analysis of module dependencies
Recursive selection

Enhanced tool support

17

Quality checks by T3Q (on behalf of ETSI)

Common tool to get dependencies based on T3D (on behalf of ETSI)
Project-specific front-ends (approved objects, recursive selection)
Further project specific tools:
replace tabs, find non-ASCII characters in comments

ETSI 2011. All rights reserved

CONCLUSIONS
18

ETSI 2011. All rights reserved

Conclusions (TTCN-3 language)

Important TTCN-3 language features

optional parameters

backward compatibility, maintenance

template restrictions

in combination with naming conventions and appropriate tools

no problems with receive templates used in send direction anymore

encvalue/decvalue

maintenance (no external functions needed)

check operation, interleave

race conditions

testcase.stop

well-defined termination of a test case in case of unrecoverable error situations

visibility: "private", "friend

not used yet (may improve quality regarding module dependencies)

19

ETSI 2011. All rights reserved

Common Conclusions

Feedback of long-terms projects to the core language is vital

Tool Compatibility is needed
Shortened release cycle of TTCN-3 standards

TTCN-3 development is SW Development

SW Engineering
System Engineering
Long-term characteristic of a project needs to be considered from the beginning
Conceptual errors will be expensive
Tool support is vital
Common and project-specific tools

20

ETSI 2011. All rights reserved

ADDITIONAL SLIDES
21

ETSI 2011. All rights reserved

Release Upgrades - 1
Release Upgrade for Protocol and Signalling
(Baseline Moving)

Typically every year

In general changes are backward compatible
Critical and non-critical extensions (TS 36.331 Annex A)
Critical extensions

redefinition of message content or parts of it

(branching, e.g. r8 and r9 branch)

sender needs to know which release the receiver supports

Non-critical extensions

Provision of additional information

e.g. new fields at the end of a record, new enumerated value
when not supported receiver shall skip the information
Messages in general contain hooks for critical and non-critical
extensions

22

ETSI 2011. All rights reserved

Release Upgrades - 2
Impact on writing TTCN-3

Non-critical extensions

In general no code duplication is necessary

Enhancement of existing templates by new optional parameters
(initialised with * for receiving and omit for sending)

Critical extensions

New templates are needed

Challenge: minimised code duplication
Example:

template of a plain RRC PDU for UMTS

up to several hundred lines of code
clever template structure reduces duplication of code
parameterised templates
modified templates

23

ETSI 2011. All rights reserved