ROSE Tutorial: A Tool for Building Source-to-Source Translators Draft Tutorial (version 0.9.

5a)
Daniel Quinlan, Markus Schordan, Richard Vuduc, Qing Yi Thomas Panas, Chunhua Liao, and Jeremiah J. Willcock Lawrence Livermore National Laboratory Livermore, CA 94550 925-423-2668 (oce) 925-422-6278 (fax) {dquinlan,panas2,liao6}@llnl.gov markus@complang.tuwien.ac.at qingyi@cs.utsa.edu richie@cc.gatech.edu jewillco@osl.iu.edu Project Web Page: www.rosecompiler.org UCRL Number for ROSE User Manual: UCRL-SM-210137-DRAFT UCRL Number for ROSE Tutorial: UCRL-SM-210032-DRAFT UCRL Number for ROSE Source Code: UCRL-CODE-155962 ROSE User Manual (pdf) ROSE Tutorial (pdf) ROSE HTML Reference (html only) July 8, 2013

ii July 8, 2013

Contents
1 Introduction 1.1 What is ROSE . . . . . . . . . . . . . . . . . 1.2 Why you should be interested in ROSE . . . 1.3 Problems that ROSE can address . . . . . . . 1.4 Examples in this ROSE Tutorial . . . . . . . 1.5 ROSE Documentation and Where To Find It 1.6 Using the Tutorial . . . . . . . . . . . . . . . 1.7 Required Makele for Tutorial Examples . . . 1 1 2 2 3 10 11 11

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

Working with the ROSE AST

13
15 19 23 29 31 35 35 36 37 37 37 38 45 47 48 51 54 56

2 Identity Translator 3 Simple AST Graph Generator 4 AST Whole Graph Generator 5 Advanced AST Graph Generation 6 AST PDF Generator 7 Introduction to AST Traversals 7.1 Input For Example Traversals . . . . . . . . . . . . . . . . . 7.2 Traversals of the AST Structure . . . . . . . . . . . . . . . . 7.2.1 Classic Object-Oriented Visitor Pattern for the AST 7.2.2 Simple Traversal (no attributes) . . . . . . . . . . . 7.2.3 Simple Pre- and Postorder Traversal . . . . . . . . . 7.2.4 Inherited Attributes . . . . . . . . . . . . . . . . . . 7.2.5 Synthesized Attributes . . . . . . . . . . . . . . . . . 7.2.6 Accumulator Attributes . . . . . . . . . . . . . . . . 7.2.7 Inherited and Synthesized Attributes . . . . . . . . . 7.2.8 Persistent Attributes . . . . . . . . . . . . . . . . . . 7.2.9 Nested Traversals . . . . . . . . . . . . . . . . . . . . 7.2.10 Combining all Attributes and Using Primitive Types iii

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

iv 7.2.11 Combined Traversals . . . . . . . . . . . . . . . . . . . . . 7.2.12 Short-Circuiting Traversals . . . . . . . . . . . . . . . . . Memory Pool Traversals . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 ROSE Memory Pool Visit Traversal . . . . . . . . . . . . 7.3.2 Classic Object-Oriented Visitor Pattern for Memory Pool 7.3.3 ROSE IR Type Traversal (uses Memory Pools) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

CONTENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 63 66 66 68 70 73 73 77 77 78 81 81 81 87 87 87 87 87 91 91 92 92

7.3

8 Graph Processing Tutorial 8.1 Traversal Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Scopes of Declarations 9.1 Input For Examples Showing Scope Information . . . . . . . . . . . . . . . . . . 9.2 Generating the code representing any IR node . . . . . . . . . . . . . . . . . . . . 10 AST Query 10.1 Simple Queries on the AST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Nested Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 AST File I/O 11.1 Source Code for File I/O . . . . . . . . . . . 11.2 Input to Demonstrate File I/O . . . . . . . 11.3 Output from File I/O . . . . . . . . . . . . 11.4 Final Code After Passing Through File I/O

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

12 Debugging Techniques 12.1 Input For Examples Showing Debugging Techniques . . . . . . . . . . . . . . . . 12.2 Generating the code from any IR node . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Displaying the source code position of any IR node . . . . . . . . . . . . . . . . .

II

Complex Types

95
97 97 98 101 105 109

13 Type and Declaration Modiers 13.1 Input For Example Showing use of Volatile type modier . . . . . . . . . . . . . 13.2 Generating the code representing the seeded bug . . . . . . . . . . . . . . . . . . 14 Function Parameter Types 15 Resolving Overloaded Functions 16 Template Parameter Extraction

17 Template Support 111 17.1 Example Template Code #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 17.2 Example Template Code #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

CONTENTS

III

Program Analyses
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

113
115 115 117 117 117 119 128 130 133 137 137 138 138 138 139 146 146 146 146 146 146 146 147 147 147 148 148 151

18 Generic Dataow Analysis Framework 18.1 Basics of DataFlowAnalysis . . . . . . . . . . . . . 18.2 ROSE Dataow Framework . . . . . . . . . . . . . 18.2.1 Call and Control-Flow Graphs . . . . . . . 18.2.2 Analyses . . . . . . . . . . . . . . . . . . . . 18.2.3 Dataow . . . . . . . . . . . . . . . . . . . 18.2.4 Transferring Information Between Analyses 18.2.5 CFG Transformations . . . . . . . . . . . . 19 Recognizing Loops 20 Virtual CFG 20.1 CFGNode Index values . . . . . . . . . . . . 20.2 Important functions . . . . . . . . . . . . . 20.2.1 Node methods . . . . . . . . . . . . 20.2.2 Edge methods . . . . . . . . . . . . . 20.3 Drawing a graph of the CFG . . . . . . . . 20.4 Robustness to AST changes . . . . . . . . . 20.5 Limitations . . . . . . . . . . . . . . . . . . 20.5.1 Fortran support . . . . . . . . . . . 20.5.2 Exception handling . . . . . . . . . . 20.5.3 Interprocedural control ow analysis 20.6 Node ltering . . . . . . . . . . . . . . . . . 20.6.1 Interesting node lter . . . . . . . 20.6.2 Arbitrary ltering . . . . . . . . . . 20.7 Static CFG . . . . . . . . . . . . . . . . . . 20.7.1 Class methods . . . . . . . . . . . . 20.7.2 Drawing a graph of the CFG . . . . 20.8 Static, Interprocedural CFGs . . . . . . . . 21 Generating Control Flow Graphs

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

22 Graph Processing Tutorial 155 22.1 Traversal Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 23 Dataow Analysis 23.1 Def-Use Analysis . . . . . . . . . . . . . 23.1.1 Def-use Example implementation 23.1.2 Accessing the Def-Use Results . 23.2 Liveness Analysis . . . . . . . . . . . . . 23.2.1 Access live variables . . . . . . . 24 Generating the Call Graph (CG) 25 Dataow Analysis based Virtual Function Analysis 159 159 159 161 163 163 169 173

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

vi 26 Generating the Class Hierarchy Graph

CONTENTS 177

27 Database Support 181 27.1 ROSE DB Support for Persistent Analysis . . . . . . . . . . . . . . . . . . . . . . 181 27.2 Call Graph for Multi-le Application . . . . . . . . . . . . . . . . . . . . . . . . . 181 27.3 Class Hierarchy Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 28 Building Custom Graphs 187

IV

Program Transformations and Optimizations

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

189
191 192 192 193 193 193

29 Generating Unique Names for Declarations 29.1 Example Code Showing Generation of Unique Names . . . . . . . . . 29.2 Input For Examples Showing Unique Name Generation for Variables 29.3 Example Output Showing Unique Variable Names . . . . . . . . . . 29.4 Input For Examples Showing Unique Name Generation for Functions 29.5 Example Output Showing Unique Function Names . . . . . . . . . .

30 Command-line Processing Within Translators 199 30.1 Commandline Selection of Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 31 Tailoring The Code Generation Format 31.1 Source Code for Example that Tailors the Code Generation . . . . . . . . . . . . 31.2 Input to Demonstrate Tailoring the Code Generation . . . . . . . . . . . . . . . . 31.3 Final Code After Tailoring the Code Generation . . . . . . . . . . . . . . . . . . 32 AST Construction 32.1 Variable Declarations . . . . . . . . . . . 32.2 Expressions . . . . . . . . . . . . . . . . 32.3 Assignment Statements . . . . . . . . . 32.4 Functions . . . . . . . . . . . . . . . . . 32.5 Function Calls . . . . . . . . . . . . . . 32.6 Creating a struct for Global Variables . 203 203 203 203 207 207 211 213 215 220 220 233 234 235 236 236 238

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

33 Parser Building Blocks 33.1 Grammar Examples . . . . . . . . . . . . . . . 33.2 AstAttribute to Store results . . . . . . . . . . 33.3 The AstFromString Namespace . . . . . . . . . 33.4 Write your parsers using parser building blocks 33.5 Limitations . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

34 Handling Comments, Preprocessor Directives, And Adding Arbitrary Text to Generated Code 239 34.1 How to Access Comments and Preprocessor Directives . . . . . . . . . . . . . . . 239 34.1.1 Source Code Showing How to Access Comments and Preprocessor Directives240 34.1.2 Input to example showing how to access comments and CPP directives . 240

CONTENTS

vii

34.1.3 Comments and CPP Directives collected from source le (skipping headers)240 34.1.4 Comments and CPP Directives collected from source le and all header les240 34.2 Collecting #dene C Preprocessor Directives . . . . . . . . . . . . . . . . . . . . 240 34.2.1 Source Code Showing How to Collect #dene Directives . . . . . . . . . . 240 34.2.2 Input to example showing how to access comments and CPP directives . 242 34.2.3 Comments and CPP Directives collected from source le and all header les243 34.3 Automated Generation of Comments . . . . . . . . . . . . . . . . . . . . . . . . . 243 34.3.1 Source Code Showing Automated Comment Generation . . . . . . . . . . 244 34.3.2 Input to Automated Addition of Comments . . . . . . . . . . . . . . . . . 244 34.3.3 Final Code After Automatically Adding Comments . . . . . . . . . . . . . 244 34.4 Addition of Arbitrary Text to Unparsed Code Generation . . . . . . . . . . . . . 244 34.4.1 Source Code Showing Automated Arbitrary Text Generation . . . . . . . 244 34.4.2 Input to Automated Addition of Arbitrary Text . . . . . . . . . . . . . . 245 34.4.3 Final Code After Automatically Adding Arbitrary Text . . . . . . . . . . 245 35 Partial Redundancy Elimination (PRE) 253 35.1 Source Code for example using PRE . . . . . . . . . . . . . . . . . . . . . . . . . 253 35.2 Input to Example Demonstrating PRE . . . . . . . . . . . . . . . . . . . . . . . . 254 35.3 Final Code After PRE Transformation . . . . . . . . . . . . . . . . . . . . . . . . 255 36 Calling the Inliner 257 36.1 Source Code for Inliner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 36.2 Input to Demonstrate Function Inlining . . . . . . . . . . . . . . . . . . . . . . . 257 36.3 Final Code After Function Inlining . . . . . . . . . . . . . . . . . . . . . . . . . . 257 37 Using the AST Outliner 37.1 An Outlining Example . . . . . . . . . . . . . . . . . . . . 37.2 Limitations of the Outliner . . . . . . . . . . . . . . . . . 37.3 User-Directed Outlining via Pragmas . . . . . . . . . . . . 37.4 Outlining via Abstract Handles . . . . . . . . . . . . . . . 37.5 Calling Outliner Directly on AST Nodes . . . . . . . . . . 37.5.1 Selecting the outlineable if statements . . . . . . . 37.5.2 Properly ordering statements for in-place outlining 37.6 Outliners Preprocessing Phase . . . . . . . . . . . . . . . 261 261 262 264 264 266 267 267 271 277 277 280 282 282 285 287 289 291

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

38 Loop Optimization 38.1 Example Loop Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . 38.2 Matrix Multiply Example . . . . . . . . . . . . . . . . . . . . . . . . . 38.3 Loop Fusion Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.4 Example Loop Processor (LoopProcessor.C) . . . . . . . . . . . . . . . 38.5 Matrix Multiplication Example (mm.C) . . . . . . . . . . . . . . . . . 38.6 Matrix Multiplication Example Using Linearized Matrices (dgemm.C) 38.7 LU Factorization Example (lufac.C) . . . . . . . . . . . . . . . . . . . 38.8 Loop Fusion Example (tridvpk.C) . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

viii

CONTENTS

39 Parameterized Code Translation 293 39.1 Loop Unrolling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 39.2 Loop Interchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 39.3 Loop Tiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

Correctness Checking

299
301

40 Code Coverage

41 Bug Seeding 309 41.1 Input For Examples Showing Bug Seeding . . . . . . . . . . . . . . . . . . . . . . 309 41.2 Generating the code representing the seeded bug . . . . . . . . . . . . . . . . . . 310

VI

Binary Support

313

42 Instruction Semantics 315 42.1 The FindConstantsPolicy Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 42.2 Sample Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 42.3 Building on Instruction Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . 321 43 Binary Analysis 43.1 The ControlFlowGraph . . . . . . . . . . . . . . . . . . . . . . . . 43.2 DataFlow Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.2.1 Def-Use Analysis . . . . . . . . . . . . . . . . . . . . . . . . 43.2.2 Variable Analysis . . . . . . . . . . . . . . . . . . . . . . . . 43.3 Dynamic Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.4 Analysis and Transformations on Binaries . . . . . . . . . . . . . . 43.4.1 Source-to-source transformations to introduce NOPs . . . . 43.4.2 Detection of NOP sequences in the binary AST . . . . . . . 43.4.3 Transformations on the NOP sequences in the binary AST 43.4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Binary Construction 44.1 Constructors . . . . . . . . . . . . . . . . . 44.2 Read-Only Data Members . . . . . . . . . . 44.3 Constructing the Executable File Container 44.4 Constructing the ELF File Header . . . . . 44.5 Constructing the ELF Segment Table . . . 44.6 Constructing the .text Section . . . . . . . . 44.7 Constructing a LOAD Segment . . . . . . . 44.8 Constructing a PAX Segment . . . . . . . . 44.9 Constructing a String Table . . . . . . . . . 44.10Constructing an ELF Section Table . . . . . 44.11Allocating Space . . . . . . . . . . . . . . . 44.12Produce a Debugging Dump . . . . . . . . . 323 323 323 323 325 325 326 326 328 329 329 333 333 333 334 334 335 335 337 338 338 338 339 339

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

CONTENTS

ix

44.13Produce the Executable File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 45 Dwarf Debug Support 341 45.1 ROSE AST of Dwarf IR nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 45.2 Source Position to Instruction Address Mapping . . . . . . . . . . . . . . . . . . 342

VII

Interacting with Other Tools

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

347
349 350 350 351 352 352 358 361 363 363 370 370 370 370 371 372

46 Abstract Handles to Language Constructs 46.1 Use Case . . . . . . . . . . . . . . . . . . . 46.2 Syntax . . . . . . . . . . . . . . . . . . . . . 46.3 Examples . . . . . . . . . . . . . . . . . . . 46.4 Reference Implementation . . . . . . . . . . 46.4.1 Connecting to ROSE . . . . . . . . . 46.4.2 Connecting to External Tools . . . . 46.5 Summary . . . . . . . . . . . . . . . . . . .

47 ROSE-HPCToolKit Interface 47.1 An HPCToolkit Example Run . . . . . . . . . . 47.2 Attaching HPCToolkit Data to the ROSE AST 47.2.1 Calling ROSE-HPCT . . . . . . . . . . 47.2.2 Retrieving the attribute values . . . . . 47.2.3 Metric propagation . . . . . . . . . . . . 47.3 Working with GNU gprof . . . . . . . . . . . . 47.4 Command-line options . . . . . . . . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

48 TAU Instrumentation 377 48.1 Input For Examples Showing Information using Tau . . . . . . . . . . . . . . . . 377 48.2 Generating the code representing any IR node . . . . . . . . . . . . . . . . . . . . 377 49 The Haskell Interface 381 49.1 Traversals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382 49.2 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382

VIII

Parallelism

385
387 391

50 Shared-Memory Parallel Traversals 51 Distributed-Memory Parallel Traversals

52 Parallel Checker 395 52.1 Dierent Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 52.2 Running through PSUB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 53 Reduction Recognition 397

CONTENTS

IX

Tutorial Summary

399
401

54 Tutorial Wrap-up

Appendix 403 54.1 Location of To Do List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 54.2 Abstract Grammar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Glossary 411

List of Figures
1.1 Example Makefile showing how to use an installed version of ROSE (generated by make install). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Source code for translator to read an input program and generate an object code (with no translation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to identity translator. . . . . . . . . . . . . . Generated code, from ROSE identity translator, sent to the backend (vendor) compiler. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code to read an input program and generate an AST graph. . . . Example source code used as input to generate the AST graph. . . . . . . . . . . AST representing the source code le: inputCode ASTGraphGenerator.C. . . . . Example source code to read an input program and generate a whole AST graph. Example tiny source code used as input to generate the small AST graph with attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AST representing the tiny source code le: inputCode wholeAST 1.C. This graphs shows types, symbols, and other attributes that are dened on the attributed AST. Example source code used as input to generate a larger AST graph with attributes. AST representing the small source code le: inputCode wholeAST 2.C. This graph shows the signicantly greater internal complexity of a slightly larger input source code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code to read an input program and generate a PDF le to represent the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to generate the PDF le of the AST. . . . . . Example output from translator which outputs PDF representation of AST. The generated PDF le makes use of the bookmark mechanism to expand and collapse parts of the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code used as input to program in traversals shown in this chapter. Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . Output of input le to the visitor pattern traversal over the memory pools. . . . Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . xi

12

2.1 2.2 2.3

16 16 17 20 20 21 24 24 25 26

3.1 3.2 3.3 4.1 4.2 4.3 4.4 4.5

27

6.1 6.2 6.3

31 32

33 36 38 40 41

7.1 7.2 7.3 7.4

xii 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15 7.16 7.17 7.18 7.19 7.20 7.21 7.22 7.23 7.24 7.25 7.26 7.27 7.28 7.29 7.30 7.31 7.32 7.33 7.34 7.35 8.1 8.2

LIST OF FIGURES Output of input le to the visitor traversal. . . . . . . . . . . . . . . . . . . . . . Example source showing simple pre- and postorder pattern. . . . . . . . . . . . . Output of input le to the pre- and postorder traversal. . . . . . . . . . . . . . . Example source code showing use of inherited attributes (passing context information down the AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the inherited attribute traversal. . . . . . . . . . . . . . . Example source code showing use of synthesized attributed (passing analysis information up the AST). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the synthesized attribute traversal. . . . . . . . . . . . . . Example source code showing use of accumulator attributes (typically to count things in the AST). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the accumulator attribute traversal. . . . . . . . . . . . . Example source code showing use of both inherited and synthesized attributes working together (part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the inherited and synthesized attribute traversal. . . . . . Example source code showing use of persistent attributes used to pass information across multiple passes over the AST. . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the persistent attribute traversal showing the passing of information from one AST traversal to a second AST traversal. . . . . . . . . . . Example source code showing use nested traversals. . . . . . . . . . . . . . . . . . Output of input le to the nested traversal example. . . . . . . . . . . . . . . . . Input code with nested loops for nesting info processing . . . . . . . . . . . . . . Example source code showing use of inherited, synthesized, accumulator, and persistent attributes (part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code showing use of inherited, synthesized, accumulator, and persistent attributes (part 2). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output code showing the result of using inherited, synthesized, and accumulator attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source showing the combination of traversals. . . . . . . . . . . . . . . . Output of input le to the combined traversals. Note that the order of outputs changes as execution of several analyzers is interleaved. . . . . . . . . . . . . . . Input code with used to demonstrate the traversal short-circuit mechanism. . . . Example source code showing use of short-circuit mechanism to avoid traversal of full AST. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output code showing the result of short-circuiting the traversal. . . . . . . . . . Example source showing simple visit traversal over the memory pools. . . . . . . Output of input le to the visitor traversal over the memory pool. . . . . . . . . Example source showing simple visitor pattern. . . . . . . . . . . . . . . . . . . . Output of input le to the visitor pattern traversal over the memory pools. . . . Example source showing simple visit traversal over each type of IR node (one only) in the memory pools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Output of input le to the IR Type traversal over the memory pool. . . . . . . . Example of output using -rose:verbose 2 (memory use report for AST). . . . . . . Source CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . Binary CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 42 42 43 44 45 46 47 48 49 50 52 53 54 55 56 58 59 60 61 62 63 64 65 67 67 68 69 71 72 72 74 75

LIST OF FIGURES 9.1 9.2 9.3 Example source code used as input to program in codes used in this chapter. . . Example source code showing how to get scope information for each IR node. . Output of input code using scopeInformation.C . . . . . . . . . . . . . . . . . . .

xiii 78 79 80

10.1 Example source code for translator to read an input program and generate a list of functions in the AST (queryLibraryExample.C). . . . . . . . . . . . . . . . . . 10.2 Example source code used as input to program in gure 10.1 (queryLibraryExample.C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3 Output of input le to the AST query processor (queryLibraryExample.C). . . . 10.4 Example source code for translator to read an input program and generate a list of access functions in the AST (nestedQueryExample.C). . . . . . . . . . . . . . 10.5 Example source code used as input to program in gure 10.4 (nestedQueryExample.C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Output of input le to the AST query processor (nestedQueryExample.C). . . . 11.1 11.2 11.3 11.4 Example source code showing how to use the AST le I/O support. . . . . . Example source code used as input to demonstrate the AST le I/O support. Output of input code after inlining transformations. . . . . . . . . . . . . . . Output of input code after le I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

82 83 84 85 86 86 88 89 89 90

12.1 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node. . . . 12.3 Output of input code using debuggingIRnodeToString.C . . . . . . . . . . . . . . 12.4 Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node. . . . 12.5 Output of input code using debuggingSourceCodePositionInformation.C . . . . . 13.1 Example source code used as input to program in codes used in this chapter. . . 13.2 Example source code showing how to detect volatile modier. . . . . . . . . . . 13.3 Output of input code using volatileTypeModier.C . . . . . . . . . . . . . . . . .

91 93 93 94 94 97 98 99

14.1 Example source code showing how to get type information from function parameters.102 14.2 Example source code used as input to typeInfoFromFunctionParameters.C. . . . 103 14.3 Output of input to typeInfoFromFunctionParameters.C. . . . . . . . . . . . . . . 104 15.1 Example source code showing mapping of function calls to overloaded function declarations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 15.2 Example source code used as input to resolveOverloadedFunction.C. . . . . . . . 107 15.3 Output of input to resolveOverloadedFunction.C. . . . . . . . . . . . . . . . . . . 107 16.1 Example source code used to extract template parameter information. . . . . . . 109 16.2 Example source code used as input to templateParameter.C. . . . . . . . . . . . 110 16.3 Output of input to templateParameter.C. . . . . . . . . . . . . . . . . . . . . . . 110 17.1 Example source code showing use of a C++ template. . . . . . . . . . . . . . . . 111

xiv

LIST OF FIGURES 17.2 Example source code after processing using identityTranslator (shown in gure 2.1).112 17.3 Example source code showing use of a C++ template. . . . . . . . . . . . . . . . 112 17.4 Example source code after processing using identityTranslator (shown in gure 2.1).112 18.1 18.2 18.3 18.4 18.5 18.6 18.7 19.1 19.2 19.3 19.4 Example of a constant propagation analysis. . . . . . . . Example of a dataow analysis with abstraction of ane Example of simple analyses . . . . . . . . . . . . . . . . Each variables lattice for constant-propagation analysis Example of Transformation on the CFG . . . . . . . . . Example of the Transformation on the Source Code . . Code Replacement Transformation . . . . . . . . . . . . . . . . . . . constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 117 118 119 130 131 132 134 135 135 136 140 141 142

Example source code showing loop recognition (part 1). . . . . . Example source code showing loop recognition (part 2). . . . . . Example source code used as input to loop recognition processor. Output of input to loop recognition processor. . . . . . . . . . . .

20.1 Example source code showing visualization of virtual control ow graph. . . . . . 20.2 Example source code used as input to build virtual control graphs. . . . . . . . . 20.3 The debug virtual control ow graph for function main() shows all virtual CFG nodes and edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.4 The virtual control ow graph for function main() shows only interesting virtual CFG nodes and edges. Each CFGNodes caption tells associated source line number and CFGNode index value (@line-num:index-value) . . . . . . . . . . . . 20.5 The debug virtual control ow graph for function testIf() shows all virtual CFG nodes and edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20.6 The virtual control ow graph for function testIf() shows only interesting virtual CFG nodes and edges. Each CFGNodes caption tells associated source line number and CFGNode index value (@line-num:index-value) . . . . . . . . . . . . 20.7 Example source code showing visualization of static control ow graph. . . . . .

143 144

145 148

21.1 Example source code showing visualization of control ow graph. . . . . . . . . . 152 21.2 Example source code used as input to build control ow graph. . . . . . . . . . . 153 21.3 Control ow graph for function in input code le: inputCode 1.C. . . . . . . . . 153 22.1 Source CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 22.2 Binary CFG Traversal Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 23.1 23.2 23.3 23.4 23.5 23.6 23.7 23.8 Example input code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example source code using def use analysis . . . . . . . . . . . . . . . . . Output of the program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Def-Use graph for example program. . . . . . . . . . . . . . . . . . . . . . Example source code using liveness analysis . . . . . . . . . . . . . . . . . Example input code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Control ow graph annotated with live variables for example program. . . Example code retrieving live variables based on virtual control ow graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 160 160 162 164 165 166 167

LIST OF FIGURES

xv

24.1 Example source code showing visualization of call graph. . . . . . . . . . . . . . . 170 24.2 Example source code used as input to build call graph. . . . . . . . . . . . . . . . 171 24.3 Call graph for function in input code le: inputCode BuildCG.C. . . . . . . . . . 172 25.1 25.2 25.3 25.4 Source code to perform virtual function analysis . . . . . . . . . . . . . . . . . . Example source code used as input for Virtual Function Analysis. . . . . . . . . . Call graph generated by Call Graph Analysis for input code in inputCode vfa.C. Call graph resulted from Virtual Function Analysis for input code in inputCode vfa.C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 175 175 176

26.1 Example source code showing visualization of class hierarchy graph. . . . . . . . 177 26.2 Example source code used as input to build class hierarchy graph. . . . . . . . . 178 26.3 Class hierarchy graph in input code le: inputCode ClassHierarchyGraph.C. . . . 179 27.1 27.2 27.3 27.4 Example translator (part 1) using database connection to store function names. . Example translator (part 2) using database connection to store function names. . Example source code used as input to database example. . . . . . . . . . . . . . . Output from processing input code through database example dataBaseTranslator27.1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 183 184 185

29.1 Example source code showing the output of mangled name. The string represents the code associated with the subtree of the target IR node. . . . . . . . . . . . . 29.2 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29.3 Output of input code using generatingUniqueNamesFromDeclaration.C . . . . . 29.4 Example source code used as input to program in codes showing debugging techniques shown in this section. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29.5 Output of input code using generatingUniqueNamesFromDeclaration.C . . . . . 30.1 Example source code showing simple command-line processing within ROSE translator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.2 Output of input code using commandlineProcessing.C . . . . . . . . . . . . . . . 30.3 Example source code showing simple command-line processing within ROSE translator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30.4 Output of input code using commandlineProcessing.C . . . . . . . . . . . . . . .

194 195 196 197 198

200 200 201 201

31.1 Example source code showing how to tailor the code generation format. . . . . . 204 31.2 Example source code used as input to program to the tailor the code generation. 205 31.3 Output of input code after changing the format of the generated code. . . . . . . 206 32.1 AST construction and insertion for a variable using the high level interfaces . . . 32.2 Example source code to read an input program and add a new variable declaration at the top of each block. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Example source code used as input to the translators adding new variable. . . . . 32.4 Output of input to the translators adding new variable. . . . . . . . . . . . . . . 32.5 Example translator to add expressions . . . . . . . . . . . . . . . . . . . . . . . . 32.6 Example source code used as input . . . . . . . . . . . . . . . . . . . . . . . . . . 208 209 210 210 211 212

xvi

LIST OF FIGURES 32.7 Output of the input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.8 Example source code to add an assignment statement . . . . . . . . . . . . . . . 32.9 Example source code used as input . . . . . . . . . . . . . . . . . . . . . . . . . . 32.10Output of the input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.11Addition of function to global scope using high level interfaces . . . . . . . . . . 32.12Addition of function to global scope using high level interfaces and a scope stack 32.13Example source code shows addition of function to global scope (part 1). . . . . 32.14Example source code shows addition of function to global scope (part 2). . . . . 32.15Example source code used as input to translator adding new function. . . . . . . 32.16Output of input to translator adding new function. . . . . . . . . . . . . . . . . . 32.17Example source code to instrument any input program. . . . . . . . . . . . . . . 32.18Example source code using the high level interfaces . . . . . . . . . . . . . . . . . 32.19Example source code used as input to instrumenting translator. . . . . . . . . . . 32.20Output of input to instrumenting translator. . . . . . . . . . . . . . . . . . . . . 32.21Example source code instrumenting end of functions . . . . . . . . . . . . . . . . 32.22Example input code of the instrumenting translator for end of functions. . . . . . 32.23Output of instrumenting translator for end of functions. . . . . . . . . . . . . . . 32.24Example source code shows repackaging of global variables to a struct (part 1). . 32.25Example source code shows repackaging of global variables to a struct (part 2). . 32.26Example source code shows repackaging of global variables to a struct (part 3). . 32.27Example source code shows repackaging of global variables to a struct (part 4). . 32.28Example source code shows repackaging of global variables to a struct (part 5). . 32.29Example source code used as input to translator adding new function. . . . . . . 32.30Output of input to translator adding new function. . . . . . . . . . . . . . . . . . 34.1 Example source code showing how to access comments. . . . . . . . . . . . . . . 34.2 Example source code used as input to collection of comments and CPP directives. 34.3 Output from collection of comments and CPP directives on the input source le only. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.4 Output from collection of comments and CPP directives on the input source le and all header les. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.5 Example source code showing how to access comments. . . . . . . . . . . . . . . 34.6 Example source code used as input to collection of comments and CPP directives. 34.7 Output from collection of comments and CPP directives on the input source le and all header les. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34.8 Example source code showing how automate comments. . . . . . . . . . . . . . . 34.9 Example source code used as input to automate generation of comments. . . . . 34.10Output of input code after automating generation of comments. . . . . . . . . . . 34.11Example source code showing how automate the introduction of arbitrary text. 34.12Example source code used as input to automate generation of arbitrary text. . . 34.13Output of input code after automating generation of arbitrary text. . . . . . . . 212 213 213 214 215 216 217 218 219 219 221 222 223 223 224 224 225 226 227 228 229 230 231 231 241 242 242 243 246 247 247 248 249 249 250 250 251

35.1 Example source code showing how use Partial Redundancy Elimination (PRE). 253 35.2 Example source code used as input to program to the Partial Redundancy Elimination (PRE) transformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 35.3 Output of input code after Partial Redundancy Elimination (PRE) transformation.256

LIST OF FIGURES

xvii

36.1 Example source code showing how to instrument using Tau. . . . . . . . . . . . 258 36.2 Example source code used as input to program to the inlining transformation. . . 259 36.3 Output of input code after inlining transformations. . . . . . . . . . . . . . . . . 260 37.1 inputCode OutlineLoop.cc: Sample input program. The #pragma directive marks the nested for loop for outlining. . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 37.2 rose outlined-inputCode OutlineLoop.cc: The nested for loop of Figure 37.1 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 37.3 outline.cc: A basic outlining translator, which generates Figure 37.2 from Figure 37.1. This outliner relies on the high-level driver, Outliner::outlineAll(), which scans the AST for outlining pragma directives (#pragma rose outline) that mark outline targets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 37.4 inputCode OutlineLoop2.c: Sample input program without pragmas. . . . . . . . 265 37.5 rose inputCode OutlineLoop2.c: The loop at line 12 of Figure 37.12 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 37.6 rose inputCode OutlineLoop2b.c: The 2nd loop within a function named initializefrom Figure 37.12 has been outlined. . . . . . . . . . . . . . . . . . . . . . . . 267 37.7 outlineIfs.cc: A lower-level outlining translator, which calls Outliner::outline() directly on SgStatement nodes. This particular translator outlines all SgIfStmt nodes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 37.8 inputCode Ifs.cc: Sample input program, without explicit outline targets specied using #pragma rose outline, as in Figures 37.1 and 37.12. . . . . . . . . . . . . 269 37.9 rose inputCode Ifs.cc: Figure 37.8, after outlining using the translator in Figure 37.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 37.10outlinePreproc.cc: The basic translator of Figure 37.3, modied to execute the Outliners preprocessing phase only. In particular, the original call to Outliner::outlineAll() has been replaced by a call to Outliner::preprocessAll(). . . . . . . . . . . . 271 37.11rose outlined pp-inputCode OutlineLoop.cc: Figure 37.1 after outline preprocessing only, i.e., specifying -rose:outline:preproc-only as an option to the translator of Figure 37.3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 37.12inputCode OutlineNonLocalJumps.cc: Sample input program, with an outlining target that contains two non-local jumps (here, break statements). . . . . . . . . 273 37.13rose outlined pp-inputCode OutlineNonLocalJumps.cc: The non-local jump example of Figure 37.12 after outliner preprocessing, but before the actual outlining. The non-local jump is handled by an additional ag, EXIT TAKEN , which indicates what non-local jump is to be taken. . . . . . . . . . . . . . . . . . . . . 274 37.14rose outlined-inputCode OutlineNonLocalJumps.cc: Figure 37.12 after outlining. 275 38.1 Example source code showing use of loop optimization mechanisms. . . . . . . . 38.2 Example source code used as input to loop optimization processor. . . . . . . . . 38.3 Output of loop optimization processor showing matrix multiply optimization (using options: -bk1 -fs0). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.4 Example source code used as input to loop optimization processor. . . . . . . . . 38.5 Output of loop optimization processor showing loop fusion (using options: -fs2). 38.6 Detailed example source code showing use of loop optimization mechanisms (loopProcessor.C part 1). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 280 281 282 282 283

xviii

LIST OF FIGURES 284 285 286 287 288 289 290 291 292 294 295 296 297 297 298 298

38.7 loopProcessor.C source code (Part 2). . . . . . . . . . . . . . . . . . . . . . . . . 38.8 Example source code used as input to loopProcessor, show in gure 38.6. . . . . 38.9 Output of loopProcessor using input from gure 38.8 (using options: -bk1 -fs0). 38.10Example source code used as input to loopProcessor, show in gure 38.6. . . . . 38.11Output of loopProcessor using input from gure 38.10 (using options: -bk1 -unroll nvar 16). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.12Example source code used as input to loopProcessor, show in gure 38.6. . . . . 38.13Output of loopProcessor using input from gure 38.12 (using options: -bk1 -fs0 -splitloop -annotation). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38.14Example source code used as input to loopProcessor, show in gure 38.6. . . . . 38.15Output of loopProcessor input from gure 38.14 (using options: -fs2 -ic1 -opt 1 ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39.1 39.2 39.3 39.4 39.5 39.6 39.7 Example source code used as input to loopUnrolling . . . . . . . Output for a unrolling factor which can divide the iteration space Output for the case when divisibility is unknown at compile-time Example source code used as input to loopInterchange . . . . . . Output for loop interchange . . . . . . . . . . . . . . . . . . . . . Example source code used as input to loopTiling . . . . . . . . . Output for loop tiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . evenly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

40.1 Example source code shows instrumentation to call a test function from of each function body in the application (part 1). . . . . . . . . . . . . . 40.2 Example source code shows instrumentation to call a test function from of each function body in the application (part 2). . . . . . . . . . . . . . 40.3 Example source code shows instrumentation to call a test function from of each function body in the application (part 3). . . . . . . . . . . . . . 40.4 Example source code used as input to translator adding new function. . 40.5 Output of input to translator adding new function. . . . . . . . . . . . .

the top . . . . . the top . . . . . the top . . . . . . . . . . . . . . .

304 305 306 307 308

41.1 Example source code used as input to program in codes used in this chapter. . . 309 41.2 Example source code showing how to seed bugs. . . . . . . . . . . . . . . . . . . 311 41.3 Output of input code using seedBugsExample arrayIndexing.C . . . . . . . . . . 312 43.1 Dataowow graph for example program. . . . . . . . . . . . . . . . . . . . . . . 43.2 Source-to-source transformation to introduce NOP assemble instructions in the generated binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.3 Header le for the traversal used to identify the NOP sequences in the binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.4 Example code to identify the NOP sequences in the binary executable. . . . . . . 43.5 Main program using the traversal to identify the NOP sequences in the binary executable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.6 Example code showing the transformation of the binary executable. . . . . . . . 324 327 328 330 331 332

45.1 Example source code used to generate Dwarf AST for analysis. . . . . . . . . . . 341 45.2 Dwarf AST (subset of ROSE binary AST). . . . . . . . . . . . . . . . . . . . . . 343

LIST OF FIGURES 45.3 Example source code (typical for reading in a binary or source le). 45.4 Example source code (typical for reading in a binary or source le).

xix . . . . . . . 344 . . . . . . . 345

46.1 Example 1: Generated handles for loops: using constructors with or without a specied handle type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46.2 Example 1: Example source code with some loops, used as input. . . . . . . . . . 46.3 Example 1: Abstract handles generated for loops. . . . . . . . . . . . . . . . . . . 46.4 Example 2: Generated handles from strings representing handle items. . . . . . . 46.5 Example 2: Source code with some language constructs. . . . . . . . . . . . . . . 46.6 Example 2: Handles generated from string and their language constructs. . . . . 46.7 Example 3: A simple data structure used to represent a loop in an arbitrary tool. 46.8 Example 3: A test program for simple loops abstract handles. . . . . . . . . . . 46.9 Example 3: Output of the test program for simple loops abstract handles (as strings). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47.1 proled.c (part 1 of 2): Sample input program, proled using the HPCToolkit. . 47.2 proled.c (part 2 of 2): Sample input program, proled using the HPCToolkit. . 47.3 XML schema for HPCToolkit data les: This schema, prepended to each of the HPCToolkit-generated XML les, describes the format of the proling data. This particular schema was generated by HPCToolkit 1.0.4. . . . . . . . . . . . . . . . 47.4 PAPI TOT CYC.xml: Sample cycle counts observed during proling, generated from running the HPCToolkit on proled.c (Figures 47.147.2.) These lines would appear after the schema shown in Figure 47.3. . . . . . . . . . . . . . . . . . . . . 47.5 PAPI FP OPS.xml: Sample op counts observed during proling, generated from running the HPCToolkit on proled.c (Figures 47.147.2.) These lines would appear after the schema shown in Figure 47.3. . . . . . . . . . . . . . . . . . . . . 47.6 attachMetrics.cc: Sample translator to attach HPCToolkit metrics to the AST. . 47.7 Sample output, when running attachMetrics.cc (Figure 47.6) with the XML inputs in Figures 47.447.5. Here, we only show the output sent to standard output (i.e., cout and not cerr). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47.8 Sample PDF showing attributes. . . . . . . . . . . . . . . . . . . . . . . . . . . .

353 354 355 356 357 357 358 359 360 365 366

367

368

369 374

375 375

48.1 Example source code used as input to program in codes used in this chapter. . . 378 48.2 Example source code showing how to instrument using Tau. . . . . . . . . . . . 379 48.3 Output of input code using tauInstrumenter.C . . . . . . . . . . . . . . . . . . . 380 49.1 Haskell version of identity translator. . . . . . . . . . . . . . . . . . . . . . . . . . 381 49.2 Haskell version of constant folding transformation. . . . . . . . . . . . . . . . . . 383 50.1 Example source showing the shared-memory parallel execution of traversals. . . . 388 50.2 Output of input le to the shared-memory parallel traversals. Output may be garbled depending on the multi-threaded behavior of the underlying I/O libraries. 389 51.1 Example source demonstrating the use of the distributed-memory parallel analysis framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 51.2 Example output of a distributed-memory analysis running on four processors. . . 394

xx

LIST OF FIGURES 53.1 Example source code showing reduction recognition. . . . . . . . . . . . . . . . . 397 53.2 Example source code used as input to loop reduction recognition processor. . . . 398 53.3 Output of input to reduction recognition processor. . . . . . . . . . . . . . . . . . 398

Chapter 1

Introduction
1.1 What is ROSE

ROSE is an open source compiler infrastructure for building tools that can read and write source code in multiple languages (C/C++/Fortran) and/or analyze binary executables (using the x86, Power-PC, and ARM instruction sets). The target audience for ROSE is people building tools for the analysis and transformation of software generally as well as code generation tools. ROSE provides a library (librose) that can be used to support the universal requirements of tools that do custom analysis and/or transformations on source code and custom analysis of binary forms of software. ROSE is portable across and expanding range of operating systems and work with an growing number of compilers. ROSE provides a common level of infrastructure support to user-dened tools, so that they need not implement the complex support required for software analysis and transformation operations. For source code based tools these include parsing, common forms of compiler analysis, common transformations, and code generation. For binary analysis based tools these include disassembly, function boundary detection, and common forms of analysis. User dened tools may also mix source code and binary analysis to form more interesting tools for specialized purposes. ROSE is part of research work to unify the analysis of both source code and binaries within general compiler research and dene mixed forms of static and dynamic analysis. ROSE works by reading the source code and/or binary and generating an Abstract Syntax Tree (AST). The AST forms a graph representing the structure of the source code and/or binary executable and is held in memory to provide the fastest possible means of operating on the graph. The nodes used to dene the AST graph are an intermediate representation (IR); common within compiler research as a way of representing the structure of software absent syntax details (commas, semi-colons, white-space, etc.). ROSE provides mechanisms to traverse and manipulate the AST. Finally, in the case of source code, ROSE provides mechanisms to regenerate source code from the AST. As a trivial example, if the input source code program contains a variable declaration for an integer, all of this information will be available in the AST generated from the input code passed on the command line to any tool built using ROSE. Similarly, an automated transformation of the variable declaration held in the AST would be expressed using a traversal over the AST 1

CHAPTER 1. INTRODUCTION

and code semantic actions to mutate the AST. Then the transformed source code would be generated (unparsed) from the AST. In the case of binaries (including executables, object les, and libraries), the AST will represent the structure of the binary. The AST for a binary also includes the binary le format (symbol table, debug format, import tables, etc.), disassembled instructions, all instruction operands, etc. ROSE provides a rich set of tools to support the analysis of software including the support for users to build their own forms of analysis and specialized transformations. As an example, ROSE includes a full OpenMP compiler built using the internal ROSE infrastructure for analysis and transformation. A wide assortment of AST traversals are provided to express both analysis and transformations of the AST. A set of common forms of analysis are provided (call graph, control ow, etc.) most work uniformly on both source code and binary executables. Visualization support in included to help users understand and debug their tools. GUI support is available to support building professional level tools using ROSE. ROSE is actively supported by a small group at LLNL and is used as a basis for compiler research work within DOE at LLNL. Technically, ROSE is designed to build what are called translators, ROSE uses a source-tosource approach to dene such translators. Note that translators are signicantly more sophisticated than preprocessors but the terms are frequently confused. A translator must understand the source code at a fundamentally deeper level using a grammar for the whole language and on the whole source code, where as a preprocessor only understands the source code using a simpler grammar and on a subset of the source code. It is loosely the dierence between any language compiler and the C preprocessor (cpp).

1.2

Why you should be interested in ROSE

ROSE is a tool for building source-to-source translators. You should be interested in ROSE if you want to understand or improve any aspect of your software. ROSE makes it easy to build tools that read and operate on source code from large scale applications (millions of lines). Whole projects may be analyzed and even optimized using tools built using ROSE. For example, ROSE is itself analyzed nightly using ROSE. To get started immediately consult the ROSE User Manual, chapter Getting Started for details).

1.3

Problems that ROSE can address

ROSE is a mechanism to build source-to-source analysis or optimization tools that operate directly on the source code of large scale applications. Example tools that have been built include: OpenMP translator, Array class abstraction optimizer, Source-to-source instrumenter, Loop analyzer, Symbolic complexity analyzer,

1.4. EXAMPLES IN THIS ROSE TUTORIAL Inliner and outliner, Code coverage tools, and many more... Example tools that can be built include: Custom optimization tools, Custom documentation generators, Custom analysis tools, Code pattern recognition tools, Security analysis tools, and many more...

1.4

Examples in this ROSE Tutorial

This tutorial lays out a set of progressively complex example programs (located in <ROSE SOURCE>/tutorial/*) that serve as a tutorial for the use of ROSE. Translators built using ROSE can either just analyze (and output results) or compile the input programs just like a compiler (generating object les or executables). Many of the examples in this tutorial just do simple analysis of the input source code, and a few show the full compilation of the input source code. Where the translators generate either object les of executables, the vendors compiler is used to compile the nal ROSE-generated code. Within ROSE, the call to generate source code from the AST and call the vendors compiler is referred to as the backend processing. The specication of the vendors compiler as a backend is done within the conguration step within ROSE (see options for configure in the ROSE User Manual). Within the example programs below, the user can provide alternative input programs for more complex evaluation of the tutorial examples and ROSE. The end of the chapter, section 1.7, shows the makeles required to compile the tutorial programs using an installed version of ROSE (compiled using make install). This example makefile is run as part of the testing using the make installcheck rule. Chapters are organized in topics including simple ROSE AST visualization, dealing with complex data types, program analysis, program transformation and optimization, correctness checking, binary support, interacting with other tools, and parallelism. We hope readers can easily nd the information they want. FIXME: We should constantly Specic chapters in this tutorial include:
update this

Introduction 1. Introduction (this chapter) 2. Problems that ROSE can address 3. Getting Started This chapter covers where to nd ROSE documentation and how to install ROSE.

CHAPTER 1. INTRODUCTION 4. Example Makeles demonstrating the command lines to compile and link the example translators in this tutorial are found in <ROSE Compile Tree>/tutorial/exampleMakefile. Working with the ROSE AST: 1. Identity Translator This example translator reads a C or C++ application, builds the AST internally, generates source code from the AST (unparsing), and calls the backend vendor compiler to compile the generated C or C++ application code. Thus the translator acts like and can be used to replace any compiler since it takes in source code and outputs an object code (or executable). This example also shows that the output generated from and ROSE translator is a close reproduction of the input; preserving all comments, preprocessor control structure, and most formating. 2. Scopes of Declarations (scopeInformation.C) This example shows the scopes represented by dierent IR nodes in the AST. 3. AST Graph Generator This translator reads a C or C++ application code and builds the AST, internally. The translator does not regenerate code from the AST and so does not call the backend vendors compiler. This shows how simple it could be to build source code analysis tools; the code calls an internal ROSE function to generate a dot graph of the AST, the makele has the details of converting the dot graph into a postscript le (also shown). 4. AST PDF Generator This translator reads an C or C++ application code builds the AST internally. The translator does not regenerate code from the AST and so does not call the backend vendors compiler. This shows how simple it could be to build source code analysis tools, the code calls an internal ROSE function to generate a pdf le with bookmarks representing the AST. The pdf le show as output is in this case a previously generated gure of a screen shot obtained by viewing the output pdf le using acroread. 5. Introduction to AST Traversals and Attributes This collection of examples show the use of the simple visitor pattern for the traversal of the AST within ROSE. The simple visitor pattern permits operations to be programmed which will be invoked on dierent nodes within the AST. To handle communication of context information down into the AST and permit communication of analysis information up the AST, we have provided inherited and synthesized attributes (respectively). Note that an AST is most often represented as a tree with extra edges and with shared IR nodes that make the full graph (representing all edges) not a tree. We present two styles of traversal, one over the tree representing the AST (which excludes some types of IR nodes) and one over the full AST with all extra nodes and shared nodes. Extra nodes are nodes such as SgType and SgSymbol IR nodes. (a) AST traversals These traversals visit each node of the tree embedded within the AST (excluding shared SgType and SgSymbol IR nodes). These traversals visit the IR nodes is

1.4. EXAMPLES IN THIS ROSE TUTORIAL

an order dependent upon the structure of the AST (the source code from which the AST is built). i. Classic Object-Oriented Visitor Patterns This example, classicObjectOrientedVisitorPatternMemoryPoolTraversal.C, show the use of a classic visitor patterns. At the moment this example uses the ASTs memory pools as a basis but it is identical to a future traversal. The ROSE visitor Pattern (below) is generally more useful. The classic visitor pattern traversals are provided for completeness. ii. Visitor Traversal (visitorTraversal.C) Conventional visitor patterns without no attributes. This pattern can explicitly access global variables to provide the eect of accumulator attributes (using static data members we later show the handling of accumulator attributes). iii. Inherited Attributes (inheritedAttributeTraversal.C) Inherited attributes are used to communicate the context of any location within the AST in terms of other parent AST nodes. iv. Synthesized Attributes (synthesizedAttributeTraversal.C) Synthesized attributes are used to pass analysis results from the leaves of the AST to the parents (all the way to the root of the AST if required). v. Accumulator Attributes (accumulatorAttributeTraversal.C) Accumulator attributes permit the interaction of data within inherited attributes with data in synthesized attributes. In our example program we will show the use of accumulator attributes implemented as static data members. Accumulator attributes are a fancy name for what is essentially global variables (or equivalently a data structure passed by reference to all the IR nodes in the AST). vi. Inherited and Synthesized Attributes (inheritedAndSynthesizedAttributeTraversal.C) The combination of using inherited and synthesized attributes permits more complex analysis and is often required to compute analysis results on the AST within a specic context (e.g. number of loop nests of specic depth). vii. Persistent Attributes (persistantAttributes.C) Persistent attributes may be added the AST for access to stored results for later traversals of the AST. The user controls the lifetime of these persistent attributes. viii. Nested traversals Complex operations upon the AST can require many subordinate operations. Such subordinate operations can be accommodated using nested traversals. All traversals can operate on any subtree of the AST, and may even be nested arbitrarily. Interestingly, ROSE traversals may also be applied recursively (though care should be take using recursive traversals using accumulator attributes to avoid over accumulation). (b) Memory Pool traversals These traversals visit all IR nodes (including shared IR nodes such as SgTypes and SgSymbols). By design this traversal can visit ALL IR nodes without the worry

CHAPTER 1. INTRODUCTION of getting into cycles. These traversals are mostly useful for building specialized tools that operate on the AST. i. Visit Traversal on Memory Pools This is a similar traversal as to the Visitor Traversal over the tree in the AST. ii. Classic Object-Oriented Visitor Pattern on Memory Pools This is similar to the Classic Object-Oriented Visitor Pattern on the AST. iii. IR node Type Traversal on Memory Pools This is a specialized traversal which visits each type of IR node, but one one of each type of IR nodes. This specialized traversal is useful for building tools that call static member functions on each type or IR node. A number of memory based tools for ROSE are built using this traversal. 6. AST Query Library This example translator shows the use of the AST query library to generate a list of function declarations for any input program (and output the list of function names). It can be trivially modied to return a list of any IR node type (C or C++ language construct). 7. Symbol Table Handling (symbolTableHandling.C) This example shows how to use the symbol tables held within the AST for each scope. 8. AST File I/O (astFileIO GenerateBinaryFile.C) This example demonstrates the le I/O for AST. This is part of ROSE support for whole program analysis. 9. Debugging Tips There are numerous methods ROSE provides to help debug the development of specialized source-to-source translators. This section shows some of the techniques for getting information from IR nodes and displaying it. Show how to use the PDF generator for ASTs. This section may contain several subsections. (a) Generating the code representing any IR node (b) Displaying the source code position of any IR node Complex Types 1. Getting the type parameters in function declaration (functionParameterTypes.C) This example translator builds a list to record the types used in each function. It shows an example of the sort of type information present within the AST. ROSE specically maintains all type information. 2. Resolving overloaded functions (resolvingOverloadedFunctions.C C++ specic) The AST has all type information pre-evaluated, particularly important for C++ applications where type resolution is required for determining function invocation. This example translator builds a list of functions called within each function, showing that overloaded function are fully resolved within the AST. Thus the user is not required to compute the type resolution required to identify which over loaded functions are called. 3. Getting template parameters to a templated class (templateParameters.C C++ specic)

1.4. EXAMPLES IN THIS ROSE TUTORIAL

All template information is saved within the AST. Templated classes and functions are separately instantiated as specializations, as such they can be transformed separately depending upon their template values. This example code shows the template types used the instantiate a specic templated class. Program Analysis 1. Recognizing loops within applications (loopRecognition.C) This example program shows the use of inherited and synthesized attributes form a list of loop nests and report their depth. The inherited attributes are required to record when the traversal is within outer loop and the synthesized attributes are required to pass the list of loop nests back up of the AST. 2. Generating a CFG (buildCFG.C) This example shows the generation of a control ow graph within ROSE. The example is intended to be simple. Many other graphs can be built, we need to show them as well. 3. Generating a CG (buildCallGraph.C) This example shows the generation of a call graph within ROSE. 4. Generating a CH (classHierarchyGraph.C) This example shows the generation of a class hierarchy graph within ROSE. 5. Building custom graphs of program information The mechanisms used internally to build dierent graphs of program data is also made externally available. This section shows how new graphs of program information can be built or existing graphs customized. 6. Database Support (dataBaseUsage.C) This example shows how to use the optional (see configure --help) SQLite database to hold persistent program analysis results across the compilation of multiple les. This mechanism may become less critical as the only mechanism to support global analysis once we can support whole program analysis more generally within ROSE. Program Transformations and Optimizations 1. Generating Unique Names for Declarations (generatingUniqueNamesFromDeclaration.C) A recurring issue in the development of many tools and program analysis is the representation of unique strings from language constructs (functions, variable declarations, etc.). This example demonstrated support in ROSE for the generation of unique names. Names are unique across dierent ROSE tools and compilation of dierent les. 2. Command-line processing ROSE includes mechanism to simplify the processing of command-line arguments so that translators using ROSE can trivially replace compilers within makeles. This example shows some of the many command-line handling options within ROSE and the ways in which customized options may be added. (a) Recognizing custom command-line options

CHAPTER 1. INTRODUCTION (b) Adding options to internal ROSE command-line driven mechanisms 3. Tailoring the code generation format: how to indent the generated source code and others. 4. AST construction: how to build AST pieces from scratch and attach them to the existing AST tree. (a) Adding a variable declaration (addingVariableDeclaration.C) Here we show how to add a variable declaration to the input application. Perhaps we should show this in two ways to make it clear. This is a particularly simple use of the AST IR nodes to build an AST fragment and add it to the applications AST. (b) Adding a function (addingFunctionDeclaration.C) This example program shows the addition of a new function to the global scope. This example is a bit more involved than the previous example. (c) Simple Instrumentor Translator (simpleInstrumentor.C) This example modies an input application to place new code at the top and bottom of each block. The output is show with the instrumentation in place in the generated code. (d) Other examples for creating expressions, structures and so on. 5. Handling source comments, preprocessor directives. 6. Calling the inliner (inlinerExample.C) This example shows the use of the inliner mechanism within ROSE. The function to be inlined in specied and the transformation upon the AST is done to inline the function where it is called and clean up the resulting code. 7. Calling the outliner (outlinerExample.C) This example shows the use of the outliner mechanism within ROSE. A segment of code is selected and a function is generated to hold the resulting code. Any required variables (including global variables) are passed through the generated functions interface. The outliner is a useful part of the empirical optimization mechanisms being developed within ROSE. 8. Call loop optimizer on set of loops (loopOptimization.C) This example program shows the optimization of a loop in C. This section contains several subsections each of which shows dierent sorts of optimizations. There are a large number of loop optimizations only two are shown here, we need to add more. (a) Optimization of Matrix Multiply (b) Loop Fusion Optimizations 9. Parameterized code translation: How to use command line options and abstract handles to have the translations you want, the order you want, and the behaviors you want. 10. Program slicing (programSlicingExample.C) This example shows the interface to the program slicing mechanism within ROSE. Program slicing has been implemented to two ways within ROSE. Correctness Checking

Does this tutorial still exist?

1.4. EXAMPLES IN THIS ROSE TUTORIAL

1. Code Coverage Analysis (codeCoverage.C) Code coverage is a useful tool by itself, but is particularly useful when combined with automated detection of bugs in programs. This is part of work with IBM, Haifa. 2. Bug seeding: how to purposely inject bugs into source code. Binary Support 1. Instruction semantics 2. Binary Analysis 3. Binary construction 4. DWarf debug support Interacting with Other Tools 1. Abstract handles: uniform ways of specifying language constructs. 2. ROSE-HPCT interface: How to annotate AST with performance metrics generated by third-party performance tools. 3. Tau Performance Analysis Instrumentation (tauInstrumenter.C) Tau currently uses an automate mechanism that modied the source code text le. This example shows the modication of the AST and the generation of the correctly instrumented les (which can otherwise be a problem when macros are used). This is part of collaborations with the Tau project. 4. The Haskell interface: interacting with a function programming language. Parallelism 1. Shared-memory parallel traversals 2. Distributed-memory parallel traversals 3. Parallel checker 4. Reduction variable recognition Other examples included come specically from external collaborations and are more practically oriented. Each is useful as an example because each solves a specic technical problem. More of these will be included over time. FIXME: The following tutorials
no longer exist?

1. Fortran promotion of constants to double precision (typeTransformation.C) Fortran constants are by default singe precision, and must be modied to be double precision. This is a common problem in older Fortran applications. This is part of collaborations with LANL to eventually automatically update/modify older Fortran applications. 2. Automated Runtime Library Support (charmSupport.C) Getting research runtime libraries into use within large scale applications requires automate mechanism to make minor changes to large amounts of code. This is part of collaborations with the Charm++ team (UIUC).

10

CHAPTER 1. INTRODUCTION (a) Shared Threaded Variable Detection Instrumentation (interveneAtVariables.C) Instrumentation support for variables, required to support detection of threaded bugs in applications. (b) Automated Modication of Function Parameters (changeFunction.C) This example program addresses a common problem where an applications function must be modied to include additional information. In this case each function in a threaded library is modied to include additional information to a corresponding wrapper library which instruments the librarys use.

Add make installcheck e.am to build example ors using the installed libraries.

1.5

ROSE Documentation and Where To Find It

There are three forms of documentation for ROSE, and also a ROSE web Page and email lists. For more detailed information on getting started, see the ROSE User Manual, chapter Getting Started for more details). 1. ROSE User Manual The User Manual presents how to get started with ROSE and documents features of the ROSE infrastructure. The User Manual is found in ROSE/docs/Rose directory, or at: ROSE User Manual (pdf version, relative link) 2. ROSE Tutorial The ROSE Tutorial presents a collection of examples of how to use ROSE (found in the ROSE/tutorial directory). The ROSE Tutorial documentation is found in ROSE/docs/Rose/Tutorial directory. The tutorial documentation is built in the following steps: (a) actual source code for each example translator in the ROSE/tutorial directory is included into the tutorial documentation (b) each example is compiled (c) inputs to the examples are taken from the ROSE/tutorial directory (d) output generated from running each example is placed into the tutorial documentation Thus the ROSE/tutorial directory contains the exact examples in the tutorial and each example may be modied (changing either the example translators or the inputs to the examples). The ROSE Tutorial can also be found in the ROSE/docs/Rose/Tutorial directory (the LaTeX document; ps or pdf le) : ROSE Tutorial (pdf version, relative link), 3. ROSE HTML Reference: Intermediate Representation (IR) documentation This web documentation presents the detail interfaces for each IR nodes (documentation generated by Doxygen). The HTML IR documentation is found in ROSE/docs/Rose directory (available as html only): ROSE HTML Reference (relative link) 4. ROSE Web Page The ROSE web pages are located at: http://www.rosecompiler.org

1.6. USING THE TUTORIAL

11

5. ROSE Email List The ROSE project maintains an external mailing list (see information at: www.roseCompiler.org and click on the Mailing Lists link for how to join).

1.6

Using the Tutorial

First install ROSE (see ROSE User Manual, chapter Getting Started for details). Within the ROSE distribution at the top level is the tutorial directory. All of the examples in this documentation are represented there with Makeles and sample input codes to the example translators.

1.7

Required Makele for Tutorial Examples

FIXME: The exampleMakele needs to be modied to include the support for compiling all of the tutorial examples.

This section shows an example makele 1.1 required for the compilation of many of the tutorial example programs using the installed libraries (assumed to be generated from make install). The build process can be tested by running make installcheck from within the ROSE compile tree. This makefile can be found in the compile tree (not the source tree) for ROSE in the tutorial directory.

12

CHAPTER 1. INTRODUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

# # # # # #

Example Makefile for ROSE users This makefile is provided as an example of how to use ROSE when ROSE is installed (using "make install"). This makefile is tested as part of the "make distcheck" rule (run as part of tests before any SVN checkin). The test of this makefile can also be run by using the "make installcheck" rule (run as part of "make distcheck").

# Location of include directory after "make install" ROSE_INCLUDE_DIR = /home/hudson-rose/.hudson/tempInstall/include # Location of Boost include directory BOOST_CPPFLAGS = -pthread -I/export/tmp.hudson-rose/opt/boost_1_40_0-inst/include # Location of Dwarf include and lib (if ROSE is configured to use Dwarf) ROSE_DWARF_INCLUDES = ROSE_DWARF_LIBS_WITH_PATH = # Location of library directory after "make install" ROSE_LIB_DIR = /home/hudson-rose/.hudson/tempInstall/lib CC CXX CPPFLAGS #CXXCPPFLAGS CXXFLAGS LDFLAGS = gcc = g++ = = @CXXCPPFLAGS@ = -g -Wall =

ROSE_LIBS = $(ROSE_LIB_DIR)/librose.la # Location of source code ROSE_SOURCE_DIR = \ ../../tutorial executableFiles = identityTranslator ASTGraphGenerator \ visitorTraversal inheritedAttributeTraversal \ synthesizedAttributeTraversal \ inheritedAndSynthesizedAttributeTraversal \ accumulatorAttributeTraversal persistantAttributes \ queryLibraryExample nestedTraversal \ loopRecognition \ typeInfoFromFunctionParameters \ resolveOverloadedFunction templateParameter \ instrumentationExample addVariableDeclaration \ addFunctionDeclaration loopOptimization \ buildCFG debuggingIRnodeToString \ debuggingSourceCodePositionInformation \ commandlineProcessing \ loopNestingInfoProcessing # Default make rule to use all: $(executableFiles) @if [ x$${ROSE_IN_BUILD_TREE:+present} = xpresent ]; then echo "ROSE_IN_BUILD_TREE should not be set" >&2; exit 1; fi

# Example of how to use ROSE (linking to dynamic library, which is must faster # and smaller than linking to static libraries). Dynamic linking requires the # use of the "-L$(ROSE_LIB_DIR) -Wl,-rpath" syntax if the LD_LIBRARY_PATH is not # modified to use ROSE_LIB_DIR. We provide two example of this; one using only # the "-lrose -ledg" libraries, and one using the many separate ROSE libraries. $(executableFiles): # g++ -I$(ROSE_INCLUDE_DIR) -o $@ $(ROSE_SOURCE_DIR)/$@.C -L$(ROSE_LIB_DIR) -Wl,-rpath $(ROSE_LIB_DIR) $(ROSE_LIBS) # g++ -I$(ROSE_INCLUDE_DIR) -o $@ $(ROSE_SOURCE_DIR)/$@.C $(LIBS_WITH_RPATH) $(ROSE_LIBS) # /bin/sh ..//libtool --mode=link $(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) -I$(ROSE_INCLUDE_DIR) $(BOOST_CPPFLAGS) -o $@ $(ROSE_SOU /bin/sh ..//libtool --mode=link $(CXX) $(CPPFLAGS) $(CXXFLAGS) $(LDFLAGS) -I$(ROSE_INCLUDE_DIR) $(BOOST_CPPFLAGS) $(ROSE_DWARF_INC

Figure 1.1: Example Makefile showing how to use an installed version of ROSE (generated by make install).

Part I

Working with the ROSE AST

Getting familiar with the ROSE AST is the basis for any advanced usage of ROSE. This part of tutorial collects examples for AST visualization, traversal, query, and debugging.

13

Chapter 2

Identity Translator
What To Learn From This Example This example shows a trivial ROSE translator which does not transformation, but eectively wraps the the backend vendor compiler in an extra layer of indirection. Using the input code in Figure 2.2 we show a translator which builds the AST (calling frontend()), generates the source code from the AST, and compiles the generated code using the backend vendor compiler1 . Figure 2.1 shows the source code for this translator. The AST graph is generated by the call to the frontend() using the standard argc and argv parameters from the C/C++ main() function. In this example code, the variable project represents the root of the AST2 . The source code also shows what is an optional call to check the integrity of the AST (calling function AstTests::runAllTests()); this function has no side-eects on the AST. The source code generation and compilation to generate the object le or executable are done within the call to backend(). The identity translator (identityTranslator ) is probably the simplest translator built using ROSE. It is built by default and can be found in ROSE BUILD/exampleTranslators/documentedExamples/simpleTranslatorExamples or ROSE INSTALL/bin. It is often used to test if ROSE can compile input applications. Typing identityTranslator help will give you more information about how to use the translator. Figure 2.3 shows the generated code from the processing of the identityTranslator build using ROSE and using the input le shown in gure 2.2. This example also shows that the output generated from and ROSE translator is a close reproduction of the input; preserving all comments, preprocessor control structure, and most formating. Note that all macros are expanded in the generated code. In this trivial case of a program in a single le, the translator compiles the application to build an executable (since -c was not specied on the command-line).

that the backend vendor compiler is selected at conguration time. The AST is technically a tree with additional attributes that are represented by edges and additional nodes, so the AST is a tree and the AST with attributes is a more general graph containing edges that would make it technically not a tree.
2

1 Note:

15

16

CHAPTER 2. IDENTITY TRANSLATOR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

// Example ROSE T r a n s l a t o r : u s e d f o r #i n c l u d e r o s e . h

t e s t i n g ROSE i n f r a s t r u c t u r e

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Run i n t e r n a l c o n s i s t e n c y t e s t s on AST AstTests : : runAllTests ( p r o j e c t ) ; // I n s e r t your own m a n i p u l a t i o n o f t h e AST h e r e . . . // G e n e r a t e s o u r c e c o d e from AST and c a l l r e t u r n backend ( p r o j e c t ) ; } t h e vendor s c o m p i l e r

Figure 2.1: Source code for translator to read an input program and generate an object code (with no translation).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

// Example i n p u t f i l e #i n c l u d e < i o s t r e a m > typedef f l o a t R ea l ;

f o r ROSE t u t o r i a l

// Main f u n c t i o n i n t main ( ) { int x = 0; bool value = f a l s e ; // f o r l o o p f o r ( i n t i =0; i < 4 ; { int x ; } return 0; }

i ++)

Figure 2.2: Example source code used as input to identity translator.

17

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

// Example i n p u t f i l e #i n c l u d e < i o s t r e a m > t y p e d e f f l o a t Real ; // Main f u n c t i o n

f o r ROSE t u t o r i a l

i n t main ( ) { int x = 0; bool value = f a l s e ; // f o r l o o p for ( int i = 0; i < 4; int x ; } return 0; }

i ++) {

Figure 2.3: Generated code, from ROSE identity translator, sent to the backend (vendor) compiler.

18

CHAPTER 2. IDENTITY TRANSLATOR

Chapter 3

Simple AST Graph Generator

What To Learn From This Example This example shows how to generate a DOT le to visualize a simplied AST from any input program. DOT is a graphics le format from the AT&T GraphViz project used to visualize moderate sized graphs. It is one of the visualization tools used within the ROSE project. More inforamtion can be readily found at www.graphviz.org/. We have found the zgrviewer to be an especially useful program for visualizing graphs generated in the DOT le format (see chapter 5 for more information on zgrviewer). Each node of the graph in gure 3.3 shows a node of the Intermediate Representation (IR); the graphs demonstrates that the AST is formally a tree. Each edge shows the connection of the IR nodes in memory. The generated graph shows the connection of dierent IR nodes that form the AST for an input program source code. Binary executables can similarly be vizualized using DOT les. The generation of such graphs is appropriate for small input programs, chapter 6 shows a mechanism using PDF les that is more appropriate to larger programs (e.g. 100K lines of code). More information about generation of specialized AST graphs can be found in 5 and custom graph generation in 28. Note that a similar utility program named dotGenerator already exists within ROSE/exampleTranslators/DOTGenerator. It is also installed to ROSE INS/bin. The program in gure 3.1 calls an internal ROSE function that traverses the AST and generates an ASCII le in dot format. Figure 3.2 shows an input code which is processed to generate a graph of the AST, generating a dot le. The dot le is then processed using dot to generate a postscript le 3.3 (within the Makefile). Figure 3.3 (../../..//tutorial/test.ps) can be found in the compile tree (in the tutorial directory) and viewed directly using ghostview or any postscript viewer to see more detail. Figure 3.3 displays the individual C++ nodes in ROSEs intermediate representation (IR). Each circle represents a single IR node, the name of the C++ construct appears in the center of the circle, with the edge numbers of the traversal on top and the number of child nodes appearing below. Internal processing to build the graph generates unique values for each IR node, a pointer address, which is displays at the bottom of each circle. The IR nodes are connected for form a tree, and abstract syntax tree (AST). Each IR node is a C++ class, see SAGE III reference for 19

20

CHAPTER 3. SIMPLE AST GRAPH GENERATOR

1 2 3 4 5 6 7 8 9 10 11 12 13 14

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // G e n e r a t e a DOT f i l e t o u s e i n generateDOT ( p r o j e c t ) ; return 0; } v i s u a l i z i n g t h e AST graph .

Figure 3.1: Example source code to read an input program and generate an AST graph.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; }

Figure 3.2: Example source code used as input to generate the AST graph. details, the edges represent the values of data members in the class (pointers which connect the IR nodes to other IR nodes). The edges are labeled with the names of the data members in the Use this rst example classes representing the IR nodes.

e to explain the use of ader les (cong.h and d the code to build the SgProject object.

21

0:57 SgSourceFile 1 0x2abc0be90010

globalScope

1:56 SgGlobal 424 0x2abc0beec010

*[421]

*[422]

*[423]

2:3 SgTemplateDeclaration !isForward 0 0x2abc0c19e2a8

4:9 SgFunctionDeclaration foo isForward 3 0x2abc0bff6908

10:55 SgFunctionDeclaration foo !isForward 3 0x2abc0bff6ca0

parameterList decoratorList definition

parameterList decoratorList

definition

5:8 SgFunctionParameterList !isForward 1 0x2abc0c0b3628

11:14 SgFunctionParameterList !isForward 1 0x2abc0c0b3860

15:54 SgFunctionDefinition 1 0x2abc0c241010

*[0]

*[0]

body

6:7 SgInitializedName 1 0x2abc0c139438

12:13 SgInitializedName 1 0x2abc0c139560

16:53 SgBasicBlock 3 0x2abc0c28a010

initptr

initptr

*[0]

*[1]

*[2]

17:24 SgVariableDeclaration !isForward 2 0x2abc0c2cd010

25:28 SgVariableDeclaration !isForward 2 0x2abc0c2cd290

29:52 SgIfStmt 3 0x2abc0c3f3010

baseTypeDefiningDeclaration *[1]

baseTypeDefiningDeclaration *[1]

conditional

true_body

false_body

18:23 SgInitializedName x 1 0x2abc0c139688

26:27 SgInitializedName y 1 0x2abc0c1397b0

30:35 SgExprStatement 1 0x1d358540

36:43 SgExprStatement 1 0x1d358598

44:51 SgExprStatement 1 0x1d3585f0

initptr

initptr

expression

expression

expression

19:22 SgAssignInitializer 1 0x1d2e46e0

31:34 SgCastExp 1 0x1d3218b0

37:42 SgAssignOp 2 0x1d36de20

45:50 SgAssignOp 2 0x1d36de90

operand_i

operand_i

lhs_operand_i rhs_operand_i

lhs_operand_i rhs_operand_i

20:21 SgIntVal value = 1 0 0x1d2cb050

32:33 SgVarRefExp name = x 0 0x1d33ee90

38:39 SgVarRefExp name = y 0 0x1d33eef8

40:41 SgIntVal value = 2 0 0x1d2cb0b8

46:47 SgVarRefExp name = y 0 0x1d33ef60

48:49 SgIntVal value = 3 0 0x1d2cb120

Figure 3.3: AST representing the source code le: inputCode ASTGraphGenerator.C.

22

CHAPTER 3. SIMPLE AST GRAPH GENERATOR

Chapter 4

AST Whole Graph Generator

What To Learn From This Example This example shows how to generate and visualize the AST from any input program. This view of the AST includes all additional IR nodes and edges that form attributes to the AST, as a result this graph is not a tree. These graphs are more complex but show signicantly more detail about the AST and its additional edges and attributes. Each node of the graph in gure ?? shows a node of the Intermediate Representation (IR). Each edge shows the connection of the IR nodes in memory. The generated graph shows the connection of dierent IR nodes to form the AST and its additional attributes (e.g types, modiers, etc). The generation of such graphs is appropriate for very small input programs, chapter 6 shows a mechanism using PDF les that is more appropriate to larger programs (e.g. 100K lines of code). More information about generation of specialized AST graphs can be found in 5 and custom graph generation in 28. Again, a utility program, called dotGeneratorWholeASTGraph is provided within ROSE to generate detailed dot graph for input code. It is available from ROSE BUILD/exampleTranslators/DOTGenerator or ROSE INS/bin. A set of options is available to further customize what types of AST nodes to be shown or hidden. Please consult the screen output of dotGeneratorWholeASTGraph -help for details. Viewing these dot les is best done using: zgrviewer at http://zvtm.sourceforge.net/zgrviewer.html. This tool permits zooming in and out and viewing isolated parts of even very large graphs. Zgrviewer permits a more natural way of understanding the AST and its additional IR nodes than the pdf le displayed in these pages. The few lines of code used to generate the graphs can be used on any input code to better understand how the AST represents dierent languages and their constructs. The program in gure 4.1 calls an internal ROSE function that traverses the AST and generates an ASCII le in dot format. Figure ?? shows an tiny input code which is processed to generate a graph of the AST with its attributes, generating a dot le. The dot le is then processed using dot to generate a pdf le 4.3 (within the Makefile). Note that a similar utility program already exists within ROSE/exampleTranslators (and includes a utility to output an alternative PDF representation (suitable for larger ASTs) as well). Figure ?? (../../..//tutorial/test.ps) can be found in the compile tree (in the tutorial directory) and viewed directly using any pdf or dot viewer to see more detail (zgrviewer working with the dot le directly is strongly advised). Note that ASTs can get very large, and that the additional IR nodes required to represent 23

24

CHAPTER 4. AST WHOLE GRAPH GENERATOR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // // // // B u i l d t h e DOT f i l e t o v i s u a l i z e t h e AST w i t h a t t r i b u t e s ( t y p e s , symbols , To p r o t e c t a g a i n s t b u i l d i n g g r a p h s t h a t a r e t o o l a r g e an o p t i o n i s p r o v i d e d t o bound t h e number o f IR n o d e s f o r which a graph w i l l be generated . The l a y o u t o f l a r g e r g r a p h s i s p r o h i b i t i v e l y e x p e n s i v e . c o n s t i n t MAX NUMBER OF IR NODES = 2 0 0 0 ; g e n e r a t e A s t G r a p h ( p r o j e c t , MAX NUMBER OF IR NODES ) ; return 0; } etc . ) .

Figure 4.1: Example source code to read an input program and generate a whole AST graph. the types, modiers, etc, can generate visually complex graphs. ROSE contains the mechanisms to traverse these graphs and do analysis on them. In one case the number of IR nodes exceeded 27 million, an analysis was done through a traversal of the graph in 10 seconds on a desktop x86 machine (the memory requirements were 6 Gig). ROSE organizes the IR in ways that permit analysis of programs that can represent rather large ASTs.
1 2 3 4 5 6 7 8 // T r i v i a l f u n c t i o n u s e d t o g e n e r a t e graph o f AST // w i t h a l l t y p e s and a d d i t i o n a l e d g e s shown . // Graphs o f t h i s s o r t a r e l a r g e , and can be // v i e w e d u s i n g z g r v i e w e r f o r d o t f i l e s . int foo () { return 0; }

Figure 4.2: Example tiny source code used as input to generate the small AST graph with attributes. Figure 4.3 displays the individual C++ nodes in ROSEs intermediate representation (IR). Colors and shapes are used to represent dierent types or IR nodes. Although only visible using zgrviewer the name of the C++ construct appears in the center of each node in the graph, with the names of the data members in each IR node as edge labels. Unique pointer values are includes and printed next to the IR node name. These graphs are the single best way to develop an intuitive understanding how language constructs are organized in the AST. In these graphs, the color yellow is used for types (SgType IR nodes), the color green is used for expressions (SgExpression IR nodes), and statements are a number of dierent colors and shapes to make them more recognizable. Figure 4.5 shows a graph similar to the previous graph but larger and more complex because it is from a larger code. Larger graphs of this sort are still very useful in understanding how more signicant language constructs are organized and reference each other in the AST. Tools such as zgrviewer are essential to reviewing and understanding these graphs. Although such graphs

25

26

CHAPTER 4. AST WHOLE GRAPH GENERATOR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

// L a r g e r f u n c t i o n u s e d t o g e n e r a t e graph o f AST // w i t h a l l t y p e s and a d d i t i o n a l e d g e s shown . // Graphs o f t h i s s o r t a r e l a r g e , and can be // v i e w e d u s i n g z g r v i e w e r f o r d o t f i l e s . int foo ( int x ) ; int globalVar = 42; void foobar A ( ) { int a = 4; int b = a + 2; int c = b globalVar ; int x ; x = foo ( c ) ; int y = x + 2; int z = globalVar y ; }

void foobar B () { int p; int i = 4; i n t k = globalVar ( i +2); p = foo (k ) ; i n t r = ( p+2) g l o b a l V a r ; }

Figure 4.4: Example source code used as input to generate a larger AST graph with attributes. can be visualized, in practice this is only useful for debugging small codes in the construction of custom analysis and transformation tools. The graphs for real million line applications would never be visualized. Using ROSE one can build automated tools to operate on the AST for large scale applications where visualization would not be possible.

27

28

CHAPTER 4. AST WHOLE GRAPH GENERATOR

Chapter 5

Advanced AST Graph Generation

What To Learn From This Example This example shows a maximally complete representation of the AST (often in more detail that is useful). Where chapter 3 presented a ROSE-based translator which presented the AST as a tree, this chapter presents the more general representation of the graph in which the AST is embedded. The AST may be thought of as a subset of a more general graph or equivalently as an AST (a tree in a formal sense) with annotations (extra edges and information), sometimes referred to as a decorated AST. We present tools for seeing all the IR nodes in the graph containing the AST, including all types (SgType nodes), symbols (SgSymbol nodes), compiler generated IR nodes, and supporting IR nodes. In general it is a specic ltering of this larger graph which is more useful to communicating how the AST is designed and internally connected. We use these graphs for internal debugging (typically on small problems where the graphs are reasonable in size). The graphs presented using these graph mechanism present all back-edges, and demonstrate what IR nodes are shared internally (typically SgType IR nodes). First a few names, we will call the AST those nodes in the IR that are specied by a traversal using the ROSE traversal (SgSimpleTraversal, etc.). We will call the graph of all IR nodes the Graph of all IR nodes. the AST is embedded in the Graph of all IR nodes. The AST is a tree, while the graph of all IR nodes typically not a tree (in a Graph Theory sense) since it typically contains cycles. We cover the visualization of both the AST and the Graph of all IR nodes. AST graph These techniques dene ways of visualizing the AST and ltering IR nodes from being represented. Simple AST graphs Colored AST graphs Filtering the graph The AST graph may be generated for any subtree of the AST (not possible for the graphs of all IR nodes). Additionally runtime options permit null pointers to be ignored. . 29
FIXME: This chapter brings more confusions. I suggest to remove it until it is done right . -Leo

FIXME: Is this true?

30

CHAPTER 5. ADVANCED AST GRAPH GENERATION Graph of all IR nodes These techniques dene the ways of visualizing the whole graph of IR nodes and is based on the memory pool traversal as a means to access all IR nodes. Even disconnected portions of the AST will be presented. Simple graphs Colored graphs Filtering the graph

Removed this example newer mechanism for the whole AST graphs needs to be presented.

Chapter 6

AST PDF Generator

What To Learn From This Example This example demonstrates a mechanism for generating a visualization of the AST using pdf les. A pdf le is generated and can be viewed using acroread. The format is suitable for much larger input programs than the example shown in previous chapters using dot format 4. This mechanism can support the visualization of input les around 100K lines of code.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 // Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // G e n e r a t e a PDF f i l e f o r i n t e r a c t i v e generatePDF ( p r o j e c t ) ; return 0; } e x p l o r a t i o n o f t h e AST .

Figure 6.1: Example source code to read an input program and generate a PDF le to represent the AST. The program in gure 6.1 calls an internal ROSE function that traverses the AST and generates an ASCI le in dot format. Figure 3.2 shows an input code which is processed to generate a graph of the AST, generating a pdf le. The pdf le is then processed using acroread to generate a GUI for viewing the AST. A standalone utility tool, called pdfGenerator is provided within ROSE. It is available from ROSE BUILD/exampleTranslators/PDFGenerator or ROSE INS/bin. Users can use it to generate AST in a pdf format from an input code. Figure 6.3 displays on the left hand side the individual C++ nodes in ROSEs intermediate representation (IR). The page on the right hand side shows that IR nodes member data. Pointers in boxes can be clicked on to navigate the AST (or nodes in the tree hierarchy can be clicked on jump to any location in the AST. This representation shows only the IR nodes that are traversed 31

32

CHAPTER 6. AST PDF GENERATOR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; }

Figure 6.2: Example source code used as input to generate the PDF le of the AST. by the standard traversal (no SgSymbol or SgType IR nodes are presented in this view of the AST). The output of this translator is shown in gure 6.3. The left hand side of the screen is a tree with click-able nodes to expand/collapse the subtrees. The right hand side of the screen is a description of the data at a particular node in the AST (the node where the user has clicked the left mouse button). This relatively simple view of the AST is useful for debugging transformation and nding information in the AST required by specic sorts of analysis. It is also useful for developing an intuitive feel for what information is in the AST, how it is organized, and where it is stored.

33

Figure 6.3: Example output from translator which outputs PDF representation of AST. The generated PDF le makes use of the bookmark mechanism to expand and collapse parts of the AST.

34

CHAPTER 6. AST PDF GENERATOR

Chapter 7

Introduction to AST Traversals

An essential operation in the analysis and construction of ASTs is the denition of traversals upon the AST to gather information and modify targeted internal representation (IR) nodes. ROSE includes dierent sorts of traversals to address the dierent requirements of numerous program analysis and transformation operations. This section demonstrates the dierent types of traversals that are possible using ROSE. ROSE translators most commonly introduce transformations and analysis through a traversal over the AST. Alternatives could be to generate a simpler IR that is more suitable to a specic transformation and either convert modication to that transformation specic IR into changes to the AST or generate source code from the transformation specic IR directly. These approaches are more complex than introducing changes to the AST directly, but may be better for specic transformations. Traversals represent an essential operation on the AST and there are a number of dierent types of traversals. The suggested traversals for users are explained in Section 7.2. Section 7.3 introduces specialized traversals (that traverse the AST in dierent orders and traverse types and symbols), typically not appropriate for most translators (but perhaps appropriate for specialized tools, often internal tools within ROSE). See the ROSE User Manual for a more complete introduction to the dierent types of traversals. The purpose of this tutorial is to present examples, but we focus less on the background and philosophy here than in the ROSE User Manual. This chapter presents a number of ways of traversing the AST of any input source code. These traversals permit operations on the AST, which may either read or modify the AST in place. Modications to the AST will be reected in the source code generated when the AST is unparsed; the code generation phase of the source-to-source process dened by ROSE. Note that for all examples, the input code described in section 7.1 is used to generate all outputs shown with each translator.
FIXME: Add What to learn from this example paragraph to each example. FIXME: Add What is dierent from previous example paragraph to each example. FIXME: Add a table and/or graph at the end of this chapter to summarize the traversals.

7.1

Input For Example Traversals

The code shown in gure 7.1 shows the input code that will be used to demonstrate the traversals in this chapter. It may be modied by the user to experiment with the use of the traversals on 35

36

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; f o r ( i n t i =0; i < 4 ; { int x ; } i ++) for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 7.1: Example source code used as input to program in traversals shown in this chapter. alternative input codes.

7.2

Traversals of the AST Structure

This collection of traversals operates on the AST in an order which matches the structure of the AST and the associated source code. These types of traversals are the most common traversals for users to use. A subsequent section of this chapter demonstrated more specialized traversals over all IR nodes (more than just those IR nodes in the AST representing the structure of the source code) that are suitable for some tools, mostly tools built internally within ROSE.

7.2. TRAVERSALS OF THE AST STRUCTURE

37

Because the traversals in this section traverse the structure of the source code (see the AST graph presented in the rst tutorial example) they are more appropriate for most transformations of the source code. We suggest that the user focus on these traversals which represent the interface we promote for analysis and transformation of the AST, instead of the memory pools traversals which are suitable mostly for highly specialized internal tools. The simple traversals of both kinds have the same interface so the user may easily switch between them with out signicant diculty.

7.2.1

Classic Object-Oriented Visitor Pattern for the AST

We show this example rst, but it is rarely used in practice, and more useful traversals follow. It is however most closely similar to traversals that are available in other compiler infrastructures, and so a concept with which many people will be familar. In this case because this implementation is based on the memory pool infrstructure it will visit all node and not in any order based on the AST. The ASTSimpleProcessing traversal in section 7.2.2 is closer to a common visitor pattern that visits the IR nodes in the order in which they appear in the AST. Figure 7.2 shows the source code for a translator using the classic object-oriented visitor pattern to traverse the AST. This visitor pattern is only implemented for the memory pool based traversal. Thus it works on the whole of the attributed AST and does not work on a restricted subset of the AST (e.g. a subtree of the unattributed AST). Figure 7.3 shows the output from this traversal using the example input source from gure 7.1.

7.2.2

Simple Traversal (no attributes)

Figure 7.4 shows the source code for a translator which traverses the AST. The traversal object is from the type visitorTraversal derived from AstSimpleProcessing. The visit() function is required to be dened because it is dened as a pure virutal funciton in the AstSimpleProcessing base class. The member function traverseInputFiles() of AstSimpleProcessing is called to traverse the AST and call the visit() function on each IR node. Note that the function traverse() (not used) would visit each IR nodes while traverseInputFiles() will only visit those IR nodes that originate in the input source code (thus skipping all header les). For each node where the visit() function is called a SgNode pointer is to the node is passed into the visit function. Note that using this simple traversal the only context information available to the visit function is what is stored in its member variables (though access to other nodes is possible along any edges in the attributed AST graph). The only option is to traverse the AST in either pre-order or postorder. The atTraversalEnd() function may be dened by the user to do nal processing after all nodes have been visited (or to perform preparations before the nodes are visited, in the case of the corresponding atTraversalStart() function). Figure 7.5 shows the output from this traversal using the example input source from gure 7.1.

7.2.3

Simple Pre- and Postorder Traversal

Figure 7.6 shows the source code for a translator that traverses the AST without attributes (like the one in the previous subsection), but visiting each node twice, once in preorder (before its children) and once in postorder (after all children). Figure 7.7 shows the output from this traversal using the example input source from gure 7.1.

38

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

#i n c l u d e r o s e . h // C l a s s i c V i s i t o r P a t t e r n i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t ha s no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s C l a s s i c V i s i t o r : p u b l i c ROSE VisitorPattern { public : // O v e r r i d e v i r t u r a l f u n c t i o n d e f i n e d i n b a s e c l a s s void v i s i t ( SgGlobal g l o b a l S c o p e ) { p r i n t f ( Found t h e S g G l o b a l IR node \ n ) ; } void v i s i t ( SgFunctionDeclaration functionDeclaration ) { p r i n t f ( Found a S g F u n c t i o n D e c l a r a t i o n IR node \ n ) ; } v o i d v i s i t ( SgTypeInt i n t T y p e ) { p r i n t f ( Found a SgTypeInt IR node \ n ) ; } v o i d v i s i t ( SgTypeDouble doubleType ) { p r i n t f ( Found a SgTypeDouble IR node \ n ) ; } };

int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // C l a s s i c v i s i t o r p a t t e r n o v e r t h e memory p o o l o f IR n o d e s ClassicVisitor visitor A ; traverseMemoryPoolVisitorPattern ( v i s i t o r A ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 7.2: Example source showing simple visitor pattern.

7.2.4

Inherited Attributes

Figure 7.8 shows the use of inherited attributes associated with each IR node. Within this traversal the attributes are managed by the traversal and exist on the stack. Thus the lifetime of the attributes is only as long as the processing of the IR node and its subtree. Attributes such as this are used to communicate context information down the AST and called Inherited attributes. In the example the class Inherited Attribute is used to represent inherited attribute. Each instance of the class represents an attribute value. When the AST is traversed we obtain as output the loop nest depth at each point in the AST. The output uses the example input source from gure 7.1. Note that inherited attributes are passed by-value down the AST. In very rare cases you

7.2. TRAVERSALS OF THE AST STRUCTURE

39

might want to pass a pointer to dynamically allocated memory as an inherited attribute. In this case you can dene the virtual member function void destroyInheritedValue(SgNode *n, InheritedAttribute inheritedValue) which is called after the last use of the inherited attribute computed at this node, i. e. after all children have been visited. You can use this function to free the memory allocated for this inherited attribute.

40

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found

a SgTypeInt IR node a SgTypeDouble IR node t h e S g G l o b a l IR node a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration

IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR

node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node

7.2. TRAVERSALS OF THE AST STRUCTURE

41

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h class { visitorTraversal : public AstSimpleProcessing

public : visitorTraversal (); v i r t u a l v o i d v i s i t ( SgNode n ) ; v i r t u a l void atTraversalEnd ( ) ; }; visitorTraversal : : visitorTraversal () { } v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( Found a f o r l o o p . . . \ n ) ; } } void v i s i t o r T r a v e r s a l : : atTraversalEnd ( ) { p r i n t f ( T r a v e r s a l e n d s h e r e . \n ) ; } int main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( I n v i s i t o r T r a v e r s a l . C : main ( ) \ n ) ; SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l f u n c t i o n ( member f u n c t i o n o f A s t S i m p l e P r o c e s s i n g ) // s t a r t i n g a t t h e p r o j e c t node o f t h e AST, u s i n g a p r e o r d e r t r a v e r s a l . exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 7.4: Example source showing simple visitor pattern.

1 2 3

Found a f o r l o o p . . . Found a f o r l o o p . . . T r a v e r s a l ends here .

Figure 7.5: Output of input le to the visitor traversal.

42

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h c l a s s PreAndPostOrderTraversal : p u b l i c AstPrePostProcessing { public : v i r t u a l v o i d p r e O r d e r V i s i t ( SgNode n ) ; v i r t u a l v o i d p o s t O r d e r V i s i t ( SgNode n ) ; }; v o i d P r e A n d P o s t O r d e r T r a v e r s a l : : p r e O r d e r V i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( E n t e r i n g f o r l o o p . . . \n ) ; } } v o i d P r e A n d P o s t O r d e r T r a v e r s a l : : p o s t O r d e r V i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( L e a v i n g f o r l o o p . . . \n ) ; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( I n p r e P o s t T r a v e r s a l . C : main ( ) \ n ) ; SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t PreAndPostOrderTraversal exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t ) ; return 0; }

Figure 7.6: Example source showing simple pre- and postorder pattern.

1 2 3 4

Entering f o r loop . . . Leaving f o r loop . . . Entering f o r loop . . . Leaving f o r loop . . .

Figure 7.7: Output of input le to the pre- and postorder traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

43

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h // B u i l d an i n h e r i t e d a t t r i b u t e class InheritedAttribute { public : // Depth i n AST i n t depth ; i n t maxLinesOfOutput ; f o r the t r e e t r a v e r s a l t o t e s t t h e r e w r i t e mechanism

// S p e c i f i c c o n s t r u c t o r s a r e r e q u i r e d I n h e r i t e d A t t r i b u t e ( i n t x ) : depth ( x ) , maxLinesOfOutput ( 2 0 ) { } ; I n h e r i t e d A t t r i b u t e ( c o n s t I n h e r i t e d A t t r i b u t e & X ) : depth (X . depth ) , maxLinesOfOutput ( 2 0 ) { } ; }; class { visitorTraversal public : // v i r t u a l virtual }; InheritedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode n , I n h e r i t e d A t t r i b u t e i n h e r i t e d A t t r i b u t e ) { s t a t i c i nt linesOfOutput = 0; i f ( l i n e s O f O u t p u t++ < i n h e r i t e d A t t r i b u t e . maxLinesOfOutput ) p r i n t f ( Depth i n AST a t %s = %d \ n , n > s a g e c l a s s n a m e ( ) , i n h e r i t e d A t t r i b u t e . depth ) ; r e t u r n I n h e r i t e d A t t r i b u t e ( i n h e r i t e d A t t r i b u t e . depth + 1 ) ; } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // DQ ( 1 / 1 8 / 2 0 0 6 ) : P a r t o f d e b u g g i n g SgFile & l o c a l F i l e = project >g e t f i l e ( 0 ) ; l o c a l F i l e . g e t f i l e i n f o ()> d i s p l a y ( l o c a l F i l e // B u i l d t h e i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute (0); // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , i n h e r i t e d A t t r i b u t e ) ; // Or t h e t r a v e r s a l o v e r a l l AST IR n o d e s can be c a l l e d ! exampleTraversal . t r a v e r s e ( project , i n h e r i t e d A t t r i b u t e ) ; return 0; } : p u b l i c AstTopDownProcessing < I n h e r i t e d A t t r i b u t e >

f u n c t i o n must be d e f i n e d I n h e r i t e d A t t r i b u t e e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode n ,

InheritedAttribute

inheritedAttribute );

information );

Figure 7.8: Example source code showing use of inherited attributes (passing context information down the AST.

44

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Inside of Sg File Info : : display ( l o c a l F i l e information ) isTransformation = false isCompilerGenerated = false isOutputInCodeGeneration = false isShared = false isFrontendSpecific = false isSourcePositionUnavailableInFrontend = f a l s e isCommentOrDirective = false isToken = false f i l e n a m e = / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p line = 1 column = 1 Depth i n AST a t S g S o u r c e F i l e = 0 Depth i n AST a t S g G l o b a l = 1 Depth i n AST a t S g T e m p l a t e D e c l a r a t i o n = 2 Depth i n AST a t S g F u n c t i o n D e c l a r a t i o n = 2 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 3 Depth i n AST a t S g I n i t i a l i z e d N a m e = 4 Depth i n AST a t S g F u n c t i o n D e c l a r a t i o n = 2 Depth i n AST a t S g F u n c t i o n P a r a m e t e r L i s t = 3 Depth i n AST a t S g I n i t i a l i z e d N a m e = 4 Depth i n AST a t S g F u n c t i o n D e f i n i t i o n = 3 Depth i n AST a t S g B a s i c B l o c k = 4 Depth i n AST a t S g V a r i a b l e D e c l a r a t i o n = 5 Depth i n AST a t S g I n i t i a l i z e d N a m e = 6 Depth i n AST a t S g A s s i g n I n i t i a l i z e r = 7 Depth i n AST a t S g I n t V a l = 8 Depth i n AST a t S g V a r i a b l e D e c l a r a t i o n = 5 Depth i n AST a t S g I n i t i a l i z e d N a m e = 6 Depth i n AST a t Sg Fo r St at ement = 5 Depth i n AST a t S g F o r I n i t S t a t e m e n t = 6 Depth i n AST a t S g V a r i a b l e D e c l a r a t i o n = 7

Figure 7.9: Output of input le to the inherited attribute traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

45

7.2.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

Synthesized Attributes

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h #i n c l u d e < a l g o r i t h m > #i n c l u d e < f u n c t i o n a l > #i n c l u d e <numeric > typedef bool SynthesizedAttribute ; class { visitorTraversal public : // v i r t u a l virtual }; SynthesizedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode n , S y n t h e s i z e d A t t r i b u t e s L i s t { // Fold up t h e l i s t o f c h i l d a t t r i b u t e s u s i n g l o g i c a l or , i . e . t h e l o c a l // r e s u l t w i l l be t r u e i f f one o f t h e c h i l d a t t r i b u t e s i s t r u e . SynthesizedAttribute localResult = s t d : : a c c u m u l a t e ( c h i l d A t t r i b u t e s . b e g i n ( ) , c h i l d A t t r i b u t e s . end ( ) , f a l s e , s t d : : l o g i c a l o r <b o o l > ( ) ) ; if ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( Found a f o r l o o p . . . localResult = true ; } : p u b l i c AstBottomUpProcessing < S y n t h e s i z e d A t t r i b u t e >

f u n c t i o n must be d e f i n e d SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode n , S y n t h e s i z e d A t t r i b u t e s L i s t c h i l d A t t r i b u t e s

);

childAttributes )

\n ) ;

return localResult ; } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST SynthesizedAttribute r e s u l t = exampleTraversal . t r a v e r s e ( p r o j e c t ) ; if ( r e s u l t == t r u e ) { p r i n t f ( The program c o n t a i n s a t l e a s t one l o o p ! \ n ) ; }

return 0; }

Figure 7.10: Example source code showing use of synthesized attributed (passing analysis information up the AST). Figure 7.10 shows the use of attributes to pass information up the AST. The lifetime of the

46

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3

Found a f o r l o o p . . . Found a f o r l o o p . . . The program c o n t a i n s a t l e a s t one l o o p !

Figure 7.11: Output of input le to the synthesized attribute traversal. attributes are similar as for inherited attributes. Attributes such as these are called synthesized attributes. This code shows the code for a translator which does an analysis of an input source code to determine the presence of loops. It returns true if a loop exists in the input code and false otherwise. The list of synthesized attributes representing the information passed up the AST from a nodes children is of type SynthesizedAttributesList, which is a type that behaves very similarly to vector<SynthesizedAttribute> (it supports iterators, can be indexed, and can be used with STL algorithms). The example determines the existence of loops for a given program.

7.2. TRAVERSALS OF THE AST STRUCTURE

47

7.2.6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

Accumulator Attributes

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h // B u i l d an a c c u m u l a t o r a t t r i b u t e , f a n c y name f o r what i s c l a s s AccumulatorAttribute { public : i n t forLoopCounter ; essentially a global v a r i a b l e : ).

// S p e c i f i c c o n s t r u c t o r s a r e o p t i o n a l AccumulatorAttribute ( ) { forLoopCounter = 0 ; } A c c u m u l a t o r A t t r i b u t e ( c o n s t A c c u m u l a t o r A t t r i b u t e & X ) {} A c c u m u l a t o r A t t r i b u t e & o p e r a t o r= ( c o n s t A c c u m u l a t o r A t t r i b u t e & X ) { r e t u r n t h i s ; } }; class { visitorTraversal : public AstSimpleProcessing

public : s t a t i c AccumulatorAttribute accumulatorAttribute ; v i r t u a l v o i d v i s i t ( SgNode n ) ; }; // d e c l a r a t i o n r e q u i r e d f o r s t a t i c d a t a member AccumulatorAttribute v i s i t o r T r a v e r s a l : : accumulatorAttribute ; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( Found a f o r l o o p . . . \ n ) ; a c c u m u l a t o r A t t r i b u t e . f o r L o o p C o u n t e r ++; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST // can be s p e c i f i e d t o be p r e o r d e r o r p o s t o r d e r ) . exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; p r i n t f ( Number o f return 0; } f o r l o o p s i n i n p u t a p p l i c a t i o n = %d \ n , e x a m p l e T r a v e r s a l . a c c u m u l a t o r A t t r i b u t e . f o r L o o p C o u n t e r ) ;

Figure 7.12: Example source code showing use of accumulator attributes (typically to count things in the AST). Figure 7.12 shows the use of a dierent sort of attribute. This attribute has a lifetime equal to the lifetime of the traversal object (much longer than the traversal of any subset of IR nodes). The same attribute is accessible from each IR node. Such attributes are called accumulator

48

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3

Found a f o r l o o p . . . Found a f o r l o o p . . . Number o f f o r l o o p s i n i n p u t a p p l i c a t i o n = 2

Figure 7.13: Output of input le to the accumulator attribute traversal. attributes and are semantically equivalent to a global variable. Accumulator attributes act as global variables which can easily be used to count application specic properties within the AST. Note that due to the limitation that the computation of inherited attributes cannot be made dependent on the values of synthesized attributes, counting operations cannot be implemented by combining these attributes as is usually done in attribute grammars. However, the use of accumulator attributes serves well for this purpose. Therefore all counting-like operations should be implemented using accumulator attributes (= member variables of traversal or processing classes). Although not shown in this tutorial explicitly, accumulator attributes may be easily mixed with inherited and/or synthesized attributes. In this example we count the number of for-loops in an input program.

7.2.7

Inherited and Synthesized Attributes

Figure 7.14 shows the combined use of inherited and synthesized attributes. The example source code shows the mixed use of such attributes to list the functions containing loop. Inherited attributes are used to communicate that the traversal is in a function, which the synthesized attributes are used to pass back the existence of loops deeper within the subtrees associated with each function. List of functions containing loops.

7.2. TRAVERSALS OF THE AST STRUCTURE

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 // ROSE i s #i n c l u d e a tool for building preprocessors , this file is an e x a m p l e preprocessor built w i t h ROSE .

49

r o s e . h

#i n c l u d e < a l g o r i t h m > #i n c l u d e < f u n c t i o n a l > #i n c l u d e <n u m e r i c > typedef typedef class { bool bool InheritedAttribute ; SynthesizedAttribute ; : public SgTopDownBottomUpProcessing< I n h e r i t e d A t t r i b u t e , S y n t h e s i z e d A t t r i b u t e >

Traversal

public : // F u n c t i o n s r e q u i r e d InheritedAttribute evaluateInheritedAttribute SgNode a s t N o d e , InheritedAttribute inheritedAttribute );

SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute , SubTreeSynthesizedAttributes synthesizedAttributeList }; InheritedAttribute Traversal : : evaluateInheritedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute ) { i f ( i s S g F u n c t i o n D e f i n i t i o n ( astNode ) ) { // The i n h e r i t e d a t t r i b u t e i s t r u e i f f return true ; } return inheritedAttribute ; }

);

we a r e

inside

function .

SynthesizedAttribute Traversal : : evaluateSynthesizedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute , SynthesizedAttributesList childAttributes ) { i f ( i n h e r i t e d A t t r i b u t e == f a l s e ) { // The i n h e r i t e d a t t r i b u t e i s f a l s e , i . e . we a r e n o t i n s i d e any // f u n c t i o n , s o t h e r e c a n be no l o o p s h e r e . return f a l s e ; } else { // F o l d up t h e l i s t o f c h i l d a t t r i b u t e s u s i n g l o g i c a l o r , i . e . t h e // r e s u l t w i l l be t r u e i f f o n e o f t h e c h i l d a t t r i b u t e s i s t r u e . SynthesizedAttribute localResult = s t d : : a c c u m u l a t e ( c h i l d A t t r i b u t e s . b e g i n ( ) , c h i l d A t t r i b u t e s . end ( ) , f a l s e , s t d : : l o g i c a l o r <b o o l > ( ) ) ; i f ( i s S g F u n c t i o n D e f i n i t i o n ( a s t N o d e ) && l o c a l R e s u l t == t r u e ) { p r i n t f ( Found a f u n c t i o n c o n t a i n i n g a f o r l o o p . . . \ n ) ; } i f ( isSgForStatement ( astNode ) ) { localResult = true ; } return localResult ; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e a b s t r a c t s y n t a x t r e e SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // Build the i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute = false ; Define the t r a v e r s a l T r a v e r s a l myTraversal ; C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t ( r o o t ) node o f t h e AST myTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t , i n h e r i t e d A t t r i b u t e ) ; T h i s program return 0; only does analysis , so it need not call the backend to

local

//

//

// }

generate

code .

Figure 7.14: Example source code showing use of both inherited and synthesized attributes working together (part 1).

50

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2

Found a f u n c t i o n c o n t a i n i n g a f o r l o o p Found a f u n c t i o n c o n t a i n i n g a f o r l o o p

... ...

Figure 7.15: Output of input le to the inherited and synthesized attribute traversal.

7.2. TRAVERSALS OF THE AST STRUCTURE

51

7.2.8

Persistent Attributes

Figure 7.16 shows the use of another form of attribute. This attribute has a lifetime which is controlled explicitly by the user; it lives on the heap typically. These attributes are explicitly attached to the IR nodes and are not managed directly by the traversal. There attributes are called persistent attributes and are not required to be associated with any traversal. Persistent attributes are useful for storing information across multiple traversals (or permanently within the AST) for later traversal passes. Persistent attributes may be used at any time and combined with other traversals (similar to accumulator attributes). Traversals may combine any or all of the types of attributes within in ROSE as needed to store, gather, or propagate information within the AST for complex program analysis.

52

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h class { persistantAttribute : public AstAttribute

public : int value ; p e r s i s t a n t A t t r i b u t e ( i n t v ) : v a l u e ( v ) {} }; class { }; v o i d v i s i t o r T r a v e r s a l S e t A t t r i b u t e : : v i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( Found a f o r l o o p ( s e t t h e a t t r i b u t e ) visitorTraversalSetAttribute public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

...

\n ) ;

// B u i l d an a t t r i b u t e ( on t h e heap ) A s t A t t r i b u t e n e w A t t r i b u t e = new p e r s i s t a n t A t t r i b u t e ( 5 ) ; ROSE ASSERT( n e w A t t r i b u t e != NULL ) ; // Add i t t o t h e AST ( s o i t can be f o u nd l a t e r i n a n o t h e r p a s s o v e r t h e AST) n >addNewAttribute ( MyNewAttribute , n e w A t t r i b u t e ) ; } } class { }; v o i d v i s i t o r T r a v e r s a l R e a d A t t r i b u t e : : v i s i t ( SgNode n ) { i f ( i s S g F o r S t a t e m e n t ( n ) != NULL) { p r i n t f ( Found a f o r l o o p ( r e a d t h e a t t r i b u t e ) // // // // // visitorTraversalReadAttribute public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

...

\n ) ;

Add i t t o t h e AST ( s o i t can be f o u nd l a t e r i n a n o t h e r p a s s o v e r t h e AST) A s t A t t r i b u t e e x i s t i n g A t t r i b u t e = n > a t t r i b u t e [ MyNewAttribute ] ; DQ ( 1 / 2 / 2 0 0 6 ) : Added s u p p o r t f o r new a t t r i b u t e i n t e r f a c e . p r i n t f ( v i s i t o r T r a v e r s a l R e a d A t t r i b u t e : : v i s i t ( ) : u s i n g new a t t r i b u t e i n t e r f a c e \ n ) ; A s t A t t r i b u t e e x i s t i n g A t t r i b u t e = n > a t t r i b u t e ( ) [ MyNewAttribute ] ; A s t A t t r i b u t e e x i s t i n g A t t r i b u t e = n > g e t A t t r i b u t e ( MyNewAttribute ) ; ROSE ASSERT( e x i s t i n g A t t r i b u t e != NULL ) ; p r i n t f ( Existing a t t r i b u t e a t %p v a l u e = %d \ n , n , d y n a m i c c a s t < p e r s i s t a n t A t t r i b u t e >( e x i s t i n g A t t r i b

} } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t t o s e t p e r s i s t a n t AST a t t r i b u t e s visitorTraversalSetAttribute exampleTraversalSettingAttribute ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversalSettingAttribute . traverseInputFiles ( project , preorder ) ; // B u i l d t h e t r a v e r s a l o b j e c t t o r e a d any e x i s t i n g AST a t t r i b u t e s visitorTraversalReadAttribute exampleTraversalReadingAtribute ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversalReadingAtribute . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 7.16: Example source code showing use of persistent attributes used to pass information across multiple passes over the AST.

7.2. TRAVERSALS OF THE AST STRUCTURE

53

1 2 3 4 5 6

Found a Found a Found a Existing Found a Existing

f o r loop ( s e t the a t t r i b u t e ) . . . f o r loop ( s e t the a t t r i b u t e ) . . . f o r loop ( read the a t t r i b u t e ) . . . a t t r i b u t e at 0 x2af0dbb95010 value = 5 f o r loop ( read the a t t r i b u t e ) . . . a t t r i b u t e at 0 x2af0dbb95130 value = 5

Figure 7.17: Output of input le to the persistent attribute traversal showing the passing of information from one AST traversal to a second AST traversal.

54

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.2.9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

Nested Traversals

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h class { }; class { }; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { i f ( i s S g F u n c t i o n D e c l a r a t i o n ( n ) != NULL) { p r i n t f ( Found a f u n c t i o n d e c l a r a t i o n nestedVisitorTraversal public : v i r t u a l void : public AstSimpleProcessing visitorTraversal public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

v i s i t ( SgNode n ) ;

...

\n ) ;

// B u i l d t h e n e s t e d t r a v e r s a l o b j e c t ne st ed V i s i t o r T r a v e r sa l exampleTraversal ;

// C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST ( t r a v e r s e i n p o s t o r d e r j u s t t o be d i f f e r e // Note t h a t we c a l l t h e t r a v e r s e f u n c t i o n i n s t e a d o f t r a v e r s e I n p u t F i l e s , b e c a u s e we a r e n o t s t a r t i n g a // t h e AST r o o t . exampleTraversal . t r a v e r s e (n , postorder ) ; } } v o i d n e s t e d V i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { i f ( i s S g F u n c t i o n D e f i n i t i o n ( n ) != NULL) { p r i n t f ( Found a f u n c t i o n d e f i n i t i o n } }

within the f u n c t i o n

declaration

...

\n ) ;

int main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( I n v i s i t o r T r a v e r s a l . C : main ( ) \ n ) ; SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 7.18: Example source code showing use nested traversals. Figure 7.18 shows the use of multiple traversals in composition. Figure 7.19 shows the output

7.2. TRAVERSALS OF THE AST STRUCTURE

55

1 2 3 4 5 6 7 8

Found Found Found Found Found Found Found Found

a a a a a a a a

function function function function function function function function

declaration . . . declaration . . . d e f i n i t i o n within the f u n c t i o n declaration . . . declaration . . . declaration . . . declaration . . . d e f i n i t i o n within the f u n c t i o n

declaration

...

declaration

...

Figure 7.19: Output of input le to the nested traversal example. of the nested traversal.

56

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.2.10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

Combining all Attributes and Using Primitive Types

i n t main ( ) { i n t x =1; f o r ( i n t i =1; i < 10; i ++) f o r ( i n t j=i ; j < 10; j ++) f o r ( i n t k=i ; k < 10; k++) f o r ( i n t l=i ; l < 10; l ++) for ( int m =i ;m< 10;m++) x++; i n t i =5 , j =7; w h i l e ( i > 0) { w h i l e ( j > 0) { x++; j ; i ; } } i =10; do { x++; i ; } w h i l e ( i >0); return x ; }

Figure 7.20: Input code with nested loops for nesting info processing The previous examples have shown cases where attributes were classes, alternatively attributes can be any primitive type (int, bool, etc.). This example demonstrates how to use AstTopDownBottomUpProcessing to compute inherited and synthesized attributes, generate pdf and dot output, how to accumulate information, and how to attach attributes to AST nodes in the same pass. The attributes are used to compute the nesting level and the nesting depth of for/while/dowhile loops: The nesting level is computed using an inherited attribute. It holds that nesting level(innerloop) = nesting level(outerloop) + 1 starting with 1 at the outer most loop. The nesting depth is computed using a synthesized attribute. It holds that nesting depth(innerloop) = nesting level(outerloop) 1 starting with 1 at the inner most loop. To compute the values we use a primitive type (unsigned int). This example also shows how to use defaultSynthesizedAttribute to initialize a synthesized attribute of primitive type. The values of the attributes are attached to the AST using AstAttribute and the AST node attribute mechanism available at every AST node (which can be accessed with node->attribute). (see loopNestingInfoProcessing.C) For the entire program the maximum nesting level (= max nesting depth) is computed as accumulated value using member variable _maxNestingLevel of class LoopNestingInfoProcessing. We also demonstrate how to customize an AstAttribute such that the value of the attribute is printed in a pdf output. (by overriding toString, see LoopNestingInfo class) In the generated pdf le (for some C++ input le) the values of the attributes can be viewed for each node (see printLoopInfo implementation). Further more we also generate a dot le, to visualize the tree using the graph visualization tool dot. The generated le can be converted to

7.2. TRAVERSALS OF THE AST STRUCTURE postscript (using dot) and viewed with gv.

57

7.2.11

Combined Traversals

Performing a large number of program analyses as separate traversals of the AST can be somewhat inecient as there is some overhead associated with visiting every node several times. ROSE therefore provides a mechanism for combining traversal objects of the same base type and evaluating them in a single traversal of the AST. This is entirely transparent to the individual traversal object, so existing code can be reused with the combination mechanism, and analyzers can be developed and tested in isolation and combined when needed. The one requirement that is placed on traversals to be combined is that they be independent of each other; in particular, this means that they should not modify the AST or any shared global data. Any output produced by the analyzers will be interleaved. Figure 7.24 shows the source code for a translator that combines three dierent analyzers into one traversal, each one counting the occurrences of a dierent type of AST node (as determined by a VariantT value). First three traversals are run after each other, as usual; then three traversal objects are passed (by pointer) to an object of type AstCombinedSimpleProcessing using its addTraversal method. One then invokes one of the usual traverse methods on this combined object with the same eect as if it had been called for each of the traversal objects individually. Any operation on the list of analyzers is possible using the get traversalPtrListRef method of the combined processing class that returns a reference to its internal list of analyzers (an object of type vector<AstSimpleProcessing *>). Any changes made through this reference will be reected in further traversals. In addition to AstCombinedSimpleProcessing, there is also a combined class for each of the other types of traversals discussed above: AstCombinedTopDownProcessing, AstCombinedBottomUpProcessing, etc. Where traversals using attributes are combined, all of the combined traversals must have the same attribute types (i. e. the same template parameters). Attributes are passed to and returned from the combined traversal as a vector.

58

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

// Author : Markus Schordan , Vienna U n i v e r s i t y o f Technology , 2 0 0 4 . // $ I d : l o o p N e s t i n g I n f o P r o c e s s i n g . C, v 1 . 1 2 0 0 6 / 0 4 / 2 4 0 0 : 2 2 : 0 0 d q u i n l a n Exp $ // #i n c l u d e < s t r i n g > // #i n c l u d e < i o s t r e a m > #i n c l u d e r o s e . h u s i n g namespace s t d ; typedef typedef typedef typedef unsigned i n t unsigned i n t NestingLevel NestingDepth NestingLevel ; NestingDepth ; InhNestingLevel ; SynNestingDepth ;

/ ! T h i s c l a s s i s u s e d t o a t t a c h i n f o r m a t i o n t o AST n o d e s . Method t o S t r i n g i s o v e r r i d d e n and c a l l e d when a p d f f i l e i s g e n e r a t e d . T h i s a l l o w s t o d i s p l a y t h e v a l u e o f an AST node a t t r i b u t e ( a n n o t a t i o n ) i n a p d f f i l e . / c l a s s NestingLevelAnnotation : public AstAttribute { public : NestingLevelAnnotation ( N e s t i n g L e v e l n , NestingDepth d ) : n e s t i n g L e v e l ( n ) , n e s t i n g D e p t h ( d ) {} NestingLevel getNestingLevel () { return nestingLevel ; } NestingDepth getNestingDepth ( ) { r e t u r n nestingDepth ; } string toString () { ost ringstream s s ; ss< < n e s t i n g L e v e l <<,<< n e s t i n g D e p t h ; return ss . str ( ) ; } private : NestingLevel nestingLevel ; NestingDepth nestingDepth ; }; / ! The l o o p n e s t i n g l e v e l and n e s t i n g depth f o r e a c h w h i l e / d o w h i l e / f o r l o o p n e s t i s computed . I t i s a t t a c h e d t o t h e AST a s a n n o t a t i o n and can be a c c e s s e d a s node >a t t r i b u t e [ l o o p N e s t i n g I n f o ] a f t e r t h e p r o c e s s i n g ha s been p e r f o r m e d . The maximum n e s t i n g l e v e l o f t h e whole AST i s computed a s a c c u m u l a t e d v a l u e i n a member v a r i a b l e and can be a c c e s s e d w i t h getMaxNestingLevel ( ) . / c l a s s L o o p L e v e l P r o c e s s i n g : p u b l i c AstTopDownBottomUpProcessing <I n h N e s t i n g L e v e l , SynNestingDepth > { public : L o o p L e v e l P r o c e s s i n g ( ) : m a x N e s t i n g L e v e l ( 0 ) {} / ! P e r f o r m s a t r a v e r s a l o f t h e AST and computes l o o p n e s t i n g i n f o r m a t i o n by u s i n g i n h e r i t e d and s y n t h e s i z e d a t t r i b u t e s . The r e s u l t s a r e a t t a c h e d t o t h e AST a s annotation . / v o i d a t t a c h L o o p N e s t i n g A n n o t a t o n ( S g P r o j e c t node ) { t r a v e r s e I n p u t F i l e s ( node , 0 ) ; } / ! R e t u r n s t h e maximum n e s t i n g l e v e l o f t h e e n t i r e AST ( o f t h e i n p u t R e q u i r e s a t t a c h L o o p N e s t i n g A n n o t a t i o n ( t o be c a l l e d b e f o r e ) / NestingLevel getMaxNestingLevel ( ) { return maxNestingLevel ; } file ).

protected : / / ! computes t h e n e s t i n g l e v e l I n h N e s t i n g L e v e l e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode , I n h N e s t i n g L e v e l ) ; / / ! computes t h e n e s t i n g depth SynNestingDepth e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode , I n h N e s t i n g L e v e l , S y n t h e s i z e d A t t r i b u t e s L i s t ) ; / / ! p r o v i d e s t h e d e f a u l t v a l u e 0 f o r t h e n e s t i n g depth SynNestingDepth d e f a u l t S y n t h e s i z e d A t t r i b u t e ( I n h N e s t i n g L e v e l i n h ) ; private : NestingLevel maxNestingLevel ; };

NestingLevel L o o p L e v e l P r o c e s s i n g : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode node ,

NestingLevel loopNestingLevel ) {

/ / ! compute maximum n e s t i n g l e v e l o f e n t i r e program i n a c c u m u l a t o r ( member v a r i a b l e ) i f ( loopNestingLevel > maxNestingLevel ) m a x N e s t i n g L e v e l=l o o p N e s t i n g L e v e l ; s w i t c h ( node >v a r i a n t T ( ) ) { c a s e V SgGotoStatement :

7.2. TRAVERSALS OF THE AST STRUCTURE

59

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79

c o u t << WARNING: Goto s t a t e m e n t f o u n d . We do n o t c o n s i d e r g o t o l o o p s . \ n ; // DQ ( 1 1 / 1 7 / 2 0 0 5 ) : Added r e t u r n s t a t m e n t t o a v o i d g++ w a r n i n g : c o n t r o l r e a c h e s end o f nonv o i d f u n c t i o n return loopNestingLevel ; break ; c a s e V SgDoWhileStmt : c a s e V SgForStatement : c a s e V SgWhileStmt : r e t u r n l o o p N e s t i n g L e v e l +1; default : return loopNestingLevel ; } } SynNestingDepth LoopLevelProcessing : : d e f a u l t S y n t h e s i z e d A t t r i b u t e ( InhNestingLevel inh ) { / ! we do no t need t h e i n h e r i t e d a t t r i b u t e h e r e a s d e f a u l t v a l u e f o r s y n t h e s i z e d a t t r i b u t e we s e t 0 , r e p r e s e n t i n g n e s t i n g depth 0 . / return 0; } SynNestingDepth L o o p L e v e l P r o c e s s i n g : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode node , I n h N e s t i n g L e v e l n e s t i n g L e v e l , S y n t h e s i z e d A t t r i b u t e s L i s t { i f ( nestingLevel > maxNestingLevel ) m a x N e s t i n g L e v e l=n e s t i n g L e v e l ; // compute maximum n e s t i n g depth o f s y n t h e s i z e d a t t r i b u t e s SynNestingDepth n e s t i n g D e p t h =0; f o r ( S y n t h e s i z e d A t t r i b u t e s L i s t : : i t e r a t o r i=l . b e g i n ( ) ; i != l . end ( ) ; i ++) { i f ( i >n e s t i n g D e p t h ) n e s t i n g D e p t h= i ; } s w i t c h ( node >v a r i a n t T ( ) ) { c a s e V SgDoWhileStmt : c a s e V SgForStatement : c a s e V SgWhileStmt : { n e s t i n g D e p t h ++; c o u t << N e s t i n g l e v e l : << n e s t i n g L e v e l << , n e s t i n g depth : << n e s t i n g D e p t h << e n d l ; break ; } default : { // DQ ( 1 1 / 1 7 / 2 0 0 5 ) : Nothing t o do h e r e , but e x p l i c i t } } // add l o o p n e s t i n g l e v e l a s a n n o t a t i o n t o AST N e s t i n g L e v e l A n n o t a t i o n n l a = new N e s t i n g L e v e l A n n o t a t i o n ( n e s t i n g L e v e l , n e s t i n g D e p t h ) ; ROSE ASSERT( n l a != NULL ) ; // DQ ( 1 / 2 / 2 0 0 6 ) : Added s u p p o r t f o r new a t t r i b u t e i n t e r f a c e . // p r i n t f ( L o o p L e v e l P r o c e s s i n g : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( ) : u s i n g new a t t r i b u t e #i f 0 i f ( node > g e t a t t r i b u t e ( ) == NULL) { A s t A t t r i b u t e M e c h a n i s m a t t r i b u t e P t r = new A s t A t t r i b u t e M e c h a n i s m ( ) ; ROSE ASSERT( a t t r i b u t e P t r != NULL ) ; node >s e t a t t r i b u t e ( a t t r i b u t e P t r ) ; } #e n d i f // node > a t t r i b u t e . add ( l o o p N e s t i n g I n f o , n l a ) ; // node > a t t r i b u t e ( ) . add ( l o o p N e s t i n g I n f o , n l a ) ; node >addNewAttribute ( l o o p N e s t i n g I n f o , n l a ) ; / / ! r e t u r n t h e maximum n e s t i n g depth a s s y n t h e s i z e d return nestingDepth ; } i n t main ( i n t a r g c , c h a r a r g v ) { // command l i n e p a r a m e t e r s a r e p a s s e d t o EDG // non EDG p a r a m e t e r s a r e p a s s e d ( t h r o u g h ) t o ROSE ( and t h e v e n d o r c o m p i l e r ) S g P r o j e c t r o o t=f r o n t e n d ( a r g c , a r g v ) ; LoopLevelProcessing t ; attribute i n t e r f a c e \n ) ;

l)

default in switch avoids

lots

of warnings .

Figure 7.22: Example source code showing use of inherited, synthesized, accumulator, and per-

60

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11

Output : Nesting Nesting Nesting Nesting Nesting Nesting Nesting Nesting Max l o o p

level :5 , level :4 , level :3 , level :2 , level :1 , level :2 , level :1 , level :1 , nesting

n e s t i n g depth : 1 n e s t i n g depth : 2 n e s t i n g depth : 3 n e s t i n g depth : 4 n e s t i n g depth : 5 n e s t i n g depth : 1 n e s t i n g depth : 2 n e s t i n g depth : 1 level : 5

Figure 7.23: Output code showing the result of using inherited, synthesized, and accumulator attributes.

7.2. TRAVERSALS OF THE AST STRUCTURE

61

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

#i n c l u d e < r o s e . h> c l a s s NodeTypeCounter : p u b l i c A s t S i m p l e P r o c e s s i n g { public : NodeTypeCounter ( enum VariantT v a r i a n t , s t d : : s t r i n g typeName ) : myVariant ( v a r i a n t ) , typeName ( typeName ) , c o u n t ( 0 ) { } protected : v i r t u a l v o i d v i s i t ( SgNode node ) { i f ( node >v a r i a n t T ( ) == myVariant ) { s t d : : c o u t << Found << typeName << s t d : : e n d l ; c o u n t++; } } v i r t u a l void atTraversalEnd ( ) { s t d : : c o u t << typeName << t o t a l : << c o u n t << s t d : : e n d l ; } private : enum VariantT myVariant ; s t d : : s t r i n g typeName ; unsigned i n t count ; }; i n t main ( i n t a r g c , c h a r a r g v ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; s t d : : c o u t << s e q u e n t i a l e x e c u t i o n o f t r a v e r s a l s << s t d : : e n d l ; NodeTypeCounter f o r S t a t e m e n t C o u n t e r ( V SgForStatement , f o r l o o p ) ; NodeTypeCounter i n t V a l u e C o u n t e r ( V SgIntVal , i n t c o n s t a n t ) ; NodeTypeCounter v a r D e c l C o u n t e r ( V S g V a r i a b l e D e c l a r a t i o n , v a r i a b l e d e c l a r a t i o n ) ; // t h r e e c a l l s t o t r a v e r s e , e x e c u t e d s e q u e n t i a l l y forStatementCounter . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; intValueCounter . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; varDeclCounter . t r a v e r s e I n p u t F i l e s ( p r o j e ct , preorder ) ; s t d : : c o u t << s t d : : e n d l ; s t d : : c o u t << combined e x e c u t i o n o f t r a v e r s a l s << s t d : : e n d l ; AstCombinedSimpleProcessing combinedTraversal ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V SgForStatement , f o r l o o p ) ) ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V SgIntVal , i n t c o n s t a n t ) ) ; c o m b i n e d T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeCounter ( V S g V a r i a b l e D e c l a r a t i o n , v a r i a b l e // one c a l l t o t r a v e r s e , e x e c u t i o n i s i n t e r l e a v e d combinedTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; }

declaration ));

Figure 7.24: Example source showing the combination of traversals.

62

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

sequential execution of traversals Found f o r l o o p Found f o r l o o p f o r loop t o t a l : 2 Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t Found i n t c o n s t a n t i n t c o n s t a n t t o t a l : 10 Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n Found v a r i a b l e d e c l a r a t i o n variable declaration total : 8 combined e x e c u t i o n o f t r a v e r s a l s Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found f o r l o o p Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found f o r l o o p Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t Found i n t c o n s t a n t Found v a r i a b l e d e c l a r a t i o n Found i n t c o n s t a n t f o r loop t o t a l : 2 i n t c o n s t a n t t o t a l : 10 variable declaration total : 8

Figure 7.25: Output of input le to the combined traversals. Note that the order of outputs changes as execution of several analyzers is interleaved.

7.2. TRAVERSALS OF THE AST STRUCTURE

63

7.2.12

Short-Circuiting Traversals

The traversal short-circuit mechanism is a simple way to cut short the traversal of a large AST once specic information has been obtained. It is purely an optimization mechanism, and a bit of a hack, but common within the C++ Boost community. Since the technique works we present it as a way of permitting users to avoid the full traversal of an AST that they might deam to be redundant of inappropriate. We dont expect that this mechanism will be particularly useful to most users and we dont recommend it. It may even at some point not be supported. However, we present it because it is a common technique used in the C++ Boost community and it happens to work (at one point it didnt work and so we have no idea what we xed that permitted it to work now). We have regarded this technique as a rather ugly hack. It is presented in case you really need it. It is, we think, better than the direct use of lower level mechanisms that are used to support the AST traversal.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// I n p u t f o r

t r a n s l a t o r t o show e x c e p t i o n b a s e d e x i t i n g from a t r a n s l a t o r .

namespace A { int go ; struct B { static int }; };

stop

void foo ( void ) { e x t e r n void bar ( i n t ) ; b a r (A : : go ); b a r (A : : B : : stop ); }

Figure 7.26: Input code with used to demonstrate the traversal short-circuit mechanism. Figure 7.27 shows the example code demonstrating a traversal setup to support the shortcircuit mechanism (a conventional mechanism used often within the C++ Boost community). The input code shown in gure 7.26 is compiled using the example translator, the output is shown in gure 7.28. The output shown in gure 7.28 demonstrates the initiation of a traversal over the AST and that traversal being short-circuited after a specic point in the evaluation. The result is that there is no further traversal of the AST after that point where it is short-circuited.

64

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

// Example o f an AST t r a v e r s a l t h a t u s e s t h e Boost i d i o m o f t h r o w i n g // an e x c e p t i o n t o e x i t t h e t r a v e r s a l e a r l y . #i n c l u d e < r o s e . h> #i n c l u d e < s t r i n g > #i n c l u d e < i o s t r e a m > u s i n g namespace s t d ; // E x c e p t i o n t o i n d i c a t e an e a r l y e x i t from a t r a v e r s a l a t some node . c l a s s StopEarly { public : S t o p E a r l y ( c o n s t SgNode n ) : e x i t n o d e ( n ) {} ( e . e x i t n o d e ) {} S t o p E a r l y ( c o n s t S t o p E a r l y& e ) : e x i t n o d e // P r i n t s i n f o r m a t i o n about t h e e x i t node . v o i d p r i n t ( o s t r e a m& o ) c o n s t { i f ( exit node ) { o << \ t << ( c o n s t v o i d ) e x i t n o d e << : << e x i t n o d e >c l a s s n a m e ( ) << e n d l ; c o n s t SgLocatedNode l o c n = i s S g L o c a t e d N o d e ( e x i t n o d e ) ; i f ( loc n ) { const S g F i l e I n f o info = loc n >g e t s t a r t O f C o n s t r u c t ( ) ; ROSE ASSERT ( i n f o ) ; o << \ tAt << i n f o > g e t f i l e n a m e ( ) << : << i n f o > g e t l i n e ( ) << e n d l ; } } } private : c o n s t SgNode e x i t n o d e ; // Node a t e a r l y };

e x i t from t r a v e r s a l

// P r e o r d e r t r a v e r s a l t o f i n d t h e f i r s t SgVarRefExp o f a p a r t i c u l a r name . c l a s s VarRefFinderTraversal : public AstSimpleProcessing { public : // I n i t i a t e t r a v e r s a l t o f i n d t a r g e t i n p r o j . v o i d f i n d ( S g P r o j e c t p r o j , c o n s t s t r i n g& t a r g e t ) { target = target ; t r a v e r s e I n p u t F i l e s ( proj , preorder ) ; } v o i d v i s i t ( SgNode node ) { c o n s t SgVarRefExp r e f = isSgVarRefExp ( node ) ; if ( ref ) { c o n s t S g V a r i a b l e S y m b o l sym = r e f >g e t s y m b o l ( ) ; ROSE ASSERT ( sym ) ; c o u t << V i s i t i n g SgVarRef << sym >get name ( ) . s t r ( ) << << e n d l ; i f ( sym >get name ( ) . s t r ( ) == t a r g e t ) // E a r l y e x i t a t f i r s t match . throw S t o p E a r l y ( r e f ) ; } } private : string };

t a r g e t ; // Symbol r e f e r e n c e name t o f i n d .

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j = frontend ( argc , argv ) ; VarRefFinderTraversal f i n d e r ; // Look f o r a r e f e r e n c e t o try { f i n d e r . f i n d ( proj , s t o p c o u t << R e f e r e n c e t o a } c a t c h ( S t o p E a r l y& s t o p ) { c o u t << R e f e r e n c e t o a stop . p r i n t ( cout ) ; } stop ); symbol symbol . n o t f o u n d . << e n d l ; f o u n d . << e n d l ;

stop stop

// Look f o r a r e f e r e n c e t o g o . try { f i n d e r . f i n d ( proj , g o ); c o u t << R e f e r e n c e t o a symbol } c a t c h ( S t o p E a r l y& go ) { c o u t << R e f e r e n c e t o a symbol go . p r i n t ( c o u t ) ; }

go go

n o t f o u n d . << e n d l ; f o u n d . << e n d l ;

7.2. TRAVERSALS OF THE AST STRUCTURE

65

1 2 3 4 5 6 7 8 9

V i s i t i n g SgVarRef g o V i s i t i n g SgVarRef s t o p R e f e r e n c e t o a symbol s t o p f o u n d . 0 x36327d8 : SgVarRefExp At / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e t r a v e r s a l V i s i t i n g SgVarRef g o f o u n d . R e f e r e n c e t o a symbol g o 0 x3632770 : SgVarRefExp At / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e t r a v e r s a l

Figure 7.28: Output code showing the result of short-circuiting the traversal.

66

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.3

Memory Pool Traversals

Allocation of IR nodes in ROSE is made more ecient through the use of specialized allocators implemented at member function new operators for each class of the IR in Sage III. Such specialized memory allocators avoid signicant fragmentation of memory, provide more ecient packing of memory, improve performance of allocation of memory and IR node access, and additionally provide a secondary mechanism to accessing all the IR nodes. Each IR node has a memory pool which is an STL vector of blocks (a xed or variable sized array of contiguously stored IR nodes). The three types of traversals are: 1. ROSE Memory Pool Visit Traversal This traversal is similar to the one provided by the SimpleProcessing Class (using the visit() function and no inherited or synthesized attributes). 2. Classic Object-Oriented Visitor Pattern for Memory Pool This is a classic object-oriented visitor pattern. 3. IR node type traversal, visits one type of IR node for all IR types in the AST. This is useful for building specialized tools.

7.3.1

ROSE Memory Pool Visit Traversal

Figure 7.29 shows the source code for a translator which traverses the memory pool containing the AST. At each node the visit() function is called using only the input information represented by the current node. Note that using this simple traversal no context information is available to the visit function. All the IR nodes for a given memory pool are iterated over at one time. The order of the traversal of the dierent memory pools is random but xed. Thus the order of the traversal of the IR nodes is in no way connected to the structure of the AST (unlike the previous non-memory pool traversals that were very much tied to the structure of the AST and which matched the structure of the original input source code being compiled).

7.3. MEMORY POOL TRAVERSALS

67

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

#i n c l u d e r o s e . h // ROSE V i s i t T r a v e r s a l ( s i m i l a r i n t e r f a c e a s Markus s v i s i t t r a v e r s a l ) // i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t ha s no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s RoseVisitor : public ROSE VisitTraversal { public : int counter ; v o i d v i s i t ( SgNode node ) ; RoseVisitor () }; v o i d R o s e V i s i t o r : : v i s i t ( SgNode node ) { // p r i n t f ( r o s e V i s i t o r : : v i s i t : c o u n t e r %4d node = %s \ n , c o u n t e r , node >c l a s s n a m e ( ) . c s t r ( ) ) ; c o u n t e r ++; } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // ROSE v i s i t t r a v e r s a l RoseVisitor v i s i t o r ; v i s i t o r . traverseMemoryPool ( ) ; p r i n t f ( Number o f IR n o d e s i n AST = %d \ n , v i s i t o r . c o u n t e r ) ; r e t u r n backend ( p r o j e c t ) ; } : c o u n t e r ( 0 ) {}

Figure 7.29: Example source showing simple visit traversal over the memory pools.

Number o f IR n o d e s i n AST = 7705

Figure 7.30: Output of input le to the visitor traversal over the memory pool.

68

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.3.2

Classic Object-Oriented Visitor Pattern for Memory Pool

Figure 7.31 shows the source code for a translator which traverses the memory pools containing the AST. At each node the visit() function is called using only the input information represented by the current node. Note that using this simple traversal no context information is available to the visit function. The traversal order is the same as in the 7.29.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

#i n c l u d e r o s e . h // C l a s s i c V i s i t o r P a t t e r n i n ROSE ( implemented u s i n g t h e t r a v e r s a l o v e r // t h e e l e m e n t s s t o r e d i n t h e memory p o o l s s o i t ha s no c y c l e s and v i s i t s // ALL IR n o d e s ( i n c l u d i n g a l l S g F i l e I n f o , SgSymbols , SgTypes , and t h e // s t a t i c b u i l t i n SgTypes ) . c l a s s C l a s s i c V i s i t o r : p u b l i c ROSE VisitorPattern { public : // O v e r r i d e v i r t u r a l f u n c t i o n d e f i n e d i n b a s e c l a s s void v i s i t ( SgGlobal g l o b a l S c o p e ) { p r i n t f ( Found t h e S g G l o b a l IR node \ n ) ; } void v i s i t ( SgFunctionDeclaration functionDeclaration ) { p r i n t f ( Found a S g F u n c t i o n D e c l a r a t i o n IR node \ n ) ; } v o i d v i s i t ( SgTypeInt i n t T y p e ) { p r i n t f ( Found a SgTypeInt IR node \ n ) ; } v o i d v i s i t ( SgTypeDouble doubleType ) { p r i n t f ( Found a SgTypeDouble IR node \ n ) ; } };

int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // C l a s s i c v i s i t o r p a t t e r n o v e r t h e memory p o o l o f IR n o d e s ClassicVisitor visitor A ; traverseMemoryPoolVisitorPattern ( v i s i t o r A ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 7.31: Example source showing simple visitor pattern.

7.3. MEMORY POOL TRAVERSALS

69

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found Found

a SgTypeInt IR node a SgTypeDouble IR node t h e S g G l o b a l IR node a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration a SgFunctionDeclaration

IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR IR

node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node

70

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

7.3.3

ROSE IR Type Traversal (uses Memory Pools)

Figure 7.33 shows the source code for a translator which traverses only one type of IR node using the memory pool containing the AST. This traversal is useful for building specialized tools (often tools which only call static functions on each type of IR node). This example shows the use of an alternative traversal which traverses a representative of each type or IR node just one, but only if it exists in the AST (memory pools). This sort of traversal is useful for building tools that need only operate on static member functions of the IR nodes or need only sample one of each type or IR node present in the AST. this specic example also appears in: ROSE/src/midend/astDiagnostics/AstStatistics.C. The users use of the traversal is the same as for other ROSE AST traversals except that the ROSE VisitTraversal::traverseRepresentativeIRnodes() member function is called instead of ROSE VisitTraversal::traverseMemoryPool(). This mechanism can be used to generate more complete reports of the memory consumption of the AST, which is reported on if -rose:verbose 2 is used. Figure 7.35 shows a partial snapshot of current IR node frequency and memory consumption for a moderate 40,000 line source code le (one le calling a number of header les), sorted by memory consumption. The AST contains approximately 280K IR nodes. Note that the Sg File Info IR nodes is most frequent and consumes the greatest amount of memory, this reects our bias toward preserving signicant information about the mapping of language constructs back to the positions in the source le to support a rich set of source-to-source functionality. Note: more complete information about the memory use of the AST in in the ROSE User Manual appendix.

7.3. MEMORY POOL TRAVERSALS

71

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

// T h i s example c o d e shows t h e t r a v e r s a l #i n c l u d e r o s e . h u s i n g namespace s t d ;

o f IR t y p e s n o t a v a i l a b l e u s i n g t h e o t h e r t r a v e r s a l mechanism .

// CPP Macro t o implement c a s e f o r e a c h IR node ( we c o u l d a l t e r n a t i v e l y u s e a v i s i t o r p a t t e r n and a f u n c t i o n t e m p l a t e , ma #d e f i n e IR NODE VISIT CASE (X) \ c a s e V ## X: \ { \ X c a s t N o d e = i s## X( node ) ; \ i n t numberOfNodes = castNode >numberOfNodes ( ) ; \ i n t memoryFootprint = castNode >memoryUsage ( ) ; \ p r i n t f ( c o u n t = %7d , memory u s e = %7d b y t e s , node name = %s \ n , numberOfNodes , memoryFootprint , castNode >c break ; \ } c l a s s RoseIRnodeVisitor : public ROSE VisitTraversal { public : int counter ; v o i d v i s i t ( SgNode node ) ; R o s e I R n o d e V i s i t o r ( ) : c o u n t e r ( 0 ) {} }; void { // // // RoseIRnodeVisitor : : v i s i t ( SgNode node )

Using a c l a s s i c v i s i t o r p a t t e r n s h o u l d a v o i d a l l t h i s c a s t i n g , but e a c h f u n c t i o n must be c r e a t e d s e p a r a t e l y ( s o i t i s wash i f we want t o do a l l IR nodes , a s we do h e r e ) . s w i t c h ( node >v a r i a n t T ( ) ) { IR NODE VISIT CASE ( S g F i l e I n f o ) IR NODE VISIT CASE ( S g P a r t i a l F u n c t i o n T y p e ) IR NODE VISIT CASE ( SgFunctionType ) IR NODE VISIT CASE ( S g P o i n t e r T y p e ) IR NODE VISIT CASE ( S g F u n c t i o n D e c l a r a t i o n ) IR NODE VISIT CASE ( SgFunctionSymbol ) IR NODE VISIT CASE ( SgSymbolTable ) IR NODE VISIT CASE ( S g I n i t i a l i z e d N a m e ) IR NODE VISIT CASE ( S g S t o r a g e M o d i f i e r ) IR NODE VISIT CASE ( Sg Fo rS ta t eme nt ) IR NODE VISIT CASE ( S g F o r I n i t S t a t e m e n t ) IR NODE VISIT CASE ( S g C t o r I n i t i a l i z e r L i s t ) IR NODE VISIT CASE ( S g I f S t m t ) IR NODE VISIT CASE ( SgExprStatement ) IR NODE VISIT CASE ( S g T e m p l a t e D e c l a r a t i o n ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n D e c l ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n D e f n ) IR NODE VISIT CASE ( S g T e m p l a t e I n s t a n t i a t i o n M e m b e r F u n c t i o n D e c l ) IR NODE VISIT CASE ( SgClassSymbol ) IR NODE VISIT CASE ( SgTemplateSymbol ) IR NODE VISIT CASE ( SgMemberFunctionSymbol ) default : {

#i f 0 p r i n t f ( Case n o t h a n d l e d : %s \ n , node >c l a s s n a m e ( ) . c s t r ( ) ) ; #e n d i f } } }

int main ( i n t a r g c , c h a r a r g v [ ] ) { // ROSE v i s i t t r a v e r s a l SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // ROSE v i s i t t r a v e r s a l RoseIRnodeVisitor v i s i t o r ; v i s i t o r . traverseRepresentativeIRnodes ( ) ; p r i n t f ( Number o f t y p e s o f IR n o d e s ( a f t e r

b u i l d i n g AST) = %d \ n , v i s i t o r . c o u n t e r ) ;

#i f 1 // IR n o d e s s t a t i s t i c s i f ( project >g e t v e r b o s e ( ) > 1 ) c o u t << A s t N o d e S t a t i s t i c s : : I R n o d e U s a g e S t a t i s t i c s ( ) ; #e n d i f i n t errorCode = 0 ; e r r o r C o d e = backend ( p r o j e c t ) ; return errorCode ;

72

CHAPTER 7. INTRODUCTION TO AST TRAVERSALS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

count = 2 4 , memory u s e = 3840 b y t e s count = 7 5 0 , memory u s e = 30000 b y t e s count = 3 4 2 4 , memory u s e = 246528 b y t e s No r e p r e s e n t a t i v e f o r S g P a r t i a l F u n c t i o n T y p e count = 3 9 9 , memory u s e = 47880 b y t e s count = 1 2 , memory u s e = 1152 b y t e s count = 2 , memory u s e = 576 b y t e s count = 2 , memory u s e = 208 b y t e s count = 4 , memory u s e = 2272 b y t e s count = 1 , memory u s e = 296 b y t e s count = 9 , memory u s e = 792 b y t e s count = 6 , memory u s e = 3984 b y t e s count = 5 , memory u s e = 3840 b y t e s count = 1 , memory u s e = 296 b y t e s count = 3 , memory u s e = 3048 b y t e s count = 4 2 4 , memory u s e = 390080 b y t e s count = 1 , memory u s e = 48 b y t e s count = 3 , memory u s e = 144 b y t e s count = 2 , memory u s e = 96 b y t e s count = 3 9 9 , memory u s e = 19152 b y t e s count = 7 5 0 , memory u s e = 222000 b y t e s Number o f t y p e s o f IR n o d e s ( a f t e r b u i l d i n g

, node name = SgSymbolTable , node name = S g S t o r a g e M o d i f i e r , node name = S g F i l e I n f o f o u n d i n memory p o o l s , node name = SgFunctionType , node name = S g P o i n t e r T y p e , node name = Sg Fo rS ta t ement , node name = S g F o r I n i t S t a t e m e n t , node name = S g C t o r I n i t i a l i z e r L i s t , node name = S g I f S t m t , node name = SgExprStatement , node name = S g T e m p l a t e D e c l a r a t i o n , node name = S g T e m p l a t e I n s t a n t i a t i o n D e c l , node name = S g T e m p l a t e I n s t a n t i a t i o n D e f n , node name = S g T e m p l a t e I n s t a n t i a t i o n M e m b e r F u n c t i o n D e c l , node name = S g F u n c t i o n D e c l a r a t i o n , node name = SgClassSymbol , node name = SgTemplateSymbol , node name = SgMemberFunctionSymbol , node name = SgFunctionSymbol , node name = S g I n i t i a l i z e d N a m e AST) = 0

Figure 7.34: Output of input le to the IR Type traversal over the memory pool.
AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST AST Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Memory Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Pool Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: Statistics: numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes numberOfNodes = 114081 memory consumption = 5019564 node = Sg_File_Info = 31403 memory consumption = 628060 node = SgTypedefSeq = 14254 memory consumption = 285080 node = SgStorageModifier = 14254 memory consumption = 1140320 node = SgInitializedName = 8458 memory consumption = 169160 node = SgFunctionParameterTypeList = 7868 memory consumption = 1101520 node = SgModifierType = 7657 memory consumption = 398164 node = SgClassType = 7507 memory consumption = 2071932 node = SgClassDeclaration = 7060 memory consumption = 282400 node = SgTemplateArgument = 6024 memory consumption = 385536 node = SgPartialFunctionType = 5985 memory consumption = 1388520 node = SgFunctionParameterList = 4505 memory consumption = 1477640 node = SgTemplateInstantiationDecl = 3697 memory consumption = 162668 node = SgReferenceType = 3270 memory consumption = 758640 node = SgCtorInitializerList = 3178 memory consumption = 76272 node = SgMemberFunctionSymbol = 2713 memory consumption = 119372 node = SgPointerType = 2688 memory consumption = 161280 node = SgThrowOp = 2503 memory consumption = 60072 node = SgFunctionSymbol = 2434 memory consumption = 107096 node = SgFunctionTypeSymbol = 2418 memory consumption = 831792 node = SgFunctionDeclaration = 2304 memory consumption = 55296 node = SgVariableSymbol = 2298 memory consumption = 101112 node = SgVarRefExp = 2195 memory consumption = 114140 node = SgSymbolTable = 2072 memory consumption = 721056 node = SgMemberFunctionDeclaration = 1668 memory consumption = 400320 node = SgVariableDeclaration = 1667 memory consumption = 393412 node = SgVariableDefinition = 1579 memory consumption = 101056 node = SgMemberFunctionType = 1301 memory consumption = 31224 node = SgTemplateSymbol = 1300 memory consumption = 364000 node = SgTemplateDeclaration = 1198 memory consumption = 455240 node = SgTemplateInstantiationMemberFunctionDecl = 1129 memory consumption = 54192 node = SgIntVal = 1092 memory consumption = 56784 node = SgAssignInitializer = 1006 memory consumption = 52312 node = SgExpressionRoot

Truncated results presented ...

Figure 7.35: Example of output using -rose:verbose 2 (memory use report for AST).

Chapter 8

Graph Processing Tutorial

8.1 Traversal Tutorial

ROSE can collect and analyze paths in both source and binary CFGs. At moment it doesnt attempt to save paths because if you save them directly the space necessary is extremely large, as paths grow 2n with successive if statements and even faster when for loops are involved. Currently a path can only cannot complete the same loop twice. However it is possible for a graph such that 1 - 2 , 2-3, 3-1, 3-5, has paths, 1,2,3,1,2,3,5 and 1,2,3,5 because the loop 1,2,3,1 is not repeated. The tutorial example works as such:

73

74

CHAPTER 8. GRAPH PROCESSING TUTORIAL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

#include < i o s t r e a m > #include < f s t r e a m > //#i n c l u d e < r o s e . h> #include < s t r i n g > #include < e r r . h> #include SgGraphTemplate . h #include g r a p h P r o c e s s i n g . h

#include staticCFG . h #include i n t e r p r o c e d u r a l C F G . h / T e s t i n g t h e graph t r a v e r s a l mechanism now i mple menting i n A s t P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g / #include < s y s / t i m e . h> #include < s y s / r e s o u r c e . h> using namespace s t d ; using namespace b o o s t ;

/ You need t o use myGraph t y p e h e r e b e c a u s e t h e c o n v e r s i o n o f StaticCFG : : InterproceduralCFG or StaticCFG : :CFG i n a b o o s t form . The SgGraphTemplate . h f i l e h a n d l e s t h i s c o n v e r s i o n and myGraph i s s p e c i f i c t o t h a t f i l e / typedef myGraph CFGforT ;

/ Your b a s i c v i s i t o r t r a v e r s a l s u b c l a s s e d from SgGraphTraversal on t h e CFGforT t e m p l a t e as d e f i n e d above / c l a s s v i s i t o r T r a v e r s a l : public S g G r a p h T r a v e r s a l <CFGforT> { public : int paths ; / This i s t h e f u n c t i o n run by t h e a l g o r i t h m on e v e r y path , VertexID i s a t y p e implemented i n SgGraphTe void a n a l y z e P a t h ( v e c t o r <VertexID >& pth ) ; }; / d e f i n i n g t h e a n a l y z e P a t h f u n c t i o n . This s i m p l y c o u n t s p a t h s as s h o u l d be o b v i o u s . Again , VertexID i s d e f i n e void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r <VertexID >& pth ) { p a t h s ++; } i n t main ( i n t a r g c , char a r g v [ ] ) { / F i r s t you need t o produce t h e p r o j e c t f i l e / SgProject p r o j = frontend ( argc , argv ) ; ROSE ASSERT ( p r o j != NULL ) ; / G e t t i n g t h e Function D e c l a r a t i o n and D e f i n i t i o n f o r p r o d u c i n g t h e graph / S g F u n c t i o n D e c l a r a t i o n mainDefDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j ) ; S g F u n c t i o n D e f i n i t i o n mainDef = mainDefDecl >g e t d e f i n i t i o n ( ) ; / I n s t a n t i a t i n g t h e v i s i t o r T r a v e r s a l / v i s i t o r T r a v e r s a l v i s = new v i s i t o r T r a v e r s a l ( ) ; / This c r e a t e s t h e StaticCFG : : InterproceduralCFG o b j e c t t o be c o n v e r t e d t o a b o o s t graph / StaticCFG : : I n t e r p r o c e d u r a l C F G c f g ( mainDef ) ; stringstream ss ; S g I n c i d e n c e D i r e c t e d G r a p h g = new S g I n c i d e n c e D i r e c t e d G r a p h ( ) ; / We g o t t h e n e c e s s a r y i n t e r n a l S g I n c i d e n c e D i r e c t e d G r a p h from t h e c f g / g = c f g . getGraph ( ) ; myGraph mg = new myGraph ( ) ; / C o n v e r t i n g t h e c f g t o a b o o s t graph / mg = i n s t a n t i a t e G r a p h ( g , c f g , mainDef ) ; / S e t i n t e r n a l v a r i a b l e s / vis >p a t h s = 0 ; / i n v o k i n g t h e t r a v e r s a l , t h e f i r s t argument i s t h e graph , t h e second i s t r u e i f you do not want bounds , f a l s e i f you do , t h e t h i r d and f o u r t h arguments a r e s t a r t i n g and s t o p p i n g v e r t i c e s r e s p e c t i v e l y , i f you a r e not bounding s i m p l y i n s e r t 0 . F i n a l l y t h e l a s t argument i s c u r r e n t l y d e p r e c a t e d / vis >c o n s t r u c t P a t h A n a l y z e r (mg , true , 0 , 0 , true ) ; s t d : : c o u t << f i n i s h e d << s t d : : e n d l ; s t d : : c o u t << p a t h s : << v i s >p a t h s << s t d : : e n d l ; delete v i s ; }

Figure 8.1: Source CFG Traversal Example

8.1. TRAVERSAL TUTORIAL

75

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

#include < i o s t r e a m > #include < f s t r e a m > #include < r o s e . h> //#i n c l u d e interproceduralCFG . h #include < s t r i n g > #include < e r r . h> / These a r e n e c e s s a r y f o r any b i n a r y T r a v e r s a l / #include g r a p h P r o c e s s i n g . h #include B i n a r y C o n t r o l F l o w . h #include B i n a r y L o a d e r . h / T e s t i n g t h e graph t r a v e r s a l mechanism now i mple menting i n g r a p h P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g /) / using namespace s t d ; using namespace b o o s t ; / These s h o u l d j u s t be c o p i e d v e r b a t i m / typedef b o o s t : : g r a p h t r a i t s <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph > : : v e r t e x d e s c r i p t o r V e r t e x ; / < Graph v e r t e x t y p e . / typedef b o o s t : : g r a p h t r a i t s <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph > : : e d g e d e s c r i p t o r Edge ; / < Graph ed g e t y p e . /

/ We f i r s t make a v i s i t o r T r a v e r s a l , s u b c l a s s e d from SgGraphTraversal t e m p l a t e d on t h e B i n a r y A n a l y s i s : ControlFlow : : Graph which i s implemented as a b o o s t graph /

class v i s i t o r T r a v er sa l {

: public S g G r a p h T r a v e r s a l <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph>

public : long i n t p t h s ; long i n t t l t n o d e s ; / This needs t o be i n any v i s i t o r T r a v e r s a l , i t i s t h e f u n c t i o n t h a t w i l l be run on e v e r y p a t h by t h e graph p a t h a n a l y s i s a l g o r i t h m , n o t i c e t h e V e r t e x t y p e i s from t h e above t y p e d e f s / v i r t u a l void a n a l y z e P a t h ( v e c t o r <Vertex >& pth ) ; }; / This i s a v e r y s i m p l e i n c a r n a t i o n , i t j u s t c o u n t s p a t h s / void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r <Vertex >& pth ) { p t h s ++; } i n t main ( i n t a r g c , char a r g v [ ] ) { / Parse t h e b i n a r y f i l e / SgProject p r o j e c t = fr ontend ( argc , argv ) ; s t d : : v e c t o r <S g A s m I n t e r p r e t a t i o n > i n t e r p s = S a g e I n t e r f a c e : : querySubTree <S g A s m I n t e r p r e t a t i o n >( p r o j e c t ) ; i f ( i n t e r p s . empty ( ) ) { f p r i n t f ( s t d e r r , no b i n a r y i n t e r p r e t a t i o n s f o u n d \ n ) ; exit (1); } / C a l c u l a t e p l a i n o l d CFG. / BinaryAnalysis : : ControlFlow c f g a n a l y z e r ; B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph c f g = new B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph ; c f g a n a l y z e r . b u i l d c f g f r o m a s t ( i n t e r p s . back ( ) , c f g ) ; s t d : : o f s t r e a m mf ; mf . open ( a n a l y s i s . d o t ) ; / D e c l a r i n g t h e v i s i t o r T r a v e r s a l / v i s i t o r T r a v e r s a l v i s = new v i s i t o r T r a v e r s a l ; / S e t t i n g i n t e r n a l v a r i a b l e s / vis >t l t n o d e s = 0 ; vis >p t h s = 0 ;

/ v i s i t o r T r a v e r s a l has 5 arguments , t h e f i r s t i s t h e ambient CFG, t h e second i d e n t i f i e s w h e t h e r or not you ar e bounding t h e graph , t h a t i s , w h e t h e r you want a l l your p a t h s t o s t a r t a t one s p e c i f i c node and end a t a n o t h e r s p e c i f i c node , t h e f o u r t h and f i f t h would be s t a r t and end i f t h e graph were bounded . S i n c e t h e y aren t you can s i m p l y i n p u t 0 , f o r t h e moment t h e f i n a l argument i s d e p r e c a t e d , t h o u g h i t s p u r p o s e was t o t e l l t h e progra t h a t your a n a l y s i s f u n c t i o n was t h r e a d s a f e , t h a t i s t h a t openMP c o u l d run i t w i t h o u t h a v i n g a c r i t i c a l command . C u r r e n t l y a c r i t i c a l i s a l w a y s used / vis >c o n s t r u c t P a t h A n a l y z e r ( c f g , true , 0 , 0 , f a l s e ) ; s t d : : c o u t << p t h s : << v i s >p t h s << s t d : : e n d l ; s t d : : c o u t << t l t n o d e s : << v i s > t l t n o d e s << s t d : : e n d l ;

76

CHAPTER 8. GRAPH PROCESSING TUTORIAL

Chapter 9

Scopes of Declarations
The scope of an IR node may be either stored explicitly in the IR node or obtained through computation through its parent information in the AST. Figure X shows an example where the variable denition for a variable is the scope of namespace X. The declaration for variable a is in the namespace X. In a more common way, the function foo is a member function of B with a declaration appearing in class B, but with a function denition in global scope.
namespace X{ extern int a; } int X::a = 0; class B { void foo(); }; void B::foo() {}

In C++, using name qualication the scope of a declaration can be independent of it structural location in the AST. The get parent() member function (available on most IR nodes) communicates the structural information of the original source code (also represented in the AST). The scope information must at times be stored explicitly when it can not be interpreted structurally. The example in this chapter show how to nd the scope of each C++ construct . Note that SgExpression IR nodes can take their scope from that of the statement where they are found. SgStatement and SgInitializedName IR nodes are the interesting IR nodes from the point of scope. The SgInitializedName and all SgStatement IR nodes have a member function get scope() which returns the scope of the associated IR nodes. The example code in this chapter traverses the AST and reports the scope of any SgInitializedName and all SgStatement IR nodes. It is intended to provide a simple intuition about what the scope can be expected to be in an application. The example code is also useful as a simple means of exploring the scopes of any other input application.

9.1

Input For Examples Showing Scope Information

Figure 9.1 shows the input example code form this tutorial example. 77

78

CHAPTER 9. SCOPES OF DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12

i n t xyz ; void foo ( i n t { int y ; for ( int { int z = } } x)

i =0; i < 1 0 ; z; 42;

i ++)

Figure 9.1: Example source code used as input to program in codes used in this chapter.

9.2

Generating the code representing any IR node

The following code traverses each IR node and for a SgInitializedName of SgStatement outputs the scope information. The input code is shown in gure 9.1; the output of this code is shown in gure 9.3.

9.2. GENERATING THE CODE REPRESENTING ANY IR NODE

79

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

// T h i s example shows t h e s c o p e o f e a c h s t a t e m e n t and name ( v a r i a b l e names , b a s e c l a s s names , #i n c l u d e r o s e . h class { }; visitorTraversal public : v i r t u a l void : public AstSimpleProcessing

etc . ) .

v i s i t ( SgNode n ) ;

v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { // There a r e t h r e e t y p e s i r IR n o d e s t h a t can be q u e r i e d f o r s c o p e : // SgStatement , and // SgInitializedName SgStat ement s t a t e m e n t = i s S g S t a t e m e n t ( n ) ; i f ( s t a t e m e n t != NULL) { SgScopeStatement scope = statement >g e t s c o p e ( ) ; ROSE ASSERT( s c o p e != NULL ) ; p r i n t f ( S gS ta t ement = %12p = %30 s h as s c o p e = %12p = %s ( t o t a l number = %d ) \ n , statement , statement >c l a s s n a m e ( ) . c s t r ( ) , scope , scope >c l a s s n a m e ( ) . c s t r ( ) , ( i n t ) s c o p e >numberOfNodes ( ) ) ; } SgInitializedName initializedName = isSgInitializedName (n ) ; i f ( i n i t i a l i z e d N a m e != NULL) { SgScopeStatement scope = i n i t i a l i z e d N a m e >g e t s c o p e ( ) ; ROSE ASSERT( s c o p e != NULL ) ; p r i n t f ( S g I n i t i a l i z e d N a m e = %12p = %30 s h as s c o p e = %12p = %s ( t o t a l number = %d ) \ n , initializedName , initializedName >get name ( ) . s t r ( ) , scope , scope >c l a s s n a m e ( ) . c s t r ( ) , ( i n t ) s c o p e >numberOfNodes ( ) ) ; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; p r i n t f ( Number o f s c o p e s ( S g S c o p e S t a t e m e n t ) = %d \ n , ( i n t ) S g S c o p e S t a t e m e n t : : numberOfNodes ( ) ) ; p r i n t f ( Number o f s c o p e s ( S g B a s i c B l o c k ) = %d \ n , ( i n t ) S g B a s i c B l o c k : : numberOfNodes ( ) ) ; #i f 0 p r i n t f ( \ n\n ) ; p r i n t f ( Now o u t p u t a l l t h e s y m b o l s i n e a c h symbol t a b l e \ n ) ; S a g e I n t e r f a c e : : outputLocalSymbolTables ( p r o j e c t ) ; p r i n t f ( \ n\n ) ; #e n d i f return 0; }

Figure 9.2: Example source code showing how to get scope information for each IR node.

80

CHAPTER 9. SCOPES OF DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

SgStatement = 0 x2b1280ce1010 = S g G l o b a l ha s s c o p e SgStatement = 0 x2b1280f93010 = S g V a r i a b l e D e c l a r a t i o n ha s s c o p e SgInitializedName = 0 x2b1280f2e438 = xyz has s c o p e = 0 x 2 b 1 2 8 0 c e 1 0 1 0 = S g G l o b a l ( t o t a l number = 0 ) SgStatement = 0 x2b1280deb908 = S g F u n c t i o n D e c l a r a t i o n ha s s c o p e SgStatement = 0 x2b1280ea8628 = S g F u n c t i o n P a r a m e t e r L i s t ha s s c o p e SgInitializedName = 0 x2b1280f2e560 = x has s c o p e = 0 x2b12810b9010 = S g F u n c t i o n D e f i n i t i o n ( t o t a l number = 0 ) SgStatement = 0 x2b12810b9010 = S g F u n c t i o n D e f i n i t i o n ha s s c o p e SgStatement = 0 x 2b 12 81 1 02 0 10 = S g B a s i c B l o c k ha s s c o p e SgStatement = 0 x2b1280f93290 = S g V a r i a b l e D e c l a r a t i o n ha s s c o p e SgInitializedName = 0 x2b1280f2e688 = y has s c o p e = 0 x2 b1 28 11 0 20 10 = S g B a s i c B l o c k ( t o t a l number = 0 ) SgStatemen t = 0 x2 b1 28 11 4 50 1 0 = Sg Fo r Sta t ement ha s s c o p e SgStatemen t = 0 x1a8b0190 = S g F o r I n i t S t a t e m e n t h as s c o p e = SgStatemen t = 0 x2b1280f93510 = S g V a r i a b l e D e c l a r a t i o n ha s s c o p e SgInitializedName = 0 x2b1280f2e7b0 = i has s c o p e = 0 x 2b 12 81 1 45 01 0 = Sg Fo rS ta t ement ( t o t a l number = 0 ) SgStatemen t = 0 x1 a 9 3 1 6 6 0 = SgExprStatement has s c o p e = SgStatemen t = 0 x2 b1 28 11 0 21 2 0 = S g B a s i c B l o c k ha s s c o p e SgStatemen t = 0 x2b1280f93790 = S g V a r i a b l e D e c l a r a t i o n ha s s c o p e SgInitializedName = 0 x2b1280f2e8d8 = z has s c o p e = 0 x2 b12 8 11 02 1 20 = S g B a s i c B l o c k ( t o t a l number = 0 ) SgStatemen t = 0 x1a9316b8 = SgExprStatement has s c o p e = Number o f s c o p e s ( S g S c o p e S t a t e m e n t ) = 0 Number o f s c o p e s ( S g B a s i c B l o c k ) = 2

= 0 x2b1280ce1010 = SgGlobal ( t o t = 0 x2b1280ce1010 = SgGlobal ( t o t

= 0 x2b1280ce1010 = SgGlobal ( t o t = 0 x2b1280ce1010 = SgGlobal ( t o t

= 0 x2b1280ce1010 = SgGlobal ( t o t = 0 x2b12810b9010 = S g F u n c t i o n D e = 0 x 2b1 2 81 102010 = S g B a s i c B l o c k

= 0 x2 b12 8 11 02010 = S g B a s i c B l o c k 0 x2 b1 28 11 4 5010 = SgForStatement = 0 x2 b12 8 1145010 = SgForStatemen

0 x2 b1 28 1 14 5010 = SgForStatement = 0 x2 b12 8 1145010 = SgForStatemen = 0 x2 b12 8 1102120 = S g B a s i c B l o c k

0 x2 b1 28 1 10 2120 = S g B a s i c B l o c k ( t

Figure 9.3: Output of input code using scopeInformation.C

Chapter 10

AST Query
This chapter presents a mechanism for simple queries on the AST. Such queries are typically a single line of code, instead of the class that must be declared and dened when using the traversal mechanism. While the traversal mechanism is more sophisticated and more powerful, the AST Query mechanism is particularly simple to use.

10.1

Simple Queries on the AST

This section demonstrates a simple query on the AST. The program in gure 10.1 calls an internal ROSE Query Library. Queries of the AST using the query library are particularly simple and often are useful as nested queries within more complex analysis. More information of the ROSE AST Query Library is available within ROSE User Manual. Using the input program in gure 10.2 the translator processes the code and generates the output in gure 10.3.

FIXME: Put an example of composition of AST queries into the example input code.

10.2

Nested Query

This section demonstrates a nested AST query, showing how to use composition in the construction of more elaborate queries from simple ones. The number of traversals of the AST can be reduced by using nested queries. Nested queries permits queries on the result from a NodeQuery. Another advantage is that nested (combined) queries can be formed to query for information without writing new query, the nested query is a new query. The program in gure 10.4 calls an internal ROSE Query Library. Two dierent queries are performed to nd all access functions within the AST. The rst query is nested, the returned list from a query is used in a traversal, and the second query queries the AST for the same nodes. Using the input program in gure 10.5 the translator processes the code and generates the output in gure 10.6.

81

82

CHAPTER 10. AST QUERY

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ;

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r

int counter = 0; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = i sS gFu ncti o nDecl a ra ti o n ( i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // DQ ( 3 / 5 / 2 0 0 6 ) : Only o u t p u t t h e non c o m p i l e r g e n e r a t e d IR n o d e s i f ( ( i )> g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) { // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( F u n c t i o n #%2d name i s %s a t l i n e %d \ n , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n >get name ( ) . s t r ( ) , functionDeclaration > g e t f i l e i n f o ()> g e t l i n e ( ) ) ; } else { // Output s o m e t h i n g about t h e c o m p i l e r g e n e r a t e d b u i l t i n f u n c t i o n s p r i n t f ( Compiler g e n e r a t e d ( b u i l t i n ) f u n c t i o n #%2d name i s %s \ n , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n >get name ( ) . s t r ( ) ) ; } } // Note : Show c o m p o s i t i o n o f AST q u e r i e s return 0; }

Figure 10.1: Example source code for translator to read an input program and generate a list of functions in the AST (queryLibraryExample.C).

10.2. NESTED QUERY

83

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 10.2: Example source code used as input to program in gure 10.1 (queryLibraryExample.C).

84
Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d Compiler g e n e r a t e d

CHAPTER 10. AST QUERY

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (

builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin builtin

) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) )

function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function

# 0 # 1 # 2 # 3 # 4 # 5 # 6 # 7 # 8 # 9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82

name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name

is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is

builtin copysign builtin copysignf builtin copysignl builtin acosf builtin acosl builtin asinf builtin asinl builtin atanf builtin atanl builtin atan2f builtin atan2l builtin ceilf builtin ceill builtin coshf builtin coshl builtin floorf builtin floorl builtin fmodf builtin fmodl builtin frexpf builtin frexpl builtin ldexpf builtin ldexpl builtin log10f builtin log10l builtin modff builtin modfl builtin powf builtin powl builtin sinhf builtin sinhl builtin tanf builtin tanl builtin tanhf builtin tanhl builtin powil builtin powi builtin powif builtin strchr builtin strrchr builtin strpbrk builtin strstr builtin nansf builtin nans builtin nansl builtin fabs builtin fabsf builtin fabsl builtin cosf builtin cosl builtin sinf builtin sinl builtin sqrtf builtin sqrtl builtin fpclassify builtin return address builtin frame address builtin expect builtin prefetch builtin huge val builtin huge valf builtin huge vall builtin inf builtin inff builtin infl builtin nan builtin nanf builtin nanl builtin nans builtin nansf builtin nansl builtin clz builtin ctz builtin popcount builtin parity builtin ffsl builtin clzl builtin ctzl builtin popcountl builtin parityl builtin ffsll builtin clzll builtin ctzll

10.2. NESTED QUERY

85

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ;

// F u n c t i o n q u e r y S o l v e r A c c e s s F u n c t i o n s ( ) // f i n d a c c e s s f u n c t i o n s ( f u n c t i o n name s t a r t s w i t h g e t o r s e t ) NodeQuerySynthesizedAttributeType q u e r y S o l v e r A c c e s s F u n c t i o n s ( SgNode astNode ) { ROSE ASSERT ( astNode != 0 ) ; NodeQuerySynthesizedAttributeType returnNodeList ; S g F u n c t i o n D e c l a r a t i o n f u n c D e c l = i s S g F u n c t i o n D e c l a r a t i o n ( astNode ) ; if ( f u n c D e c l != NULL) { s t r i n g functionName = f u n c D e c l >get name ( ) . s t r ( ) ; i f ( ( functionName . l e n g t h ( ) >= 4 ) && ( ( functionName . s u b s t r ( 0 , 4 ) == g e t ) r e t u r n N o d e L i s t . p u s h b a c k ( astNode ) ; }

||

( functionName . s u b s t r ( 0 , 4 ) == s e t

return returnNodeList ; } // F u n c t i o n p r i n t F u n c t i o n D e c l a r a t i o n L i s t w i l l p r i n t a l l f u n c t i o n names i n t h e l i s t v o i d p r i n t F u n c t i o n D e c l a r a t i o n L i s t ( R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t ) { int counter = 0; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L i s t . end ( ) ; { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = i sS gFu ncti o nDecl a ra ti o n ( i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( f u n c t i o n name #%d i s %s a t l i n e %d \ n , c o u n t e r ++, f u n c t i o n D e c l a r a t i o n >get name ( ) . s t r ( ) , functionDeclaration > g e t f i l e i n f o ()> g e t l i n e ( ) ) ; } } i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST and f i n d // ( f u n c t i o n name s t a r t s w i t h g e t o r s e t ) all access functions

// B u i l d l i s t u s i n g a q u e r y o f t h e whole AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n ) ; // B u i l d l i s t u s i n g n e s t e d Q u e r i e s ( o p e r a t i n g on r e t u r n r e s u l t o f p r e v i o u s q u e r y ) R o s e S T L C o n t a i n e r <SgNode> a c c e s s F u n c t i o n s L i s t ; a c c e s s F u n c t i o n s L i s t = NodeQuery : : q u e r y N o d e L i s t ( f u n c t i o n D e c l a r a t i o n L i s t ,& q u e r y S o l v e r A c c e s s F u n c t i o n s ) ; printFunctionDeclarationList ( accessFunctionsList ); // A l t e r n a t i v e form o f same q u e r y b u i l d i n g t h e l i s t u s i n g a q u e r y o f t h e whole AST a c c e s s F u n c t i o n s L i s t = NodeQuery : : querySubTree ( p r o j e c t ,& q u e r y S o l v e r A c c e s s F u n c t i o n s ) ; printFunctionDeclarationList ( accessFunctionsList ); // Another way t o q u e r y f o r c o l l e c t i o n s o f IR n o d e s V a r i a n t V e c t o r vv1 = V S g C l a s s D e f i n i t i o n ; s t d : : c o u t << Number o f c l a s s d e f i n i t i o n s i n t h e memory p o o l

i s : << NodeQuery : : queryMemoryPool ( vv1 ) . s i z e ( ) << s t d :

// Another way t o q u e r y f o r c o l l e c t i o n s o f m u l t i p l e IR n o d e s . // V a r i a n t V e c t o r ( V SgType ) i s i n t e r n a l l y expanded t o a l l IR n o d e s d e r i v e d from SgType . V a r i a n t V e c t o r vv2 = V a r i a n t V e c t o r ( V S g C l a s s D e f i n i t i o n ) + V a r i a n t V e c t o r ( V SgType ) ; s t d : : c o u t << Number o f c l a s s d e f i n i t i o n s AND t y p e s i n t h e memory p o o l i s : << NodeQuery : : queryMemoryPool ( vv2 ) . s i z e // Note : Show c o m p o s i t i o n o f AST q u e r i e s return 0; }

Figure 10.4: Example source code for translator to read an input program and generate a list of access functions in the AST (nestedQueryExample.C).

86

CHAPTER 10. AST QUERY

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 10.5: Example source code used as input to program in gure 10.4 (nestedQueryExample.C).

1 2 3 4 5 6 7 8 9 10

f u n c t i o n name #0 i s g e t f o o f u n c t i o n name #1 i s s e t f o o f u n c t i o n name #2 i s g e t f o o f u n c t i o n name #3 i s s e t f o o f u n c t i o n name #0 i s g e t f o o f u n c t i o n name #1 i s s e t f o o f u n c t i o n name #2 i s g e t f o o f u n c t i o n name #3 i s s e t f o o Number o f c l a s s d e f i n i t i o n s Number o f c l a s s d e f i n i t i o n s

at l i n e 0 at l i n e 0 a t l i n e 28 a t l i n e 29 at l i n e 0 at l i n e 0 a t l i n e 28 a t l i n e 29 i n t h e memory p o o l i s : 1 AND t y p e s i n t h e memory p o o l

i s : 436

Figure 10.6: Output of input le to the AST query processor (nestedQueryExample.C).

Chapter 11

AST File I/O

Figure 11.1 shows an example of how to use the AST File I/O mechanism. This chapter presents an example translator to write out an AST to a le and then read it back in.

11.1

Source Code for File I/O

Figure 11.1 shows an example translator which reads an input application, forms the AST, writes out the AST to a le, then deletes the AST and reads the AST from the previously written le. The input code is shown in gure 11.2, the output of this code is shown in gure 11.3.

11.2

Input to Demonstrate File I/O

Figure 11.2 shows the example input used for demonstration of the AST le I/O. In this case we are reusing the example used in the inlining example.

11.3

Output from File I/O

Figure 11.3 shows the output from the example le I/O tutorial example.

11.4

Final Code After Passing Through File I/O

Figure 11.4 shows the same le as the input demonstrating that the le I/O didnt change the resulting generated code. Much more sophisticated tests are applied internally to verify the correctness of the AST after AST le I/O.

87

88

CHAPTER 11. AST FILE I/O

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

// Example d e m o n s t r a t i n g f u n c t i o n #i n c l u d e r o s e . h u s i n g namespace s t d ;

i n l i n i n g ( maximal i n l i n i n g , up t o p r e s e t number o f

inlinings ).

// T h i s i s a f u n c t i o n i n Qing s AST i n t e r f a c e v o i d F i x S g P r o j e c t ( S g P r o j e c t& p r o j ) ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // DQ ( 7 / 2 0 / 2 0 0 4 ) : Added i n t e r n a l c o n s i s t a n c y AstTests : : runAllTests ( p r o j e c t ) ; b o o l modifiedAST = t r u e ; int count = 0; // I n l i n e one c a l l a t a t i m e u n t i l do { modifiedAST = f a l s e ; a l l have been i n l i n e d . Loops on r e c u r s i v e c o d e . t e s t s on AST

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) // Loop o v e r a l l f u n c t i o n c a l l s // f o r ( l i s t <SgNode > : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; i != f u n c t i o n C a l l L i s t . end ( ) ; R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; w h i l e ( modifiedAST == f a l s e && i != f u n c t i o n C a l l L i s t . end ( ) ) { SgFunctionCallExp f u n c t i o n C a l l = isSgFunctionCallExp ( i ) ; ROSE ASSERT( f u n c t i o n C a l l != NULL ) ; // Not a l l f u n c t i o n c a l l s can be i n l i n e d i n C++, s o r e p o r t bool s u c e s s f u l l y I n l i n e d = doInline ( functionCall ) ; if if successful .

i ++)

( s u c e s s f u l l y I n l i n e d == t r u e ) { // As s o o n a s t h e AST i s m o d i f i e d recompute t h e l i s t o f f u n c t i o n // c a l l s ( and r e s t a r t t h e i t e r a t i o n s o v e r t h e m o d i f i e d l i s t ) modifiedAST = t r u e ; } else { modifiedAST = f a l s e ; } list iterator

// I n c r e m e n t t h e i ++; }

// Q u i t e when we have c e a s e d t o do any i n l i n e t r a n s f o r m a t i o n s // and o n l y do a p r e d e f i n e d number o f i n l i n e t r a n s f o r m a t i o n s c o u n t++; } w h i l e ( modifiedAST == t r u e && c o u n t < 1 0 ) ; // C a l l f u n c t i o n t o p o s t p r o c e s s t h e AST and f i x u p symbol t a b l e s FixSgProject ( p r o j e c t ) ; // Rename e a c h v a r i a b l e d e c l a r a t i o n renameVariables ( p r o j e c t ) ; // Fold up b l o c k s flattenBlocks ( project ); // Clean up i n l i n e r g e n e r a t e d c o d e cleanupInlinedCode ( project ) ; // Change members t o p u b l i c changeAllMembersToPublic ( p r o j e c t ) ; // DQ ( 3 / 1 1 / 2 0 0 6 ) : T h i s f a i l s s o t h e i n l i n i n g , o r t h e AST I n t e r f a c e // s u p p o r t , n e e d s more work even though i t g e n e r a t e d good c o d e . // A s t T e s t s : : r u n A l l T e s t s ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

11.4. FINAL CODE AFTER PASSING THROUGH FILE I/O

89

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t x v o i d incrementX ( ) { x++; } int foo () { int a = 0; while ( a < 5) { ++a ; } return a + 3; } i n t main ( i n t , c h a r ) { // Two t r i v a l f u n c t i o n incrementX ( ) ; incrementX ( ) ;

calls

to i n l i n e

// Somthing more i n t e r e s t i n g t o i n l i n e for ( ; foo () < 7;) { x++; } return x ; }

Figure 11.2: Example source code used as input to demonstrate the AST le I/O support.

Figure 11.3: Output of input code after inlining transformations.

90

CHAPTER 11. AST FILE I/O

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t x v o i d incrementX ( ) { x++; } int foo () { int a 0 = 0; while ( a 0 < 5){ ++a 0 ; } return a 0 + 3; } i n t main ( i n t , c h a r ) { x++; x++; // Somthing more i n t e r e s t i n g t o i n l i n e for ( ; true ; ) { int a 1 = 0; while ( a 1 < 5){ ++a 1 ; } int rose temp 7 0 = a 1 + 3; bool rose temp 2 = ( bool )( rose temp i f (! rose temp 2 ) { break ; } else { } x++; } return x ; }

0 < 7);

Figure 11.4: Output of input code after le I/O.

Chapter 12

Debugging Techniques
There are numerous methods ROSE provides to help debug the development of specialized source-to-source translators. This section shows some of the techniques for getting information from IR nodes and displaying it. More information about generation of specialized AST graphs to support debugging can be found in chapter 5 and custom graph generation in section 28.

12.1

Input For Examples Showing Debugging Techniques

Figure 12.1 shows the input code used for the example translators that report useful debugging information in this chapter.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 // Example program show ing m a t r i x m u l t i p l y // ( f o r u s e w i t h l o o p o p t i m i z a t i o n t u t o r i a l example ) #d e f i n e N 50 i n t main ( ) { int i , j , k ; d o u b l e a [ N ] [ N] , b [ N ] [ N ] , c [ N ] [ N ] ; f o r ( i = 0 ; i <= N 1; i +=1) { f o r ( j = 0 ; j <= N 1; j +=1) { f o r ( k = 0 ; k <= N 1; k+=1) { c [ i ][ j ] = c[ i ][ j ] + a[ i ][ k] b[k ][ j ]; } } } return 0; }

Figure 12.1: Example source code used as input to program in codes showing debugging techniques shown in this section.

91

92

CHAPTER 12. DEBUGGING TECHNIQUES

12.2

Generating the code from any IR node

Any IR node may be converted to the string that represents its subtree within the AST. If it is a type, then the string will be the value of the type; if it is a statement, the value will be the source code associated with that statement, including any sub-statements. To support the generation for strings from IR nodes we use the unparseToString() member function. This function strips comments and preprocessor control structure. The resulting string is useful for both debugging and when forming larger strings associated with the specication of transformations using the string-based rewrite mechanism. Using ROSE, IR nodes may be converted to strings, and strings converted to AST fragments of IR nodes. Note that unparsing associated with generating source code for the backend vendor compiler is more than just calling the unparseToString member function, since it introduces comments, preprocessor control structure and formating. Figure 12.2 shows a translator which generates a string for a number of predened IR nodes. Figure 12.1 shows the sample input code and gure 12.5 shows the output from the translator when using the example input application.

12.3

Displaying the source code position of any IR node

This example shows how to obtain information about the position of any IR node relative to where it appeared in the original source code. New IR nodes (or subtrees) that are added to the AST as part of a transformation will be marked as part of a transformation and have no position in the source code. Shared IR nodes (as generated by the AST merge mechanism are marked as shared explicitly (other IR nodes that are shared by denition dont have a SgFileInfo object and are thus not marked explicitly as shared. The example translator to output the source code position is shown in gure 12.4. Using the input code in gure 12.1 the output code is shown in gure 12.5.

12.3. DISPLAYING THE SOURCE CODE POSITION OF ANY IR NODE

93

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h u s i n g namespace s t d ; int main ( i n t a r g c , c h a r a r g v [ ] { ios : : sync with stdio (); if

) // Syncs C++ and C I /O s u b s y s t e m s !

( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( I n p r e p r o c e s s o r . C : main ( ) \ n ) ;

SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // AST d i a g n o s t i c t e s t s A s t T e s t s : : r u n A l l T e s t s ( c o n s t c a s t <S g P r o j e c t >( p r o j e c t ) ) ; // t e s t s t a t i s t i c s i f ( project >g e t v e r b o s e ( ) > 1 ) { c o u t << A s t N o d e S t a t i s t i c s : : t r a v e r s a l S t a t i s t i c s ( p r o j e c t ) ; c o u t << A s t N o d e S t a t i s t i c s : : I R n o d e U s a g e S t a t i s t i c s ( ) ; } ( project >g e t v e r b o s e ( ) > 0 ) p r i n t f ( G e n e r a t e t h e p d f o u t p u t o f t h e SAGE I I I AST \ n ) ; generatePDF ( p r o j e c t ) ; if ( project >g e t v e r b o s e ( ) > 0 ) p r i n t f ( G e n e r a t e t h e DOT o u t p u t o f t h e SAGE I I I AST \ n ) ; generateDOT ( p r o j e c t ) ; if R o s e S T L C o n t a i n e r <SgNode> n o d e L i s t ; // n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgType , NodeQuery : : E x t r a c t T y p e s ) ; n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgForStatement ) ; p r i n t f ( \ n n o d e L i s t . s i z e ( ) = %zu \ n , n o d e L i s t . s i z e ( ) ) ; R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { p r i n t f ( Query node = %p = %s = %s \ n , i , ( i )> s a g e c l a s s n a m e ( ) , ( i )> u n p a r s e T o S t r i n g ( ) . c s t r ( ) ) ; i ++; } return 0; }

Figure 12.2: Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node.

1 2 3 4 5

nodeList . s i z e () = 3 Query node = 0 x2b3d3d7bd010 = SgF or St a temen t = f o r ( i = 0 ; i <= 50 1 ; i += 1 ) { f o r ( j = 0 ; j <= 50 1 ; j += 1 ) { f o r ( k = 0 ; k < Query node = 0 x2b3d3d7bd130 = SgF or St a temen t = f o r ( j = 0 ; j <= 50 1 ; j += 1 ) { f o r ( k = 0 ; k <= 50 1 ; k += 1 ) { c [ i ] [ j ] =( c [ Query node = 0 x2b3d3d7bd250 = SgF or St a temen t = f o r ( k = 0 ; k <= 50 1 ; k += 1 ) { c [ i ] [ j ] =( c [ i ] [ j ] +(a [ i ] [ k ] b [ k ] [ j ] ) ) ; }

Figure 12.3: Output of input code using debuggingIRnodeToString.C

94

CHAPTER 12. DEBUGGING TECHNIQUES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h u s i n g namespace s t d ; int main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( SgProject : : g e t v e r b o s e ( ) > 0) p r i n t f ( I n p r e p r o c e s s o r . C : main ( ) \ n ) ; SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; R o s e S T L C o n t a i n e r <SgNode> n o d e L i s t ; n o d e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgForStatement ) ; p r i n t f ( \ n n o d e L i s t . s i z e ( ) = %zu \ n , n o d e L i s t . s i z e ( ) ) ; R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { S g F i l e I n f o & f i l e I n f o = ( ( i )> g e t f i l e i n f o ( ) ) ; p r i n t f ( Query node = %p = %s i n %s \ n a t l i n e %d on column %d \ n , i , ( i )> s a g e c l a s s n a m e ( ) , f i l e I n f o . g e t f i l e n a m e ( ) , f i l e I n f o . get line () , f i l e I n f o . get col ()); i ++; } if ( project >g e t v e r b o s e ( ) > 0 ) p r i n t f ( C a l l i n g t h e backend ( ) \ n ) ;

return 0; }

Figure 12.4: Example source code showing the output of the string from an IR node. The string represents the code associated with the subtree of the target IR node.

1 2 3 4 5 6 7 8

nodeList . s i z e () = 3 Query node = 0 x2b100348b010 a t l i n e 11 on column Query node = 0 x2b100348b130 a t l i n e 13 on column Query node = 0 x2b100348b250 a t l i n e 15 on column

= SgF orSt a temen t i n / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s 6 = SgF or St a temen t i n / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s 11 = SgF or St a temen t i n / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s 16

hared IR nodes (as generated by the AST merge mechanism are mar

Figure 12.5: Output of input code using debuggingSourceCodePositionInformation.C

Part II

Complex Types

This part elaborates some details for handling complex types in ROSE.

95

Chapter 13

Type and Declaration Modiers

Most languages support the general concept of modiers to types, declarations, etc. The keyword volatile for example is a modier to the type where it is used in a declaration. Searching for the modiers for types and declarations,however, can be confusing. They are often not where one would expect, and most often because of corner cases in the language that force them to be handled in specic ways. This example tutorial code is a demonstration of a how to access the volatile modier used in the declaration of types for variables. We demonstrate that the modier is not present in the SgVariableDeclaration or the SgVariableDenitoon, but is located in the SgModierType used to wrap the type returned from the SgInitializedName (the variable in the variable declaration).

13.1

Input For Example Showing use of Volatile type modier

Figure 13.1 shows the example input used for demonstration of test for the volatile type modier.

1 2 3 4 5 6 7 8 9 10

// I n p u t example o f u s e o f v o l a t i l e t y p e m o d i f i e r volatile int a ,b ;

void foo () { for ( volatile { } }

i n t y = 0 ; y < 1 0 ; y++)

Figure 13.1: Example source code used as input to program in codes used in this chapter.

97

98

CHAPTER 13. TYPE AND DECLARATION MODIFIERS

13.2

Generating the code representing the seeded bug

Figure 13.2 shows a code that traverses each IR node and for and SgInitializedName IR node checks its type. The input code is shown in gure 13.1, the output of this code is shown in gure 13.3.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 #i n c l u d e using class { r o s e . h std ; : public AstSimpleProcessing

namespace

visitorTraversal public : void

v i s i t ( SgNode n ) ;

}; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { // The v o l a t i l e m a d i f i e r i s i n t h e t y p e o f t h e S g I n i t i a l i z e d N a m e SgInitializedName initializedName = isSgInitializedName (n ) ; i f ( i n i t i a l i z e d N a m e != NULL) { p r i n t f ( Found a S g I n i t i a l i z e d N a m e = %s \ n , i n i t i a l i z e d N a m e >g e t n a m e ( ) . s t r ( ) ) ; SgType t y p e = i n i t i a l i z e d N a m e >g e t t y p e ( ) ; p r i n t f ( i n i t i a l i z e d N a m e : t y p e = %p = %s \ n , t y p e , t y p e >c l a s s n a m e ( ) . c s t r ( ) ) ; SgModifierType modifierType = isSgModifierType ( type ) ; i f ( m o d i f i e r T y p e != NULL) { bool i s V o l a t i l e = modifierType >g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V o l a t i l e ( ) ; p r i n t f ( i n i t i a l i z e d N a m e : S g M o d i f i e r T y p e : i s V o l a t i l e = %s \ n , ( i s V o l a t i l e == t r u e ) ? t r u e } S g M o d i f i e r N o d e s m o d i f i e r N o d e s = t y p e >g e t m o d i f i e r s ( ) ; p r i n t f ( i n i t i a l i z e d N a m e : m o d i f i e r N o d e s = %p \ n , m o d i f i e r N o d e s ) ; i f ( m o d i f i e r N o d e s != NULL) { SgModifierTypePtrVector m o d i f i e r L i s t = modifierNodes >g e t n o d e s ( ) ; f o r ( S g M o d i f i e r T y p e P t r V e c t o r : : i t e r a t o r i = m o d i f i e r L i s t . b e g i n ( ) ; i != m o d i f i e r L i s t . end ( ) ; { p r i n t f ( i n i t i a l i z e d N a m e : m o d i f i e r s : i = %s \ n , ( i )> c l a s s n a m e ( ) . c s t r ( ) ) ; } } }

false );

i ++)

// Note t h a t t h e v o l a t i l e m a d i f i e r i s n o t i n t h e S g V a r i a b l e D e c l a r a t i o n n o r t h e S g V a r i a b l e D e f i n i t i o n SgVariableDeclaration variableDeclaration = isSgVariableDeclaration (n ) ; i f ( v a r i a b l e D e c l a r a t i o n != NULL) { bool i s V o l a t i l e = variableDeclaration >g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V o l a t p r i n t f ( S g V a r i a b l e D e c l a r a t i o n : i s V o l a t i l e = %s \ n , ( i s V o l a t i l e == t r u e ) ? t r u e : f a l s e ) ; SgVariableDefinition variableDefinition = variableDeclaration >g e t d e f i n i t i o n ( ) ; // p r i n t f ( v a r i a b l e D e f i n i t i o n = %p \ n , v a r i a b l e D e f i n i t i o n ) ; i f ( v a r i a b l e D e f i n i t i o n != NULL) { bool i s V o l a t i l e = v a r i a b l e D e f i n i t i o n >g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t t y p e M o d i f i e r ( ) . g e t c o n s t V o l a t i l e M o d i f i e r ( ) . i s V p r i n t f ( S g V a r i a b l e D e f i n i t i o n : i s V o l a t i l e = %s \ n , ( i s V o l a t i l e == t r u e ) ? t r u e : f a l s e ) ; } } } // must h a v e a r g c and a r g v h e r e ! ! i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject project = frontend

( argc ,

argv ) ;

visitorTraversal myvisitor ; myvisitor . traverseInputFiles ( project , preorder ) ; return } backend ( p r o j e c t ) ;

Figure 13.2: Example source code showing how to detect volatile modier.

13.2. GENERATING THE CODE REPRESENTING THE SEEDED BUG

99

1 2 3 4 5 6 7 8 9

// I n p u t example o f u s e o f v o l a t i l e t y p e m o d i f i e r volatile int a ; v o l a t i l e i n t b ; void foo () { for ( volatile } }

i n t y = 0 ; y < 1 0 ; y++) {

Figure 13.3: Output of input code using volatileTypeModier.C

100

CHAPTER 13. TYPE AND DECLARATION MODIFIERS

Chapter 14

Function Parameter Types

The analysis of functions often requires the query of the function types. This tutorial example shows how to obtain the function parameter types for any function. Note that functions also have a type which is based on their signature, a combination of their return type and functions parameter types. Any functions sharing the same return type and function parameter types have the same function type (the function type, a SgFunctionType IR node, will be shared between such functions). Figure 14.1 shows a translator which reads an application (shown in gure 14.2) and outputs information about the function parameter types for each function, shown in gure 14.3. This information includes the order of the function declaration in the global scope, and name of the function, and the types of each parameter declared in the function declaration. Note that there are a number of builtin functions dened as part of the GNU g++ and gcc compatibility and these are output as well. These are marked as compiler generated functions within ROSE. The code shows how to dierentiate between the two dierent types. Notice also that instantiated template functions are classied as compiler generated.

101

102

CHAPTER 14. FUNCTION PARAMETER TYPES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ;

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r

int functionCounter = 0; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = i sS gFu ncti o nDecl a ra ti o n ( i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // DQ ( 3 / 5 / 2 0 0 6 ) : Only o u t p u t t h e non c o m p i l e r g e n e r a t e d IR n o d e s i f ( ( i )> g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) { SgFunctionParameterList functionParameters = functionDeclaration >g e t p a r a m e t e r L i s t ( ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( Non c o m p i l e r g e n e r a t e d f u n c t i o n name #%3d i s %s \ n , f u n c t i o n C o u n t e r ++, f u n c t i o n D e c l a r a t

SgInitializedNamePtrList & parameterList = functionParameters >g e t a r g s ( ) ; i n t parameterCounter = 0 ; f o r ( S g I n i t i a l i z e d N a m e P t r L i s t : : i t e r a t o r j = p a r a m e t e r L i s t . b e g i n ( ) ; j != p a r a m e t e r L i s t . end ( ) ; j + { SgType parameterType = ( j )> g e t t y p e ( ) ; p r i n t f ( parameterType #%2d = %s \ n , p a r a m e t e r C o u n t e r ++,parameterType >u n p a r s e T o S t r i n g ( }

} else { p r i n t f ( C o m p i l e r g e n e r a t e d f u n c t i o n name #%3d i s %s \ n , f u n c t i o n C o u n t e r ++, f u n c t i o n D e c l a r a t i o n } } return 0; }

Figure 14.1: Example source code showing how to get type information from function parameters.

103

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( i n t a r g c , c h a r a r g v [ ] { foo (42); foo (3.14159265); )

t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 14.2: Example source code used as input to typeInfoFromFunctionParameters.C.

104

CHAPTER 14. FUNCTION PARAMETER TYPES

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

Compiler C o mp i l e r C o mp i l e r C o mp i l e r C o mp i l e r C o mp i l e r C o mp i l e r C o mp i l e r C o mp i l e r Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler Compiler

generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated generated

function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function function

name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name name

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82

is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is

builtin copysign builtin copysignf builtin copysignl builtin acosf builtin acosl builtin asinf builtin asinl builtin atanf builtin atanl builtin atan2f builtin atan2l builtin ceilf builtin ceill builtin coshf builtin coshl builtin floorf builtin floorl builtin fmodf builtin fmodl builtin frexpf builtin frexpl builtin ldexpf builtin ldexpl builtin log10f builtin log10l builtin modff builtin modfl builtin powf builtin powl builtin sinhf builtin sinhl builtin tanf builtin tanl builtin tanhf builtin tanhl builtin powil builtin powi builtin powif builtin strchr builtin strrchr builtin strpbrk builtin strstr builtin nansf builtin nans builtin nansl builtin fabs builtin fabsf builtin fabsl builtin cosf builtin cosl builtin sinf builtin sinl builtin sqrtf builtin sqrtl builtin fpclassify builtin return address builtin frame address builtin expect builtin prefetch builtin huge val builtin huge valf builtin huge vall builtin inf builtin inff builtin infl builtin nan builtin nanf builtin nanl builtin nans builtin nansf builtin nansl builtin clz builtin ctz builtin popcount builtin parity builtin ffsl builtin clzl builtin ctzl builtin popcountl builtin parityl builtin ffsll builtin clzll builtin ctzll

Chapter 15

Resolving Overloaded Functions

Figure 15.1 shows a translator which reads an application and reposts on the mapping between function calls and function declarations. This is trivial since all overloaded function resolution is done within the frontend and so need not be computed (this is because all type resolution is done in the frontend and stored in the AST explicitly). Other compiler infrastructures often require this to be gured out from the AST, when type resolution is unavailable, and while not too hard for C, this is particularly complex for C++ (due to overloading and type promotion within function arguments). Figure 15.2 shows the input code used to get the translator. Figure 15.3 shows the resulting output.

105

106

CHAPTER 15. RESOLVING OVERLOADED FUNCTIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) ; int functionCounter = 0; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; i != f u n c t i o n C a l l L i s t . end ( ) ; { S g E x p r e s s i o n f u n c t i o n E x p r e s s i o n = i s S g F u n c t i o n C a l l E x p ( i )> g e t f u n c t i o n ( ) ; ROSE ASSERT( f u n c t i o n E x p r e s s i o n != NULL ) ; SgFunctionRefExp f u n c t i o n R e f E x p = i s S g F u n c t i o n R e f E x p ( f u n c t i o n E x p r e s s i o n ) ; SgFunctionSymbol f u n c t i o n S y m b o l = NULL ; i f ( f u n c t i o n R e f E x p != NULL) { // Case o f nonmember f u n c t i o n functionSymbol = functionRefExp >g e t s y m b o l ( ) ; } else { // Case o f member f u n c t i o n ( h i d d e n i n r h s o f b i n a r y d o t o p e r a t o r e x p r e s s i o n ) SgDotExp dotExp = isSgDotExp ( f u n c t i o n E x p r e s s i o n ) ; ROSE ASSERT( dotExp != NULL ) ; f u n c t i o n E x p r e s s i o n = dotExp >g e t r h s o p e r a n d ( ) ; SgMemberFunctionRefExp memberFunctionRefExp = isSgMemberFunctionRefExp ( f u n c t i o n E x p r e s s i o n ) ; ROSE ASSERT( memberFunctionRefExp != NULL ) ; f u n c t i o n S y m b o l = memberFunctionRefExp >g e t s y m b o l ( ) ; } ROSE ASSERT( f u n c t i o n S y m b o l != NULL ) ; SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = functionSymbol >g e t d e c l a r a t i o n ( ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ;

i ++

// Output mapping o f f u n c t i o n c a l l s t o f u n c t i o n d e c l a r a t i o n s p r i n t f ( L o c a t i o n o f f u n c t i o n c a l l #%d a t l i n e %d r e s o l v e d by o v e r l o a d e d f u n c t i o n d e c l a r e d a t l i n e % f u n c t i o n C o u n t e r ++, i s S g F u n c t i o n C a l l E x p ( i )> g e t f i l e i n f o ()> g e t l i n e ( ) , functionDeclaration > g e t f i l e i n f o ()> g e t l i n e ( ) ) ; } return 0; }

Figure 15.1: Example source code showing mapping of function calls to overloaded function declarations.

107

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; }; // O v e r l o a d e d f u n c t i o n s void foo ( i n t ) ; void foo ( double ) { int x = 1; int y ; for t e s t i n g overloaded function resolution

// Added t o a l l o w non t r i v i a l CFG i f (x) y = 2; else y = 3; } i n t main ( ) { foo (42); foo (3.14159265); t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 15.2: Example source code used as input to resolveOverloadedFunction.C.

1 2 3 4

Location Location Location Location

of of of of

function function function function

call call call call

#0 #1 #2 #3

at at at at

line line line line

29 30 33 34

resolved resolved resolved resolved

by by by by

overloaded overloaded overloaded overloaded

function function function function

declared declared declared declared

at at at at

line line line line

14 15 0 0

Figure 15.3: Output of input to resolveOverloadedFunction.C.

108

CHAPTER 15. RESOLVING OVERLOADED FUNCTIONS

Chapter 16

Template Parameter Extraction

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> t e m p l a t e I n s t a n t i a t i o n D e c l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g T e m p l a t e I n s t a n t i a t i o n D e c l ) ; i n t classTemplateCounter = 0 ; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = t e m p l a t e I n s t a n t i a t i o n D e c l L i s t . b e g i n ( ) ; i != t e m p l a t e I n s t a n t i a t i o n D e c l L i s t . end ( ) ; i ++) { SgTemplateInstantiationDecl instantiatedTemplateClass = i s S g T e m p l a t e I n s t a n t i a t i o n D e c l ( i ) ; ROSE ASSERT( i n s t a n t i a t e d T e m p l a t e C l a s s != NULL ) ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( C l a s s name #%d i s %s \ n , c l a s s T e m p l a t e C o u n t e r ++, instantiatedTemplateClass >g e t t e m p l a t e N a m e ( ) . s t r ( ) ) ; c o n s t SgTemplateArgumentPtrList& t e m p l a t e P a r a m e t e r L i s t = i n s t a n t i a t e d T e m p l a t e C l a s s >g e t t e m p l a t e A r g u m e n t s ( ) ; i n t parameterCounter = 0 ; f o r ( SgTemplateArgumentPtrList : : c o n s t i t e r a t o r j = t e m p l a t e P a r a m e t e r L i s t . b e g i n ( ) ; j != t e m p l a t e P a r a m e t e r L i s t . end ( ) ; j ++) { p r i n t f ( TemplateArgument #%d = %s \ n , p a r a m e t e r C o u n t e r ++,( j )> u n p a r s e T o S t r i n g ( ) . c s t r ( ) ) ; } } return 0; }

Figure 16.1: Example source code used to extract template parameter information. 109

110

CHAPTER 16. TEMPLATE PARAMETER EXTRACTION

Figure 16.1 shows a translator which reads an application and gathers a list of loop nests. At the end of the traversal it reports information about each instantiated template, including the template arguments. Figure 16.2 shows the input code used to get the translator. Figure 16.3 shows the resulting output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

// Templated c l a s s d e c l a r a t i o n u s e d i n t e m p l a t e p a r a m e t e r example c o d e t e m p l a t e <typename T> c l a s s templateClass { public : int x ; void foo ( i n t ) ; void foo ( double ) ; };

i n t main ( ) { t e m p l a t e C l a s s <char > i n s t a n t i a t e d C l a s s ; instantiatedClass . foo ( 7 ) ; instantiatedClass . foo ( 7 . 0 ) ; t e m p l a t e C l a s s <i n t > i n s t a n t i a t e d C l a s s I n t ; t e m p l a t e C l a s s <f l o a t > i n s t a n t i a t e d C l a s s F l o a t ; t e m p l a t e C l a s s <t e m p l a t e C l a s s <char > > i n s t a n t i a t e d C l a s s N e s t e d C h a r ; f o r ( i n t i =0; i < 4 ; { int x ; } return 0; } i ++)

Figure 16.2: Example source code used as input to templateParameter.C.

1 2 3 4 5 6 7 8

C l a s s name #0 i s t e m p l a t e C l a s s TemplateArgument #0 = c h a r C l a s s name #1 i s t e m p l a t e C l a s s TemplateArgument #0 = i n t C l a s s name #2 i s t e m p l a t e C l a s s TemplateArgument #0 = f l o a t C l a s s name #3 i s t e m p l a t e C l a s s TemplateArgument #0 = t e m p l a t e C l a s s < c h a r >

Figure 16.3: Output of input to templateParameter.C.

Chapter 17

Template Support
This chapter is specic to demonstrating the C++ template support in ROSE. This section is not an introduction to the general subject of C++ templates. ROSE provides special handling for C++ templates because template instantiation must be controlled by the compiler. Templates that require instantiation are instantiated by ROSE and can be seen in the traversal of the AST (and transformed). Any templates that can be instantiated by the backend compiler and are not transformed are not output within the code generation phase.

17.1

Example Template Code #1

FIXME: Provide a list of when templates are generated internally in the AST and when template instantiations are output.

This section presents gure 17.4, a simple C++ source code using a template. It is used as a basis for showing how template instantiations are handled within ROSE.
1 2 3 4 5 6 7 8 9 10 11 12 t e m p l a t e <typename T> class X { public : void foo ( ) ; }; X <i n t > x ; void X <i n t > : : f o o ( ) { }

Figure 17.1: Example source code showing use of a C++ template.

17.2

Example Template Code #2

This section presents gure 17.4, a simple C++ source code using a template function. It is used as a basis for showing how template instantiations are handled within ROSE.

111

112

CHAPTER 17. TEMPLATE SUPPORT

1 2 3 4 5 6 7

t e m p l a t e < typename T > class X { public : void foo ( ) ; }; class X < int > x ;

Figure 17.2: Example source code after processing using identityTranslator (shown in gure 2.1).

1 2 3 4 5 6 7 8 9 10 11 12 13

// t e m p l a t e f u n c t i o n t e m p l a t e <typename T> void foo ( T t ) { } // S p e c i a l i z a t i o n from u s e r template< > v o i d f o o <i n t >( i n t x ) {} i n t main ( ) { foo ( 1 ) ; }

Figure 17.3: Example source code showing use of a C++ template.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

// t e m p l a t e f u n c t i o n t e m p l a t e < typename T > void foo ( T t ) { } // S p e c i a l i z a t i o n from u s e r template< > void foo < i n t > ( i n t x ) { } i n t main ( ) { foo< i n t > ( 1 ) ; return 0; }

Figure 17.4: Example source code after processing using identityTranslator (shown in gure 2.1).

Part III

Program Analyses

This part exemplies the use of existing ROSE analyses and how to build customized analyses using ROSE.

113

Chapter 18

Generic Dataow Analysis Framework

This chapter summarizes the basic considerations behind the ROSE dataow framework as well as how its API can be used to implement inter- and intra-procedural dataow analyses. It is oriented towards potential users of the framework.

18.1

Basics of DataFlowAnalysis

Dataow analysis is a technique for determining an applications possible states at various points in a program. It works as a xed-point iteration over a space of possible facts about each node in the applications Control Flow Graph (CFG). The algorithm starts with no information about each node and iterates by accumulating all the constraints on the applications state at each node until it reaches a xed point where no additional constraints can be discovered. The designer of a given dataow analysis must specify an abstract representation of the set of all possible application states that maintains the relevant details of the state (e.g. whether a variable has a constant value or the linear relationships between variable pairs), while igoring the rest. For example, a state abstraction for the constant propagation analysis may have three dierent values: the special symbol if the variable is uninitialized, a numeric value if this is the only value the variable may hold at a given CFG node or if it may have more than one value. More sophisticated abstractions represent application state using polyhedral constraints or predicate logic. Further, the designer must specify a transfer function that maps the abstract state before any application operation to the state after it. For example, if before statement i + + it is known that i == n then after the statement it is known that i 1 == n. To deal with control ow the designer also species a meet function that conjoins the abstract states along multiple control paths. For example, if at the end of the if branch of a conditional it is known that i=5 and at the end of the else branch i<10, the strongest fact that is true immediately after both branches of the conditional is 5 i <10. The set of possible abstract states must form a lattice, which is a partial order where for any pair of elements there exists a unique least upper bound. Intuitively, states that are lower in the partial order represent fewer constraints on the application state and higher states represent 115

116

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.1: Example of a constant propagation analysis.

more constraints. The special state corresponds to the least state in the partial order, where the application has done nothing to constrain its state (e.g. all variables are uninitialized). The meet function must guarantee the uniqueness of the least upper bound and the transfer function must be monotonic (if AB then transfer(A)transfer(B)). The dataow xed-point iteration ensures that the abstract state of every CFG node rises monotonically as it incorporates information about more possible application behaviors. When the analysis reaches a xed point, the abstract state at each CFG node corresponds to the tightest set of constraints that are can be specied by the abstraction about the application state at that location. For an intuition about how dataow analyses work, 18.1 presents an example of a constant propagation analysis. The CFG is on the left and the table on the right shows the xed-point solution of the abstract state immediately after each node. At each node the abstract application state records for each variable one of the following values: (i) , which indicates that the variable is uninitialized, (ii) a specic constant value if the variable may only have this value at node n or (iii) which indicates that the variable may have more than one value (i.e., is not representable as a single constant).It shows that immediately after node A it is known that i=0 and similarly after node B, i=0 and j=5. The same is true after node C since it has no side-eects and after the assignment in node D, the state changes to i=2, j=5. When the two conditional branches meet, the abstract state is the union of the states on both branches: the strongest assertions that are true of both states. Since j has the same value and i has two dierent values, the abstract state after node E is i= , j=5. ?? presents an example with a more complex abstraction: conjunction of linear relationships between variables. At node B the dataow analysis computes that i=0 and j=5. When this state is propagated through the loop, the analysis discovers that after the rst iteration i=4 and j=5. It then computes the meet of i=0 j=5 and i=4 j=5, the facts along both paths. Since this abstraction represents linear relationships, the union nds the tightest linear relationships that are true of both input states. It thus infers that i=0 (mod 4), i=0 (mod 5) (i is divisible by 4 and j by 5) and that 5i=4j-5. When this state is propagated again through the body of the loop, these assertions are discovered to be the invariants of this loop and become the xed-point solution after node C. If they were not invariants, the algorithm would iterate until invariants were found or it reached the abstract state which means that no linear constraints are known. Further, since the conditional j<100 is also linear, j<100 is recorded in the states of the nodes inside the loop and j 100 is recorded at node F after the loop.

18.2. ROSE DATAFLOW FRAMEWORK

117

Figure 18.2: Example of a dataow analysis with abstraction of ane constraints.

18.2

ROSE Dataow Framework

ROSE provides a framework for implementing dataow analyses. It allows users to specify their dataow analysis by implement the standard dataow components: (i) an abstraction of the applications state, (ii) a transfer function that species the eects of code on the application state and (iii) a meet operator that combines multiple possible abstract states into one. These are implemented by extending base classes provided by the framework and implementing key virtual methods that correspond to the above functionality. The framework then solves the dataow equations using the user-provided classes and saves the results at each CFG node. This section describes the functionality provided by the framework and how it can be used to implement analyses. 18.1 summarizes the functionality provided by the framework.

18.2.1

Call and Control-Flow Graphs

The ROSE dataow analysis framework operates on top of the ROSE Call Graph (CG) and Virtual Control-Flow Graph (VCFG). The CG documents the caller/callee relationships between application functions. The VCFG connects SgNodes in the applications AST to identify the possible execution orders between them. The VCFG is dynamic in that instead of being computing once for the entire application, it computes the outgoing and incoming edges of a given SgNode fresh every time this information is needed. This makes the VCFG very exible because it automatically responds to changes in the AST with no need for complex adjustments to the graph.

18.2.2

Analyses

ROSE supports both inter-and intra-procedural analyses. Users implement basic, non-dataow analyses by extending the IntraProceduralAnalysis and InterProceduralAnalysis classes. Intra analyses iterate over the CFG of each function, and inter analyses apply intra analyses to individual

118
Class Analysis

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Purpose Implement Simple CFG passes Implement intraprocedural dataow iteration Implement the inter-procedural dataow Generic interface for abstractions of the applications state Stores dataow information of each CFG node Transforms the CFG Interface Classes IntraProceduralAnalysis InterProceduralAnalysis Classes IntraFWDataow IntraBWDataow User Responsibilities Extend the classes and implement their runAnalysis and transfer methods Extend classes and implement genInitState and transfer methods Execute on a given instance of IntraDataow Extend the Lattice class and implement interface methods

Intra Procedural Dataow Inter Procedural Dataow Lattice

Classes - Context Insensitive InterProcedural Dataow Methods initialize copy meetUpdate operator== str

Nodestate

AstInterface

Methods setLatticeAbove getLatticeAbove deleteLatticeAbove setFact getFact deleteFacts Methods insertBeforeUsing CommaOp, insertAfterUsing CommaOp, replaceWithPattern

Call methods to access dataow information

Call methods

Table 18.1: The Functionality of the Dataow Interface functions. To implement an analysis an application developer must derive a class from the IntraProceduralAnalysis and/or InterProceduralAnalysis classes and implement the runAnalysis method. Classes UnstructuredPassInterAnalysis and UnstructuredPassIntraAnalysis Figure 18.2.3 provide examples of simple analyses. UnstructuredPassInterAnalysis takes as an argument a reference to an InterProceduralAnalysis and iterates once through all functions. It applies the runAnalysis method of the intra analysis to each function. UnstructuredPassIntraAnalysis iterates once through all the CFG nodes in the given function, applying its visit method to each node. These analyses can be used to implement simple passes through the applications CFG and serve as the foundation of the dataow analysis framework. For example, src/simpleAnalyses/saveDotAnalysis.C and src/simpleAnalyses/printAnalysisStates.C are examples of simple one-pass analyses. saveDotAnalysis prints the applications CFG as a DOT le and printAnalysisStates prints the dataow

1 2 3 4 5 6 7 8 9 10

c l a s s UnstructuredPassInterAnalysis : virtual public InterProceduralAnalysis { U n s t r u c t u r e d P a s s I n t e r A n a l y s i s ( I n t r a P r o c e d u r a l A n a l y s i s& i n t r a A n a l y s i s ) void runAnalysis ( ) ; }; c l a s s UnstructuredPassIntraAnalysis : virtual public IntraProceduralAnalysis { b o o l r u n A n a l y s i s ( c o n s t Function& func , NodeState s t a t e ) ; v i r t u a l v o i d v i s i t ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e )=0; }; Figure 18.3: Example of simple analyses

18.2. ROSE DATAFLOW FRAMEWORK

119

Figure 18.4: Each variables lattice for constant-propagation analysis states of all CFG nodes in the application, which is useful for debugging.

18.2.3

Dataow

To implement a dataow analysis in ROSE users must rst extend the Lattice class to create an abstraction of the applications state that will be used by the analysis. Lattices implement methods such as meet, equality, ordering and operators that allow the Lattice to be moved from one lexical scope to another (e.g. from a caller function to the callee). Users then create an intra-procedural analysis by extending the IntraFWDataow to create a forward analysis and from IntraBWDataow to create a backward analysis. Within this class they must implement a function that returns the default abstract state of any given node at the start of the analysis. Further, they implement a transfer function that maps the applications abstract state from before a given CFG node to the state that results from the execution of the nodes expression or statement. Finally, users combine the intra-procedural analysis that they have developed with an inter-procedural analysis of their choice. This analysis will apply the intra-procedural analysis the user has implemented to the applications functions and resolve the eects of function calls on the applications abstract state, utilizing the users own state abstraction. For a concrete example, consider how the classical constant-propagation analysis is implemented using ROSE. This analysis uses a simple abstraction of application state, where the abstract state of each variable may be a value the lattice shown in 18.4 The code in below shows a class that implements this lattice. This class derives from the FiniteLattice class because the distance between the smallest and largest value in the lattice is nite. Similar functionality is provided for innite lattices. Its state (lines 4-15) consists of its current level in the lattice as well as its value if the level is valKnown. Since the type of value is long, this abstraction can only represent integral constants. Further, the class has a special uninitialized level that means that the object has not yet been used as part of a dataow analysis. This class implements methods meetUpdate (lines 42-65) and the equality operator (lines 68-74) to provide the basic semantics of a lattice. meetUpdate computes least upper bound of the constraints in this lattice object and another one, storing the results in this object.If both lattices have the same state, the meet is equal to this state and if they have dierent states, the meet is since the variable represented by the lattice may be set to multiple values on dierent execution paths. The equality operator determines whether two lattice objects have the same information content. Further, the class implements utility methods that help the dataow framework manipulate it. The initialize method (lines 24-27) ensures the object is ready to be used. Further, two copy methods (lines 30-38) make it easy to clone lattice objects. Finally, an str method (lines 82-88)

120

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

simplies analysis debugging by printing the abstract states at CFG nodes.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

c l a s s constPropLat : p u b l i c F i n i t e L a t t i c e { // The d i f f e r e n t l e v e l s o f t h i s o b j e c t s l a t t i c e t y p e d e f enum { u n i n i t i a l i z e d =0, // This o b j e c t i s u n i n i t i a l i z e d bottom =1, // No c o n s t r a i n s on t h i s o b j e c t s v a l u e a r e known valKnown=2, // The v a l u e o f t h e v a r i a b l e i s known ( one a s s i g n m e n t s e e n ) top=3 // This v a r i a b l e may have more than one v a l u e } latticeLevels ; // The l e v e l o f t h i s o b j e c t w i t h i n i t s l a t t i c e latticeLevels level ; // The v a l u e o f t h e v a r i a b l e ( i f l e v e l == valKnown ) long value ; n o d e C o n s t L a t t i c e ( ) { l e v e l=u n i n i t i a l i z e d ; } n o d e C o n s t L a t t i c e ( c o n s t n o d e C o n s t L a t t i c e& t h a t ) : v a l u e ( t h a t . v a l u e ) , l e v e l ( t h a t . l e v e l ) {} // I n i t i a l i z e s t h i s L a t t i c e t o i t s d e f a u l t s t a t e , // i f i t i s not a l r e a d y i n i t i a l i z e d void i n i t i a l i z e ( ) { i f ( l e v e l == u n i n i t i a l i z e d ) l e v e l=bottom ; } // Returns a copy o f t h i s l a t t i c e L a t t i c e copy ( ) c o n s t { r e t u r n new n o d e C o n s t L a t t i c e ( t h i s ) ; } // O v e r w r i t e s t h e s t a t e o f t h i s L a t t i c e with t h a t o f t h a t L a t t i c e v o i d copy ( L a t t i c e t h a t ) { n o d e C o n s t L a t t i c e t h a t = d y n a m i c c a s t <n o d e C o n s t L a t t i c e >( t h a t a r g ) ; v a l u e = that >v a l u e ; l e v e l = that > l e v e l ; } // Computes t h e meet o f t h i s and t h a t and s a v e s t h e r e s u l t i n t h i s // r e t u r n s t r u e i f t h i s c a u s e s t h i s t o change and f a l s e o t h e r w i s e b o o l meetUpdate ( L a t t i c e t h a t ) { // Record t h i s o b j e c t s o r i g i n a l s t a t e t o e n a b l e change d e t e c t i o n

18.2. ROSE DATAFLOW FRAMEWORK 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 unsigned long oldValue = value ; l a t t i c e L e v e l s oldLevel = l e v e l ;

121

// Cast t h a t i n t o a n o d e C o n s t L a t t i c e and a b o r t i f t h i s i s not p o s s i b l e n o d e C o n s t L a t t i c e t h a t = d y n a m i c c a s t <n o d e C o n s t L a t t i c e >( t h a t a r g ) ; ROSE ASSERT( t h a t ) ; // I f t h a t i s a t a h i g h e r l a t t i c e l e v e l than t h i s , t h e v a r i a b l e must have // m u l t i p l e p o s s i b l e v a l u e on d i f f e r e n t e x e c u t i o n p a t h s i f ( that > l e v e l > l e v e l ) l e v e l = top ; // I f both a r e a t t h e same l e v e l e l s e i f ( that > l e v e l == l e v e l ) { // I f l a t t i c e s c o r r e s p o n d t o d i f f e r e n t v a l u e s o f t h e v a r i a b l e i f ( l e v e l == valKnown && v a l u e != that >v a l u e ) l e v e l = top ; // The union o f both t h e s e f a c t s i s top } // Otherwise , t h i s l a t t i c e d o e s n t change // Return whether t h i s o b j e c t was m o d i f i e d r e t u r n ( oldValID != valID ) | | ( o l d L e v e l != l e v e l ) ; } // E q u a l i t y Operator b o o l o p e r a t o r==( L a t t i c e t h a t a r g ) { // Cast t h a t i n t o a n o d e C o n s t L a t t i c e and a b o r t i f t h i s i s not p o s s i b l e n o d e C o n s t L a t t i c e t h a t = d y n a m i c c a s t <n o d e C o n s t L a t t i c e >( t h a t a r g ) ; ROSE ASSERT( t h a t ) ; r e t u r n l e v e l==that > l e v e l && ( l e v e l !=valKnown | | v a l u e==that >v a l u e ) ; } // Returns a s t r i n g r e p r e s e n t a t i o n o f t h i s o b j e c t ( t h i s f u n c t i o n i s // r e q u i r e d t o s i m p l i f y debugging ) s t r i n g s t r ( s t r i n g i n d e n t =) { } // S e t s t h e s t a t e o f t h i s l a t t i c e t o t h e g i v e n v a l u e . Returns t r u e i f t h i s // c a u s e s t h e l a t t i c e s s t a t e t o change , f a l s e o t h e r w i s e bool set ( long value ) { b o o l m o d i f i e d = t h i s > l e v e l != valKnown | | t h i s >v a l u e != v a l u e ; t h i s > v a l u e = v a l u e ; l e v e l = valKnown ; return modified ; } };

122

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

The second step in implementing constant propagation is to provide a class that implements the dataow analysis itself. This is done by extending the IntraFWDataow class, which implements forward intra-procedural analyses and implementing the genInitState and transfer methods, described below. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 c l a s s c o n s t P r o p A n a l y s i s : p u b l i c IntraFWDataflow { c o n s t P r o p A n a l y s i s ( ) : IntraFWDataflow ( ) { } // G e n e r a t e s t h e i n i t i a l l a t t i c e s t a t e f o r t h e g i v e n d a t a f l o w node , i n t h e // g i v e n f u n c t i o n , with t h e g i v e n NodeState v o i d g e n I n i t S t a t e ( c o n s t Function& func , c o n s t DataflowNode& n , c o n s t NodeState& s t a t e , v e c t o r < L a t t i c e >& i n i t L a t t i c e s , v e c t o r <NodeFact>& i n i t F a c t s ) ; // The t r a n s f e r f u n c t i o n t h a t i s a p p l i e d t o e v e r y node i n t h e CFG // n The d a t a f l o w node t h a t i s b e i n g p r o c e s s e d // s t a t e The NodeState o b j e c t t h a t d e s c r i b e s t h e s t a t e o f t h e node , a s // e s t a b l i s h e d by e a r l i e r a n a l y s i s p a s s e s // d f I n f o The L a t t i c e s t h a t t h i s t r a n s f e r f u n c t i o n o p e r a t e s on . The // f u n c t i o n t a k e s t h e s e l a t t i c e s a s i n p u t and o v e r w r i t e s them with // the r e s u l t of the t r a n s f e r . // Returns t r u e i f any o f t h e i n p u t l a t t i c e s changed a s a r e s u l t o f t h e // t r a n s f e r f u n c t i o n and f a l s e o t h e r w i s e . b o o l t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r < L a t t i c e >& d f I n f o ) ; } The constPropAnalysis implementation of method genInitState creates a lattice (lines 5-7) that maintains the initial abstract state of the application at CFG node n. This lattice is an instance of the utility class FiniteVarsExprsProductLattice, which creates one copy of constPropLat for each variable that is live at node n. Since it is a product of lattices, this class is also a lattice with well-dened meet and equality operators based on the operators of its constituent lattices. The dataow framework provides an identical class for innite lattices as well as a generic ProductLattice class for arbitrary products of lattices. The function then adds (line 4) the lattice to vector initLattices, which is read by the dataow framework. This function can also specify one or more facts that the framework will maintain at each node. These facts are not subject to dataow iteration and can be used to maintain information that is useful independently of the current dataow state. 1 2 3 4 5 v o i d c o n s t P r o p A n a l y s i s : : g e n I n i t S t a t e ( c o n s t Function& func , c o n s t DataflowNode& n , c o n s t NodeState& s t a t e , v e c t o r < L a t t i c e >& i n i t L a t t i c e s , v e c t o r <NodeFact>& i n i t F a c t s ) { i n i t L a t t i c e s . push back ( new F i n i t e V a r s E x p r s P r o d u c t L a t t i c e ( t r u e , f a l s e ,

18.2. ROSE DATAFLOW FRAMEWORK 6 7 8 new constPropLat ( ) , NULL, func , n , s t a t e ) ) ; }

123

The transfer method maps the abstract state before the CFG node n to the state that results from its execution. It begins by accessing the applications abstract state above node n from the dfInfo argument (lines 6-7) . This is the vector of lattices created by genInitState for node n. It can also be obtained from the state object, which maintains the state of the lattices both below and above each node, as well as the facts at each node. The function then initializes all the constPropLats in the product lattice (10-13) and advances to analyze the eects of the current node on the abstract state. Lines 16-127 show how the transfer function operates on dierent types of SgNodes. This code leverages a key feature of how ROSE represents the applications structure. Since ROSE focuses on source-to-source transformations that minimize the changes in the applications source code, all analyses must work on the original AST and cannot perform large normalization passes such as transforming the application into SSA form. Since it is dicult to implement complex analyses on top of the AST, we have developed an on-demand normalization that signicantly simplies the analysis development without changing the AST. Working with AST is dicult because AST sub-trees that describe the structure of expressions are complex and dicult to parse (e.g. consider analyzing all the side-eects of a=b=foo(c=d)). As such, our framework treats every SgExpression that does not correspond to an actual memory object as if it produces a temporary object that is read by its parent SgExpression. For example, in the SgExpression a=(b=c*5+d), SgIntVal 5 produces a temporary variable that is consumed by SgMultiplyOp c*5. SgVarRefExp c produces a real application variable, which is also consumed by the SgMultiplyOp. The SgMultiplyOp in turn produces a temporary variable that is consumed by SgAddOp c*5+d, which produces a temporary variable that is consumed by SgAssignOp b=c*5+d, the result of which is consumed by SgAssignOp a=(b=c*5+d). The use of these temporary variables makes it possible for user analyses to focus on just the eects of individual AST nodes without having to analyze sub-trees of the AST. Section 18.2.4 discusses how this on-demand normalization in maintained when updating to the AST. The eects of integral constants (e.g. SgIntVal or SgLongLongIntVal) are transferred on lines 16-31. On line 20 the analysis calls function SgExpr2Var to convert the SgExpression into a varID, which is an abstract representation of the memory object (either a real or temporary variable) denoted by the SgExpression. On line 21 it queries the FiniteVarsExprsProductLattice prod with this varID to get the constPropLat associated with this memory object. If this variable is live (a nonNULL lattice is returned), on lines 26-28 it sets the lattice object to be at level valKnown and sets the value to be equal to the constant represented by the SgExpression. The same logic is used for non-integral constants on lines 33-40. However, since our abstraction cannot represent such constants, their lattices are set to . Lines 44-60 manage assignments and the lattices of the left-hand-side expression and the assignment SgAssignOp itself are set to be equal to the lattice of the right-hand-side expression. The code for variable declaration (lines 63-81) and initialization (lines 84-98) are similar in that the lattice of the right-hand-side is copied to the lattice of the initialized variable. Finally, lines 101-127 focus on arithmetic operations. If the lattices of the left- and right-hand-side expressions are both at levels valKnown,the operation is performed immediately by the analysis on their statically known values and the result is stored in the lattices of the left-hand-side expression and the SgExpression itself. Finally,

124

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

on line 129 the function returns the modied variable, which keeps track of whether the state of the downstream lattices has changed. Since these lattices are inputs to other SgExpressions, this informs the dataow framework whether it needs to analyze how these lattices are transferred by those expressions.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

b o o l c o n s t P r o p A n a l y s i s : : t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r < L a t t i c e >& d f I n f o ) { b o o l m o d i f i e d=f a l s e ; // Get t h e l a t t i c e o b j e c t F i n i t e V a r s E x p r s P r o d u c t L a t t i c e prodLat = d y n a m i c c a s t <F i n i t e V a r s E x p r s P r o d u c t L a t t i c e > ( ( d f I n f o . b e g i n ( ) ) ) ; // Make s u r e t h a t a l l t h e non c o n s t a n t L a t t i c e s a r e i n i t i a l i z e d c o n s t v e c t o r < L a t t i c e >& l a t t i c e s = prodLat > g e t L a t t i c e s ( ) ; f o r ( v e c t o r < L a t t i c e > :: c o n s t i t e r a t o r i t = l a t t i c e s . b e g i n ( ) ; i t != l a t t i c e s . end ( ) ; i t ++) ( d y n a m i c c a s t <n o d e C o n s t L a t t i c e > ( i t ))> i n i t i a l i z e ( ) ; // I n t e g r a l Numeric C o n s t a n t s i f ( isSgLongLongIntVal ( n . getNode ( ) ) || // Other t y p e s o f i n t e g r a l c o n s t a n t s ...) { // Memory o b j e c t and l a t t i c e o f t h e e x p r e s s i o n s r e s u l t varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; constPropLat r e s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; // I f t h e r e s u l t e x p r e s s i o n i s l i v e i f ( resLat ) { i f ( isSgLongLongIntVal ( n . getNode ( ) ) ) m o d i f i e d = r e s L a t > s e t ( is SgLongLongIntVal ( n . getNode ()) > g e t v a l u e ( ) ) | | modified ; // Same f o r o t h e r t y p e s o f i n t e g r a l c o n s t a n t s ... } // Non i n t e g r a l C o ns t a nt s } e l s e i f ( isSgValueExp ( n . getNode ( ) ) ) { // Memory o b j e c t and l a t t i c e o f t h e e x p r e s s i o n s r e s u l t varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; constPropLat r e s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; // I f t h e r e s u l t e x p r e s s i o n i s l i v e , s e t i t t o top s i n c e we o n l y work // with i n t e g r a l c o n s t a n t s i f ( r e s L a t ) m o d i f i e d = r e s L a t >setTop ( ) | | m o d i f i e d ;

18.2. ROSE DATAFLOW FRAMEWORK 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86

125

// P l a i n a s s i g n m e n t : l h s = r h s } e l s e i f ( isSgAssignOp ( n . getNode ( ) ) ) { // Memory o b j e c t s denoted by t h e e x p r e s s i o n s l e f t and r i g h t hand // s i d e s a s w e l l a s t h e SgAssignOp i t s e l f varID l h s = SgExpr2Var ( isSgAssignOp ( n . getNode ()) > g e t l h s o p e r a n d ( ) ) ; varID r h s = SgExpr2Var ( isSgAssignOp ( n . getNode ()) > g e t r h s o p e r a n d ( ) ) ; varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; // The l a t t i c e s a s s o c i a t e d t h e t h r e e memory o b j e c t s constPropLat r e s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; constPropLat l h s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( l h s ) ) ; constPropLat r h s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r h s ) ) ; // I f t h e l h s and/ o r t h e SgAssignOp a r e l i v e , copy l a t t i c e from t h e r h s i f ( l h s L a t ) { l h s L a t >copy ( r h s L a t ) ; m o d i f i e d = t r u e ; } i f ( r e s L a t ) { r e s L a t >copy ( r h s L a t ) ; m o d i f i e d = t r u e ; } // V a r i a b l e D e c l a r a t i o n } e l s e i f ( i s S g I n i t i a l i z e d N a m e ( n . getNode ( ) ) ) { varID var ( i s S g I n i t i a l i z e d N a m e ( n . getNode ( ) ) ) ; constPropLat varLat = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( var ) ) ; // I f t h i s v a r i a b l e i s l i v e i f ( varLat ) { // I f t h e r e was no i n i t i a l i z e r , i n i t i a l i z e i t s l a t t i c e t o Bottom i f ( initName > g e t i n i t i a l i z e r ()==NULL) m o d i f i e d = varLat >s e t B o t ( ) | | m o d i f i e d ; // Otherwise , copy t h e l a t t i c e o f t h e i n i t i a l i z e r t o t h e v a r i a b l e else { varID i n i t = SgExpr2Var ( i s S g I n i t i a l i z e d N a m e ( n . getNode ()) > g e t i n i t i a l i z e r ( ) ) ; ConstPropLat i n i t L a t = d y n a m i c c a s t <ConstPropLat >( prodLat > g e t V a r L a t t i c e ( i n i t ) ) ; i f ( i n i t L a t ) { varLat >copy ( i n i t L a t ) ; m o d i f i e d = t r u e ; } } } // I n i t i a l i z e r f o r a v a r i a b l e } e l s e i f ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ( ) ) ) { // Memory o b j e c t s o f t h e i n i t i a l i z e d v a r i a b l e and t h e // i n i t i a l i z a t i o n e x p r e s s i o n

126 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK varID r e s = SgExpr2Var ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ( ) ) ) ; varID asgn = SgExpr2Var ( i s S g A s s i g n I n i t i a l i z e r ( n . getNode ()) > g e t o p e r a n d ( ) ) ; // The l a t t i c e s a s s o c i a t e d both memory o b j e c t s constPropLat r e s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; constPropLat asgnLat = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( asgn ) ) ; // I f t h e v a r i a b l e i s l i v e , copy l a t t i c e from t h e a s s i g n m e n t i f ( r e s L a t ) { r e s L a t >copy ( asgnLat ) ; m o d i f i e d = t r u e ; }

// += A r i t h m e t i c O p e r a t i o n } e l s e i f ( i s S g P l u s A s s i g n O p ( n . getNode ( ) ) ) { // Memory o b j e c t s denoted by t h e e x p r e s s i o n s l e f t and r i g h t hand // s i d e s a s w e l l a s t h e SgAssignOp i t s e l f varID l h s = SgExpr2Var ( isSgAssignOp ( n . getNode ()) > g e t l h s o p e r a n d ( ) ) ; varID r h s = SgExpr2Var ( isSgAssignOp ( n . getNode ()) > g e t r h s o p e r a n d ( ) ) ; varID r e s = SgExpr2Var ( i s S g E x p r e s s i o n ( n . getNode ( ) ) ) ; // The l a t t i c e s a s s o c i a t e d t h e t h r e e memory o b j e c t s constPropLat r e s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; constPropLat l h s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( l h s ) ) ; constPropLat r h s L a t = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r h s ) ) ; // I f t h e l h s and/ o r t h e SgAssignOp a r e l i v e and we know both t h e i r // v a l u e s o f t h e v a l u e o f t h e r h s e x p r e s s i o n , s e t t h e i r l a t t i c e t o be t h e // sum o f t h e two . i f ( l h s L a t && l h s L a t > l e v e l==constPropLat : : valKnown && rhsLat > l e v e l==constPropLat : : valKnown ) { m o d i f i e d = l h s L a t > s e t ( l h s L a t >v a l u e + rhsLat >v a l u e ) | | m o d i f i e d ; } i f ( r e s L a t && r e s L a t > l e v e l==constPropLat : : valKnown && rhsLat > l e v e l==constPropLat : : valKnown ) { m o d i f i e d = r e s L a t > s e t ( r e s L a t >v a l u e + rhsLat >v a l u e ) | | m o d i f i e d ; } } // Same f o r o t h e r a r i t h m e t i c o p e r a t i o n s ... return modified ; } Once the intra-procedural analysis has been implemented, it can be executed on the application

18.2. ROSE DATAFLOW FRAMEWORK

127

by combining it with an inter-procedural analysis. Currently two such analyses are implemented. ContextInsensitiveInterProceduralDataow implements a basic context-insensitive analysis that propagates abstract state from callers to callees but does not dierentiate between dierent call sites of the same function. As such, it is sensitive to inter-procedural data ows but can be imprecise because it takes into account control ows that are actually impossible, such as entering a function from one call site but returning to another. The code below provides an example of how this analysis is used to create an inter-procedural constant propagation analysis. The dataow framework is initialized on line 6 and the applications call graph is built on lines 9-11. The intra-procedural analysis object is created on line 17 and the context-insensitive inter-procedural analysis is created on line 20. The user passes into its constructor references to their intra-procedural analysis and the call graph. Finally, on line 23 the user applies the full inter-procedural analysis to the entire application.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

i n t main ( i n t argc , c h a r argv [ ] ) { // B u i l d t h e AST used by ROSE S g P r o j e c t p r o j e c t = f r o n t e n d ( argc , argv ) ; // I n i t i a l i z e t h e ROSE d a t a f l o w framework initAnalysis ( project ); // B u i l d t h e c a l l graph C a l l G r a p h B u i l d e r cgb ( p r o j e c t ) ; cgb . b u i l d C a l l G r a p h ( ) ; S g I n c i d e n c e D i r e c t e d G r a p h graph = cgb . getGraph ( ) ; // S e t t h e debug l e v e l t o p r i n t t h e p r o g r e s s o f t h e d a t a f l o w a n a l y s i s analysisDebugLevel = 1; // C r e a t e t h e i n t r a p r o c e d u r a l c o n s t a n t p r o p a g a t i o n a n a l y s i s c o n s t P r o p A n a l y s i s cp ( p r o j e c t ) ; // C r e a t e t h e i n t e r p r o c e d u r a l a n a l y s i s f o r i n t r a a n a l y s i s cp C o n t e x t I n s e n s i t i v e I n t e r P r o c e d u r a l D a t a f l o w i n t e r c p (&cp , graph ) ; // Run i n t e r p r o c e d u r a l c o n s t a n t p r o p a g a t i o n on t h e e n t i r e a p p l i c a t i o n inte r c p . runAnalysis ( ) ; }

To simplify debugging the framework also provides the UnstructuredPassInterDataow analysis, which simply applies the users intra-procedural analysis on each function within the application. While this produces globally incorrect results, it simplies debugging analyses on individual functions.

128

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

18.2.4

Transferring Information Between Analyses

Since in practice users need to implement multiple analyses where one depends on the results of another, the ROSE dataow framework maintains the results of all analyses at each CFG nodes and makes it easy for analyses to access this data. The lattices and facts of a given CFG node are stored in its associated NodeState object. The data produced by an analysis can be retrieved by using its pointer, as shown in the example below. This code shows analysis exAnalysis, which takes in its constructor a pointer to the constPropAnalysis described above (lines 4-5). Inside its transfer function this analysis calls the getLatticeBelow method of its argument state (instance of the NodeState class) to get the lattice associated with constPropAnalysis that has index 0 (lines 11-13). It then gets the constPropLat of any variable it cares about and make analysis decisions based on what is statically known about its state. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 c l a s s exAnalysis { // C l a s s m a i n t a i n s a p o i n t e r t o t h e c o n s t a n t p r o p a g a t i o n a n a l y s i s t o make // i t p o s s i b l e t o a c c e s s i t s r e s u l t s c o n s t P r o p A n a l y s i s& c p A n a l y s i s ; e x A n a l y s i s ( c o n s t P r o p A n a l y s i s c p A n a l y s i s ) : c p A n a l y s i s ( c p A n a l y s i s ) {} b o o l t r a n s f e r ( c o n s t Function& func , c o n s t DataflowNode& n , NodeState& s t a t e , c o n s t v e c t o r < L a t t i c e >& d f I n f o ) { // Get t h e L a t t i c e s computed by t h e c o n s t a n t p r o p a g a t i o n a n a l y s i s f o r t h e // c u r r e n t CFG node F i n i t e V a r s E x p r s P r o d u c t L a t t i c e prodLat = d y n a m i c c a s t <F i n i t e V a r s E x p r s P r o d u c t L a t t i c e >( s t a t e > g e t L a t t i c e B e l o w ( c p A n a l y s i s , 0 ) ) ; // Some a p p l i c a t i o n v a r i a b l e o f i n t e r e s t varID var = . . . ; // The constPropLat o f t h i s v a r i a b l e constPropLat varCPLat = d y n a m i c c a s t <constPropLat >( prodLat > g e t V a r L a t t i c e ( r e s ) ) ; // Analyze d i f f e r e n t l y depending on what i s known about t h e // v a r i a b l e s v a l u e i f ( varCPLat ) i f ( varCPLat> l e v e l == constPropLat : : bottom ) { ... } e l s e i f ( varCPLat> l e v e l == constPropLat : : valKnown ) { ... } e l s e i f ( varCPLat> l e v e l == constPropLat : : top ) { ... } } ...

18.2. ROSE DATAFLOW FRAMEWORK 34 };

129

The code below shows the full functionality of the NodeState class. . Lines 5-16 show the functions to set, get and delete the lattices above and below the associated CFG node. Lines 20-26 provide the same functionality for facts. The str method on line 20 returns a string representation of the lattices and facts associated with the CFG node, which is very useful for debugging. Lines 37-50 show the objects static methods. The getNodeState method on line 37 returns the NodeState object of a given CFG node. Since the ROSE virtual CFG can have multiple CFG nodes for the same AST node, this method requires an additional index argument to identify the node in question. Finally, method copyLattices aEQa and related methods (lines 39-50) copy lattice information from above a CFG node to below it and vice versa, from one node to another or from one analysis at a given node to another analysis at the same node. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 c l a s s NodeState { // S e t s t h e l a t t i c e s above / below t h i s node f o r t h e g i v e n a n a l y s i s t o t h e // g i v e n l a t t i c e v e c t o r v o i d s e t L a t t i c e A b o v e ( c o n s t A n a l y s i s a n a l y s i s , v e c t o r < L a t t i c e >& l a t t i c e s ) ; v o i d s e t L a t t i c e B e l o w ( c o n s t A n a l y s i s a n a l y s i s , v e c t o r < L a t t i c e >& l a t t i c e s ) ; // Returns t h e l a t t i c e l a t t i c e N a m e g e n e r a t e d by t h e g i v e n a n a l y s i s from // above / below t h e node Lattice getLatticeAbove ( const Analysis analysis , int latticeName ) const ; Lattice getLatticeBelow ( const Analysis analysis , int latticeName ) const ; // D e l e t e s a l l l a t t i c e s above / below t h i s node t h a t a r e a s s o c i a t e d with t h e // g i v e n a n a l y s i s void deleteLatticeAbove ( const Analysis a n a l y s i s ) ; void deleteLatticeBelow ( const Analysis a n a l y s i s ) ; // S e t s t h e f a c t s a t t h i s node f o r t h e g i v e n a n a l y s i s t o t h e g i v e n // f a c t v e c t o r v o i d s e t F a c t s ( c o n s t A n a l y s i s a n a l y s i s , c o n s t v e c t o r <NodeFact>& newFacts ) ; // Returns t h e g i v e n f a c t , owned by t h e g i v e n a n a l y s i s NodeFact g e t F a c t ( c o n s t A n a l y s i s a n a l y s i s , i n t factName ) c o n s t ; // D e l e t e s a l l f a c t s a t t h i s node a s s o c i a t e d with t h e g i v e n a n a l y s i s void deleteFacts ( const Analysis a n a l y s i s ) ; // Returns a s t r i n g r e p r e s e n t a t i o n o f a l l t h e l a t t i c e s and f a c t s // a s s o c i a t e d with t h e CFG node s t r i n g s t r ( Analysis analysis , s t r i n g indent ) const ; // S t a t i c Methods // Returns t h e NodeState o b j e c t a s s o c i a t e d with t h e g i v e n d a t a f l o w node

130

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.5: Example of Transformation on the CFG 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 // i n d e x i s used when m u l t i p l e NodeState o b j e c t s a r e a s s o c i a t e d with a // g i v e n node // ( ex : SgFunctionCallExp has 3 NodeStates : entry , f u n c t i o n body , e x i t ) s t a t i c NodeState g e t N o d e S t a t e ( c o n s t DataflowNode& n , i n t i n d e x =0); // C o p i e s from s above l a t t i c e s f o r t h e g i v e n a n a l y s i s t o to s above // l a t t i c e s f o r t h e same a n a l y s i s s t a t i c v o i d c o p y L a t t i c e s a E Q a ( A n a l y s i s a n a l y s i s , NodeState& to , c o n s t NodeState& from ) ; // C o p i e s from s above l a t t i c e s f o r a n a l y s i s A t o to s above l a t t i c e s f o r // a n a l y s i s B s t a t i c v o i d c o p y L a t t i c e s a E Q a ( A n a l y s i s a n a l y s i s A , NodeState& to , A n a l y s i s a n a l y s i s B , c o n s t NodeState& from ) ; // S i m i l a r methods f o r c o p y i n g i n d i f f e r e n t p e r m u t a t i o n s ... };

18.2.5

CFG Transformations

ROSE makes it easy to modify the applications AST as a result of dataow analyses. The dataow framework maintains an on-demand normal form, where analyses can focus on the actions of individual SgNodes and ignore how they are arranged within the AST. ROSE maintains this abstraction by providing an API that inserts new SgExpressions into the applications CFG, making all the needed changes in the AST to make sure that the correct control ow is maintained. To get the intuition of this functionality consider the expression foo()=(bar()+baz()). Suppose the user has decided based on the dataow state before the SgAddOp + that it they want to add a call to function print immediately before it. From the perspective of the CFG, this is a simple and well-dened operation, as shown at the top of Figure 4. The side-eects of the calls to bar and baz must complete before the call to print and the side-eects of print must complete before the execution of the + operation. The call to foo is not well-ordered relative print or the other

18.2. ROSE DATAFLOW FRAMEWORK

131

Figure 18.6: Example of the Transformation on the Source Code operations by the structure of the CFG. Unfortunately, it is dicult to implement these semantics in the context of the AST because (i) there is no way to add a function call to an SgAddOp and (ii) because in C++ the sequence points required by the above semantics (some side-eects much complete before others) are provided by a few specic constructs such as statement boundaries and the comma operator. As such, the transformation requires the complex set of AST changes shown in the Figure 5. We must create temporary variables to hold the results of the calls to bar and baz. We then transform the original SgAddOp into a longer SgCommaOpExp, where we rst call bar and baz, saving their results into the temporary variables, then call print and nally perform the addition. The result of the addition is the result of the entire comma expression, so this transformation correctly enforces the semantics of the desired transformation. ROSE provides the three functions to make it easy to insert expressions into the CFG. Functions insertBeforeUsingCommaOp and insertAfterUsingCommaOp insert SgExpressions before or after existing SgExpressions using a generalization of the transformation described in Figure 18.6. 1 // I n s e r t an e x p r e s s i o n ( new exp ) b e f o r e a n o t h e r e x p r e s s i o n ( a n c h o r e x p ) has 2 // p o s s i b l e s i d e e f f e c t s , w i t h o u t c h a n g i n g t h e o r i g i n a l s e m a n t i c s . This i s 3 // a c h i e v e d by u s i n g a comma o p e r a t o r : ( new exp , a n c h o r e x p ) . The comma 4 // o p e r a t o r i s r e t u r n e d . 5 SgCommaOpExp insertBeforeUsingCommaOp ( S g E x p r e s s i o n new exp , 6 SgExpression anchor exp ) ; 7 // I n s e r t an e x p r e s s i o n ( new exp ) a f t e r a n o t h e r e x p r e s s i o n ( a n c h o r e x p ) has 8 // p o s s i b l e s i d e e f f e c t s , w i t h o u t c h a n g i n g t h e o r i g i n a l s e m a n t i c s . This i s 9 // done by u s i n g two comma o p e r a t o r s : 10 // type T1 ; . . . ( ( T1 = anchor exp , new exp ) , T1) ) . . . , 11 // where T1 i s a temp v a r i a b l e s a v i n g t h e p o s s i b l e s i d e e f f e c t o f a n c h o r e x p . 12 // The top l e v e l comma op exp i s r e t u r n e d . The r e f e r e n c e t o T1 i n T1 = 13 // a n c h o r e x p i s saved i n t e m p r e f . 14 SgCommaOpExp insertAfterUsingCommaOp ( S g E x p r e s s i o n new exp , 15 S g E x p r e s s i o n anchor exp , SgStatement t e m p d e c l = NULL, 16 SgVarRefExp t e m p r e f = NULL ) ; Function replaceWithPattern (Figure 18.2.5) replaces one SgExpression with another. However, since the original expression may still be valuable, it allows the original expression to be

132

CHAPTER 18. GENERIC DATAFLOW ANALYSIS FRAMEWORK

Figure 18.7: Code Replacement Transformation included at one or more locations inside the new expression that contain nodes of type SgVariantExpression. 1 2 3 4 // Re pla c e an anchor node with a s p e c i f i e d p a t t e r n s u b t r e e with o p t i o n a l // S g V a r i a n t E x p r e s s i o n . A l l S g V a r i a n t E x p r e s s i o n i n t h e p a t t e r n w i l l be // r e p l a c e d with c o p i e s o f t h e anchor node . SgNode r e p l a c e W i t h P a t t e r n ( SgNode anchor , SgNode n e w p a t t e r n ) ; An example of this transformation is shown Figure 18.7, where the original code is the same as in the example above and the new pattern expression is a single SgMultOp where the arguments are both SgVariantExpressions. The result of the transformation is that the original SgAddOp is replaced with a multiplication the arguments of which are copies of the SgAddOp: (bar()+baz())*(bar()+baz()).

Chapter 19

Recognizing Loops
Figures 19.1 and 19.2 show a translator which reads an application and gathers a list of loop nests. At the end of the traversal it reports information about each loop nest, including the function where it occurred and the depth of the loop nest. FIXME: This example program Using this translator we can compile the code shown in gure 19.3. The output is shown in is unnished. It will output a list of objects representing gure 19.4. information about perfectly
nested loops.

133

134

CHAPTER 19. RECOGNIZING LOOPS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

// ROSE i s // r o s e . C : #i n c l u d e class {

a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an e x a m p l e p r e p r o c e s s o r b u i l t Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e

w i t h ROSE .

r o s e . h

InheritedAttribute public : int

loopNestDepth ; ( ) : loopNestDepth ( 0 ) {}; ( const InheritedAttribute & X )

InheritedAttribute InheritedAttribute }; class { SynthesizedAttribute

{};

public : SynthesizedAttribute () }; class { Traversal : public

{};

SgTopDownBottomUpProcessing< I n h e r i t e d A t t r i b u t e , S y n t h e s i z e d A t t r i b u t e >

public : // F u n c t i o n s r e q u i r e d InheritedAttribute evaluateInheritedAttribute SgNode a s t N o d e , InheritedAttribute inheritedAttribute );

SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute , SubTreeSynthesizedAttributes synthesizedAttributeList }; InheritedAttribute Traversal : : evaluateInheritedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute ) { s w i t c h ( a s t N o d e >v a r i a n t T ( ) ) { c a s e V SgForStatement : { p r i n t f ( Found a S g F o r S t a t e m e n t \ n ) ; // This l o o p i s one d e e p p e r than t h e depth i n h e r i t e d A t t r i b u t e . l o o p N e s t D e p t h ++; break ; } default : { // g++ n e e d s } } return } SynthesizedAttribute Traversal : : evaluateSynthesizedAttribute ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute , inheritedAttribute ; of the

);

parent s

inherited

attribute

block

here

Figure 19.1: Example source code showing loop recognition (part 1).

135

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

SubTreeSynthesizedAttributes { SynthesizedAttribute

synthesizedAttributeList

returnAttribute ;

s w i t c h ( a s t N o d e >v a r i a n t T ( ) ) { c a s e V SgForStatement : { break ; } default : { // g++ n e e d s } } return } returnAttribute ;

block

here

int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; // Build the i n h e r i t e d a t t r i b u t e InheritedAttribute inheritedAttribute ; Traversal // myTraversal ;

C a l l t h e t r a v e r s a l s t a r t i n g a t t h e s a g e P r o j e c t node o f t h e AST myTraversal . t r a v e r s e I n p u t F i l e s ( p r o j e c t , i n h e r i t e d A t t r i b u t e ) ; return 0;

Figure 19.2: Example source code showing loop recognition (part 2).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

i n t main ( ) { int x [ 4 ] ; int y [ 4 ] [ 4 ] ; f o r ( i n t i =0; i < 4 ; { x [ i ] = 7; } i ++)

f o r ( i n t i =0; i < 4 ; i ++) { f o r ( i n t j =0; j < 4 ; j ++) { y [ i ] [ j ] = 42; } } return 0; }

Figure 19.3: Example source code used as input to loop recognition processor.

136

CHAPTER 19. RECOGNIZING LOOPS

1 2 3

Found a Sg Fo r Sta t ement Found a Sg Fo r Sta t ement Found a Sg Fo r Sta t ement

Figure 19.4: Output of input to loop recognition processor.

Chapter 20

Virtual CFG
The ROSE virtual control ow graph interface provides a higher level of detail than ROSEs other control ow graph interfaces. It expresses control ow even within expressions, and handles shortcircuited logical and conditional operators properly1 . The interface is referred to as virtual because no explicit graph is ever created: only the particular CFG nodes and edges used in a given program ever exist. CFG nodes and edges are value classes (they are copied around by value, reducing the need for explicit memory management). A CFG node consists of two components: an AST node pointer, and an index of a particular CFG node within that AST node. There can be several CFG nodes corresponding to a given AST node, and thus the AST node pointers cannot be used to index CFG nodes. The particular index values for the dierent AST node types are explained in Section 20.1.

20.1

CFGNode Index values

To facilitate traversal and represent sucient details, each eligible ROSE AST node (expression, statement and SgInitializedName) has several corresponding CFGNodes in virtual CFG. These CFGNodes have indices from 0 to n. CFGNode of index value of 0 is used to represent the beginning CFG node for an AST node, while the CFGNode of index n is the end CFGNode for the AST node. The beginning node represents the point in the control ow immediately before the construct starts to execute, and the ending node represents the point immediately after the construct has nished executing. Note that these two nodes do not dominate the other CFG nodes in the construct due to goto statements and labels. Reimplementation of S gNode::cfgIndexForEnd() returns the index value for n of each eligible SgNode type. See source le src/frontend/SageIII/virtualCFG/memberFunctions.C for valid index values for each type of eligible SgNode.
1 It assumes operands of expressions are computed in left-to-right order, unlike the actual language semantics, however.

137

138

CHAPTER 20. VIRTUAL CFG

20.2

Important functions

The main body of the virtual CFG interface is in virtualCFG.h; the source code is in src/frontend/SageIII/virtualCFG/ and is linked into librose . The ltered CFG interface explained below is in lteredCFG.h, and functions for converting the CFG to a graph in Dot format are in cfgToDot.h. Two functions provide the basic way of converting from AST nodes to CFG nodes. Each SgNode has two methods, cfgForBeginning() and cfgForEnd(), to generate the corresponding beginning and end CFG nodes, respectively. These functions require that the AST node is either an expression, a statement, or a SgInitializedName.

20.2.1

Node methods

CFGNode(SgNode node, unsigned int index): Build a CFG node from the given AST node and index. Valid index values are in Section 20.1. toString(): Produce a string representing the information in the node. toStringForDebugging(): Similar, but with more internal debugging information. id(): A C identier representing the node. getNode(): Get the underlying AST node. getIndex(): Get the index (as explained in Section 20.1) for this CFG node within its underlying AST node. outEdges(): Return a vector of outgoing CFG edges from this node. This function internally calls cfgOutEdges(unsigned int idx) to generate out edges for each CFGNode of a given index value. inEdges(): Return a vector of CFG edges coming into this node (note that the sources and targets of the edges are not reversed, and so each in edge has its target as the current node). This function internally calls cfgInEdges(unsigned int idx) to generate in edges for each CFGNode of a given index value. isInteresting (): See Section 20.6.1. Nodes are also comparable using the operators ==, !=, and <.

20.2.2

Edge methods

toString(): Produce a string representing the information in the node. toStringForDebugging(): Similar, but with more internal debugging information. id(): A C identier representing the node. source(): The starting CFG node for this edge. target (): The ending CFG node for this edge.

20.3. DRAWING A GRAPH OF THE CFG

139

condition(): When there are multiple CFG edges from the same starting node, each of them is taken under certain conditions. The condition() method returns the condition, of type EdgeConditionKind. The possible return values are: eckUnconditional: An edge that is always taken. eckTrue: True case of a two-way branch (either an if statement or a loop eckFalse: False case of a two-way branch eckCaseLabel: Case label in a switch statement (key is given by caseLabel()) eckDefault: Default label of a switch statement eckDoConditionPassed: Enter Fortran do loop body eckDoConditionFailed: Fortran do loop nished eckForallIndicesInRange: Start testing forall mask eckForallIndicesNotInRange: End of forall loop eckComputedGotoCaseLabel: Case in computed goto number needs to be computed separately eckArithmeticIfLess: Edge for the arithmetic if expression being less than zero eckArithmeticIfEqual: Edge for the arithmetic if expression being equal to zero eckArithmeticIfGreater: Edge for the arithmetic if expression being greater than zero caseLabel(): For an edge with condition eckCaseLabel, an expression representing the key for the case label. computedGotoCaseIndex(): The index of this edges case within a Fortran computed goto (an edge of kind eckComputedGotoCaseLabel). conditionBasedOn(): The test expression or switch expression that is tested by this edge. scopesBeingExited(), scopesBeingEntered(): Variables leaving and entering scope during this edge. This information has not been extensively veried, and should not be relied upon. Edges can also be compared using the operators == and !=. They are not ordered to avoid dependencies on pointer comparison on dierent computers.

20.3

Drawing a graph of the CFG

Fig. 20.3 shows a translator to dump full (debug) virtual control ow graphs for all functions within input source les. It also dumps a simplied version (interesting) version of virtual control ow graphs. A standalone tool named virtualCFG is installed under ROSE INSTALL TREE/bin for users to generate both debug and interesting dot les of virtual CFGs. The example input code is given in Fig. 20.3. Debug and interesting virtualCFG of function main() are shown in Fig. 20.3 and Fig. 20.4, respectively. Debug and interesting virtualCFG of function testIf () are shown in Fig. 20.5 and Fig. 20.6, respectively.

140

CHAPTER 20. VIRTUAL CFG

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

// Example t r a n s l a t o r t o g e n e r a t e d o t #i n c l u d e r o s e . h #i n c l u d e < s t r i n g > u s i n g namespace s t d ;

files

of

virtual

c o n t r o l flow graphs

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject s a g e P r o j e c t = fro ntend ( argc , argv ) ; // P r o c e s s a l l f u n c t i o n d e f i n i t i o n b o d i e s f o r v i r t u a l c o n t r o l f l o w graph g e n e r a t i o n R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n s = NodeQuery : : querySubTree ( s a g e P r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : c o n s t i t e r a t o r i = f u n c t i o n s . b e g i n ( ) ; i != f u n c t i o n s . end ( ) ; ++i ) { SgFunctionDefinition proc = i s S g F u n c t i o n D e f i n i t i o n ( i ) ; ROSE ASSERT ( p r o c != NULL ) ; s t r i n g f i l e N a m e= S t r i n g U t i l i t y : : st r i p Pa t hF r o m Fi l eN a m e ( proc > g e t f i l e i n f o ()> g e t f i l e n a m e S t r i n g ( ) ) ; s t r i n g dotFileName1=f i l e N a m e +.+ proc > g e t d e c l a r a t i o n ()> get name ( ) +. debug . d o t ; s t r i n g dotFileName2=f i l e N a m e +.+ proc > g e t d e c l a r a t i o n ()> get name ( ) +. i n t e r e s t i n g . do t ; // Dump o u t t h e f u l l CFG, i n c l u d i n g b o o k k e e p i n g n o d e s VirtualCFG : : cfgToDotForDebugging ( proc , dotFileName1 ) ; // Dump o u t o n l y t h o s e n o d e s which a r e i n t e r e s t i n g f o r VirtualCFG : : i n t e r e s t i n g C f g T o D o t ( proc , dotFileName2 ) ; } return 0; } analyses

Figure 20.1: Example source code showing visualization of virtual control ow graph. As we can see in Fig. 20.5, the debug dot graph has all CFGNodes for each eligible SgNode. For example, there are three CFGNodes for SgIfStmt, with indices from 0 to 2. The interesting CFGNode of SgIfStmt has solid line oval while non-essential CFGNodes have dashed line ovals in the graph. The caption of each node has a format of <SgNode-Enum-value>@linenumber:CFGNode-index-value. It is obvious from the graph that SgIfStmts interesting CFGNode has an index value of 1. In comparison, Fig. 20.6 only shows interesting CFGNodes with solid line ovals. Again, the captions tells line numbers and CFGNodes index values for each CFGNode.

20.3. DRAWING A GRAPH OF THE CFG

141

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

#i n c l u d e < s t d i o . h> #i n c l u d e < s t r i n g . h> #i n c l u d e < a s s e r t . h> size t i ; char b u f f e r [ 1 0 ] ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { f o r ( i =0; i < s t r l e n ( a r g v [ 1 ] ) ; { b u f f e r [ i ] = argv [ 1 ] [ i ] ; } return 0; } int t e s t I f ( int i ) { int rt ; i f ( i %2 ==0) r t =0; else r t =1; return rt ; }

i ++)

Figure 20.2: Example source code used as input to build virtual control graphs.

142

CHAPTER 20. VIRTUAL CFG

Start(::main) <SgFunctionDefinition> @line=7 :idx=0

main_parameter_list_ <SgFunctionParameterList> @line=7 :idx=0

initialized_name_argc <SgInitializedName> argc :idx=0

main_parameter_list_ <SgFunctionParameterList> @line=7 :idx=1

initialized_name_argv <SgInitializedName> argv :idx=0

main_parameter_list_ <SgFunctionParameterList> @line=7 :idx=2

After parameters(::main) <SgFunctionDefinition> @line=7 :idx=1

After pre-initialization(::main) <SgFunctionDefinition> @line=7 :idx=2

0x2b01cdc8b010 <SgBasicBlock> @line=8 :idx=0

0x2b01cdcce010 <SgForStatement> @line=9 :idx=0

SgForInitStatement <SgForInitStatement> @line=9 :idx=0

SgExprStatement <SgExprStatement> @line=9 :idx=0

SgAssignOp_undef_name <SgAssignOp> @line=9 :idx=0

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

SgAssignOp_undef_name <SgAssignOp> @line=9 :idx=1

SgCastExp_undef_name <SgCastExp> @line=0 :idx=0

integer_value_exp_0 <SgIntVal> @line=9 :idx=0

integer_value_exp_0 <SgIntVal> @line=9 :idx=1

SgCastExp_undef_name <SgCastExp> @line=0 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=9 :idx=2

SgExprStatement <SgExprStatement> @line=9 :idx=1

SgForInitStatement <SgForInitStatement> @line=9 :idx=1

0x2b01cdcce010 <SgForStatement> @line=9 :idx=1

SgExprStatement <SgExprStatement> @line=9 :idx=0

SgLessThanOp_undef_name <SgLessThanOp> @line=9 :idx=0

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

SgLessThanOp_undef_name <SgLessThanOp> @line=9 :idx=1

function_call_function_ref_strlen <SgFunctionCallExp> @line=9 :idx=0

function_ref_strlen <SgFunctionRefExp> @line=9 :idx=0

function_call_function_ref_strlen <SgFunctionCallExp> @line=9 :idx=1

expr_list_exp_SgCastExp_undef_name <SgExprListExp> @line=9 :idx=0

SgCastExp_undef_name <SgCastExp> @line=0 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=9 :idx=0

var_ref_of_argv <SgVarRefExp> @line=9 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=9 :idx=1

integer_value_exp_1 <SgIntVal> @line=9 :idx=0

integer_value_exp_1 <SgIntVal> @line=9 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=9 :idx=2

SgCastExp_undef_name <SgCastExp> @line=0 :idx=1

expr_list_exp_SgCastExp_undef_name <SgExprListExp> @line=9 :idx=1

function_call_function_ref_strlen <SgFunctionCallExp> @line=9 :idx=2

function_call_function_ref_strlen <SgFunctionCallExp> @line=9 :idx=3

SgLessThanOp_undef_name <SgLessThanOp> @line=9 :idx=2

SgExprStatement <SgExprStatement> @line=9 :idx=1

0x2b01cdcce010 <SgForStatement> @line=9 :idx=2

key(true)

key(false)

0x2b01cdc8b120 <SgBasicBlock> @line=10 :idx=0

0x2b01cdcce010 <SgForStatement> @line=9 :idx=4

SgExprStatement <SgExprStatement> @line=11 :idx=0

0x2b01cdc8b010 <SgBasicBlock> @line=8 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=11 :idx=0

SgReturnStmt <SgReturnStmt> @line=13 :idx=0

array_ref_of_var_ref_of_buffer_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=0

integer_value_exp_0 <SgIntVal> @line=13 :idx=0

var_ref_of_buffer <SgVarRefExp> @line=0 :idx=0

integer_value_exp_0 <SgIntVal> @line=13 :idx=1

array_ref_of_var_ref_of_buffer_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=1

0x2b01cdc8b010 <SgBasicBlock> @line=8 :idx=2

SgReturnStmt <SgReturnStmt> @line=13 :idx=1

var_ref_of_i <SgVarRefExp> @line=11 :idx=0

End(::main) <SgFunctionDefinition> @line=7 :idx=3

array_ref_of_var_ref_of_buffer_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=2

SgAssignOp_undef_name <SgAssignOp> @line=11 :idx=1

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=11 :idx=0

var_ref_of_argv <SgVarRefExp> @line=11 :idx=0

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=11 :idx=1

integer_value_exp_1 <SgIntVal> @line=11 :idx=0

integer_value_exp_1 <SgIntVal> @line=11 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=11 :idx=2

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=1

var_ref_of_i <SgVarRefExp> @line=11 :idx=0

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=2

SgAssignOp_undef_name <SgAssignOp> @line=11 :idx=2

SgExprStatement <SgExprStatement> @line=11 :idx=1

0x2b01cdc8b120 <SgBasicBlock> @line=10 :idx=1

0x2b01cdcce010 <SgForStatement> @line=9 :idx=3

SgPlusPlusOp_undef_name <SgPlusPlusOp> @line=9 :idx=0

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

SgPlusPlusOp_undef_name <SgPlusPlusOp> @line=9 :idx=1

Figure 20.3: The debug virtual control ow graph for function main() shows all virtual CFG nodes and edges

20.3. DRAWING A GRAPH OF THE CFG

Start(::main) <SgFunctionDefinition> @line=7 :idx=0

143

initialized_name_argc <SgInitializedName> argc :idx=0

initialized_name_argv <SgInitializedName> argv :idx=0

main_parameter_list_ <SgFunctionParameterList> @line=7 :idx=2

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

integer_value_exp_0 <SgIntVal> @line=9 :idx=1

SgCastExp_undef_name <SgCastExp> @line=0 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=9 :idx=2

SgExprStatement <SgExprStatement> @line=9 :idx=1

SgForInitStatement <SgForInitStatement> @line=9 :idx=1

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

function_ref_strlen <SgFunctionRefExp> @line=9 :idx=0

var_ref_of_argv <SgVarRefExp> @line=9 :idx=0

integer_value_exp_1 <SgIntVal> @line=9 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=9 :idx=2

SgCastExp_undef_name <SgCastExp> @line=0 :idx=1

expr_list_exp_SgCastExp_undef_name <SgExprListExp> @line=9 :idx=1

function_call_function_ref_strlen <SgFunctionCallExp> @line=9 :idx=3

SgLessThanOp_undef_name <SgLessThanOp> @line=9 :idx=2

SgExprStatement <SgExprStatement> @line=9 :idx=1

0x2b01cdcce010 <SgForStatement> @line=9 :idx=2

key(true)

key(false)

var_ref_of_buffer <SgVarRefExp> @line=0 :idx=0

integer_value_exp_0 <SgIntVal> @line=13 :idx=1

var_ref_of_i <SgVarRefExp> @line=11 :idx=0

SgReturnStmt <SgReturnStmt> @line=13 :idx=1

array_ref_of_var_ref_of_buffer_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=2

End(::main) <SgFunctionDefinition> @line=7 :idx=3

var_ref_of_argv <SgVarRefExp> @line=11 :idx=0

integer_value_exp_1 <SgIntVal> @line=11 :idx=1

array_ref_of_var_ref_of_argv_at_integer_value_exp_1 <SgPntrArrRefExp> @line=11 :idx=2

var_ref_of_i <SgVarRefExp> @line=11 :idx=0

array_ref_of_array_ref_of_var_ref_of_argv_at_integer_value_exp_1_at_var_ref_of_i <SgPntrArrRefExp> @line=11 :idx=2

SgAssignOp_undef_name <SgAssignOp> @line=11 :idx=2

SgExprStatement <SgExprStatement> @line=11 :idx=1

var_ref_of_i <SgVarRefExp> @line=9 :idx=0

SgPlusPlusOp_undef_name <SgPlusPlusOp> @line=9 :idx=1

Figure 20.4: The virtual control ow graph for function main() shows only interesting virtual CFG nodes and edges. Each CFGNodes caption tells associated source line number and CFGNode index value (@line-num:index-value)

144

CHAPTER 20. VIRTUAL CFG

Start(::testIf) <SgFunctionDefinition> @line=16 :idx=0

testIf_parameter_list_ <SgFunctionParameterList> @line=16 :idx=0

initialized_name_i <SgInitializedName> i :idx=0

testIf_parameter_list_ <SgFunctionParameterList> @line=16 :idx=1

After parameters(::testIf) <SgFunctionDefinition> @line=16 :idx=1

After pre-initialization(::testIf) <SgFunctionDefinition> @line=16 :idx=2

0x2b01cdc8b230 <SgBasicBlock> @line=17 :idx=0

_variable_declaration_rt <SgVariableDeclaration> @line=18 :idx=0

initialized_name_rt <SgInitializedName> rt :idx=0

_variable_declaration_rt <SgVariableDeclaration> @line=18 :idx=1

0x2b01cdc8b230 <SgBasicBlock> @line=17 :idx=1

0x2b01cdd15010 <SgIfStmt> @line=19 :idx=0

SgExprStatement <SgExprStatement> @line=19 :idx=0

SgEqualityOp_undef_name <SgEqualityOp> @line=19 :idx=0

SgModOp_undef_name <SgModOp> @line=19 :idx=0

var_ref_of_i <SgVarRefExp> @line=19 :idx=0

SgModOp_undef_name <SgModOp> @line=19 :idx=1

integer_value_exp_2 <SgIntVal> @line=19 :idx=0

integer_value_exp_2 <SgIntVal> @line=19 :idx=1

SgModOp_undef_name <SgModOp> @line=19 :idx=2

SgEqualityOp_undef_name <SgEqualityOp> @line=19 :idx=1

integer_value_exp_0 <SgIntVal> @line=19 :idx=0

integer_value_exp_0 <SgIntVal> @line=19 :idx=1

SgEqualityOp_undef_name <SgEqualityOp> @line=19 :idx=2

SgExprStatement <SgExprStatement> @line=19 :idx=1

0x2b01cdd15010 <SgIfStmt> @line=19 :idx=1

key(true)

key(false)

SgExprStatement <SgExprStatement> @line=20 :idx=0

SgExprStatement <SgExprStatement> @line=22 :idx=0

SgAssignOp_undef_name <SgAssignOp> @line=20 :idx=0

SgAssignOp_undef_name <SgAssignOp> @line=22 :idx=0

var_ref_of_rt <SgVarRefExp> @line=20 :idx=0

var_ref_of_rt <SgVarRefExp> @line=22 :idx=0

SgAssignOp_undef_name <SgAssignOp> @line=20 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=22 :idx=1

integer_value_exp_0 <SgIntVal> @line=20 :idx=0

integer_value_exp_1 <SgIntVal> @line=22 :idx=0

integer_value_exp_0 <SgIntVal> @line=20 :idx=1

integer_value_exp_1 <SgIntVal> @line=22 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=20 :idx=2

SgAssignOp_undef_name <SgAssignOp> @line=22 :idx=2

SgExprStatement <SgExprStatement> @line=20 :idx=1

SgExprStatement <SgExprStatement> @line=22 :idx=1

0x2b01cdd15010 <SgIfStmt> @line=19 :idx=2

0x2b01cdc8b230 <SgBasicBlock> @line=17 :idx=2

SgReturnStmt <SgReturnStmt> @line=24 :idx=0

var_ref_of_rt <SgVarRefExp> @line=24 :idx=0

SgReturnStmt <SgReturnStmt> @line=24 :idx=1

0x2b01cdc8b230 <SgBasicBlock> @line=17 :idx=3

End(::testIf) <SgFunctionDefinition> @line=16 :idx=3

Figure 20.5: The debug virtual control ow graph for function testIf() shows all virtual CFG nodes and edges

20.3. DRAWING A GRAPH OF THE CFG

Start(::testIf) <SgFunctionDefinition> @line=16 :idx=0

145

initialized_name_i <SgInitializedName> i :idx=0

testIf_parameter_list_ <SgFunctionParameterList> @line=16 :idx=1

initialized_name_rt <SgInitializedName> rt :idx=0

_variable_declaration_rt <SgVariableDeclaration> @line=18 :idx=1

var_ref_of_i <SgVarRefExp> @line=19 :idx=0

integer_value_exp_2 <SgIntVal> @line=19 :idx=1

SgModOp_undef_name <SgModOp> @line=19 :idx=2

integer_value_exp_0 <SgIntVal> @line=19 :idx=1

SgEqualityOp_undef_name <SgEqualityOp> @line=19 :idx=2

SgExprStatement <SgExprStatement> @line=19 :idx=1

0x2b01cdd15010 <SgIfStmt> @line=19 :idx=1

key(true)

key(false)

var_ref_of_rt <SgVarRefExp> @line=20 :idx=0

var_ref_of_rt <SgVarRefExp> @line=22 :idx=0

integer_value_exp_0 <SgIntVal> @line=20 :idx=1

integer_value_exp_1 <SgIntVal> @line=22 :idx=1

SgAssignOp_undef_name <SgAssignOp> @line=20 :idx=2

SgAssignOp_undef_name <SgAssignOp> @line=22 :idx=2

SgExprStatement <SgExprStatement> @line=20 :idx=1

SgExprStatement <SgExprStatement> @line=22 :idx=1

var_ref_of_rt <SgVarRefExp> @line=24 :idx=0

SgReturnStmt <SgReturnStmt> @line=24 :idx=1

End(::testIf) <SgFunctionDefinition> @line=16 :idx=3

Figure 20.6: The virtual control ow graph for function testIf() shows only interesting virtual CFG nodes and edges. Each CFGNodes caption tells associated source line number and CFGNode index value (@line-num:index-value)

146

CHAPTER 20. VIRTUAL CFG

20.4

Robustness to AST changes

Control ow graph nodes and edges can be kept (i.e., are not invalidated) in many cases when the underlying AST changes. However, there are some limitations to this capability. Changing the AST node that is pointed to by a given CFG node is not safe. CFG nodes for deleted AST nodes are of course invalid, as are those pointing to AST nodes whose parent pointers become invalid.

20.5

Limitations

Although workable for intraprocedural analysis of C code, the virtual CFG code has several limitations for other languages and uses.

20.5.1

Fortran support

The virtual control ow graph includes support for many Fortran constructs, but that support is fairly limited and not well tested. It is not recommended for production use.

20.5.2

Exception handling

The virtual CFG interface does not support control ow due to exceptions or the setjmp/longjmp constructs. It does, however, support break, continue, goto, and early returns from functions.

20.5.3

Interprocedural control ow analysis

In virtual CFGs, interprocedural control ow analysis is disabled by default. It can be enabled by setting the parameter virtualInteproceduralControlFlowGraphs in SgNode::cfgInEdges, SgNode::cfgOutEdges, and their subclasses denitions. Interprocedural edges are labeled with the eckInterprocedural EdgeConditionKind. In cases where the ow of control cannot be determined statically (calls of virtual functions, function pointers, or functors), the interprocedural control ow graph contains all possible options. In keeping with the virtual nature of ROSEs control ow graphs, the set of options is computed on-the-y; therefore, changes to the AST will be reected in subsequent interaction with the control ow graph.

20.6
FIXME

Node ltering

20.6.1

Interesting node lter

To simplify the virtual CFG, non-essential CFGNodes, such as the beginning and the end CFGNodes for each AST node, can be ltered out. Each eligible SgNode type has a most important CFGNode out of its all CFGNodes. The interesting CFGNodes index value for each Node type is

20.7. STATIC CFG

147

returned by calling the derived implementation of virtual bool SgNode::cfgIsIndexInteresting(int idx).

20.6.2

Arbitrary ltering

20.7

Static CFG

Since a virtual CFG does not produce any real graph, it is quite inecient to traverse a virtual CFG frequently. It is necessary to build a static CFG which may improve the performance of some specic operations. A SgGraph object (actually, its a SgIncidenceDirectedGraph object) is created to store the static CFG. Each node in the graph is a SgGraphNode object. In a virtual CFG, each node contains two members: node and index. A SgGraphNode already holds a pointer to SgNode, and we have to add the other property index to our SgGraphNode. This can be done by adding the corresponding attribute to SgGraphNode.

20.7.1

Class methods

CFG(): The default constructor. CFG(SgNode node, bool isFiltered = false): Initialize a static CFG with the start node to build from and a ag indicating if the CFG is a full or ltered one. setStart (SgNode node): Set the start node for building a static CFG. Note that the argument has to be an object of any of the following classes: SgProject, SgStatement, SgExpression, and SgInitializedName. If a SgProject object is passed in, several graphs are built for every function denition. isFilteredCFG(): Return the isFiltered ag. setFiltered (bool ag ): Set the isFiltered ag. buildCFG(): Build a full or ltered CFG according to the isFiltered ag. buildFullCFG(): Build a full CFG for debugging. buildFilteredCFG(): Build a ltered CFG which only contains interesting nodes. getOutEdges(SgGraphNode node): Return a vector of outgoing CFG edges (SgDirectedGraphEdge objects) from the given node. getInEdges(SgGraphNode node): Return a vector of CFG edges coming into the given node. cfgForBeginning(SgNode node): Return the CFG node for just before this AST node. cfgForEnd(SgNode node): Return the CFG node for just after this AST node. getIndex(SgGraphNode node): Return the index of the given CFG node.

148

CHAPTER 20. VIRTUAL CFG cfgToDot(SgNode node, const std::string& lename): Generate a DOT le for the current CFG. Note that the start node to be drawn can be indicated which is not necessary to be the start node of the CFG.

20.7.2

Drawing a graph of the CFG

Figure 20.7.2 shows a translator to dump full (debug) static control ow graphs for all functions within input source les. It also dumps a simplied version (interesting) version of static control ow graphs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 // Example t r a n s l a t o r t o g e n e r a t e d o t #i n c l u d e r o s e . h #i n c l u d e < s t r i n g > u s i n g namespace s t d ; files of static c o n t r o l flow graphs

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject s a g e P r o j e c t = fro ntend ( argc , argv ) ; // P r o c e s s a l l f u n c t i o n d e f i n i t i o n b o d i e s f o r s t a t i c c o n t r o l f l o w graph g e n e r a t i o n R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n s = NodeQuery : : querySubTree ( s a g e P r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : c o n s t i t e r a t o r i = f u n c t i o n s . b e g i n ( ) ; i != f u n c t i o n s . end ( ) ; ++i ) { SgFunctionDefinition proc = i s S g F u n c t i o n D e f i n i t i o n ( i ) ; ROSE ASSERT ( p r o c != NULL ) ; s t r i n g f i l e N a m e= S t r i n g U t i l i t y : : st r i p Pa t hF r o m Fi l eN a m e ( proc > g e t f i l e i n f o ()> g e t f i l e n a m e S t r i n g ( ) ) ; s t r i n g dotFileName1=f i l e N a m e +.+ proc > g e t d e c l a r a t i o n ()> get name ( ) +. debug . d o t ; s t r i n g dotFileName2=f i l e N a m e +.+ proc > g e t d e c l a r a t i o n ()> get name ( ) +. i n t e r e s t i n g . do t ; StaticCFG : : CFG c f g ( p r o c ) ; // Dump o u t t h e f u l l CFG, i n c l u d i n g b o o k k e e p i n g n o d e s c f g . buildFullCFG ( ) ; c f g . cfgToDot ( proc , dotFileName1 ) ; // Dump o u t o n l y t h o s e n o d e s which a r e i n t e r e s t i n g f o r c f g . buildFilteredCFG ( ) ; c f g . cfgToDot ( proc , dotFileName2 ) ; } return 0; } analyses

Figure 20.7: Example source code showing visualization of static control ow graph. The example input code is given in Fig. 20.3. Debug and interesting static CFG are shown in Fig. 20.5 and Fig. 20.6, respectively.

20.8

Static, Interprocedural CFGs

ROSE supports construction of interprocedural control ow graphs using the InterproceduralCFG class, a subclass of StaticCFG. Like the StaticCFG, the InterproceduralCFG can be constructed from any SgNode that aects control ow. If an InterproceduralCFG is constructed from a given node, it will contain all possible paths of execution from that point. Edges between procedures will be labelled with the eckInterprocedural EdgeConditionKind.

20.8. STATIC, INTERPROCEDURAL CFGS

149

In cases where a function call cannot be statically resolved to a function denition, the InterproceduralCFG includes edges from the call node to all possible denitions, which are determined by ROSEs CallGraph.

150

CHAPTER 20. VIRTUAL CFG

Chapter 21

Generating Control Flow Graphs

The control ow of a program is broken into basic blocks as nodes with control ow forming edges between the basic blocks. Thus the control ow forms a graph which often labeled edges (true and false), and basic blocks representing sequentially executed code. This chapter presents the Control Flow Graph (CFG) and the ROSE application code for generating such graphs for any function in an input code. The CFG forms a fundamental building block for more complex forms of program analysis. Figure 21.1 shows the code required to generate the control ow graph for each function of an application. Using the input code shown in gure 21.2 the rst functions control ow graph is shown in gure 21.3. Figure 21.3 shows the control ow graph for the function in the input code in gure 21.2.

151

152

CHAPTER 21. GENERATING CONTROL FLOW GRAPHS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e #i n c l u d e r o s e . h <GraphUpdate . h> CFGImpl . h GraphDotOutput . h c o n t r o l F l o w G r a p h . h CommandOptions . h

u s i n g namespace s t d ; // Use t h e ControlFlowGraph i s d e f i n e d i n both PRE // and t h e D o m i n a t o r T r ee s A n d D o m i n a n c e F r o n t i e r s namespaces . // We want t o u s e t h e one i n t h e PRE namespace . u s i n g namespace PRE ; class { }; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { SgFunctionDeclaration functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { SgFunctionDefinition functionDefinition = functionDeclaration >g e t d e f i n i t i o n ( ) ; i f ( f u n c t i o n D e f i n i t i o n != NULL) { SgBasicBlock functionBody = f u n c t i o n D e f i n i t i o n >g e t b o d y ( ) ; ROSE ASSERT( f u n c t i o n B o d y != NULL ) ; ControlFlowGraph c o n t r o l f l o w ; // The CFG can o n l y be c a l l e d on a f u n c t i o n d e f i n i t i o n ( a t p r e s e n t ) makeCfg ( f u n c t i o n D e f i n i t i o n , c o n t r o l f l o w ) ; s t r i n g fileName = functionDeclaration >get name ( ) . s t r ( ) ; f i l e N a m e += . d o t ; ofstream d o t f i l e ( fileName . c s t r ( ) ) ; printCfgAsDot ( d o t f i l e , c o n t r o l f l o w ) ; } } } i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; CmdOptions : : G e t I n s t a n c e ()> S e t O p t i o n s ( a r g c , a r g v ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; } visitorTraversal public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

Figure 21.1: Example source code showing visualization of control ow graph.

153

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

#i n c l u d e < s t d i o . h> #i n c l u d e < s t r i n g . h> #i n c l u d e < a s s e r t . h> size t i ; char b u f f e r [ 1 0 ] ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { f o r ( i =0; i < s t r l e n ( a r g v [ 1 ] ) ; { b u f f e r [ i ] = argv [ 1 ] [ i ] ; } return 0; } int t e s t I f ( int i ) { int rt ; i f ( i %2 ==0) r t =0; else r t =1; return rt ; }

i ++)

Figure 21.2: Example source code used as input to build control ow graph.

1 i = 0; in SgForInitStatement

4 i < strlen(argv[1]) in SgForStatement

2 return 0;

3 buffer[i] = argv[1][i];

5 i++ in SgBasicBlock

Figure 21.3: Control ow graph for function in input code le: inputCode 1.C.

154

CHAPTER 21. GENERATING CONTROL FLOW GRAPHS

Chapter 22

Graph Processing Tutorial

22.1 Traversal Tutorial

ROSE can collect and analyze paths in both source and binary CFGs. At moment it doesnt attempt to save paths because if you save them directly the space necessary is extremely large, as paths grow 2n with successive if statements and even faster when for loops are involved. Currently a path can only cannot complete the same loop twice. However it is possible for a graph such that 1 - 2 , 2-3, 3-1, 3-5, has paths, 1,2,3,1,2,3,5 and 1,2,3,5 because the loop 1,2,3,1 is not repeated. The tutorial example works as such:

155

156

CHAPTER 22. GRAPH PROCESSING TUTORIAL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

#include < i o s t r e a m > #include < f s t r e a m > //#i n c l u d e < r o s e . h> #include < s t r i n g > #include < e r r . h> #include SgGraphTemplate . h #include g r a p h P r o c e s s i n g . h

#include staticCFG . h #include i n t e r p r o c e d u r a l C F G . h / T e s t i n g t h e graph t r a v e r s a l mechanism now i mple menting i n A s t P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g / #include < s y s / t i m e . h> #include < s y s / r e s o u r c e . h> using namespace s t d ; using namespace b o o s t ;

/ You need t o use myGraph t y p e h e r e b e c a u s e t h e c o n v e r s i o n o f StaticCFG : : InterproceduralCFG or StaticCFG : :CFG i n a b o o s t form . The SgGraphTemplate . h f i l e h a n d l e s t h i s c o n v e r s i o n and myGraph i s s p e c i f i c t o t h a t f i l e / typedef myGraph CFGforT ;

/ Your b a s i c v i s i t o r t r a v e r s a l s u b c l a s s e d from SgGraphTraversal on t h e CFGforT t e m p l a t e as d e f i n e d above / c l a s s v i s i t o r T r a v e r s a l : public S g G r a p h T r a v e r s a l <CFGforT> { public : int paths ; / This i s t h e f u n c t i o n run by t h e a l g o r i t h m on e v e r y path , VertexID i s a t y p e implemented i n SgGraphTe void a n a l y z e P a t h ( v e c t o r <VertexID >& pth ) ; }; / d e f i n i n g t h e a n a l y z e P a t h f u n c t i o n . This s i m p l y c o u n t s p a t h s as s h o u l d be o b v i o u s . Again , VertexID i s d e f i n e void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r <VertexID >& pth ) { p a t h s ++; } i n t main ( i n t a r g c , char a r g v [ ] ) { / F i r s t you need t o produce t h e p r o j e c t f i l e / SgProject p r o j = frontend ( argc , argv ) ; ROSE ASSERT ( p r o j != NULL ) ; / G e t t i n g t h e Function D e c l a r a t i o n and D e f i n i t i o n f o r p r o d u c i n g t h e graph / S g F u n c t i o n D e c l a r a t i o n mainDefDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j ) ; S g F u n c t i o n D e f i n i t i o n mainDef = mainDefDecl >g e t d e f i n i t i o n ( ) ; / I n s t a n t i a t i n g t h e v i s i t o r T r a v e r s a l / v i s i t o r T r a v e r s a l v i s = new v i s i t o r T r a v e r s a l ( ) ; / This c r e a t e s t h e StaticCFG : : InterproceduralCFG o b j e c t t o be c o n v e r t e d t o a b o o s t graph / StaticCFG : : I n t e r p r o c e d u r a l C F G c f g ( mainDef ) ; stringstream ss ; S g I n c i d e n c e D i r e c t e d G r a p h g = new S g I n c i d e n c e D i r e c t e d G r a p h ( ) ; / We g o t t h e n e c e s s a r y i n t e r n a l S g I n c i d e n c e D i r e c t e d G r a p h from t h e c f g / g = c f g . getGraph ( ) ; myGraph mg = new myGraph ( ) ; / C o n v e r t i n g t h e c f g t o a b o o s t graph / mg = i n s t a n t i a t e G r a p h ( g , c f g , mainDef ) ; / S e t i n t e r n a l v a r i a b l e s / vis >p a t h s = 0 ; / i n v o k i n g t h e t r a v e r s a l , t h e f i r s t argument i s t h e graph , t h e second i s t r u e i f you do not want bounds , f a l s e i f you do , t h e t h i r d and f o u r t h arguments a r e s t a r t i n g and s t o p p i n g v e r t i c e s r e s p e c t i v e l y , i f you a r e not bounding s i m p l y i n s e r t 0 . F i n a l l y t h e l a s t argument i s c u r r e n t l y d e p r e c a t e d / vis >c o n s t r u c t P a t h A n a l y z e r (mg , true , 0 , 0 , true ) ; s t d : : c o u t << f i n i s h e d << s t d : : e n d l ; s t d : : c o u t << p a t h s : << v i s >p a t h s << s t d : : e n d l ; delete v i s ; }

Figure 22.1: Source CFG Traversal Example

22.1. TRAVERSAL TUTORIAL

157

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

#include < i o s t r e a m > #include < f s t r e a m > #include < r o s e . h> //#i n c l u d e interproceduralCFG . h #include < s t r i n g > #include < e r r . h> / These a r e n e c e s s a r y f o r any b i n a r y T r a v e r s a l / #include g r a p h P r o c e s s i n g . h #include B i n a r y C o n t r o l F l o w . h #include B i n a r y L o a d e r . h / T e s t i n g t h e graph t r a v e r s a l mechanism now i mple menting i n g r a p h P r o c e s s i n g . h ( i n s i d e s r c /midend/ a s t P r o c e s s i n g /) / using namespace s t d ; using namespace b o o s t ; / These s h o u l d j u s t be c o p i e d v e r b a t i m / typedef b o o s t : : g r a p h t r a i t s <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph > : : v e r t e x d e s c r i p t o r V e r t e x ; / < Graph v e r t e x t y p e . / typedef b o o s t : : g r a p h t r a i t s <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph > : : e d g e d e s c r i p t o r Edge ; / < Graph ed g e t y p e . /

/ We f i r s t make a v i s i t o r T r a v e r s a l , s u b c l a s s e d from SgGraphTraversal t e m p l a t e d on t h e B i n a r y A n a l y s i s : ControlFlow : : Graph which i s implemented as a b o o s t graph /

class v i s i t o r T r a v er sa l {

: public S g G r a p h T r a v e r s a l <B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph>

public : long i n t p t h s ; long i n t t l t n o d e s ; / This needs t o be i n any v i s i t o r T r a v e r s a l , i t i s t h e f u n c t i o n t h a t w i l l be run on e v e r y p a t h by t h e graph p a t h a n a l y s i s a l g o r i t h m , n o t i c e t h e V e r t e x t y p e i s from t h e above t y p e d e f s / v i r t u a l void a n a l y z e P a t h ( v e c t o r <Vertex >& pth ) ; }; / This i s a v e r y s i m p l e i n c a r n a t i o n , i t j u s t c o u n t s p a t h s / void v i s i t o r T r a v e r s a l : : a n a l y z e P a t h ( v e c t o r <Vertex >& pth ) { p t h s ++; } i n t main ( i n t a r g c , char a r g v [ ] ) { / Parse t h e b i n a r y f i l e / SgProject p r o j e c t = fr ontend ( argc , argv ) ; s t d : : v e c t o r <S g A s m I n t e r p r e t a t i o n > i n t e r p s = S a g e I n t e r f a c e : : querySubTree <S g A s m I n t e r p r e t a t i o n >( p r o j e c t ) ; i f ( i n t e r p s . empty ( ) ) { f p r i n t f ( s t d e r r , no b i n a r y i n t e r p r e t a t i o n s f o u n d \ n ) ; exit (1); } / C a l c u l a t e p l a i n o l d CFG. / BinaryAnalysis : : ControlFlow c f g a n a l y z e r ; B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph c f g = new B i n a r y A n a l y s i s : : C o n t r o l F l o w : : Graph ; c f g a n a l y z e r . b u i l d c f g f r o m a s t ( i n t e r p s . back ( ) , c f g ) ; s t d : : o f s t r e a m mf ; mf . open ( a n a l y s i s . d o t ) ; / D e c l a r i n g t h e v i s i t o r T r a v e r s a l / v i s i t o r T r a v e r s a l v i s = new v i s i t o r T r a v e r s a l ; / S e t t i n g i n t e r n a l v a r i a b l e s / vis >t l t n o d e s = 0 ; vis >p t h s = 0 ;

/ v i s i t o r T r a v e r s a l has 5 arguments , t h e f i r s t i s t h e ambient CFG, t h e second i d e n t i f i e s w h e t h e r or not you ar e bounding t h e graph , t h a t i s , w h e t h e r you want a l l your p a t h s t o s t a r t a t one s p e c i f i c node and end a t a n o t h e r s p e c i f i c node , t h e f o u r t h and f i f t h would be s t a r t and end i f t h e graph were bounded . S i n c e t h e y aren t you can s i m p l y i n p u t 0 , f o r t h e moment t h e f i n a l argument i s d e p r e c a t e d , t h o u g h i t s p u r p o s e was t o t e l l t h e progra t h a t your a n a l y s i s f u n c t i o n was t h r e a d s a f e , t h a t i s t h a t openMP c o u l d run i t w i t h o u t h a v i n g a c r i t i c a l command . C u r r e n t l y a c r i t i c a l i s a l w a y s used / vis >c o n s t r u c t P a t h A n a l y z e r ( c f g , true , 0 , 0 , f a l s e ) ; s t d : : c o u t << p t h s : << v i s >p t h s << s t d : : e n d l ; s t d : : c o u t << t l t n o d e s : << v i s > t l t n o d e s << s t d : : e n d l ;

158

CHAPTER 22. GRAPH PROCESSING TUTORIAL

Chapter 23

Dataow Analysis
The dataow analysis in Rose is based on the control ow graph (CFG). One type of dataow analysis is called def-use analysis, which is explained next.

23.1

Def-Use Analysis

The denition-usage (def-use) analysis allows to query the denition and usage for each control ow node (CFN). Any statement or expression within ROSE is represented as a sequence of CFNs. For instance, the CFG for the following program
1 2 3 4 5 i n t main ( ) { int x = 9; x = x + 1; }

Figure 23.1: Example input code. is illustrated in Figure 23.4.

23.1.1

Def-use Example implementation

Fig. 23.2 shows an example program of how the def-use analysis is called. It generates a dot graph showing def/use information within a control ow graph. It also outputs reaching definition information for each variable references of an input code. This program (named as defuseAnalysis ) is installed under ROSE INST/bin as a standalone tool for users to experiment the def/use analysis of ROSE. Figure 23.3 shows the screen output of the code(Fig. 23.2) running on the input code(Fig. 23.1). Each variable reference in the input code has at least one reaching denition node. The associated denition statement is also printed out. 159

160

CHAPTER 23. DATAFLOW ANALYSIS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

#i n c l u d e r o s e . h #i n c l u d e D e f U s e A n a l y s i s . h #i n c l u d e < s t r i n g > #i n c l u d e < i o s t r e a m > u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v + a r g c ) ; SgProject project = frontend ( argvList ) ; // C a l l t h e DefUse A n a l y s i s DFAnalysis d e f u s e = new D e f U s e A n a l y s i s ( p r o j e c t ) ; b o o l debug = f a l s e ; defuse >run ( debug ) ; // Output d e f u s e a n a l y s i s r e s u l t s i n t o a d o t f i l e defuse >dfaToDOT ( ) ; // Find a l l v a r i a b l e r e f e r e n c e s N o d e Q u e r y S y n t h e s i z e d A t t r i b u t e T y p e v a r s = NodeQuery : : querySubTree ( p r o j e c t , V SgVarRefExp ) ; NodeQuerySynthesizedAttributeType : : c o n s t i t e r a t o r i = vars . begin ( ) ; f o r ( ; i != v a r s . end ();++ i ) { SgVarRefExp v a r R e f = isSgVarRefExp ( i ) ; S g I n i t i a l i z e d N a m e initName = i s S g I n i t i a l i z e d N a m e ( var Ref >g e t s y m b o l ()> g e t d e c l a r a t i o n ( ) ) ; s t d : : s t r i n g name = initName >g e t q u a l i f i e d n a m e ( ) . s t r ( ) ; // Find r e a c h i n g d e f i n i t i o n o f initName a t t h e c o n t r o l f l o w node v a r R e f v e c t o r <SgNode > v e c = d e f u s e >g e t D e f F o r ( v ar Ref , initName ) ; ROSE ASSERT ( v e c . s i z e ( ) >0 ) ; // e a c h v a r i a b l e r e f e r e n c e must have a d e f i n i t i o n somewhere // Output e a c h r e a c h i n g d e f i n i t i o n node and t h e c o r r e s p o n d i n g s t a t e m e n t . s t d : : cout < < < <s t d : : e n d l ; s t d : : c o u t << v e c . s i z e ( ) << d e f i n i t i o n e n t r y / e n t r i e s f o r << va rR ef >u n p a r s e T o S t r i n g ( ) << @ l i n e << va rR ef > g e t f i l e i n f o ()> g e t l i n e () <<: << va rR ef > g e t f i l e i n f o ()> g e t c o l ( ) << s t d : : e n d l ; f o r ( s i z e t j =0; j <v e c . s i z e ( ) ; j ++) { cout < <v e c [ j ]> c l a s s n a m e () << << v e c [ j ]<< e n d l ; SgSt at emen t d e f s t m t = S a g e I n t e r f a c e : : g e t E n c l o s i n g S t a t e m e n t ( v e c [ j ] ) ; ROSE ASSERT( d e f s t m t ) ; cout < <d e f s t m t >u n p a r s e T o S t r i n g () << @ l i n e << d e f s t m t > g e t f i l e i n f o ()> g e t l i n e ( ) <<:<< d e f s t m t > g e t f i l e i n f o ()> g e t c o l ( ) << e n d l ; } } return 0; }

Figure 23.2: Example source code using def use analysis

1 2 3 4 5 6 7 8 1 d e f i n i t i o n entry / e n t r i e s fo r x @ l i n e 4:5 SgAssignInitializer 0 x97cf6f0 int x = 9; @ line 3:3 1 d e f i n i t i o n entry / e n t r i e s fo r x @ l i n e 4:7 SgAssignInitializer 0 x97cf6f0 int x = 9; @ line 3:3

Figure 23.3: Output of the program

23.1. DEF-USE ANALYSIS

161

23.1.2

Accessing the Def-Use Results

For each CFN in the CFG, the denition and usage for variable references can be determined with the public function calls: vector <SgNode*> getDefFor(SgNode*, SgInitializedName*) vector <SgNode*> getUseFor(SgNode*, SgInitializedName*) where SgNode* represents any control ow node and SgInitializedName any variable (being used or dened at that CFN). The result is a vector of possible CFNs that either dene (getDefFor) or use (getUseFor) a specic variable. Figure 23.4 shows how the variable x is being declared and dened in CFNs between node 1 and 6. Note that the denition is annotated along the edge. For instance at node 6, the edge reads (6) DEF: x (3) = 5. This means that variable x was declared at CFN 3 but dened at CFN 5. The second statement x=x+1 is represented by CFNs from 7 to 12. One can see in the gure that x is being re-dened at CFN 11. However, the denition of x within the same statement happens at CFN 8. Hence, the denition of the right hand side x in the statement is at CFN 5 : (8) DEF: x (3) = 5. Another usage of the def-use analysis is to determine which variables actually are dened at each CFN. The following function allows to query a CFN for all its variables (SgInitializedNames) and the positions those variables are dened (SgNode): std::multimap <SgInitializedName*, SgNode*> getDefMultiMapFor(SgNode*) std::multimap <SgInitializedName*, SgNode*> getUseMultiMapFor(SgNode*) All public functions are described in DefuseAnalysis.h. To use the def-use analysis, one needs to create an object of the class DefUseAnalysis and execute the run function. After that, the described functions above help to evaluate denition and usage for each CFN.

162

CHAPTER 23. DATAFLOW ANALYSIS

::main : ( 2 ) - [0x2ade7aa41010] <SgFunctionDefinition> @line=1 :idx=0 DEF: x ( 4 ) - 12 : ( 3 ) - [0x2ade7a956628] <SgFunctionParameterList> @line=1 :idx=0

: ( 4 ) - [0x2ade7a9dc438] initVar : x<SgInitializedName> x :idx=0 DEF: x ( 4 ) - 4 : ( 5 ) - [0x97b6060] <SgIntVal> @line=3 :idx=1 DEF: x ( 4 ) - 4 : ( 6 ) - [0x97cf6f0] <SgAssignInitializer> @line=3 :idx=1 DEF: x ( 4 ) - 6 : ( 7 ) - [0x2ade7aacd010] varDecl : x,<SgVariableDeclaration> @line=3 :idx=1 DEF: x ( 4 ) - 6 : ( 8 ) - [0x97f4b40] varRef : x<SgVarRefExp> @line=4 :idx=0 DEF: x ( 4 ) - 6 : ( 9 ) - [0x97f4ba8] varRef : x<SgVarRefExp> @line=4 :idx=0 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 10 ) - [0x97b60c8] <SgIntVal> @line=4 :idx=1 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 11 ) - [0x980e3a0] <SgAddOp> @line=4 :idx=2 DEF: x ( 4 ) - 6 USE: x ( 4 ) - 9 : ( 12 ) - [0x9829930] <SgAssignOp> @line=4 :idx=2 DEF: x ( 4 ) - 12 : ( 13 ) - [0x9844f20] <SgExprStatement> @line=4 :idx=1 DEF: x ( 4 ) - 12 : ( 14 ) - [0x97b6130] <SgIntVal> @line=5 :idx=1 DEF: x ( 4 ) - 12 : ( 15 ) - [0x985a6f0] <SgReturnStmt> @line=0 :idx=1 DEF: x ( 4 ) - 12 ::main : ( 2 ) - [0x2ade7aa41010] <SgFunctionDefinition> @line=1 :idx=3

Figure 23.4: Def-Use graph for example program.

23.2. LIVENESS ANALYSIS

163

23.2

Liveness Analysis

Liveness analysis is a classic data ow analysis performed by compilers to calculate for each program point the variables that may be potentially read before their next write (re-denition). A variable is live at a point in a programs execution path if it holds a value that may be needed in the future. Fig. 23.5 shows an example program of how the liveness analysis is called in a ROSE-based translator. It generates a dot graph showing def/use information within a control ow graph, alone with live-in and live-out variables. This program (named as livenessAnalysis ) is installed under ROSE INST/bin as a standalone tool for users to experiment the liveness analysis of ROSE. Figure 23.7 shows control ow graph with live variable information for the code(Fig. 23.5) running on the input code(Fig. 23.6).

23.2.1

Access live variables

After calling liveness analysis, one can access live-in and live-out variables from a translator based on the virtual control ow graph node. Figure 23.8 shows an example function which retrieves the live-in and live-out variables for a for loop. The code accesses the CFG node (showing in Figure 23.7) of a for statement and retrieve live-in variables of the true edges target node as the live-in variables of the loop body. Similarly, the live-out variables of the loop are obtained by getting the live-in variables of the node right after the loop (target node of the false edge).

164

CHAPTER 23. DATAFLOW ANALYSIS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

#i n c l u d e r o s e . h #i n c l u d e < s t r i n g > #i n c l u d e < i o s t r e a m > #i n c l u d e < f s t r e a m > u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v + a r g c ) ; SgProject project = frontend ( argvList ) ; i f ( project > g e t f i l e L i s t ( ) . s i z e ( ) ==0) return 0; // P r e p a r e t h e DefUse A n a l y s i s DFAnalysis d e f u s e = new D e f U s e A n a l y s i s ( p r o j e c t ) ; b o o l debug = f a l s e ; defuse >run ( debug ) ; i f ( debug ) defuse >dfaToDOT ( ) ; // P r e p a r e l i v e n e s s a n a l y s i s L i v e n e s s A n a l y s i s l i v = new L i v e n e s s A n a l y s i s ( debug , ( D e f U s e A n a l y s i s ) d e f u s e ) ; ROSE ASSERT ( l i v != NULL ) ; // Find a l l f u n c t i o n d e f i n i t i o n s R o s e S T L C o n t a i n e r <SgNode> n o d e L i s t= NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; s t d : : v e c t o r <FilteredCFGNode < I s D F A F i l t e r > > d f a F u n c t i o n s ; R o s e S T L C o n t a i n e r <SgNode > : : c o n s t i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; b o o l abortme= f a l s e ; f o r ( ; i != n o d e L i s t . end ();++ i ) { SgFunctionDefinition func = i s S g F u n c t i o n D e f i n i t i o n ( i ) ; // run l i v e n e s s a n a l y s i s FilteredCFGNode < I s D F A F i l t e r > r e m s o u r c e = l i v >run ( f u n c , abortme ) ; i f ( abortme ) { c e r r <<E r r o r : L i v e n e s s a n a l y s i s i s ABORTING . << e n d l ; ROSE ASSERT( f a l s e ) ; } i f ( r e m s o u r c e . getNode ( ) ! =NULL) dfaFunctions . push back ( rem source ) ; } SgFilePtrList f i l e l i s t = project >g e t f i l e L i s t ( ) ; s t d : : s t r i n g f i r s t F i l e N a m e = S t r i n g U t i l i t y : : st r i p Pa t hF r o m Fi l eN a m e ( f i l e l i s t [0] > g e t F i l e N a m e ( ) ) ; s t d : : s t r i n g f i l e N a m e = f i r s t F i l e N a m e + l i v e n e s s . d o t ; std : : ofstream f s ( fileName . c s t r ( ) ) ; dfaToDot ( f s , s t r i n g ( v a r ) , d f a F u n c t i o n s , ( D e f U s e A n a l y s i s ) d e f u s e , l i v ) ; fs . close (); return 0; }

Figure 23.5: Example source code using liveness analysis

23.2. LIVENESS ANALYSIS

165

1 2 3 4 5 6 7 8 9 10 11 12 13 14

int a [100]; void foo2 () { int i ; i n t tmp ; tmp = 1 0 ; f o r ( i =0; i < 100; i ++) { a [ i ] = tmp ; tmp =a [ i ]+ i ; } a[0] = 1 ; }

Figure 23.6: Example input code.

166

CHAPTER 23. DATAFLOW ANALYSIS

::foo2 : ( 3 ) - [0x2ac006c4a010] <SgFunctionDefinition> @line=3 :idx=0 out : visited : 3 in : : ( 4 ) - [0x2ac006a391b8] <SgFunctionParameterList> @line=3 :idx=0 out : visited : 1 in : : ( 5 ) - [0x2ac006abf1e8] initVar : i<SgInitializedName> i :idx=0 out : visited : 1 in : DEF: i ( 5 ) - 5 : ( 6 ) - [0x2ac006b24290] varDecl : i,<SgVariableDeclaration> @line=5 :idx=1 out : visited : 2 in : : ( 7 ) - [0x2ac006abf310] initVar : tmp<SgInitializedName> tmp :idx=0 out : visited : 2 in : DEF: tmp ( 7 ) - 7 : ( 8 ) - [0x2ac006b24510] varDecl : tmp,<SgVariableDeclaration> @line=6 :idx=1 out : visited : 2 in : : ( 9 ) - [0x19593b20] varRef : tmp<SgVarRefExp> @line=7 :idx=0 out : visited : 2 in : : ( 10 ) - [0x195ad240] <SgIntVal> @line=7 :idx=1 out : visited : 2 in : : ( 11 ) - [0x195c68f0] <SgAssignOp> @line=7 :idx=2 out : tmp, visited : 2 in : tmp, DEF: tmp ( 7 ) - 11 : ( 12 ) - [0x195e1e80] <SgExprStatement> @line=7 :idx=1 out : tmp, visited : 2 in : tmp, : ( 13 ) - [0x19593b88] varRef : i<SgVarRefExp> @line=8 :idx=0 out : tmp, visited : 2 in : tmp, : ( 14 ) - [0x195ad2a8] <SgIntVal> @line=8 :idx=1 out : tmp, visited : 2 in : tmp, : ( 15 ) - [0x195c6960] <SgAssignOp> @line=8 :idx=2 out : tmp,i, visited : 2 in : tmp,i, DEF: i ( 5 ) - 15 : ( 16 ) - [0x195e1ed8] <SgExprStatement> @line=8 :idx=1 out : tmp,i, visited : 2 in : tmp,i, : ( 17 ) - [0x195f98d0] <SgForInitStatement> @line=8 :idx=1 out : tmp,i, visited : 3 in : tmp,i, : ( 18 ) - [0x19593bf0] varRef : i<SgVarRefExp> @line=8 :idx=0 out : tmp,i, visited : 3 in : tmp,i, USE: i ( 5 ) - 18 : ( 19 ) - [0x195ad310] <SgIntVal> @line=8 :idx=1 out : tmp,i, visited : 3 in : tmp,i, : ( 20 ) - [0x19613390] <SgLessThanOp> @line=8 :idx=2 out : tmp,i, visited : 2 in : tmp,i, : ( 21 ) - [0x195e1f30] <SgExprStatement> @line=8 :idx=1 out : tmp,i, visited : 2 in : tmp,i, : ( 22 ) - [0x2ac006cd6010] <SgForStatement> @line=8 :idx=2 out : tmp,i, visited : 2 in : tmp,i, : ( 23 ) - [0x19593d28] varRef : a<SgVarRefExp> @line=0 :idx=0 out : tmp,i, visited : 1 in : tmp,i, : ( 25 ) - [0x19593d90] varRef : i<SgVarRefExp> @line=10 :idx=0 out : tmp,i, visited : 1 in : tmp,i, USE: i ( 5 ) - 25 : ( 27 ) - [0x1964a4a0] <SgPntrArrRefExp> @line=10 :idx=2 out : tmp,i, visited : 2 in : tmp,i, : ( 29 ) - [0x19593df8] varRef : tmp<SgVarRefExp> @line=10 :idx=0 out : i, visited : 2 in : i, USE: tmp ( 7 ) - 29 : ( 31 ) - [0x195c69d0] <SgAssignOp> @line=10 :idx=2 out : i,a, visited : 2 in : i,a, DEF: ::a ( 2 ) - 31 out : tmp,i, visited : 2 in :

: ( 24 ) - [0x195940d0] varRef : a<SgVarRefExp> @line=0 :idx=0 out : visited : 2 in : : ( 26 ) - [0x195ad378] <SgIntVal> @line=13 :idx=1

out : visited : 2 in :

: ( 28 ) - [0x1964a580] <SgPntrArrRefExp> @line=13 :idx=2 out : visited : 2 in : : ( 30 ) - [0x195ad3e0] <SgIntVal> @line=13 :idx=1

out : visited : 2 in :

: ( 32 ) - [0x195c6ab0] <SgAssignOp> @line=13 :idx=2 out : visited : 2 in : DEF: ::a ( 2 ) - 32 out : tmp,i, visited : 3 in : tmp,i, DEF: i ( 5 ) - 44

: ( 33 ) - [0x195e1f88] <SgExprStatement> @line=10 :idx=1

: ( 34 ) - [0x195e2038] <SgExprStatement> @line=13 :idx=1

out : i,a, visited : 2 in : i,a, : ( 35 ) - [0x19593e60] varRef : tmp<SgVarRefExp> @line=11 :idx=0 out : i,a, visited : 2 in : i,a,

out : visited : 2 in : ::foo2 : ( 3 ) - [0x2ac006c4a010] <SgFunctionDefinition> @line=3 :idx=3

: ( 36 ) - [0x19593f30] varRef : a<SgVarRefExp> @line=0 :idx=0 out : i, visited : 2 in : i, USE: ::a ( 2 ) - 36 : ( 37 ) - [0x19593f98] varRef : i<SgVarRefExp> @line=11 :idx=0 out : i, visited : 2 in : i, USE: i ( 5 ) - 37 : ( 38 ) - [0x1964a510] <SgPntrArrRefExp> @line=11 :idx=2 out : i, visited : 2 in : i, : ( 39 ) - [0x19594000] varRef : i<SgVarRefExp> @line=11 :idx=0 out : i, visited : 2 in : i, USE: i ( 5 ) - 39 : ( 40 ) - [0x19665dd0] <SgAddOp> @line=11 :idx=2 out : i, visited : 2 in : i, : ( 41 ) - [0x195c6a40] <SgAssignOp> @line=11 :idx=2 out : tmp,i, visited : 2 in : tmp,i, DEF: tmp ( 7 ) - 41 : ( 42 ) - [0x195e1fe0] <SgExprStatement> @line=11 :idx=1 out : tmp,i, visited : 2 in : tmp,i, : ( 43 ) - [0x19593c58] varRef : i<SgVarRefExp> @line=8 :idx=0 out : tmp, visited : 3 in : tmp, USE: i ( 5 ) - 43 : ( 44 ) - [0x1962e940] <SgPlusPlusOp> @line=8 :idx=1

Figure 23.7: Control ow graph annotated with live variables for example program.

23.2. LIVENESS ANALYSIS

167

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

v o i d g e t L i v e V a r i a b l e s ( L i v e n e s s A n a l y s i s l i v , S gF or St at ement f o r s t m t ) { ROSE ASSERT( l i v != NULL ) ; ROSE ASSERT( f o r s t m t != NULL ) ; s t d : : v e c t o r < S g I n i t i a l i z e d N a m e > l i v e I n s , l i v e O u t s ; // For SgForStatement , v i r t u a l CFG node which i s i n t e r e s t i n g ha s an i n d e x number o f 2 , // a s shown i n i t s d o t graph s node c a p t i o n . // < SgForStatement > @ 8 : 2 means t h i s node i s f o r a f o r s t a t e m e n t a t s o u r c e l i n e 8 , w i t h an i n d e x 2 . CFGNode c f g n o d e ( f o r s t m t , 2 ) ; FilteredCFGNode <I s D F A F i l t e r > f i l t e r n o d e= FilteredCFGNode <I s D F A F i l t e r > ( c f g n o d e ) ; // Check e d g e s v e c t o r <FilteredCFGEdge < I s D F A F i l t e r > > o u t e d g e s = f i l t e r n o d e . out Edges ( ) ; ROSE ASSERT( o u t e d g e s . s i z e ( ) == 2 ) ; v e c t o r <FilteredCFGEdge < I s D F A F i l t e r > > : : i t e r a t o r i t e r = o u t e d g e s . b e g i n ( ) ; f o r ( ; i t e r != o u t e d g e s . end ( ) ; i t e r ++) { FilteredCFGEdge < I s D F A F i l t e r > e d g e= i t e r ; // one t r u e e d g e g o i n g i n t o t h e l o o p body // x . Live i n ( l o o p ) = l i v e i n ( f i r s t stmt i n l o o p ) i f ( e d g e . c o n d i t i o n ()== eckTrue ) { SgNode f i r s t n o d e= e d g e . t a r g e t ( ) . getNode ( ) ; liveIns = liv >g e t I n ( f i r s t n o d e ) ; } // one f a l s e e d g e g o i n g o u t o f l o o p // x . l i v e o u t ( l o o p ) = l i v e i n ( f i r s t stmt a f t e r l o o p ) e l s e i f ( e d g e . c o n d i t i o n ()== e c k F a l s e ) { SgNode f i r s t n o d e= e d g e . t a r g e t ( ) . getNode ( ) ; liveOuts0 = liv >g e t I n ( f i r s t n o d e ) ; } else { c e r r <<Unexpected CFG o u t e d g e t y p e f o r SgForStmt!<< e n d l ; ROSE ASSERT( f a l s e ) ; } } // end f o r ( e d g e s ) }

Figure 23.8: Example code retrieving live variables based on virtual control ow graph

168

CHAPTER 23. DATAFLOW ANALYSIS

Chapter 24

Generating the Call Graph (CG)

The formal denition of a call graph is: A diagram that identies the modules in a system or computer program and shows which modules call one another. IEEE A call graph shows all function call paths of an arbitrary code. These paths are found by following all function calls in a function, where a function in the graph is represented by a node and each possible function call by an edge (arrow). To make a call graph this process is redone for every called function until all edges are followed and there are no ungraphed functions. ROSE has an in-build mechanism for generating call graphs. ROSE provides support for generating call graphs, as dened in src/midend/programAnalysis/CallGraphAnalysis/CallGraph.h. Figure 24 shows the code required to generate the call graph for each function of an application. Using the input code shown in gure 24 the rst functions call graph is shown in gure 24.3. A standalone tool named buildCallGraph is installed under ROSE INSTALL/bin so users can use it to generate call graphs in dot format.

169

170

CHAPTER 24. GENERATING THE CALL GRAPH (CG)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

#i n c l u d e r o s e . h #i n c l u d e <CallGraph . h> #i n c l u d e < i o s t r e a m > u s i n g namespace s t d ; u s i n g namespace S t r i n g U t i l i t y ; // A F u n c t i o n o b j e c t u s e d a s a p r e d i c a t e t h a t d e t e r m i n e s which f u n c t i o n s a r e // t o be r e p r e s e n t e d i n t h e c a l l graph . s t r u c t k e e p F u n c t i o n : p u b l i c u n a r y f u n c t i o n <b o o l , S g F u n c t i o n D e c l a r a t i o n > { bool operator ( ) ( SgFunctionDeclaration funcDecl ) { bool returnValue = true ; ROSE ASSERT( f u n c D e c l != NULL ) ; s t r i n g filename = funcDecl > g e t f i l e i n f o ()> g e t f i l e n a m e ( ) ; s t d : : s t r i n g func name = f u n c D e c l >get name ( ) . g e t S t r i n g ( ) ; s t r i n g s t r i p p e d f i l e n a m e = st r i p P a t h Fr o mF i l eN a me ( f i l e n a m e ) ; // s t r i n g : : s i z e t y p e l o c ; // F i l t e r o u t f u n c t i o n s from t h e ROSE p r e i n c l u d e h e a d e r f i l e i f ( f i l e n a m e . f i n d ( r o s e e d g r e q u i r e d m a c r o s a n d f u n c t i o n s ) ! = s t r i n g : : npos ) returnValue = f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s e l s e i f ( funcDecl > g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ()== t r u e ) r e t u r n V a l u e= f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s e l s e i f ( funcDecl > g e t f i l e i n f o ()> i s F r o n t e n d S p e c i f i c ()== t r u e ) r e t u r n V a l u e= f a l s e ; // f i l t e r o u t o t h e r b u i l t i n f u n c t i o n s // e l s e i f ( func name . f i n d ( ,0)== 0 ) ; // returnValue = f a l s e ; // I O g e t c I O p u t c IO feof , etc . // l o c = func name . f i n d ( I O , 0 ) ; // i f ( l o c == 0 ) r e t u r n V a l u e = f a l s e ; // s k i p f u n c t i o n s from s t a n d a r d s y s t e m h e a d e r s // TODO Need more r i g i d c h e c k else if ( s t r i p p e d f i l e n a m e==s t r i n g ( s t d i o . h ) | | s t r i p p e d f i l e n a m e==s t r i n g ( l i b i o . h ) | | s t r i p p e d f i l e n a m e==s t r i n g ( math . h ) | | s t r i p p e d f i l e n a m e==s t r i n g ( t i m e . h ) | | s t r i p p e d f i l e n a m e==s t r i n g ( s e l e c t . h ) | | s t r i p p e d f i l e n a m e==s t r i n g ( m a t h c a l l s . h ) ) r e t u r n V a l u e= f a l s e ; i f ( returnValue ) cout <<Debug:<< func name << from f i l e :<< s t r i p p e d f i l e n a m e << Keep : << r e t u r n V a l u e < <e n d l ; return returnValue ; } }; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; if { ( project > g e t f i l e L i s t ( ) . s i z e ( ) >=1) // C o n s t r u c t a c a l l Graph C a l l G r a p h B u i l d e r CGBuilder ( p r o j e c t ) ; CGBuilder . b u i l d C a l l G r a p h ( k e e p F u n c t i o n ( ) ) ; CGBuilder . b u i l d C a l l G r a p h ( b u i l t i n F i l t e r ( ) ) ; // Output t o a d o t f i l e AstDOTGeneration d o t g e n ; SgFilePtrList f i l e l i s t = project >g e t f i l e L i s t ( ) ; s t d : : s t r i n g f i r s t F i l e N a m e = S t r i n g U t i l i t y : : s t r i p Pa t h F r o mF i l e N a me ( f i l e l i s t [0] > g e t F i l e N a m e ( ) ) ; d o t g e n . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( CGBuilder . getGraph ( ) , f i r s t F i l e N a m e + c a l l G r a p h . d o t ) ; } r e t u r n 0 ; // backend ( p r o j e c t ) ; }

//

Figure 24.1: Example source code showing visualization of call graph.

171

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55

// s i m p l e ( class A { public : int int int }; void void void void foo1 foo2 foo3 foo4

t r i v i a l ) example c o d e u s e d t o d e m o n s t r a t e t h e

c a l l graph g e n e r a t i o n

f1 () { return 0;} f 2 ( ) { p f = &A : : f 1 ; (A : : p f ) ( ) ;

r e t u r n ( t h i s > p f ) ( ) ; }

(); (); (); ();

void foo1 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo2 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo3 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } void foo4 () { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; } i n t main ( ) { foo1 ( ) ; foo2 ( ) ; foo3 ( ) ; foo4 ( ) ; return 0; }

Figure 24.2: Example source code used as input to build call graph.

172

CHAPTER 24. GENERATING THE CALL GRAPH (CG)

::main 0 0x1025bc80

::A::f2 0 0x1025bd60

::foo1 0 0x1025bac0

::A::f1 0 0x1025bcf0

::foo2 0 0x1025bb30

::foo4 0 0x1025bc10

::foo3 0 0x1025bba0

Figure 24.3: Call graph for function in input code le: inputCode BuildCG.C.

Chapter 25

Dataow Analysis based Virtual Function Analysis

C++ Virtual function provides polymorphism to the developer but makes it dicult for compilers to do optimizations. Virtual functions are usually resolved at runtime from the vtable. Its very dicult for a compiler to know which functions will be called at compile time. ROSE provides a ow sensitive dataow analysis based approach to cut down the set of possible function calls. The code for Virtual Function Analysis is located in src/midend/programAnalysis/VirtualFunctionAnalysis/VirtualFunctionAnalysis.h. It also provides a mechanism to resolve any function calls. Its a whole program analysis and supposed to be expensive. It memorizes all the resolved function calls for any call site, so that subsequent calls are resolved faster. Figure 25.1 shows the code required to generate the pruned call graph. Using the input code shown in gure 25 Call Graph Analysis generates call graph shown in gure 25.3. Executing dataow analysis to resolve virtual function calls resulted in the gure 25.4.

173

174

CHAPTER 25. DATAFLOW ANALYSIS BASED VIRTUAL FUNCTION ANALYSIS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

#i n c l u d e s a g e 3 b a s i c . h #i n c l u d e < s t r i n g > #i n c l u d e < i o s t r e a m > #i n c l u d e V i r t u a l F u n c t i o n A n a l y s i s . h u s i n g namespace b o o s t ;

u s i n g namespace s t d ; u s i n g namespace b o o s t ; // A F u n c t i o n o b j e c t u s e d a s a p r e d i c a t e t h a t d e t e r m i n e s which f u n c t i o n s a r e // t o be r e p r e s e n t e d i n t h e c a l l graph . s t r u c t k e e p F u n c t i o n : p u b l i c u n a r y f u n c t i o n <b o o l , S g F u n c t i o n D e c l a r a t i o n >{ public : bool operator ( ) ( SgFunctionDeclaration funcDecl ){ bool returnValue = true ; ROSE ASSERT( f u n c D e c l != NULL ) ; s t r i n g filename = funcDecl > g e t f i l e i n f o ()> g e t f i l e n a m e ( ) ; // F i l t e r o u t f u n c t i o n s from t h e ROSE p r e i n c l u d e h e a d e r f i l e i f ( f i l e n a m e . f i n d ( r o s e e d g r e q u i r e d m a c r o s a n d f u n c t i o n s ) ! = s t r i n g : : npos ) returnValue = f a l s e ; // F i l t e r o u t c o m p i l e r g e n e r a t e d f u n c t i o n s i f ( funcDecl > g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ()== t r u e ) r e t u r n V a l u e= f a l s e ; return returnValue ; } }; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; CallGraphBuilder b u i l d e r ( p r o j e c t ) ; b u i l d e r . buildCallGraph ( keepFunction ( ) ) ; // G e n e r a t e c a l l graph i n d o t f o r m a t AstDOTGeneration d o t g e n ; d o t g e n . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( b u i l d e r . getGraph ( ) , o r i g i n a l c a l l g r a p h . d o t ) ; S g F u n c t i o n D e c l a r a t i o n mainDecl = S a g e I n t e r f a c e : : f i n d M a i n ( p r o j e c t ) ; i f ( mainDecl == NULL) { std : : cerr < < Can t e x e c u t e V i r t u a l F u n c t i o n A n a l y s i s w i t h o u t main f u n c t i o n \ n ; return 0; } V i r t u a l F u n c t i o n A n a l y s i s a n a l = new V i r t u a l F u n c t i o n A n a l y s i s ( p r o j e c t ) ; anal >run ( ) ; anal >p r u n e C a l l G r a p h ( b u i l d e r ) ; AstDOTGeneration d o t g e n 2 ; d o t g e n 2 . w r i t e I n c i d e n c e G r a p h T o D O T F i l e ( b u i l d e r . getGraph ( ) , p r u n e d c a l l g r a p h . d o t ) ; d e l e t e anal ;

return 0; }

Figure 25.1: Source code to perform virtual function analysis

175

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

c l a s s Animal { public : v i r t u a l v o i d s h o u t ( ) {} }; c l a s s dog : p u b l i c Animal { public : v i r t u a l v o i d s h o u t ( ) {} }; c l a s s t e r r i e r : p u b l i c dog { public : v i r t u a l v o i d s h o u t ( ) {} }; class yterrier : public t e r r i e r { public : v i r t u a l v o i d s h o u t ( ) {} }; i n t main ( v o i d ) { Animal p , q ; dog x , d ; t e r r i e r y ; y = new y t e r r i e r ; x = &d ; p = ( Animal )&x ; q = p; p = y ; x >s h o u t ( ) ; return 0; }

Figure 25.2: Example source code used as input for Virtual Function Analysis.

::main 0 0xecac4c0

::Animal::shout 0 0xecac530

::dog::shout 0 0xecac5a0

::terrier::shout 0 0xecac610

::yterrier::shout 0 0xecac680

Figure 25.3: Call graph generated by Call Graph Analysis for input code in inputCode vfa.C.

176

CHAPTER 25. DATAFLOW ANALYSIS BASED VIRTUAL FUNCTION ANALYSIS

::main 0 0xecac4c0

::Animal::shout 0 0xecac530

::dog::shout 0 0xecac5a0

::terrier::shout 0 0xecac610

::yterrier::shout 0 0xecac680

Figure 25.4: Call graph resulted from Virtual Function Analysis for input code in inputCode vfa.C.

Chapter 26

Generating the Class Hierarchy Graph

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #i n c l u d e r o s e . h #i n c l u d e CallGraph . h #i n c l u d e < b o o s t / f o r e a c h . hpp> #d e f i n e f o r e a c h BOOST FOREACH u s i n g namespace s t d ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // C o n s t r u c t c l a s s h i e r a r c h y graph ClassHierarchyWrapper h i e r ( p r o j e c t ) ; // D i s p l a y t h e a n c e s t o r s o f e a c h c l a s s v e c t o r < S g C l a s s D e f i n i t i o n > a l l C l a s s e s = S a g e I n t e r f a c e : : querySubTree < S g C l a s s D e f i n i t i o n >( p r o j e c t , foreach ( SgClassDefinition classDef , a l l C l a s se s ) { p r i n t f ( \ n%s s u b c l a s s e s : , c l a s s D e f > g e t d e c l a r a t i o n ()> get name ( ) . s t r ( ) ) ; foreach ( SgClassDefinition subclass , hier . getSubclasses ( classDef )) { p r i n t f (% s , , s u b c l a s s > g e t d e c l a r a t i o n ()> get name ( ) . s t r ( ) ) ; } } return 0; }

V SgClassDefinitio

Figure 26.1: Example source code showing visualization of class hierarchy graph. For C++, because of multiple inheritance, a class hierarchy graph is a directed graph with pointers from a class to a superclass. A superclass is a class which does not inherit from any other class. A class may inherit from a superclass by inheriting from another class which does rather than by a direct inheritance. Figure 26 shows the code required to generate the class hierarchy graph for each class of an 177

178

CHAPTER 26. GENERATING THE CLASS HIERARCHY GRAPH

application. Using the input code shown in gure 26 the rst functions call graph is shown in gure 26.3.
1 2 3 4 5 c l a s s A{ } ; class B : class C : p u b l i c A{ } ; p u b l i c B{ } ;

Figure 26.2: Example source code used as input to build class hierarchy graph. Figure 26.3 shows the class hierarchy graph for the classes in the input code in gure 26.

179

Figure 26.3: Class hierarchy graph in input code le: inputCode ClassHierarchyGraph.C.

180

CHAPTER 26. GENERATING THE CLASS HIERARCHY GRAPH

Chapter 27

Database Support
This chapter is specic to support in ROSE for persistent storage. ROSE uses the SQLite database and makes it simple to store data in the database for retrieval in later phases of processing large multiple le projects. FIXME: Need more information
here.

27.1

ROSE DB Support for Persistent Analysis

This section presents gure 27.3, a simple C++ source code using a template. It is used as a basis for showing how template instantiations are handled within ROSE. An example translator using a database connection to store function information is shown in Fig.27.1 and Fig.27.2. The output by the translator operating on the C++ source code is shown in Fig. 27.4.

27.2

Call Graph for Multi-le Application

This section shows an example of the use of the ROSE Database mechanism where information is stored after processing each le as part of generating the call graph for a project consisting of multiple les. The separate les are show in gures 27.3 and ??. These les are processed using the translator in gure ?? to generate the nal project call graph shown in gure ??.

27.3

Class Hierarchy Graph

FIXME: This example still needs to be implemented to use the new ROSE call graph generator.

This section presents a translator in gure ??, to generate the class hierarchy graph of the example shown in gure ??. The input is a multi-le application show in gure ?? and gure ??. This example is incomplete.

FIXME: This example is still incomplete.

181

182

CHAPTER 27. DATABASE SUPPORT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

// Example ROSE T r a n s l a t o r : u s e d f o r #i n c l u d e r o s e . h u s i n g namespace s t d ;

t e s t i n g ROSE i n f r a s t r u c t u r e

// DQ ( 9 / 9 / 2 0 0 5 ) : Don t i n c l u d e t h e d a t a b a s e by d e f a u l t // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL #i n c l u d e GlobalDatabaseConnectionMYSQL . h #e n d i f i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL // B u i l d t h e Data b a s e G l o b a l D a t a b a s e C o n n e c t i o n gDB ; gDB = new G l o b a l D a t a b a s e C o n n e c t i o n ( functionNameDataBase ) ; gDB >i n i t i a l i z e ( ) ; s t r i n g command = ; command = command + CREATE TABLE F u n c t i o n s ( name TEXT, c o u n t e r Query q = gDB >getQuery ( ) ; q > s e t ( command ) ; q >e x e c u t e ( ) ; ( q > s u c c e s s ( ) != 0 ) c o u t << E r r o r c r e a t i n g schema : << q > e r r o r ( ) << \ n ; // A l t e r n a t i v e s y n t a x , but d o e s n o t p e r m i t a c c e s s t o e r r o r m e s s a g e s and e x i t c o d e s // gDB >e x e c u t e ( command . c s t r ( ) ) ; #e n d i f if // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Run i n t e r n a l c o n s i s t e n c y t e s t s on AST AstTests : : runAllTests ( p r o j e c t ) ; // B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n ) ; int counter = 0; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L i s t . end ( ) ; i ++) { // B u i l d a p o i n t e r t o t h e c u r r e n t t y p e s o t h a t we can c a l l // t h e get name ( ) member f u n c t i o n . SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = i sS gFu ncti o nDecl a ra ti o n ( i ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n != NULL ) ; SgName func name = f u n c t i o n D e c l a r a t i o n >get name ( ) ; // S k i p b u i l t i n f u n c t i o n s f o r s h o r t e r output , L i a o 4 / 2 8 / 2 0 0 8 i f ( func name . g e t S t r i n g ( ) . f i n d ( b u i l t i n ,0)==0) continue ; // o u t p u t t h e f u n c t i o n number and t h e name o f t h e f u n c t i o n p r i n t f ( f u n c t i o n name #%d i s %s a t l i n e %d \ n , c o u n t e r ++,func name . s t r ( ) ,

);;

Figure 27.1: Example translator (part 1) using database connection to store function names.

27.3. CLASS HIERARCHY GRAPH

183

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

functionDeclaration > g e t f i l e i n f o ()> g e t l i n e ( ) ) ; s t r i n g functionName = f u n c t i o n D e c l a r a t i o n >g e t q u a l i f i e d n a m e ( ) . s t r ( ) ; // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL command = INSERT INTO F u n c t i o n s v a l u e s ( \ + functionName + \ , + S t r i n g U t i l i t y : : numberToString ( c o u n t e r ) + ) ; ; // A l t e r n a t i v e i n t e r f a c e // q > s e t ( command ) ; // c o u t << E x e c u t i n g : << q >p r e v i e w ( ) << \ n ; // q >e x e c u t e ( ) ; gDB >e x e c u t e ( command . c s t r ( ) ) ; #e n d i f } // TPS ( 0 1 Dec2008 ) : Enabled mysql and t h i s f a i l s . // seems l i k e i t i s n o t s u p p o s e d t o be i n c l u d e d #i f 0 //# i f d e f HAVE MYSQL command = SELECT from F u n c t i o n s ; ; // A l t e r n a t i v e I n t e r f a c e ( u s i n g q u e r y o b j e c t s ) // q << command ; q > s e t ( command ) ; c o u t << E x e c u t i n g : << q >p r e v i e w ( ) << \ n ; // e x e c u t e and r e t u r n r e s u l t ( a l t e r n a t i v e u s a g e : gDB >s e l e c t ( ) ) R e s u l t r e s = q >s t o r e ( ) ; i f ( q > s u c c e s s ( ) != 0 ) c o u t << E r r o r r e a d i n g v a l u e s : << q > e r r o r ( ) << \ n ; else { // Read t h e t a b l e r e t u r n e d from t h e q u e r y // r e s >s h o w R e s u l t ( ) ; f o r ( Result : : i t e r a t o r i = res >b e g i n ( ) ; i != r e s >end ( ) ; i++ ) { // A l t e r n a t i v e s y n t a x i s p o s s i b l e : Row r = i ; s t r i n g functionName = ( i ) [ 0 ] . g e t s t r i n g ( ) ; i n t counter = ( i ) [ 1 ] ; p r i n t f ( functionName = %s c o u n t e r = %d \ n , functionName . c s t r ( ) , c o u n t e r ) ; } } gDB >shutdown ( ) ; #e l s e p r i n t f ( Program c o m p i l e d w i t h o u t d a t a b a s e c o n n e c t i o n s u p p o r t ( add u s i n g ROSE c o n f i g u r e o p t i o n ) \ n ) ; #e n d i f return 0; }

Figure 27.2: Example translator (part 2) using database connection to store function names.

184

CHAPTER 27. DATABASE SUPPORT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

// T h i s example c o d e i s u s e d t o r e c o r d names o f f u n c t i o n s i n t o t h e d a t a b a s e . class A { public : virtual int f1 () = 0; v i r t u a l i n t f 2 ( ) {} int f3 ( ) ; virtual int f4 ( ) ; }; int A: : f3 () { f1 ( ) ; i n t A : : f 4 ( ) {} return f3 ( ) ; }

class B : public A { public : virtual int f1 ( ) ; v i r t u a l i n t f 2 ( ) {} }; i n t B : : f 1 ( ) {} class C : public A { public : v i r t u a l i n t f 1 ( ) {} i n t f 3 ( ) {} }; class D : public B { public : v i r t u a l i n t f 2 ( ) {} }; class E : public D { public : virtual int f1 () { return 5; } }; class G : public E { public : virtual int f1 ( ) ; }; i n t G : : f 1 ( ) {} class F : public D { public : v i r t u a l i n t f 1 ( ) {} virtual int f2 () { return 5;} int f3 () { return 2;} }; class H : public C { public : v i r t u a l i n t f 1 ( ) {} v i r t u a l i n t f 2 ( ) {} i n t f 3 ( ) {} };

Figure 27.3: Example source code used as input to database example.

27.3. CLASS HIERARCHY GRAPH

185

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

f u n c t i o n name #0 i s s y n c l o c k t e s t a n d s e t at l i n e 0 f u n c t i o n name #1 i s s y n c l o c k r e l e a s e at l i n e 0 f u n c t i o n name #2 i s f 1 a t l i n e 6 f u n c t i o n name #3 i s f 2 a t l i n e 7 f u n c t i o n name #4 i s f 3 a t l i n e 8 f u n c t i o n name #5 i s f 4 a t l i n e 9 f u n c t i o n name #6 i s f 3 a t l i n e 12 f u n c t i o n name #7 i s f 4 a t l i n e 13 f u n c t i o n name #8 i s f 1 a t l i n e 18 f u n c t i o n name #9 i s f 2 a t l i n e 19 f u n c t i o n name #10 i s f 1 a t l i n e 22 f u n c t i o n name #11 i s f 1 a t l i n e 27 f u n c t i o n name #12 i s f 3 a t l i n e 28 f u n c t i o n name #13 i s f 2 a t l i n e 34 f u n c t i o n name #14 i s f 1 a t l i n e 40 f u n c t i o n name #15 i s f 1 a t l i n e 46 f u n c t i o n name #16 i s f 1 a t l i n e 49 f u n c t i o n name #17 i s f 1 a t l i n e 54 f u n c t i o n name #18 i s f 2 a t l i n e 55 f u n c t i o n name #19 i s f 3 a t l i n e 56 f u n c t i o n name #20 i s f 1 a t l i n e 62 f u n c t i o n name #21 i s f 2 a t l i n e 63 f u n c t i o n name #22 i s f 3 a t l i n e 64 Program c o m p i l e d w i t h o u t d a t a b a s e c o n n e c t i o n s u p p o r t ( add u s i n g ROSE c o n f i g u r e o p t i o n )

Figure 27.4: Output from processing input code through database example dataBaseTranslator27.1.

186

CHAPTER 27. DATABASE SUPPORT

Chapter 28

Building Custom Graphs

What To Learn From This Example This example shows how to generate custom graphs using SgGraph class. Rose provides a collection type SgGraph to store a graph. Two specic graphs are also provided which are derived from SgGraph : SgIncidenceDirectedGraph and SgIncidenceUndirectedGraph. Nodes and edges in a SgGraph are represented by SgGraphNode and SgGraphEdge separately. A SgGraph is built by adding SgGraphNode s and SgGraphEdge s using its member function addNode and addEdge. You can get all nodes and edges of a SgGraph by calling its functions computeNodeSet and computeEdgeSet separately. More interfaces of SgGraph and its subclasses can be found in doxygen of Rose. Since SgGraph is for Rose use, each node in it holds a pointer to SgNode, which is the default attribute of a SgGraphNode. If you want to add more attributes inside, you can use SgGraphNode s member function addNewAttribute by providing a name and an AstAttribute object to add a new attribute to a node. Normally, you have to build your own attribute class which should be derived from class AstAttribute. Three attribute classes are provided by Rose: AstRegExAttribute, AstTextAttribute, and MetricAttribute. For more information about them, please refer to Roses doxygen.

187

188

CHAPTER 28. BUILDING CUSTOM GRAPHS

Part IV

Program Transformations and Optimizations

This part gives examples of building source-to-source program transformations and optimizations.

189

Chapter 29

Generating Unique Names for Declarations

There are many instances where a unique name must be generated for either a function or variable declaration. ROSE denes a mechanism to make the generation of unique names from all SgDeclarationStatment IR nodes and the SgInitializedName IR node. This simplies ROSEbased applications that require this sort of mechanism. Our experience has found that a significant number of tools require such a mechanism and that its correct implementation can have subtle points. The specic translator described in this chapter traverses an AST and outputs the unique names that can be generated for each declaration showing the use of the unique name generation mechanism. This tool is intended as an example of how to generate unique names using ROSE. Not all IR nodes can be used to generate a unique name. The generated names are unique under the following rules: 1. Any two generated names are the same if the declarations are the same. Declaration can be the same across les or within the same le. Declarations that are the same can have dierent location in the same le (be represented multiple times) or be in dierent les. Language constructs that are the same must follow the One-time Denition Rule (ODR) across les. 2. Declarations in dierent unnamed scopes (e.g. for loop bodies) will generate dierent names. 3. Names are the same when generated by dierent ROSE tools. Pointer values could be used to generate unique names of all IR nodes, but this would work only within a single invocation of the ROSE based tool. Generated names are not based on internal pointer values and are thus insensitive to pointer values. Generated names of the same declaration are thus the same even if generated from dierent tools. This allows multiple ROSE tools to inter-operate. This unique name generation mechanism is only applicable to specic IR nodes, specically: SgInitializedName 191

192

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS SgDeclarationStatement IR nodes: Obvious IR nodes supported: SgClassDeclaration SgFunctionDeclaration SgEnumDeclaration SgNamespaceDeclarationStatement SgTypedefDeclaration Less obvious IR nodes not supported (support for these would not make sense): SgAsmStmt SgCtorInitializerList SgFunctionParameterList SgNamespaceAliasDeclarationStatement SgPragmaDeclaration SgTemplateDeclaration (can this have a mangled name?) SgTemplateInstantiationDirectiveStatement SgUsingDeclarationStatement SgUsingDirectiveStatement SgVariableDeclaration Note that the SgVariableDeclaration contains a list of SgInitializedName nodes and the mangled names are best queried from each SgInitializedName instead of the SgVariableDeclaration. SgVariableDenition Un-named scopes A number of scopes are un-names and so there is an opportunity to generate non-unique names from declarations in such scopes. To x this we generate names for each un-named scope to guarantee uniqueness. Nodes handled are: SgForStatement SgBasicBlock SgIfStmt get the complete list ...

Other language constructs can generate unique names as well, but their name could be invalid after certain transformation that move it structurally within the generated source code.

29.1 29.2

Example Code Showing Generation of Unique Names Input For Examples Showing Unique Name Generation for Variables

Figure 29.1, shows an example translator demonstrating the generation of unique names from declarations in the AST. For each SgInitializedName we generate the mangled name. Figure 29.2

29.3. EXAMPLE OUTPUT SHOWING UNIQUE VARIABLE NAMES

193

shows the input code and gure 29.3 shows the generated output from the translator (the mangled names from the AST associated with the input application).

29.3 29.4

Example Output Showing Unique Variable Names Input For Examples Showing Unique Name Generation for Functions

Figure 29.1, shows an example translator demonstrating the generation of unique names from declarations in the AST. For each SgInitializedName we generate the mangled name. Figure 29.4 shows the input code and gure 29.5 shows the generated output from the translator (the mangled names from the AST associated with the input application).

29.5

Example Output Showing Unique Function Names

194

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

// T h i s example shows t h e g e n e r a t i o n o f u n i q u e names from d e c l a r a t i o n s . // // // // // // Mangled name demo T h i s t r a n s l a t o r q u e r i e s t h e AST f o r a l l S g I n i t i a l i z e d N a m e s and S g F u n c t i o n D e c l a r a t i o n s , and f o r e a c h one p r i n t s ( a ) t h e s o u r c e l o c a t i o n , ( b ) t h e s o u r c e name o f t h e o b j e c t , and ( c ) t h e mangled name .

#i n c l u d e < r o s e . h> u s i n g namespace s t d ; // R e t u r n s a S g F i l e I n f o o b j e c t a s a d i s p l a y f r i e n d l y s t a t i c s t r i n g toString ( const S g F i l e I n f o info ) { ostringstream i n f o s t r ; i f ( info ) i n f o s t r << [ << i n f o >g e t r a w f i l e n a m e ( ) << : << i n f o >g e t r a w l i n e ( ) << ] ; return i n f o s t r . str ( ) ; } string , [ source : l i n e ] .

// D i s p l a y s l o c a t i o n and mangled name o f an S g I n i t i a l i z e d N a m e o b j e c t . s t a t i c v o i d p r i n t I n i t i a l i z e d N a m e ( c o n s t SgNode node ) { c o n s t S g I n i t i a l i z e d N a m e name = i s S g I n i t i a l i z e d N a m e ( node ) ; ROSE ASSERT ( name != NULL ) ; if ( name > g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) c o u t // << t o S t r i n g ( name >g e t f i l e i n f o ( ) ) // << << name >get name ( ) . s t r ( ) << > << name >g e t m a n g l e d n a m e ( ) . s t r ( ) << e n d l ;

} // D i s p l a y s l o c a t i o n and mangled name o f an S g F u n c t i o n D e c l a r a t i o n o b j e c t . s t a t i c v o i d p r i n t F u n c t i o n D e c l a r a t i o n ( c o n s t SgNode node ) { c o n s t S g F u n c t i o n D e c l a r a t i o n d e c l = i s S g F u n c t i o n D e c l a r a t i o n ( node ) ; ROSE ASSERT ( d e c l != NULL ) ; if ( decl > g e t f i l e i n f o ()> i s C o m p i l e r G e n e r a t e d ( ) == f a l s e ) c o u t // << t o S t r i n g ( d e c l >g e t s t a r t O f C o n s t r u c t ( ) ) // << << d e c l >g e t q u a l i f i e d n a m e ( ) . s t r ( ) << > << d e c l >g e t m a n g l e d n a m e ( ) . s t r ( ) << e n d l ;

} i n t main ( i n t a r g c , c h a r a r g v ) { SgProject p r o j = frontend ( argc , argv ) ; c o u t << e n d l << BEGIN i n i t i a l i z e d names << e n d l ; R o s e S T L C o n t a i n e r <SgNode > i n i t n a m e s = NodeQuery : : querySubTree ( p r o j , V S g I n i t i a l i z e d N a m e ) ; f o r e a c h ( i n i t n a m e s . b e g i n ( ) , i n i t n a m e s . end ( ) , p r i n t I n i t i a l i z e d N a m e ) ; c o u t << END i n i t i a l i z e d names << e n d l ; c o u t << e n d l << BEGIN f u n c t i o n d e c l a r a t i o n s << e n d l ; R o s e S T L C o n t a i n e r <SgNode > f u n c d e c l s = NodeQuery : : querySubTree ( p r o j , V S g F u n c t i o n D e c l a r a t i o n ) ; f o r e a c h ( f u n c d e c l s . b e g i n ( ) , f u n c d e c l s . end ( ) , p r i n t F u n c t i o n D e c l a r a t i o n ) ; c o u t << END f u n c t i o n d e c l a r a t i o n s << e n d l ; r e t u r n backend ( p r o j ) ; }

Figure 29.1: Example source code showing the output of mangled name. The string represents the code associated with the subtree of the target IR node.

29.5. EXAMPLE OUTPUT SHOWING UNIQUE FUNCTION NAMES

195

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

// I n p u t int x ;

file

to t e s t mangling o f S g I n i t i a l i z e d N a m e o b j e c t s .

// G l o b a l c l a s s class A { private : int x ; // N e s t e d c l a s s class B { private : int x ; public : void foo ( i n t x arg ) { i n t x ; } }; }; t e m p l a t e <typename T> void f o o (T x a r g ) { T x; f o r ( x = 0 ; x < 1 0 ; x++) { T x = 0; do { // L o c a l c l a s s class A { private : // N e s t e d c l a s s class B { T x; }; public : v o i d f o o (T x ) {} }; T x = 0; } while (x > 0 ) ; do { T x = 0; } while (x > 0 ) ; // N e s t e d s c o p e { T x = 0; } } } t e m p l a t e v o i d f o o <i n t > ( i n t x ) ; t e m p l a t e v o i d f o o <d o u b l e > ( d o u b l e x ) ; void bar ( void ) { f o r ( i n t x = 0 ; x != 0 ; x++) f o r ( i n t x = 0 ; x != 0 ; x++) f o r ( l o n g x = 0 ; x != 0 ; x++) ; try { f o r ( i n t x = 0 ; x != 0 ; x++) ; } c a t c h ( i n t ) {} c a t c h ( c h a r x ) {} }

Figure 29.2: Example source code used as input to program in codes showing debugging techniques shown in this section.

196

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

BEGIN i n i t i a l i z e d names x > x scope x x > A x > A scope B scope x > L2R L3R ARG1 x arg x > L2R L3R scope SgSS2 scope x > foo tas i tae Fb v Gb i Fe ARG1 x arg x arg > foo tas i tae Fb v Gb i Fe ARG1 x arg > foo tas d tae Fb v Gb d Fe ARG1 x arg > foo tas d tae Fb v Gb d Fe ARG1 x > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS3 scope x x > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS3 scope SgSS4 x > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS3 scope SgSS4 x > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS6 scope SgSS7 > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS8 scope CATCHARG x > bar Fb v Gb Fe L4R scope SgSS2 scope SgSS10 scope x > foo tas i tae Fb v Gb i Fe ARG1 x arg x > foo tas i tae Fb v Gb i Fe scope SgSS2 scope x tas i tae Fb v Gb i Fe scope SgSS2 scope SgSS3 scope x > foo scope B scope x x > L0R x > L5R L6R ARG1 x > foo tas i tae Fb v Gb i Fe scope SgSS2 scope SgSS3 scope x > foo tas i tae Fb v Gb i Fe scope SgSS2 scope SgSS3 scope tas i tae Fb v Gb i Fe scope SgSS2 scope SgSS3 scope x > foo x arg > foo tas d tae Fb v Gb d Fe ARG1 tas d tae Fb v Gb d Fe scope SgSS2 scope x x > foo x > foo tas d tae Fb v Gb d Fe scope SgSS2 scope SgSS3 scope x > L1R scope B scope x x > L7R L8R ARG1 x > foo tas d tae Fb v Gb d Fe scope SgSS2 scope SgSS3 scope x > foo tas d tae Fb v Gb d Fe scope SgSS2 scope SgSS3 scope tas d tae Fb v Gb d Fe scope SgSS2 scope SgSS3 scope x > foo END i n i t i a l i z e d names BEGIN f u n c t i o n d e c l a r a t i o n s : :A: : B : : foo > L2R L3R : : foo < int > > foo tas i tae Fb : : foo < int > > foo tas i tae Fb : : foo < double > > foo tas d tae : : foo < double > > foo tas d tae : : bar > bar Fb v Gb Fe L4R tas i tae Fb : : foo < int > > foo A: : foo > L5R L6R : : foo < double > > foo tas d tae A: : foo > L7R L8R END f u n c t i o n d e c l a r a t i o n s

scope scope scope

x SgSS5 x scope x

SgSS4

scope

SgSS4 SgSS4 SgSS4

scope scope scope

SgSS5 SgSS11 SgSS13

scope scope scope

SgSS6 SgSS1 x

SgSS4

scope

SgSS4 SgSS4 SgSS4

scope scope scope

SgSS5 SgSS11 SgSS13

scope scope scope

SgSS6 SgSS1 x

v Gb i Fe v Gb i Fe Fb v Gb d Fe Fb v Gb d Fe v Gb i Fe Fb v Gb d Fe

Figure 29.3: Output of input code using generatingUniqueNamesFromDeclaration.C

29.5. EXAMPLE OUTPUT SHOWING UNIQUE FUNCTION NAMES

197

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

// I n p u t

file

to t e s t mangling o f S g F u n c t i o n D e c l a r a t i o n o b j e c t s .

long long long long long long { }

foobar ( ) ; foobar ( int foobar ( int foobar ( int foobar ( int foobar ( int

); y); x); x = 0); xyz )

r e t u r n xyz ;

char foobarChar ( char ) ; char foobarChar ( char c ) ; // I n p u t file to t e s t mangling o f S g F u n c t i o n D e c l a r a t i o n o b j e c t s .

typedef int value0 t ; typedef value0 t value t ; namespace N { typedef struct { int a ; } s t ; c l a s s A { p u b l i c : A ( ) {} v i r t u a l v o i d f o o ( i n t ) {} } ; c l a s s B { p u b l i c : B ( ) {} v o i d f o o ( v a l u e t ) c o n s t {} } ; c l a s s C : p u b l i c A { p u b l i c : C ( ) {} v o i d f o o ( i n t ) {} v o i d f o o ( c o n s t v o i d f o o ( c o n s t s t ) {} } typedef N: : s t s2 t ; void foo ( v a l u e t ) ; v o i d f o o ( s 2 t ) {} v o i d f o o ( f l o a t x [ ] ) {} void foo ( value t , s 2 t ) ; t e m p l a t e <typename T> v o i d f o o (T) {} namespace P { typedef long double t y p e t ; namespace Q { t e m p l a t e <typename T> v o i d f o o (T) {} class R { public : R ( ) {} t e m p l a t e <typename T> v o i d f o o (T) {} v o i d f o o (P : : t y p e t ) {} t e m p l a t e <typename T, i n t x> i n t f o o (T) { r e t u r n x ; } }; } } t e m p l a t e <typename T, i n t x> i n t f o o (T) { r e t u r n x ; } template template template template void void void void f o o <char > ( c h a r ) ; f o o <c o n s t v a l u e t > ( c o n s t v a l u e t P : : Q : : f o o <l o n g > ( l o n g ) ; P : : Q : : R : : f o o <v a l u e t > ( v a l u e t ) ; );

s t &) {} } ;

Figure 29.4: Example source code used as input to program in codes showing debugging techniques shown in this section.

198

CHAPTER 29. GENERATING UNIQUE NAMES FOR DECLARATIONS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

BEGIN i n i t i a l i z e d names > foobar Fb l Gb i Fe L4R ARG1 Fb l Gb i Fe L4R ARG1 y > foobar x > foobar Fb l Gb i Fe L4R ARG1 x > foobar Fb l Gb i Fe L4R ARG1 Fb l Gb i Fe L4R ARG1 xyz > foobar Fb c Gb c Fe L5R ARG1 > foobarChar Fb c Gb c Fe L5R ARG1 c > foobarChar a > L2R scope a > L6R L7R ARG1 > L8R L9R ARG1 > L10R L11R ARG1 > L12R L13R ARG1 > L14R L15R ARG1 Fb v Gb L0R Fe L16R ARG1 > foo > foo Fb v Gb L3R Fe L17R ARG1 x > L18R L19R ARG1 > foo Fb v Gb L0R sep L3R Fe L20R ARG1 Fb v Gb L0R sep L3R Fe L20R ARG2 > foo > L21R L22R ARG1 > foo tas c tae Fb v Gb c Fe ARG1 > foo tas c tae Fb v Gb c Fe ARG1 > L23R ARG1 > L23R ARG1 > L24R ARG1 > L24R ARG1 > L25R ARG1 > L25R ARG1 tas c tae Fb v Gb c Fe ARG1 > foo > L23R ARG1 > L24R ARG1 > L25R ARG1 END i n i t i a l i z e d names BEGIN f u n c t i o n d e c l a r a t i o n s : : foobar > foobar Fb l Gb Fe L26R : : foobar > foobar Fb l Gb i Fe L4R Fb l Gb i Fe L4R : : foobar > foobar : : foobar > foobar Fb l Gb i Fe L4R : : foobar > foobar Fb l Gb i Fe L4R : : foobar > foobar Fb l Gb i Fe L4R : : foobarChar > foobarChar Fb c Gb c Fe L5R : : foobarChar > foobarChar Fb c Gb c Fe L5R : :N: :A: :A > L27R L28R : :N: :A: : foo > L6R L7R : :N: :B: :B > L29R L30R : :N: : B : : foo > L8R L9R : :N: :C: :C > L31R L32R : :N: :C: : foo > L10R L11R : :N: :C: : foo > L12R L13R : :N: : foo > L14R L15R : : foo > foo Fb v Gb L0R Fe L16R : : foo > foo Fb v Gb L3R Fe L17R : : foo > L18R L19R : : foo > foo Fb v Gb L0R sep L3R Fe L20R : : P : :Q: :R: :R > L33R L34R : : P : :Q: :R: : foo > L21R L22R tas c tae Fb v Gb c Fe : : foo < char > > foo : : foo < char > > foo tas c tae Fb v Gb c Fe : : foo < const value t > > L23R : : foo < const value t > > L23R : : P : :Q: : foo < long > > L24R : : P : :Q: : foo < long > > L24R : : P : :Q: :R: : foo < value t > > L25R : : P : :Q: :R: : foo < value t > > L25R : : foo < char > > foo tas c tae Fb v Gb c Fe : : foo < const value t > > L23R : : P : :Q: : foo < long > > L24R : : P : :Q: :R: : foo < value t > > L25R END f u n c t i o n d e c l a r a t i o n s

Figure 29.5: Output of input code using generatingUniqueNamesFromDeclaration.C

Chapter 30

Command-line Processing Within Translators

ROSE includes mechanism to simplify the processing of command-line arguments so that translators using ROSE can trivially replace compilers within makeles. This example shows some of the many command-line handling options within ROSE and the ways in which customized options may be added for specic translators.

30.1

Commandline Selection of Files

Overview This example shows the optional processing of specic les selected after the call to the frontend to build the project. First the SgProject if build and then the les are selected for processing via ROSE or the backend compiler directly. This example demonstrates the separation of the construction of a SgProject with valid SgFile objects for each le on the command line, but with an empty SgGlobal scope, and the call to the frontend, called for each SgFile in a separate loop over all the SgFile objects.

199

200

CHAPTER 30. COMMAND-LINE PROCESSING WITHIN TRANSLATORS

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h u s i n g namespace s t d ; int main ( i n t a r g c , c h a r a r g v [ ] ) { R o s e S T L C o n t a i n e r <s t r i n g > l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( P r e p r o c e s s o r ( b e f o r e ) : a r g v = \ n%s \ n , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // Remove c e r t a i n s o r t s o f o p t i o n s from t h e command l i n e C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l , edg : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l , edg : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l , e d g p a r a m e t e r : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l , e d g p a r a m e t e r : ) ; // Add a t e s t f o r a custom command l i n e o p t i o n int integerOptionForVerbose = 0; i f ( C o m m a n d l i n e P r o c e s s i n g : : i s O p t i o n W i t h P a r a m e t e r ( l , m y T r a n s l a t o r : , ( v | v e r b o s e ) , i n t e g e r O p t i o n F o r V e r b o s { p r i n t f ( Turning on my t r a n s l a t o r s v e r b o s e mode ( s e t t o %d ) \ n , i n t e g e r O p t i o n F o r V e r b o s e ) ; } // Adding a new command l i n e p a r a m e t e r ( f o r mechanisms i n ROSE t h a t t a k e command l i n e s ) // p r i n t f ( a r g c = %zu \ n , l . s i z e ( ) ) ; // l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( l . s i z e ( ) = %zu \ n , l . s i z e ( ) ) ; p r i n t f ( P r e p r o c e s s o r ( a f t e r ) : a r g v = \ n%s \ n , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // // // // SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; G e n e r a t e t h e s o u r c e c o d e and c o m p i l e u s i n g t h e vendor s c o m p i l e r r e t u r n backend ( p r o j e c t ) ; g e n e r a t e t h e s o u r c e c o d e and c a l l t h e backend c o m p i l e r ...

// B u i l d t h e AST, frontend ( l ) ; return 0; }

Figure 30.1: Example source code showing simple command-line processing within ROSE translator.

1 2 3 4 5 6

P r e p r o c e s s o r ( b e f o r e ) : argv = / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x /ROSE b u i l d / t u t o r i a l / . l i b s / l t Turning on my t r a n s l a t o r s v e r b o s e mode ( s e t t o 4 2 ) l . size () = 4 P r e p r o c e s s o r ( a f t e r ) : argv = / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x /ROSE b u i l d / t u t o r i a l / . l i b s / l t

Figure 30.2: Output of input code using commandlineProcessing.C

30.1. COMMANDLINE SELECTION OF FILES

201

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // r o s e . C : Example ( d e f a u l t ) ROSE P r e p r o c e s s o r : u s e d f o r t e s t i n g ROSE i n f r a s t r u c t u r e #i n c l u d e r o s e . h u s i n g namespace s t d ; int main ( i n t a r g c , c h a r a r g v [ ] ) { R o s e S T L C o n t a i n e r <s t r i n g > l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( P r e p r o c e s s o r ( b e f o r e ) : a r g v = \ n%s \ n , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // Remove c e r t a i n s o r t s o f o p t i o n s from t h e command l i n e C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l , edg : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgs ( l , edg : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l , e d g p a r a m e t e r : ) ; C o m m a n d l i n e P r o c e s s i n g : : removeArgsWithParameters ( l , e d g p a r a m e t e r : ) ; // Add a t e s t f o r a custom command l i n e o p t i o n int integerOptionForVerbose = 0; i f ( C o m m a n d l i n e P r o c e s s i n g : : i s O p t i o n W i t h P a r a m e t e r ( l , m y T r a n s l a t o r : , ( v | v e r b o s e ) , i n t e g e r O p t i o n F o r V e r b o s e , t r u e ) ) { p r i n t f ( Turning on my t r a n s l a t o r s v e r b o s e mode ( s e t t o %d ) \ n , i n t e g e r O p t i o n F o r V e r b o s e ) ; } // Adding a new command l i n e p a r a m e t e r ( f o r mechanisms i n ROSE t h a t t a k e command l i n e s ) // p r i n t f ( a r g c = %zu \ n , l . s i z e ( ) ) ; // l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; p r i n t f ( l . s i z e ( ) = %zu \ n , l . s i z e ( ) ) ; p r i n t f ( P r e p r o c e s s o r ( a f t e r ) : a r g v = \ n%s \ n , S t r i n g U t i l i t y : : l i s t T o S t r i n g ( l ) . c s t r ( ) ) ; // // // // SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; G e n e r a t e t h e s o u r c e c o d e and c o m p i l e u s i n g t h e vendor s c o m p i l e r r e t u r n backend ( p r o j e c t ) ; g e n e r a t e t h e s o u r c e c o d e and c a l l t h e backend c o m p i l e r ...

// B u i l d t h e AST, frontend ( l ) ; return 0; }

Figure 30.3: Example source code showing simple command-line processing within ROSE translator.

1 2 3 4 5 6

P r e p r o c e s s o r ( b e f o r e ) : argv = / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x /ROSE b u i l d / t u t o r i a l / . l i b s / l t c o m m a n d l i n Turning on my t r a n s l a t o r s v e r b o s e mode ( s e t t o 4 2 ) l . size () = 4 P r e p r o c e s s o r ( a f t e r ) : argv = / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x /ROSE b u i l d / t u t o r i a l / . l i b s / l t c o m m a n d l i n

Figure 30.4: Output of input code using commandlineProcessing.C

202

CHAPTER 30. COMMAND-LINE PROCESSING WITHIN TRANSLATORS

Chapter 31

Tailoring The Code Generation Format

Figure 31.1 shows an example of how to use the mechanisms in ROSE to tailor the format and style of the generated code. This chapter presents an example translator that modies the formatting of the code that is generated within ROSE. The details of functionality are hidden from the user and a high level interface is provided that permits key parameters to be specied. This example will be made more sophisticated later, for now it just modies the indentation of nested code blocks (from 2 spaces/block to 5 spaces/block).

31.1

Source Code for Example that Tailors the Code Generation

Figure 31.1 shows an example translator which calls the inliner mechanism. The code is designed to only inline up to ten functions. the list of function calls is recomputed after any function call is successfully inlined. The input code is shown in gure 31.2, the output of this code is shown in gure 31.3.

31.2

Input to Demonstrate Tailoring the Code Generation

Figure 31.2 shows the example input used for demonstration of how to control the formatting of generated code.

31.3

Final Code After Tailoring the Code Generation

Figure 31.3 shows the results from changes to the formatting of generated code.

203

204

CHAPTER 31. TAILORING THE CODE GENERATION FORMAT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

// T h i s example w i l l be made more s o p h i s t i c a t e d l a t e r , f o r now i t j u s t // m o d i f i e s t h e i n d e n t a t i o n o f n e s t e d c o d e b l o c k s ( from 2 s p a c e s / b l o c k // t o 5 s p a c e s / b l o c k ) . #i n c l u d e r o s e . h #i n c l u d e unparseFormatHelp . h c l a s s CustomCodeFormat : p u b l i c UnparseFormatHelp { public : CustomCodeFormat ( ) ; CustomCodeFormat ( ) ; virtual virtual i n t g e t L i n e ( SgLocatedNode , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) ; i n t g e t C o l ( SgLocatedNode , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) ;

// r e t u r n t h e v a l u e f o r i n d e n t a t i o n o f c o d e ( p a r t o f c o n t r o l o v e r s t y l e ) v i r t u a l i n t tabIndent ( ) ; // r e t u r n t h e v a l u e f o r where l i n e wrapping s t a r t s ( p a r t o f c o n t r o l o v e r s t y l e ) v i r t u a l i n t maxLineLength ( ) ; private : int defaultLineLength ; int defaultIndentation ; };

CustomCodeFormat : : CustomCodeFormat ( ) { // d e f a u l t v a l u e s h e r e ! defaultLineLength = 20; defaultIndentation = 5; } CustomCodeFormat : : CustomCodeFormat ( ) {} // r e t u r n : > 0 : s t a r t new l i n e s ; == 0 : u s e same l i n e ; < 0 : d e f a u l t int CustomCodeFormat : : g e t L i n e ( SgLocatedNode , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) { // Use d e f a u l t mechanism t o s e l e c t t h e l i n e where t o o u t p u t g e n e r a t e d c o d e r e t u r n 1; } // r e t u r n s t a r t i n g column . i f < 0 , u s e d e f a u l t int CustomCodeFormat : : g e t C o l ( SgLocatedNode , S g U n p a r s e I n f o& i n f o , FormatOpt o p t ) { // Use d e f a u l t mechanism t o s e l e c t t h e column where t o o u t p u t g e n e r a t e d c o d e r e t u r n 1; } int CustomCodeFormat : : t a b I n d e n t ( ) { // Modify t h e i n d e n t a t i o n o f t h e g e n e r a t e d c o d e ( t r i v a l example o f return defaultIndentation ; } int CustomCodeFormat : : maxLineLength ( ) { return defaultLineLength ; } i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; CustomCodeFormat f o r m a t C o n t r o l = new CustomCodeFormat ( ) ; r e t u r n backend ( p r o j e c t , f o r m a t C o n t r o l ) ; }

t a i l o r i n g code g e n e r a t i o n )

Figure 31.1: Example source code showing how to tailor the code generation format.

31.3. FINAL CODE AFTER TAILORING THE CODE GENERATION

205

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

e x t e r n i n t min ( i n t

, int

);

v o i d dgemm( d o u b l e a , d o u b l e b , d o u b l e c , i n t n ) { int var 1 ; int var 0 ; int i ; int j ; int k ; for ( var 1 = 0; v a r 1 <= 1 + n ; v a r 1 += 1 6 ) { v a r 0 <= 1 + n ; v a r 0 += 1 6 ) { for ( var 0 = 0; f o r ( i = 0 ; i <= 1 + n ; i += 1 ) { f o r ( k = v a r 1 ; k <= min( 1 + n , v a r 1 + 1 5 ) ; k += 1 ) { i n t dummy 1 = k n + i ; f o r ( j = v a r 0 ; j <= min ( n + 16 , v a r 0 ) ; j += 1 6 ) { int var 2 = ( j ); c [ j n + i ] = c[ j n + i ] + a[k n + i ] b[ j n var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] var 2 = 1 + var 2 ; c [ var 2 n + i ] = c [ var 2 n + i ] + a [ k n + i ] } f o r ( ; j <= min( 1 + n , v a r 0 + 1 5 ) ; j += 1 ) { c [ j n + i ] = c[ j n + i ] + a[k n + i ] b[ j n } } } } } }

+ k]; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ; b [ var 2 n + k ] ;

+ k];

Figure 31.2: Example source code used as input to program to the tailor the code generation.

206

CHAPTER 31. TAILORING THE CODE GENERATION FORMAT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56

i n t min ( i n t

, int

);

v o i d dgemm( d o u b l e a , d o u b l e b , d o u b l e c , i n t n ) { int var 1 ; int var 0 ; int i ; int j ; int k ; for ( var 1 = 0; v a r 1 <= ( 1 + n ) ; v a r 1 += 1 6 ) { v a r 0 <= ( 1 + n ) ; v a r 0 += 1 6 ) { for ( var 0 = 0; f o r ( i = 0 ; i <= ( 1 + n ) ; i += 1 ) { f o r ( k = v a r 1 ; k <= min (( 1 + n ) , ( v a r 1 + 1 5 ) ) ; k += 1 ) { i n t dummy 1 = ( ( k n ) + i ) ; f o r ( j = v a r 0 ; j <= min ( ( n + 16) , v a r 0 ) ; j += 1 6 ) { int var 2 = j ; c [ ( j n) + i ] = ( c [ ( j n) + i ] + (a [ ( k n) + i ] b [ ( j n) v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] v a r 2 = (1 + v a r 2 ) ; c [ ( var 2 n) + i ] = ( c [ ( var 2 n) + i ] + (a [ ( k n) + i ] } f o r ( ; j <= min (( 1 + n ) , ( v a r 0 + 1 5 ) ) ; j += 1 ) { c [ ( j n) + i ] = ( c [ ( j n) + i ] + (a [ ( k n) + i ] b [ ( j n) } } } } } }

+ k])); b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ; b [ ( var 2 n) + k ] ) ) ;

+ k]));

Figure 31.3: Output of input code after changing the format of the generated code.

Chapter 32

AST Construction
AST construction is a fundamental operation needed for building ROSE source-to-source translators. Several levels of interfaces are available in ROSE for users to build AST from scratch. High level interfaces are recommended to use whenever possible for their simplicity. Low level interfaces can give users the maximum freedom to manipulate some details in AST trees. This chapter uses several examples to demonstrate how to create AST fragments for common language constructs (such as variable declarations, functions, function calls, etc.) and how to insert them into an existing AST tree. More examples of constructing AST using high level interfaces can be found at rose/tests/roseTests/astInterfaceTests. The source les of the high level interfaces are located in rose/src/frontend/SageIII/sageInterface.

32.1

Variable Declarations

What To Learn Two examples are given to show how to construct a SAGE III AST subtree for a variable declaration and its insertion into the existing AST tree. Example 1. Building a variable declaration using the high level AST construction and manipulation interfaces dened in namespace SageBuilder and SageInterface. Figure 32.1 shows the high level construction of an AST fragment (a variable declaration) and its insertion into the AST at the top of each block. buildVariableDeclaration() takes the name and type to build a variable declaration node. prependStatement() inserts the declaration at the top of a basic block node. Details for parent and scope pointers, symbol tables, source le position information and so on are handled transparently. Example 2. Building the variable declaration using low level member functions of SAGE III node classes. Figure 32.2 shows the low level construction of the same AST fragment (for the same variable declaration) and its insertion into the AST at the top of each block. SgNode constructors and their member functions are used. Side eects for scope, parent pointers and symbol tables have to be handled by programmers explicitly. 207

208

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

// S a g e B u i l d e r c o n t a i n s a l l h i g h l e v e l buildXXX ( ) f u n c t i o n s , // s u c h a s b u i l d V a r i a b l e D e c l a r a t i o n ( ) , b u i l d L a b e l S t a t e m e n t ( ) e t c . // S a g e I n t e r f a c e c o n t a i n s h i g h l e v e l AST m a n i p u l a t i o n and u t i l i t y f u n c t i o n s , // e . g . appendStatement ( ) , l o o k u p F u n c t i o n S y m b o l I n P a r e n t S c o p e s ( ) e t c . #i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; c l a s s SimpleInstrumentation : public SgSimpleProcessing { public : v o i d v i s i t ( SgNode astNode ) ; }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode astNode ) { S g B a s i c B l o c k b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { SgVariableDeclaration variableDeclaration = b u i l d V a r i a b l e D e c l a r a t i o n ( newVariable , buildIntType prependStatement ( v a r i a b l e D e c l a r a t i o n , block ) ; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

());

preorder ) ;

Figure 32.1: AST construction and insertion for a variable using the high level interfaces

32.1. VARIABLE DECLARATIONS

209

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

// // // // //

ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e a t t h e t o p and bottom o f e a c h b a s i c b l o c k . Member f u n c t i o n s o f SAGE I I I AST node c l a s s e s a r e d i r e c t l y u s e d . So a l l d e t a i l s f o r S g F i l e I n f o , s c o p e , p a r e n t , symbol t a b l e s have t o be e x p l i c i t l y h a n d l e d .

#i n c l u d e r o s e . h class { }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode astNode ) { S g B a s i c B l o c k b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { // Mark t h i s a s a t r a n s f o r m a t i o n ( r e q u i r e d ) Sg File Info sourceLocation = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; ROSE ASSERT( s o u r c e L o c a t i o n != NULL ) ; SgType t y p e = new SgTypeInt ( ) ; ROSE ASSERT( t y p e != NULL ) ; SgName name = n e w V a r i a b l e ; S g V a r i a b l e D e c l a r a t i o n v a r i a b l e D e c l a r a t i o n = new S g V a r i a b l e D e c l a r a t i o n ( s o u r c e L o c a t i o n , name , t y p e ) ; ROSE ASSERT( v a r i a b l e D e c l a r a t i o n != NULL ) ; SgInitializedName i n i t i a l i z e d N a m e = ( v a r i a b l e D e c l a r a t i o n >g e t v a r i a b l e s ( ) . b e g i n ( ) ) ; // DQ ( 6 / 1 8 / 2 0 0 7 ) : The u n p a r s e r r e q u i r e s t h a t t h e s c o p e be s e t ( f o r name q u a l i f i c a t i o n t o work ) . initializedName >s e t s c o p e ( b l o c k ) ; // L i a o ( 2 / 1 3 / 2 0 0 8 ) : A s t T e s t s r e q u i r e s t h i s t o be s e t variableDeclaration >s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( v a r i a b l e D e c l a r a t i o n ) ; ROSE ASSERT( b l o c k >g e t s t a t e m e n t s ( ) . s i z e ( ) > 0 ) ; block >g e t s t a t e m e n t s ( ) . i n s e r t ( b l o c k >g e t s t a t e m e n t s ( ) . b e g i n ( ) , v a r i a b l e D e c l a r a t i o n ) ; variableDeclaration >s e t p a r e n t ( b l o c k ) ; // Add a symbol t o t h e s y b o l t a b l e f o r t h e new v a r i a b l e S g V a r i a b l e S y m b o l v a r i a b l e S y m b o l = new S g V a r i a b l e S y m b o l ( i n i t i a l i z e d N a m e ) ; block > i n s e r t s y m b o l ( name , v a r i a b l e S y m b o l ) ; } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; } SimpleInstrumentation : public : void public SgSimpleProcessing

visit

( SgNode astNode ) ;

preorder

);

Figure 32.2: Example source code to read an input program and add a new variable declaration at the top of each block.

210

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9

i n t main ( ) { f o r ( i n t i =0; i < 4 ; { int x ; } return 0; }

i ++)

Figure 32.3: Example source code used as input to the translators adding new variable.
1 2 3 4 5 6 7 8 9 10

i n t main ( ) { i n t newVariable ; for ( int i = 0; i < 4; i n t newVariable ; int x ; } return 0; }

i ++) {

Figure 32.4: Output of input to the translators adding new variable. Figure 32.3 shows the input code used to test the translator. Figure 32.4 shows the resulting output.

32.2. EXPRESSIONS

211

32.2

Expressions

Figure 32.5 shows a translator using the high level AST builder interface to add an assignment statement right before the last statement in a main() function. Figure 32.6 shows the input code used to test the translator. Figure 32.7 shows the resulting output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 // E x p r e s s i o n s can be b u i l t u s i n g both bottomup ( recommended ) and topdown o r d e r s . // Bottomup : b u i l d o p e r a n d s f i r s t , o p e r a t i o n l a t e r // Topdown : b u i l d o p e r a t i o n f i r s t , s e t o p e r a n d s l a t e r on . #i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; // go t o t h e f u n c t i o n body S g F u n c t i o n D e c l a r a t i o n mainFunc= f i n d M a i n ( p r o j e c t ) ; S g B a s i c B l o c k body= mainFunc > g e t d e f i n i t i o n ()> g e t b o d y ( ) ; p u s h S c o p e S t a c k ( body ) ; // bottomup : b u i l d o p e r a n d s f i r s t , c r e a t e e x p r e s s i o n l a t e r on // d o u b l e r e s u l t = 2 ( 1 gama gama ) ; SgExpression i n i t e x p = buildMultiplyOp ( buildDoubleVal ( 2 . 0 ) , buildSubtractOp ( buildDoubleVal ( 1 . 0 ) , b u i l d M u l t i p l y O p ( buildVarRefExp ( gama ) , buildVarRefExp ( gama ) ))); S g V a r i a b l e D e c l a r a t i o n d e c l = b u i l d V a r i a b l e D e c l a r a t i o n ( r e s u l t , buildDoubleType ( ) , b u i l d A s s i g n I n i t i a l i z e r ( i n i t e x p ) ) ; S gS tatement l a s t s t m t = g e t L a s t S t a t e m e n t ( t o p S c o p e S t a c k ( ) ) ; insertStatementBefore ( laststmt , decl ) ; // topdown : b u i l d e x p r e s s i o n f i r s t , s e t o p e r a n d s l a t e r on // d o u b l e r e s u l t 2 = a l p h a b e t a ; SgExpression i n i t e x p 2 = buildMultiplyOp ( ) ; set L hsOperand ( i n i t e x p 2 , buildVarRefExp ( a l p h a ) ) ; setRhsOperand ( i n i t e x p 2 , buildVarRefExp ( b e t a ) ) ; S g V a r i a b l e D e c l a r a t i o n d e c l 2 = b u i l d V a r i a b l e D e c l a r a t i o n ( r e s u l t 2 , buildDoubleType ( ) , b u i l d A s s i g n I n i t i a l i z e r ( i n i t e x p 2 ) ) ; l a s t s t m t = getLastStatement ( topScopeStack ( ) ) ; insertStatementBefore ( laststmt , decl2 ) ; popScopeStack ( ) ; AstTests : : runAllTests ( p r o j e c t ) ; // i n v o k e backend c o m p i l e r t o g e n e r a t e o b j e c t / b i n a r y r e t u r n backend ( p r o j e c t ) ; } files

Figure 32.5: Example translator to add expressions

212

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8

i n t main ( ) { d o u b l e a l p h a= 0 . 5 ; double beta = 0 . 1 ; d o u b l e gama = 0 . 7 ; return 0; }

Figure 32.6: Example source code used as input

1 2 3 4 5 6 7 8 9 10

i n t main ( ) { double alpha = 0 . 5 ; double beta = 0 . 1 ; d o u b l e gama = 0 . 7 ; d o u b l e r e s u l t = 2 . 0 0 0 0 0 ( 1 . 0 0 0 0 0 gama gama ) ; double r e s u l t 2 = alpha beta ; return 0; }

Figure 32.7: Output of the input

32.3. ASSIGNMENT STATEMENTS

213

32.3

Assignment Statements

Figure 32.8 shows a translator using the high level AST builder interface to add an assignment statement right before the last statement in a main() function. Figure 32.9 shows the input code used to test the translator. Figure 32.10 shows the resulting output.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 // S a g e B u i l d e r c o n t a i n s a l l h i g h l e v e l buildXXX ( ) f u n c t i o n s , // s u c h a s b u i l d V a r i a b l e D e c l a r a t i o n ( ) , b u i l d L a b e l S t a t e m e n t ( ) e t c . // S a g e I n t e r f a c e c o n t a i n s h i g h l e v e l AST m a n i p u l a t i o n and u t i l i t y f u n c t i o n s , // e . g . appendStatement ( ) , l o o k u p F u n c t i o n S y m b o l I n P a r e n t S c o p e s ( ) e t c . #i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; // go t o t h e f u n c t i o n body o f main ( ) // and push i t t o t h e s c o p e s t a c k S g F u n c t i o n D e c l a r a t i o n mainFunc= f i n d M a i n ( p r o j e c t ) ; S g B a s i c B l o c k body= mainFunc > g e t d e f i n i t i o n ()> g e t b o d y ( ) ; p u s h S c o p e S t a c k ( body ) ; // b u i l d a v a r i a b l e a s s i g n m e n t s t a t e m e n t : i =9; // buildVarRefExp ( s t r i n g varName ) w i l l a u t o m a t i c a l l y s e a r c h f o r a matching v a r i a b l e symbol s t a r t i n g // from t h e c u r r e n t s c o p e t o t h e g l o b a l s c o p e . SgExprStatement a s s i g n S t m t = b u i l d A s s i g n S t a t e m e n t ( buildVarRefExp ( i ) , b u i l d I n t V a l ( 9 ) ) ; // i n s e r t i t b e f o r e t h e l a s t r e t u r n s t a t e m e n t S gS tatement l a s t S t m t = g e t L a s t S t a t e m e n t ( t o p S c o p e S t a c k ( ) ) ; insertStatementBefore ( lastStmt , assignStmt ) ; popScopeStack ( ) ; // A s t T e s t s e n s u r e s t h e r e i s no d a n g l i n g SgVarRefExp w i t h o u t a mathing symbol AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 32.8: Example source code to add an assignment statement

1 2 3 4 5

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { int i ; return 0; }

Figure 32.9: Example source code used as input

214

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { int i ; i = 9; return 0; }

Figure 32.10: Output of the input

32.4. FUNCTIONS

215

32.4

Functions

This section shows how to add a function at the top of a global scope in a le. Again, examples for both high level and low level constructions of AST are given. Figure 32.11 shows the high level construction of a dening function (a function with a function body). Scope information is passed to builder functions explicitly when it is needed.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 // T h i s e x a m p l e sh ow s how t o c o n s t r u c t a d e f i n i n g // u s i n g h i g h l e v e l AST c o n s t r u c t i o n i n t e r f a c e s . // #i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; class { SimpleInstrumentation public : void }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode a s t N o d e ) { SgGlobal g l o b a l S c o p e = i s S g G l o b a l ( astNode ) ; i f ( g l o b a l S c o p e != NULL) { // // C r e a t e a p a r a m e t e r l i s t w i t h a p a r a m e t e r // SgName v a r 1 n a m e = v a r n a m e ; SgReferenceType r e f t y p e = buildReferenceType ( buildIntType ( ) ) ; S g I n i t i a l i z e d N a m e v a r 1 i n i t n a m e = b u i l d I n i t i a l i z e d N a m e ( var1 name , r e f t y p e ) ; SgFunctionParameterList parameterList = buildFunctionParameterList ( ) ; appendArg ( p a r a m e t e r L i s t , v a r 1 i n i t n a m e ) ; // // C r e a t e a d e f i n i n g f u n c t i o n D e c l a r a t i o n ( w i t h a f u n c t i o n body ) // SgName f u n c n a m e = my function ; SgFunctionDeclaration func = buildDefiningFunctionDeclaration ( func name , b u i l d I n t T y p e ( ) , p a r a m e t e r L i s t , g l o b a l S c o p e ) ; SgBasicBlock func body = func > g e t d e f i n i t i o n ()> g e t b o d y ( ) ; // // I n s e r t a s t a t e m e n t i n t h e f u n c t i o n body // SgVarRefExp v a r r e f = b u i l d V a r R e f E x p ( v ar 1 n a m e , f u n c b o d y ) ; SgPlusPlusOp p p e x p r e s s i o n = b u i l d P l u s P l u s O p ( v a r r e f ) ; SgExprStatement new stmt = buildExprStatement ( p p e x p r e s s i o n ) ; // i n s e r t a s t a t e m e n t i n t o t h e f u n c t i o n body p r e p e n d S t a t e m e n t ( n ew stm t , f u n c b o d y ) ; prependStatement ( func , g l o b a l S c o p e ) ; : public function ( with a function body )

SgSimpleProcessing

visit

SgNode a s t N o d e

);

} } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

preorder

);

Figure 32.11: Addition of function to global scope using high level interfaces

Figure 32.12 shows almost the same high level construction of the dening function, but

216

CHAPTER 32. AST CONSTRUCTION with an additional scope stack. Scope information is passed to builder functions implicitly when it is needed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

// T h i s e x a m p l e sh ow s how t o c o n s t r u c t a d e f i n i n g f u n c t i o n ( w i t h // u s i n g h i g h l e v e l AST c o n s t r u c t i o n i n t e r f a c e s . // A s c o p e s t a c k i s u s e d t o p a s s s c o p e i n f o r m a t i o n i m p l i c i t l y t o #i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SgGlobal globalSc ope = g e t F i r s t G l o b a l S c o p e // push g l o b a l s c o p e i n t o s t a c k pushScopeStack ( isSgScopeStatement //

function

body ) functions

some

builder

( project );

( globalScope ) ) ;

Create a parameter l i s t with a parameter SgName v a r 1 n a m e = v a r n a m e ; SgReferenceType r e f t y p e = buildReferenceType ( buildIntType ( ) ) ; S g I n i t i a l i z e d N a m e v a r 1 i n i t n a m e = b u i l d I n i t i a l i z e d N a m e ( var1 name , r e f t y p e ) ; SgFunctionParameterList parameterList = buildFunctionParameterList ( ) ; appendArg ( p a r a m e t e r L i s t , v a r 1 i n i t n a m e ) ; C r e a t e a d e f i n i n g f u n c t i o n D e c l a r a t i o n ( w i t h a f u n c t i o n body ) = my function ; SgName f u n c n a m e SgFunctionDeclaration func = buildDefiningFunctionDeclaration ( func name , b u i l d I n t T y p e ( ) , p a r a m e t e r L i s t ) ; SgBasicBlock func body = func > g e t d e f i n i t i o n ()> g e t b o d y ( ) ; push f u n c t i o n body s c o p e i n t o s t a c k pushScopeStack ( isSgScopeStatement ( func body ) ) ; b u i l d a s t a t e m e n t i n t h e f u n c t i o n body SgVarRefExp v a r r e f = b u i l d V a r R e f E x p ( v a r 1 n a m e ) ; SgPlusPlusOp p p e x p r e s s i o n = b u i l d P l u s P l u s O p ( v a r r e f ) ; SgExprStatement new stmt = buildExprStatement ( p p e x p r e s s i o n ) ; i n s e r t a statement in to the fu n c t io n appendStatement ( new stmt ) ; pop f u n c t i o n body o f f t h e s t a c k popScopeStack ( ) ; i n s e r t the f u n c t i o n d e c l a r a t i o n prependStatement ( func ) ; popScopeStack ( ) ; AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; into body

//

//

//

// //

//

the

scope

at

the

top

of

the

scope

stack

Figure 32.12: Addition of function to global scope using high level interfaces and a scope stack The low level construction of the AST fragment of the same function declaration and its insertion is separated into two portions and shown in two gures (Figure 32.13 and Figure 32.14 ). Figure 32.29 and Figure 32.30 give the input code and output result for the translators above.

32.4. FUNCTIONS
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 // ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t sh ow s t h e d e s i g n o f a t r a n s f o r m a t i o n // a t t h e t o p o f t h e s o u r c e f i l e . #i n c l u d e r o s e . h is to an e x a m p l e instrument preprocessor s o u r c e code ,

217
b u i l t w i t h ROSE . p l a c i n g s o u r c e code

#d e f i n e TRANSFORMATION FILE INFO S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) class { SimpleInstrumentation public : void }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode a s t N o d e ) { SgGlobal g l o b a l S c o p e = i s S g G l o b a l ( astNode ) ; i f ( g l o b a l S c o p e != NULL) { // // C r e a t e t h e f u n c t i o n D e c l a r a t i o n // = new S g T y p e I n t ( ) ; SgType f u n c r e t u r n t y p e = my function ; SgName f u n c n a m e SgFunctionType f u n c t y p e = new S g F u n c t i o n T y p e ( f u n c r e t u r n t y p e , f a l s e ) ; SgFunctionDeclaration func = new S g F u n c t i o n D e c l a r a t i o n ( TRANSFORMATION FILE INFO , f u n c n a m e , SgFunctionDefinition func def = new S g F u n c t i o n D e f i n i t i o n ( TRANSFORMATION FILE INFO , f u n c ) ; SgBasicBlock func body = new S g B a s i c B l o c k (TRANSFORMATION FILE INFO ) ; // // s e t t h e end s o u r c e p o s i t i o n a s t r a n s f o r m a t i o n g e n e r a t e d s i n c e the c o n s t r u c t o r s only s e t the beginning source p o s i t i o n func >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; func >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( f u n c ) ; func def >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; func def >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( f u n c d e f ) ; >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; func body func body >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( f u n c b o d y ) ; // // Sets func Sets func t h e body i n t o t h e d e f i n i t i o n def >s e t b o d y ( f u n c b o d y ) ; the d e f i n t i o n s parent to the def >s e t p a r e n t ( f u n c ) ; : public SgSimpleProcessing

visit

SgNode a s t N o d e

);

func type ) ;

by

default

declaration

// DQ ( 9 / 8 / 2 0 0 7 ) : F ix u p t h e d e f i n i n g and non d e f i n i n g d e c l a r a t i o n s ROSE ASSERT ( f u n c > g e t d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; func >s e t d e f i n i n g D e c l a r a t i o n ( f u n c ) ; ROSE ASSERT ( f u n c >g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; ROSE ASSERT ( f u n c > g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != f u n c ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : We h a v e n o t b u i l d a non d e f i n i n g d e c l a r a t i o n , s o > g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; ROSE ASSERT ( f u n c this should be NULL .

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add f u n c t i o n s y m b o l t o g l o b a l s c o p e ! // p r i n t f ( F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r f u n c t i o n = %p = %s \ n , g l o b a l S c o p e , g l o b a l S c o p e >c l a s s n a m e ( ) . c s t r ( ) , f u S g F u n c t i o n S y m b o l f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( f u n c ) ; >g e t n a m e ( ) , f u n c t i o n S y m b o l ) ; globalScope >i n s e r t s y m b o l ( f u n c ROSE ASSERT ( g l o b a l S c o p e >l o o k u p f u n c t i o n s y m b o l ( f u n c >g e t n a m e ( ) ) != NULL ) ; // // C r e a t e t h e I n i t i a l i z e d N a m e f o r a p a r a m e t e r w i t h i n t h e p a r a m e t e r l i s t // SgName v a r 1 n a m e = v a r n a m e ; SgTypeInt v a r 1 t y p e = new S g T y p e I n t ( ) ; S g R e f e r e n c e T y p e r e f t y p e = new S g R e f e r e n c e T y p e ( v a r 1 t y p e ) ; S g I n i t i a l i z e r v a r 1 i n i t i a l i z e r = NULL ; S g I n i t i a l i z e d N a m e v a r 1 i n i t n a m e = new S g I n i t i a l i z e d N a m e ( v a r 1 n a m e , var1 init name > s e t f i l e i n f o (TRANSFORMATION FILE INFO ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : We now t e s t t h i s , s o i t var1 init name >s e t s c o p e ( f u n c d e f ) ; has to be set explicitly . ref type , v a r 1 i n i t i a l i z e r , NULL ) ;

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add v a r i a b l e s y m b o l t o g l o b a l s c o p e ! // p r i n t f ( F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r S g I n i t i a l i z e d N a m e = %p = %s \ n , g l o b a l S c o p e , g l o b a l S c o p e >c l a s s n a m e ( ) . c S g V a r i a b l e S y m b o l v a r s y m b o l = new S g V a r i a b l e S y m b o l ( v a r 1 i n i t n a m e ) ; func def >i n s e r t s y m b o l ( v a r 1 i n i t n a m e >g e t n a m e ( ) , v a r s y m b o l ) ;

Figure 32.13: Example source code shows addition of function to global scope (part 1).

218
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

CHAPTER 32. AST CONSTRUCTION

ROSE ASSERT ( f u n c d e f >l o o k u p v a r i a b l e s y m b o l ( v a r 1 i n i t n a m e >g e t n a m e ( ) ) ROSE ASSERT ( v a r 1 i n i t n a m e > g e t s y m b o l f r o m s y m b o l t a b l e ( ) != NULL ) ; // Done // constructing the InitializedName variable != NULL ) ;

I n s e r t argument i n f u n c t i o n p a r a m e t e r l i s t ROSE ASSERT ( f u n c != NULL ) ; = new S g F i l e I n f o ( ) ; // S g F i l e I n f o p a r a m e t e r L i s t F i l e I n f o // S g F i l e I n f o p a r a m e t e r L i s t F i l e I n f o = S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) ; S g F u n c t i o n P a r a m e t e r L i s t p a r a m e t e r L i s t = new S g F u n c t i o n P a r a m e t e r L i s t (TRANSFORMATION FILE INFO ) ; ROSE ASSERT ( p a r a m e t e r L i s t != NULL ) ; func >s e t p a r a m e t e r L i s t ( p a r a m e t e r L i s t ) ; ROSE ASSERT ( f u n c > g e t p a r a m e t e r L i s t ( ) != NULL ) ; func > g e t p a r a m e t e r L i s t ()> a p p e n d a r g ( v a r 1 i n i t n a m e ) ; // // I n s e r t a s t a t e m e n t i n t h e f u n c t i o n body // // // c r e a t e a VarRefExp S g V a r i a b l e S y m b o l v a r s y m b o l = new S g V a r i a b l e S y m b o l ( v a r 1 i n i t n a m e ) ; SgVarRefExp v a r r e f = new SgVarRefExp ( TRANSFORMATION FILE INFO , v a r s y m b o l ) ; >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; var ref >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( v a r r e f ) ; var ref c r e a t e a ++ e x p r e s s i o n , 0 f o r p r e f i x ++ S g P l u s P l u s O p p p e x p r e s s i o n = new S g P l u s P l u s O p (TRANSFORMATION FILE INFO , v a r r e f , 0 ) ; pp expression >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; pp expression >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( p p e x p r e s s i o n ) ; c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t n e w s t m t = new S g E x p r S t a t e m e n t ( TRANSFORMATION FILE INFO , p p e x p r e s s i o n ) ; >s e t e n d O f C o n s t r u c t (TRANSFORMATION FILE INFO ) ; n e w stm t n e w stm t >g e t e n d O f C o n s t r u c t ()> s e t p a r e n t ( n e w s t m t ) ;

//

//

#i f

0 // DQ ( 9 / 8 / 2 0 0 7 ) : T h i s i s no l o n g e r r e q u i r e d , // c r e a t e an e x p r e s s i o n t y p e S g T y p e I n t e x p r t y p e = new S g T y p e I n t ( ) ; // SgExpressionRoot is not longer used in t h e ROSE IR .

c r e a t e an e x p r e s s i o n r o o t S g E x p r e s s i o n R o o t e x p r r o o t = new S g E x p r e s s i o n R o o t ( TRANSFORMATION FILE INFO , p p e x p r e s s i o n , e x p r t y p e ) ; expr root >s e t p a r e n t ( n e w s t m t ) ; of SgExpression instead of SgExpressionRoot

// DQ ( 1 1 / 8 / 2 0 0 6 ) : M o d i f i e d t o r e f l e c t u s e >s e t e x p r e s s i o n ( e x p r r o o t ) ; n e w stm t

pp expression > s e t p a r e n t ( ne w stm t >g e t e x p r e s s i o n ( ) ) ; #e n d i f pp expression >s e t p a r e n t ( n e w s t m t ) ; // i n s e r t a s t a t e m e n t i n t o t h e f u n c t i o n body func body >p r e p e n d s t a t e m e n t ( n e w s t m t ) ; s e t t i n g the parent e x p l i c i t l y i s func >s e t p a r e n t ( g l o b a l S c o p e ) ; not required since it would be done w i t h i n AST p o s t p r o c e s s i n g

//

// //

s c o p e s o f s t a t m e n t s must be s e t e x p l i c i t l y s i n c e w i t h i n C ++ t h e y a r e n o t t o be t h e same a s t h a t i n d i c a t e d by t h e p a r e n t ( s e e ChangeLog f o r S p r i n g func >s e t s c o p e ( g l o b a l S c o p e ) ;

guaranteed 2005).

// // I n s e r t t h e f u n c t i o n d e c l a r a t i o n i n t h e c o d e // globalScope >p r e p e n d d e c l a r a t i o n ( f u n c ) ; // // } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , R e q u i r e d p o s t p r o c e s s i n g o f AST r e q u i r e d temporaryAstFixes ( globalScope ) ; AstPostProcessing ( globalScope ) ; to set parent pointers and fixup template names , etc .

preorder

);

Figure 32.14: Example source code shows addition of function to global scope (part 2).

32.4. FUNCTIONS

219

1 2 3 4 5 6 7 8 9 10

i n t main ( ) { f o r ( i n t i =0; i < 4 ; { int x ; } return 0; }

i ++)

Figure 32.15: Example source code used as input to translator adding new function.

1 2 3 4 5 6 7 8 9 10 11 12 13

i n t m y f u n c t i o n ( i n t &var name ) { ++var name ; } i n t main ( ) { for ( int i = 0; int x ; } return 0; }

i < 4;

i ++) {

Figure 32.16: Output of input to translator adding new function.

220

CHAPTER 32. AST CONSTRUCTION

32.5

Function Calls

Adding functions calls is a typical task for instrumentation translator. Figure 32.17 shows the use of the AST string based rewrite mechanism to add function calls to the top and bottom of each block within the AST. Figure 32.18 shows the use of the AST builder interface to do the same instrumentation work. Figure 32.19 shows the input code used to get the translator. Figure 32.20 shows the resulting output. Another example shows how to add a function call at the end of each function body. A utility function, instrumentEndOfFunction(), from SageInterface name space is used. The interface tries to locate all return statements of a target function and rewriting return expressions with side eects, if there are any. Figure 32.21 shows the translator code. Figure 32.22 shows the input code. The instrumented code is shown in Figure 32.23.

32.6

Creating a struct for Global Variables

: TODO: This tutorial vel AST manipulation. d have a more concise sing SageInterface and SageBuilder functions.

This is an example written to support the Charm++ tool. This translator extracts global variables from the program and builds a structure to hold them. The support is part of a number of requirements associated with using Charm++ and AMPI. Figure 32.24 shows repackaging of global variables within an application into a struct. All reference to the global variables are also transformed to reference the original variable indirectly through the structure. This processing is part of preprocessing to use Charm++. This example shows the low level handling directly at the level of the IR.

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

221

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e // a t t h e t o p and bottome o f e a c h b a s i c b l o c k . #i n c l u d e r o s e . h u s i n g namespace s t d ; class { }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode astNode ) { S g B a s i c B l o c k b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { c o n s t u n s i g n e d i n t SIZE OF BLOCK = 1 ; i f ( block > g e t s t a t e m e n t s ( ) . s i z e ( ) > SIZE OF BLOCK ) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e s t r i n g codeAtTopOfBlock = v o i d my TimerF unct io nSta rt ( ) ; myTi mer Functi onSt art ( ) ; ; s t r i n g codeAtBottomOfBlock = v o i d myTimerFunctionEnd ( ) ; myTimerFunctionEnd ( ) ; ; // I n s e r t new c o d e i n t o t h e s c o p e r e p r e s e n t e d by t h e s t a t e m e n t ( a p p l i e s t o S g S c o p e S t a t e m e n t s ) MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ; // I n s e r t t h e new c o d e a t t h e t o p and bottom o f t h e s c o p e r e p r e s e n t e d by b l o c k M i d d l e L e v e l R e w r i t e : : i n s e r t ( b l o c k , codeAtTopOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : TopOfCurrentScope ) ; M i d d l e L e v e l R e w r i t e : : i n s e r t ( b l o c k , codeAtBottomOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : BottomOfCurrentScope ) ; } } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; } SimpleInstrumentation : public : void public SgSimpleProcessing

visit

( SgNode astNode ) ;

preorder

);

Figure 32.17: Example source code to instrument any input program.

222

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e i s an example p r e p r o c e s s o r b u i l t w i t h ROSE . // S p e c i f i c a l l y i t shows t h e d e s i g n o f a t r a n s f o r m a t i o n t o i n s t r u m e n t s o u r c e code , p l a c i n g s o u r c e c o d e // a t t h e t o p and bottom o f e a c h b a s i c b l o c k . #i n c l u d e r o s e . h u s i n g namespace s t d ; u s i n g namespace S a g e I n t e r f a c e ; u s i n g namespace S a g e B u i l d e r ; c l a s s SimpleInstrumentation : public SgSimpleProcessing { public : v o i d v i s i t ( SgNode astNode ) ; }; void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode astNode ) { S g B a s i c B l o c k b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { c o n s t u n s i g n e d i n t SIZE OF BLOCK = 1 ; i f ( block > g e t s t a t e m e n t s ( ) . s i z e ( ) > SIZE OF BLOCK ) { SgName name1 ( my TimerF unct io nSta rt ) ; // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e SgFunctionDeclaration decl 1 = buildNondefiningFunctionDeclaration ( name1 , b u i l d V o i d T y p e ( ) , b u i l d F u n c t i o n P a r a m e t e r L i s t ( ) , b l o c k ) ; (( decl 1 >g e t d e c l a r a t i o n M o d i f i e r ( ) ) . g e t s t o r a g e M o d i f i e r ( ) ) . s e t E x t e r n ( ) ; SgExprStatement c a l l S t m t 1 = b u i l d F u n c t i o n C a l l S t m t ( name1 , b u i l d V o i d T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , b l o c k ) ; prependStatement ( call Stmt 1 , block ) ; prependStatement ( d e c l 1 , block ) ; SgName name2 ( myTimerFunctionEnd ) ; // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e SgFunctionDeclaration decl 2 = buildNondefiningFunctionDeclaration ( name2 , b u i l d V o i d T y p e ( ) , b u i l d F u n c t i o n P a r a m e t e r L i s t ( ) , b l o c k ) ; (( decl 2 >g e t d e c l a r a t i o n M o d i f i e r ( ) ) . g e t s t o r a g e M o d i f i e r ( ) ) . s e t E x t e r n ( ) ; SgExprStatement c a l l S t m t 2 = b u i l d F u n c t i o n C a l l S t m t ( name2 , b u i l d V o i d T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , b l o c k ) ; appendStatement ( d e c l 2 , b l o c k ) ; appendStatement ( c a l l S t m t 2 , b l o c k ) ; } } } int main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ; SimpleInstrumentation treeTraversal ; treeTraversal . traverseInputFiles ( project , AstTests : : runAllTests ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

preorder ) ;

Figure 32.18: Example source code using the high level interfaces

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

223

1 2 3 4 5 6 7 8 9

// O v e r l o a d e d f u n c t i o n s void foo ( double ) { int x = 1; int y ;

for

t e s t i n g overloaded function

resolution

// I t h i n k t h a t t h i s c a s e f a i l s // i f ( x ) y = 1 ; e l s e y = 2 ; }

currently

Figure 32.19: Example source code used as input to instrumenting translator.

1 2 3 4 5 6 7 8 9 10 11 12 13

// O v e r l o a d e d f u n c t i o n s

for

t e s t i n g overloaded function

resolution

void foo ( double ) { v o i d myT imerFunct io nSt ar t ( ) ; myT imerFunctionSta rt ( ) ; int x = 1; int y ; v o i d myTimerFunctionEnd ( ) ; ; myTimerFunctionEnd ( ) ; // I t h i n k t h a t t h i s c a s e f a i l s // i f ( x ) y = 1 ; e l s e y = 2 ; }

currently

Figure 32.20: Output of input to instrumenting translator.

224

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

/! \ b r i e f test instrumentation right / #i n c l u d e r o s e . h #i n c l u d e < i o s t r e a m > u s i n g namespace S a g e I n t e r f a c e ; u s i n g namespace S a g e B u i l d e r ;

b e f o r e t h e end o f a f u n c t i o n

i n t main ( i n t a r g c , c h a r a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Find a l l f u n c t i o n d e f i n i t i o n s we want t o i n s t r u m e n t s t d : : v e c t o r <SgNode > f u n c D e f L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e f i n i t i o n ) ; s t d : : v e c t o r <SgNode > : : i t e r a t o r i t e r ; f o r ( i t e r = f u n c D e f L i s t . b e g i n ( ) ; i t e r != f u n c D e f L i s t . end ( ) ; i t e r ++) { SgFunctionDefinition c u r d e f = i s S g F u n c t i o n D e f i n i t i o n ( i t e r ) ; ROSE ASSERT( c u r d e f ) ; S g B a s i c B l o c k body = c u r d e f >g e t b o d y ( ) ; // B u i l d t h e c a l l s t a t e m e n t f o r e a c h p l a c e SgExprStatement c a l l S t m t 1 = b u i l d F u n c t i o n C a l l S t m t ( c a l l 1 , b u i l d I n t T y p e ( ) , b u i l d E x p r L i s t E x p ( ) , body ) ; // i n s t r u m e n t t h e f u n c t i o n i n t i= i n s t r u m e n t E n d O f F u n c t i o n ( c u r d e f >g e t d e c l a r a t i o n ( ) , c a l l S t m t 1 ) ; s t d : : cout <<I n s t r u m e n t e d << i << p l a c e s . << s t d : : e n d l ; } // end o f i n s t r u m e n t a t i o n AstTests : : runAllTests ( p r o j e c t ) ; // t r a n s l a t i o n o n l y project >u n p a r s e ( ) ; }

Figure 32.21: Example source code instrumenting end of functions

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

/ Example c o d e : a f u n c t i o n with m u l t i p l e r e t u r n s some r e t u r n s have e x p r e s s i o n s w i t h s i d e a f u n c t i o n w i t h o u t any r e t u r n / extern int foo ( ) ; extern int c a l l 1 ( ) ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( a r g c > 1) return foo ( ) ; else return foo ( ) ; return 0; } void bar ( ) { int i ; }

effects

Figure 32.22: Example input code of the instrumenting translator for end of functions.

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

225

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

/ Example c o d e : a f u n c t i o n with m u l t i p l e r e t u r n s some r e t u r n s have e x p r e s s i o n s w i t h s i d e a f u n c t i o n w i t h o u t any r e t u r n / int foo ( ) ; int call1 (); i n t main ( i n t a r g c , c h a r a r g v [ ] ) { i f ( argc > 1) { int rose temp 1 = foo ( ) ; call1 (); return rose temp 1 ; } else { int rose temp 2 = foo ( ) ; call1 (); return rose temp 2 ; } call1 (); return 0; } void bar ( ) { int i ; call1 (); }

effects

Figure 32.23: Output of instrumenting translator for end of functions.

226

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

// ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t sh ows t h e d e s i g n o f a t r a n s f o r m a t i o n #i n c l u d e using r o s e . h std ;

is to

an e x a m p l e p r e p r o c e s s o r b u i l t w i t h ROSE . do a t r a n s f o r m a t i o n s p e c i f i c f o r Charm++.

namespace

R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > buildListOfGlobalVariables ( SgSourceFile { // T h i s f u n c t i o n b u i l d s a l i s t o f g l o b a l a s s e r t ( f i l e != NULL ) ;

file

) ( from a SgFile ) .

variables

R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > g l o b a l V a r i a b l e L i s t ; SgGlobal globalSc ope = f i l e >g e t g l o b a l S c o p e ( ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ; R o s e S T L C o n t a i n e r <S g D e c l a r a t i o n S t a t e m e n t > : : i t e r a t o r i = g l o b a l S c o p e >g e t d e c l a r a t i o n s ( ) . b e g i n ( ) ; w h i l e ( i != g l o b a l S c o p e > g e t d e c l a r a t i o n s ( ) . end ( ) ) { S g V a r i a b l e D e c l a r a t i o n v a r i a b l e D e c l a r a t i o n = i s S g V a r i a b l e D e c l a r a t i o n ( i ) ; i f ( v a r i a b l e D e c l a r a t i o n != NULL) { >g e t v a r i a b l e s ( ) ; R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > & v a r i a b l e L i s t = v a r i a b l e D e c l a r a t i o n R o s e S T L C o n t a i n e r <S g I n i t i a l i z e d N a m e > : : i t e r a t o r v a r = v a r i a b l e L i s t . b e g i n ( ) ; w h i l e ( v a r != v a r i a b l e L i s t . end ( ) ) { g l o b a l V a r i a b l e L i s t . push back ( var ) ; v a r ++; } } i ++; } return } // T h i s f u n c t i o n i s n o t u s e d , b u t i s u s e f u l f o r // g e n e r a t i n g t h e l i s t o f a l l g l o b a l v a r i a b l e s R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > buildListOfGlobalVariables ( SgProject project ) { // T h i s f u n c t i o n b u i l d s a l i s t o f g l o b a l v a r i a b l e s globalVariableList ;

( from a

SgProject ) .

R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > g l o b a l V a r i a b l e L i s t ; c o n s t S g F i l e P t r L i s t& f i l e L i s t = p r o j e c t >g e t f i l e L i s t ( ) ; SgFilePtrList : : c o n s t i t e r a t o r f i l e = f i l e L i s t . begin ( ) ; // Loop o v e r t h e // when m u l t i p l e w h i l e ( f i l e != { Rose STL f i l e s in the p r o j e c t ( multiple f i l e s e x i s t s o u r c e f i l e s a r e p l a c e d on t h e command l i n e ) . f i l e L i s t . end ( ) ) C o n t a i n e r < S g I n i t i a l i z e d N a m e > f i l e G l o b a l V a r i a b l e L i s t = b u i l d L i s t O f G l o b a l V a r i a b l e s ( i s S g S o u r c e F i l e ( f i l e ) ) ;

// DQ ( 9 / 2 6 / 2 0 0 7 ) : Moved f r o m s t d : : l i s t t o s t d : : v e c t o r // g l o b a l V a r i a b l e L i s t . merge ( f i l e G l o b a l V a r i a b l e L i s t ) ; g l o b a l V a r i a b l e L i s t . i n s e r t ( g l o b a l V a r i a b l e L i s t . b e g i n ( ) , f i l e G l o b a l V a r i a b l e L i s t . b e g i n ( ) , f i l e G l o b a l V a r i a b l e L i s t . end ( ) ) ; f i l e ++; } return } R o s e S T L C o n t a i n e r <SgVarRefExp > b u i l d L i s t O f V a r i a b l e R e f e r e n c e E x p r e s s i o n s U s i n g G l o b a l V a r i a b l e s ( SgNode node ) { // T h i s f u n c t i o n b u i l d s a l i s t o f u s e s o f v a r i a b l e s ( SgVarRefExp IR n o d e s ) // return variable R o s e S T L C o n t a i n e r <SgVarRefExp > g l o b a l V a r i a b l e U s e L i s t ; l i s t o f a l l v a r i a b l e s ( t h e n s e l e c t o u t t h e g l o b a l v a r i a b l e s by R o s e S T L C o n t a i n e r <SgNode> n o d e L i s t = NodeQuery : : q u e r y S u b T r e e testing ( node , the scope ) V SgVarRefExp globalVariableList ;

within

t h e AST .

//

);

R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = n o d e L i s t . b e g i n ( ) ; w h i l e ( i != n o d e L i s t . end ( ) ) { SgVarRefExp v a r i a b l e R e f e r e n c e E x p r e s s i o n = i s S g V a r R e f E x p ( i ) ;

Figure 32.24: Example source code shows repackaging of global variables to a struct (part 1).

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

227

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

assert ( variableReferenceExpression

!= NULL ) ;

assert ( variableReferenceExpression >g e t s y m b o l ( ) != NULL ) ; assert ( variableReferenceExpression >g e t s y m b o l ()> g e t d e c l a r a t i o n ( ) != NULL ) ; assert ( variableReferenceExpression >g e t s y m b o l ()> g e t d e c l a r a t i o n ()> g e t s c o p e ( )

!= NULL ) ;

// Note t h a t v a r i a b l e R e f e r e n c e E x p r e s s i o n >g e t s y m b o l ()> g e t d e c l a r a t i o n ( ) r e t u r n s t h e // S g I n i t i a l i z e d N a m e ( n o t t h e S g V a r i a b l e D e c l a r a t i o n w h e r e i t was d e c l a r e d ) ! SgInitializedName variable = variableReferenceExpression >g e t s y m b o l ()> g e t d e c l a r a t i o n ( ) ; SgScopeStatement variableScope = variable >g e t s c o p e ( ) ; it

// Check i f t h i s i s a v a r i a b l e d e c l a r e d i n g l o b a l s c o p e , i f s o , t h e n s a v e i f ( i s S g G l o b a l ( v a r i a b l e S c o p e ) != NULL) { g l o b a l V a r i a b l e U s e L i s t . push back ( v a r i a b l e R e f e r e n c e E x p r e s s i o n ) ; } i ++; } return } globalVariableUseList ;

SgClassDeclaration b u i l d C l a s s D e c l a r a t i o n A n d D e f i n i t i o n ( s t r i n g name , S g S c o p e S t a t e m e n t { // T h i s f u n c t i o n b u i l d s a c l a s s d e c l a r a t i o n and d e f i n i t i o n // ( b o t h t h e d e f i n i n g and n o n d e f i n i n g d e c l a r a t i o n s a s r e q u i r e d ) . //

scope )

B u i l d a f i l e i n f o o b j e c t marked a s a t r a n s f o r m a t i o n Sg File Info f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ; This i s the c l a s s d e f i n i t i o n ( the f i l e I n f o SgClassDefinition classDefinition = new a s s e r t ( c l a s s D e f i n i t i o n != NULL ) ; i s the p o s i t i o n of the opening SgClassDefinition ( f i l e I n f o ); brace )

//

//

S e t t h e end o f c o n s t r u c t e x p l i c t l y ( w h e r e n o t a classDefinition >s e t e n d O f C o n s t r u c t ( f i l e I n f o ) ;

transformation

this

is

the

location

of

the

closing

brace )

//

This i s the d e f i n i n g d e c l a r a t i o n f o r the c l a s s ( with a r e f e r e n c e to the c l a s s d e f i n i t i o n ) S g C l a s s D e c l a r a t i o n c l a s s D e c l a r a t i o n = new S g C l a s s D e c l a r a t i o n ( f i l e I n f o , name . c s t r ( ) , S g C l a s s D e c l a r a t i o n : : e s t r u c t , NULL, c l a s s D e f i n i t i o n ) ; a s s e r t ( c l a s s D e c l a r a t i o n != NULL ) ; Set the d e f i n i n g d e c l a r a t i o n in the d e f i n i n g d e c l a r a t i o n ! classDeclaration >s e t d e f i n i n g D e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ; S e t t h e non d e f i n i n g d e c l a r a t i o n i n t h e d e f i n i n g d e c l a r a t i o n ( b o t h a r e r e q u i r e d ) S g C l a s s D e c l a r a t i o n n o n d e f i n i n g C l a s s D e c l a r a t i o n = new S g C l a s s D e c l a r a t i o n ( f i l e I n f o , name . c s t r ( ) , S g C l a s s D e c l a r a t i o n : : e s t r u c t , NULL, NULL ) ; a s s e r t ( c l a s s D e c l a r a t i o n != NULL ) ; nondefiningClassDeclaration >s e t t y p e ( S g C l a s s T y p e : : c r e a t e T y p e ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ) ; S e t t h e i n t e r n a l r e f e r e n c e t o t h e non d e f i n i n g d e c l a r a t i o n classDeclaration >s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ; classDeclaration >s e t t y p e ( n o n d e f i n i n g C l a s s D e c l a r a t i o n >g e t t y p e ( ) ) ; S e t t h e d e f i n i n g and no d e f i n i n g d e c l a r a t i o n s i n t h e non d e f i n i n g c l a s s d e c l a r a t i o n ! nondefiningClassDeclaration >s e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( n o n d e f i n i n g C l a s s D e c l a r a t i o n ) ; nondefiningClassDeclaration >s e t d e f i n i n g D e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ; Set the nondefining d e c l a r a t i o n as a forward nondefiningClassDeclaration >s e t F o r w a r d ( ) ; declaration !

//

//

//

//

//

// Don t f o r g e t t h e s e t t h e d e c l a r a t i o n i n t h e d e f i n i t i o n classDefinition >s e t d e c l a r a t i o n ( c l a s s D e c l a r a t i o n ) ; // s e t t h e s c o p e e x p l i c i t l y ( name q u a l i f i c a t i o n t r i c k s classDeclaration >s e t s c o p e ( s c o p e ) ; nondefiningClassDeclaration >s e t s c o p e ( s c o p e ) ; can

( IR node

constructors

are

s i d e e f f e c t

free !)!

imply

it

is

not

always

the

parent

IR node ! )

// some e r r o r c h e c k i n g assert ( classDeclaration > g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; assert ( classDeclaration > g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; assert ( classDeclaration > g e t d e f i n i t i o n ( ) != NULL ) ;

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add f u n c t i o n s y m b o l t o g l o b a l s c o p e ! p r i n t f ( F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r c l a s s = %p = %s \ n , s c o p e , s c o p e >c l a s s n a m e ( ) . c s t r ( ) , c l a s s D e c l a r a t i o n , c l a s s D e S g C l a s s S y m b o l c l a s s S y m b o l = new S g C l a s s S y m b o l ( c l a s s D e c l a r a t i o n ) ; scope >i n s e r t s y m b o l ( c l a s s D e c l a r a t i o n >g e t n a m e ( ) , c l a s s S y m b o l ) ;

Figure 32.25: Example source code shows repackaging of global variables to a struct (part 2).

228

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

ROSE ASSERT( s c o p e >l o o k u p c l a s s s y m b o l ( c l a s s D e c l a r a t i o n >g e t n a m e ( ) ) return } classDeclaration ;

!= NULL ) ;

SgVariableSymbol putGlobalVariablesIntoClass { // T h i s f u n c t i o n i t e r a t e s SgVariableSymbol for { //

( R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > & g l o b a l V a r i a b l e s , over the list of global variables and inserts them into the

SgClassDeclaration iput class

classDeclaration

definition

g l o b a l C l a s s V a r i a b l e S y m b o l = NULL ; var = g l o b a l V a r i a b l e s . begin ( ) ; var != g l o b a l V a r i a b l e s . end ( ) ; v a r++)

( R o s e S T L C o n t a i n e r <S g I n i t i a l i z e d N a m e > : : i t e r a t o r

p r i n t f ( Appending g l o b a l v a r i a b l e = %s t o new g l o b a l V a r i a b l e C o n t a i n e r \ n , ( v a r)> g e t n a m e ( ) . s t r ( ) ) ; S g V a r i a b l e D e c l a r a t i o n g l o b a l V a r i a b l e D e c l a r a t i o n = i s S g V a r i a b l e D e c l a r a t i o n ( ( v a r)> g e t p a r e n t ( ) ) ; a s s e r t ( g l o b a l V a r i a b l e D e c l a r a t i o n != NULL ) ;

// Get t h e g l o b a l s c o p e f r o m t h e g l o b a l v a r i a b l e d i r e c t l y SgGlobal globalScope = i sS gG l ob a l ( g l o b a l V a r i a b l e D e c l a r a t i o n >g e t s c o p e ( ) ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ; if ( v a r == g l o b a l V a r i a b l e s . b e g i n ( ) ) { // T h i s i s t h e f i r s t t i m e i n t h i s l o o p , r e p l a c e t h e f i r s t g l o b a l v a r i a b l e w i t h // t h e c l a s s d e c l a r a t i o n / d e f i n i t i o n c o n t a i n i n g a l l t h e g l o b a l v a r i a b l e s ! // Note t h a t i n i t i a l i z e r s i n t h e g l o b a l v a r i a b l e d e c l a r a t i o n s r e q u i r e m o d i f i c a t i o n // o f t h e p r e i n i t i a l i z a t i o n l i s t i n t h e c l a s s s c o n s t r u c t o r ! I am i g n o r i n g t h i s f o r globalScope >r e p l a c e s t a t e m e n t ( g l o b a l V a r i a b l e D e c l a r a t i o n , c l a s s D e c l a r a t i o n ) ; //

now !

B u i l d s o u r c e p o s i t i o n i n f o r m a i t o n ( marked a s t r a n s f o r m a t i o n ) Sg File Info f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ;

// Add t h e v a r i a b l e o f t h e c l a s s t y p e t o t h e g l o b a l s c o p e ! S g C l a s s T y p e v a r i a b l e T y p e = new S g C l a s s T y p e ( c l a s s D e c l a r a t i o n >g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) ) ; a s s e r t ( v a r i a b l e T y p e != NULL ) ; S g V a r i a b l e D e c l a r a t i o n v a r i a b l e D e c l a r a t i o n = new S g V a r i a b l e D e c l a r a t i o n ( f i l e I n f o , A M P I g l o b a l s , v a r i a b l e T y p e ) ; a s s e r t ( v a r i a b l e D e c l a r a t i o n != NULL ) ; globalScope >i n s e r t s t a t e m e n t ( c l a s s D e c l a r a t i o n , v a r i a b l e D e c l a r a t i o n , f a l s e ) ; assert ( variableDeclaration > g e t v a r i a b l e s ( ) . empty ( ) == f a l s e ) ; S g I n i t i a l i z e d N a m e variableName = ( v a r i a b l e D e c l a r a t i o n >g e t v a r i a b l e s ( ) . b e g i n ( ) ) ; a s s e r t ( v a r i a b l e N a m e != NULL ) ; // DQ ( 9 / 8 / 2 0 0 7 ) : Need t o s e t t h e s c o p e o f variableName >s e t s c o p e ( g l o b a l S c o p e ) ; // t h e new variable .

build the return value g l o b a l C l a s s V a r i a b l e S y m b o l = new S g V a r i a b l e S y m b o l ( v a r i a b l e N a m e ) ;

// DQ ( 9 / 8 / 2 0 0 7 ) : Need t o add t h e s y m b o l t o t h e g l o b a l s c o p e ( new t e s t i n g r e q u i r e s t h i s ) . globalScope >i n s e r t s y m b o l ( v a r i a b l e N a m e >g e t n a m e ( ) , g l o b a l C l a s s V a r i a b l e S y m b o l ) ; ROSE ASSERT ( g l o b a l S c o p e >l o o k u p v a r i a b l e s y m b o l ( v a r i a b l e N a m e >g e t n a m e ( ) ) != NULL ) ; } else { // f o r a l l o t h e r i t e r a t i o n s o f t h i s l o o p . . . // remove v a r i a b l e d e c l a r a t i o n f r o m t h e g l o b a l s c o p e globalScope >r e m o v e s t a t e m e n t ( g l o b a l V a r i a b l e D e c l a r a t i o n ) ; } // add t h e v a r i a b l e d e c l a r a t i o n t o t h e c l a s s d e f i n i t i o n classDeclaration > g e t d e f i n i t i o n ()> append member ( g l o b a l V a r i a b l e D e c l a r a t i o n ) ; } return } globalClassVariableSymbol ;

void f i x u p R e f e r e n c e s T o G l o b a l V a r i a b l e s ( R o s e S T L C o n t a i n e r <SgVarRefExp > & v a r i a b l e R e f e r e n c e L i s t , S g V a r i a b l e S y m b o l g l o b a l C l a s s V a r i a b l e { // Now f i x u p t h e SgVarRefExp t o r e f e r e n c e t h e g l o b a l v a r i a b l e s t h r o u g h a s t r u c t f o r ( R o s e S T L C o n t a i n e r <SgVarRefExp > : : i t e r a t o r v a r = v a r i a b l e R e f e r e n c e L i s t . b e g i n ( ) ; v a r != v a r i a b l e R e f e r e n c e L i s t . end ( ) ; v a r { a s s e r t ( v a r != NULL ) ;

Figure 32.26: Example source code shows repackaging of global variables to a struct (part 3).

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

229

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80

//

printf

( Variable

reference

f o r %s \ n , ( v a r)> g e t s y m b o l ()> g e t d e c l a r a t i o n ()> g e t n a m e ( ) . s t r ( ) ) ;

SgNode p a r e n t = ( v a r)> g e t p a r e n t ( ) ; a s s e r t ( p a r e n t != NULL ) ; // I f t h i s i s n o t an e x p r e s s i o n t h e n i s l i k e l y a m e a n i n g l e s s SgExpression parentExpression = isSgExpression ( parent ) ; a s s e r t ( p a r e n t E x p r e s s i o n != NULL ) ; Build the reference through the global class variable statement such as ( x ; )

// //

( x > AMPI globals . x )

B u i l d s o u r c e p o s i t i o n i n f o r m a i t o n ( marked a s t r a n s f o r m a t i o n ) Sg File Info f i l e I n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; a s s e r t ( f i l e I n f o != NULL ) ;

Build AMPI globals S g E x p r e s s i o n l h s = new SgVarRefExp ( f i l e I n f o , g l o b a l C l a s s V a r i a b l e S y m b o l ) ; a s s e r t ( l h s != NULL ) ; // B u i l d A M P I g l o b a l s . x f r o m x SgDotExp g l o b a l V a r i a b l e R e f e r e n c e = new SgDotExp ( f i l e I n f o , l h s , v a r ) ; a s s e r t ( g l o b a l V a r i a b l e R e f e r e n c e != NULL ) ; // if ( p a r e n t E x p r e s s i o n != NULL) { // I n t r o d u c e r e f e r e n c e t o v a r //

through

the

data

structure

case of binary operator SgUnaryOp u n a r y O p e r a t o r = isSgUnaryOp ( p a r e n t E x p r e s s i o n ) ; i f ( u n a r y O p e r a t o r != NULL) { unaryOperator >s e t o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } else { // c a s e o f b i n a r y o p e r a t o r SgBinaryOp b i n a r y O p e r a t o r = i s S g B i n a r y O p ( p a r e n t E x p r e s s i o n ) ; i f ( b i n a r y O p e r a t o r != NULL) { // f i g u r e o u t i f t h e v a r i s on t h e l h s o r t h e r h s i f ( binaryOperator > g e t l h s o p e r a n d ( ) == v a r ) { binaryOperator >s e t l h s o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } else { a s s e r t ( binaryOperator > g e t r h s o p e r a n d ( ) == v a r ) ; binaryOperator >s e t r h s o p e r a n d ( g l o b a l V a r i a b l e R e f e r e n c e ) ; } } else { // i g n o r e t h e s e c a s e s f o r now ! switch ( parentExpression >v a r i a n t T ( ) ) { // Where t h e v a r i a b l e a p p e r s i n t h e f u n c t i o n argument l i s t t h e p a r e n t i s a S g E x p r L i s t E x p c a s e V SgExprListExp : { p r i n t f ( S o r r y n o t i m p l e m e n t e d , c a s e o f g l o b a l v a r i a b l e i n f u n c t i o n argument l i s t assert ( false ); break ; } case V S g I n i t i a l i z e r : c a s e V SgRefExp : c a s e V SgVarArgOp : default : { p r i n t f ( Error : d e f a u l t reached in switch p a r e n t E x p r e s s i o n = %p = %s \ n , p a r e n t E x p r e s s i o n , p a r e n t E x p r e s s i o n >c l a s s n a m e ( ) . c s t r ( ) ) ; assert ( false ); } } } } } } } #d e f i n e OUTPUT NAMES OF GLOBAL VARIABLES 0 #d e f i n e OUTPUT NAMES OF GLOBAL VARIABLE REFERENCES 0 void

...

\n ) ;

Figure 32.27: Example source code shows repackaging of global variables to a struct (part 4).

230

CHAPTER 32. AST CONSTRUCTION

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

transformGlobalVariablesToUseStruct { a s s e r t ( f i l e != NULL ) ; //

SgSourceFile

file

T h e s e a r e t h e g l o b a l v a r i a b l e s i n t h e i n p u t program ( p r o v i d e d a s h e l p f u l i n f o r m a t i o n ) R o s e S T L C o n t a i n e r < S g I n i t i a l i z e d N a m e > g l o b a l V a r i a b l e s = b u i l d L i s t O f G l o b a l V a r i a b l e s ( f i l e ) ;

#i f

OUTPUT NAMES OF GLOBAL VARIABLES p r i n t f ( g l o b a l v a r i a b l e s ( declared in g l o b a l scope ) : f o r ( R o s e S T L C o n t a i n e r <S g I n i t i a l i z e d N a m e > : : i t e r a t o r { p r i n t f ( %s \ n , ( v a r)> g e t n a m e ( ) . s t r ( ) ) ; } p r i n t f (\n ) ; #e n d i f // get the g l o b a l scope within the f i r s t f i l e ( c u r r e n t l y SgGlobal globalSc ope = f i l e >g e t g l o b a l S c o p e ( ) ; a s s e r t ( g l o b a l S c o p e != NULL ) ;

\n ) ; var = g l o b a l V a r i a b l e s . begin ( ) ;

var

!=

g l o b a l V a r i a b l e s . end ( ) ;

v a r++)

ignoring

all

other

files )

//

Build the c l a s s d e c l a r a t i o n SgClassDeclaration c l a s s D e c l a r a t i o n = buildClassDeclarationAndDefinition ( AMPI globals t , globalScope ) ;

// Put t h e g l o b a l v a r i a b l e s i n t o t h e c l a s s SgVariableSymbol globalClassVariableSymbol = p u t G l o b a l V a r i a b l e s I n t o C l a s s ( g l o b a l V a r i a b l e s , c l a s s D e c l a r a t i o n ) ; // // T h e i r a s s o c i a t e d s y m b o l s w i l l be l o c a t e d w i t h i n t h e p r o j e c t s AST ( where they o c c u r i n v a r i a b l e r e f e r e n c e e x p r e s s i o n s ) . R o s e S T L C o n t a i n e r <SgVarRefExp > v a r i a b l e R e f e r e n c e L i s t = b u i l d L i s t O f V a r i a b l e R e f e r e n c e E x p r e s s i o n s U s i n g G l o b a l V a r i a b l e s ( f i l e ) ;

OUTPUT NAMES OF GLOBAL VARIABLE REFERENCES p r i n t f ( g l o b a l v a r i a b l e s a p p e a r i n g i n t h e a p p l i c a t i o n : \n ) ; f o r ( R o s e S T L C o n t a i n e r <SgVarRefExp > : : i t e r a t o r v a r = v a r i a b l e R e f e r e n c e L i s t . b e g i n ( ) ; { p r i n t f ( %s \ n , ( v a r)> g e t s y m b o l ()> g e t d e c l a r a t i o n ()> g e t n a m e ( ) . s t r ( ) ) ; } p r i n t f (\n ) ; #e n d i f // }

#i f

var

!=

v a r i a b l e R e f e r e n c e L i s t . end ( ) ;

var

Fixup a l l r e f e r e n c e s t o g l o b a l v a r i a b l e t o a c c e s s t h e v a r i a b l e through t h e c l a s s ( x > AMPI globals . x ) fixupReferencesToGlobalVariables ( variableReferenceList , globalClassVariableSymbol ) ;

void transformGlobalVariablesToUseStruct ( SgProject project ) { // C a l l t h e t r a n s f o r m a t i o n o f e a c h f i l e ( t h e r e a r e m u l t i p l e S g F i l e // o b j e c t s when m u l t i p l e f i l e s a r e s p e c f i e d on t h e command l i n e ! ) . a s s e r t ( p r o j e c t != NULL ) ; c o n s t S g F i l e P t r L i s t& f i l e L i s t = p r o j e c t >g e t f i l e L i s t ( ) ; SgFilePtrList : : c o n s t i t e r a t o r f i l e = f i l e L i s t . begin ( ) ; w h i l e ( f i l e != f i l e L i s t . end ( ) ) { t r a n s f o r m G l o b a l V a r i a b l e s T o U s e S t r u c t ( i s S g S o u r c e F i l e ( f i l e ) ) ; f i l e ++; } } // // MAIN PROGRAM // int main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; a s s e r t ( p r o j e c t != NULL ) ; // transform a p p l i c a t i o n as r e q u i r e d transformGlobalVariablesToUseStruct ( project ) ; o u t new application r o s e <i n p u t file name > )

// Code g e n e r a t i o n p h a s e ( w r i t e r e t u r n backend ( p r o j e c t ) ; }

Figure 32.28: Example source code shows repackaging of global variables to a struct (part 5).

32.6. CREATING A STRUCT FOR GLOBAL VARIABLES

231

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

int x ; int y ; long z ; float pressure ; i n t main ( ) { int a = 0; int b = 0; fl oa t density = 1.0; x++; b++; x = a + y; return 0; }

Figure 32.29: Example source code used as input to translator adding new function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

s t r u c t AMPI globals t { int x ; int y ; long z ; float pressure ; } ; s t r u c t A M P I g l o b a l s t A MP I g l o b a l s ; i n t main ( ) { int a = 0; int b = 0; f l oa t density = 1.0; A M PI gl oba ls . x++; b++; A M PI gl oba ls . x = ( a + A M P I g l o b a l s . y ) ; return 0; }

Figure 32.30: Output of input to translator adding new function.

232

CHAPTER 32. AST CONSTRUCTION

Chapter 33

Parser Building Blocks

It is often needed to write a small parser to parse customized source code annotations in forms of C/C++ pragmas or Fortran comments. The parser is responsible for recognizing keywords, variables, expressions in the annotations and storing the recognized information in AST, often in a form of persistent AstAttribute. ROSE provides a set of helper functions which can be used to create such a simple parser using recursive descent parsing1 . These functions are collected in the namespace named AstFromString, documented under the Namespace List of ROSEs online Doxygen web references. A suggested workow to build your own pragma (or comment) parser is: input: dene the grammar of your pragma, including keywords, directives and optional clauses. Borrowing grammars of OpenMP is a good idea. output: dene your own AstAttribute in order to store any possible pragma information generated by your parser. The attribute data structure should store the AST subtrees generated by a parser. The Attribute itself should be attached to relevant statement node (pragma declarations or others) in the original AST. parser: write your recursive descent pragma parser by leveraging the functions dened in rose sourcetree/src/frontend/SageIII/astFromString/AstFromString.h. The parsing results will be used to generate your attribute which should be attached to your pragma declaration statement (or a following statement for a Fortran comment). use of parsing results: in another phase, write your traversal to recognize the attached attribute to do whatever you plan to do with the pragma information. A full example is provided under rose/projects/pragmaParsing to demonstrate the entire workow of using parser building blocks (helper functions within AstFromString) to create a parser to parse user-dened pragmas. We will use this example in this tutorial.
1 Description of basic recursive descent parsing techniques can be found at http://en.wikipedia.org/wiki/ Recursive descent parser

233

234

CHAPTER 33. PARSER BUILDING BLOCKS

33.1

Grammar Examples

The rst step of building a parser is to formally dene the grammar of input strings. The online Doxygen web reference lists helper functions from the AstFromString namespace, with detailed information about the underneath grammars they try to recognize. These grammar can be used as example about how to prepare grammars. For example bool afs match cast expression () follows the grammar like: cast expression : ( type name ) cast expression | unary expression, which means a cast expression is either a unary expression or another cast expression prepended with a type name enclosed in a pair of parenthesis. Note that the grammar has a rule with a right recursion here (cast expression : ( type name ) cast expression). The grammars should try to avoid left recursion (e.g., result : result something else ), which may leads innite recursive calls in your parser. Again, a helper function in AstFromString often implements a grammar. Please take a look at some of them to see how grammars are written to facilitate building recursive descent parsers. The pragma in the pragmaParsing project has the following grammar (documented in rose/projects/pragmaParsing/hcpragma.h ): ---------- grammar begin ----------% string means literal string to be matched % | means alternation hcc_pragma = #pragma hc_simple_part | hc_cuda_part hc_simple_part = hc entry| suspendable | entry suspendable | suspendable entry hc_cuda_part = CUDA kernel_part| place_part kernel_part = kernel % place could be an expression % the grammar uses assignment_expression instead of expression to disallow comma expressions % (list of expressions connected with comma) e.g. exp1, exp2 will be parsed to be (ex1, exp2) % otherwise. % place_part = assignment_expression autodim_part | dim_part % autodim(<dim1>[, <dim2>, <dim3>, <shared_size>]) % [ ] means optional % , means , to be simple % assignment_expression is used to avoid parsing exp1, exp2, exp3 to be one single comma % expression ((exp1,exp2),exp3) autodim_part = autodim ( assignment_expression [, assignment_expression [, assignment_expression [, assignment_expression ] ] ] ) % dim(blocksPerGrid, threadsPerBlock[, shared_size]) dim_part = dim ( assignment_expression , assignment_expression ,

33.2. ASTATTRIBUTE TO STORE RESULTS [ , assignment_expression ] )

235

The example grammar allows a list of expressions inside a pragma. A tricky part here is that C allows single comma expression like ((exp1,exp2),exp3). We use assignment expression to avoid parsing exp1, exp2, exp3 to be one such single comma expression. The point here is that the terms in the grammar have to be accurately mapped to formal C grammar terms. Some familiarity of formal C grammar terms, as shown at http://www.antlr.org/grammar/ 1153358328744/C.g, is required since helper functions have names matching the formal terms in C grammar. The assignment expressions, not expressions, are what we care about in this particular simple grammar.

33.2

AstAttribute to Store results

Once the grammar is dened with terms matching helper functions of AstFromString, a data structure is needed to store the results of parsing. It is recommended to create your data structure by inheriting AstAttribute, which can be attached to any AST nodes with location information. As in the pragmaParsing project, we dene a few classes as the following: class HC_PragmaAttribute: public AstAttribute { public: SgNode * node; enum hcpragma_enum pragma_type; ... }; class HC_CUDA_PragmaAttribute: public HC_PragmaAttribute { public: SgExpression* place_exp; ... }; class HC_CUDA_autodim_PragmaAttribute: public HC_CUDA_PragmaAttribute { public: SgExpression* dim1_exp; SgExpression* dim2_exp; SgExpression* dim3_exp; SgExpression* shared_size_exp; ... };

The point is that the class is inherited from AstAttribute and it contains elds to store all terms dened in the grammar.

236

CHAPTER 33. PARSER BUILDING BLOCKS

33.3

The AstFromString Namespace

AstFromString has a few namespace scope variables, such as: char c char: this indicates the current position of the input string being parsed. SgNode c sgnode: a SgNode variable storing the current anchor AST node, which servers as a start point to nd enclosing scopes for resolving identiers/symbols discovered by a parser. SgNode c parsed node: a SgNode variable storing the root of an AST subtree generated by the parser. In general, your parser should initialize c char with the rst position of an input string to be parsed. It should also set c sgnode to be the pragma statement when you parse pragma strings, or the immediate following statement when you parse Fortran comments. The results often are stored in c parsed node. Your parser should timely check the results and lling your AstAttribute structure with the AST substrees for identiers, constants, expressions, etc. Helper functions within AstFromString include functions to parse identiers, types, substrings, expressions. AST pieces will be generated automatically as needed. So users can focus on building their grammar and parser without doing repetitive chores of parsing common language constructs. Take bool afs match assignment expression() as an example, this function will try to match an expression that satises a grammar rule like: assignment expression : lvalue assignment operator assignment expression | conditional expression . If a successful match is found, the function returns true. In the meantime, the function builds an AST subtree to represent the matched expression and stores the subtree into the variable SgNode c sgnode.

33.4

Write your parsers using parser building blocks

rose/src/frontend/SageIII/astFromString/AstFromString.cpp has the implementation of all parser building blocks (helper functions) for a wide range of grammar rules. They can serve as examples about how to hand-write additional functions to recognize your own grammars. For example, to implement a simple grammar rule like type qualier : const | volatile , we have the following helper function: /* type_qualifier : const | volatile ; */ bool afs_match_type_qualifier() { bool result = false;

33.4. WRITE YOUR PARSERS USING PARSER BUILDING BLOCKS

237

const char* old_char = c_char; if (afs_match_substr("const")) { c_parsed_node = buildConstVolatileModifier (SgConstVolatileModifier::e_const); result = true; } else if (afs_match_substr("volatile")) { c_parsed_node = buildConstVolatileModifier (SgConstVolatileModifier::e_volatile); result = true; } if (result == false) c_char = old_char; return result; } Please note that the function above tries to undo any side eects if the parsing fails. If successful, the parsed result will be stored into c parsed node. Here is another example with right recursion: /* Grammar is cast_expression : ( type_name ) cast_expression | unary_expression ; */ bool afs_match_cast_expression() { bool result = false; const char* old_char = c_char; if (afs_match_unary_expression()) { if (isSgExpression(c_parsed_node)) result = true; } else if (afs_match_char(()) { if (afs_match_type_name()) { SgType* t = isSgType(c_parsed_node); assert (t!= NULL); if (afs_match_char())) { if (afs_match_cast_expression()) { SgExpression* operand = isSgExpression(c_parsed_node);

238

CHAPTER 33. PARSER BUILDING BLOCKS c_parsed_node = buildCastExp(operand, t); result = true; // must set this!! } else { c_char = old_char; } } else { c_char = old_char; } } else { c_char = old_char; result = false; } } if (result == false) return result; c_char = old_char;

} sourcetree/projects/pragmaParsing/hcpragma.C gives a full example of how to use helper functions inside your own parsing functions.

33.5

Limitations

Currently, the parser building blocks support C only. Expressions parsing functions are ready to be used by users. Statement parsing is still ongoing work.

Chapter 34

Handling Comments, Preprocessor Directives, And Adding Arbitrary Text to Generated Code
What To Learn From These Examples Learn how to access existing comments and CPP directives and modify programs to include new ones. Where such comments can be automated they can be used to explain transformations or for more complex transformations using other tools designed to read the generated comments. Also included is how to add arbitrary text to the generated code (often useful for embedded system programming to support back-end compiler specic functionality). This chapter deals with comments and preprocessor directives. These are often dropped from compiler infrastructures and ignored by make tools. ROSE takes great care to preserve all comments and preprocessor directives. Where they exist in the input code we save them (note that EDG drops them from their AST) and weave them back into the AST. Note that #pragma is not a CPP directive and is part of the C and C++ grammar, thus represented explicitly in the AST (see SgPragmaDeclaration).

34.1

How to Access Comments and Preprocessor Directives

Comments and CPP directives are treated identically within ROSE and are saved as special preprocessor attributes to IR nodes within the AST. Not all IR nodes can have these specic type of attributes, only SgLocatedNodes can be associated with such preprocessor attributes. The more general persistent attribute mechanism within ROSE is separate from this preprocessor attribute mechanism and is available on a wider selection of IR nodes. 239

240CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

34.1.1

Source Code Showing How to Access Comments and Preprocessor Directives

Figure 34.1 shows an example translator which access the comments and preprocessor directives on each statement. Note that in this example the AST is traversed twice, rst header les are ignored and then the full AST (including header les) are traversed (generated additional comments). The input code is shown in gure 34.2, the output of this code is shown in gure 34.3 for the source le only. Figure 34.4 shows the same input code processed to output comments and preprocessor directives assembled from the source le and all header les.

34.1.2

Input to example showing how to access comments and CPP directives

Figure 34.2 shows the example input used for demonstration of how to collect comments and CPP directives.

34.1.3

Comments and CPP Directives collected from source le (skipping headers)

Figure 34.3 shows the results from the collection of comments and CPP directives within the input source le only (without -rose:collectAllCommentsAndDirectives).

34.1.4

Comments and CPP Directives collected from source le and all header les

Figure 34.4 shows the results from the collection of comments and CPP directives within the input source le and all headers (with -rose:collectAllCommentsAndDirectives).

34.2

Collecting #dene C Preprocessor Directives

This example shows how to collect the #dene directives as a list for later processing.

34.2.1

Source Code Showing How to Collect #dene Directives

Figure 34.5 shows an example translator which access the comments and preprocessor directives on each statement. Note that in this example the AST is traversed twice, rst header les are ignored and then the full AST (including header les) are traversed (generated additional comments). The input code is shown in gure 34.6, the output of this code is shown in Figure 34.7 shows the same input code processed to output comments and preprocessor directives assembled from the source le and all header les.

34.2. COLLECTING #DEFINE C PREPROCESSOR DIRECTIVES

241

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; // C l a s s d e c l a r a t i o n c l a s s v i s i t o r T r a v e r s a l : public AstSimpleProcessing { public : v i r t u a l v o i d v i s i t ( SgNode n ) ; }; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { // On e a c h node l o o k f o r any comments o f CPP d i r e c t i v e s SgLocatedNode l o c a t e d N o d e = i s S g L o c a t e d N o d e ( n ) ; i f ( l o c a t e d N o d e != NULL) { A t t a c h e d P r e p r o c e s s i n g I n f o T y p e comments = l o c a t e d N o d e >g e t A t t a c h e d P r e p r o c e s s i n g I n f o ( ) ; if ( comments != NULL) { p r i n t f ( Found a t t a c h e d comments ( t o IR node a t %p o f t y p e : %s ) : \ n , l o c a t e d N o d e , l o c a t e d N o d e >c l a s s n a m e ( ) int counter = 0; AttachedPreprocessingInfoType : : i t e r a t o r i ; f o r ( i = comments >b e g i n ( ) ; i != comments >end ( ) ; i ++) { p r i n t f ( Attached Comment #%d i n f i l e %s ( r e l a t i v e P o s i t i o n=%s ) : c l a s s i f i c a t i o n %s : \ n%s \ n , c o u n t e r ++,( i )> g e t f i l e i n f o ()> g e t f i l e n a m e S t r i n g ( ) . c s t r ( ) , ( ( i )> g e t R e l a t i v e P o s i t i o n ( ) == P r e p r o c e s s i n g I n f o : : b e f o r e ) ? b e f o r e : a f t e r , P r e p r o c e s s i n g I n f o : : d i r e c t i v e T y p e N a m e ( ( i )> g e t T y p e O f D i r e c t i v e ( ) ) . c s t r ( ) , ( i )> g e t S t r i n g ( ) . c s t r ( ) ) ; } } else { p r i n t f ( No a t t a c h e d comments ( a t %p o f t y p e : %s ) : \ n , l o c a t e d N o d e , l o c a t e d N o d e >s a g e c l a s s n a m e ( ) ) ; }

} } i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // // // // // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST T r a v e r s e a l l h e a d e r f i l e s and s o u r c e f i l e ( t h e r o s e : c o l l e c t A l l C o m m e n t s A n d D i r e c t i v e s commandline o p t i o n c o n t r o l s i f comments and CPP d i r e c t i v e s a r e s e p a r a t e l y e x t r a c t e d from h e a d e r f i l e s ) . exampleTraversal . t r a v e r s e ( project , preorder ) ; exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; return 0; }

Figure 34.1: Example source code showing how to access comments.

242CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14

// #i n c l u d e < s t d i o . h> #d e f i n e SOURCE CODE BEFORE INCLUDE A #d e f i n e SOURCE CODE BEFORE INCLUDE B #i n c l u d e <i n p u t C o d e c o l l e c t C o m m e n t s . h> #d e f i n e SOURCE CODE AFTER INCLUDE A #d e f i n e SOURCE CODE AFTER INCLUDE B // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e i n t main ( ) { return 0; }

Figure 34.2: Example source code used as input to collection of comments and CPP directives.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

No a t t a c h e d comments ( a t 0 x 2 b 7 b 1 c 3 7 f 0 1 0 o f t y p e : S g G l o b a l ) : Found a t t a c h e d comments ( t o IR node a t 0 x2 b7 b1 c48 99 0 8 o f t y p e : S g F u n c t i o n D e c l a r a t i o n ) : Attached Comment #0 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am // #i n c l u d e < s t d i o . h>

Attached Comment #1 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am #d e f i n e SOURCE CODE BEFORE INCLUDE A

Attached Comment #2 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am #d e f i n e SOURCE CODE BEFORE INCLUDE B

Attached Comment #3 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am #i n c l u d e <i n p u t C o d e c o l l e c t C o m m e n t s . h>

Attached Comment #4 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am #d e f i n e SOURCE CODE AFTER INCLUDE A

Attached Comment #5 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am #d e f i n e SOURCE CODE AFTER INCLUDE B

Attached Comment #6 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /am // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e No No No No No attached attached attached attached attached comments comments comments comments comments ( at ( at ( at ( at ( at 0 x2 b7b1 c5 46 6 28 o f t y p e : S g F u n c t i o n P a r a m e t e r L i s t ) : 0 x2 b7b1 c7 57 0 10 o f t y p e : S g F u n c t i o n D e f i n i t i o n ) : 0 x2 b7b1 c7 a0 0 10 o f t y p e : S g B a s i c B l o c k ) : 0 x f 5 4 c 3 2 0 o f t y p e : SgReturnStmt ) : 0 xf561c00 o f type : SgIntVal ) :

Figure 34.3: Output from collection of comments and CPP directives on the input source le only.

34.2.2

Input to example showing how to access comments and CPP directives

Figure 34.6 shows the example input used for demonstration of how to collect comments and CPP directives.

34.3. AUTOMATED GENERATION OF COMMENTS

243

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

No a t t a c h e d comments ( a t 0 x 2 a e f f 1 7 2 7 0 1 0 o f t y p e : S g G l o b a l ) : Found a t t a c h e d comments ( t o IR node a t 0 x 2 a e f f 1 8 1 0 9 0 8 o f t y p e : S g F u n c t i o n D e c l a r a t i o n ) : Attached Comment #0 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t // #i n c l u d e < s t d i o . h> Attached Comment #1 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t #d e f i n e SOURCE CODE BEFORE INCLUDE A Attached Comment #2 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t #d e f i n e SOURCE CODE BEFORE INCLUDE B Attached Comment #3 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t #i n c l u d e <i n p u t C o d e c o l l e c t C o m m e n t s . h> Attached Comment #4 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t #d e f i n e SOURCE CODE AFTER INCLUDE A Attached Comment #5 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t #d e f i n e SOURCE CODE AFTER INCLUDE B Attached Comment #6 i n f i l e / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t // main program : c o l l e c t C o m m e n t s i n p u t t e s t c o d e No No No No No attached attached attached attached attached comments comments comments comments comments ( at ( at ( at ( at ( at 0 x2aeff18cd628 o f type : SgFunctionParameterList ) : 0 x2aeff1ade010 o f type : S g F u n c t i o n D e f i n i t i o n ) : 0 x2aeff1b27010 o f type : SgBasicBlock ) : 0 xd979420 o f t y p e : SgReturnStmt ) : 0 xd98ed00 o f t y p e : S g I n t V a l ) :

Figure 34.4: Output from collection of comments and CPP directives on the input source le and all header les.

34.2.3

Comments and CPP Directives collected from source le and all header les

Figure 34.7 shows the results from the collection of comments and CPP directives within the input source le and all headers (with -rose:collectAllCommentsAndDirectives).

34.3

Automated Generation of Comments

Figure 34.8 shows an example of how to introduce comments into the AST which will then show up in the generated source code. The purpose for this is generally to add comments to where transformations are introduced. If the code is read by the use the generated comments can be useful in identifying, marking, and/or explaining the transformation. This chapter presents an example translator which just introduces a comment at the top of each function. The comment includes the name of the function and indicates that the comment is automatically generated. Where appropriate such techniques could be used to automate the generation of documentation templates in source code that would be further lled in by the used. In this case the automatically generated templates would be put into the generated source code and a patch formed between the generated source and the original source. The patch could be easily inspected and applied to the original source code to place the documentation templates into the original source. The skeleton of the documentation in the source code could been be lled in

244CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT by the use. The template would have all relevant information obtained by analysis (function parameters, system functions used, security information, side-eects, anything that could come from an analysis of the source code using ROSE).

34.3.1

Source Code Showing Automated Comment Generation

Figure 34.8 shows an example translator which calls the mechanism to add a comment to the IR node representing a function declaration (SgFunctionDeclaration). The input code is shown in gure 34.9, the output of this code is shown in gure 34.10.

34.3.2

Input to Automated Addition of Comments

Figure 34.9 shows the example input used for demonstration of an automated commenting.

34.3.3

Final Code After Automatically Adding Comments

Figure 34.10 shows the results from the addition of comments to the generated source code.

34.4

Addition of Arbitrary Text to Unparsed Code Generation

This section is dierent from the comment generation (section 34.3) because it is more exible and does not introduce any formatting. It also does not use the same internal mechanism, this mechanism supports the addition of new strings or the replacement of the IR node (where the string is attached) with the new string. It is fundamentally lower level and a more powerful mechanism to support generation of tailored output where more than comments, CPP directives, or AST transformation are required. It is also much more dangerous to use. This mechanism is expected to be used rarely and sparingly since no analysis of the AST is likely to leverage this mechanism and search for code that introduced as a transformation here. Code introduced using this mechanism is for the most part unanalyzable since it would have to be reparsed in the context of the location in the AST were it is attached. (Technically this is possible and is the subject of the existing ROSE AST Rewrite mechanism, but that is a dierent subject). Figure 34.11 shows an example of how to introduce arbitrary text into the AST for output by the unparser which will then show up in the generated source code. The purpose for this is generally to add backend compiler or tool specic code generation which dont map to any formal language constructs and so cannot be represented in the AST. However, since most tools that require specialized annotations read them as comments, the mechanism in the previous section 34.3 may be more appropriate. It is because this is not always that case that we have provide this more general mechanism (often useful for embedded system compilers).

34.4.1

Source Code Showing Automated Arbitrary Text Generation

Figure 34.11 shows an example translator which calls the mechanism to add a arbitrary text to the IR node representing a function declaration (SgFunctionDeclaration).

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

245

The input code is shown in gure 34.12, the output of this code is shown in gure 34.13.

34.4.2

Input to Automated Addition of Arbitrary Text

Figure 34.12 shows the example input used for demonstration of the automated introduction of text via the unparser.

34.4.3

Final Code After Automatically Adding Arbitrary Text

Figure 34.13 shows the results from the addition of arbitrary text to the generated source code.

246CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; // B u i l d a s y n t h e s i z e d a t t r i b u t e f o r t h e t r e e t r a v e r s a l class SynthesizedAttribute { public : // L i s t o f #d e f i n e d i r e c t i v e s ( s a v e t h e P r e p r o c e s s i n g I n f o o b j e c t s // s o t h a t we have a l l t h e s o u r c e c o d e p o s i t i o n i n f o r m a t i o n ) . l i s t < P r e p r o c e s s i n g I n f o > a c c u m u l a t e d L i s t ; void display () const ; }; void SynthesizedAttribute : : display () const { l i s t <P r e p r o c e s s i n g I n f o > : : c o n s t i t e r a t o r i = a c c u m u l a t e d L i s t . b e g i n ( ) ; w h i l e ( i != a c c u m u l a t e d L i s t . end ( ) ) { p r i n t f ( CPP d e f i n e d i r e c t i v e = %s \ n , ( i )> g e t S t r i n g ( ) . c s t r ( ) ) ; i ++; } } class { visitorTraversal public : // v i r t u a l virtual }; SynthesizedAttribute v i s i t o r T r a v e r s a l : : e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode n , { SynthesizedAttribute localResult ; : p u b l i c AstBottomUpProcessing < S y n t h e s i z e d A t t r i b u t e >

f u n c t i o n must be d e f i n e d SynthesizedAttribute evaluateSynthesizedAttribute ( SgNode n , S y n t h e s i z e d A t t r i b u t e s L i s t c h i l d A t t r i b u t e s

);

SynthesizedAttributesList

childAttributes )

// p r i n t f ( I n e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( n = %p = %s ) \ n , n , n >c l a s s n a m e ( ) . c s t r ( ) ) ;

// B u i l d t h e l i s t from c h i l d r e n ( i n r e v e r s e o r d e r t o p r e s e r v e t h e f i n a l o r d e r i n g ) f o r ( S y n t h e s i z e d A t t r i b u t e s L i s t : : r e v e r s e i t e r a t o r c h i l d = c h i l d A t t r i b u t e s . r b e g i n ( ) ; c h i l d != c h i l d A t t r i b u t { l o c a l R e s u l t . accumulatedList . s p l i c e ( l o c a l R e s u l t . accumulatedList . begin ( ) , child >a c c u m u l a t e d L i s t ) ; } // Add i n t h e i n f o r m a t i o n from t h e c u r r e n t node SgLocatedNode l o c a t e d N o d e = i s S g L o c a t e d N o d e ( n ) ; i f ( l o c a t e d N o d e != NULL) { AttachedPreprocessingInfoType commentsAndDirectives = locatedNode >g e t A t t a c h e d P r e p r o c e s s i n g I n f o ( ) ; if

( c o m m e n t s A n d D i r e c t i v e s != NULL) { // p r i n t f ( Found a t t a c h e d comments ( t o IR node a t %p o f t y p e : %s ) : \ n , l o c a t e d N o d e , l o c a t e d N o d e >c // i n t c o u n t e r = 0 ;

// Use a r e v e r s e i t e r a t o r s o t h a t we p r e s e r v e t h e o r d e r when u s i n g p u s h f r o n t t o add e a c h d i r e c t i v AttachedPreprocessingInfoType : : r e v e r s e i t e r a t o r i ; f o r ( i = commentsAndDir ectiv es > r b e g i n ( ) ; i != commentsAndDir ect ives >r e n d ( ) ; i ++) { // The d i f f e r e n t c l a s s i f i c a t i o n s o f comments and d i r e c t i v e s a r e i n ROSE/ s r c / f r o n t e n d / S a g e I I I / i f ( ( i )> g e t T y p e O f D i r e c t i v e ( ) == P r e p r o c e s s i n g I n f o : : C p r e p r o c e s s o r D e f i n e D e c l a r a t i o n ) { #i f 0 p r i n t f ( Attached Comment #%d i n f i l e %s ( r e l a t i v e P o s i t i o n=%s ) : c l a s s i f i c a t c o u n t e r ++,( i )> g e t f i l e i n f o ()> g e t f i l e n a m e S t r i n g ( ) . c s t r ( ) , ( ( i )> g e t R e l a t i v e P o s i t i o n ( ) == P r e p r o c e s s i n g I n f o : : b e f o r e ) ? b e f o r e : a f t e r , P r e p r o c e s s i n g I n f o : : d i r e c t i v e T y p e N a m e ( ( i )> g e t T y p e O f D i r e c t i v e ( ) ) . c s t r ( ) , ( i )> g e t S t r i n g ( ) . c s t r ( ) ) ; #e n d i f // u s e p u s h f r o n t ( ) t o end up w i t h s o u r c e o r d e r i n g o f l o c a l R e s u l t . accumulatedList . p u s h f r o n t ( i ) ; } } } } // p r i n t f ( l o c a l R e s u l t a f t e r a d d i n g c u r r e n t node i n f o \ n ) ; // l o c a l R e s u l t . d i s p l a y ( ) ; final list of directives

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

247

1 2 3 4

#d e f i n e JUST A MACRO j u s t a m a c r o #d e f i n e ANOTHER MACRO a n o t h e r m a c r o

Figure 34.6: Example source code used as input to collection of comments and CPP directives.

1 2 3 4 5 6 7 8 9 10 11

CPP d e f i n e CPP d e f i n e CPP d e f i n e CPP d e f i n e CPP d e f i n e CPP d e f i n e

d i r e c t i v e = #d e f i n e max ( a , b ) ( ( a ) > ( b ) ? ( a ) : ( b ) ) d i r e c t i v e = #d e f i n e maxint ( a , b ) ( { i n t a = (a) , b = (b ) ; a > b ? a : b ; })

d i r e c t i v e = #d e f i n e SOURCE CODE BEFORE INCLUDE A d i r e c t i v e = #d e f i n e SOURCE CODE BEFORE INCLUDE B d i r e c t i v e = #d e f i n e SOURCE CODE AFTER INCLUDE A d i r e c t i v e = #d e f i n e SOURCE CODE AFTER INCLUDE B

Figure 34.7: Output from collection of comments and CPP directives on the input source le and all header les.

248CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; class { }; v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { SgFunctionDeclaration functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { s t r i n g comment = s t r i n g ( Autocomment f u n c t i o n name : ) + functionDeclaration >get name ( ) . s t r ( ) + i s now a commented f u n c t i o n ; // Note t h a t t h i s f u n c t i o n w i l l add t h e // o r / / comment s y n t a x a s r e q u i r e d S a g e I n t e r f a c e : : attachComment ( f u n c t i o n D e c l a r a t i o n , comment ) ; } visitorTraversal public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

f o r C o r C++, o r F o r

SgValueExp valueExp = i s S g V a l u e E x p ( n ) ; i f ( valueExp != NULL) { // Check i f t h e r e i s an e x p r e s s i o n t r e e from t h e o r i g i n a l u n f o l d e d e x p r e s s i o n . // T h i s i s a t r i v i a l example o u f t h e o u t p u t o f an a n a l y s i s r e s u l t . s t r i n g comment = s t r i n g ( Autocomment v a l u e : ) + ( ( valueExp > g e t o r i g i n a l E x p r e s s i o n T r e e ( ) != NULL) ? t h i s I S a c o n s t a n t f o l d e d v a l u e : t h i s i s NOT a c o n s t a n t f o l d e d v a l u e ) ; S a g e I n t e r f a c e : : attachComment ( valueExp , comment ) ; } } // T y p i c a l main f u n c t i o n f o r ROSE t r a n s l a t o r i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 34.8: Example source code showing how automate comments.

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

249

1 2 3 4 5 6 7 8

int foo () { int x = 2; x += 3 + 4 ; return x ; }

Figure 34.9: Example source code used as input to automate generation of comments.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

// Autocomment f u n c t i o n name : int foo () { int x = // Autocomment v a l u e : 2; x += // Autocomment v a l u e : 3 + // Autocomment v a l u e : 4; return x ; }

foo

i s now a commented f u n c t i o n

this

i s NOT a c o n s t a n t f o l d e d v a l u e

this this

i s NOT a c o n s t a n t f o l d e d v a l u e i s NOT a c o n s t a n t f o l d e d v a l u e

Figure 34.10: Output of input code after automating generation of comments.

250CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

// Example ROSE T r a n s l a t o r : u s e d w i t h i n ROSE/ t u t o r i a l #i n c l u d e r o s e . h u s i n g namespace s t d ; class { }; visitorTraversal public : v i r t u a l void : public AstSimpleProcessing

v i s i t ( SgNode n ) ;

v o i d v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { SgFunctionDeclaration functionDeclaration = isSgFunctionDeclaration (n ) ; i f ( f u n c t i o n D e c l a r a t i o n != NULL) { // T h i s i s an example o f a XYZ t o o l s p e c i f i c a n n o t a t i o n s t r i n g c o m p i l e r S p e c i f i c D i r e c t i v e = \ n# i f XYZ TOOL \ n \ b u i l t i n \ \ n#e n d i f \ n ; S a g e I n t e r f a c e : : addTextForUnparser ( f u n c t i o n D e c l a r a t i o n , c o m p i l e r S p e c i f i c D i r e c t i v e , A s t U n p a r s e A t t r i b u t e : } SgValueExp valueExp = i s S g V a l u e E x p ( n ) ; i f ( valueExp != NULL) { // Add a backend s p e c i f i c c o m p i l e r d i r e c t i v e s t r i n g c o m p i l e r S p e c i f i c D i r e c t i v e = \ n# i f CRAY \ n c r a y s p e c i f i c a t t r i b u t e \ n#e n d i f \ n ; S a g e I n t e r f a c e : : addTextForUnparser ( valueExp , c o m p i l e r S p e c i f i c D i r e c t i v e , A s t U n p a r s e A t t r i b u t e : : e b e f o r e ) ; } } // T y p i c a l main f u n c t i o n f o r ROSE t r a n s l a t o r i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e AST u s e d by ROSE SgProject p r o j e c t = fr ontend ( argc , argv ) ; // B u i l d t h e t r a v e r s a l o b j e c t v i s i t o r T r a v e r s a l exampleTraversal ; // C a l l t h e t r a v e r s a l s t a r t i n g a t t h e p r o j e c t node o f t h e AST exampleTraversal . t r a v e r s e I n p u t F i l e s ( project , preorder ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 34.11: Example source code showing how automate the introduction of arbitrary text.

1 2 3 4 5 6

int foo () { int x = 42; return x ; }

Figure 34.12: Example source code used as input to automate generation of arbitrary text.

34.4. ADDITION OF ARBITRARY TEXT TO UNPARSED CODE GENERATION

251

1 2 3 4 5 6 7 8 9 10 11 12 13

# i f XYZ TOOL builtin #e n d i f int foo () { int x = # i f CRAY cray specific attribute #e n d i f 42; return x ; }

Figure 34.13: Output of input code after automating generation of arbitrary text.

252CHAPTER 34. HANDLING COMMENTS, PREPROCESSOR DIRECTIVES, AND ADDING ARBITRARY TEXT

Chapter 35

Partial Redundancy Elimination (PRE)

Figure 35.1 shows an example of how to call the Partial Redundancy Elimination (PRE) implemented by Jeremiah Willcock. This transformation is useful for cleaning up code generated from other transformations (used in Qings loop optimizations).

35.1

Source Code for example using PRE

Figure 35.1 shows an example translator which calls the PRE mechanism. The input code is shown in gure 35.2, the output of this code is shown in gure 35.3.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// Example t r a n s l a t o r d e m o n t r a t i n g P a r t i a l Redundancy E l i m i n a t i o n (PRE ) . #i n c l u d e r o s e . h #i n c l u d e CommandOptions . h i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . s t d : : v e c t o r <s t d : : s t r i n g > l = C o m m a n d l i n e P r o c e s s i n g : : g e n e r a t e A r g L i s t F r o m A r g c A r g v ( a r g c , a r g v ) ; CmdOptions : : G e t I n s t a n c e ()> S e t O p t i o n s ( a r g c , a r g v ) ; SgProject project = frontend ( l ) ; PRE : : p a r t i a l R e d u n d a n c y E l i m i n a t i o n ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

Figure 35.1: Example source code showing how use Partial Redundancy Elimination (PRE).

253

254

CHAPTER 35. PARTIAL REDUNDANCY ELIMINATION (PRE)

35.2

Input to Example Demonstrating PRE

Figure 35.2 shows the example input used for demonstration of Partial Redundancy Elimination (PRE) transformation.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

// Program , b a s e d on example i n Knoop e t a l ( Optimal c o d e motion : t h e o r y and // p r a c t i c e , ACM TOPLAS 1 6 ( 4 ) , 1 9 9 4 , pp . 1117 1155 , a s c i t e d i n P a l e r i e t a l // ( s e e p r e . C ) ) , c o n v e r t e d t o C++ i n t unknown ( ) ; // ROSE bug : void foo () { i n t a , b , c , x , y , z , w; ( unknown ( ) ) { y = a + b; a = c; // Added by J e r e m i a h W i l l c o c k t o t e s t w = a + b; a = b; x = a + b; w = a + b; a = c; // End o f added p a r t x = a + b; } if if i n c l u d i n g body r e t u r n 0 ; h e r e doesn t work

l o c a l PRE

( unknown ( ) ) { w h i l e ( unknown ( ) ) { y = a + b ; } } e l s e i f ( unknown ( ) ) { w h i l e ( unknown ( ) ) {} i f ( unknown ( ) ) { y = a + b ; } e l s e { g o t o L9 ; } // FIXME : t h e PRE c o d e c r a s h e s } else { g o t o L10 ; } z = a + b; a = c; L9 : x = a + b ; L10 : } i n t unknown ( ) { 0 ; // Works around ROSE bug return 0; } i n t main ( i n t , c h a r ) { foo ( ) ; return 0; } 0 ; // ROSE bug : u s i n g r e t u r n ; h e r e doesn t work

if

t h i s isn t in a block

Figure 35.2: Example source code used as input to program to the Partial Redundancy Elimination (PRE) transformation.

35.3. FINAL CODE AFTER PRE TRANSFORMATION

255

35.3

Final Code After PRE Transformation

Figure 35.3 shows the results from the use of PRE on an the example input code.

256

CHAPTER 35. PARTIAL REDUNDANCY ELIMINATION (PRE)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74

// Program , b a s e d on example i n Knoop e t a l ( Optimal c o d e motion : t h e o r y and // p r a c t i c e , ACM TOPLAS 1 6 ( 4 ) , 1 9 9 4 , pp . 1117 1155 , a s c i t e d i n P a l e r i e t a l // ( s e e p r e . C ) ) , c o n v e r t e d t o C++ // ROSE bug : i n c l u d i n g body r e t u r n 0 ; h e r e doesn t work i n t unknown ( ) ; void foo () { // P a r t i a l redundancy e l i m i n a t i o n : c a c h e v a r 1 i s a c a c h e o f ( a + b ) int cachevar 1 ; int a ; int b; int c ; int x ; int y ; int z ; i n t w; i f ( ( unknown ( ) ) ) { y = (a + b); a = c; // Added by J e r e m i a h W i l l c o c k t o t e s t l o c a l PRE w = (a + b); a = b; cachevar 1 = (a + b ) ; x = cachevar 1 ; w = cachevar 1 ; a = c; // End o f added p a r t x = (a + b); } else { } i f ( ( unknown ( ) ) ) { cachevar 1 = (a + b ) ; w h i l e ( ( unknown ( ) ) ) { y = cachevar 1 ; } } e l s e i f ( ( unknown ( ) ) ) { w h i l e ( ( unknown ( ) ) ) { } // FIXME : t h e PRE c o d e c r a s h e s i f t h i s i s n t i n a b l o c k i f ( ( unknown ( ) ) ) { cachevar 1 = (a + b ) ; y = cachevar 1 ; } else { g o t o L9 ; } } else { g o t o L10 ; } z = cachevar 1 ; a = c; L9 : x = (a + b); // ROSE bug : u s i n g r e t u r n ; h e r e doesn t work L10 : 0; } i n t unknown ( ) { // Works around ROSE bug 0; return 0; } i n t main ( i n t { foo ( ) ; return 0; } , char )

Figure 35.3: Output of input code after Partial Redundancy Elimination (PRE) transformation.

Chapter 36

Calling the Inliner

Figure 36.1 shows an example of how to use the inline mechanism. This chapter presents an example translator to to inlining of function calls where they are called. Such transformations are quite complex in a number of cases (one case is shown in the input code; a function call in a for loop conditional test). The details of functionality are hidden from the user and a high level interface is provided.

36.1

Source Code for Inliner

Figure 36.1 shows an example translator which calls the inliner mechanism. The code is designed to only inline up to ten functions. the list of function calls is recomputed after any function call is successfully inlined. The input code is shown in gure 36.2, the output of this code is shown in gure 36.3.

36.2

Input to Demonstrate Function Inlining

Figure 36.2 shows the example input used for demonstration of an inlining transformation.

36.3

Final Code After Function Inlining

Figure 36.3 shows the results from the inlining of three function calls. The rst two function calls are the same, and trivial. The second function call appears in the test of a for loop and is more complex.

257

258

CHAPTER 36. CALLING THE INLINER

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81

// Example d e m o n s t r a t i n g f u n c t i o n #i n c l u d e r o s e . h u s i n g namespace s t d ;

i n l i n i n g ( maximal i n l i n i n g , up t o p r e s e t number o f

inlinings ).

// T h i s i s a f u n c t i o n i n Qing s AST i n t e r f a c e v o i d F i x S g P r o j e c t ( S g P r o j e c t& p r o j ) ; i n t main ( i n t a r g c , c h a r a r g v [ ] ) { // B u i l d t h e p r o j e c t o b j e c t (AST) which we w i l l f i l l up w i t h m u l t i p l e f i l e s and u s e a s a // h a n d l e f o r a l l p r o c e s s i n g o f t h e AST( s ) a s s o c i a t e d w i t h one o r more s o u r c e f i l e s . S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g c , a r g v ) ; // DQ ( 7 / 2 0 / 2 0 0 4 ) : Added i n t e r n a l c o n s i s t a n c y AstTests : : runAllTests ( p r o j e c t ) ; b o o l modifiedAST = t r u e ; int count = 0; // I n l i n e one c a l l a t a t i m e u n t i l do { modifiedAST = f a l s e ; a l l have been i n l i n e d . Loops on r e c u r s i v e c o d e . t e s t s on AST

// B u i l d a l i s t o f f u n c t i o n s w i t h i n t h e AST R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n C a l l L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n C a l l E x p ) // Loop o v e r a l l f u n c t i o n c a l l s // f o r ( l i s t <SgNode > : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; i != f u n c t i o n C a l l L i s t . end ( ) ; R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n C a l l L i s t . b e g i n ( ) ; w h i l e ( modifiedAST == f a l s e && i != f u n c t i o n C a l l L i s t . end ( ) ) { SgFunctionCallExp f u n c t i o n C a l l = isSgFunctionCallExp ( i ) ; ROSE ASSERT( f u n c t i o n C a l l != NULL ) ; // Not a l l f u n c t i o n c a l l s can be i n l i n e d i n C++, s o r e p o r t bool s u c e s s f u l l y I n l i n e d = doInline ( functionCall ) ; if if successful .

i ++)

( s u c e s s f u l l y I n l i n e d == t r u e ) { // As s o o n a s t h e AST i s m o d i f i e d recompute t h e l i s t o f f u n c t i o n // c a l l s ( and r e s t a r t t h e i t e r a t i o n s o v e r t h e m o d i f i e d l i s t ) modifiedAST = t r u e ; } else { modifiedAST = f a l s e ; } list iterator

// I n c r e m e n t t h e i ++; }

// Q u i t e when we have c e a s e d t o do any i n l i n e t r a n s f o r m a t i o n s // and o n l y do a p r e d e f i n e d number o f i n l i n e t r a n s f o r m a t i o n s c o u n t++; } w h i l e ( modifiedAST == t r u e && c o u n t < 1 0 ) ; // C a l l f u n c t i o n t o p o s t p r o c e s s t h e AST and f i x u p symbol t a b l e s FixSgProject ( p r o j e c t ) ; // Rename e a c h v a r i a b l e d e c l a r a t i o n renameVariables ( p r o j e c t ) ; // Fold up b l o c k s flattenBlocks ( project ); // Clean up i n l i n e r g e n e r a t e d c o d e cleanupInlinedCode ( project ) ; // Change members t o p u b l i c changeAllMembersToPublic ( p r o j e c t ) ; // DQ ( 3 / 1 1 / 2 0 0 6 ) : T h i s f a i l s s o t h e i n l i n i n g , o r t h e AST I n t e r f a c e // s u p p o r t , n e e d s more work even though i t g e n e r a t e d good c o d e . // A s t T e s t s : : r u n A l l T e s t s ( p r o j e c t ) ; r e t u r n backend ( p r o j e c t ) ; }

36.3. FINAL CODE AFTER FUNCTION INLINING

259

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t x v o i d incrementX ( ) { x++; } int foo () { int a = 0; while ( a < 5) { ++a ; } return a + 3; } i n t main ( i n t , c h a r ) { // Two t r i v a l f u n c t i o n incrementX ( ) ; incrementX ( ) ;

calls

to i n l i n e

// Somthing more i n t e r e s t i n g t o i n l i n e for ( ; foo () < 7;) { x++; } return x ; }

Figure 36.2: Example source code used as input to program to the inlining transformation.

260

CHAPTER 36. CALLING THE INLINER

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

// T h i s t e s t c o d e i s a c o m b i n a t i o n o f p a s s 1 and p a s s 7 , s e l e c t e d somewhat randomly // from Jeremiah s t e s t c o d e o f h i s i n l i n i n g t r a n s f o r m a t i o n from summer 2 0 0 4 . int x = 0; // F u n c t i o n i t i n c r e m e n t x v o i d incrementX ( ) { x++; } int foo () { int a 0 = 0; while ( a 0 < 5){ ++a 0 ; } return a 0 + 3; } i n t main ( i n t , c h a r ) { x++; x++; // Somthing more i n t e r e s t i n g t o i n l i n e for ( ; true ; ) { int a 1 = 0; while ( a 1 < 5){ ++a 1 ; } int rose temp 7 0 = a 1 + 3; bool rose temp 2 = ( bool )( rose temp i f (! rose temp 2 ) { break ; } else { } x++; } return x ; }

0 < 7);

Figure 36.3: Output of input code after inlining transformations.

Chapter 37

Using the AST Outliner

Outlining is the process of replacing a block of consecutive statements with a function call to a new function containing those statements. Conceptually, outlining the inverse of inlining (Chapter 36). This chapter shows how to use the basic experimental outliner implementation included in the ROSE projects directory. There are two basic ways to use the outliner. The rst is a user-level method, in which you may use a special pragma to mark outline targets in the input program, and then call a high-level driver routine to process these pragmas. You can also use command line option to specify outlining targets using abstract handle strings (detailed in Chapter 46). The second method is to call low-level outlining routines that operate directly on AST nodes. After a brief example of what the outliner can do and a discussion of its limits (Sections 37.137.2), we discuss each of these methods in Sections 37.3 and 37.5, respectively.

37.1

An Outlining Example

Figure 37.1 shows a small program with a pragma marking the outline target, a nested for loop, and Figure 37.2 shows the result. The outliner extracts the loop and inserts it into the body of a new function, and inserts a call to that function. The outlined codes input and output variables are wrapped up as parameters to this function. We make the following observations about this output. Placement and forward declarations. The function itself is placed, by default, at the end of the input le to guarantee that it has access to all of the same declarations that were available at the outline target site. The outliner inserts any necessary forward declarations as well, including any necessary friend declarations if the outline target appeared in a class member function. Calling convention. The outliner generates a C-callable function (extern C, with pointer arguments). This design choice is motivated by our need to use the outliner to extract code into external, dynamically loadable library modules. 261

262

CHAPTER 37. USING THE AST OUTLINER

37.2

Limitations of the Outliner

The main limitation of the outliner implementation is that it can only outline single SgStatement nodes. However, since an SgStatement node may be a block (i.e., an SgBasicBlock node), a single statement may actually comprise a sequence of complex statements. The rationale for restricting to single SgStatement nodes is to avoid subtly changing the programs semantics when outlining code. Consider the following example, in which we wish to outline the middle 3 lines of executable code. 2 4 6 int x = 5 ; // START o u t l i n i n g h e r e . foo (x ) ; Object y ( x ) ; y . foo ( ) ; // STOP o u t l i n i n g h e r e . y . bar ( ) ; This example raises a number of issues. How should an outliner handle the declaration of y, which constructs an object in local scope? It cannot just cut-and-paste the declaration of y to the body of the new outlined function because that will change its scope and lifetime, rendering the call to y.bar() impossible. Additionally, it may be unsafe to move the declaration of y so that it precedes the outlined region because the constructor call may have side-eects that could aect the execution of foo(x). It is possible to heap-allocate y inside the body of the

10

15

namespace N { class A { i n t f o o ( void ) const { return 7 ; } i n t b a r ( void ) const { return f o o ( ) / 2 ; } public : i n t b i z ( void ) const { int r e s u l t = 0 ; # pragma r o s e o u t l i n e f o r ( i n t i = 1 ; i <= f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= b a r ( ) ; j ++) r e s u l t += i j ; return r e s u l t ; } }; } extern C i n t p r i n t f ( const char fmt , i n t main ( ) { N: :A x ; p r i n t f ( %d \ n , x . b i z return 0 ; } ...);

20

25

( ) ) ; // P r i n t s

168

Figure 37.1: inputCode OutlineLoop.cc: Sample input program. The #pragma directive marks the nested for loop for outlining.

37.2. LIMITATIONS OF THE OUTLINER

( int r e s u l t p , const void t h i s

263

extern C void O U T namespace N { 5

13785

ptr

);

class A { public : f r i e n d void : : O U T

13785 ( int r e s u l t p , const void t h i s p t r

);

10

private : i n l i n e i n t f o o ( ) const { return 7 ; } i n l i n e i n t b a r ( ) const { return ( t h i s )> f o o ( ) / 2 ; } public : i n l i n e i n t b i z ( ) const { // //A d e c l a r a t i o n f o r t h i s p o i n t e r const c l a s s A t h i s p t r = this ; int r e s u l t = 0 ; OUT 1 13785 (& r e s u l t , & t h i s p t r return r e s u l t ; } } ; } extern C { i n t p r i n t f ( const char fmt , . . . ) ; } i n t main ( ) { class N: :A x ; // P r i n t s 1 6 8 p r i n t f ( %d \ n , x . b i z return 0 ; }

15

20

25

);

30

35

40

());

45

50

extern C void O U T 1 1 3 7 8 5 ( int r e s u l t p , const void t h i s p t r p ) { i n t &r e s u l t = ( ( i n t ) r e s u l t p ); = ( ( const c l a s s N : : A ) t h i s p t r p ); const c l a s s N : : A & t h i s p t r f o r ( i n t i = 1 ; i <= t h i s p t r > f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= t h i s p t r >b a r ( ) ; j ++) r e s u l t += ( i j ) ; }

Figure 37.2: rose outlined-inputCode OutlineLoop.cc: The nested for loop of Figure 37.1 has been outlined.

outlined function so that it can be returned to the caller and later freed, but it is not clear if changing y from a stack-allocated variable to a heap-allocated one will always be acceptable, particularly if the developer of the original program has, for instance, implemented a customized memory allocator. Restricting outlining to well-dened SgStatement objects avoids these issues. It is possible to build a higher-level outliner that extends the outliners basic infrastructure to handle these and other issues.

264

CHAPTER 37. USING THE AST OUTLINER

The outliner cannot outline all possible SgStatement nodes. However, the outliner interface provides a routine, outliner::isOutlineable(s), for testing whether an SgStatement object s is known to satisfy the outliners preconditions (see Section 37.5 for details).

37.3

User-Directed Outlining via Pragmas

Figure 37.3 shows the basic translator, outline, that produces Figure 37.2 from Figure 37.1. This translator extends the identity translator with an include directive on line 5 of Figure 37.3, and a call to the outliner on line 16. All outliner routines live in the Outliner namespace. Here, the call to Outliner::outlineAll (proj) on line 16 traverses the AST, looks for #pragma rose outline directives, outlines the SgStatement objects to which each pragma is attached, and returns the number of outlined objects. A slightly lower-level outlining primitive. The Outliner::outlineAll() routine is a wrapper around calls to a simpler routine, Outliner::outline(), that operates on pragmas: Outliner : : Result O u t l i n e r : : o u t l i n e ( SgPragmaDeclaration s ) ;

Given a pragma statement AST node s, this routine checks if s is a rose outline directive, and if so, outlines the statement with which s is associated. It returns a Outliner::Result object, which is simply a structure that points to (a) the newly generated outlined function and (b) the statement corresponding to the new function call (i.e., the outlined function call-site). See Outliner.hh or the ROSE Programmers Reference for more details. The Outliner::outlineAll() wrapper. The advantage of using the wrapper instead of the lower-level primitive is that the wrapper processes the pragmas in an order that ensures the outlining can be performed correctly in-place. This order is neither a preorder nor a postorder traversal, but in fact a reverse preorder traversal; refer to the wrappers documentation for an explanation.

37.4

Outlining via Abstract Handles

The ROSE AST outliner also allows users to specify outlining targets using abstract handles (details are given in Chapter 46) without relying on planting pragmas into the source code. For the translator (e.g. named outline) built from the source shown in Figure 37.3, it accepts a command line option in a form of -rose:outline:abstract handle handle string. The outline program is able to locate a language construct matching the handle string within an input source le and then outline the construct. For example, a handle string ForStatementposition,12 will tell the outliner to outline the for loop at source position line 12. Another handle, FunctionDeclarationname,initialize::ForStatementnumbering,2 indicates that the outlining target is the second loop within a function named initializer. Figure 37.5 shows the outlining results using the rst handle(ForStatementposition,12 ) from an input source le (shown in Figure 37.4). Figure 37.6 shows the results using the second handle string for the same input.

37.4. OUTLINING VIA ABSTRACT HANDLES

// ! o u t l i n e . cc : Demonstrates t h e pragma i n t e r f a c e o f t h e O u t l i n e r . #include < r o s e . h> #include < i o s t r e a m > 5 #include < O u t l i n e r . hh> #include < v e c t o r > #include < s t r i n g > using namespace s t d ; 10 int main ( i n t a r g c , char a r g v [ ] ) { // ! A c c e p t i n g command l i n e o p t i o n s t o t h e o u t l i n e r v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v+a r g c ) ; O u t l i n e r : : commandLineProcessing ( a r g v L i s t ) ; SgProject proj = frontend ( argvList ) ; ROSE ASSERT ( p r o j ) ; 20 c e r r << [ O u t l i n i n g . . . ] << e n d l ; s i z e t count = O u t l i n e r : : o u t l i n e A l l ( p r o j ) ; c e r r << [ P r o c e s s e d << c o u n t << o u t l i n e return backend ( p r o j ) ; } d i r e c t i v e s . ] << e n d l ;

265

15

25

Figure 37.3: outline.cc: A basic outlining translator, which generates Figure 37.2 from Figure 37.1. This outliner relies on the high-level driver, Outliner::outlineAll(), which scans the AST for outlining pragma directives (#pragma rose outline) that mark outline targets.

#d e f i n e MSIZE 500 i n t n ,m, m i t s ; double t o l , r e l a x = 1 . 0 , a l p h a = 0 . 0 5 4 3 ; double u [ MSIZE ] [ MSIZE ] , f [ MSIZE ] [ MSIZE ] , u o l d [ MSIZE ] [ MSIZE ] ; double dx , dy ; void i n i t i a l i z e ( ) { i n t i , j , xx , yy ; dx = 2 . 0 / ( n 1 ); dy = 2 . 0 / (m 1 ); f o r ( i =0; i <n ; i ++) f o r ( j =0; j < m; j ++) { xx =( i n t ) ( 1.0 + dx ( i 1 ) ) ; yy = ( i n t ) ( 1 . 0 + dy ( j 1 ) ) ; u[ i ][ j ] = 0.0; f [ i ] [ j ] = 1.0 a l p h a (1.0 xx xx ) ( 1 . 0 yy yy ) \ 2 . 0 ( 1 . 0 xx xx ) 2.0 (1.0 yy yy ) ; } }

10

15

20

Figure 37.4: inputCode OutlineLoop2.c: Sample input program without pragmas.

266

CHAPTER 37. USING THE AST OUTLINER

10

#d e f i n e MSIZE 500 int n ; i n t m; int mits ; double t o l ; double r e l a x = 1 . 0 ; double a l p h a = 0 . 0 5 4 3 ; double u [ 5 0 0UL ] [ 5 0 0 UL ] ; double f [ 5 0 0UL ] [ 5 0 0 UL ] ; double u o l d [ 5 0 0UL ] [ 5 0 0 UL ] ; double dx ; double dy ; void O U T 1 1 1 8 9 0 ( i n t i p

, int j p

, int xxp

, int yyp

);

15

20

void i n i t i a l i z e ( ) { int i ; int j ; i n t xx ; i n t yy ; dx = ( 2 . 0 / ( n 1 ) ) ; dy = ( 2 . 0 / (m 1 ) ) ; O U T 1 1 1 8 9 0 (& i ,& j ,&xx ,& yy ) ; } void O U T 1 1 1 8 9 0 ( i n t i p , int j p , int xxp , int yyp ) { for ( i p = 0; ip < n ; ( i p )++) = 0; jp < m; ( j p )++) { for ( j p xxp = ( ( i n t ) ( 1 . 0 + ( dx ( i p 1)))); yyp = ( ( i n t ) ( 1 . 0 + ( dy ( j p 1)))); u[ ip ][ jp ] = 0.0; f [ ip ][ jp ] = ((((( 1.0 alpha ) ( 1 . 0 ( xxp yyp ) ) ) (2.0 (1.0 ( xxp xxp ) ) ) (1.0 ( yyp xxp ) ) ) ) (2.0 (1.0 ( yyp yyp ) ) ) ) ; } }

25

30

35

Figure 37.5: rose inputCode OutlineLoop2.c: The loop at line 12 of Figure 37.12 has been outlined.

37.5

Calling Outliner Directly on AST Nodes

The preceding examples rely on the outliners #pragma interface to identify outline targets. In this section, we show how to call the outliner directly on SgStatement nodes from within your translator. Figure 37.7 shows an example translator that nds all if statements and outlines them. A sample input appears in Figure 37.8, with the corresponding output shown in Figure 37.9. Notice that valid preprocessor control structure is accounted for and preserved in the output. The translator has two distinct phases. The rst phase selects all outlineable if-statements, using the CollectOutlineableIfs helper class. This class produces a list that stores the targets in an order appropriate for outlining them in-place. The second phase iterates over the list of statements and outlines each one. The rest of this section explains these phases, as well as various aspects of the sample input and output.

37.5. CALLING OUTLINER DIRECTLY ON AST NODES

267

10

#d e f i n e MSIZE 500 int n ; i n t m; int mits ; double t o l ; double r e l a x = 1 . 0 ; double a l p h a = 0 . 0 5 4 3 ; double u [ 5 0 0UL ] [ 5 0 0 UL ] ; double f [ 5 0 0UL ] [ 5 0 0 UL ] ; double u o l d [ 5 0 0UL ] [ 5 0 0 UL ] ; double dx ; double dy ; void O U T 1 1 1 8 9 0 ( i n t i , i n t j p void i n i t i a l i z e ( ) { int i ; int j ; i n t xx ; i n t yy ; dx = ( 2 . 0 / ( n 1 ) ) ; dy = ( 2 . 0 / (m 1 ) ) ; f o r ( i = 0 ; i < n ; i ++) O U T 1 1 1 8 9 0 ( i ,& j ,&xx ,& yy ) ; }

, int xxp

, int yyp

);

15

20

25

30

35

void O U T 1 1 1 8 9 0 ( i n t i , i n t j p , int xxp , int yyp ) { for ( j p = 0; jp < m; ( j p )++) { xxp = ( ( i n t ) ( 1 . 0 + ( dx ( i 1 ) ) ) ) ; yyp = ( ( i n t ) ( 1 . 0 + ( dy ( j p 1)))); u[ i ][ jp ] = 0.0; f [ i ][ jp ] = ((((( 1.0 alpha ) ( 1 . 0 ( xxp xxp ) ) ) (1.0 ( yyp xxp ) ) ) ) (2.0 (1.0 ( yyp yyp ) ) ) (2.0 (1.0 ( xxp yyp ) ) ) ) ; } }

Figure 37.6: rose inputCode OutlineLoop2b.c: The 2nd loop within a function named initializefrom Figure 37.12 has been outlined.

37.5.1

Selecting the outlineable if statements

Line 45 of Figure 37.7 builds a list, ifs (declared on line 44), of outlineable if-statements. The helper class, CollectOutlineableIfs in lines 1235, implements a traversal to build this list. Notice that a node is inserted into the target list only if it satises the outliners preconditions; this check is the call to Outliner::isOutlineable() on line 28. The function Outliner::isOutlineable() also accepts an optional second boolean parameter (not shown). When this parameter is true and the statement cannot be outlined, the check will print an explanatory message to standard error. Such messages are useful for discovering why the outliner will not outline a particular statement. The default value of this parameter is false.

37.5.2

Properly ordering statements for in-place outlining

Each call to Outliner::outline(*i) on line 50 of Figure 37.7 outlines a target if-statement *i in if targets. However, in order for these statements to be outlined in-place, it is essential to

268

CHAPTER 37. USING THE AST OUTLINER

// o u t l i n e I f s . cc : C a l l s O u t l i n e r d i r e c t l y t o o u t l i n e #include < r o s e . h> #include < i o s t r e a m > #include < s e t > #include < l i s t > #include < O u t l i n e r . hh> using namespace s t d ;

if

statements .

10 // T r a v e r s a l t o g a t h e r a l l o u t l i n e a b l e S g I f S t m t nodes . c l a s s C o l l e c t O u t l i n e a b l e I f s : public A s t S i m p l e P r o c e s s i n g { public : // Container o f l i s t s t a t e m e n t s i n o u t l i n e a b l e o r d e r . typedef l i s t <S g I f S t m t > I f L i s t t ; // C a l l t h i s r o u t i n e t o g a t h e r t h e o u t l i n e t a r g e t s . s t a t i c void c o l l e c t ( S g P r o j e c t p , I f L i s t t & f i n a l ) { CollectOutlineableIfs collector ( final ); c o l l e c t o r . traverseInputFiles (p , postorder ) ; } v i r t u a l void v i s i t ( SgNode n ) { SgIfStmt s = isSgIfStmt (n ) ; i f ( Outliner : : isOutlineable ( s )) f i n a l t a r g e t s . push back ( s ) ; } private : C o l l e c t O u t l i n e a b l e I f s ( I f L i s t t& f i n a l ) : f i n a l t a r g e t s ( f i n a l ) {} I f L i s t t & f i n a l t a r g e t s ; // F i n a l l i s t o f o u t l i n e t a r g e t s . }; //= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = i n t main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j = frontend ( argc , argv ) ; ROSE ASSERT ( p r o j ) ; #i f 1 // B u i l d a s e t o f o u t l i n e a b l e i f s t a t e m e n t s . CollectOutlineableIfs : : IfList t i f s ; C o l l e c t O u t l i n e a b l e I f s : : c o l l e c t ( proj , i f s ) ; // O u t l i n e them a l l . for ( C o l l e c t O u t l i n e a b l e I f s : : I f L i s t t : : i t e r a t o r i = i f s . begin ( ) ; i != i f s . end ( ) ; ++i ) Outliner : : o u t l i n e ( i ) ; #e l s e p r i n t f ( S k i p p i n g o u t l i n i n g due t o r e c e n t move from s t d : : l i s t t o s t d : : v e c t o r i n ROSE \ n ) ; #e n d i f // Unparse return backend ( p r o j ) ; }

15

20

25

30

35

40

45

50

55

Figure 37.7: outlineIfs.cc: A lower-level outlining translator, which calls Outliner::outline() directly on SgStatement nodes. This particular translator outlines all SgIfStmt nodes.

outline the statements in the proper order.

37.5. CALLING OUTLINER DIRECTLY ON AST NODES

269

#include < i o s t r e a m > using namespace s t d ; 5 #d e f i n e LEAP YEAR 0 i n t main ( i n t a r g c , char a r g v [ ] ) { f o r ( i n t i = 1 ; i < a r g c ; ++i ) { s t r i n g month ( a r g v [ i ] ) ; s i z e t days = 0 ; i f ( month == January | | month == March | | month == May | | month == J u l y | | month == August | | month == Oc t o b er | | month == December ) days = 3 1 ; #i f LEAP YEAR e l s e i f ( month == F eb r u a r y ) days = 2 9 ; #e l s e e l s e i f ( month == F eb r u a r y ) days = 2 8 ; #e n d i f e l s e i f ( month == A p r i l | | month == June | | month == September | | month == November ) days = 3 0 ; c o u t << a r g v [ i ] << << days << e n d l ; } return 0 ; }

10

15

20

25

30

35

Figure 37.8: inputCode Ifs.cc: Sample input program, without explicit outline targets specied using #pragma rose outline, as in Figures 37.1 and 37.12. The postorder traversal implemented by the helper class, CollectOutlineableIfs, produces the correct ordering. To see why, consider the following example code: 2 4 6 8 i f ( a ) // [ 1 ] { i f ( b ) f o o ( ) ; // [ 2 ] } e l s e i f ( c ) // [ 3 ] { i f ( d ) bar ( ) ; // [ 4 ] } The corresponding AST is (roughly) SgIfStmt:[1] / \ / \ SgIfStmt:[2] SgIfStmt:[3]

270 | SgIfStmt:[4]

CHAPTER 37. USING THE AST OUTLINER

The postorder traversal2, 4, 3, 1ensures that child if-statements are outlined before their parents.
#include < i o s t r e a m > using namespace s t d ; #d e f i n e LEAP YEAR 0 extern C void O U T extern C void O U T extern C void O U T

1 2 3

11083 11083 11083

( void monthp ( void monthp ( void monthp

, s i z e t daysp , s i z e t daysp , s i z e t daysp

); ); );

10

15

i n t main ( i n t a r g c , char a r g v [ ] ) { f o r ( i n t i = 1 ; i < a r g c ; ++i ) { s t d : : s t r i n g month ( a r g v [ i ] ) ; s i z e t days = 0 ; O U T 3 1 1 0 8 3 (&month ,& days ) ; ( ( (& s t d : : c o u t)<< a r g v [ i ]<< ) << days ) << s t d : : e n d l < char , s t d : : c h a r t r a i t s < char > > ; } return 0 ; } extern C void O U T 1 1 1 0 8 3 ( void monthp , s i z e t d a y s p ) { s t d : : s t r i n g &month = ( ( s t d : : s t r i n g ) monthp ) ; s i z e t &days = ( ( s i z e t ) d a y s p ) ; i f ( ( ( month== A p r i l | | month== June ) | | month== September ) days = 3 0 ; } extern C void O U T 2 1 1 0 8 3 ( void monthp , s i z e t d a y s p { s t d : : s t r i n g &month = ( ( s t d : : s t r i n g ) monthp ) ; s i z e t &days = ( ( s i z e t ) d a y s p ) ; #i f 1 / #i f LEAP YEAR . . . #e l s e / ; i f ( month== F e br ua r y ) days = 2 8 ; else #e n d i f O U T 1 1 1 0 8 3 (&month ,& days ) ; } )

20

| | month== November )

25

30

35

40

45

50

extern C void O U T 3 1 1 0 8 3 ( void monthp , s i z e t d a y s p ) { s t d : : s t r i n g &month = ( ( s t d : : s t r i n g ) monthp ) ; s i z e t &days = ( ( s i z e t ) d a y s p ) ; i f ( ( ( ( ( ( month== January | | month==March ) | | month==May ) days = 3 1 ; else #i f LEAP YEAR #e l s e { #e n d i f / #i f LEAP YEAR . . . #e l s e / O U T 2 1 1 0 8 3 (&month ,& days ) ; } }

| | month== J u l y )

| | month== August )

| | month=

Figure 37.9: rose inputCode Ifs.cc: Figure 37.8, after outlining using the translator in Figure 37.7.

37.6. OUTLINERS PREPROCESSING PHASE

271

37.6

Outliners Preprocessing Phase

Internally, the outliner implementation itself has two distinct phases. The rst is a preprocessing phase, in which an arbitrary outlineable target is placed into a canonical form that is relatively simple to extract. The second phase then creates the outlined function, replacing the original target with a call to the outlined function. It is possible to run just the preprocessing phase, which is useful for understanding or even debugging the outliner implementation.
// o u t l i n e P r e p r o c . cc : Shows t h e o u t l i n e r s p r e p r o c e s s o r o n l y phase . #include < r o s e . h> #include < i o s t r e a m > 5 #include < O u t l i n e r . hh> using namespace s t d ; int main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j = frontend ( argc , argv ) ; ROSE ASSERT ( p r o j ) ; #i f 1 c e r r << [ Running o u t l i n e r s p r e p r o c e s s i n g p h a s e o n l y . . . ] << e n d l ; s i z e t count = O u t l i n e r : : p r e p r o c e s s A l l ( p r o j ) ; [ P r o c e s s e d << c o u n t << o u t l i n e d i r e c t i v e s . ] << e n d l ; c e r r << #e l s e p r i n t f ( S k i p p i n g o u t l i n i n g due t o r e c e n t move from s t d : : l i s t t o s t d : : v e c t o r i n ROSE \ n ) ; #e n d i f c e r r << [ U n p a r s i n g . . . ] << e n d l ; return backend ( p r o j ) ; 25 }

10

15

20

Figure 37.10: outlinePreproc.cc: The basic translator of Figure 37.3, modied to execute the Outliners preprocessing phase only. In particular, the original call to Outliner::outlineAll() has been replaced by a call to Outliner::preprocessAll(). To call just the preprocessor, simply replace a call to Outliner::outlineAll(s) or Outliner::outline(s) with a call to Outliner::preprocessAll(s) or Outliner::preprocess(s), respectively. The translator in Figure 37.10 modies the translator in Figure 37.3 in this way to create a preprocessingonly translator. The preprocessing phase consists of a sequence of initial analyses and transformations that the outliner performs in order to put the outline target into a particular canonical form. Roughly speaking, this form is an enclosing SgBasicBlock node, possibly preceded or followed by start-up and tear-down code. Running just the preprocessing phase on Figure 37.1 produces the output in Figure 37.11. In this example, the original loop is now enclosed in two additional SgBasicBlocks (Figure 37.11, lines 2435), the outermost of which contains a declaration that shadows the objects this pointer, replacing all local references to this with the new shadow pointer. In this case, this initial transformation is used by the main underlying outliner implementation to explicitly identify all references to the possibly implicit references to this. The preprocessing phase is more interesting in the presence of non-local control ow outside the outline target. Consider Figure 37.12, in which the outline target contains two break

272

CHAPTER 37. USING THE AST OUTLINER

namespace N { class A { private : i n l i n e i n t f o o ( ) const { return 7 ; } i n l i n e i n t b a r ( ) const { return ( t h i s )> f o o ( ) / 2 ; } public : i n l i n e i n t b i z ( ) const { // //A d e c l a r a t i o n f o r t h i s p o i n t e r const c l a s s A t h i s p t r = this ; int r e s u l t = 0 ; # pragma r o s e o u t l i n e { f o r ( i n t i = 1 ; i <= t h i s p t r > f o o ( ) ; i ++) f o r ( i n t j = 1 ; j <= t h i s p t r >b a r ( ) ; j ++) r e s u l t += ( i j ) ; } return r e s u l t ; } } ; } extern C { i n t p r i n t f ( const char fmt , } i n t main ( ) { class N: :A x ; // P r i n t s 1 6 8 p r i n t f ( %d \ n , x . b i z return 0 ; }

10

15

20

25

30

35

...);

40

());

45

Figure 37.11: rose outlined pp-inputCode OutlineLoop.cc: Figure 37.1 after outline preprocessing only, i.e., specifying -rose:outline:preproc-only as an option to the translator of Figure 37.3. statements, which require jumping to a regions of code outside the target. We show the preprocessed code in Figure 37.13. The original non-local jumps are rst transformed into assignments to a ag, EXIT TAKEN (lines 1820 and 2629), and then relocated to a subsequent block of code (lines 3853) with their execution controlled by the value of the ag. The nal outlined result appears in Figure 37.14; the initial preprocessing simplies this nal step of extracting the outline target.

37.6. OUTLINERS PREPROCESSING PHASE

273

#include < i o s t r e a m > s i z e t f a c t o r i a l ( s i z e t n) { s i z e t i = 1; s i z e t r = 1; while ( 1 ) { # pragma r o s e o u t l i n e i f ( i <= 1 ) break ; // Non l o c a l jump #1 e l s e i f ( i >= n ) break ; // Non l o c a l jump #2 else r = ++i ; } return r ; } i n t main ( i n t a r g c , char a r g v [ ] ) { s t d : : c o u t << 7 ! == << f a c t o r i a l return 0 ; }

10

15

20

( 7 ) << s t d : : e n d l ; // P r i n t s 5040

Figure 37.12: inputCode OutlineNonLocalJumps.cc: Sample input program, with an outlining target that contains two non-local jumps (here, break statements).

274

CHAPTER 37. USING THE AST OUTLINER

#include < i o s t r e a m > size t factorial { s i z e t i = 1; s i z e t r = 1; while ( 1 ) { ( s i z e t n)

10

15

20

25

30

35

40

45

# pragma r o s e o u t l i n e { i n t EXIT TAKEN = 0 ; { i f ( i <= 1 ) { EXIT TAKEN = 1 ; goto NON LOCAL EXIT ; } e l s e i f ( i >= n ) { EXIT TAKEN = 2 ; goto NON LOCAL EXIT ; } else r = ++i ; NON LOCAL EXIT : ; } i f ( EXIT TAKEN == 1 ) { // Non l o c a l jump #1 break ; } else { i f ( EXIT TAKEN == 2 ) { // Non l o c a l jump #2 break ; } else { } } } } return r ; } i n t main ( i n t a r g c , char a r g v [ ] ) { // P r i n t s 5040 ( ( ( & s t d : : c o u t ) << 7 ! == ) << f a c t o r i a l s t d : : c h a r t r a i t s < char >>; return 0 ; }

50

( 7 ) ) << s t d : : e n d l < char ,

55

Figure 37.13: rose outlined pp-inputCode OutlineNonLocalJumps.cc: The non-local jump example of Figure 37.12 after outliner preprocessing, but before the actual outlining. The non-local jump is handled by an additional ag, EXIT TAKEN , which indicates what non-local jump is to be taken.

37.6. OUTLINERS PREPROCESSING PHASE

275

#include < i o s t r e a m > extern C void O U T

14692

( s i z e t np , s i z e t i p i n t EXIT TAKEN p ) ;

s i z e t rp

10

15

20

25

30

s i z e t f a c t o r i a l ( s i z e t n) { s i z e t i = 1; s i z e t r = 1; while ( 1 ) { { i n t EXIT TAKEN = 0 ; OUT 1 14692 (&n , &i , &r , &EXIT TAKEN i f ( EXIT TAKEN == 1 ) { // Non l o c a l jump #1 break ; } else { i f ( EXIT TAKEN == 2 ) { // Non l o c a l jump #2 break ; } else { } } } } return r ; } i n t main ( i n t a r g c , char a r g v [ ] ) { // P r i n t s 5040 ( ( ( & s t d : : c o u t ) << 7 ! == ) << f a c t o r i a l s t d : : c h a r t r a i t s < char >>; return 0 ; } extern C void O U T 1 14692

);

35

( 7 ) ) << s t d : : e n d l < char ,

40

( s i z e t np , s i z e t i p i n t EXIT TAKEN p ) ); ); ); EXIT TAKEN p

s i z e t rp

45

{ s i z e t & n = (( s i z e t ) np s i z e t & i = (( s i z e t ) i p s i z e t & r = (( s i z e t ) r p i n t &EXIT TAKEN = ( ( i n t ) i f ( i <= 1 ) { EXIT TAKEN = 1 ; goto NON LOCAL EXIT ; } e l s e i f ( i >= n ) { EXIT TAKEN = 2 ; goto NON LOCAL EXIT ; } else r = ++i ; NON LOCAL EXIT : ; }

);

50

55

60

Figure 37.14: rose outlined-inputCode OutlineNonLocalJumps.cc: Figure 37.12 after outlining.

276

CHAPTER 37. USING THE AST OUTLINER

Chapter 38

Loop Optimization
This section is specic to loop optimization and show several tutorial examples using the optimization mechanisms within ROSE.
FIXME: We might want to reference Qings work explicitly since this is really just showing o here work. FIXME: We are not running performance tests within this tutorial, but perhaps we could later.

38.1

Example Loop Optimizer

Simple example translator showing use of pre-dened loop optimizations. Figure 38.1 shows the code required to call some loop optimizations within ROSE. The translator that we build for this tutorial is simple and takes the following command line options to control which optimizations are done.

-ic1 :loop interchange for more reuses -bk1/2/3 <blocksize> :block outer/inner/all loops -fs1/2 :single/multi-level loop fusion for more reuses -cp <copydim> :copy array -fs0 : loop fission -splitloop: loop splitting -unroll [locond] [nvar] <unrollsize> : loop unrolling -bs <stmtsize> : break up statements in loops -annot <filename>: Read annotation from a file which defines side effects of functions -arracc <funcname> : Use special function to denote array access (the special function can be replaced with macros after transformation). This option is for circumventing complex subscript expressions for linearized multi-dimensional arrays. -opt <level=0> : The level of loop optimizations to apply (By default, only the outermost level is optimized). -ta <int> : Max number of nodes to split for transitive dependence analysis (to limit the overhead of transitive dep. analysis) -clsize <int> : set cache line size in evaluating spatial locality (affect decisions in applying loop optimizations) -reuse_dist <int> : set maximum distance of reuse that can exploit cache (used to evaluate 277

278 temporal locality of loops)

CHAPTER 38. LOOP OPTIMIZATION

38.1. EXAMPLE LOOP OPTIMIZER

279

2 4

// LoopProcessor : // Assume no a l i a s i n g // apply loop opt to the bodies of

all

function definitions

// = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = 6 #include r o s e . h 8 10 12 using namespace s t d ; 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 } // Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; } t h e vendor s c o m p i l e r // JJW 10 29 2007 A d j u s t f o r i t e r a t o r i n v a l i d a t i o n and p o s s i b l e // i n s e r t e d s t a t e m e n t s p = s t d : : f i n d ( d e c l L i s t . b e g i n ( ) , d e c l L i s t . end ( ) , f u n c ) ; a s s e r t ( p != d e c l L i s t . end ( ) ) ; } // This w i l l do as much f u s i o n as p o s s i b l e ( f i n e r g r a i n e d // c o n t r o l o v e r l o o p o p t i m i z a t i o n s u s e s a d i f f e r e n t i n t e r f a c e ) . L o o p T r a n s f o r m I n t e r f a c e : : T r a n s f o r m T r a v e r s e ( f a I m p l , AstNodePtrImpl ( s t m t s ) ) ; int main ( i n t a r g c , char a r g v [ ] ) { v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v + a r g c ) ; CmdOptions : : G e t I n s t a n c e ()> S e t O p t i o n s ( a r g v L i s t ) ; AssumeNoAlias a l i a s I n f o ; LoopTransformInterface : : cmdline configure ( argvList ) ; L o o p T r a n s f o r m I n t e r f a c e : : s e t a l i a s I n f o (& a l i a s I n f o ) ; S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g v L i s t ) ; // Loop o v e r t h e number o f f i l e s i n t h e p r o j e c t int filenum = project >n u m b e r O f F i l e s ( ) ; f o r ( i n t i = 0 ; i < f i l e n u m ; ++i ) { SgSourceFile f i l e = isSgSourceFile ( project >g e t f i l e L i s t ( ) [ i ] ) ; SgGlobal r o o t = f i l e >g e t g l o b a l S c o p e ( ) ; S g D e c l a r a t i o n S t a t e m e n t P t r L i s t& d e c l L i s t = r o o t >g e t d e c l a r a t i o n s #include < A s t I n t e r f a c e R O S E . h> #include L o o p T r a n s f o r m I n t e r f a c e . h #include CommandOptions . h

();

// Loop o v e r t h e d e c l a r a t i o n i n t h e g l o b a l s c o p e o f each f i l e f o r ( S g D e c l a r a t i o n S t a t e m e n t P t r L i s t : : i t e r a t o r p = d e c l L i s t . b e g i n ( ) ; p != d e c l L i s t . end ( ) ; ++p ) { SgFunctionDeclaration func = i s S g F u n c t i o n D e c l a r a t i o n (p ) ; i f ( f u n c == NULL) continue ; S g F u n c t i o n D e f i n i t i o n defn = func >g e t d e f i n i t i o n ( ) ; i f ( d e f n == NULL) continue ; SgBasicBlock stmts = defn >g e t b o d y ( ) ; A s t I n t e r f a c e I m p l faImpl ( stmts ) ;

Figure 38.1: Example source code showing use of loop optimization mechanisms.

280

CHAPTER 38. LOOP OPTIMIZATION

38.2

Matrix Multiply Example

Using the matrix multiply example code shown in gure 38.2, we run the loop optimizer in gure 38.1 and generate the code shown in gure 38.3.
// Example program showing m a t r i x m u l t i p l y // ( f o r use w i t h l o o p o p t i m i z a t i o n t u t o r i a l example ) #d e f i n e N 50 i n t main ( ) { int i , j , k ; double a [ N ] [ N ] , b [ N ] [ N ] , c [ N ] [ N ] ; f o r ( i = 0 ; i <= N 1; i +=1) { f o r ( j = 0 ; j <= N 1; j +=1) { f o r ( k = 0 ; k <= N 1; k+=1) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] b[k ][ j ]; } } } return 0 ; }

2 4 6 8 10 12 14 16 18 20 22

Figure 38.2: Example source code used as input to loop optimization processor.

38.2. MATRIX MULTIPLY EXAMPLE

281

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } // Example program showing m a t r i x m u l t i p l y // ( f o r use w i t h l o o p o p t i m i z a t i o n t u t o r i a l example ) #d e f i n e N 50 i n t main ( ) { int i ; int j ; int k ; double a [ 5 0UL ] [ 5 0 UL ] ; double b [ 5 0UL ] [ 5 0 UL ] ; double c [ 5 0UL ] [ 5 0 UL ] ; int var 0 ; int var 1 ; v a r 1 <= 4 9 ; v a r 1 += 1 6 ) { for ( v a r 1 = 0 ; for ( v a r 0 = 0 ; v a r 0 <= 4 9 ; v a r 0 += 1 6 ) { f o r ( k = 0 ; k <= 4 9 ; k += 1 ) { f o r ( i = v a r 1 ; i <= min2 ( 4 9 , v a r 1 + 1 5 ) ; i += 1 ) { f o r ( j = v a r 0 ; j <= min2 ( 4 9 , v a r 0 + 1 5 ) ; j += 1 ) { c [ i ] [ j ] = (c [ i ] [ j ] + (a [ i ] [ k] b[k ] [ j ] ) ) ; } } } } } return 0 ; }

Figure 38.3: Output of loop optimization processor showing matrix multiply optimization (using options: -bk1 -fs0).

282

CHAPTER 38. LOOP OPTIMIZATION

38.3

Loop Fusion Example

Using the loop fusion example code shown in gure 38.4, we run the loop optimizer in gure 38.1 and generate the code shown in gure 38.5.

2 main ( ) { 4 int x [ 3 0 ] , 6 8 10 12 14 } for ( i = x[2 i } for ( i = x[2 i } 1 ; i <= 1 0 ; i += 1 ) { ] = x[2 i + 1] + 2; 1 ; i <= 1 0 ; i += 1 ) { + 3] = x[2 i ] + i ; i;

Figure 38.4: Example source code used as input to loop optimization processor.

2 4 6 8 10 12 14 16 18

i n t main ( ) { i n t x [ 3 0UL ] ; int i ; f o r ( i = 1 ; i <= 1 1 ; i += 1 ) { i f ( i <= 1 0 ) { x[2 i ] = (x [(2 i ) + 1] + 2); } else { } i f ( i >= 2 ) { x [ ( 2 ( 1 + i ) ) + 3 ] = ( x [ 2 ( 1 + i ) ] + ( 1 + i ) ) ; } else { } } return 0 ; }

Figure 38.5: Output of loop optimization processor showing loop fusion (using options: -fs2).

38.4

Example Loop Processor (LoopProcessor.C)

This section contains a more detail translator which uses the command-line for input of specic loop processing options and is more sophisticated than the previous translator used to handle the previous two examples. Figure 38.6 shows the code required to call the loop optimizations within ROSE. The translator that we build for this tutorial is simple and takes command line parameters to control which optimizations are done.

38.4. EXAMPLE LOOP PROCESSOR (LOOPPROCESSOR.C)

283

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

#include r o s e . h #include < g e n e r a l . h> #include p r e . h #include f i n i t e D i f f e r e n c i n g . h

// DQ ( 1 / 2 / 2 0 0 8 ) : I t h i n k t h i s // #i n c l u d e c o p y u n p a r s e r . h #include #include #include #include #include #include

i s no l o n g e r used !

r e w r i t e . h <CommandOptions . h> < A s t I n t e r f a c e R O S E . h> < L o o p T r a n s f o r m I n t e r f a c e . h> < A n n o t C o l l e c t . h> < O p e r a t o r A n n o t a t i o n . h>

using namespace s t d ; #i f d e f USE OMEGA #include < D e p T e s t S t a t i s t i c s . h> extern D e p T e s t S t a t i s t i c s D e p S t a t s ; #e n d i f extern bool DebugAnnot ( ) ; extern void F i x F i l e I n f o ( SgNode n ) ; c l a s s UnparseFormatHelp ; class UnparseDelegate ; void u n p a r s e P r o j e c t ( S g P r o j e c t p r o j e c t , UnparseFormatHelp u n p a r s e H e l p / = NULL / , U n p a r s e D e l e g a t e r e p l / = NULL / ) ; void P r i n t U s a g e ( char name ) { c e r r << name << < o p t i o n s > << <program name> << \ n ; c e r r << g o b j : g e n e r a t e o b j e c t f i l e \ n ; c e r r << o r i g : copy nonm o d i f i e d s t a t e m e n t s from o r i g i n a l f i l e \ n ; c e r r << s p l i t l o o p : a p p l y i n g l o o p s p l i t t i n g t o remove c o n d i t i o n a l s i n s i d e c e r r << ReadAnnotation : : g e t i n s t ()> O p t i o n S t r i n g ( ) << e n d l ; c e r r << p r e : a p p l y p a r t i a l redundancy e l i m i n a t i o n \ n ; c e r r << f d : a p p l y f i n i t e d i f f e r e n c i n g t o a r r a y i n d e x e x p r e s s i o n s \ n ; LoopTransformInterface : : PrintTransformUsage ( c e r r ) ; }

32 34 36 38 40 42

l o o p s \ n ;

Figure 38.6: Detailed example source code showing use of loop optimization mechanisms (loopProcessor.C part 1).

284

CHAPTER 38. LOOP OPTIMIZATION

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42

bool GenerateObj ( ) { return CmdOptions : : G e t I n s t a n c e ()> HasOption ( g o b j ) ; }

int main ( i n t a r g c , { if

char a r g v [ ]

( a r g c <= 1 ) { PrintUsage ( argv [ 0 ] ) ; return 1;

} v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v + a r g c ) ; CmdOptions : : G e t I n s t a n c e ()> S e t O p t i o n s ( a r g v L i s t ) ; AssumeNoAlias a l i a s I n f o ; LoopTransformInterface : : cmdline configure ( argvList ) ; L o o p T r a n s f o r m I n t e r f a c e : : s e t a l i a s I n f o (& a l i a s I n f o ) ; #i f d e f USE OMEGA D e p S t a t s . SetFileName ( b u f f e r . s t r ( ) ) ; #e n d i f OperatorSideEffectAnnotation funcInfo = OperatorSideEffectAnnotation : : g e t i n s t ( ) ; funcInfo >r e g i s t e r a n n o t ( ) ; ReadAnnotation : : g e t i n s t ()> r e a d ( ) ; i f ( DebugAnnot ( ) ) funcInfo >Dump ( ) ; LoopTransformInterface : : s e t s i d e E f f e c t I n f o ( funcInfo ) ; S g P r o j e c t p r o j e c t = new S g P r o j e c t ( a r g v L i s t ) ; int filenum = project >n u m b e r O f F i l e s ( ) ; f o r ( i n t i = 0 ; i < f i l e n u m ; ++i ) { // S g F i l e &s a g e F i l e = s a g e P r o j e c t >g e t f i l e ( i ) ; // S g G l o b a l r o o t = s a g e F i l e . g e t r o o t ( ) ; SgSourceFile f i l e = isSgSourceFile ( project >g e t f i l e L i s t ( ) [ i ] ) ; SgGlobal r o o t = f i l e >g e t g l o b a l S c o p e ( ) ; S g D e c l a r a t i o n S t a t e m e n t P t r L i s t& d e c l L i s t = r o o t >g e t d e c l a r a t i o n s ( ) ; f o r ( S g D e c l a r a t i o n S t a t e m e n t P t r L i s t : : i t e r a t o r p = d e c l L i s t . b e g i n ( ) ; p != d e c l L i s t . end ( ) ; ++p ) {

Figure 38.7: loopProcessor.C source code (Part 2).

38.5. MATRIX MULTIPLICATION EXAMPLE (MM.C)

285

38.5

Matrix Multiplication Example (mm.C)

Using the matrix multiplication example code shown in gure 38.8, we run the loop optimizer in gure 38.6 and generate the code shown in gure 38.9.

2 4 6 8 10 12 14 16 18 20 22 24 26 28

#d e f i n e N 50 void p r i n t m a t r i x ( double x [ ] [ N ] ) ; void i n i t m a t r i x ( double x [ ] [ N ] , double s ) ; main ( ) { int i , j , k ; double a [ N ] [ N] , b [ N ] [ N ] , c [ N ] [ N ] ; double s ; s = 235.0; initmatrix (a , s ); s = 321.0; initmatrix (b , s ) ; printmatrix (a ) ; printmatrix (b ) ; f o r ( i = 0 ; i <= N 1; i +=1) { f o r ( j = 0 ; j <= N 1; j +=1) { f o r ( k = 0 ; k <= N 1; k+=1) { c [ i ][ j ] = c [ i ][ j ] + a[ i ][ k] b[k ][ j ]; } } } printmatrix ( c ) ; }

Figure 38.8: Example source code used as input to loopProcessor, show in gure 38.6.

286

CHAPTER 38. LOOP OPTIMIZATION

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } #d e f i n e N 50 void p r i n t m a t r i x ( double x [ ] [ 5 0 UL ] ) ; void i n i t m a t r i x ( double x [ ] [ 5 0 UL ] , double s ) ; i n t main ( ) { int i ; int j ; int k ; double a [ 5 0UL ] [ 5 0 UL ] ; double b [ 5 0UL ] [ 5 0 UL ] ; double c [ 5 0UL ] [ 5 0 UL ] ; double s ; int var 0 ; int var 1 ; s = 235.0; initmatrix (a , s ); s = 321.0; initmatrix (b , s ) ; printmatrix (a ) ; printmatrix (b ) ; for ( v a r 1 = 0 ; v a r 1 <= 4 9 ; v a r 1 += 1 6 ) { for ( v a r 0 = 0 ; v a r 0 <= 4 9 ; v a r 0 += 1 6 ) { f o r ( k = 0 ; k <= 4 9 ; k += 1 ) { f o r ( i = v a r 1 ; i <= min2 ( 4 9 , v a r 1 + 1 5 ) ; i += 1 ) { f o r ( j = v a r 0 ; j <= min2 ( 4 9 , v a r 0 + 1 5 ) ; j += 1 ) { c [ i ] [ j ] = (c [ i ] [ j ] + (a [ i ] [ k] b[k ] [ j ] ) ) ; } } } } } printmatrix ( c ) ; return 0 ; }

Figure 38.9: Output of loopProcessor using input from gure 38.8 (using options: -bk1 -fs0).

38.6. MATRIX MULTIPLICATION EXAMPLE USING LINEARIZED MATRICES (DGEMM.C)287

38.6

Matrix Multiplication Example Using Linearized Matrices (dgemm.C)

Using the matrix multiplication example code shown in gure 38.10, we run the loop optimizer in gure 38.6 and generate the code shown in gure 38.11.

2 4 6 8 10 12 14 16 18

// Function p r o t o t y p e void dgemm( double a , double b , double c , i n t n ) ; // Function d e f i n i t i o n void dgemm( double a , double b , double c , i n t n ) { int i , j , k ; // i n t n ; f o r ( k =0;k<n ; k+=1){ f o r ( j =0; j <n ; j +=1){ f o r ( i =0; i <n ; i +=1){ c [ j n+i ]= c [ j n+i ]+ a [ k n+i ] b [ j n+k ] ; } } } }

Figure 38.10: Example source code used as input to loopProcessor, show in gure 38.6.

288

CHAPTER 38. LOOP OPTIMIZATION

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56

i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } i n t min2 ( i n t a0 , i n t a1 ) { return a0 < a1 ? a0 : a1 ; } // Function p r o t o t y p e void dgemm( double a , double b , double c , i n t n ) ; // Function d e f i n i t i o n void dgemm( double a , double b , double c , i n t n ) { int i ; int j ; int k ; int var 0 ; int var 1 ; for ( v a r 1 = 0 ; v a r 1 <= 1 + n ; v a r 1 += 1 6 ) { for ( v a r 0 = 0 ; v a r 0 <= 1 + n ; v a r 0 += 1 6 ) { f o r ( i = 0 ; i <= 1 + n ; i += 1 ) { f o r ( k = v a r 1 ; k <= min2( 1 + n , v a r 1 + 1 5 ) ; k += 1 ) { f o r ( j = v a r 0 ; j <= : : min2 ( n + 16 , v a r 0 ) ; j += 1 6 ) { c [ ( j n) + i ] = ( c [ ( j n) + i ] + (a [ ( k n) + i ] b [ ( j c [ ( ( 1 + j ) n) + i ] = ( c [ ( ( 1 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 2 + j ) n) + i ] = ( c [ ( ( 2 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 3 + j ) n) + i ] = ( c [ ( ( 3 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 4 + j ) n) + i ] = ( c [ ( ( 4 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 5 + j ) n) + i ] = ( c [ ( ( 5 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 6 + j ) n) + i ] = ( c [ ( ( 6 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 7 + j ) n) + i ] = ( c [ ( ( 7 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 8 + j ) n) + i ] = ( c [ ( ( 8 + j ) n) + i ] + (a [ ( k n) + c [ ( ( 9 + j ) n) + i ] = ( c [ ( ( 9 + j ) n) + i ] + (a [ ( k n) + c [((10 + j ) n) + i ] = ( c [((10 + j ) n) + i ] + (a [ ( k n) c [((11 + j ) n) + i ] = ( c [((11 + j ) n) + i ] + (a [ ( k n) c [((12 + j ) n) + i ] = ( c [((12 + j ) n) + i ] + (a [ ( k n) c [((13 + j ) n) + i ] = ( c [((13 + j ) n) + i ] + (a [ ( k n) c [((14 + j ) n) + i ] = ( c [((14 + j ) n) + i ] + (a [ ( k n) c [((15 + j ) n) + i ] = ( c [((15 + j ) n) + i ] + (a [ ( k n) } f o r ( ; j <= : : min2( 1 + n , 1 5 + v a r 0 ) ; j += 1 ) { c [ ( j n) + i ] = ( c [ ( j n) + i ] + (a [ ( k n) + i ] b [ ( j } } } } } }

n) + i] i] i] i] i] i] i] i] i] + i] + i] + i] + i] + i] + i]

k])); b[((1 + b[((2 + b[((3 + b[((4 + b[((5 + b[((6 + b[((7 + b[((8 + b[((9 + b[((10 b[((11 b[((12 b[((13 b[((14 b[((15

j) j) j) j) j) j) j) j) j) + + + + + +

n) n) n) n) n) n) n) n) n) j) j) j) j) j) j)

+ k])); + k])); + k])); + k])); + k])); + k])); + k])); + k])); + k])); n) + k ] ) ) ; n) + k ] ) ) ; n) + k ] ) ) ; n) + k ] ) ) ; n) + k ] ) ) ; n) + k ] ) ) ;

n) + k ] ) ) ;

Figure 38.11: Output of loopProcessor using input from gure 38.10 (using options: -bk1 -unroll nvar 16).

38.7. LU FACTORIZATION EXAMPLE (LUFAC.C)

289

38.7

LU Factorization Example (lufac.C)

Using the LU factorization example code shown in gure 38.12, we run the loop optimizer in gure 38.6 and generate the code shown in gure 38.13.
double abs ( double x ) { i f 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 } printmatrix (a ) ; } initmatrix (a , 5.0); printmatrix (a ) ; f o r ( k = 0 ; k<=n 2; k+=1) { p[k] = k; mu = abs ( a [ k ] [ k ] ) ; f o r ( i = k +1; i <= n 1; i +=1) { i f (mu < abs ( a [ i ] [ k ] ) ) { mu = abs ( a [ i ] [ k ] ) ; p[k] = i ; } } f o r ( j = k ; j <= n 1; j +=1) { t = a[k][ j ]; a[k ][ j ] = a[p[k ] ] [ j ]; a[p[k ] ] [ j ] = t ; } f o r ( i = k +1; i <= n 1; i +=1) { a [ i ] [ k ] = a [ i ] [ k]/ a [ k ] [ k ] ; } f o r ( j = k +1; j <=n 1; j +=1) { f o r ( i = k +1; i <=n 1; i +=1) { a [ i ] [ j ] = a [ i ] [ j ] a [ i ] [ k] a [ k ] [ j ] ; } } main ( i n t a r g c , char a r g v [ ] ) { int p [ n ] , i , j , k ; double a [ n ] [ n ] , mu, t ; #d e f i n e n 50 void p r i n t m a t r i x ( double x [ ] [ n ] ) ; void i n i t m a t r i x ( double x [ ] [ n ] , double s ) ; ( x < 0 ) return x ; e l s e return x ; }

Figure 38.12: Example source code used as input to loopProcessor, show in gure 38.6.

290

CHAPTER 38. LOOP OPTIMIZATION

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

double abs ( double x ) { i f ( x < 0) return x ; else return x ; } #d e f i n e n 50 void p r i n t m a t r i x ( double x [ ] [ 5 0 UL ] ) ; void i n i t m a t r i x ( double x [ ] [ 5 0 UL ] , double s ) ; i n t main ( i n t a r g c , char a r g v [ ] ) { i n t p [ 5 0UL ] ; int i ; int j ; int k ; double a [ 5 0UL ] [ 5 0 UL ] ; double mu ; double t ; initmatrix (a , 5 . 0 ) ; printmatrix (a ) ; f o r ( k = 0 ; k <= 4 8 ; k += 1 ) { p[k] = k; mu = a bs ( a [ k ] [ k ] ) ; f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { i f (mu < a bs ( a [ i ] [ k ] ) ) { mu = ab s ( a [ i ] [ k ] ) ; p[k] = i ; } } f o r ( j = k ; j <= 4 9 ; j += 1 ) { t = a[k][ j ]; a[k ][ j ] = a[p[k ] ] [ j ]; a[p[k ] ] [ j ] = t ; } f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { a [ i ] [ k] = (a [ i ] [ k] / a [k ] [ k ] ) ; } f o r ( j = 1 + k ; j <= 4 9 ; j += 1 ) { f o r ( i = 1 + k ; i <= 4 9 ; i += 1 ) { a [ i ] [ j ] = (a [ i ] [ j ] (a [ i ] [ k] a [k ] [ j ] ) ) ; } } } printmatrix (a ) ; return 0 ; }

Figure 38.13: Output of loopProcessor using input from gure 38.12 (using options: -bk1 -fs0 -splitloop -annotation).

38.8. LOOP FUSION EXAMPLE (TRIDVPK.C)

291

38.8

Loop Fusion Example (tridvpk.C)

Using the loop fusion example code shown in gure 38.14, we run the loop optimizer in gure 38.6 and generate the code shown in gure 38.15.
#d e f i n e n 100 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 } f o r ( k =0; k<=n 2; k+=1) f o r ( j =0; j <=n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) t o t [ i ] [ j ] = t o t [ i ] [ j ] + d [ k ] duz [ i ] [ j ] [ k ] ; f o r ( j =0; j <=n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) duz [ i ] [ j ] [ n 1] = ( duz [ i ] [ j ] [ n 1] t o t [ i ] [ j ] ) b [ n 1 ] ; f o r ( j =0; j <=n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) duz [ i ] [ j ] [ n 2]=duz [ i ] [ j ] [ n 2] e [ n 2] duz [ i ] [ j ] [ n 1 ] ; f o r ( k=n 3; k >=0; k+=1) f o r ( j = 0 ; j <= n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) duz [ i ] [ j ] [ k ] = duz [ i ] [ j ] [ k ] c [ k ] duz [ i ] [ j ] [ k +1] e [ k ] duz [ i ] [ j ] [ n 1 ] ; f o r ( j =0; j <=n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) tot [ i ] [ j ] = 0; main ( ) { int i , j , k ; f o r ( j =0; j <=n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) duz [ i ] [ j ] [ 0 ] = duz [ i ] [ j ] [ 0 ] b [ 0 ] ; f o r ( k =1; k<=n 2; k+=1) f o r ( j = 0 ; j <= n 1; j +=1) f o r ( i =0; i <=n 1; i +=1) duz [ i ] [ j ] [ k ] = ( duz [ i ] [ j ] [ k] a [ k ] duz [ i ] [ j ] [ k 1 ] ) b [ k ] ; double a [ n ] , b [ n ] , c [ n ] , d [ n ] , e [ n ] ; double t o t [ n ] [ n ] ; double dux [ n ] [ n ] [ n ] , duy [ n ] [ n ] [ n ] , duz [ n ] [ n ] [ n ] ;

Figure 38.14: Example source code used as input to loopProcessor, show in gure 38.6.

292

CHAPTER 38. LOOP OPTIMIZATION

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36

#d e f i n e n 100 double a [ 1 0 0UL ] ; double b [ 1 0 0UL ] ; double c [ 1 0 0UL ] ; double d [ 1 0 0UL ] ; double e [ 1 0 0UL ] ; double t o t [ 1 0 0UL ] [ 1 0 0 UL ] ; double dux [ 1 0 0UL ] [ 1 0 0 UL ] [ 1 0 0 UL ] ; double duy [ 1 0 0UL ] [ 1 0 0 UL ] [ 1 0 0 UL ] ; double duz [ 1 0 0UL ] [ 1 0 0 UL ] [ 1 0 0 UL ] ; i n t main ( ) { int i ; int j ; int k ; f o r ( i = 0 ; i <= 9 9 ; i += 1 ) { f o r ( j = 0 ; j <= 9 9 ; j += 1 ) { duz [ i ] [ j ] [ 0 ] = ( duz [ i ] [ j ] [ 0 ] b [ 0 ] ) ; tot [ i ] [ j ] = 0; f o r ( k = 0 ; k <= 9 8 ; k += 1 ) { i f ( k >= 1 ) { duz [ i ] [ j ] [ k ] = ( ( duz [ i ] [ j ] [ k ] ( a [ k ] duz [ i ] [ j ] [ k 1 ] ) ) b [ k ] ) ; } else { } t o t [ i ] [ j ] = ( t o t [ i ] [ j ] + ( d [ k ] duz [ i ] [ j ] [ k ] ) ) ; } duz [ i ] [ j ] [ 1 0 0 1 ] = ( ( duz [ i ] [ j ] [ 1 0 0 1 ] t o t [ i ] [ j ] ) b [ 1 0 0 1 ] ) ; duz [ i ] [ j ] [ 1 0 0 2 ] = ( duz [ i ] [ j ] [ 1 0 0 2 ] ( e [ 1 0 0 2 ] duz [ i ] [ j ] [ 1 0 0 1 ] ) ) ; f o r ( k = 9 7 ; k >= 0 ; k += 1) { duz [ i ] [ j ] [ k ] = ( ( duz [ i ] [ j ] [ k ] ( c [ k ] duz [ i ] [ j ] [ k + 1 ] ) ) ( e [ k ] duz [ i ] [ j ] [ 1 0 0 1 ] ) ) ; } } } return 0 ; }

Figure 38.15: Output of loopProcessor input from gure 38.14 (using options: -fs2 -ic1 -opt 1 ).

Chapter 39

Parameterized Code Translation

This chapter gives examples of using ROSEs high level loop translation interfaces to perform parameterized loop transformations, including loop unrolling, interchanging and tiling. The motivation is to give users the maximized exibility to orchestrate code transformations on the targets they want, the order they want, and the parameters they want. One typical application scenario is to support generating desired code variants for empirical tuning. The ROSE internal interfaces (declared within the SageInterface namespace) to call loop transformations are: bool loopUnrolling (SgForStatement *loop, size t unrolling factor) : This function needs two parameters: one for the loop to be unrolled and the other for the unrolling factor. bool loopInterchange (SgForStatement *loop, size t depth, size t lexicoOrder) : The loop interchange function has three parameters, the rst one to specify a loop which starts a perfectly-nested loop and is to be interchanged, the 2nd for the depth of the loop nest to be interchanged, and nally the lexicographical order for the permutation. bool loopTiling (SgForStatement *loopNest, size t targetLevel, size t tileSize) The loop tiling interface needs to know the loop nest to be tiled, which loop level to tile, and the tile size for the level. For eciency concerns, those functions only perform the specied translations without doing any legitimacy check. It is up to the users to make sure the transformations wont generate wrong code. We will soon provide interfaces to do the eligibility check for each transformation. We also provide standalone executable programs (loopUnrolling,loopInterchange, and loopTiling under ROSE INSTALL/bin) for the transformations mentioned above so users can directly use them via command lines and abstract handles (explained in Chapter 46) to orchestrate transformations they want.

39.1

Loop Unrolling

Figure 39.1 gives an example input code for loopUnrolling. An example command line to invoke loop unrolling on the example can look like the following: 293

294

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

2 4 6 8 10 12

int a [ 1 0 0 ] [ 1 0 0 ] ; i n t main ( void ) { int j ; f o r ( i n t i =0; i < 100; i ++) f o r ( j =0; j < 100; j ++) { i n t k =3; a [ i ] [ j ]= i+j+k ; } return 0 ; }

Figure 39.1: Example source code used as input to loopUnrolling # unroll a for statement 5 times. The loop is a statement at line 6 within # an input file. loopUnrolling -c inputloopUnrolling.C \ -rose:loopunroll:abstract_handle "Statement<position,6>" -rose:loopunroll:factor 5 Two kinds of output can be expected after loop unrolling. One (Shown in Figure 39.2) is the case that the loop iteration count is known at compile-time and can be evenly divisible by the unrolling factor. The other case (Shown in Figure 39.3 is when the divisibility is unknown and a fringe loop has to be generated to run possible leftover iterations.

39.1. LOOP UNROLLING

295

i n t a [ 1 0 0UL ] [ 1 0 0 UL ] ; 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 int main ( ) { int j ; f o r ( i n t i = 0 ; i < 1 0 0 ; i ++) { f o r ( j = 0 ; j <= 9 9 ; j += 5 ) { int k = 3 ; a [ i ] [ j ] = (( i + j ) + k ); { int k = 3 ; a [ i ] [ ( j + 1)] = (( i + ( j } { int k = 3 ; a [ i ] [ ( j + 2)] = (( i + ( j } { int k = 3 ; a [ i ] [ ( j + 3)] = (( i + ( j } { int k = 3 ; a [ i ] [ ( j + 4)] = (( i + ( j } } } return 0 ; }

+ 1)) + k ) ;

+ 2)) + k ) ;

+ 3)) + k ) ;

+ 4)) + k ) ;

Figure 39.2: Output for a unrolling factor which can divide the iteration space evenly

296

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

i n t a [ 1 0 0UL ] [ 1 0 0 UL ] ; 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 int main ( ) { int j ; f o r ( i n t i = 0 ; i < 1 0 0 ; i ++) { // i t e r c o u n t = ( ub l b +1)%s t e p ==0?(ub l b +1)/ s t e p : ( ub l b +1)/ s t e p +1; // f r i n g e = i t e r c o u n t%u n r o l l f a c t o r==0 ? 0 : u n r o l l f a c t o r s t e p int lu fringe 1 = 3; f o r ( j = 0 ; j <= 99 l u f r i n g e 1 ; j += 3 ) { int k = 3 ; a [ i ] [ j ] = (( i + j ) + k ); { int k = 3 ; a [ i ] [ ( j + 1)] = (( i + ( j + 1)) + k ) ; } { int k = 3 ; a [ i ] [ ( j + 2)] = (( i + ( j + 2)) + k ) ; } } f o r ( ; j <= 9 9 ; j += 1 ) { int k = 3 ; a [ i ] [ j ] = (( i + j ) + k ); } } return 0 ; }

Figure 39.3: Output for the case when divisibility is unknown at compile-time

39.2. LOOP INTERCHANGE

297

39.2

Loop Interchange

Figure 39.4 gives an example input code for loopInterchange.

void O U T 1 6 1 1 9 ( i n t r i , double rp , i n t s t e n c i l s i z e , i n t hypre { i n t s i , i i , j j , kk ; // t h e f o l l o w i n g 4 l e v e l l o o p n e s t i s t o be i n t e r c h a n g e d f o r ( s i = 0 ; s i < s t e n c i l s i z e ; s i ++) f o r ( kk = 0 ; kk < hypr e m ; kk++) f o r ( j j = 0 ; j j < hypr e m ; j j ++) f o r ( i i = 0 ; i i < hypr e m ; i i ++) rp [ ( r i + i i ) + j j + kk ] = Ap 0 [ i i + j j + kk ] ; } m , const double Ap 0 )

2 4 6 8 10

Figure 39.4: Example source code used as input to loopInterchange An example command line to invoke loop interchange: # interchange a loop nest starting from the first loop within the input file, # interchange depth is 4 and # the lexicographical order is 1 (swap the innermost two levels) loopInterchange -c inputloopInterchange.C -rose:loopInterchange:abstract_handle \ "ForStatement<numbering,1>" -rose:loopInterchange:depth 4 \ -rose:loopInterchange:order 1 Figure 39.5 shows the output.

2 4 6 8 10 12 14

void O U T 1 6 1 1 9 ( i n t r i , double rp , i n t s t e n c i l s i z e , i n t hypre { int s i ; int i i ; int j j ; i n t kk ; // t h e f o l l o w i n g 4 l e v e l l o o p n e s t i s t o be i n t e r c h a n g e d f o r ( s i = 0 ; s i < s t e n c i l s i z e ; s i ++) f o r ( kk = 0 ; kk < hypr e m ; kk++) f o r ( i i = 0 ; i i < hypr e m ; i i ++) f o r ( j j = 0 ; j j < hypr e m ; j j ++) rp [ ( ( r i + i i ) + j j ) + kk ] = Ap 0 [ ( i i + j j ) + kk ] ; }

m , const double Ap 0 )

Figure 39.5: Output for loop interchange

298

CHAPTER 39. PARAMETERIZED CODE TRANSLATION

39.3

Loop Tiling

Figure 39.6 gives an example input code for loopTiling.

#d e f i n e N 100 int i , j , k ; double a [ N ] [ N ] , b [ N ] [ N ] , c [ N ] [ N ] ; i n t main ( ) { f o r ( i = 0 ; i < N ; i ++) f o r ( j = 0 ; j < N ; j ++) f o r ( k = 0 ; k < N ; k++) c [ i ] [ j ]= c [ i ] [ j ]+ a [ i ] [ k ] b [ k ] [ j ] ; return 0 ; }

2 4 6 8 10 12

Figure 39.6: Example source code used as input to loopTiling An example command line to invoke loop tiling: # Tile the loop with a depth of 3 within the first loop of the input file # tile size is 5 loopTiling -c inputloopTiling.C -rose:loopTiling:abstract_handle \ "ForStatement<numbering,1>" -rose:loopTiling:depth 3 -rose:loopTiling:tilesize 5 Figure 39.7 shows the output.
#d e f i n e N 100 int i ; int j ; int k ; double a [ 1 0 0UL ] [ 1 0 0 UL ] ; double b [ 1 0 0UL ] [ 1 0 0 UL ] ; double c [ 1 0 0UL ] [ 1 0 0 UL ] ; i n t main ( ) { int lt var k ; for ( l t v a r k = 0 ; l t v a r k <= 9 9 ; l t v a r k += 5 ) { f o r ( i = 0 ; i < 1 0 0 ; i ++) f o r ( j = 0 ; j < 1 0 0 ; j ++) f o r ( k = l t v a r k ; k <= ( ( ( 9 9 < ( l t v a r k + 5 1 ) ) ? 9 9 c [ i ] [ j ] = (c [ i ] [ j ] + (a [ i ] [ k] b[k ] [ j ] ) ) ; } } return 0 ; }

2 4 6 8 10 12 14 16 18 20

: ( l t v a r k + 5 1 ) ) ) ; k += 1 ) {

Figure 39.7: Output for loop tiling

Part V

Correctness Checking

Tutorials of using ROSE to help program correctness checking or debugging.

299

Chapter 40

Code Coverage
This translator is part of ongoing collaboration with IBM on the support of code coverage analysis tools for C, C++ and F90 applications. the subject of code coverage is much more complex than this example code would cover. The following web site: http://www.bullseye.com/coverage.html contains more information and is the source for the descriptions below. Code coverage can include: Statement Coverage This measure reports whether each executable statement is encountered. Decision Coverage This measure reports whether boolean expressions tested in control structures (such as the if-statement and while-statement) evaluated to both true and false. The entire boolean expression is considered one true-or-false predicate regardless of whether it contains logicaland or logical-or operators. Additionally, this measure includes coverage of switch-statement cases, exception handlers, and interrupt handlers. Condition Coverage Condition coverage reports the true or false outcome of each boolean sub-expression, separated by logical-and and logical-or if they occur. Condition coverage measures the subexpressions independently of each other. Multiple Condition Coverage Multiple condition coverage reports whether every possible combination of boolean subexpressions occurs. As with condition coverage, the sub-expressions are separated by logical-and and logical-or, when present. The test cases required for full multiple condition coverage of a condition are given by the logical operator truth table for the condition. Condition/Decision Coverage Condition/Decision Coverage is a hybrid measure composed by the union of condition coverage and decision coverage. This measure was created at Boeing and is required for aviation software by RCTA/DO-178B. 301

302

CHAPTER 40. CODE COVERAGE Modied Condition/Decision Coverage This measure requires enough test cases to verify every condition can aect the result of its encompassing decision. Path Coverage This measure reports whether each of the possible paths in each function have been followed. A path is a unique sequence of branches from the function entry to the exit. Function Coverage This measure reports whether you invoked each function or procedure. It is useful during preliminary testing to assure at least some coverage in all areas of the software. Broad, shallow testing nds gross deciencies in a test suite quickly. Call Coverage This measure reports whether you executed each function call. The hypothesis is that faults commonly occur in interfaces between modules. Linear Code Sequence and Jump (LCSAJ) Coverage This variation of path coverage considers only sub-paths that can easily be represented in the program source code, without requiring a ow graph. An LCSAJ is a sequence of source code lines executed in sequence. This linear sequence can contain decisions as long as the control ow actually continues from one line to the next at run-time. Sub-paths are constructed by concatenating LCSAJs. Researchers refer to the coverage ratio of paths of length n LCSAJs as the test eectiveness ratio (TER) n+2. Data Flow Coverage This variation of path coverage considers only the sub-paths from variable assignments to subsequent references of the variables. Object Code Branch Coverage This measure reports whether each machine language conditional branch instruction both took the branch and fell through. Loop Coverage This measure reports whether you executed each loop body zero times, exactly once, and more than once (consecutively). For do-while loops, loop coverage reports whether you executed the body exactly once, and more than once. The valuable aspect of this measure is determining whether while-loops and for-loops execute more than once, information not reported by others measure. Race Coverage This measure reports whether multiple threads execute the same code at the same time. It helps detect failure to synchronize access to resources. It is useful for testing multi-threaded programs such as in an operating system. Relational Operator Coverage This measure reports whether boundary situations occur with relational operators (, =, , =). The hypothesis is that boundary test cases nd o-by-one errors and mistaken uses of wrong relational operators such as instead of =.

303 Weak Mutation Coverage This measure is similar to relational operator coverage but much more general [Howden1982]. It reports whether test cases occur which would expose the use of wrong operators and also wrong operands. It works by reporting coverage of conditions derived by substituting (mutating) the programs expressions with alternate operators, such as - substituted for +, and with alternate variables substituted. Table Coverage This measure indicates whether each entry in a particular array has been referenced. This is useful for programs that are controlled by a nite state machine. The rest of this text must be changed to refer to the code coverage example within ROSE/tutorial. Figure 40 shows the low level construction of a more complex AST fragment (a function declaration) and its insertion into the AST at the top of each block. Note that the code does not handle symbol table issues, yet. Building a function in global scope.

304
/ / ROSE i s a t o o l f o r b u i l d i n g p r e p r o c e s s o r s , t h i s f i l e // S p e c i f i c a l l y i t s h o w s t h e d e s i g n o f a t r a n s f o r m a t i o n // a t t h e t o p and b o t t o m e o f e a c h b a s i c b l o c k . #i n c l u d e 6 u s i n g namespace s t d ; 8 / 10 12 14 16 18 / 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 Design of t h i s code . I n p u t s : s o u r c e code ( f i l e . C) Outputs : instrumented source r o s e . h is to

CHAPTER 40. CODE COVERAGE

an e x a m p l e instrument preprocessor s o u r c e code , b u i l t w i t h ROSE . p l a c i n g source code

2 4

code

( rose

f i l e . C and

file .o)

P r o p e r t i e s of instrumented source code : 1) added d e c l a r a t i o n f o r coverage sup po r t f u n c t i o n ( e i t h e r forward f u n c t i o n d e c l a r a t i o n or a #i n c l u d e to i n c l u d e a header f i l e ) . 2) Each f u n c t i o n i n t h e s o u r c e program i s i n s t r u m e n t e d t o i n c l u d e a c a l l t o t h e coverage support function /

// G l o b a l v a r i a b l e s so t h a t t h e g l o b a l f u n c t i o n d e c l a r a t i o n can be r e u s e d t o b u i l d / / e a c h f u n c t i o n c a l l e x p r e s s i o n i n t h e AST t r a v e r s a l t o i n s t r u m e n t a l l f u n c t i o n s . S g F u n c t i o n D e c l a r a t i o n g l o b a l F u n c t i o n D e c l a r a t i o n = NULL ; SgFunctionType globalFunctionType = NULL ; S g F u n c t i o n S y m b o l f u n c t i o n S y m b o l = NULL ; / / S i m p l e ROSE t r a v e r s a l c l a s s : T h i s a l l o w s u s t o v i s i t a l l t h e // new c o d e t o i n s t r u m e n t / r e c o r d t h e i r u s e . c l a s s S i m p l e I n s t r u m e n t a t i o n : public S g S i m p l e P r o c e s s i n g { public : // r e q u i r e d v i s i t f u n c t i o n t o d e f i n e what i s t o be done void v i s i t ( SgNode a s t N o d e ) ; }; functions and add

// Code t o b u i l d f u n c t i o n d e c l a r a t i o n : T h i s d e c l a r e s Shmuel s f u n c t i o n c a l l w h i c h // w i l l be i n s e r t e d ( as a f u n c t i o n c a l l ) i n t o e a c h f u n c t i o n b o d y o f t h e i n p u t // a p p l i c a t i o n . void b u i l d F u n c t i o n D e c l a r a t i o n ( S g P r o j e c t p r o j e c t ) { / / // C r e a t e t h e f u n c t i o n D e c l a r a t i o n / / // SgGlobal globalScope = project >g e t f i l e ( 0 ) . g e t r o o t ( ) ; SgSourceFile sourceFile = isSgSourceFile ( project >g e t f i l e L i s t ( ) [ 0 ] ) ; ROSE ASSERT( s o u r c e F i l e != NULL ) ; SgGlobal globalScope = s o u r c e F i l e >g e t g l o b a l S c o p e ( ) ; ROSE ASSERT( g l o b a l S c o p e != NULL ) ; Sg File Info fil e in fo SgType f u n c t i o n r e t u r n t y p e = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; = new SgTypeVoid ( ) ;

SgName f u n c t i o n n a m e = coverageTraceFunc1 ; = new S g F u n c t i o n T y p e ( f u n c t i o n r e t u r n t y p e , f a l s e ) ; SgFunctionType f u n c t i o n t y p e S g F u n c t i o n D e c l a r a t i o n f u n c t i o n D e c l a r a t i o n = new S g F u n c t i o n D e c l a r a t i o n ( f i l e i n f o , f u n c t i o n n a m e , / / DQ ( 9 / 8 / 2 0 0 7 ) : F i x u p t h e d e f i n i n g a n d n o n d e f i n i n g d e c l a r a t i o n s ROSE ASSERT( f u n c t i o n D e c l a r a t i o n > g e t d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; functionDeclaration >s e t d e f i n i n g D e c l a r a t i o n ( f u n c t i o n D e c l a r a t i o n ) ; >g e t d e f i n i n g D e c l a r a t i o n ( ) != NULL ) ; ROSE ASSERT( f u n c t i o n D e c l a r a t i o n ROSE ASSERT( f u n c t i o n D e c l a r a t i o n > g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) != f u n c t i o n D e c l a r a t i o n ) ; / / DQ ( 9 / 8 / 2 0 0 7 ) : We h a v e n o t b u i l d a n o n d e f i n i n g d e c l a r a t i o n , s o t h i s s h o u l d b e ROSE ASSERT( f u n c t i o n D e c l a r a t i o n > g e t f i r s t N o n d e f i n i n g D e c l a r a t i o n ( ) == NULL ) ; NULL .

function type );

/ / DQ ( 9 / 8 / 2 0 0 7 ) : N e e d t o a d d f u n c t i o n s y m b o l t o g l o b a l s c o p e ! p r i n t f ( F i x i n g up t h e s y m b o l t a b l e i n s c o p e = %p = %s f o r f u n c t i o n = %p = %s \ n , g l o b a l S c o p e , g l o b a l S c o p e >c l a s s n a m e ( ) . c s t f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( f u n c t i o n D e c l a r a t i o n ) ; globalScope >i n s e r t s y m b o l ( f u n c t i o n D e c l a r a t i o n >g e t n a m e ( ) , f u n c t i o n S y m b o l ) ; >l o o k u p f u n c t i o n s y m b o l ( f u n c t i o n D e c l a r a t i o n >g e t n a m e ( ) ) != NULL ) ; ROSE ASSERT( g l o b a l S c o p e // // // Create the InitializedName for a parameter within the parameter l i s t SgName v a r 1 n a m e = t e x t S t r i n g ;

Figure 40.1: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 1).

305
SgTypeChar v a r 1 t y p e SgPointerType p o i n t e r t y p e SgInitializer var1 initializer SgInitializedName var1 init name var1 init name >s e t f i l e i n f o ( S g // 8 10 12 14 16 18 20 // 22 24 // 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 // 72 74 76 // 78 80 // c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t n e w s t m t = new S g E x p r S t a t e m e n t ( f i l e i n f o , f u n c t i o n C a l l E x p ) ; expr root >s e t p a r e n t ( n e w s t m t ) ; // // c r e a t e an e x p r e s s i o n t y p e SgTypeVoid e x p r t y p e = new SgTypeVoid ( ) ; c r e a t e an e x p r e s s i o n r o o t SgExpressionRoot expr root void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode a s t N o d e ) { S g F u n c t i o n D e c l a r a t i o n f u n c t i o n D e c l a r a t i o n = i s S g F u n c t i o n D e c l a r a t i o n ( astNode ) ; SgFunctionDefinition functionDefinition = f u n c t i o n D e c l a r a t i o n != NULL ? f u n c t i o n D e c l a r a t i o n >g e t d e f i n i t i o n ( ) i f ( f u n c t i o n D e c l a r a t i o n != NULL && f u n c t i o n D e f i n i t i o n != NULL) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s o f t h e s e f u n c t i o n s l i n k t i m e s t r i n g functionName = f u n c t i o n D e c l a r a t i o n >g e t n a m e ( ) . s t r ( ) ; s t r i n g fileName = functionDeclaration > g e t f i l e i n f o ()> g e t f i l e n a m e ( ) ; // // B u i l d a s o u r c e f i l e l o c a t i o n o b j e c t ( f o r c o n s t r u c t i o n o f e a c h IR n o d e ) N o t e t h a t we s h o u l d n o t b e s h a r i n g t h e s a m e S g F i l e I n f o o b j e c t i n m u l t i p l e IR n o d e s . Sg File Info f i l e i n f o = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; SgFunctionSymbol functionSymbol = new S g F u n c t i o n S y m b o l ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ; S g F u n c t i o n R e f E x p f u n c t i o n R e f E x p r e s s i o n = new S g F u n c t i o n R e f E x p ( f i l e i n f o , f u n c t i o n S y m b o l , g l o b a l F u n c t i o n T y p e ) ; SgExprListExp e x p r e s s i o n L i s t = new S g E x p r L i s t E x p ( f i l e i n f o ) ; s t r i n g c on v e r ag e F u n c ti on I n pu t = functionName + s t r i n g ( i n ) + fileName ; S g S t r i n g V a l f u n c t i o n N a m e S t r i n g V a l u e = new S g S t r i n g V a l ( f i l e i n f o , ( char ) c o n v e r a g e F u n c t i o n I n p u t . c s t r ( ) ) ; expressionList >a p p e n d e x p r e s s i o n ( f u n c t i o n N a m e S t r i n g V a l u e ) ; SgFunctionCallExp functionCallExp = new S g F u n c t i o n C a l l E x p ( f i l e i n f o , f u n c t i o n R e f E x p r e s s i o n , e x p r e s s i o n L i s t , g l o b a l F u n c t i o n T y p e ) ; } #i f 0 / / DQ ( 1 2 / 1 / 2 0 0 5 ) : T h i s v e r s i o n o f // i n s t u m e n t a t i o n a t e a c h f u n c t i o n / / A t IBM w e m o d i f i e d t h i s t o b e a // // // f u n c t i o n S y m b o l = new S g F u n c t i o n S y m b o l ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ; A l l any m o d i f i c a t i o n s t o b e f i x e d up ( p a r e n t s e t c ) A s t P o s t P r o c e s s i n g ( p r o j e c t ) ; // T h i s i s n o t a l l o w e d and s h o u l d be f i x e d ! AstPostProcessing ( globalScope ) ; // globalFunctionType = fu n ct ion ty p e ; globalFunctionDeclaration = functionDeclaration ; Add f u n c t i o n d e c l a r a t i o n t o g l o b a l s c o p e ! globalScope >p r e p e n d d e c l a r a t i o n ( g l o b a l F u n c t i o n D e c l a r a t i o n ) ; M a r k f u n c t i o n a s e x t e r n C functionDeclaration >g e t d e c l a r a t i o n M o d i f i e r ( ) . g e t s t o r a g e M o d i f i e r ( ) . s e t E x t e r n ( ) ; // T h i s mechanism c o u l d be i m p r o v e d ! functionDeclaration > s e t l i n k a g e ( C ) ; I f i t i s not a forward d e c l a r a t i o n then the unparser w i l l functionDeclaration >s e t F o r w a r d ( ) ; ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n >i s F o r w a r d ( ) == t r u e ) ; skip the ; at the end ( need to fix this better ) // // Set t h e scope e x p l i c i t l y ( s i n c e i t c o u l d be d i f f e r e n t from t h e p a r e n t ?) T h i s c a n t b e d o n e b y t h e A s t P o s t P r o c e s s i n g ( u n l e s s we r e l a x s o m e c o n s t r a i n t s ) functionDeclaration >s e t s c o p e ( g l o b a l S c o p e ) ; // > g e t p a r a m e t e r L i s t ( ) != NULL ) ; ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n functionDeclaration > g e t p a r a m e t e r L i s t ()> a p p e n d a r g ( v a r 1 i n i t n a m e ) ; S e t t h e p a r e n t n o d e i n t h e AST ( t h i s c o u l d b e functionDeclaration >s e t p a r e n t ( g l o b a l S c o p e ) ; done by the AstPostProcessing = new SgTypeChar ( ) ; = new S g P o i n t e r T y p e ( v a r 1 t y p e ) ; = NULL ; = new S g I n i t i a l i z e d N a m e ( v a r 1 n a m e , p o i n t e r t y p e , v a r 1 i n i t i a l i z e r , NULL ) ; File Info : : generateDefaultFileInfoForTransformationNode ( ) ) ;

2 4 6

I n s e r t argument in f u n c t i o n parameter l i s t ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n != NULL ) ; ROSE ASSERT ( f u n c t i o n D e c l a r a t i o n >g e t p a r a m e t e r L i s t ( )

!= NULL ) ;

the v i s i t function handles the special case of ( a f u n c t i o n c a l l at the top of each f u n c t i o n ) . version which instrumented every bloc k .

: NULL ;

= new

SgExpressionRoot ( f i l e

info , functionCallExp , expr type );

Figure 40.2: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 2).

306
// // new stmt >s e t e x p r e s s i o n r o o t ( e x p r r o o t ) ; functionCallExp >s e t p a r e n t ( n e w s t m t >g e t functionCallExp >s e t p a r e n t ( n e w s t m t ) ; //

CHAPTER 40. CODE COVERAGE

2 4 6 8 #i f 10 12 14 16 18 20 22 0

expression

root ());

i n s e r t a statement i n t o the f u n c t i o n body functionDefinition >g e t b o d y ()> p r e p e n d s t a t e m e n t ( n e w s t m t ) ;

// // // //

T h i s s h o w s t h e a l t e r n a t i v e u s e o f t h e ROSE However , t h e p e r f o r m a n c e i s n o t as good as b u i l d i n g t h e IR n o d e s ) . string string string string

R e w r i t e Mechanism t o do t h e same t h i n g ! t h e v e r s i o n w r i t t e n a b o v e ( more d i r e c t l y

c o d e A t T o p O f B l o c k = v o i d p r i n t f ( c h a r ) ; p r i n t f ( \ FUNCTION NAME i n FILE NAME \\ n \ ) ; ; c od e AtT op OfB loc k = c o v e r a g e T r a c e F u n c 1 ( \ FUNCTION NAME i n FILE NAME \ ) ; ; functionTarget fileTarget = FUNCTION NAME ; = FILE NAME ;

cod e AtT op OfB loc k . r e p l a c e ( co de AtT o pO fBlo ck . f i n d ( f u n c t i o n T a r g e t ) , f u n c t i o n T a r g e t . s i z e ( ) , f u n c t i o n N a m e ) ; c ode At T opO fBl oc k . r e p l a c e ( c od eA tT op OfBl oc k . f i n d ( f i l e T a r g e t ) , f i l e T a r g e t . s i z e ( ) , f i l e N a m e ) ; // printf printf ( c o d e A t T o p O f B l o c k = %s \ n , c o d e A t T o p O f B l o c k . c s t r ( ) ) ; ( %s i n %s \ n , f u n c t i o n N a m e . c s t r ( ) , f i l e N a m e . c s t r ( ) ) ;

24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 // 68 70 72 74 // 76 78 } } // // // // // // // #e n d i f } } #e n d i f // //

I n s e r t new c o d e i n t o t h e s c o p e r e p r e s e n t e d b y t h e s t a t e m e n t ( a p p l i e s t o S g S c o p e S t a t e m e n t s ) MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ; SgBasicBlock functionBody = f u n c t i o n D e f i n i t i o n >g e t b o d y ( ) ; ROSE ASSERT ( f u n c t i o n B o d y != NULL ) ; I n s e r t t h e new c o d e a t t h e t o p o f t h e s c o p e r e p r e s e n t e d b y b l o c k M i d d l e L e v e l R e w r i t e : : i n s e r t ( f u n c t i o n B o d y , codeAtTopOfBlock , s c o p e , M i d L e v e l C o l l e c t i o n T y p e d e f s : : T o p O f C u r r e n t S c o p e ) ;

void S i m p l e I n s t r u m e n t a t i o n : : v i s i t ( SgNode a s t N o d e ) { S g B a s i c B l o c k b l o c k = NULL ; b l o c k = i s S g B a s i c B l o c k ( astNode ) ; i f ( b l o c k != NULL) { // I t i s up t o t h e u s e r t o l i n k t h e i m p l e m e n t a t i o n s S g F i l e I n f o f i l e I n f o = block >g e t f i l e i n f o ( ) ; s t r i n g fileName = f i l e I n f o >g e t f i l e n a m e ( ) ; i n t lineNum = f i l e I n f o >g e t l i n e ( ) ;

of

these

functions

link

time

/ / B u i l d a s o u r c e f i l e l o c a t i o n o b j e c t ( f o r c o n s t r u c t i o n o f e a c h IR n o d e ) / / N o t e t h a t we s h o u l d n o t b e s h a r i n g t h e s a m e S g F i l e I n f o o b j e c t i n m u l t i p l e IR n o d e s . Sg File Info newCallfileInfo = Sg File Info : : generateDefaultFileInfoForTransformationNode ( ) ; ROSE ASSERT ( f u n c t i o n S y m b o l != NULL ) ; S g F u n c t i o n R e f E x p f u n c t i o n R e f E x p r e s s i o n = new S g F u n c t i o n R e f E x p ( n e w C a l l f i l e I n f o , f u n c t i o n S y m b o l , g l o b a l F u n c t i o n T y p e ) ; SgExprListExp e x p r e s s i o n L i s t = new S g E x p r L i s t E x p ( n e w C a l l f i l e I n f o ) ;

s t r i n g c o d e L o c a t i o n = f i l e N a m e + + S t r i n g U t i l i t y : : n u m b e r T o S t r i n g ( lineNum ) ; S g S t r i n g V a l f u n c t i o n N a m e S t r i n g V a l u e = new S g S t r i n g V a l ( n e w C a l l f i l e I n f o , ( char ) c o d e L o c a t i o n . c s t r ( ) ) ; expressionList >a p p e n d e x p r e s s i o n ( f u n c t i o n N a m e S t r i n g V a l u e ) ; SgFunctionCallExp functionCallExp = new S g F u n c t i o n C a l l E x p ( n e w C a l l f i l e I n f o , f u n c t i o n R e f E x p r e s s i o n , e x p r e s s i o n L i s t , g l o b a l F u c r e a t e an e x p r e s s i o n t y p e S g T y p e V o i d e x p r t y p e = new S g T y p e V o i d ( ) ; c r e a t e an e x p r e s s i o n r o o t S g E x p r e s s i o n R o o t e x p r r o o t = new S g E x p r e s s i o n R o o t ( n e w C a l l f i l e I n f o , f u n c t i o n C a l l E x p , e x p r t y p e ) ; c r e a t e an e x p r e s s i o n s t a t e m e n t S g E x p r S t a t e m e n t n e w s t m t = new S g E x p r S t a t e m e n t ( n e w C a l l f i l e I n f o , f u n c t i o n C a l l E x p ) ; expr root >s e t p a r e n t ( n e w s t m t ) ; new stmt >s e t e x p r e s s i o n ( e x p r r o o t ) ; functionCallExp >s e t p a r e n t ( n e w s t m t >g e t e x p r e s s i o n ( ) ) ; functionCallExp >s e t p a r e n t ( n e w s t m t ) ; insert a statement into the function block >p r e p e n d s t a t e m e n t ( n e w s t m t ) ; body

Figure 40.3: Example source code shows instrumentation to call a test function from the top of each function body in the application (part 3).

307

2 4 6 8 10 12 14 16 18 20

void f o o ( ) { // Should i f ( true ) int x } else { int x } }

d e t e c t t h a t f o o IS c a l l e d { = 3;

= 4;

void f o o b a r ( ) { i n t y =4; switch ( y ) { case 1 : // h e l l o w o r l d break ; case 2 : case 3 :

22 24 26 28 30 32 34 36 i n t main ( ) { i f ( true ) { } foo ( ) ; return 0 ; } } // Should d e t e c t t h a t f o o b a r i s NOT c a l l e d } default : { // }

Figure 40.4: Example source code used as input to translator adding new function.

308

CHAPTER 40. CODE COVERAGE

extern C void c o v e r a g e T r a c e F u n c 1 ( char t e x t S t r i n g ) ; 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

void f o o ( ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i // Should d e t e c t t h a t f o o IS c a l l e d i f ( true ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o int x = 3 ; } else { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o int x = 4 ; } }

void f o o b a r ( ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i int y = 4 ; switch ( y ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o case 1 : { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u // h e l l o w o r l d break ; } default : { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / // } } } // Should d e t e c t t h a t f o o b a r i s NOT c a l l e d }

i n t main ( ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i i f ( true ) { c o v e r a g e T r a c e F u n c 1 ( / e x p o r t /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o } foo ( ) ; return 0 ; }

Figure 40.5: Output of input to translator adding new function.

Chapter 41

Bug Seeding
Bug seeding is a technique used to construct example codes from existing codes which can be used to evaluate tools for the nding bugs in randomly selected existing applications. The idea is to seed an existing application with a known number of known bugs and evaluate the bug nding or security tool based on the percentage of the number of bugs found by the tool. If the bug nding tool can identify all known bugs then there can be some condence that the tool detects all bugs of that type used to seed the application. This example tutorial code is a demonstration of a more complete technique being developed in collaboration with NIST to evaluate tools for nding security aws in applications. It will in the future be a basis for testing of tools built using ROSE, specically Compass, but the techniques are not in any way specic to ROSE or Compass.

41.1

Input For Examples Showing Bug Seeding

Figure ?? shows the example input used for demonstration of bug seeding as a transformation.

2 4 6

void f o o b a r ( ) { // S t a t i c a r r a y d e c l a r a t i o n float array [ 1 0 ] ; array [ 0 ] = 0;

8 10 12 } f o r ( i n t i =0; i < 5 ; { array [ i ] = 0; } i ++)

Figure 41.1: Example source code used as input to program in codes used in this chapter.

309

310

CHAPTER 41. BUG SEEDING

41.2

Generating the code representing the seeded bug

Figure 41.2 shows a code that traverses each IR node and for and modies array reference index expressions to be out of bounds. The input code is shown in gure 41.1, the output of this code is shown in gure 41.3.

41.2. GENERATING THE CODE REPRESENTING THE SEEDED BUG

// // // This example demonstrates the of bug ( b u f f e r o v e r f l o w ) i n t o t e s t bug f i n d i n g t o o l s . seeding of a any e x i s t i n g s p e c i f i c type a p p l i c a t i o n to

311

2 4 6 8

#i n c l u d e r o s e . h u s i n g namespace S a g e B u i l d e r ; u s i n g namespace S a g e I n t e r f a c e ; namespace S e e d B u g s A r r a y I n d e x i n g {

10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 // 42 44 46 if 48 50 52 54 56 58 60 62 64 66 // 68 70 // 72 74 // 76 78 80 } 82 return 84 } 86 88 90 92 94 96 98 100 // 102 } Output return t h e new backend code seeded ( project ); with a specific form of bug . // SeedBugsArrayIndexing : : BugSeeding t r e e T r a v e r s a l ; SeedBugsArrayIndexing : : I n h e r i t e d A t t r i b u t e i n h e r i t e d A t t r i b u t e ; treeTraversal . traverseInputFiles Running i n t e r n a l t e s t s ( o p t i o n a l ) AstTests : : runAllTests ( p r o j e c t ) ; ( project , inheritedAttribute ) ; int main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , ROSE ASSERT ( p r o j e c t != NULL ) ; } inheritedAttribute ; // } B u i l d a new e x p r e s s i o n : a r r a y [ n ] > a r r a y [ n+ a r r a y S i z e C o p y ] , w h e r e t h e a r r a y S i z e C o p y S g E x p r e s s i o n n e w I n d e x E x p r e s s i o n = buildAddOp ( i n d e x E x p r e s s i o n , a r r a y S i z e C o p y ) ; S u b s t i t u t e t h e new e x p r e s s i o n f o r t h e o l d e x p r e s s i o n arrayReference >s e t r h s o p e r a n d ( n e w I n d e x E x p r e s s i o n ) ; is a size of array This i s the e x i s t i n g index e x p r e s s i o n SgExpression indexExpression = arrayReference >g e t r h s o p e r a n d ( ) ; ROSE ASSERT( i n d e x E x p r e s s i o n != NULL ) ; ( i s L o o p == t r u e | | a p p l y E v e r y W h e r e == t r u e ) { / / The i n h e r i t e d a t t r i b u t e i s t r u e i f f we a r e i n s i d e a l o o p a n d t h i s SgPntrArrRefExp a r r a y R e f e r e n c e = isSgPntrArrRefExp ( astNode ) ; i f ( a r r a y R e f e r e n c e != NULL) { // Mark a s a v u l n e r a b i l i t y i n h e r i t e d A t t r i b u t e . i s V u l n e r a b i l i t y = true ; // // Mark f u t u r e n o e s i n t h i s s u b t r e e a s i n h e r i t e d A t t r i b u t e . isLoop = isLoop ; To t e s t t h i s o n s i m p l e c o d e s , bool a p p l y E v e r y W h e r e = t r u e ; being part of a loop }; InheritedAttribute BugSeeding : : e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode a s t N o d e , InheritedAttribute inheritedAttribute ) { / / U s e t h i s i f we o n l y w a n t t o s e e d b u g s i n l o o p s bool i s L o o p = i n h e r i t e d A t t r i b u t e . i s L o o p || ( i s S g F o r S t a t e m e n t ( a s t N o d e ) != NULL) | | ( i s S g W h i l e S t m t ( a s t N o d e ) != NULL) || ( i s S g D o W h i l e S t m t ( a s t N o d e ) != NULL ) ; / / Add F o r t r a n s u p p o r t i s L o o p = i s L o o p | | ( i s S g F o r t r a n D o ( a s t N o d e ) != NULL ) ; }; class { BugSeeding : public SgTopDownProcessing < I n h e r i t e d A t t r i b u t e > class { InheritedAttribute public : bool i s L o o p ; bool i s V u l n e r a b i l i t y ; I n h e r i t e d A t t r i b u t e ( ) : i s L o o p ( f a l s e ) , i s V u l n e r a b i l i t y ( f a l s e ) {} I n h e r i t e d A t t r i b u t e ( c o n s t I n h e r i t e d A t t r i b u t e & X) : i s L o o p (X . i s L o o p ) , i s V u l n e r a b i l i t y (X . i s V u l n e r a b i l i t y )

{}

public : InheritedAttribute evaluateInheritedAttribute SgNode a s t N o d e , InheritedAttribute inheritedAttribute );

optionally

allow

it

to

be

applied

everywhere

is

SgPntrArrRefExp .

Now c h a n g e t h e a r r a y i n d e x ( t o s e e d t h e b u f f e r o v e r f l o w b u g ) SgVarRefExp a r r a y V a r R e f = i s S g V a r R e f E x p ( a r r a y R e f e r e n c e >g e t l h s o p e r a n d ( ) ) ; ROSE ASSERT( a r r a y V a r R e f != NULL ) ; >g e t s y m b o l ( ) != NULL ) ; ROSE ASSERT( a r r a y V a r R e f S g I n i t i a l i z e d N a m e arrayName = i s S g I n i t i a l i z e d N a m e ( a r r a y V a r R e f >g e t s y m b o l ()> g e t d e c l a r a t i o n ( ) ) ; ROSE ASSERT( arrayName != NULL ) ; SgArrayType a r r a y T y p e = i s S g A r r a y T y p e ( arrayName >g e t t y p e ( ) ) ; ROSE ASSERT( a r r a y T y p e != NULL ) ; S g E x p r e s s i o n a r r a y S i z e = a r r a y T y p e >g e t i n d e x ( ) ; SgTreeCopy c o p y H e l p ; Make a c o p y o f t h e e x p r e s s i o n u s e d t o h o l d t h e a r r a y s i z e i n t h e a r r a y d e c l a r a t i o n . SgExpression arraySizeCopy = isSgExpression ( arraySize >c o p y ( c o p y H e l p ) ) ; ROSE ASSERT( a r r a y S i z e C o p y != NULL ) ;

argv ) ;

Figure 41.2: Example source code showing how to seed bugs.

312

CHAPTER 41. BUG SEEDING

2 4 6 8 10

void f o o b a r ( ) { // S t a t i c a r r a y d e c l a r a t i o n f l o a t a r r a y [ 1 0UL ] ; a r r a y [ 0 + 10UL ] = 0 ; f o r ( i n t i = 0 ; i < 5 ; i ++) { a r r a y [ i + 10UL ] = 0 ; } }

Figure 41.3: Output of input code using seedBugsExample arrayIndexing.C

Part VI

Binary Support

Tutorials of using ROSE to handle binary executable les.

313

Chapter 42

Instruction Semantics
The Instruction Semantics layer in ROSE can be used to evaluate instructions and is controlled by a policy that denes the details of what evaluate means. For instance, given the following xor instruction, the X86InstructionSemantics class species that the value of the eax and edx registers are read, those two 32-bit values are xord together, and the 32-bit result is then written to the eax register. The policy denes what a 32-bit value is (it could be an integer, some representation of a constant, etc), how it is read and written to the registers, and how to compute an xor. xor eax, edx ROSE has a collection instruction semantic classes, one for each architecture. It also has a small collection of policies. This chapter briey describes a policy that tracks constant values.

42.1

The FindConstantsPolicy Class

The FindConstantsPolicy is used to track constant values across an instruction trace. The basic idea is that ROSE executes the instructions one at a time in the instruction semantics layer, identies constants, performs operations on those constants, and assigns constants to registers and memory locations. Each constant also maintains information about which instructions led to that particular constants existence. A constant is an abstract datum that has a known integer value, or a name corresponding to some unknown value, or a name and a known integer oset. Names take the form of the letter v (for value) followed by a unique integer. Known values are represented as signed hexadecimal values in the output. The ndConstants.C program in the tests/roseTests/binaryTests directory (which is described herein) takes each function and processes the instructions of that function in address order. It makes no attempt to follow branches or any other kind of control ow, but serves to demonstrate a simple way to track constants. 1 2 #define \_\_STDC_FORMAT_MACROS #include "rose.h" 315

316 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 #include "findConstants.h" #include <inttypes.h>

CHAPTER 42. INSTRUCTION SEMANTICS

/* Returns the function name if known, or the address as a string otherwise. */ static std::string name\_or\_addr(const SgAsmFunction *f) { if (f->get\_name()!="") return f->get\_name(); char buf[128]; SgAsmBlock *first\_bb = isSgAsmBlock(f->get\_statementList().front()); sprintf(buf, "0x%"PRIx64, first\_bb->get\_id()); return buf; }

class AnalyzeFunctions : public SgSimpleProcessing { public: AnalyzeFunctions(SgProject *project) { traverse(project, postorder); } void visit(SgNode *node) { SgAsmFunction *func = isSgAsmFunction(node); if (func) { std::cout <<"==============================================\n" <<"Constant propagation in function \"" <<name\_or\_addr(func) <<"\"\n" <<"==============================================\n"; FindConstantsPolicy policy; X86InstructionSemantics<FindConstantsPolicy, XVariablePtr> t(policy); std::vector<SgNode*> instructions = NodeQuery::querySubTree(func, V\_SgAsmx86Instruct for (size\_t i=0; i<instructions.size(); i++) { SgAsmx86Instruction *insn = isSgAsmx86Instruction(instructions[i]); ROSE\_ASSERT(insn); t.processInstruction(insn); RegisterSet rset = policy.currentRset; std::cout <<unparseInstructionWithAddress(insn) <<"\n" <<rset; } } } }; int main(int argc, char *argv[]) { AnalyzeFunctions(frontend(argc, argv)); }

42.2. SAMPLE OUTPUT

317

Lines 30 through 40 are the main meat of the example. For each function, we construct a fresh policy. Since the policy holds the values of registers and memory, this resets them all to an initial state having unknown values. The instruction semantics engine depends on the policy, so we also create a new one for each function. Then we loop over the instructions of the function in order of their addresses at lines 33 through 40. Each instruction is processed in turn by the X86InstructionSemantics object, adjusting the state of the associated policy. Finally, the assembly language instruction is output followed by the values contained in the registers as a result of processing the instruction.

42.2

Sample Output

Heres some abbreviated output from running the ndConstants test on a binary executable: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ============================================== Constant propagation in function "_init" ============================================== 0x80482c8:push ebp ax = v62 cx = v63 dx = v64 bx = v65 sp = v66-0x4 [from 0x80482c8:push ebp] bp = v67 si = v68 di = v69 es = v70 cs = v71 ss = v72 ds = v73 fs = v74 gs = v75 cf = v76 ?1 = v77 pf = v78 ?3 = v79 af = v80 ?5 = v81 zf = v82 sf = v83 tf = v84 if = v85 df = v86 of = v87 iopl0 = v88

318 32 33 34 35 36 37

CHAPTER 42. INSTRUCTION SEMANTICS iopl1 = v89 nt = v90 ?15 = v91 memory = { size=4; addr=v66-0x4 [from 0x80482c8:push }

ebp]; value=v67

Line 4 indicates that the instruction push ebp is located at address 0x80482c8 and the following lines show the contents of registers and known memory addresses following execution of the push. One can readily see that each register has a unique constant of unknown value by virtue of each constant having a unique name. The stack pointer register (sp) has the constant v66-0x4 obtained from this very instruction. If we had printed the registers prior to executing the push we would have seen that the original sp constant was v66. Therefore, this push instruction reduced the value of sp by four. Line 36 indicates that the four bytes beginning at memory address v66-0x4 (which happens to be the constant stored in the stack pointer register at line 9) contain the value v67 (which is the constant stored in the bp register at line 10). Therefore, it can be determined that push ebp decrements the stack pointer by four bytes, then copies a 32-bit value from the bp register to the memory pointed to by the new stack pointer. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0x80482c9:mov ebp, esp ax = v62 cx = v63 dx = v64 bx = v65 sp = v66-0x4 [from 0x80482c8:push bp = v66-0x4 [from 0x80482c8:push si = v68 di = v69 es = v70 cs = v71 ss = v72 ds = v73 fs = v74 gs = v75 cf = v76 ?1 = v77 pf = v78 ?3 = v79 af = v80 ?5 = v81 zf = v82 sf = v83 tf = v84 if = v85

ebp] ebp]

42.2. SAMPLE OUTPUT 26 27 28 29 30 31 32 33 34 df = v86 of = v87 iopl0 = v88 iopl1 = v89 nt = v90 ?15 = v91 memory = { size=4; addr=v66-0x4 [from 0x80482c8:push }

319

ebp]; value=v67

The output after the mov ebp, esp instruction at address 0x80482c9 subsequent to the push ebp that we just saw, shows that the new stack pointer has been copied into the bp register and that nothing else has changed. A more interesting instruction follows... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 0x80482cb:sub esp, 0x8 ax = v62 cx = v63 dx = v64 bx = v65 sp = v66-0xc [from 0x80482cb:sub esp, 0x8] bp = v66-0x4 [from 0x80482c8:push ebp] si = v68 di = v69 es = v70 cs = v71 ss = v72 ds = v73 fs = v74 gs = v75 cf = -v193-0x1 [from 0x80482cb:sub esp, 0x8] ?1 = v77 pf = -v187-0x1 [from 0x80482cb:sub esp, 0x8] ?3 = v79 af = -v191-0x1 [from 0x80482cb:sub esp, 0x8] ?5 = v81 zf = v190 [from 0x80482cb:sub esp, 0x8] sf = v189 [from 0x80482cb:sub esp, 0x8] tf = v84 if = v85 df = v86 of = v197 [from 0x80482cb:sub esp, 0x8] iopl0 = v88 iopl1 = v89 nt = v90 ?15 = v91 memory = {

320 33 34

CHAPTER 42. INSTRUCTION SEMANTICS size=4; addr=v66-0x4 [from 0x80482c8:push } ebp]; value=v67

The sub esp, 0x8 subtracts eight from the value of the stack pointer register and then stores the result in the stack pointer register. This can be seen by the fact that the constant stored in the sp register has changed from v66-0x4 to v66-0xc. One can also see that various ags have been modied, although we dont know the values of any of them. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

0x80482ce:call 0x8048364 ax = v62 cx = v63 dx = v64 bx = v65 sp = v66-0x10 [from 0x80482ce:call 0x8048364] bp = v66-0x4 [from 0x80482c8:push ebp] si = v68 di = v69 es = v70 cs = v71 ss = v72 ds = v73 fs = v74 gs = v75 cf = -v193-0x1 [from 0x80482cb:sub esp, 0x8] ?1 = v77 pf = -v187-0x1 [from 0x80482cb:sub esp, 0x8] ?3 = v79 af = -v191-0x1 [from 0x80482cb:sub esp, 0x8] ?5 = v81 zf = v190 [from 0x80482cb:sub esp, 0x8] sf = v189 [from 0x80482cb:sub esp, 0x8] tf = v84 if = v85 df = v86 of = v197 [from 0x80482cb:sub esp, 0x8] iopl0 = v88 iopl1 = v89 nt = v90 ?15 = v91 memory = { size=4; addr=v66-0x10 [from 0x80482ce:call 0x8048364]; value=0x80482d3 [from 0x80482ce: size=4; addr=v66-0x4 [from 0x80482c8:push ebp]; value=v67 }

Now we get to our rst branch-type instruction, a call to a particular address. Instruction semantics describe what the call instruction does to the registers and memory, but does not actually execute a call or process the called instructions. That means that a ret was not

42.3. BUILDING ON INSTRUCTION SEMANTICS

321

processed and thus we should see the return address sitting on the stack. In fact, we do: the stack pointer has been decremented by another four bytes and the memory address to which the stack pointer points contains the address of the instruction immediately after the call.

42.3

Building on Instruction Semantics

The X86InstructionSemantics class and the policy classes can be extended to handle special cases. For instance, the X86InstructionSemantics class processes the rep stosd instruction in such a way that only one iteration of the stosd is considered. Sometimes its more useful to process the entire repeated sequence in one step rather than iterating through the loop. Subclassing X86InstructionSemantics to override individual instructions or classes of instructions is simple. The subclass should redene the translate method to do whatever is necessary for certain instructions while delegating to the superclass for all remaining instructions. For example: /* Augments super::translate() to override rep\_stos instructions */ virtual void translate(SgAsmx86Instruction *insn) { switch (insn->get\_kind()) { case x86\_rep\_stosb: updateIP(insn); rep\_stos\_semantics<1>(insn); break; case x86\_rep\_stosw: updateIP(insn); rep\_stos\_semantics<2>(insn); break; case x86\_rep\_stosd: updateIP(insn); rep\_stos\_semantics<4>(insn); break; default: super::translate(insn); break; } } Its also possible to subclass the policies. For instance, if you need to do something special for binary AND operations on the stack pointer you could override the and method in the policy.

322

CHAPTER 42. INSTRUCTION SEMANTICS

Chapter 43

Binary Analysis
This chapter discusses the capabilities of ROSE to read, analyze and transform (transformations to the binary le format) binary executables. Binary support in ROSE is currently based on a custom build ROSE Disassembler (for ARM, x86, and PowerPC). The following code reads in a binary and creates a binary ROSE AST:
SgProject* project = frontend(argc,argv);

Similarily, one can unparse the AST to assembly using a call to the backend, cf. Figure ??. The best documentation for ROSEs binary analysis capabilities is found in the doxygengenerated API reference manual which can be found on the ROSE web site. Please refer to the following documented entities: class AsmUnparser, class Assembler, class BinaryLoader, namespace BinaryAnalysis, class Disassembler, class MemoryMap, class Partitioner, class SymbolicSemantics, class PartialSymbolicSemantics, class RegisterDictionary, and class X86InstructionSemantics.

43.1

The ControlFlowGraph

Based on a control ow traversal of the binary AST, a separate control ow graph is created that can be used for further analyses. TODO: Describe recent work on binary CFG.

43.2

DataFlow Analysis

Based on the control ow many forms of dataow analysis may be performed. Dataow analyses available are:

43.2.1

Def-Use Analysis

Denition-Usage is one way to compute dataow information about a binary program. 323

324

CHAPTER 43. BINARY ANALYSIS

8048414_f:__libc_csu_fini type = removed

8048460_f:__do_global_ctors_aux type = removed

8048364_f:main type = removed

80482c0_f:_start type = removed

0x8048414:push ebp visited: 1 type = removed push

0x8048460:push ebp visited: 1 type = removed push

0x8048364:lea ecx, [esp + 0x04] visited: 1 type = removed lea

0x80482c0:xor ebp, ebp visited: 1 type = removed xor

Use of rbp at 80482c0

Def of rsp at 8048414 Use of rsp at 8048415

Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048460 Def of rbp at 8048288

Use of rsp at 8048461

Def of rcx at 8048364

Def of rbp at 80482c0

0x8048415:mov ebp, esp visited: 1 type = removed mov

0x8048461:mov ebp, esp visited: 1 type = removed mov

0x8048368:and esp, 0xf0<-0x10> visited: 1 type = removed and

0x80482c2:pop esi visited: 1 type = removed pop

Def of rsp at 8048414 Def of rbp at 8048415

Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048460 Def of rbp at 8048461

Def of rcx at 8048364 Def of rsp at 8048368

Def of rsp at 80482c2 Use of rsp at 80482c3 Def of rbp at 80482c0

0x8048417:push edi visited: 1 type = removed push

0x8048463:push ebx visited: 1 type = removed push

80482e4_f:call_gmon_start type = removed

0x804836b:push DWORD PTR ds:[ecx + 0xfc<-0x04>] visited: 1 type = removed push

804836e_f: 804836e type = removed

0x80482c3:mov ecx, esp visited: 1 type = removed mov

Def of rsp at 8048417 Def of rbp at 8048415

Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048463 Def of rbp at 8048461

Def of rcx at 8048364 Def of rsp at 804836b

Def of rcx at 80482c3 Def of rsp at 80482c2 Def of rbp at 80482c0

0x8048418:push esi visited: 1 type = removed push

0x8048464:mov

ebx, 0x080494bc<.ctors> visited: 1 type = removed mov

0x80482e4:push ebp visited: 1 type = removed push

80483c0_f:__libc_csu_init type = removed

0x804836e:push ebp visited: 1 type = removed push

0x80482c5:and esp, 0xf0<-0x10> visited: 1 type = removed and

Def of rsp at 8048418 Def of rbp at 8048415

Def of rax at 8048346 Def of rax at 804834f Def of rbx at 8048464 Def of rsp at 8048463 Def of rbp at 8048461

Def of rsp at 80482e4 Def of rbp at 804827e

Use of rsp at 80482e5

Def of rcx at 8048364 Def of rsp at 804836e

Use of rsp at 804836f

Def of rcx at 80482c3 Def of rsp at 80482c5 Def of rbp at 80482c0

0x8048419:push ebx visited: 1 type = removed push

0x8048469:sub esp, 0x04 visited: 1 type = removed sub

0x80482e5:mov ebp, esp visited: 1 type = removed mov

0x80483c0:push ebp visited: 1 type = removed push

0x804836f:mov ebp, esp visited: 1 type = removed mov

0x80482c8:push eax visited: 1 type = removed push

Def of rsp at 8048419 Def of rbp at 8048415

Def of rax at 8048346 Def of rax at 804834f Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461

Def of rsp at 80482e4 Def of rbp at 80482e5

Def of rsp at 80483c0 Use of rsp at 80483c1

Def of rcx at 8048364 Def of rsp at 804836e Def of rbp at 804836f

Def of rcx at 80482c3 Def of rsp at 80482c8 Def of rbp at 80482c0

0x804841a:call 0x0804841f<<Insn>> visited: 1 type = removed call

0x804846c:mov

eax, DWORD PTR ds:[0x080494bc<.ctors>] visited: 1 type = removed mov

0x80482e7:push ebx visited: 1 type = removed push

0x80483c1:mov ebp, esp visited: 1 type = removed mov

8048340_f:frame_dummy type = removed

0x8048371:push ecx visited: 1 type = removed push

0x80482c9:push esp visited: 1 type = removed push

call

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461

Def of rsp at 80482e7 Def of rbp at 80482e5

Def of rsp at 80483c0 Def of rbp at 80483c1

Def of rcx at 8048364 Def of rsp at 8048371 Def of rbp at 804836f

Def of rcx at 80482c3 Def of rsp at 80482c9 Def of rbp at 80482c0

0x804841f:pop ebx visited: type = removed pop

0x8048471:cmp eax, 0xff<-0x01> visited: 1 type = removed cmp

0x80482e8:call

0x080482ed<<Insn>> visited: 1 type = removed call

0x80483c3:push edi visited: 1 type = removed push

0x8048340:push ebp visited: 1 type = removed push

0x8048372:sub esp, 0x14 visited: 1 type = removed sub

0x80482ca:push edx visited: 1 type = removed push

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461

call

Def of rsp at 80483c3 Def of rbp at 80483c1

Def of rsp at 8048340

Use of rsp at 8048341

Def of rcx at 8048364 Def of rsp at 8048372 Def of rbp at 804836f

Def of rcx at 80482c3 Def of rsp at 80482ca Def of rbp at 80482c0

0x8048420:add ebx, 0x0000117d visited: type = removed add

0x8048474:je

0x0804848c<<Insn>> visited: 1 type = removed je

0x80482ed:pop ebx visited: type = removed pop

0x80483c4:push esi visited: 1 type = removed push

0x8048341:mov ebp, esp visited: 1 type = removed mov

0x8048375:mov

DWORD PTR ss:[esp], 0x00000028 visited: 1 type = removed mov

0x80482cb:push 0x08048414<__libc_csu_fini> visited: 1 type = removed push

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461

Def of rsp at 80483c4 Def of rbp at 80483c1

Def of rsp at 8048340 Def of rbp at 8048341

Def of rcx at 8048364 Def of rsp at 8048372 Def of rbp at 804836f

Def of rcx at 80482c3 Def of rsp at 80482cb Def of rbp at 80482c0

0x8048426:lea

eax, [ebx + 0xffffff20<-0x000000e0>] visited: type = removed lea

0x8048476:lea esi, [esi + 0x00] visited: 1 type = removed lea

0x80482ee:add ebx, 0x000012af visited: type = removed add

0x80483c5:push ebx visited: 1 type = removed push

0x8048343:sub esp, 0x08 visited: 1 type = removed sub

0x804837c:call 0x080482a0<malloc@plt> visited: 1 type = removed call

80482a0_f:malloc@plt type = removed

0x80482d0:push 0x080483c0<__libc_csu_init> visited: 1 type = removed push

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461 Def of rsi at 8048476

Def of rsp at 80483c5 Def of rbp at 80483c1

Def of rsp at 8048343 Def of rbp at 8048341

call Def of rcx at 8048364 Def of rsp at 804837c Def of rbp at 804837c

Def of rcx at 80482c3 Def of rsp at 80482d0 Def of rbp at 80482c0

0x804842c:lea

edi, [ebx + 0xffffff20<-0x000000e0>] visited: type = removed lea

0x8048479:lea

edi, [edi + 0x00000000] visited: 1 type = removed lea

0x80482f4:push edx visited: type = removed push

0x80483c6:sub esp, 0x0c visited: 1 type = removed sub

0x8048346:mov

eax, DWORD PTR ds:[0x080494cc<.jcr>] visited: 1 type = removed mov

0x8048381:mov

DWORD PTR ss:[ebp + 0xf0<-0x10>], eax visited: type = removed mov

0x80482a0:jmp

DWORD PTR ds:[0x080495a8<.got.plt+0x0c>] visited: 1 type = removed jmp

0x80482d5:push ecx visited: 1 type = removed push

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461 Def of rsi at 8048476 Def of rdi at 8048479

call Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048288 Def of rbp at 8048288

Def of rsp at 80483c6 Def of rbp at 80483c1

Def of rax at 8048346 Def of rsp at 8048343 Def of rbp at 8048341

Use of rax at 804834b

Def of rcx at 80482c3 Def of rsp at 80482d5 Def of rbp at 80482c0

0x8048432:sub eax, edi visited: type = removed sub

0x8048480:sub ebx, 0x04 visited: 1 type = removed sub

0x80482f5:mov

eax, DWORD PTR ds:[ebx + 0xfffffffc<-0x00000004>] visited: type = removed mov

call Def of rsp at 804827e Def of rbp at 804827e

0x80483c9:call 0x080483ce<<Insn>> visited: 1 type = removed call

0x804834b:test eax, eax visited: 1 type = removed test

0x8048384:mov

DWORD PTR ss:[ebp + 0xf4<-0x0c>], 0x00000000 visited: type = removed mov

0x80482d6:push esi visited: 1 type = removed push

Def of rax at 804846c Def of rbx at 8048480 Def of rsp at 8048469 Def of rbp at 8048461 Def of rsi at 8048476 Def of rdi at 8048479 jmp_if Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048469 Def of rbp at 8048461

call

Def of rax at 8048346 Def of rsp at 8048343 Def of rbp at 8048341

Def of rcx at 80482c3 Def of rsp at 80482d6 Def of rbp at 80482c0

0x8048434:sar eax, 0x02 visited: type = removed sar

0x8048483:call eax visited: 1 type = removed call

0x80482fb:test eax, eax visited: type = removed test

0x80483ce:pop ebx visited: type = removed pop

0x804834d:je

0x08048361<<Insn>> visited: 1 type = removed je

0x804838b:mov

DWORD PTR ss:[ebp + 0xf4<-0x0c>], 0x00000000 visited: type = removed mov

0x80482d7:push 0x08048364<main> visited: 1 type = removed push

call

Def of rax at 8048346 Def of rsp at 8048343 Def of rbp at 8048341

Def of rcx at 80482c3 Def of rsp at 80482d7 Def of rbp at 80482c0

0x8048437:sub esp, 0x0c visited: type = removed sub

0x8048485:mov

eax, DWORD PTR ds:[ebx] visited: type = removed mov

0x80482fd:je jmp_if

0x08048301<<Insn>> visited: type = removed je

0x80483cf:add ebx, 0x000011ce visited: type = removed add

call

0x804834f:mov eax, 0x00000000 visited: 1 type = removed mov

0x8048392:jmp

0x080483a6<<Insn>> visited: type = removed jmp

80482b0_f:__libc_start_main@plt type = removed

0x80482dc:call 0x080482b0<__libc_start_main@plt> visited: 1 type = removed call

Def of rax at 804834f Def of rsp at 8048343 Def of rbp at 8048341

Use of rax at 8048354

jmp

call Def of rcx at 80482c3 Def of rsp at 80482dc Def of rbp at 80482dc

8048310_f:__do_global_dtors_aux type = removed

0x804843a:lea

esi, [eax + 0xff<-0x01>] visited: type = removed lea

0x8048487:cmp eax, 0xff<-0x01> visited: type = removed cmp

jmp_if

0x80482ff:call eax visited: type = removed call

0x80483d5:call 0x08048278<_init> visited: type = removed call

8048278_f:_init type = removed

0x8048354:test eax, eax visited: 1 type = removed test

0x80483a6:cmp

DWORD PTR ss:[ebp + 0xf4<-0x0c>], 0x09 visited: type = removed cmp

0x80482b0:jmp

DWORD PTR ds:[0x080495ac<.got.plt+0x10>] visited: 1 type = removed jmp

0x80482e1:hlt visited: type = removed hlt

call

call

Def of rax at 804834f Def of rsp at 8048343 Def of rbp at 8048341

0x8048310:push ebp visited: 1 type = removed push

0x804843d:jmp

0x08048444<<Insn>> visited: type = removed jmp

0x804848a:jne

0x08048480<<Insn>> visited: type = removed jne

0x8048301:pop eax visited: type = removed pop

0x80483da:lea

eax, [ebx + 0xffffff20<-0x000000e0>] visited: 1 type = removed lea

0x8048278:push ebp visited: 1 type = removed push

jmp_if Def of rax at 8048346 Def of rsp at 8048343 Def of rbp at 8048341

0x8048356:je

0x08048361<<Insn>> visited: 1 type = removed je

0x80483aa:jle

0x08048394<<Insn>> visited: type = removed jle

Def of rsp at 8048310

Use of rsp at 8048311

jmp

Def of rax at 80483da Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e

Def of rsp at 8048278

Use of rsp at 8048279

Def of rax at 804834f Def of rsp at 8048343 Def of rbp at 8048341

jmp_if

0x8048311:mov ebp, esp visited: 1 type = removed mov

0x8048444:cmp esi, 0xff<-0x01> visited: type = removed cmp

0x804848c:pop eax visited: 1 type = removed pop

0x8048302:pop ebx visited: type = removed pop

0x80483e0:lea Use of rax at 80483e6

edx, [ebx + 0xffffff20<-0x000000e0>] visited: 1 type = removed lea

0x8048279:mov ebp, esp visited: 1 type = removed mov

0x8048358:mov

DWORD PTR ss:[esp], 0x080494cc<.jcr> visited: 1 type = removed mov

0x80483ac:mov

eax, DWORD PTR ss:[ebp + 0xf0<-0x10>] visited: type = removed mov

0x8048394:mov

eax, DWORD PTR ss:[ebp + 0xf4<-0x0c>] visited: type = removed mov

Def of rsp at 8048310 Def of rbp at 8048311

0x804843f:nop visited: type = removed nop

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804848c Def of rbp at 8048461

Def of rax at 80483da Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e

Def of rsp at 8048278 Def of rbp at 8048279

jmp_if Def of rax at 804834f Def of rsp at 8048343 Def of rbp at 8048341

Def of rax at 804834f Def of rsp at 8048343 Def of rbp at 8048341

0x8048313:sub esp, 0x08 visited: 1 type = removed sub

0x8048447:jne

0x08048440<<Insn>> visited: type = removed jne

0x804848d:pop ebx visited: 1 type = removed pop

0x8048303:leave visited: type = removed leave

0x80483e6:mov

DWORD PTR ss:[ebp + 0xf0<-0x10>], eax visited: 1 type = removed mov

Use of rdx at 80483e9

0x804827b:sub esp, 0x08 visited: 1 type = removed sub

0x804835f:call eax visited: 1 type = removed call

0x80483af:add eax, 0x30 visited: type = removed add

0x8048397:shl eax, 0x02 visited: type = removed shl

Def of rsp at 8048313 Def of rbp at 8048311

jmp_if

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804848d Def of rbp at 8048461

Def of rax at 80483da Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e

Def of rsp at 804827b Def of rbp at 8048279

call

0x8048316:cmp

BYTE PTR ds:[0x080495bc<.bss>], 0x00 visited: 1 type = removed cmp

8048494_f:_fini type = removed

0x8048449:call 0x08048494<_fini> visited: type = removed call

0x8048440:call DWORD PTR ds:[edi + esi*0x04] visited: type = removed call

0x804848e:pop ebp visited: 1 type = removed pop

0x8048304:ret visited: type = removed ret

0x80483e9:sub eax, edx visited: 1 type = removed sub

0x804827e:call 0x080482e4<call_gmon_start> visited: 1 type = removed call

0x8048361:leave visited: 1 type = removed leave

0x80483b2:mov

eax, DWORD PTR ds:[eax] visited: type = removed mov

0x804839a:add

eax, DWORD PTR ss:[ebp + 0xf0<-0x10>] visited: type = removed add

Def of rsp at 8048313 Def of rbp at 8048311

call

call

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804848e Def of rbp at 8048461

Def of rax at 80483e9 Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e

ret

ret Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e

Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048361 Def of rbp at 8048361

0x804831d:je

0x0804832b<<Insn>> visited: 1 type = removed je

0x8048494:push ebp visited: 1 type = removed push

0x804844e:add esp, 0x0c visited: 1 type = removed add

0x8048443:dec esi visited: type = removed dec

0x804848f:nop visited: 1 type = removed nop

0x80483eb:xor esi, esi visited: 1 type = removed xor

0x8048283:call Use of rsi at 80483eb

0x08048340<frame_dummy> visited: type = removed call

0x8048362:ret visited: 1 type = removed ret

0x80483b4:mov

DWORD PTR ss:[ebp + 0xf8<-0x08>], eax visited: type = removed mov

0x804839d:mov

DWORD PTR ds:[eax], 0x00000005 visited: type = removed mov

jmp_if Def of rsp at 8048313 Def of rbp at 8048311

Def of rsp at 8048494

Use of rsp at 8048495

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 804844e Def of rbp at 80484ad

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804848e Def of rbp at 8048461

Def of rax at 80483e9 Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb

Use of rdx at 80483f4

ret Def of rax at 8048346 Def of rax at 804834f Def of rsp at 8048362 Def of rbp at 8048362

0x804832b:mov

eax, DWORD PTR ds:[0x080495b8<.data+8>] visited: 1 type = removed mov

0x8048495:mov ebp, esp visited: 1 type = removed mov

0x8048451:pop ebx visited: 1 type = removed pop

0x8048490:ret visited: 1 type = removed ret

0x80483ed:sar eax, 0x02 visited: 1 type = removed sar

0x8048288:call 0x08048460<__do_global_ctors_aux> visited: 1 type = removed call

0x80483a3:inc

DWORD PTR ss:[ebp + 0xf4<-0x0c>] visited: type = removed inc

0x80483b7:add esp, 0x14 visited: type = removed add

Def of rax at 804832b Def of rsp at 8048313 Def of rbp at 8048311

Def of rsp at 8048494 Def of rbp at 8048495

call

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048451 Def of rbp at 80484ad

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Use of rax at 80483f0 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb

ret Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 8048490 Def of rbp at 8048490

0x8048330:mov

edx, DWORD PTR ds:[eax] visited: 1 type = removed mov

Def of rsp at 8048313 Def of rbp at 8048311

0x8048497:push ebx visited: 1 type = removed push

0x8048452:pop esi visited: 1 type = removed pop

0x80483f0:cmp esi, eax visited: 1 type = removed cmp

0x804828d:leave visited: 1 type = removed leave

0x80483ba:pop ecx visited: type = removed pop

Use of rdx at 8048332

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

Def of rsp at 8048497 Def of rbp at 8048495

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048452 Def of rbp at 80484ad

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb

Def of rax at 804846c Def of rbx at 8048464 Def of rsp at 804828d Def of rbp at 804828d

0x8048332:test edx, edx visited: 1 type = removed test

0x8048498:call

0x0804849d<<Insn>> visited: 1 type = removed call

0x8048453:pop edi visited: 1 type = removed pop

0x80483f2:jae

0x0804840a<<Insn>> visited: 1 type = removed jae

0x804828e:ret visited: 1 type = removed ret

0x80483bb:pop ebp visited: type = removed pop

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

call

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048453 Def of rbp at 80484ad

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb

0x8048334:jne call

0x08048321<<Insn>> visited: 1 type = removed jne

0x804831f:jmp

0x0804833d<<Insn>> visited: 1 type = removed jmp

0x804849d:pop ebx visited: type = removed pop

0x8048454:leave visited: 1 type = removed leave

0x80483f4:mov edi, edx visited: 1 type = removed mov

0x80483bc:lea

esp, [ecx + 0xfc<-0x04>] visited: type = removed lea

jmp_if Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

Def of rsp at 8048313 Def of rbp at 8048311

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048454 Def of rbp at 8048454

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb Def of rdi at 80483f4

0x8048321:add eax, 0x04 visited: 1 type = removed add

0x8048336:mov

BYTE PTR ds:[0x080495bc<.bss>], 0x01 visited: 1 type = removed mov

jmp Def of rsp at 8048313 Def of rbp at 8048311

0x804849e:add ebx, 0x000010ff visited: type = removed add

0x8048455:ret visited: 1 type = removed ret

0x80483f6:mov esi, esi visited: 1 type = removed mov

Use of rsi at 80483f6

0x80483bf:ret visited: type = removed ret

Def of rax at 8048321 Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

Use of rax at 8048324

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

ret Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 80484ad Def of rbp at 80484ad

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483f6 Def of rdi at 80483f4

0x8048324:mov

DWORD PTR ds:[0x080495b8<.data+8>], eax visited: 1 type = removed mov

0x804833d:leave visited: 1 type = removed leave

0x80484a4:push eax visited: type = removed push

0x80483f8:call DWORD PTR ds:[edx + esi*0x04] visited: 1 type = removed call

Def of rax at 8048321 Def of rdx at 8048330 Def of rsp at 8048313 Def of rbp at 8048311

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 804833d Def of rbp at 804833d

call

0x8048329:call edx visited: 1 type = removed call

0x804833e:ret visited: 1 type = removed ret

0x80484a5:call 0x08048310<__do_global_dtors_aux> visited: type = removed call

0x80483fb:mov

ecx, DWORD PTR ss:[ebp + 0xf0<-0x10>] visited: type = removed mov

ret Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 804833e Def of rbp at 804833e

0x80484aa:pop ecx visited: 1 type = removed pop

0x80483fe:sub ecx, edi visited: type = removed sub

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 80484aa Def of rbp at 804833e

jmp_if Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804828e Def of rbp at 804828e Def of rsi at 80483eb

0x80484ab:pop ebx visited: 1 type = removed pop

0x8048400:inc esi visited: type = removed inc

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 80484ab Def of rbp at 804833e

jmp_if

0x80484ac:leave visited: 1 type = removed leave

0x8048401:sar ecx, 0x02 visited: type = removed sar

Def of rax at 804832b Def of rdx at 8048330 Def of rsp at 80484ac Def of rbp at 80484ac

0x80484ad:ret visited: 1 type = removed ret

0x8048404:cmp esi, ecx visited: type = removed cmp

0x8048406:mov edx, edi visited: type = removed mov

0x8048408:jb

0x080483f8<<Insn>> visited: type = removed jb

0x804840a:add esp, 0x0c visited: 1 type = removed add

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804840a Def of rbp at 804828e Def of rsi at 80483eb

0x804840d:pop ebx visited: 1 type = removed pop

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804840d Def of rbp at 804828e Def of rsi at 80483eb

0x804840e:pop esi visited: 1 type = removed pop

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804840e Def of rbp at 804828e Def of rsi at 80483eb

0x804840f:pop edi visited: 1 type = removed pop

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 804840f Def of rbp at 804828e Def of rsi at 80483eb

0x8048410:leave visited: 1 type = removed leave

Def of rax at 80483ed Def of rdx at 80483e0 Def of rbx at 8048464 Def of rsp at 8048410 Def of rbp at 8048410 Def of rsi at 80483eb

0x8048411:ret visited: 1 type = removed ret

Figure 43.1: Dataowow graph for example program.

43.3. DYNAMIC ANALYSIS

325

43.2.2

Variable Analysis

This analysis helps to detect dierent types within a binary. Currently, we use this analysis to detect interrupt calls and their parameters together with the def-use analysis. This allows us to track back the value of parameters to the calls, such as eax and therefore determine whether a interrupt call is for instance a write or read. Another feature is the buer overow analysis. By traversing the CFG, we can detect buer overows.

43.3

Dynamic Analysis

Recent work in ROSE has added support for dynamic analysis and for mixing of dynamic and static analysis using the Intel Pin framework. This optional support in ROSE requires a congure option (--with-IntelPin=< path>). The path in the congure option is the absolute path to the top level directory of the location of the Intel Pin distribution. This support for Intel Pin has only been tested on a 64bit Linux system using the most recent distribution of Intel Pin (version 2.6). Note: The dwarf support in ROSE is currently incompatable with the dwarf support in Intel Pin. A message in the conguration of ROSE will detect if both support for Dwarf and Intel Pin are both specied and exit with an error message that they are incompatable options. See tutorial/intelPin directory for examples using static and dynamic analysis. These example will be improved in the future, at the moment they just call the generation of the binary AST. Note: We have added a x to Intel Pin pin.H le: // DQ (3/9/2009): Avoid letting "using" declarations into header files. #ifndef REQUIRE_PIN_TO_USE_NAMESPACES using namespace LEVEL_PINCLIENT; #endif so that the namespace of Intel Pin would not be a problem for ROSE. The development team have suggested that they may x their use of using declarations for namespaces in their header les. Also note that the path must be absolute since it will be the prex for the pin executable to be run in the internal tests and anything else might be a problem if the path does not contain the current directory (.). Or, perhaps we should test for this in the future. Note 2: Linking to libdwarf.a is a special problem. Both ROSE and Intel Pin use libdwarf.a and both build shred libraries that link to the static version of the library (libdwarf.a). This is a problem im building Pin Tools since both the PinTool and librose.so will use a statically linked dwarf library (internally). This causes the rst use of dwarf to fail, because there are then two versions of the same library in place. The solution is to force at most one static version of the library and let the other one be a shared library. Alternatively both the Pin tool and librose.so can be built using the shared version of dwarf (libdwarf.so). There is a makele rule in libdwarf to build the shared version of the library, but the default is to only build the static library (libdwarf.a), so use make make libdwarf.so to build the shared library. So we allow ROSE to link to the libdwarf.a (statically), which is how ROSE has always worked (this may be revisited in the future). And we force the Pin tool to link

326

CHAPTER 43. BINARY ANALYSIS

using the shared dwarf library (libdwarf.so). Note: The specication of the location of libdwarf.so in the Intel Pin directory structure is problematic using rpath, so for the case of using the Intel Pin package with ROSE please set the LD LIBRARY PATH explicitly (a better solution using rpath may be made available in the future).

43.4

Analysis and Transformations on Binaries

This section demonstrates how the binary can be analyized and transformed via operations on the AST. In this tutorial example we will recognize sequences of NOP (No OPeration) instructions (both single byte and less common multi-byte NOPs) and replace them in the binary with as few multi-byte NOP instructions as required to overwrite the identied NOP sequence (also called a nop sled). In the following subsections we will demonstrate three example codes that work together to demonstrate aspects of the binary analysis and transforamation using ROSE. All of these codes are located in the directory tutorial/binaryAnalysis of the ROSE distribution. We show specically: 1. How to insert NOP sequences into binaries via source-to-source transformations. The tutorial example will demonstrate the trival insertion of random length nop sequences into all functions of an input source code application. The purpose of this is to really just generate arbitrary test codes upon which to use in sperately demonstrating (and testing) the binary analysis support and the binary transformation support (next). This example is shown in gure 43.2. The le name is: 2. How to identify sequences of NOP instructions in the binary (nop sleds). The tutorial example show how to identify arbitrary NOP sequences in the binary. Note that our approach looks only for single instructions that have no side-eect operations and thus qualify as a NOP, and specically does not look at collections of multiple statements that taken togehter have no side-eect and thus could be considered a NOP sequence. Our test on each instruciton is isolated to the SageInterface::isNOP(SgAsmInstruction*) function. Initially this function only detects NOP instructions that are single and multibyte Intel x86 suggested NOP instructions. The catagory of NOP instructions that have no side-eects is broader than this and will be addressed more completely in the future. This example is shown in gures 43.3, 43.4, and 43.5. 3. How transformations on the binary executable are done to support rewriting all identied NOP sequences as Intel x86 suggested NOP instructions. Importantly, this tutorial example of an AST rewrite does not change the size of any part of the nal executable. This example is shown in gure 43.6.

43.4.1

Source-to-source transformations to introduce NOPs

This tutorial example (see gure 43.2) is a source-to-source trnasformation that is used to generate test codes for the binary NOP sequence detection example and the NOP sequence transformation example (the next two tutorial examples). This tutorial uses C language asm statements that are inserted into each function in the source code, the resulting generated source code (with

43.4. ANALYSIS AND TRANSFORMATIONS ON BINARIES

// This t r a n s l a t o r d o es a source to s o u r c e t r a n s f o r m a t i o n on // t h e i n p u t s o u r c e code t o i n t r o d u c e asm s t a t e m e n t s o f NOP // i n s t r u c t i o n s a t t h e t o p o f each f u n c t i o n . #include r o s e . h 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 } 42 44 46 48 50 52 54 56 NopTransform t ; t . traverse ( project , preorder ) ; // r e g e n e r a t e t h e o r i g i n a l e x e c u t a b l e . return backend ( p r o j e c t ) ; } i n t main ( i n t a r g c , char a r g v [ ] ) { // Generate t h e ROSE AST. SgProject p r o j e c t = fr ontend ( argc , argv ) ; // AST c o n s i s t e n c y t e s t s ( o p t i o n a l f o r u s e r s , b u t t h i s e n f o r c e s more o f our t e s t s ) AstTests : : runAllTests ( p r o j e c t ) ; } // Add t o t h e f r o n t o t h e l i s t o f s t a t e m e n t s i n t h e f u n c t i o n body p r e p e n d S t a t e m e n t ( nopStatement , f u n c t i o n B o d y ) ; } // Use a m u l t i b y t e NOP j u s t f o r added fun . // SgAsmStmt nopStatement = buildAsmStatement ( nop ) ; c l a s s NopTransform : public S g S i m p l e P r o c e s s i n g { public : NopTransform ( ) { s r a n d ( t i m e (NULL) ) ; } void v i s i t ( SgNode n ) ; }; void NopTransform : : v i s i t ( SgNode n ) { SgFunctionDefinition functionDefinition = isSgFunctionDefinition (n ) ; i f ( f u n c t i o n D e f i n i t i o n != NULL) { // I n t r o d u c e NOP s i n t o f u n c t i o n d e f i n i t i o n SgBasicBlock functionBody = f u n c t i o n D e f i n i t i o n >g e t b o d y ( ) ; ROSE ASSERT( f u n c t i o n B o d y != NULL ) ; using namespace s t d ; using namespace S a g e I n t e r f a c e ; using namespace S a g e B u i l d e r ;

327

2 4

// Generate a s i n g l e random m u l t i b y t e NOP (1 9 b y t e s l o n g i n c l u d i n g no NOP, when n == 0) i n t n = rand ( ) % 1 0 ; i f (n > 0) { p r i n t f ( I n t r o d u c i n g a m u l t i b y t e NOP i n s t r u c t i o n ( n = %d ) a t t h e t o p o f f u n c t i o n %s \ n , n , f u n c t i o n D e f i n i t i SgAsmStmt nopStatement = b u i l d M u l t i b y t e N o p S t a t e m e n t ( n ) ;

Figure 43.2: Source-to-source transformation to introduce NOP assemble instructions in the generated binary executable. asm statements) is then compiled to generate inputs to use in our detection and transformation of NOP sequences. Although it is possible, the backend compiler (at least the one we are using), does not optimize away these asm statements that represent NOP instructions. In general all C

328

CHAPTER 43. BINARY ANALYSIS

and C++ compilers turn o optimizations upon encountering the asm language construct, so it is easy to use this approach to build binaries from arbitrary source code that have a range of properties. Seeding the source code with such properties causes the binary to have the similar properties. We include this example in the tutorial because it is a cute example of how we can combine source code analsys with binary analysis (in this case we are only supporting testing of the binary analysis). Much more interesting example of the connection of source code and binary analysis are available.

2 4 6 8 10 12

c l a s s C o u n t T r a v e r s a l : public S g S i m p l e P r o c e s s i n g { public : // L o c a l Accumulator A t t r i b u t e int count ; bool p r e v i o u s I n s t r u c t i o n W a s N o p ; SgAsmInstruction nopSequenceStart ; s t d : : v e c t o r <s t d : : p a i r <S g A s m I n s t r u c t i o n , int > > n o p S e q u e n c e s ; C o u n t T r a v e r s a l ( ) : c o u n t ( 0 ) , p r e v i o u s I n s t r u c t i o n W a s N o p ( f a l s e ) {} void v i s i t ( SgNode n ) ; };

Figure 43.3: Header le for the traversal used to identify the NOP sequences in the binary executable.

43.4.2

Detection of NOP sequences in the binary AST

The tutorial example (see gures 43.3, 43.4, and 43.5) shows the detection of NOP sequences and is separated into three gures (the header le, the source code, and the main program). The header le and source le will be reused in the next tutorial example to demonstrate the transformation of the NOP sequences that this tutorial identies. The transformation will be done on the AST (and a new modied executable generated by unparsing the AST). Using a simple preorder traversal over the AST we identify independent sequences of NOP instructions. This traversal save the sequence so that it can be used in both the tutorial example that detects the NOP sequences and also the tutorial example that will transform them to be more compact multi-byte NOP sequences that will use the suggested Intel x86 multi-byte NOP instructions. In this example, the sequence of a NOP sequence is saved as the address of the starting NOP instruction and the number of instructions in the sequence. The function SageInterface::isNOP(SgAsmInstruction*) is used to test of an instruction is a NOP. Presently this test only identies single or multi-byte NOP instrucitons that have been suggested for use as single and multi-byte NOP instructions by Intel. Other instruction may semantically be equivalent to a NOP instruction and they are not identied in this initial work. Later work may include this capability and for this purpose we have isolated out the SageInterface::isNOP(SgAsmInstruction*). Note also that sequences of instructions may be semantically equivalent to a NOP sequence and we also do not attempt to identify these (separate interface functions in the SageInterface namespace have been dened to support this work; these functions are presently unimplemented.

43.4. ANALYSIS AND TRANSFORMATIONS ON BINARIES

329

43.4.3

Transformations on the NOP sequences in the binary AST

Using the results from the previously presented traversal to identify NOP sequences, we demonstrate the transformation to change these locations of the binary (in the AST) and then regenerate the executable to have a dierent representation. The new representation is a more clear (obvious within manual binary analysis) and likely more compressed representation using only the suggested Intel x86 multi-byte NOP.

43.4.4

Conclusions

FIXME: Not sure if the

The identication and transformation have been deomonstrated on the AST and can be ex- resulting executable will use the assembler on each instruction in pressed in the binary binary by calling the backend() function in ROSE; which will regenerate the generated binary from the the executable. The capability to regenerate the executable will not always result in a properly unparsing within ROSE. This might require some more work. formed executable that can be executed, this depends upon the transformations done and does So at the moment the AST is ROSE does not have specialized support for this beyond regnerating the binary executable from transformed and I have to look the AST. In the case of this NOP transformation we have not changed the size of any part into if the binary sees the eect of the binary so any relative osets need not be xed up. Assumeing we have not accedenof the transformation in the AST. taly interpreted data that had values that matched parts of the opcodes that were transformed, then resulting binary executable should execute without problems. This transformation however makes clear how critical it is that data not be interpreted as instructions (which could randomly be interpreted as NOPs in the case of this tutorial example).

330

CHAPTER 43. BINARY ANALYSIS

#include r o s e . h 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 n o p S e q u e n c e s . p u s h b a c k ( p a i r <S g A s m I n s t r u c t i o n , int >( n o p S e q u e n c e S t a r t , c o u n t ) ) ; 44 46 48 50 52 54 56 58 60 62 } 64 } } SgAsmBlock b l o c k = isSgAsmBlock ( n o p S e q u e n c e S t a r t >g e t p a r e n t ( ) ) ; ROSE ASSERT( b l o c k != NULL ) ; S g A sm S t a t e m e n t P t r L i s t & l = b l o c k >g e t s t a t e m e n t L i s t ( ) ; using namespace s t d ; using namespace S a g e I n t e r f a c e ; void C o u n t T r a v e r s a l : : v i s i t ( SgNode n ) { SgAsmInstruction asmInstruction = isSgAsmInstruction (n ) ; i f ( a s m I n s t r u c t i o n != NULL) { // Use t h e new i n t e r f a c e s u p p o r t f o r t h i s ( t h i s d e t e c t s a l l m u l t i b y t e nop i n s t r u c t i o n s ) . i f ( S a g e I n t e r f a c e : : isNOP ( a s m I n s t r u c t i o n ) == true ) { i f ( p r e v i o u s I n s t r u c t i o n W a s N o p == true ) { // Increment t h e l e n g t h o f t h e i d e n t i f i e d NOP s e q u e n c e c o u n t++; } else { count = 1 ; // Record t h e s t a r t i n g a d d r e s s o f t h e NOP s e q u e n c e nopSequenceStart = asmInstruction ; } #include s a g e I n t e r f a c e A s m . h #include d e t e c t N o p S e q u e n c e s T r a v e r s a l . h #include s t r i n g i f y . h

p r e v i o u s I n s t r u c t i o n W a s N o p = true ; } else { i f ( count > 0) { // Report t h e s e q u e n c e when we have d e t e c t e d t h e end o f t h e s e q u e n c e . SgAsmFunction f u n c t i o n D e c l a r a t i o n = getAsmFunction ( a s m I n s t r u c t i o n ) ; p r i n t f ( R e p o r t i n g NOP s e q u e n c e o f l e n g t h %3d a t a d d r e s s %zu i n f u n c t i o n %s ( r e a s o n f o r t count , n o p S e q u e n c e S t a r t >g e t a d d r e s s ( ) , f u n c t i o n D e c l a r a t i o n >get name ( ) . c s t r ( ) , functionDeclaration >g e t r e a s o n ( ) , stringifySgAsmFunctionFunctionReason ( functionDeclaration >g e t r e a s o n ( ) ) . c s t r ( ) ) ;

// Now i t e r a t e o v e r t h e nop i n s t r u c t i o n s i n t h e s e q u e n c e and r e p o r t t h e l e n g h t o f each ( can S g A sm S t a t e m e n t P t r L i s t : : i t e r a t o r i = f i n d ( l . b e g i n ( ) , l . end ( ) , n o p S e q u e n c e S t a r t ) ; ROSE ASSERT( i != l . end ( ) ) ; int counter = 0 ; while ( ( i != a s m I n s t r u c t i o n ) && ( i != l . end ( ) ) ) { p r i n t f ( NOP #%2d i s l e n g t h = %2d \ n , c o u n t e r ++,( i n t ) i s S g A s m I n s t r u c t i o n ( i )> g e t i ++; } } count = 0 ; previousInstructionWasNop = false ;

Figure 43.4: Example code to identify the NOP sequences in the binary executable.

43.4. ANALYSIS AND TRANSFORMATIONS ON BINARIES

331

2 4 6 8 10 12 14 16 18 20 22 24

// // // // // //

This example ROSE t r a n s l a t o r j u s t do e s an a n a l y s i s o f t h e i n p u t b i n a r y . The e x i s t e n c e o f NOP s e q u e n c e s a r e d e t e c t e d and r e p o r t e d . For each NOP i n t h e s e q u e n c e t h e s i z e o f t h e NOP i n s t r u c t i o n i s r e p o r t e d . Note t h a t a l l m u l t i b y t e NOP i n s t r u c t i o n s a r e d e t e c t e d and so t h e r e p o r t e d s i z e o f each i n s t r u c t i o n i n t h e NOP s e q u e n c e can v a r y . I n t e l m u l t i b y t e NOP i n s t r u c t i o n s can be 19 b y t e s l o n g .

#include r o s e . h #include d e t e c t N o p S e q u e n c e s T r a v e r s a l . h i n t main ( i n t a r g c , char a r g v [ ] ) { // Generate t h e ROSE AST. SgProject p r o j e c t = fr ontend ( argc , argv ) ; // AST c o n s i s t e n c y t e s t s ( o p t i o n a l f o r u s e r s , b u t t h i s e n f o r c e s more o f our t e s t s ) AstTests : : runAllTests ( p r o j e c t ) ; CountTraversal t ; t . traverse ( project , preorder ) ; // r e g e n e r a t e t h e o r i g i n a l e x e c u t a b l e . return backend ( p r o j e c t ) ; }

Figure 43.5: Main program using the traversal to identify the NOP sequences in the binary executable.

332
// // T h i s t u t o r i a l e x a m p l e s h o w how t o i n t r o d u c e t r a n s f o r m a t i o n s on t h e i n p u t b i n a r y e x e c t u t a b l e . r o s e . h s a g e I n t e r f a c e A s m . h d e t e c t N o p S e q u e n c e s T r a v e r s a l . h

CHAPTER 43. BINARY ANALYSIS

2 4 6 8 10 12 14 16 18 20 22 24 26 28

#i n c l u d e #i n c l u d e #i n c l u d e

u s i n g namespace s t d ; u s i n g namespace S a g e I n t e r f a c e ; u s i n g namespace S a g e B u i l d e r A s m ;

class {

NopReplacementTraversal

public

SgSimpleProcessing

public : // L o c a l A c c u m u l a t o r A t t r i b u t e s t d : : v e c t o r < s t d : : p a i r <S g A s m I n s t r u c t i o n , i n t > > & n o p S e q u e n c e s ; s t d : : v e c t o r < s t d : : p a i r <S g A s m I n s t r u c t i o n , i n t > > : : i t e r a t o r l i s t I t e r a t o r ; SgAsmStatementPtrList d e l e t e L i s t ; N o p R e p l a c e m e n t T r a v e r s a l ( s t d : : v e c t o r <s t d : : p a i r <S g A s m I n s t r u c t i o n , i n t > > & X) { l i s t I t e r a t o r = nopSequences . begin ( ) ; } void }; visit ( SgNode n ); : n o p S e q u e n c e s (X)

30 32 34 36 38 40 printf 42 // 44 46 // 48 50 52 54 56 58 printf 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 } 98 // 100 102 104 106 108 110 // // // R e m o v i n g t h e o r i g i n a l NOP r e m o v e S t a t e m e n t ( i ) ; r e m o v e I n s t r u c t i o n ( i ) ; d e l e t e L i s t . push back ( i ) ; instruction . // // } Add t o t h e f r o n t o t h e l i s t o f s t a t e m e n t s i n t h e f u n c t i o n b o d y prependStatement ( nopStatement , bloc k ) ; i n s e r t I n s t r u c t i o n B e f o r e ( / t a r g e t / a s m I n s t r u c t i o n , / new i n s t r u c t i o n / // // // Now r e w r i t e t h e AST t o u s e t h e n u m b e r o f m u l t i b y t e NOPS s p e c i f i e d i n t h e n u m b e r O f N o p S i z e N a r r a y f o r e a c h s i z e o f m u l t i b y t e NOP . printf ( Rewrite t h e AST h e r e ! \n ) ; // // // From t h e s i z e o f t h e NOP s l e d , c o m p u t e h o w many m u l t i b y t e T h i s c o d e i s s p e c i f i c t o x 8 6 s u p p o r t i n g m u l t i b y t e NOPs i n c o n s t i n t MAX SIZE MULTIBYTE NOP = 9 ; i n t numberOfNopSizeN [ MAX SIZE MULTIBYTE NOP + 1 ] ; NOPs w e w i l l w a n t t o u s e s i z e d 1 9 b y t e s l o n g . to cover the same space in the ( n o p s l e d s i z e = %d \ n , n o p s l e d s i z e ) ; Now i t e r a t e o v e r t h e n o p i n s t r u c t i o n s i n t h e s e q u e n c e a n d r e p o r t t h e l e n g h t o f e a c h ( c a n b e m u l t i b y t e S g A s m S t a t e m e n t P t r L i s t : : i t e r a t o r i = f i n d ( l . b e g i n ( ) , l . end ( ) , a s m I n s t r u c t i o n ) ; ROSE ASSERT ( i != l . end ( ) ) ; int n o p s l e d s i z e = 0; f o r ( i n t j = 0 ; j < n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ; j ++) { ROSE ASSERT ( i != l . end ( ) ) ; p r i n t f ( NOP #%2d i s l e n g t h = %2d \ n , j , ( i n t ) i s S g A s m I n s t r u c t i o n ( i )> g e t r a w b y t e s ( ) . s i z e ( ) ) ; n o p s l e d s i z e += ( i n t ) i s S g A s m I n s t r u c t i o n ( i )> g e t r a w b y t e s ( ) . s i z e ( ) ; i ++; } nop SgAsmBlock b l o c k = isSgAsmBlock ( a s m I n s t r u c t i o n >g e t p a r e n t ( ) ) ; ROSE ASSERT ( b l o c k != NULL ) ; SgAsmStatementPtrList & l = block >g e t s t a t e m e n t L i s t ( ) ; ( n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s = %d \ n , n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ) ; void N o p R e p l a c e m e n t T r a v e r s a l : : v i s i t ( SgNode n ) { SgAsmInstruction asmInstruction = isSgAsmInstruction (n ) ; i f ( a s m I n s t r u c t i o n != NULL && a s m I n s t r u c t i o n == l i s t I t e r a t o r >f i r s t ) { / / T h i s i s t h e i n s t r u c t i o n i n t h e AST t h a t m a t c h e s t h e s t a r t o f o n e SgAsmBlock b l o c k = i s S g A s m B l o c k ( a s m I n s t r u c t i o n >g e t p a r e n t ( ) ) ; int numberOfOriginalNopInstructions = l i s t I t e r a t o r >s e c o n d ;

of

the

NOP

sequences .

instructions

bina

0 th element of numberOfNopSizeN i s not used . numberOfNopSizeN [ 0 ] = 0 ; f o r ( i n t i = MAX SIZE MULTIBYTE NOP ; i > 0 ; i ) { // i n t n u m b e r O f N o p S i z e 9 = n o p s l e d s i z e / 9 ; numberOfNopSizeN [ i ] = n o p s l e d s i z e / i ; n o p s l e d s i z e = numberOfNopSizeN [ i ] i ; i f ( numberOfNopSizeN [ i ] > 0 ) p r i n t f ( numberOfNopSizeN [%d ] = %d \ n , i , numberOfNopSizeN [ i ] ) ; }

I g n o r e t h e 0 t h e l e m e n t o f n u m b e r O f N o p S i z e N ( s i n c e a 0 l e n g t h NOP d o e s n o t m a k e s e n s e ) . f o r ( i n t i = 1 ; i <= MAX SIZE MULTIBYTE NOP ; i ++) { / / B u i l d t h i s many b y t e s o f t h i s s i z e f o r ( i n t j = 0 ; j < numberOfNopSizeN [ i ] ; j ++) { / / We w a n t t o b u i l d a ( b i n a r y AST n o d e ) S g A s m I n s t r u c t i o n o b j e c t i n s t e a d o f a ( s o u r c e // SgAsmStmt n o p S t a t e m e n t = b u i l d M u l t i b y t e N o p S t a t e m e n t ( i ) ; / / S g A s m I n s t r u c t i o n m u l t i B y t e N o p = m a k e I n s t r u c t i o n ( x 8 6 n o p , n o p , modrm ) ; SgAsmInstruction multiByteNopInstruction = buildMultibyteNopInstruction ( i ) ;

code

AST

node )

SgAsmStmt .

multiByteNopInstruction ) ;

Now i t e r a t e o v e r t h e n o p i n s t r u c t i o n s i n t h e s e q u e n c e a n d r e p o r t t h e l e n g h t o f e a c h ( c a n b e m u l t i b y t e n o p i n s t r u c t i o n s i = f i n d ( l . b e g i n ( ) , l . end ( ) , a s m I n s t r u c t i o n ) ; ROSE ASSERT ( i != l . end ( ) ) ; f o r ( i n t j = 0 ; j < n u m b e r O f O r i g i n a l N o p I n s t r u c t i o n s ; j ++) { ROSE ASSERT ( i != l . end ( ) ) ; / / p r i n t f ( D e l e t i n g o r i g i n a l NOP i n s t r u c t i o n #%2d i s l e n g t h = %2 d \ n , j , ( i n t ) i s S g A s m I n s t r u c t i o n ( i )> g e t r a w b y t e s ( )

Chapter 44

Binary Construction
ROSE is normally used in such a way that a le (source code or binary) is parsed to construct an AST, then operations are performed on the AST, and the modied AST is unparsed to create a new source or binary le. However, it is also possible to construct an AST explicitly without parsing and then use that AST to generate the output. The AST construction interface for binary les was designed so that working les could be created simply, while still providing methods to control the ner details of the resulting le. The example in this chapter shows how to construct a statically linked ELF executable containing a small .text section that simply causes the process to exit with a specic non-zero value.

44.1

Constructors

The AST node constructors are designed to construct the tree from the root down, and thus generally take the parent node as an argument. Nodes that refer to other nodes as prerequisites also take those prerequisites as arguments. For instance, an ELF Relocation Section is a child of the ELF File Header but also needs an ELF Symbol Table and therefore takes both objects as constructor arguments.

44.2

Read-Only Data Members

When two or more le formats have a similar notion then that notion is represented in a base class. However, part of the information may continue to survive in the specialized class. In these situations modications to the specilized data will probably be overridden by the generic values from the base class. For instance, all formats have a notion of byte order which is represented in the base class SgAsmGenericHeader as little- or big-endian (an enumeration constant). The ELF specication provides an 8-bit unsigned eld to store the byte order and therefore has potentially more than two possibilities. Any value assigned to the ELF-specic byte order will likely be overwritten by the generic byte order before the AST is unparsed. A similar situation arises with section osets, sizes, memory mappings, permissions, etc. The SgAsmGenericSection class represents ELF Sections, ELF Segments, PE Objects, and other 333

334

CHAPTER 44. BINARY CONSTRUCTION

contiguous regions of a le and has methods for obtaining/setting these values. In addition, many of the formats have some sort of table that describes these sections and which also contains similar information (e.g., the ELF Segment Table, a.k.a., the ELF Program Header Table). As above, the generic representation of these notions (stored in SgAsmGenericSection) override the format-specic values (stored in SgAsmElfSegmentEntry). ROSETTA doesnt make a distinction between data members that can be user-modied and data members that should be modied only by the parser. Therefore it is up to the user to be aware that certain data members will have their values computed or copied from other locations in the AST during the unparsing phase.

44.3

Constructing the Executable File Container

All executable les are stored as children of an SgAsmGenericFile node. The children are le format headers (SgAsmGenericHeader) such as an ELF File Header (SgAsmElfFileHeader). This design allows a single executable le to potentially contain more than one executable and is used by formats like Windows-PE where the le contains a DOS File Header as well as a PE File Header. For the purposes of this example the SgAsmGenericFile node will serve as the root of the AST and therefore we do not need to specify a parent in the constructor argument list. SgAsmGenericFile *ef = new SgAsmGenericFile;

44.4

Constructing the ELF File Header

The ELF File Header is the rst thing in the le, always at oset zero. File headers are always children of an SgAsmGenericFile which is specied as the constructor argument. The section constructors (a le header is a kind of section) always create the new section so it begins at the current end-of-le and contains at least one byte. This ensures that each section has a unique starting address, which will be important when le memory is actually allocated and sections need to be moved aroundthe allocator needs to know the relative positions of the sections in order to correctly relocate them. If we were parsing an ELF le we would usually use ROSEs frontend() method. However, one can also parse the le by rst constructing the SgAsmElfFileHeader and then invoking its parse() method, which parses the ELF File Header and everything that can be reached from that header. We use the typical 0x400000 as the virtual address of the main LOAD segment, which occupies the rst part of the le up through the end of the .text section (see below). ELF File Headers dont actually store a base address, so instead of assigning one to the SgAsmElfFileHeader well leave the headers base address at the default zero and add base va explicitly whenever we need to. SgAsmElfFileHeader *fhdr = new SgAsmElfFileHeader(ef); fhdr->get_exec_format()->set_word_size(8); /* default is 32-bit; we want 64-bit */ fhdr->set_isa(SgAsmExecutableFileFormat::ISA_X8664_Family); /* instruction set architecture; default rose_addr_t base_va = 0x400000; /* base virtual address */

44.5. CONSTRUCTING THE ELF SEGMENT TABLE

335

44.5

Constructing the ELF Segment Table

ELF executable les always have an ELF Segment Table (also called the ELF Program Header Table), which usually appears immediately after the ELF File Header. The ELF Segment Table describes contiguous regions of the le that should be memory mapped by the loader. ELF Segments dont have namesnames are imparted to the segment by virtue of the segment also being described by the ELF Section Table, which well create later. Being a contiguous region of the le, an ELF Segment Table (SgAsmElfSegmentTable) is derived from SgAsmGenericSection. All non-header sections have a header as a parent, which we supply as an argument to the constructor. Since all ELF Segments will be children of the ELF File Header rather than children of the ELF Segment Table, we could dene the ELF Segment Table at the end rather than here. But dening it now is an easy way to get it located in its usuall location immediately after the ELF File Header. SgAsmElfSegmentTable *segtab = new SgAsmElfSegmentTable(fhdr);

44.6

Constructing the .text Section

ROSE doesnt treat a .text section as being anything particularly specialits just a regular SgAsmElfSection, which derives from SgAsmGenericSection. However, in this example, we want to make sure that our .text section gets initialized with some instructions. The easiest way to do that is to specialize SgAsmElfSection and override or augment a few of the virtual functions. We need to override two functions. First, the calculate sizes() function should return the size we need to store the instructions. Well treat the instructions as an array of entries each entry being one byte of the instruction stream. In other words, each entry is one byte in length consisting of one required byte and no optional bytes. We need to also override the unparse() method since the base class will just ll the .text section with zeros. The SgAsmGenericSection::write method we use will write the instructions starting at the rst byte of the section. Finally, we need to augment the reallocate() method. This method is reponsible for allocating space in the le for the section and performing any other necessary pre-unparsing actions. We dont need to allocate space since the base classs method will take care of that in conjuction with our version of calculate sizes(), but we do need to set a special ELF ag (SHF ALLOC) in the ELF Segment Table entry for this section. Theres a few ways to accomplish this. We do it this way because the ELF Section Table Entry is not created until later and we want to demonstrate how to keep all .text-related code in close proximity. class TextSection : public SgAsmElfSection { public: TextSection(SgAsmElfFileHeader *fhdr, size_t ins_size, const unsigned char *ins_bytes) : SgAsmElfSection(fhdr), ins_size(ins_size), ins_bytes(ins_bytes) {} virtual rose_addr_t calculate_sizes(size_t *entsize, size_t *required, size_t *optional, size_t *entcount) if (entsize) *entsize = 1; /* an "entry" is one byte of instruction */ if (required) *required = 1; /* each "entry" is also stored in one byte of the file */ if (optional) *optional = 0; /* there is no extra data per instruction byte */

336

CHAPTER 44. BINARY CONSTRUCTION

if (entcount) *entcount = ins_size; /* number of "entries" is the total instruction b return ins_size; /* return value is section size required */ } virtual bool reallocate() { bool retval = SgAsmElfSection::reallocate(); /* returns true if size or position of any sectio SgAsmElfSectionTableEntry *ste = get_section_entry(); ste->set_sh_flags(ste->get_sh_flags() | 0x02); /* set the SHF_ALLOC bit */ return retval; } virtual void unparse(std::ostream &f) const { write(f, 0, ins_size, ins_bytes); /* Write the instructions at offset zero in secti } size_t ins_size; const unsigned char *ins_bytes; }; The section constructors and reallocators dont worry about alignment issuesthey always allocate from the next available byte. However, instructions typically need to satisfy some alignment constraints. We can easily adjust the le oset chosen by the constructor, but we also need to tell the reallocator about the alignment constraint. Even if we didnt ever resize the .text section the reallocator could be called for some other section in such a way that it needs to move the .text section to a new le oset. For the purpose of this tutorial we want to be very picky about the location of the ELF Segment Table. We want it to immediately follow the ELF File Header without any intervening bytes of padding. At the current time, the ELF File Header has a size of one byte and will eventually be reallocated. When we reallocate the header the subsequent sections will need to be shifted to higher le osets. When this happens, the allocator shifts them all by the same amount taking care to satisfy all alignment constraints, which means that an alignment constraint of byte bytes on the .text section will induce a similar alignment on the ELF Segment Table. Since we dont want that, the best practice is to call reallocate() now, before we create the .text section.

ef->reallocate(); /* Give existing sections a chance to all static const unsigned char instructions[] = {0xb8, 0x01, 0x00, 0x00, 0x00, 0xbb, 0x56, 0x00, 0x00, 0x SgAsmElfSection *text = new TextSection(fhdr, NELMTS(instructions), instructions); text->set_purpose(SgAsmGenericSection::SP_PROGRAM); /* Program-supplied text/data/etc. */ text->set_offset(ALIGN_UP(text->get_offset(), 4)); /* Align on an 8-byte boundary */ text->set_file_alignment(4); /* Tell reallocator about alignment const text->set_mapped_alignment(4); /* Alignment constraint for memory mappin text->set_mapped_rva(base_va+text->get_offset()); /* Mapped address is based on file offset text->set_mapped_size(text->get_size()); /* Mapped size is same as file size */ text->set_mapped_rperm(true); /* Readable */ text->set_mapped_wperm(false); /* Not writable */ text->set_mapped_xperm(true); /* Executable */ At this point the text section doesnt have a name. We want to name it .text and we want those characters to eventually be stored in the ELF le in a string table which well

44.7. CONSTRUCTING A LOAD SEGMENT

337

provide later. In ELF, section names are represented by the sections entry in the ELF Section Table as an oset into an ELF String Table for a NUL-terminated ASCII string. ROSE manages strings using the SgAsmGenericString class, which has two subclasses: one for strings that arent stored in the executable le (SgAsmBasicString) and one for strings that are stored in the le (SgAsmStoredString). Both are capable of string an std::string value and querying its byte oset (although SgAsmBasicString::get oset() will always return SgAsmGenericString::unallocated). Since we havent added the .text section to the ELF Section Table yet the new section has an SgAsmBasicString name. We can assign a string to the name now and the string will be allocated in the ELF le when weve provided further information. text->get_name()->set_string(".text"); The ELF File Header needs to know the virtual address at which to start running the program. In ROSE, virtual addresses can be attached to a specic section so that if the section is ever moved the address is automatically updated. Some formats allow more than one entry address which is why the method is called add entry rva() rather than set entry rva(). ELF, however, only allows one entry address. rose_rva_t entry_rva(text->get_mapped_rva(), text); fhdr->add_entry_rva(entry_rva);

44.7

Constructing a LOAD Segment

ELF Segments dene parts of an executable le that should be mapped into memory by the loader. A program will typically have a LOAD segment that begins at the rst byte of the le and continues through the last instruction (in our case, the end of the .text section) and which is mapped to virtual address 0x400000. Weve already created the ELF Segment Table, so all we need to do now is create an ELF Segment and add it to the ELF Segment Table. ELF Segments, like ELF Sections, are represented by SgAsmElfSection. An SgAsmElfSection is an ELF Section if it has an entry in the ELF Section Table, and/or its an ELF Segment if it has an entry in the ELF Segment Table. The methods get section entry() and get segment entry() retrieve the actual entries in those tables. Recall that the constructor creates new sections located at the current end-of-le and containing one byte. Our LOAD segment needs to have a dierent oset and size. SgAsmElfSection *seg1 = new SgAsmElfSection(fhdr); seg1->get_name()->set_string("LOAD"); seg1->set_offset(0); seg1->set_size(text->get_offset() + text->get_size()); seg1->set_mapped_rva(base_va); seg1->set_mapped_size(seg1->get_size()); seg1->set_mapped_rperm(true); seg1->set_mapped_wperm(false); seg1->set_mapped_xperm(true); segtab->add_section(seg1); /* /* /* /* /* /* /* /* /* /*

ELF Segments are represented by SgAsmElfSection Segment names arent saved (but useful for debug Starts at beginning of file */ Extends to end of .text section */ Typically mapped by loader to this memory addres Make mapped size match size in the file */ Readable */ Not writable */ Executable */ Add definition to ELF Segment Table */

338

CHAPTER 44. BINARY CONSTRUCTION

44.8

Constructing a PAX Segment

This documentation shows how to construct a generic ELF Segment, giving it a particular le oset and size. ELF Segments dont have names stored in the le, but we can assign a name to the AST node to aid in debuggingit just wont be written out. When parsing an ELF le, segment names are generated based on the type stored in the entry of the ELF Segment Table. For a PAX segment we want this type to be PT PAX FLAGS (the default is PT LOAD).

SgAsmElfSection *pax = new SgAsmElfSection(fhdr); pax->get_name()->set_string("PAX Flags"); /* Name just for debugging */ pax->set_offset(0); /* Override address to be at zero rather pax->set_size(0); /* Override size to be zero rather than o segtab->add_section(pax); /* Add definition to ELF Segment Table */ pax->get_segment_entry()->set_type(SgAsmElfSegmentTableEntry::PT_PAX_FLAGS);

44.9

Constructing a String Table

An ELF String Table always corresponds to a single ELF Section of class SgAsmElfStringSection and thus youll often see the term ELF String Section used interchangeably with ELF String Table even though theyre two unique but closely tied classes internally. When the ELF String Section is created a corresponding ELF String Table is also created under the covers. Since string tables manage their own memory in reponse to the strings they contain, one should never adjust the size of the ELF String Section (its actually ne to enlarge the section and the new space will become free space in the string table). ELF les typically have multiple string tables so that section names are in a dierent section than symbol names, etc. In this tutorial well create the section names string table, typically called .shstrtab, but use it for all string storage. SgAsmElfStringSection *shstrtab = new SgAsmElfStringSection(fhdr); shstrtab->get_name()->set_string(".shstrtab");

44.10

Constructing an ELF Section Table

We do this last because we want the ELF Section Table to appear at the end of the le and this is the easiest way to achieve that. Theres really not much one needs to do to create the ELF Section Table other than provide the ELF File Header as a parent and supply a string table. The string table we created above isnt activated until we assign it to the ELF Section Table. The rst SgAsmElfStringSection added to the SgAsmElfSectionTable becomes the string table for storing section names. It is permissible to add other sections to the table before adding the string table. SgAsmElfSectionTable *sectab = new SgAsmElfSectionTable(fhdr); sectab->add_section(text); /* Add the .text section */ sectab->add_section(shstrtab); /* Add the string table to store section

44.11. ALLOCATING SPACE

339

44.11

Allocating Space

Prior to calling unparse(), we need to make sure that all sections have a chance to allocate space for themselves, and perform any other operations necessary. Its not always possible to determine sizes at an earlier time, and most constructors would have declared sizes of only one byte. The reallocate() method is dened in the SgAsmGenericFile class since it operates over the entire collection of sections simultaneously. In other words, if a section needs to grow then all the sections located after it in the le need to be shifted to higher le osets. ef->reallocate(); The reallocate() method has a shortcoming (as of 2008-12-19) in that it might not correctly update memory mappings in the case when the mapping for a section is inferred from the mapping of a containing segment. This can happen in our example since the .text sections memory mapping is a function of the LOAD Segment mapping. The work-around is to adjust mappings for these situations and then call reallocate() one nal time. This nal reallocate() call wont move any sections, but should always be the last thing to call before unparsing() (it gives sections a chance to update data dependencies which is not possible during unparse() due to its const nature).

text->set_mapped_rva(seg1->get_mapped_rva()+(text->get_offset()-seg1->get_offset())); ef->reallocate(); /*wont resize or move things this time since we didnt modify much since the last call to re

44.12

Produce a Debugging Dump

A debugging dump can be made with the following code. This dump will not be identical to the one produced by parsing and dumping the resulting le since we never parsed a le (a dump contains some information thats parser-specic). ef->dump(stdout); SgAsmGenericSectionPtrList all = ef->get_sections(true); for (size_t i=0; i<all.size(); i++) { fprintf(stdout, "Section %zu:\n", i); all[i]->dump(stdout, " ", -1); }

44.13

Produce the Executable File

The executable le is produced by unparsing the AST. std::ofstream f("a.out"); ef->unparse(f); Note that the resulting le will not be created with execute permissionthat must be added manually.

340

CHAPTER 44. BINARY CONSTRUCTION

Chapter 45

Dwarf Debug Support

DWARF is a widely used, standardized debugging data format. DWARF was originally designed along with ELF, although it is independent of object le formats. The name is a pun on ELF that has no ocial meaning but may be an acronym for Debug With Attributed Record Format. See Wikipedia for more information about the Dwarf debug format. This chapter presents the support in ROSE for Dwarf 3 debug information; its representation in the AST and its use in binary analysis tools. This work is part of general work to support as much information as possible about binaries. In the following sections we use a small example (see gure 45.1) that demonstrates various features of Dwarf. The source code of our binary example is:
// Test code t o d e m o n s t r a t i o n o f dwarf s u p p o r t . // Designed t o be s m a l l b e c a u s e Dwarf i s v e r b o s e . namespace e x a mp l e n a m e s p a c e { int a ; };

2 4 6 8 10 12 14 16 18 20

i n t main ( ) { int x = 42; // Loops a r e not r e c o g n i s e d i n Dwarf . . . f o r ( i n t i = 0 ; i < 1 0 ; i ++) { x = ( x % 2) ? x 2 : x + 1 ; } return 0 ; }

Figure 45.1: Example source code used to generate Dwarf AST for analysis. Much larger binaries can be analyzed, but such larger binary executables are more dicult to present (in this tutorial). 341

342

CHAPTER 45. DWARF DEBUG SUPPORT

45.1

ROSE AST of Dwarf IR nodes

ROSE tools process the binary into an AST that is used to represent all the information in the binary executable. Figure 45.2 shows the subset of that AST (which includes the rest of the binary le format and the disassembled instructions) that is specic to Dwarf. A command line option (rose:visualize dwarf only) is used to restrict the generated dot le visualization to just the Dwarf information. This option is used in combination with -rose:read executable le format only to process only the binary le format (thus skipping the instruction disassembly).

45.2

Source Position to Instruction Address Mapping

XME: This need to be to correctly return sets nd sets of addresses in be a second interface.

One of the valuable parts of Dwarf is the mapping between the source lines and the instruction addresses at a statement level (provided in the .debug line section). Even though Dwarf does not represent all statements in the source code, the mapping is signicantly ner granularity than that provided at only the function level by the symbol table (which identies the functions by instruction address, but not the source le line numbers; the later requires source code analysis). The example code in 45.3 shows the mapping between the source code lines and the instruction addresses. Output from the compilation of this test code and running it with the example input results in the output shown in gure 45.4. This output shows the binary executables instruction address range (binary compiled on Linux x86 system), the range of lines of source code used by the binary executable, the mapping of a source code range of line numbers to the instruction addresses, and the mapping of a range of instruction addresses to the source code line numbers.

45.2. SOURCE POSITION TO INSTRUCTION ADDRESS MAPPING

343

0:505 SgProject 1 0x2ba4cb9b2010

fileList_ptr

1:504 SgFileList 1 0x1ae97930

*[0]

2:503 SgBinaryComposite 2 0x2ba4cba4b010

genericFileList

interpretations

3:498 SgAsmGenericFileList 1 0x1aeccdc0

499:502 SgAsmInterpretationList 1 0x1aeb9530

*[0]

*[0]

4:497 SgAsmGenericFile 2 0x2ba4cba98010

500:501 SgAsmInterpretation 2 0x1b084120

headers

holes

dwarf_info global_block

5:494 SgAsmGenericHeaderList 1 0x1aef2740

495:496 SgAsmGenericSectionList 0 0x1af05fd0

*[0]

6:493 SgAsmElfFileHeader ELF File Header 3 0x2ba4cbaeb010

exec_format

dlls

sections

7:8 SgAsmGenericFormat 0 0x1af3dd60

9:10 SgAsmGenericDLLList 0 0x1af2a4d0

11:492 SgAsmGenericSectionList 34 0x1af06020

*[0]

*[1]

*[2]

*[6]

*[7]

*[5]

*[4]

*[3]

*[10] *[11]

*[9] *[12]

*[8]

*[13]

*[14]

*[15]

*[16]

*[17]

*[18]

*[19]

*[20]

*[21]

*[22]

*[23]

*[24]

*[25]

*[26]

*[27]

*[28]

*[29]

*[30]

*[31]

*[32]

*[33]

12:13 SgAsmElfSectionTable ELF Section Table 0 0x2ba4cbb4f010

14:17 SgAsmElfStringSection .shstrtab 2 0x2ba4cbba6010

18:21 SgAsmElfSection 2 0x2ba4cbc05010

22:27 SgAsmElfSection .interp 2 0x2ba4cbc050f0

28:33 SgAsmElfSection .note.ABI-tag 2 0x2ba4cbc051d0

34:37 SgAsmElfStringSection .dynstr 2 0x2ba4cbba60f8

38:53 SgAsmElfSymverNeededSection .gnu.version_r 3 0x2ba4cbc3c010

54:57 SgAsmElfSection .init 2 0x2ba4cbc052b0

58:61 SgAsmElfSection .plt 2 0x2ba4cbc05390

62:65 SgAsmElfSection .text 2 0x2ba4cbc05470

66:69 SgAsmElfSection .fini 2 0x2ba4cbc05550

70:73 SgAsmElfSection .rodata 2 0x2ba4cbc05630

74:79 SgAsmElfEHFrameSection .eh_frame 3 0x2ba4cbc75010

80:83 SgAsmElfSection .ctors 2 0x2ba4cbc05710

84:87 SgAsmElfSection .dtors 2 0x2ba4cbc057f0

88:91 SgAsmElfSection .jcr 2 0x2ba4cbc058d0

92:99 SgAsmElfDynamicSection .dynamic 3 0x2ba4cbcae010

100:103 SgAsmElfSection .got 2 0x2ba4cbc059b0

104:107 SgAsmElfSection .got.plt 2 0x2ba4cbc05a90

108:111 SgAsmElfSection .data 2 0x2ba4cbc05b70

112:115 SgAsmElfSection .bss 2 0x2ba4cbc05c50

116:119 SgAsmElfSection .comment 2 0x2ba4cbc05d30

120:123 SgAsmElfStringSection .strtab 2 0x2ba4cbba61e0

124:153 SgAsmElfSymbolSection .dynsym 3 0x2ba4cbce7010

154:171 SgAsmElfSymverSection .gnu.version 3 0x2ba4cbd46010

172:177 SgAsmElfRelocSection .rel.dyn 3 0x2ba4cbd7f010

178:183 SgAsmElfRelocSection .rel.plt 3 0x2ba4cbd7f108

184:469 SgAsmElfSymbolSection .symtab 3 0x2ba4cbce7100

470:473 SgAsmElfSection .hash 2 0x2ba4cbc05e10

474:475 SgAsmElfSegmentTable ELF Segment Table 0 0x2ba4cbdbc010

476:479 SgAsmElfSection PHDR#0 2 0x2ba4cbc05ef0

480:483 SgAsmElfSection LOAD#2 2 0x2ba4cbc05fd0

484:487 SgAsmElfSection LOAD#3 2 0x2ba4cbc060b0

488:491 SgAsmElfSection GNU_STACK#6 2 0x2ba4cbc06190

section_entrysegment_entry

section_entry segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry segment_entry

section_entry segment_entry

entries

section_entry segment_entry

section_entry segment_entry

section_entry segment_entry

section_entry segment_entry

section_entry segment_entry

section_entry segment_entry

ci_entries

section_entry segment_entry

section_entry segment_entry

section_entry segment_entry

section_entry

segment_entry

entries

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

symbols

section_entry segment_entry

entries

section_entry segment_entry

entries

section_entry segment_entry

entries

section_entry segment_entry

symbols

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

section_entry

segment_entry

15:16 SgAsmElfSectionTableEntry 0 0x2ba4cbb80ee8

19:20 SgAsmElfSectionTableEntry 0 0x2ba4cbb80010

23:24 SgAsmElfSectionTableEntry 0 0x2ba4cbb800a8

25:26 SgAsmElfSegmentTableEntry 0 0x2ba4cbded0a0

29:30 SgAsmElfSectionTableEntry 0 0x2ba4cbb80140

31:32 SgAsmElfSegmentTableEntry 0 0x2ba4cbded2e0

35:36 SgAsmElfSectionTableEntry 0 0x2ba4cbb80308

39:40 SgAsmElfSectionTableEntry 0 0x2ba4cbb80438

41:52 SgAsmElfSymverNeededEntryList 1 0x1af78710

55:56 SgAsmElfSectionTableEntry 0 0x2ba4cbb80600

59:60 SgAsmElfSectionTableEntry 0 0x2ba4cbb80698

63:64 SgAsmElfSectionTableEntry 0 0x2ba4cbb80730

67:68 SgAsmElfSectionTableEntry 0 0x2ba4cbb807c8

71:72 SgAsmElfSectionTableEntry 0 0x2ba4cbb80860

75:76 SgAsmElfSectionTableEntry 0 0x2ba4cbb808f8

77:78 SgAsmElfEHFrameEntryCIList 0 0x1afc6950

81:82 SgAsmElfSectionTableEntry 0 0x2ba4cbb80990

85:86 SgAsmElfSectionTableEntry 0 0x2ba4cbb80a28

89:90 SgAsmElfSectionTableEntry 0 0x2ba4cbb80ac0

93:94 SgAsmElfSectionTableEntry 0 0x2ba4cbb80b58

95:96 SgAsmElfSegmentTableEntry 0 0x2ba4cbded250

97:98 SgAsmElfDynamicEntryList 0 0x1afda1e0

101:102 SgAsmElfSectionTableEntry 0 0x2ba4cbb80bf0

105:106 SgAsmElfSectionTableEntry 0 0x2ba4cbb80c88

109:110 SgAsmElfSectionTableEntry 0 0x2ba4cbb80d20

113:114 SgAsmElfSectionTableEntry 0 0x2ba4cbb80db8

117:118 SgAsmElfSectionTableEntry 0 0x2ba4cbb80e50

121:122 SgAsmElfSectionTableEntry 0 0x2ba4cbb81018

125:126 SgAsmElfSectionTableEntry 0 0x2ba4cbb80270

127:152 SgAsmElfSymbolList 6 0x1b01e7d0

155:156 SgAsmElfSectionTableEntry 0 0x2ba4cbb803a0

157:170 SgAsmElfSymverEntryList 6 0x1b032060

173:174 SgAsmElfSectionTableEntry 0 0x2ba4cbb804d0

175:176 SgAsmElfRelocEntryList 0 0x1b055300

179:180 SgAsmElfSectionTableEntry 0 0x2ba4cbb80568

181:182 SgAsmElfRelocEntryList 0 0x1b055350

185:186 SgAsmElfSectionTableEntry 0 0x2ba4cbb80f80

187:468 SgAsmElfSymbolList 70 0x1b01e820

471:472 SgAsmElfSectionTableEntry 0 0x2ba4cbb801d8

477:478 SgAsmElfSegmentTableEntry 0 0x2ba4cbded010

481:482 SgAsmElfSegmentTableEntry 0 0x2ba4cbded130

485:486 SgAsmElfSegmentTableEntry 0 0x2ba4cbded1c0

489:490 SgAsmElfSegmentTableEntry 0 0x2ba4cbded370

*[0]

*[0]

*[2]

*[1]

*[3] *[4]

*[5]

*[0]

*[1]

*[2]

*[3]

*[4]

*[5]

*[0]

*[1]

*[2]

*[3]

*[4]

*[5]

*[6]

*[7]

*[8]

*[9]

*[10]

*[11]

*[12]

*[13]

*[14]

*[15]

*[16]

*[17]

*[18]

*[19]

*[20]

*[21]

*[22]

*[23]

*[24]

*[25]

*[26]

*[27]

*[28]

*[29]

*[30]

*[31]

*[32]

*[33]

*[34]

*[35]

*[36]

*[37]

*[38]

*[39]

*[40]

*[41]

*[42]

*[43]

*[44]

*[45]

*[46]

*[47]

*[48]

*[49]

*[50]

*[51]

*[52]

*[53]

*[54]

*[55]

*[56]

*[57]

*[58]

*[59]

*[60]

*[61]

*[62]

*[63]

*[64]

*[65]

*[66]

*[67]

*[68]

*[69]

42:51 SgAsmElfSymverNeededEntry 2 0x1af8bfa0

128:131 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22010

132:135 SgAsmElfSymbol malloc 1 0x2ba4cbd220a0

136:139 SgAsmElfSymbol __libc_start_main 1 0x2ba4cbd22130

140:143 SgAsmElfSymbol _IO_stdin_used 1 0x2ba4cbd221c0

144:147 SgAsmElfSymbol _Jv_RegisterClasses 1 0x2ba4cbd22250

148:151 SgAsmElfSymbol __gmon_start__ 1 0x2ba4cbd222e0

158:159 SgAsmElfSymverEntry 0 0x1b0458f0

160:161 SgAsmElfSymverEntry 0 0x1b045930

162:163 SgAsmElfSymverEntry 0 0x1b045970

164:165 SgAsmElfSymverEntry 0 0x1b0459b0

166:167 SgAsmElfSymverEntry 0 0x1b0459f0

168:169 SgAsmElfSymverEntry 0 0x1b045a30

188:191 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22370

192:195 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22400

196:199 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22490

200:203 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22520

204:207 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd225b0

208:211 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22640

212:215 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd226d0

216:219 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22760

220:223 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd227f0

224:227 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22880

228:231 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22910

232:235 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd229a0

236:239 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22a30

240:243 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22ac0

244:247 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22b50

248:251 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22be0

252:255 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22c70

256:259 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22d00

260:263 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22d90

264:267 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22e20

268:271 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22eb0

272:275 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22f40

276:279 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd22fd0

280:283 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd23060

284:287 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd230f0

288:291 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd23180

292:295 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd23210

296:299 SgAsmElfSymbol no_name_for_symbol 1 0x2ba4cbd232a0

300:303 SgAsmElfSymbol call_gmon_start 1 0x2ba4cbd23330

304:307 SgAsmElfSymbol crtstuff.c 1 0x2ba4cbd233c0

308:311 SgAsmElfSymbol __CTOR_LIST__ 1 0x2ba4cbd23450

312:315 SgAsmElfSymbol __DTOR_LIST__ 1 0x2ba4cbd234e0

316:319 SgAsmElfSymbol __JCR_LIST__ 1 0x2ba4cbd23570

320:323 SgAsmElfSymbol completed.5610 1 0x2ba4cbd23600

324:327 SgAsmElfSymbol p.5608 1 0x2ba4cbd23690

328:331 SgAsmElfSymbol __do_global_dtors_aux 1 0x2ba4cbd23720

332:335 SgAsmElfSymbol frame_dummy 1 0x2ba4cbd237b0

336:339 SgAsmElfSymbol crtstuff.c 1 0x2ba4cbd23840

340:343 SgAsmElfSymbol __CTOR_END__ 1 0x2ba4cbd238d0

344:347 SgAsmElfSymbol __DTOR_END__ 1 0x2ba4cbd23960

348:351 SgAsmElfSymbol __FRAME_END__ 1 0x2ba4cbd239f0

352:355 SgAsmElfSymbol __JCR_END__ 1 0x2ba4cbd23a80

356:359 SgAsmElfSymbol __do_global_ctors_aux 1 0x2ba4cbd23b10

360:363 SgAsmElfSymbol buffer2.c 1 0x2ba4cbd23ba0

364:367 SgAsmElfSymbol _DYNAMIC 1 0x2ba4cbd23c30

368:371 SgAsmElfSymbol _fp_hw 1 0x2ba4cbd23cc0

372:375 SgAsmElfSymbol __fini_array_end 1 0x2ba4cbd23d50

376:379 SgAsmElfSymbol __dso_handle 1 0x2ba4cbd23de0

380:383 SgAsmElfSymbol __libc_csu_fini 1 0x2ba4cbd23e70

384:387 SgAsmElfSymbol _init 1 0x2ba4cbd23f00

388:391 SgAsmElfSymbol malloc@@GLIBC_2.0 1 0x2ba4cbd23f90

392:395 SgAsmElfSymbol _start 1 0x2ba4cbd24020

396:399 SgAsmElfSymbol __fini_array_start 1 0x2ba4cbd240b0

400:403 SgAsmElfSymbol __libc_csu_init 1 0x2ba4cbd24140

404:407 SgAsmElfSymbol __bss_start 1 0x2ba4cbd241d0

408:411 SgAsmElfSymbol main 1 0x2ba4cbd24260

412:415 SgAsmElfSymbol __libc_start_main@@GLIBC_2.0 1 0x2ba4cbd242f0

416:419 SgAsmElfSymbol __init_array_end 1 0x2ba4cbd24380

420:423 SgAsmElfSymbol data_start 1 0x2ba4cbd24410

424:427 SgAsmElfSymbol _fini 1 0x2ba4cbd244a0

428:431 SgAsmElfSymbol __preinit_array_end 1 0x2ba4cbd24530

432:435 SgAsmElfSymbol _edata 1 0x2ba4cbd245c0

436:439 SgAsmElfSymbol _GLOBAL_OFFSET_TABLE_ 1 0x2ba4cbd24650

440:443 SgAsmElfSymbol _end 1 0x2ba4cbd246e0

444:447 SgAsmElfSymbol __init_array_start 1 0x2ba4cbd24770

448:451 SgAsmElfSymbol _IO_stdin_used 1 0x2ba4cbd24800

452:455 SgAsmElfSymbol __data_start 1 0x2ba4cbd24890

456:459 SgAsmElfSymbol _Jv_RegisterClasses 1 0x2ba4cbd24920

460:463 SgAsmElfSymbol __preinit_array_start 1 0x2ba4cbd249b0

464:467 SgAsmElfSymbol __gmon_start__ 1 0x2ba4cbd24a40

file_name

entries

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

name

43:44 SgAsmStoredString 0 0x1af68e80

45:50 SgAsmElfSymverNeededAuxList 1 0x1af9f830

129:130 SgAsmStoredString 0 0x1af693c0

133:134 SgAsmStoredString 0 0x1af69400

137:138 SgAsmStoredString 0 0x1af69440

141:142 SgAsmStoredString 0 0x1af69480

145:146 SgAsmStoredString 0 0x1af694c0

149:150 SgAsmStoredString 0 0x1af69500

189:190 SgAsmStoredString 0 0x1af69640

193:194 SgAsmStoredString 0 0x1af69680

197:198 SgAsmStoredString 0 0x1af696c0

201:202 SgAsmStoredString 0 0x1af69700

205:206 SgAsmStoredString 0 0x1af69740

209:210 SgAsmStoredString 0 0x1af69780

213:214 SgAsmStoredString 0 0x1af697c0

217:218 SgAsmStoredString 0 0x1af69800

221:222 SgAsmStoredString 0 0x1af69840

225:226 SgAsmStoredString 0 0x1af69880

229:230 SgAsmStoredString 0 0x1af698c0

233:234 SgAsmStoredString 0 0x1af69900

237:238 SgAsmStoredString 0 0x1af69940

241:242 SgAsmStoredString 0 0x1af69980

245:246 SgAsmStoredString 0 0x1af699c0

249:250 SgAsmStoredString 0 0x1af69a00

253:254 SgAsmStoredString 0 0x1af69a40

257:258 SgAsmStoredString 0 0x1af69a80

261:262 SgAsmStoredString 0 0x1af69ac0

265:266 SgAsmStoredString 0 0x1af69b00

269:270 SgAsmStoredString 0 0x1af69b40

273:274 SgAsmStoredString 0 0x1af69b80

277:278 SgAsmStoredString 0 0x1af69bc0

281:282 SgAsmStoredString 0 0x1af69c00

285:286 SgAsmStoredString 0 0x1af69c40

289:290 SgAsmStoredString 0 0x1af69c80

293:294 SgAsmStoredString 0 0x1af69cc0

297:298 SgAsmStoredString 0 0x1af69d00

301:302 SgAsmStoredString 0 0x1af69d40

305:306 SgAsmStoredString 0 0x1af69d80

309:310 SgAsmStoredString 0 0x1af69dc0

313:314 SgAsmStoredString 0 0x1af69e00

317:318 SgAsmStoredString 0 0x1af69e40

321:322 SgAsmStoredString 0 0x1af69e80

325:326 SgAsmStoredString 0 0x1af69ec0

329:330 SgAsmStoredString 0 0x1af69f00

333:334 SgAsmStoredString 0 0x1af69f40

337:338 SgAsmStoredString 0 0x1af69f80

341:342 SgAsmStoredString 0 0x1af69fc0

345:346 SgAsmStoredString 0 0x1af6a000

349:350 SgAsmStoredString 0 0x1af6a040

353:354 SgAsmStoredString 0 0x1af6a080

357:358 SgAsmStoredString 0 0x1af6a0c0

361:362 SgAsmStoredString 0 0x1af6a100

365:366 SgAsmStoredString 0 0x1af6a140

369:370 SgAsmStoredString 0 0x1af6a180

373:374 SgAsmStoredString 0 0x1af6a1c0

377:378 SgAsmStoredString 0 0x1af6a200

381:382 SgAsmStoredString 0 0x1af6a240

385:386 SgAsmStoredString 0 0x1af6a280

389:390 SgAsmStoredString 0 0x1af6a2c0

393:394 SgAsmStoredString 0 0x1af6a300

397:398 SgAsmStoredString 0 0x1af6a340

401:402 SgAsmStoredString 0 0x1af6a380

405:406 SgAsmStoredString 0 0x1af6a3c0

409:410 SgAsmStoredString 0 0x1af6a400

413:414 SgAsmStoredString 0 0x1af6a440

417:418 SgAsmStoredString 0 0x1af6a480

421:422 SgAsmStoredString 0 0x1af6a4c0

425:426 SgAsmStoredString 0 0x1af6a500

429:430 SgAsmStoredString 0 0x1af6a540

433:434 SgAsmStoredString 0 0x1af6a580

437:438 SgAsmStoredString 0 0x1af6a5c0

441:442 SgAsmStoredString 0 0x1af6a600

445:446 SgAsmStoredString 0 0x1af6a640

449:450 SgAsmStoredString 0 0x1af6a680

453:454 SgAsmStoredString 0 0x1af6a6c0

457:458 SgAsmStoredString 0 0x1af6a700

461:462 SgAsmStoredString 0 0x1af6a740

465:466 SgAsmStoredString 0 0x1af6a780

*[0]

46:49 SgAsmElfSymverNeededAux 1 0x1afb30c0

name

47:48 SgAsmStoredString 0 0x1af68ec0

Figure 45.2: Dwarf AST (subset of ROSE binary AST).

344

CHAPTER 45. DWARF DEBUG SUPPORT

#include r o s e . h 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 u i n t 6 4 t minInstructionAddress = addressRange . f i r s t ; 52 54 56 58 60 62 64 66 68 p r i n t f ( \ n I n s t r u c t i o n a d d r e s s e s computed from s o u r c e p o s i t i o n s : \ n ) ; // I t e r a t e o v e r l i n e numbers t o map b a c k t o i n s t r u c t i o n a d d r e s s e s f o r ( i n t lineNumber = minLine 2 ; lineNumber <= maxLine + 2 ; lineNumber++) { // Out o f range v a l u e s g e n e r a t e t h e n e x t a d d r e s s or NULL. F i l e I d L i n e C o l u m n F i l e P o s i t i o n s ( s o u r c e f i l e i d , s t d : : p a i r <int , int >(lineNumber , columnNumber ) ) ; u i n t 6 4 t i n s t r u c t i o n A d d r e s s = SgAsmDwarfLineList : : so u r c eC o de T o A dd r e ss ( s ) ; printf ( so ur c e C o d e T o A d dr es s(%d,%d,%d ) = 0x%l x \ n , s . f i r s t , s . s e c o n d . f i r s t , s . s e c o n d . s eco nd , i n s t r u c } i n t minLine = s o u r c e F i l e R a n g e . f i r s t . f i r s t ; i n t maxLine = s o u r c e F i l e R a n g e . s e c o n d . f i r s t ; i n t columnNumber = 1; // Compute t h e b i n a r y e x e c u t a b l e i n s t r u c t i o n a d d r e s s range s t d : : p a i r < u i n t 6 4 t , u i n t 6 4 t > a d d r e s s R a n g e = SgAsmDwarfLineList : : i n s t r u c t i o n R a n g e ( ) ; p r i n t f ( \ nBinary i n s t r u c t i o n a d d r e s s r a n g e = ( 0 x%l x , 0x%l x ) \ n , a d d r e s s R a n g e . f i r s t , a d d r e s s R a n g e . s e c o n d ) ; s t d : : s t r i n g s o u r c e F i l e n a m e = S g F i l e I n f o : : getFilenameFromID ( s o u r c e f i l e i d ) ; p r i n t f ( f i l e i d = %d s o u r c e F i l e n a m e = %s \ n , s o u r c e f i l e i d , s o u r c e F i l e n a m e . c s t r ( ) ) ; // Compute t h e s o u r c e l i n e range from t h e i n s t r u c t i o n s i n t h e b i n a r y e x e c u t a b l e s t d : : p a i r <L i n e C o l u m n F i l e P o s i t i o n , L i n e C o l u m n F i l e P o s i t i o n > s o u r c e F i l e R a n g e ; s o u r c e F i l e R a n g e = SgAsmDwarfLineList : : sourceCodeRange ( s o u r c e f i l e i d ) ; p r i n t f ( \ n S o u r c e f i l e l i n e number r a n g e f o r : \ n f i l e = %s ( i d = %d ) \ n [ ( l i n e=%d , sourceFilename . c s t r ( ) , s o u r c e f i l e i d , s o u r c e F i l e R a n g e . f i r s t . f i r s t , s o u r c e F i l e R a n g e . f i r s t . se c o n d , sourceFileRange . second . f i r s t , sourceFileRange . second . second ) ; c o l=%d ) , ( l i n e=%d , // Increment t o g e t t h e n e x t f i l e i d ( f o r t h e s o u r c e f i l e , i n s t e a d o f t h e b i n a r y int s o u r c e f i l e i d = b i n a r y f i l e i d + 1 ; file ) // This i s c o n t r o l e d by u s i n g t h e with dwarf c o n f i g u r e command l i n e o p t i o n . #i f USE ROSE DWARF SUPPORT // The i n p u t f i l e i s t h e b i n a r y f i l e . . . int b i n a r y f i l e i d = project > g e t f i l e L i s t ()[0] > g e t f i l e i n f o ()> g e t f i l e i d ( ) ; int main ( i n t a r g c , char a r g v ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; ROSE ASSERT ( p r o j e c t != NULL ) ;

s t d : : s t r i n g b i n a r y F i l e n a m e = S g F i l e I n f o : : getFilenameFromID ( b i n a r y f i l e i d ) ; p r i n t f ( f i l e i d = %d b i n a r y F i l e n a m e = %s \ n , b i n a r y f i l e i d , b i n a r y F i l e n a m e . c s t r ( ) ) ;

// I t e r a t e o v e r t h e a d d r e s s e s o f t h e b i n a r y and compute t h e s o u r c e code l i n e numbers ( l i m i t t h e range so t h e p r i n t f ( \ n S o u r c e l i n e s computed from a d d r e s s r a n g e ( t r u n c a t e d t o keep o u t p u t s h o r t ) : \ n ) ; f o r ( u i n t 6 4 t a d d r e s s = m i n I n s t r u c t i o n A d d r e s s 1 ; a d d r e s s < m i n I n s t r u c t i o n A d d r e s s + 2 5 ; a d d r e s s++) { F i l e I d L i n e C o l u m n F i l e P o s i t i o n s map = SgAsmDwarfLineList : : a d d r e s s T o S o u r c e C o d e ( a d d r e s s ) ; printf ( a d d r e s s T o S o u r c e C o d e : a d d r e s s 0x%l x = (%d,%d,%d ) \ n , a d d r e s s , s map . f i r s t , s map . s e c o n d . f i r s } #e l s e p r i n t f ( \ n \nROSE must be c o n f i g u r e d w i t h with d w a r f= <path t o l i b d w a r f > t o u s e Dwarf s u p p o r t . \ n \ n ) ; #e n d i f p r i n t f ( \ nProgram Terminated Normally ! \ n \ n ) ; // S k i p c a l l t o backend s i n c e t h i s return backend ( p r o j e c t ) ; } is j u s t an a n a l y s i s .

Figure 45.3: Example source code (typical for reading in a binary or source le).

45.2. SOURCE POSITION TO INSTRUCTION ADDRESS MAPPING

345

2 ROSE must be c o n f i g u r e d w i t h with d w a r f= <path t o l i b d w a r f > t o u s e Dwarf s u p p o r t . 4 6 Program Terminated Normally !

Figure 45.4: Example source code (typical for reading in a binary or source le).

346

CHAPTER 45. DWARF DEBUG SUPPORT

Part VII

Interacting with Other Tools

How to build interoperable tools using ROSE.

347

Chapter 46

Abstract Handles to Language Constructs

This chapter describes a reference design and its corresponding implementation for supporting abstract handles to language constructs in source code and optimization phases. It can be used to facilitate the interoperability between compilers and tools. We dene an abstract handle as a representation for a unique language construct in a specic program. Interoperability between tools is achieved by writing out the abstract handles as strings and reading them within other tools to build the equivalent abstract handle. 1 The idea is to dene identiers for unique statements, loops, functions, and other language constructs in source code. Given the diverse user requirements, an ideal specication should include multiple forms to specify a language construct. Currently, we are interested in the following forms for specifying language constructs: Source le position information including path, lename, line and column number etc. GNU standard source position from http://www.gnu.org/prep/standards/html node/ Errors.html presents some examples. Global or local numbering of specied language construct in source le (e.g. 2nd do loop in a global scope). The le is itself specied using an abstract handle (typically generated from the le name). Global or local names of constructs. Some language constructs, such as les, function denitions and namespace, have names which can be used as their handle within a context. Language-specic label mechanisms. These include named constructs in Fortran, numbered labels in Fortran, and statement labels in C and C++, etc.
1 Abstract Handles are not appropriate for program analysis since they are not intended to be used to capture the full structure of a program. Instead, Abstract Handles represent references to language constructs in a program, capturing only a programs local structure; intended to support interoperability between source based tools (including compilers). We dont advise the use of abstract handles in an aggressive way to construct an alternative intermediate representation (IR) for a full program.

349

350

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

In addition to human-readable forms, compilers and tools can generate internal IDs for language constructs. It is up to compiler/tool developers to provide a way to convert their internal representations into human-readable formats. Abstract Handles can have any of the human-readable or machine-generated forms. A handle can be used alone or combined with other handles to specify a language construct. A handle can also be converted from one form to another (e.g. from a compiler specic form to an human readable form relative to the source position; lename, line number, etc.). Abstract handles can have dierent lifetimes depending on their use and implementation. An abstract handle might be required to be persistent if it is used to reference a language construct that would be optimized over multiple executions of one or more dierent tools. Where as an abstract-handle might be internally generated only for purposes of optimizations used in a single execution (e.g. optimization within a compiler).

46.1

Use Case

A typical use can for Abstract Handles might be for a performance tool to identify a collection of loops in functions that are computationally intensive and to construct Abstract Handles that refer to this specic loops. Then pass the Abstract Handles to a second tool that might analyze the source code and/or the binary executable to evaluate if the computational costs are reasonable or if optimizations might be possible. The specic goal of the Abstract Handles is to support these sorts of uses within autotuning using diverse tools used and/or developed as part of autotuning research within the DOE SciDAC PERI project.

46.2

Syntax

A possible grammar for abstract handles could be: /* a handle is a single handle item or a link of them separated by ::, or other delimiters */ handle ::= handle_item | handle :: handle_item /* Each handle item consists of construct_type and a specifier. Or it can be a compiler generated id of any forms. */ handle_item ::= construct_type specifier | compiler_generated_handle /* Construct types are implementation dependent. An implementation can support a subset of legal constructs or all of them. We define a minimum set of common construct type names here and will grow this list as needed. */ construct_type ::= Project|SourceFile|FunctionDeclaration|ForStatement|... /* A specifier is used to locate a particular construct

46.3. EXAMPLES e.g: <name, "foo"> */ specifier::= < specifier_type , specifier_value >

351

/* tokens for specifier types could be name, position,numbering, label, etc. specifier type is necessary to avoid ambiguity for specifier values, because a same value could be interpreted in different specifier types otherwise */ specifier_type::= name | position | numbering | label /* Possible values for a specifier */ specifier_value::= string_lit|int_lit|position_value| label_value /*A label could be either integer or string */ label_value::= int_lit | string_lit /* Start and end source line and column information e.g.: 13.5-55.4, 13, 13.5 , 13.5-55 */ position_value:: = line_number[ . column_number][ - line_number[ . column_number]] /* Integer value: one or more digits */ int_lit ::= [0-9]+ /* String value: start with a letter, followed by zero or more letters or digits */ string_lit ::= [a-z][a-z0-9]*

46.3

Examples

We give some examples of language handles using the syntax mentioned above. Canonical ASTs node type names are used as the construct type names. Other implementations can use their own construct type names. A le handle consisting of only one handle item: SourceFile<name,"/home/PERI/test111.f"> A function handle using a named handle item, combined with a parent handle using a name also: SourceFile<name,"/home/PERI/test111.f">::FunctionDeclaration<name,"foo">

352

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS A function handle using source position(A function starting at line 12, column 1 till line 30, column 1 within a le): SourceFile<name,"/home/PERI/test111.f">::FunctionDeclaration<position,"12.1-30.1">

A function handle using numbering(The rst function denition in a le): SourceFile<name,/home/PERI/test111.f">::FunctionDeclaration<numbering,1> A return statement using source position (A return statement at line 100): SourceFile<name,/home/PERI/test222.c>::ReturnStatement<position,"100"> A loop using numbering information (The second loop in function main()): SourceFile<name,"/home/PERI/test222.c">::FunctionDeclaration<name,"main">:: ForStatement<numbering,2> A nested loop using numbering information (The rst loop inside the second loop in function main()): SourceFile<name,"/home/PERI/test222.c">::FunctionDeclaration<name,"main">:: ForStatement<numbering,2>::ForStatement<numbering,1>

46.4

Reference Implementation

Abstract Handles are fundamentally compiler and tool independent, however to clarify the concepts, provide meaningful examples, a working reference implementation we have provided a reference implementation in ROSE. The source les are located in src/midend/abstractHandle in the ROSE distribution. A generic interface (abstract handle.h and abstract handle.cpp) provides data structures and operations for manipulating abstract handles using source le positions, numbering, or names. Any compilers and tools can have their own implementations using the same interface.

46.4.1

Connecting to ROSE

A ROSE adapter (roseAdapter.h and roseAdapter.cpp) using the interface is provided as a concrete implementation for the maximum capability of the implementation (within a sourceto-source compiler). Figure 46.1 shows the code (using ROSE) to generate abstract handles for loops in an input source le (as in Figure 46.2). Abstract handle constructors generate handles from abstract nodes, which are implemented using ROSE AST nodes. Source position is used by default to generate a handle item. Names or numbering are used instead when source position information is not available. The Constructor can also be used to generate a handle item using a specied handle type (numbering handles in the example). Figure 46.3 is the output showing the generated handles for the loops.

46.4. REFERENCE IMPLEMENTATION

/ Example code t o g e n e r a t e a b s t r a c t h a n d l e s f o r l a n g u a g e c o n s t r u c t s by Liao , / #include #include #include #include #include 10/6/2008 r o s e . h <i o s t r e a m > a b s t r a c t h a n d l e . h r o s e A d a p t e r . h < s t r i n g . h>

353

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42

using namespace s t d ; using namespace A b s t r a c t H a n d l e ; // a g l o b a l h a n d l e f o r t h e c u r r e n t f i l e s t a t i c a b s t r a c t h a n d l e f i l e h a n d l e = NULL ; c l a s s v i s i t o r T r a v e r s a l : public A s t S i m p l e P r o c e s s i n g { protected : v i r t u a l void v i s i t ( SgNode n ) ; }; void v i s i t o r T r a v e r s a l : : v i s i t ( SgNode n ) { S gF orStatement f o r l o o p = i s S g F o r S t a t e m e n t ( n ) ; if { ( forloop ) cout < < C r e a t i n g h a n d l e s f o r a l o o p c o n s t r u c t . . . << e n d l ; // Create an a b s t r a c t node a b s t r a c t n o d e anode= b u i l d r o s e N o d e ( f o r l o o p ) ; // Create an a b s t r a c t h a n d l e from t h e a b s t r a c t node // Using s o u r c e p o s i t i o n s p e c i f i e r s by d e f a u l t a b s t r a c t h a n d l e a h a n d l e = new a b s t r a c t h a n d l e ( anode ) ; cout < <a h a n d l e > t o S t r i n g ()<< e n d l ; // Create h a n d l e s b a s e d on numbering s p e c i f i e r s w i t h i n t h e f i l e a b s t r a c t h a n d l e b h a n d l e = new a b s t r a c t h a n d l e ( anode , e number in g , f i l e h a n d l e ) ; cout < <bhandle > t o S t r i n g ()<< e n d l < <e n d l ; } } i n t main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Generate a f i l e h a n d l e abstract node f i l e n o d e = buildroseNode (( project >g e t f i l e L i s t ( ) ) [ 0 ] ) ; f i l e h a n d l e = new a b s t r a c t h a n d l e ( f i l e n o d e ) ; // Generate h a n d l e s f o r l a n g u a g e c o n s t r u c t s visitorTraversal myvisitor ; myvisitor . traverseInputFiles ( project , preorder ) ; // Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; } t h e vendor s c o m p i l e r

44 46 48 50 52 54 56 58

Figure 46.1: Example 1: Generated handles for loops: using constructors with or without a specied handle type.

354

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

2 4 6 8 10 12 14 16

/ t e s t i n p u t f o r g e n e r a t e d a b s t r a c t h a n d l e s / int a [ 1 0 0 ] [ 1 0 0 ] [ 1 0 0 ] ; void f o o ( ) { int i , j , k ; f o r ( i =0; i ++; i < 100) f o r ( j =0; j ++; j < 100) f o r ( k =0; k++;k < 100) a [ i ] [ j ] [ k]= i+j+k ; f o r ( i =0; i ++; i < 100) f o r ( j =0; j ++; j < 100) f o r ( k =0; k++;k < 100) a [ i ] [ j ] [ k ]+=5; }

Figure 46.2: Example 1: Example source code with some loops, used as input.

46.4. REFERENCE IMPLEMENTATION

355

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46

Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,7.3 10.25 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,1 > Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,8.5 10.25 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,2 > Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,9.7 10.25 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,3 > Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,12.3 15.22 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,4 > Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,13.5 15.22 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,5 > Creating handles for a loop construct . . . P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement < p o s i t i o n ,14.7 15.22 > P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 1 . cpp > : : ForStatement <numbering ,6 >

Figure 46.3: Example 1: Abstract handles generated for loops.

356

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

A second example (shown in Figure 46.4) demonstrates how to create handles using userspecied strings representing handle items for language constructs within a source le (shown in Figure 46.5). This is particularly useful to grab internal language constructs from handles provided by external software tools. The output of the example is given in Figure 46.6.
/ Example code t o g e n e r a t e l a n g u a g e h a n d l e s from i n p u t s t r i n g s a b o u t source p o s i t i o n information numbering i n f o r m a t i o n by Liao , / #include #include #include #include #include 10/9/2008 r o s e . h <i o s t r e a m > < s t r i n g . h> a b s t r a c t h a n d l e . h r o s e A d a p t e r . h

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48

using namespace s t d ; using namespace A b s t r a c t H a n d l e ; i n t main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Generate a f i l e h a n d l e from t h e f i r s t f i l e o f t h e p r o j e c t a b s t r a c t n o d e f i l e n o d e= b u i l d r o s e N o d e ( ( p r o j e c t >g e t f i l e L i s t ( ) ) [ 0 ] ) ; a b s t r a c t h a n d l e h a n d l e 0 = new a b s t r a c t h a n d l e ( f i l e n o d e ) ; cout < < C r e a t e d a f i l e h a n d l e : \ n<<h a n d l e 0 > t o S t r i n g ()<< e n d l < <e n d l ; ; // Create a h a n d l e t o a namespace g i v e n i t s name and p a r e n t h a n d l e s t r i n g i n p u t 1= N a m e s p a c e D e c l a r a t i o n S t a t e m e n t <name , s p a c e 1 > ; a b s t r a c t h a n d l e h a n d l e 1 = new a b s t r a c t h a n d l e ( h a n d l e 0 , i n p u t 1 ) ; cout < < C r e a t e d a h a n d l e : \ n<<h a n d l e 1 > t o S t r i n g ()<< e n d l < <e n d l ; >getNode()> t o S t r i n g ()<< e n d l < <e n d l ; cout < < I t p o i n t s t o : \ n<<h a n d l e 1 // Create a h a n d l e w i t h i n t h e f i l e , g i v e n a s t r i n g s p e c i f y i n g // i t s c o n s t r u c t t y p e ( c l a s s d e c l a r a t i o n ) and s o u r c e p o s i t i o n s t r i n g i n p u t= C l a s s D e c l a r a t i o n < p o s i t i o n ,4.3 9.2 > ; a b s t r a c t h a n d l e h a n d l e 2 = new a b s t r a c t h a n d l e ( h a n d l e 0 , i n p u t ) ; > t o S t r i n g ()<< e n d l < <e n d l ; cout < < C r e a t e d a h a n d l e : \ n<<h a n d l e 2 cout < < I t p o i n t s t o : \ n<<h a n d l e 2 >getNode()> t o S t r i n g ()<< e n d l < <e n d l ; ; // f i n d t h e second f u n c t i o n d e c l a r a t i o n w i t h i n h a n d l e 2 a b s t r a c t h a n d l e h a n d l e 3 ( h a n d l e 2 , F u n c t i o n D e c l a r a t i o n <numbering ,2 > ) ; cout < < C r e a t e d a h a n d l e : \ n<< h a n d l e 3 . t o S t r i n g ()<< e n d l < <e n d l ; cout < < I t p o i n t s t o : \ n<< h a n d l e 3 . getNode()> t o S t r i n g ()<< e n d l ; // Generate s o u r c e code from AST and c a l l return backend ( p r o j e c t ) ; } t h e vendor s c o m p i l e r

Figure 46.4: Example 2: Generated handles from strings representing handle items.

46.4. REFERENCE IMPLEMENTATION

357

2 4 6 8 10

void b a r ( i n t x ) ; namespace s p a c e 1 { class A { public : void b a r ( i n t x ) ; void f o o ( ) ; }; }

Figure 46.5: Example 2: Source code with some language constructs.

2 4 6 8 10 12 14 16 18 20 22 24 26 28

Created a f i l e handle : P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 2 . cpp> Created a handle : P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 2 . cpp > : : N a m e s p a c e D e c l a r a t i o n S t a t e m e n t <name , s p a c e 1 > I t points to : namespace s p a c e 1 { c l a s s A { public : void b a r ( i n t x ) ; void f o o ( ) ; } ; } Created a handle : P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 2 . cpp > : : C l a s s D e c l a r a t i o n < p o s i t i o n ,4.3 9.2 > I t points to : c l a s s A { public : void b a r ( i n t x ) ; void f o o ( ) ; } ; Created a handle : P r o j e c t <numbering , 1 > : : F i l e L i s t <numbering , 1 > : : S o u r c e F i l e <name , / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / inputCod e A b s t r a c t H a n d l e 2 . cpp > : : C l a s s D e c l a r a t i o n < p o s i t i o n , 4 . 3 9 . 2 > : : MemberFunctionDeclar a t i o n <numbering ,2 > I t points to : public : void f o o ( ) ;

Figure 46.6: Example 2: Handles generated from string and their language constructs.

358

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

46.4.2

Connecting to External Tools

A third example is provided to demonstrate how to use the abstract interface with any other tools, which may have less features in terms of supported language constructs and their correlations compared to a compiler. Assume a tool operating on some simple for-loops within an arbitrary source le (the input le is not shown in this example). Such a tool might have an internal data structure representing loops; such as that in given in Figure 46.7. We will show how the tool specic data structure for loops can be used to generate abstract handles and output as strings that can be used by other tools which use abstract handles (which would generate the abstract handles by reading the strings).
/ A toy loop data s t r u c t u r e demonstrating a t h i n c l i e n t of a b s t r a c t handles : A s i m p l e s t l o o p t o o l which k e e p s a t r e e o f l o o p s i n a f i l e / #i f n d e f my loop INCLUDED #d e f i n e my loop INCLUDED #include < s t r i n g > #include < v e c t o r > c l a s s MyLoop { public : std : : s t r i n g sourceFileName ; s i z e t line number ; s t d : : v e c t o r <MyLoop> c h i l d r e n ; MyLoop p a r e n t ; }; #e n d i f

2 4 6 8 10 12 14 16 18

Figure 46.7: Example 3: A simple data structure used to represent a loop in an arbitrary tool. An adapter (loopAdapter.h and loopAdapter.cpp) using the proposed abstract handle interface is given in src/midend/abstractHandle. It provides a concrete implementation for the interface for the simple loops and adds a node to support le nodes (Compared to a full-featured IR for a compiler, the le node is an additional detail for tools without data structures to support les). The test program is given in Figure 46.8. Again, it creates a top level le handle rst. Then a loop handle (loop handle1) is created within the le handle using its relative numbering information. The loop handle2 is created from from its string format using le position information (using GNU standard le position syntax). The loop handle3 uses its relative numbering information within loop handle1. The output of the program is shown in Figure 46.9. It demonstrates the generated strings to represent the abstract handles in the arbitrary code operated upon by the tool. Interoperability is achieved by another tool reading in the generated string representation to generate an abstract handle to the same source code language construct.

46.4. REFERENCE IMPLEMENTATION

359

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54

#include #include #include #include #include #include

<i o s t r e a m > <s t r i n g > <v e c t o r > a b s t r a c t h a n d l e . h myloop . h l o o p A d a p t e r . h

using namespace s t d ; using namespace A b s t r a c t H a n d l e ; i n t main ( ) { // P r e p a r i n g t h e i n t e r n a l l o o p r e p r e s e n t a t i o n // d e c l a r e and i n i t i a l i z e a l i s t o f l o o p s u s i n g MyLoop // The l o o p t o o l s h o u l d be a b l e t o g e n e r a t e i t s r e p r e s e n t a t i o n from // s o u r c e code somehow . We f i l l i t up manually h e r e . v e c t o r <MyLoop > l o o p s ; MyLoop l o o p 1 , l o o p 2 , l o o p 3 ; l o o p 1 . s o u r c e F i l e N a m e= f i l e 1 . c ; loop1 . line number = 7; l o o p 1 . p a r e n t = NULL ; l o o p 2 . s o u r c e F i l e N a m e= f i l e 1 . c ; loop2 . line number = 8; l o o p 2 . p a r e n t=&l o o p 1 ; l o o p 1 . c h i l d r e n . p u s h b a c k (& l o o p 2 ) ; l o o p 3 . s o u r c e F i l e N a m e= f i l e 1 . c ; loop3 . line number = 12; l o o p 3 . p a r e n t=NULL ; l o o p s . p u s h b a c k (& l o o p 1 ) ; l o o p s . p u s h b a c k (& l o o p 3 ) ; // using a b s t r a c t handles // Generate t h e a b s t r a c t h a n d l e for the source f i l e f i l e N o d e f i l e n o d e = new f i l e N o d e ( f i l e 1 . c ) ; filenode >setMLoops ( l o o p s ) ; a b s t r a c t h a n d l e f i l e h a n d l e = new a b s t r a c t h a n d l e ( f i l e n o d e ) ; cout < < C r e a t e d a f i l e h a n d l e : <<e n d l < <f i l e h a n d l e > t o S t r i n g ()<< e n d l ; // Create a l o o p h a n d l e w i t h i n t h e f i l e u s i n g numbering i n f o . a b s t r a c t n o d e l o o p n o d e 1= new loopNode (& l o o p 1 ) ; a b s t r a c t h a n d l e l o o p h a n d l e 1= new a b s t r a c t h a n d l e ( l o o p n o d e 1 , e number in g , f i l e h a n d l e ) ; cout < < C r e a t e d a l o o p h a n d l e : <<e n d l < <l o o p h a n d l e 1 > t o S t r i n g ()<< e n d l ; // Create a n o t h e r l o o p h a n d l e w i t h i n a f i l e u s i n g i t s s o u r c e p o s i t i o n i n f o r m a t i o n s t r i n g i n p u t 1 ( ForStatement < p o s i t i o n ,12 > ) ; a b s t r a c t h a n d l e l o o p h a n d l e 2= new a b s t r a c t h a n d l e ( f i l e h a n d l e , i n p u t 1 ) ; cout < < C r e a t e d a l o o p h a n d l e : <<e n d l < <l o o p h a n d l e 2 > t o S t r i n g ()<< e n d l ; // Create y e t a n o t h e r l o o p h a n d l e w i t h i n a l o o p u s i n g i t s r e l a t i v e numbering i n f o r m a t i o n s t r i n g i n p u t 2 ( ForStatement <numbering ,1 > ) ; a b s t r a c t h a n d l e l o o p h a n d l e 3= new a b s t r a c t h a n d l e ( l o o p h a n d l e 1 , i n p u t 2 ) ; <l o o p h a n d l e 3 > t o S t r i n g ()<< e n d l ; cout < < C r e a t e d a l o o p h a n d l e : <<e n d l < return 0 ; }

56

Figure 46.8: Example 3: A test program for simple loops abstract handles.

360

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

2 4 6 8

bash 3 . 0 0 : . / testMyLoop Created a f i l e handle : S o u r c e F i l e <name , f i l e 1 . c> Created a loop handle : S o u r c e F i l e <name , f i l e 1 . c > : : ForStatement <numbering ,1 > Created a loop handle : S o u r c e F i l e <name , f i l e 1 . c > : : ForStatement < p o s i t i o n ,12 > Created a loop handle : S o u r c e F i l e <name , f i l e 1 . c > : : ForStatement <numbering , 1 > : : ForStatement <numbering ,1 >

Figure 46.9: Example 3: Output of the test program for simple loops abstract handles (as strings).

46.5. SUMMARY

361

46.5

Summary

Abstract handles are low level mechanisms to support multiple tools to exchange references to source code. Several examples are used to present the dierent features of abstract handles. Importantly, the specication of abstract handles is tool independent. A reference implementation is provided and is publically available within the ROSE compiler framework. We encourage debate on the pros and cons of this concept to support interoperability of tools which must pass references to source code between them. This work is expected to a small piece of the infrastructure to suport autotuning research.

362

CHAPTER 46. ABSTRACT HANDLES TO LANGUAGE CONSTRUCTS

Chapter 47

ROSE-HPCToolKit Interface
ROSE-HPCToolKit is designed to read in performance data generated by HPCToolkit (and recently GNU gprof) and annotate ROSE AST with performance metrics. It is included in the ROSE distribution and enabled by default if an existing installation of the Gnome XML library, libxml2 (http://xmlsoft.org) can be detected by ROSEs congure script. Or it can be enabled explicitly by specifying the --enable-rosehpct option when running congure. The HPCToolkit (http://www.hipersoft.rice.edu/hpctoolkit) is a set of tools for analyzing the dynamic performance behavior of applications. It includes a tool that instruments a programs binary, in order to observe CPU hardware counters during execution; additional post-processing tools attribute the observed data to statements in the original source code. HPCToolkit stores this data and the source attributions in XML les. In this chapter, we give an overview of simple interfaces in ROSE that can read this data and attach it to the AST. GNU gprof is a basic but easy to use proling tool. It produces an execution prole of applications. gprofs output is a at prole for each function by default, which is not very interesting to us. We use a line-by-line output with full le path information generated by using option -l -L with gprof.

47.1

An HPCToolkit Example Run

Consider the sample source program shown in Figures 47.147.2. This program takes an integer n on the command line, and has a number of loops whose op and memory-operation complexity are either (n) or (n2 ). For this example, we would expect the loop nest at line 56, which has O(n2 ) cost, to be the most expensive loop in the program for large n. Suppose we use the HPCToolkit to prole this program, collecting cycle counts and oatingpoint instructions.1 HPCToolkit will generate one XML le for each metric. A schema specifying the format of these XML les appears in Figure 47.3. In essence, this schema species that the XML le will contain a structured, abstract representation of the program in terms of abstract program entities such as modules, procedures, loops, and statements. Each of these entities may have line number information and a metric value.
1 In

this example, HPCToolkit uses the PAPI to read CPU counters (http://icl.cs.utk.edu/papi).

363

364

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

(Refer to the HPCToolkit documentation for more information.) This schema is always the rst part of an HPCToolkit-generated XML prole data le. We ran HPCToolkit on the program in Figures 47.147.2, and collected cycle and op counter data. The actual XML les storing this data appear in Figures 47.4 and 47.5. By convention, these metrics are named according to their PAPI symbolic name, as shown on line 67 in both Figures 47.4 and 47.5. According to the cycle data on line 90 of Figure 47.4, the most expensive statement in proled.c is line 62 of Figure 47.1, as expected.

47.1. AN HPCTOOLKIT EXAMPLE RUN

365

#include < s t d i o . h> #include < s t d l i b . h> #include < s t r i n g . h> 5 typedef double v a l t ; static val t gen ( s i z e t n ) { v a l t x = ( v a l t ) malloc ( sizeof ( v a l t ) n ) ; i f ( x != NULL) { size t i ; f o r ( i = 0 ; i < n ; i ++) x [ i ] = ( val t )1.0 / n; } return x ; } static val t d o t ( s i z e t n , const v a l t x , const v a l t y ) { size t i ; v a l t sum ; f o r ( i = 0 , sum = 0 . 0 ; i < n ; i ++) sum += x [ i ] y [ i ] ; return sum ; } static s i z e t max ( s i z e t n , const v a l t x ) { size t i ; s i z e t i max ; v a l t v max ; if ( n <= 0 ) return 0 ;

10

15

20

25

30

35

40

45

v max = x [ 0 ] ; i max = 0 ; f o r ( i = 1 ; i < n ; i ++) i f ( x [ i ] > v max ) { v max = x [ i ] ; i max = i ; } return i max ; }

50 s t a t i c void mv ( s i z e t n , const v a l t A, const v a l t x , v a l t y ) { size t j ; memset ( y , 0 , s i z e o f ( v a l t ) n ) ; f o r ( j = 0 ; j < n ; j ++) { const v a l t Ap ; register v a l t xj = x [ j ] ; size t i ; f o r ( i = 0 , Ap = A + j n ; i < n ; i ++, Ap++) y [ i ] += Ap [ 0 ] x j ; } }

55

60

Figure 47.1: proled.c (part 1 of 2): Sample input program, proled using the HPCToolkit.

366

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

70

int main ( i n t a r g c , char a r g v [ ] ) { size t n; val t x; val t y; v a l t A; size t i ; / o u t p u t s / v a l t sum ; s i z e t i max ; v a l t y max ; if { f p r i n t f ( s t d e r r , u s a g e : %s <n>\n , a r g v [ 0 ] ) ; return 1 ; } n = a t o i ( argv [ 1 ] ) ; i f ( n <= 0 ) return 1 ; A = gen ( n n ) ; x = gen ( n ) ; y = gen ( n ) ; if { (A == NULL | | x == NULL | | y == NULL) f p r i n t f ( s t d e r r , Out o f memory \ n ) ; return 1 ; } sum = 0 ; f o r ( i = 0 ; i < n ; i ++) sum += d o t ( n , x , y ) ; mv ( n , A, x , y ) ; i max = max ( n , y ) ; y max = y [ i max ] ; p r i n t f ( %g %l u %g \ n , sum , ( unsigned long ) i max , y max ) ; return 0 ; } ( a r g c != 2 )

75

80

85

90

95

100

105

110 / e o f /

Figure 47.2: proled.c (part 2 of 2): Sample input program, proled using the HPCToolkit.

47.1. AN HPCTOOLKIT EXAMPLE RUN

367

10

15

20

25

30

35

40

45

50

55

60

<? xml version= 1 . 0 ?> < !D O C T Y P E PROFILE [ < ! P r o f i l e : c o r r e l a t e s p r o f i l i n g i n f o w i t h program s t r u c t u r e . > < !E L E M E N T PROFILE (PROFILEHDR, PROFILEPARAMS, PROFILESCOPETREE)> < ! ATTLIST PROFILE version C D A T A# REQUIRED > < !E L E M E N T PROFILEHDR (# P C D A T A)> < !E L E M E N T PROFILEPARAMS (TARGET, METRICS)> < !E L E M E N T TARGET E M P T Y > < ! ATTLIST TARGET name C D A T A# REQUIRED > < !E L E M E N T METRICS (METRIC)+> < !E L E M E N T METRIC E M P T Y > < ! ATTLIST METRIC shortName C D A T A# REQUIRED nativeName C D A T A# REQUIRED period C D A T A# REQUIRED units C D A T A# IMPLIED displayName C D A T A# IMPLIED display ( true | f a l s e ) # IMPLIED > < !E L E M E N T PROFILESCOPETREE (PGM) > < ! This i s e s s e n t i a l l y t h e PGM d t d w i t h M e l e m e n t added . > < !E L E M E N T PGM (G | LM | F |M)+> < ! ATTLIST PGM n C D A T A# REQUIRED > < ! Groups c r e a t e a r b i t r a r y s e t s o f o t h e r e l e m e n t s e x c e p t PGM. > < !E L E M E N T G (G | LM | F | P | L | S |M) > < ! ATTLIST G n C D A T A# IMPLIED > < ! Runtime l o a d modules f o r PGM ( e . g . , DSOs, e x e ) > < !E L E M E N T LM (G | F |M) > < ! ATTLIST LM n C D A T A# REQUIRED > < ! F i l e s c o n t a i n p r o c e d u r e s and s o u r c e l i n e i n f o > < !E L E M E N T F (G | P | L | S |M) > < ! ATTLIST F n C D A T A# REQUIRED > < ! Procedures c o n t a i n s o u r c e l i n e i n f o n : p r o c e s s e d name ; l n : l i n k name > < !E L E M E N T P (G | L | S |M) > < ! ATTLIST P n C D A T A# REQUIRED ln C D A T A# IMPLIED b C D A T A# IMPLIED e C D A T A# IMPLIED > < ! Loops > < !E L E M E N T L (G | L | S |M) > < ! ATTLIST L b C D A T A# IMPLIED e C D A T A# IMPLIED > < ! Statement / Statement range > < !E L E M E N T S (M) > < ! ATTLIST S b C D A T A# REQUIRED e C D A T A# IMPLIED id C D A T A# IMPLIED > < !E L E M E N TME M P T Y > < ! ATTLIST M n C D A T A# REQUIRED v C D A T A# REQUIRED > ]>

Figure 47.3: XML schema for HPCToolkit data les: This schema, prepended to each of the HPCToolkit-generated XML les, describes the format of the proling data. This particular schema was generated by HPCToolkit 1.0.4.

368

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

65

70

75

80

85

90

95

<PROFILE version= 3 . 0 > <PROFILEHDR > </PROFILEHDR > <PROFILEPARAMS > <TARGET name= example /> <METRICS > <METRIC shortName= 0 nativeName=PAPI TOT CYC displayName=PAPI TOT CYC p e r i o d= 32767 u n i t s=PAPI e v e n </METRICS > </PROFILEPARAMS > <PROFILESCOPETREE > < PGM n= example > <LM n= /home/ vuduc2 / p r o j e c t s /ROSE/tmp/ xml / x e r c e s c / h p c t i f / e x a m p l e s / d a t a /01/ example > <F n= . / p r o f i l e d . c > <P n= main > <S b= 15 > < M n= 0 v= 163835 /> </S> <S b= 25 > < M n= 0 v= 65534 /> </S> <S b= 26 > < M n= 0 v= 65534 /> </S> </P> <P n=mv> <S b= 61 > < M n= 0 v= 65534 /> </S> <S b= 62 > < M n= 0 v= 327670 /> </S> </P> </F> </LM > </PGM > </PROFILESCOPETREE > </PROFILE>

Figure 47.4: PAPI TOT CYC.xml: Sample cycle counts observed during proling, generated from running the HPCToolkit on proled.c (Figures 47.147.2.) These lines would appear after the schema shown in Figure 47.3.

47.1. AN HPCTOOLKIT EXAMPLE RUN

369

65

70

75

80

85

90

<PROFILE version= 3 . 0 > <PROFILEHDR > </PROFILEHDR > <PROFILEPARAMS > <TARGET name= example /> <METRICS > <METRIC shortName= 0 nativeName=PAPI FP OPS displayName=PAPI FP OPS p e r i o d= 32767 u n i t s=PAPI e v e n t s /> </METRICS > </PROFILEPARAMS > <PROFILESCOPETREE > < PGM n= example > <LM n= /home/ vuduc2 / p r o j e c t s /ROSE/tmp/ xml / x e r c e s c / h p c t i f / e x a m p l e s / d a t a /01/ example > <F n= . / p r o f i l e d . c > <P n= main > <S b= 25 > < M n= 0 v= 32767 /> </S> <S b= 26 > < M n= 0 v= 98301 /> </S> </P> <P n=mv> <S b= 61 > < M n= 0 v= 131068 /> </S> <S b= 62 > < M n= 0 v= 262136 /> </S> </P> </F> </LM > </PGM > </PROFILESCOPETREE > </PROFILE>

Figure 47.5: PAPI FP OPS.xml: Sample op counts observed during proling, generated from running the HPCToolkit on proled.c (Figures 47.147.2.) These lines would appear after the schema shown in Figure 47.3.

370

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

47.2

Attaching HPCToolkit Data to the ROSE AST

To attach the data of Figures 47.4 and 47.5 to the AST, we augment a basic ROSE translator with two additional calls, as shown in Figure 47.6, lines 4748 and 54. We describe these calls below.

47.2.1

Calling ROSE-HPCT

We must rst include rosehpct/rosehpct.hh, as shown on line 6 of Figure 47.6. All ROSE-HPCT routines and intermediate data structures reside in the RoseHPCT namespace. Next, lines 4748 of Figure 47.6 store the contents of the raw XML le into an intermediate data structure of type RoseHPCT::ProgramTreeList t. The RoseHPCT::loadProlingFiles() routine processes command-line arguments, extracting ROSE-HPCT-specic options that specify the les. We discuss these options in Section 47.4. Line 54 of Figure 47.6, attaches the intermediate prole data structure to the ROSE AST. The RoseHPCT::attachMetrics() routine creates new persistent attributes that store the counter data.2 The attributes are named using the metric name taken from the XML le (see lines 67 of Figures 47.447.5); in this example, the attributes are named PAPI TOT CYC and PAPI FP OPS. Following the conventions of persistent attribute mechanism as described in Chapter 7, the attributes themselves are of type RoseHPCT::MetricAttr, which derives from the AstAttribute type.

47.2.2

Retrieving the attribute values

We retrieve the attribute values as described in Chapter 7. In particular, given a located node with cycle and op attribute values, the printFlopRate() routine dened in lines 1142 of Figure 47.6 prints the source position, AST node type, and estimated ops per cycle. We call printFlopRate() for each expression statement (SgExpressionStmt), for-initializer (SgForInitStatement), and for-statement (SgForStatement) in lines 5966 of Figure 47.6. The output is shown in Figure 47.7. Inspecting the output carefully, you may notice seeming discrepancies between the values shown and the values that appear in the XML les, or other values which seem unintuitive. We explain how these values are derived in Section 47.2.3. This example dumps the AST as a PDF le, as shown on line 68 of Figure 47.6. You can inspect this le to conrm where attributes have been attached. We show an example of such a page in Figure 47.8. This page is the SgExprStatement node representing the sum-accumulate on line 26 of Figure 47.1.

47.2.3

Metric propagation

The example program in Figure 47.6 dumps metric values at each expression statement, forinitializer, and for-statement, but the input XML les in Figure 47.447.5 only attribute the prole data to statements that are not loop constructs. (The <S ...> XML tags refer to statements, intended to be simple non-scoping executable statements; a separate <L ...> tag
2 The last parameter to RoseHPCT::attachMetrics() is a boolean that, when true, enables verbose (debugging) messages to standard error.

47.3. WORKING WITH GNU GPROF

371

would refer to a loop.) Since the XML le species statements only by source line number, RoseHPCT::attachMetrics() attributes measurements to AST nodes in a heuristic way. For example, lines 7880 of Figure 47.4 indicate that all executions of the simple statements of line 25 of the original source (Figure 47.1) accounted for 65534 observed cycles, and that line 26 accounted for an additional 65534 cycles. In the AST, there are multiple statement and expression nodes that occur on line 25; indeed, Figure 47.7 lists 4 such statements. The ROSE-HPCT modules uses a heuristic which only assigns <S ...> metric values to non-scoping nodes derived from SgStatement. When multiple SgStatement nodes occur at a particular source line, ROSE-HPCT simply attaches the metric to each of them. But only one of them will be used for propagating metrics to parent scopes. How is the measurement of 65534 cycles attributed to all of the AST nodes corresponding to line 25 of Figure 47.1? Indeed, line 25 actually contains four dierent SgStatement nodes: an SgForStatement representing the whole loop on lines 2526, an SgForInitStatement (initializer), and two SgExprStatements (one which is a child of the SgForInitStatement, and another for the for-loops test expression). The loops increment is stored in the SgForStatement node as an SgExpression, not an SgStatement. The SgForStatement node is a scoping statement, and so it receives none of the 65534 cycles. Since the increment is not a statement and one of the SgExprStatements is a child of the initializer, this leaves only two direct descendants of the SgForStatementthe initializer and the test expression statementamong which to divide the 65534 cycles. Thus, each receives 32767 cycles. The initializers SgExprStatement child gets the same 32767 as its parent, since the two nodes are equivalent (see rst two cases of Figure 47.7). For the entire loop on lines 2526 of Figure 47.1, the original XML les attribute 65534 cycles to line 25, and another 65534 cycles to line 26 (see Figure 47.4). Moreover, the XML les do not attribute any costs to this loop via an explicit <L ...> tag. Thus, the best we can infer is that the entire for-statements costs is the sum of its immediate child costs; in this case, 131068 cycles. The RoseHPCT::attachMetrics() routine will heuristically accumulate and propagate metrics in this way to assign higher-level scopes approximate costs. The RoseHPCT::attachMetrics() routine automatically propagates metric values through parent scopes. A given metric attribute, RoseHPCT::MetricAttr* x, is derived through propagation if x->isDerived() returns true. In fact, if you call x->toString() to obtain a string representation of the metrics value, two asterisks will be appended to the string as a visual indicator that the metric is derived. We called RoseHPCT::MetricAttr::toString() on lines 27 and 29 of Figure 47.6, and all of the SgForStatement nodes appearing in the output in Figure 47.7 are marked as derived. Alternatively, you cann call RoseHPCT::attachMetricsRaw(), rather than calling RoseHPCT::attachMetrics(). The raw routine takes the same arguments but only attaches the raw data, i.e., without attempting to propagate metric values through parent scopes.

47.3

Working with GNU gprof

ROSE-HPCT can also accept the line-by-line proling output generated by GNU gprof. Currently, we only use the self seconds associated with each line and attach them to ROSE AST as AST attributes named WALLCLK. A typical session to generate compatible gprof proling le for ROSE-HPCT is given below: [liao@codes]$ gcc -g seq-pi.c -pg

372

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

[liao@codes]$ ./a.out [liao@codes]$ gprof -l -L a.out gmon.out &>profile.result -l tells gprof to output line-by-line proling information and -L causes gprof to output full le path information. An excerpt of an output le looks like the following:
Flat profile: Each sample counts as 0.01 seconds. % cumulative self self time seconds seconds calls Ts/call 38.20 8.84 8.84 36.43 17.27 8.43 11.00 19.82 2.54 5.66 21.12 1.31 3.93 22.04 0.91 3.24 22.79 0.75 0.95 23.00 0.22 0.50 23.12 0.12 0.09 23.14 0.02 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00 0.00 23.14 0.00 1 0.00

total Ts/call

0.00 0.00 0.00 0.00

name jacobi (/home/liao6/temp/jacobi.c:193 @ 804899c) jacobi (/home/liao6/temp/jacobi.c:196 @ 8048a3f) jacobi (/home/liao6/temp/jacobi.c:188 @ 804893e) jacobi (/home/liao6/temp/jacobi.c:187 @ 8048968) jacobi (/home/liao6/temp/jacobi.c:197 @ 8048a71) jacobi (/home/liao6/temp/jacobi.c:191 @ 8048a7f) jacobi (/home/liao6/temp/jacobi.c:186 @ 8048976) jacobi (/home/liao6/temp/jacobi.c:187 @ 8048935) jacobi (/home/liao6/temp/jacobi.c:190 @ 8048a94) driver (/home/liao6/temp/jacobi.c:91 @ 8048660) error_check (/home/liao6/temp/jacobi.c:220 @ 8048b7c) initialize (/home/liao6/temp/jacobi.c:116 @ 8048722) jacobi (/home/liao6/temp/jacobi.c:160 @ 8048892)

47.4

Command-line options

The call to RoseHPCT::loadProlingFiles() on line 49 of Figure 47.6 processes and extracts ROSE-HPCT-specic command-line options. To generate the output in this chapter, we invoked Figure 47.6 with the following command-line:

./attachMetrics \ -rose:hpctprof ../../../tutorial/roseHPCT/PAPI_TOT_CYC.xml \ -rose:hpctprof ../../../tutorial/roseHPCT/PAPI_FP_OPS.xml \ -rose:hpcteqpath .=/export/tmp.hudson-rose/hudson/workspace/a90-ROSE-daily-release/label/amd64-linu -c ../../../tutorial/roseHPCT/profiled.c The main option is -rose:hpct:prof <file>, which species the HPCToolkit-generated XML le containing metric data. Here, we use this option twice to specify the names of the cycle and op data les (Figures 47.447.5). To accept gprof output le, please use another option -rose:gprof:linebyline <file>. This option cannot be used with -rose:hpct:prof <file> currently. We need the other option, -rose:hpct:eqpath <A>=<B>, to specify how paths in the HPCToolkit XML les can be mapped to le paths in the ROSE AST. This option allows users to generate performance les on one machine and analyze the results on another machine. In this example, the XML les specify the source le as, ./proled.c (line 73 of Figures 47.4 and 47.5); the eqpath command-line option above remaps the relative path . to an absolute path as it would appear in the ROSE AST. Another example is to use the same performance le even after the original source tree is moved to another location. ROSE-HPCT can still correctly match performance data if the root source paths are given as -rose:hpct:eqpath <oldRootPath>=<newRootPath>.

47.4. COMMAND-LINE OPTIONS

373

Yet another option -rose:hpct:enable debug is provided to display runtime debugging information such as metrics reading, attaching, and propagating. It also adds performance metrics into the ROSE output source le as source comments as shown below. Users can examine the source comments to make sure performance metrics are attached and propagated properly. As we can see, ROSE-HPCT attaches each performance metric to each matching statement. If there are multiple statements showing in the same line, the same metric will be attached to each of them. The metric propagation step will only propagate one of them to upper-level language constructs to ensure the correctness.
/* ROSE-HPCT propagated metrics WALLCLK:18.95[SgForStatement at 0xb7beb218] */ for ( /* ROSE-HPCT raw data: Statement WALLCLK:0.02@File jacobi.c 190-0 -> SgForInitStatement 0x94e8d08 at 190 */ i = 1; /* ROSE-HPCT raw data: Statement WALLCLK:0.02@File jacobi.c 190-0 -> SgExprStatement 0x94516d8 at 190 */ i < (n - 1); i++) /* ROSE-HPCT propagated metrics WALLCLK:18.93[SgForStatement at 0xb7beb29c] */ for ( /* ROSE-HPCT raw data: Statement WALLCLK:0.75@File jacobi.c 191-0 -> SgForInitStatement 0x94e8d38 at 191 */ j = 1; /* ROSE-HPCT raw data: Statement WALLCLK:0.75@File jacobi.c 191-0 -> SgExprStatement 0x9451728 at 191 */ j < (m - 1); j++) /* ROSE-HPCT propagated metrics WALLCLK:18.18[SgBasicBlock at 0x93f60b4] */ { /* ROSE-HPCT raw data: Statement WALLCLK:8.84@File jacobi.c 193-0 -> SgExprStatement 0x9451750 at 193 */ resid = (((((ax * (((( *uold)[i - 1])[j]) + ((( *uold)[i + 1])[j]))) + (ay * (((( *uold)[i])[j - 1]) + ((( *uold)[i])[j + 1])))) + (b * ((( *uold)[i])[j]))) - ((( *f)[i])[j])) / b); /* ROSE-HPCT raw data: Statement WALLCLK:8.43@File jacobi.c 196-0 -> SgExprStatement 0x9451778 at 196 */ (( *u)[i])[j] = (((( *uold)[i])[j]) - (omega * resid)); /* ROSE-HPCT raw data: Statement WALLCLK:0.91@File jacobi.c 197-0 -> SgExprStatement 0x94517a0 at 197 */ error = (error + (resid * resid)); }

374

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

// a t t a c h M e t r i c s . cc Sample t r a n s l a t o r showing how t o a t t a c h // HPCToolkit d a t a t o t h e ROSE AST. #include #include #include #include r o s e . h <i o s t r e a m > <l i s t > < r o s e h p c t / r o s e h p c t . hh>

using namespace s t d ; 10 // P r i n t s t h e e s t i m a t e d f l o p s / c y c l e a t a l o c a t e d node . s t a t i c void p r i n t F l o p R a t e ( const SgNode n ) { const SgLocatedNode n l o c = i s S g L o c a t e d N o d e ( n ) ; if ( n loc && n l o c > a t t r i b u t e E x i s t s ( PAPI TOT CYC ) && n l o c > a t t r i b u t e E x i s t s ( PAPI FP OPS ) ) { // E x t r a c t a t t r i b u t e s . const RoseHPCT : : M e t r i c A t t r c y c l e s a t t r = dynamic cast<const RoseHPCT : : M e t r i c A t t r > ( n > g e t A t t r i b u t e ( PAPI TOT CYC ) ) ; const RoseHPCT : : M e t r i c A t t r f l o p s a t t r = dynamic cast<const RoseHPCT : : M e t r i c A t t r > ( n > g e t A t t r i b u t e ( PAPI FP OPS ) ) ; ROSE ASSERT ( c y c l e s a t t r && f l o p s a t t r ) ; // Get double string double string t h e m e t r i c v a l u e s , as d o u b l e s and s t r i n g s . cycles = cycles attr >g e t V a l u e ( ) ; c y c l e s s = const cast <RoseHPCT : : M e t r i c A t t r > ( c y c l e s a t t r )> t o S t r i n g ( ) ; >g e t V a l u e ( ) ; flops = flops attr f l o p s s = const cast <RoseHPCT : : M e t r i c A t t r > ( f l o p s a t t r )> t o S t r i n g ( ) ;

15

20

25

30

35

40 } } 45

// P r i n t node p o i n t e r / t y p e , parent , e s t i m a t e d f l o p r a t e , and s o u r c e p o s i t i o n . const SgNode n p a r = n l o c >g e t p a r e n t ( ) ; >c l a s s n a m e ( ) << > c o u t << ( const void ) n l o c << : < << n l o c << ( Par= << ( const void ) n p a r << : < << n p a r >c l a s s n a m e ( ) << >) << = ( << f l o p s s << f l o p s ) << / ( << c y c l e s s << cy ) << = << f l o p s / c y c l e s << f l o p s / cy << e n d l << [ << n l o c > g e t s t a r t O f C o n s t r u c t ()> g e t r a w f i l e n a m e ( ) << : << n l o c > g e t s t a r t O f C o n s t r u c t ()> g e t r a w l i n e ( ) << ] << e n d l ;

50

i n t main ( i n t a r g c , char a r g v [ ] ) { v e c t o r < s t r i n g > a r g v L i s t ( argv , a r g v+a r g c ) ; c e r r << [ Loading HPCToolkit o r Gprof p r o f i l i n g d a t a . . . ] << e n d l ; RoseHPCT : : P r o g r a m T r e e L i s t t p r o f i l e s = RoseHPCT : : l o a d P r o f i l i n g F i l e s ( a r g v L i s t ) ; c e r r << [ B u i l d i n g t h e ROSE AST . . . ] << e n d l ; SgProject proj = frontend ( argvList ) ;

55

c e r r << [ A t t a c h i n g m e t r i c s t o t h e AST . . . ] << e n d l ; RoseHPCT : : a t t a c h M e t r i c s ( p r o f i l e s , p r o j , p r o j >g e t v e r b o s e ( ) > 0 ) ; c e r r << [ E s t i m a t i n g f l o p e x e c u t i o n r a t e s . . . ] << e n d l ; typedef R o s e S T L C o n t a i n e r <SgNode > N o d e L i s t t ;

60 N o d e L i s t t e s t m t s = NodeQuery : : querySubTree ( p r o j , V SgExprStatement ) ; f o r e a c h ( e s t m t s . b e g i n ( ) , e s t m t s . end ( ) , p r i n t F l o p R a t e ) ; N o d e L i s t t f o r i n i t s = NodeQuery : : querySubTree ( p r o j , V S g F o r I n i t S t a t e m e n t ) ; f o r e a c h ( f o r i n i t s . b e g i n ( ) , f o r i n i t s . end ( ) , p r i n t F l o p R a t e ) ; N o d e L i s t t f o r s = NodeQuery : : querySubTree ( p r o j , V SgForStatement ) ; f o r e a c h ( f o r s . b e g i n ( ) , f o r s . end ( ) , p r i n t F l o p R a t e ) ; 70 c e r r << [ Dumping a PDF . . . ] << e n d l ; generatePDF ( p r o j ) ; // DQ ( 1 / 2 / 2 0 0 8 ) : This o u t p u t a p p e a r s t o have p r o v i d e d enough s y n c h r o n i z a t i o n i n // t h e o u t p u t t o a l l o w j 3 2 t o work . S i n c e I can t debug t h e problem f u r t h e r f o r // now I w i l l l e a v e i t . c e r r << [ C a l l i n g backend . . . ] << e n d l ; return backend ( p r o j ) ; }

65

75

Figure 47.6: attachMetrics.cc: Sample translator to attach HPCToolkit metrics to the AST.

47.4. COMMAND-LINE OPTIONS

375

Figure 47.7: Sample output, when running attachMetrics.cc (Figure 47.6) with the XML inputs in Figures 47.447.5. Here, we only show the output sent to standard output (i.e., cout and not cerr).

pointer:0x2ba35d14b010 SgVariableDeclaration /export/tmp.hudson-rose/hudson/workspace/a90-ROSE-daily-release/label/amd64-linux/tutorial/roseHPCT/profiled.c 24:3 IsTransformation:0 IsOutputInCodeGeneration:0 Declaration mangled name: _variable_declaration__variable_type_val_td__typedef_declaration_variable_name_L66R__L67R__scope____S Click here to go to the parent node

SgNode* p_parent : 0x2ba35d387560 bool p_isModified : 1 $CLASSNAME* p_freepointer : 0xffffffffffffffff Sg_File_Info* p_startOfConstruct : 0x18ab3f08 Sg_File_Info* p_endOfConstruct : 0x18ab3f50 AttachedPreprocessingInfoType* p_attachedPreprocessingInfoPtr : 0 AstAttributeMechanism* p_attributeMechanism : 0x18d3b3b0 SgLabelRefExp* p_numeric_label : 0 int p_source_sequence_value : 1737 unsigned int p_decl_attributes : 0 std::string p_linkage : Figure 47.8: Sample PDF showing attributes. SgDeclarationModifier p_declarationModifier : <too long>: <too long>: SgDeclarationModifier(isUnknown() = false isDefault() = false isFrie bool p_nameOnly : 1 bool p_forward : 0 bool p_externBrace : 0 bool p_skipElaborateType : 0 SgDeclarationStatement* p_definingDeclaration : 0 SgDeclarationStatement* p_firstNondefiningDeclaration : 0x2ba35d14b010 SgQualifiedNamePtrList p_qualifiedNameList : [] std::string p_binding_label : SgDeclarationStatement* p_baseTypeDefiningDeclaration : 0 SgInitializedNamePtrList p_variables : [0x2ba35d312580] bool p_variableDeclarationContainsBaseTypeDefiningDeclaration : 0 SgDeclarationStatement::template_specialization_enum p_specialization : 1 bool p_requiresGlobalNameQualificationOnType : 0 std::string p_gnu_extension_section : std::string p_gnu_extension_alias : unsigned short p_gnu_extension_initialization_priority : 0 unsigned long p_gnu_extension_alignment : 0

376

CHAPTER 47. ROSE-HPCTOOLKIT INTERFACE

Chapter 48

TAU Instrumentation
Tau is a performance analysis tool from University of Oregon. They have mechanisms for automating instrumentation of the source codes text le directly, but these can be problematic in the present of macros. We present an example of instrumentation combined with code generation to provide a more robust means of instrumentation for source code. This work is preliminary and depends upon two separate mechanisms for the rewrite of the AST (one high level using strings and one low level representing a more direct handling of the AST at the IR level).

48.1

Input For Examples Showing Information using Tau

Figure 48.1 shows the example input used for demonstration of Tau instrumentation.

48.2

Generating the code representing any IR node

Figure 48.2 shows a code that traverses each IR node and for a SgInitializedName of SgStatement output the scope information. The input code is shown in gure 48.2, the output of this code is shown in gure 48.3.

377

378

CHAPTER 48. TAU INSTRUMENTATION

2 4 6 8 10 12 14 16

// #i n c l u d e <math . h> // #i n c l u d e < s t d l i b . h> double f o o ( double x ) { double t h e V a l u e = x ; t h e V a l u e = x ; return t h e V a l u e ; } i n t main ( i n t a r g c , char a r g v [ ] ) { int j , i ; double t S q u a r e d , t ; t = 1.0; tSquared = t t ;

18 20 22 24 26 } i = 1000; f o r ( j =1; j < i ; j += 2 ) { t S q u a r e d += 1 . 0 ; t S q u a r e d += f o o ( 2 . 2 ) ; } return 0 ;

Figure 48.1: Example source code used as input to program in codes used in this chapter.

48.2. GENERATING THE CODE REPRESENTING ANY IR NODE

379

// Demonstration o f i n s t r u m e n t a t i o n u s i n g t h e TAU performance m o n i t o r i n g t o o l s ( U n i v e r s i t y o f Oregon ) 2 #include r o s e . h 4 using namespace s t d ; 6 #d e f i n e NEW FILE INFO S g F i l e I n f o : : g e n e r a t e D e f a u l t F i l e I n f o F o r T r a n s f o r m a t i o n N o d e ( ) 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 MiddleLevelRewrite : : ScopeIdentifierEnum scope = MidLevelCollectionTypedefs : : StatementScope ; M i d d l e L e v e l R e w r i t e : : PlacementPositionEnum l o c a t i o n I n S c o p e = M i d L e v e l C o l l e c t i o n T y p e d e f s : : TopOfCurrentScope ; // Add a TAU i n c l u d e d i r e c t i v e t o t h e t o p o f t h e g l o b a l s c o p e R o s e S T L C o n t a i n e r <SgNode> g l o b a l S c o p e L i s t = NodeQuery : : querySubTree ( p r o j e c t , V SgGlobal ) ; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = g l o b a l S c o p e L i s t . b e g i n ( ) ; i != g l o b a l S c o p e L i s t . end ( ) ; { // Need t o c a s t i from SgNode t o a t l e a s t a SgStatement SgGlobal g l o b a l S c o p e = i s S g G l o b a l ( i ) ; ROSE ASSERT ( g l o b a l S c o p e != NULL ) ; // Output t h e s o u r c e code // generateDOT ( p r o j e c t ) ; file ( as r e p r e s e n t e d by t h e SAGE AST) as a DOT f i l e ( graph ) int main ( i n t a r g c , char a r g v [ ] ) { // This t e s t code t e s t s t h e AST r e w r i t e mechanism t o add TAU I n s t r u m e n t i o n t o t h e AST. SgProject p r o j e c t = fr ontend ( argc , argv ) ; // Output t h e s o u r c e code // generatePDF ( p r o j e c t ) ; file ( as r e p r e s e n t e d by t h e SAGE AST) as a PDF f i l e ( w i t h bookmarks ) } } ROSE ASSERT( r e t u r n C l a s s D e c l a r a t i o n != NULL ) ; return r e t u r n C l a s s D e c l a r a t i o n ; SgClassDeclaration getProfilerClassDeclaration ( SgProject project ) { // Note t h a t i t would be more e l e g a n t t o l o o k t h i s up i n t h e Symbol t a b l e ( do t h i s n e x t ) S g C l a s s D e c l a r a t i o n r e t u r n C l a s s D e c l a r a t i o n = NULL ; R o s e S T L C o n t a i n e r <SgNode> c l a s s D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g C l a s s D e c l a r a t i o n ) ; f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = c l a s s D e c l a r a t i o n L i s t . b e g i n ( ) ; i != c l a s s D e c l a r a t i o n L i s t . end ( ) ; { // Need t o c a s t i from SgNode t o a t l e a s t a SgStatement SgClassDeclaration c l a s s D e c l a r a t i o n = i s S g C l a s s D e c l a r a t i o n ( i ) ; ROSE ASSERT ( c l a s s D e c l a r a t i o n != NULL ) ; // p r i n t f ( In g e t P r o f i l e r C l a s s D e c l a r a t i o n ( ) : if

i ++)

classDeclaration >get name ( ) = %s \ n , c l a s s D e c l a r a t i o n >get name ( ) . s

( classDeclaration >get name ( ) == P r o f i l e r ) returnClassDeclaration = classDeclaration ;

// Allow ROSE t r a n s l a t o r o p t i o n s t o i n f l u e n c e i f we t r a n s f o r m t h e AST i f ( project > g e t s k i p t r a n s f o r m a t i o n ( ) == f a l s e ) { // NOTE: There can be m u l t i p l e f i l e s on t h e command l i n e and each

file

has a g l o b a l s c o p e

52 54 56 58 60 62 64 66

i ++)

// TAU d oe s not seem t o c o m p i l e u s i n g EDG or g++ ( need t o s o r t t h i s o u t w i t h Brian ) // M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e ,# d e f i n e PROFILING ON \ n#i n c l u d e < TAU. h> \ n , scope , l o c a t i o n I n S c o p e ) ; // M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e ,# i n c l u d e < TAU. h> \ n , scope , l o c a t i o n I n S c o p e ) ; M i d d l e L e v e l R e w r i t e : : i n s e r t ( g l o b a l S c o p e , #d e f i n e PROFILING ON 1 \ n#d e f i n e TAU STDCXXLIB 1 \ n#i n c l u d e < TAU. h> } #i f 1 // Now g e t t h e c l a s s d e c l a r a t i o n r e p r e s e n t i n g t h e TAU t y p e w i t h which t o b u i l d v a r i a b l e d e c l a r a t i o n s SgClassDeclaration tauClassDeclaration = getProfilerClassDeclaration ( project ) ; ROSE ASSERT( t a u C l a s s D e c l a r a t i o n != NULL ) ; SgClassType tauType = t a u C l a s s D e c l a r a t i o n >g e t t y p e ( ) ; ROSE ASSERT( tauType != NULL ) ;

68 70 72 74 76 78 80 82

// Get a c o n s t r u c t o r t o use w i t h t h e v a r i a b l e d e c l a r a t i o n ( anyone w i l l due f o r code g e n e r a t i o n ) SgMemberFunctionDeclaration memberFunctionDeclaration = S a g e I n t e r f a c e : : g e t D e f a u l t C o n s t r u c t o r ( t a u C l a s s D e c l a r a t i o ROSE ASSERT( m e m b e r F u n c t i o n D e c l a r a t i o n != NULL ) ;

// Add t h e i n s t r u m e n t a t i o n t o each f u n c t i o n R o s e S T L C o n t a i n e r <SgNode> f u n c t i o n D e c l a r a t i o n L i s t = NodeQuery : : querySubTree ( p r o j e c t , V S g F u n c t i o n D e c l a r a t i o n ) f o r ( R o s e S T L C o n t a i n e r <SgNode > : : i t e r a t o r i = f u n c t i o n D e c l a r a t i o n L i s t . b e g i n ( ) ; i != f u n c t i o n D e c l a r a t i o n L i s t . end { SgFunctionDeclaration f u n c t i o n D e c l a r a t i o n = i sS gFu ncti o nDecl a ra ti o n ( i ) ;

380

CHAPTER 48. TAU INSTRUMENTATION

Test F a i l e d !

Figure 48.3: Output of input code using tauInstrumenter.C

Chapter 49

The Haskell Interface

ROSEs Haskell interface allows the user to analyse and transform the Sage III IR from Haskell, a statically typed pure functional programming language. See http://www.haskell.org/. The interface exposes almost all Sage III APIs to Haskell, allowing the user to call whichever APIs are required. The interface also supports an AST traversal mechanism inspired by Haskells scrap your boilerplate design pattern. The Haskell interface also provides a convenient mechanism for a user to rapidly experiment with the ROSE IR. GHCs command-line interpreter ghci can be used to explore the IR interactively by invoking API methods at will. The Haskell interface relies on the Glasgow Haskell Compiler (GHC). It is auto-congured so long as the GHC binaries are in your $PATH. If not, you will need to supply the path to the binaries at congure time with the option --with-haskell=bindir, where bindir is the path to GHCs bin directory. After installation, ROSE is available as a standard Haskell package named rose. This means that you can supply the ag -package rose to the Haskell compiler in order to make the extension available for use. To understand the usage of the interface, it is crucial to grasp how the concept of monads works in Haskell. For a useful tutorial on monads, the reader is referred to the All About Monads tutorial found at http://www.haskell.org/all about monads/. The simplest Haskell-based ROSE program is the identity translator, whose code is listed in Figure 49.1.
module Main where 2 4 6 8 import ROSE import System main = do p r o j e c t < f r o n t e n d = < < getArgs exitWith = < < backend p r o j e c t

Figure 49.1: Haskell version of identity translator.

381

382

CHAPTER 49. THE HASKELL INTERFACE

49.1

Traversals

As previously mentioned, the traversal mechanism is inspired by the scrap-your-boilerplate pattern. Our implementation of the scrap-your-boilerplate pattern provides both transformation and query traversals. A transformation traversal applies a global transformation to a tree by applying a given function to each tree node, whereas a query traversal derives a result from a tree using a function that produces a result from a node together with a combinator which combines the results from several nodes (for example in a summation query, the combinator may be the addition function). In order to carry out a traversal, two steps are necessary. Firstly one must build a type extension, a type-generic function built from one or more type-specic functions. Secondly one must employ a generic traversal combinator which applies the type extension throughout the program. In our interface type extensions for transformations are built using the functions mkMn, which builds a type extension from a type-specic function, and extMn, which extends an existing type extension with a type-specic function. Likewise mkMqn and extMqn for queries. These functions perform static and dynamic type checking such that they will only call the type-specic functions when it is safe to do so. The two generic traversal combinators are everywhereMc and everythingMc. They take two arguments: the type extension and the tree to be traversed. everywhereMc returns the transformed tree, and everythingMc the result of the query. Tying everything together, Figure 49.2 shows an example of a simple constant folding transformation.

49.2

Further Reading

Reference documentation for the interface is available on ROSEs website at: http://www.rosecompiler.org/ROSE HaskellAPI/

49.2. FURTHER READING

383

module Main where 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 s i m p l i f y op n | n == n u l l S g N o d e = return ( u p S g E x p r e s s i o n n ) | otherwise = do l h s < binaryOpGetLhsOperand n r h s < binaryOpGetRhsOperand n l h s I n t < i s S g I n t V a l l h s r h s I n t < i s S g I n t V a l r h s i f i s J u s t l h s I n t && i s J u s t r h s I n t then do l h s V a l < i n t V a l G e t V a l u e ( fromJust l h s I n t ) r h s V a l < i n t V a l G e t V a l u e ( fromJust r h s I n t ) l e t sum = l h s V a l op r h s V a l f i < s g N u l l F i l e liftM u p S g E x p r e s s i o n ( newIntVal f i sum ( show sum ) ) else return ( u p S g E x p r e s s i o n n ) main : : IO ( ) main = do t i m e 1 < getClockTime p r j < f r o n t e n d = < < getArgs t i m e 2 < getClockTime putStrLn ( Frontend t o o k ++ show ( d i f f C l o c k T i m e s t i m e 2 t i m e 1 ) ) everywhereMc (mkMn simplifyAddOp extMn s i m p l i f y S u b t r a c t O p extMn s i m p l i f y M u l t i p l y O p extMn s i m p l i f y D i v i d e O p ) p r j t i m e 3 < getClockTime putStrLn ( T r a v e r s a l t o o k ++ show ( d i f f C l o c k T i m e s t i m e 3 t i m e 2 ) ) exitWith = < < backend p r j s i m p l i f y M u l t i p l y O p : : SgMultiplyOp ( ) > IO ( S g E x p r e s s i o n ( ) ) simplifyMultiplyOp = simplify () s i m p l i f y D i v i d e O p : : SgDivideOp ( ) > IO ( S g E x p r e s s i o n ( ) ) s i m p l i f y D i v i d e O p = s i m p l i f y div simplifyAddOp : : SgAddOp ( ) > IO ( S g E x p r e s s i o n ( ) ) simplifyAddOp = s i m p l i f y (+) s i m p l i f y S u b t r a c t O p : : SgSubtractOp ( ) > IO ( S g E x p r e s s i o n ( ) ) s i m p l i f y S u b t r a c t O p = s i m p l i f y ( ) import import import import import import import Data . Maybe C o n t r o l . Monad System Data . DataMc ROSE ROSE . Sage3 Time

Figure 49.2: Haskell version of constant folding transformation.

384

CHAPTER 49. THE HASKELL INTERFACE

Part VIII

Parallelism

Topics relevant to shared or distributed parallel computing using ROSE.

385

Chapter 50

Shared-Memory Parallel Traversals

Besides the traversal classes introduced in Chapter 7, ROSE also includes classes to run multithreaded traversals to make use of multi-CPU systems with shared memory (such as typical multicore desktop systems). These shared memory traversals are like the combined traversal classes in that they run several small traversal classes simultaneously; the dierence is that here dierent visit functions may be executed concurrently on dierent CPUs, while the combined traversals always execute visit functions sequentially. Because of this similarity, the public interface for the parallel traversals is a superset of the combined traversal interface. For each Ast*Processing class there is an AstSharedMemoryParallel*Processing class that provides an interface for adding traversals to its internal list, getting a reference to the internal list, and for starting the combined traversal. The traverse() method performs the same combined traversal as in the corresponding AstCombined*Processing class, and the new traverseInParallel() method (with the same signature as traverse()) must be used to start a parallel traversal. (We currently do not provide traverseWithinFileInParallel() and traverseInputFilesInParallel() that would be needed to make the parallel processing classes a fully-featured drop-in replacement for other classes.) A example of how to use the parallel traversal classes is given in Figure 50.1 (note the similarity to Figure 7.24 on page 61). A group of traversal objects is executed rst in combined mode and then in parallel threads. It is the users responsibility to make sure that the actions executed in the parallel traversal are thread-safe. File or terminal I/O may produce unexpected results if several threads attempt to write to the same stream at once. Allocation of dynamic memory (including the use of ROSE or standard C++ library calls that allocate memory) may defeat the purpose of multi-threading as such calls will typically be serialized by the library. Two member functions in each AstSharedMemoryParallel*Processing class are available to tune the performance of the parallel traversals. The rst is void set numberOfThreads(size t threads) which can be used to set the number of parallel threads. The default value for this parameter is 2. Our experiments suggest that even on systems with more than two CPUs, running more than two traversal threads in parallel does not typically increase performance 387

388

CHAPTER 50. SHARED-MEMORY PARALLEL TRAVERSALS

#include < r o s e . h> 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 } i n t main ( i n t a r g c , char a r g v ) { SgProject p r o j e c t = fr ontend ( argc , argv ) ; s t d : : c o u t << combined e x e c u t i o n o f t r a v e r s a l s << s t d : : e n d l ; AstSharedMemoryParallelSimpleProcessing p a r a l l e l T r a v e r s a l ( 5 ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeTr aversa l ( V SgForStatement , f o r l o o p ) ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeTr aversa l ( V SgIntVal , i n t c o n s t a n t ) ) ; p a r a l l e l T r a v e r s a l . a d d T r a v e r s a l ( new NodeTypeTr aversa l ( V S g V a r i a b l e D e c l a r a t i o n , v a r i a b l e parallelTraversal . traverse ( project , preorder ) ; s t d : : c o u t << s t d : : e n d l ; s t d : : c o u t << s h a r e d memory p a r a l l e l e x e c u t i o n o f t r a v e r s a l s << s t d : : e n d l ; parallelTraversal . traverseInParallel ( project , preorder ) ; protected : v i r t u a l void v i s i t ( SgNode node ) { i f ( node >v a r i a n t T ( ) == myVariant ) { s t d : : c o u t << Found << typeName ; i f ( SgLocatedNode l o c = i s S g L o c a t e d N o d e ( node ) ) { S g F i l e I n f o f i = loc >g e t s t a r t O f C o n s t r u c t ( ) ; if ( fi >i s C o m p i l e r G e n e r a t e d ( ) ) { s t d : : c o u t << : c o m p i l e r g e n e r a t e d ; } else { >g e t f i l e n a m e S t r i n g ( ) s t d : : c o u t << : << f i << : << f i >g e t l i n e ( ) ; } } s t d : : c o u t << s t d : : e n d l ; } } private : enum VariantT myVariant ; s t d : : s t r i n g typeName ; }; c l a s s NodeTypeTr aversa l : public A s t S i m p l e P r o c e s s i n g { public : NodeTypeTraversa l ( enum VariantT v a r i a n t , s t d : : s t r i n g typeName ) : myVariant ( v a r i a n t ) , typeName ( typeName ) { }

declaration ));

Figure 50.1: Example source showing the shared-memory parallel execution of traversals. because the memory bandwidth is saturated. The second function is void set synchronizationWindowSize(size t windowSize). This sets a parameter that corresponds to the size of a window of AST nodes that the parallel threads use to synchronize. The value is, in eect, the number of AST nodes that are visited by each thread before synchronizing. Smaller values may in theory result in more locality and therefore better cache utilization at the expense of more time spent waiting for other threads. In practice, synchronization overhead appears to dominate caching eects, so making this parameter too small inhibits performance. The default value is 100000; any large values will result in comparable execution times.

389

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42

combined e x e c u t i o n o f t r a v e r s a l s Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found f o r l o o p : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e E x a m Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found v a r i a b l e d e c l a r a t i o n : c o m p i l e r g e n e r a t e d Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found f o r l o o p : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e E x a m Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e

s h a r e d memory p a r a l l e l e x e c u t i o n o f t r a v e r s a l s Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : c o m p i l e r g e n e r a t e d Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found v a r i a b l e d e c l a r a t i o n : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p Found Found i n t c o n s t a n t f o r l o o p : : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 8 Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found f o r loopFound i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e E x a m p l e T r a v e r s a l s 1 Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e Found i n t c o n s t a n t : / export /tmp . hudson r o s e / hudson / w o r k s p a c e / a90 ROSE d a i l y r e l e a s e / l a b e l /amd64 l i n u x / t u t o r i a l / i n p u t C o d e

Figure 50.2: Output of input le to the shared-memory parallel traversals. Output may be garbled depending on the multi-threaded behavior of the underlying I/O libraries.

390

CHAPTER 50. SHARED-MEMORY PARALLEL TRAVERSALS

Chapter 51

Distributed-Memory Parallel Traversals

ROSE provides an experimental distributed-memory AST traversal mechanism meant for very large scale program analysis. It allows you to distribute expensive program analyses among a distributed-memory system consisting of many processors; this can be a cluster or a network of workstations. Dierent processes in the distributed system will get dierent parts of the AST to analyze: Each process is assigned a number of dening function declarations in the AST, and a method implemented by the user is invoked on each of these. The parts of the AST outside of function denitions are shared among all processes, but there is no guarantee that all function denitions are visible to all processes. The distributed memory analysis framework uses the MPI message passing library for communicating attributes among processes. You will need an implementation of MPI to be able to build and run programs using distributed memory traversals; consult your documentation on how to run MPI programs. (This is often done using a program named mpirun, mpiexecute, or similar.) Distributed memory analyses are performed in three phases: 1. A top-down traversal (the pre-traversal) specied by the user runs on the shared AST outside of function denitions. The inherited attributes this traversal computes for dening function declaration nodes in the AST are saved by the framework for use in the next phase. 2. For every dening function declaration, the user-provided analyzeSubtree() method is invoked; these calls run concurrently, but on dierent function declarations, on all processors. It takes as arguments the AST node for the function declaration and the inherited attribute computed for this node by the pre-traversal. Within analyzeSubtree() any analysis features provided by ROSE can be used. This method returns the value that will be used as the synthesized attribute for this function declaration in the bottom-up traversal (the post-traversal). However, unlike normal bottom-up traversals, the synthesized attribute is not simply copied in memory as the AST is distributed. The user must therefore provide the methods serializeAttribute() and deserializeAttribute(). These compute a serialized 391

392

CHAPTER 51. DISTRIBUTED-MEMORY PARALLEL TRAVERSALS representation of a synthesized attribute, and convert such a representation back to the users synthesized attribute type, respectively. A serialized attribute is a pair of an integer specifying the size of the attribute in bytes and a pointer to a region of memory of that size that will be copied byte by byte across the distributed systems communication network. Attributes from dierent parts of the AST may have dierent sizes. As serialization of attributes will often involve dynamic memory allocation, the user can also implement the deleteSerializedAttribute() method to such dynamic memory after the serialized data has been copied to the communication subsystems internal buer. Within the analyzeSubtree() method the methods numberOfProcesses() and myID() can be called. These return the total number of concurrent processes, and an integer uniquely identifying the currently running process, respectively. The ID ranges from 0 to one less than the number of processes, but has no semantics other than that it is dierent for each process.

3. Finally, a bottom-up traversal is run on the shared AST outside of function denitions. The values returned by the distributed analyzers in the previous phase are used as synthesized attributes for function denition nodes in this traversal. After the bottom-up traversal has nished, the getFinalResults() method can be invoked to obtain the nal synthesized attribute. The isRootProcess() method returns true on exactly one designated process and can be used to perform output, program transformations, or other tasks that are not meant to be run on each processor. Figure 51.1 gives a complete example of how to use the distributed memory analysis framework. It implements a trivial analysis that determines for each function declaration at what depth in the AST it can be found and what its name is. Figure 51.2 shows the output produced by this program when running using four processors on some input les.

393

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82

// This i s a s m a l l example o f how t o use t h e d i s t r i b u t e d memory t r a v e r s a l mechanism . I t computes a l i s t o f f u n c t i o n // d e f i n i t i o n s i n a program and o u t p u t s t h e i r names , t h e i r d e p t h i n t h e AST, and t h e ID o f t h e p r o c e s s t h a t found i t . #include < r o s e . h> #include D i s t r i b u t e d M e m o r y A n a l y s i s . h // The pre t r a v e r s a l runs b e f o r e t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s and i s used t o p r o p a g a t e c o n t e x t i n f o r m a t i o n down // t o t h e i n d i v i d u a l f u n c t i o n d e f i n i t i o n s i n t h e AST. Here , i t j u s t computes t h e d e p t h o f nodes i n t h e AST. c l a s s F u n c t i o n N a m e s P r e T r a v e r s a l : public AstTopDownProcessing <int > { protected : i n t e v a l u a t e I n h e r i t e d A t t r i b u t e ( SgNode , i n t depth ) { return depth + 1 ; } }; // The p o s t t r a v e r s a l runs a f t e r t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s and i s used t o c o l l e c t t h e i n f o r m a t i o n i t // computed . Here , t h e s y n t h e s i z e d a t t r i b u t e s computed by t h e d i s t r i b u t e d a n a l y s i s a r e s t r i n g s r e p r e s e n t i n g i n f o r m a t i o n // a b o u t f u n c t i o n s . These s t r i n g s a r e c o n c a t e n a t e d by t h e p o s t t r a v e r s a l ( and i n t e r l e a v e d w i t h n e w l i n e s where n e c e s s a r y ) . c l a s s F u n c t i o n N a m e s P o s t T r a v e r s a l : public AstBottomUpProcessing <s t d : : s t r i n g > { protected : s t d : : s t r i n g e v a l u a t e S y n t h e s i z e d A t t r i b u t e ( SgNode node , S y n t h e s i z e d A t t r i b u t e s L i s t s y n A t t r i b u t e s ) { std : : s t r i n g r e s u l t = ; SynthesizedAttributesList : : iterator s ; f o r ( s = s y n A t t r i b u t e s . b e g i n ( ) ; s != s y n A t t r i b u t e s . end ( ) ; ++s ) { s t d : : s t r i n g &s t r = s ; r e s u l t += s t r ; i f ( s t r . s i z e ( ) > 0 && s t r [ s t r . s i z e ( ) 1 ] != \ n ) r e s u l t += \ n ; } return r e s u l t ; } std : : s t r i n g defaultSynthesizedAttribute () { return ; } }; // This i s t h e d i s t r i b u t e d p a r t o f t h e a n a l y s i s . The D i s t r i b u t e d M e m o r y T r a v e r s a l b a s e c l a s s i s a t e m p l a t e t a k i n g an // i n h e r i t e d and a s y n t h e s i z e d a t t r i b u t e t y p e as t e m p l a t e p a r am e te r s ; t h e s e a r e t h e same t y p e s used by t h e pre and // p o s t t r a v e r s a l s . c l a s s FunctionNames : public D i s t r i b u t e d M e m o r y T r a v e r s a l <int , s t d : : s t r i n g > { protected : // The a n a l y z e S u b t r e e ( ) method i s c a l l e d f o r e v e r y d e f i n i n g f u n c t i o n d e c l a r a t i o n i n t h e AST. I t s second argument i s t h e // i n h e r i t e d a t t r i b u t e computed f o r t h i s node by t h e pre t r a v e r s a l , t h e v a l u e i t r e t u r n s becomes t h e s y n t h e s i z e d // a t t r i b u t e used by t h e p o s t t r a v e r s a l . s t d : : s t r i n g a n a l y z e S u b t r e e ( S g F u n c t i o n D e c l a r a t i o n f u n c D e c l , i n t depth ) { s t d : : s t r i n g funcName = f u n c D e c l >get name ( ) . s t r ( ) ; std : : stringstream s ; s << p r o c e s s << myID ( ) << : a t depth << depth << : f u n c t i o n << funcName ; return s . s t r ( ) ; } // The u s e r must implement t h i s method t o pack a s y n t h e s i z e d a t t r i b u t e ( a s t r i n g i n t h i s c a s e ) i n t o an a r r a y o f b y t e s // f o r communication . The f i r s t component o f t h e p a i r i s t h e number o f b y t e s i n t h e b u f f e r . s t d : : p a i r <int , void > s e r i a l i z e A t t r i b u t e ( s t d : : s t r i n g a t t r i b u t e ) const { int len = a t t r i b u t e . s i z e ( ) + 1 ; char s t r = s t r d u p ( a t t r i b u t e . c s t r ( ) ) ; return s t d : : m a k e p a i r ( l e n , s t r ) ; } // This method must be implemented t o c o n v e r t t h e s e r i a l i z e d d a t a t o t h e a p p l i c a t i o n s s y n t h e s i z e d s t d : : s t r i n g d e s e r i a l i z e A t t r i b u t e ( s t d : : p a i r <int , void > s e r i a l i z e d A t t r i b u t e ) const { return s t d : : s t r i n g ( ( const char ) s e r i a l i z e d A t t r i b u t e . s e c o n d ) ; } a t t r i b u t e type .

// This method i s o p t i o n a l ( t h e d e f a u l t i m p l e m e n t a t i o n i s empty ) . I t s j o b i s t o f r e e memory t h a t may have been // a l l o c a t e d by t h e s e r i a l i z e A t t r i b u t e ( ) method . void d e l e t e S e r i a l i z e d A t t r i b u t e ( s t d : : p a i r <int , void > s e r i a l i z e d A t t r i b u t e ) const { std : : f r e e ( s e r i a l i z e d A t t r i b u t e . second ) ; } };

394

CHAPTER 51. DISTRIBUTED-MEMORY PARALLEL TRAVERSALS

----- found the following functions: -----process 0: at depth 3: function il process 0: at depth 5: function head process 0: at depth 5: function eq process 1: at depth 3: function headhead process 1: at depth 3: function List process 1: at depth 3: function find process 1: at depth 3: function head process 2: at depth 3: function operator!= process 2: at depth 3: function find process 2: at depth 3: function head process 2: at depth 3: function fib process 3: at depth 3: function xform process 3: at depth 3: function func process 3: at depth 3: function f process 3: at depth 3: function g process 3: at depth 3: function deref -------------------------------------------

Figure 51.2: Example output of a distributed-memory analysis running on four processors.

Chapter 52

Parallel Checker
This Chapter is about the project DistributedMemoryAnalysisCompass, which runs Compass Checkers in Parallel, i.e. shared, combined and distributed.

52.1

Dierent Implementations

The project contains the following les: parallel functionBased ASTBalance contains the original implementation, which is based on an AST traversal that is balanced based on the number of nodes in each function. Then the functions are distributed over all processors. It contains as well the original interfaces to the shared and combined traversal work. parallel le compass distributed on the granularity level of les. parallel functionBased dynamicBalance is the implementation of dynamically scheduling functions across processors. In addition, this algorithm weights the functions rst and then sorts them in descending order according to their weight. parallel compass performs dynamic scheduling based on nodes. The nodes are weighted and then sorted. This algorithm allows the greatest scalability.

52.2

Running through PSUB

The following represents a typical script to run parallel compass on 64 processors using CXX Grammer. CXX Grammar is a binary ROSE AST representation of a previously parsed program. We specify 65 processors because processor 0 does only communication and no computation. Furthermore, we ask for 17 nodes of which each has 8 processors giving us a total of 136 possible processes. We only need 65 but still want to use this conguration. This will average out our 65 processes over 17 nodes, resulting in about 4 processors per node. This trick is used because the AST loaded into memory takes about 400 MB per process. We end up with 1600MB per node. 395

396

CHAPTER 52. PARALLEL CHECKER

#!/bin/bash # Sample LCRM script to be submitted with psub #PSUB -r ncxx65 # sets job name (limit of 7 characters) #PSUB -b nameofbank # sets bank account #PSUB -ln 17 # == defines the amount of nodes needed #PSUB -o ~/log.log #PSUB -e ~/log.err #PSUB -tM 0:05 # Max time 5 min runtime #PSUB -x # export current env var settings #PSUB -nr # do NOT rerun job after system reboot #PSUB -ro # send output log directly to file #PSUB -re # send err log directly to file #PSUB -mb # send email at execution start #PSUB -me # send email at execution finish #PSUB -c zeus #PSUB # no more psub commands # job commands start here set echo echo LCRM job id = $PSUB_JOBID cd ~/new-src/build-rose/projects/DistributedMemoryAnalysisCompass/ srun -n 65 ./parallel_compass -load ~/CXX_Grammar.ast echo "ALL DONE" There are a few tricks that could be considered. Prioritization is based on the amount of time and nodes requested. If less time is specied, it is more likely that a job runs very soon, as processing time becomes available. To submit the job above, use psub le-name. To check the job in the queue, use squeue and to cancel the job use mjobctl -c job-number.

Chapter 53

Reduction Recognition
Figures 53.1 shows a translator which nds the rst loop of a main function and recognizes reduction operations and variables within the loop. A reduction recognition algorithm (ReductionRecognition()) is implemented in the SageInterface namespace and follows the C/C++ reduction restrictions dened in the OpenMP 3.0 specication.
// T e s t r e d u c t i o n r e c o g n i t i o n #i n c l u d e r o s e . h #i n c l u d e < i o s t r e a m > #i n c l u d e < s e t > u s i n g namespace s t d ; i n t main ( i n t a r g c , char a r g v [ ] ) { SgProject p r o j e c t = f r o n t e n d ( argc , argv ) ; // Find main ( ) f u n c t i o n SgFunctionDeclaration func = S a g e I n t e r f a c e : : findMain ( p r o j e c t ) ; ROSE ASSERT ( f u n c != NULL ) ; S g B a s i c B l o c k body = f u n c > g e t d e f i n i t i o n ()> g e t b o d y ( ) ; // Find t h e f i r s t l o o p R o s e S T L C o n t a i n e r <SgNode> n o d e l i s t = NodeQuery : : q u e r y S u b T r e e ( body , V S g F o r S t a t e m e n t ) ; SgForStatement loop = isSgForStatement ( ( n o d e l i s t . begin ( ) ) ) ; ROSE ASSERT ( l o o p != NULL ) ; // C o l l e c t r e d u c t i o n v a r i a b l e s and o p e r a t i o n s std : : set< s t d : : p a i r < S g I n i t i a l i z e d N a m e , Var ia nt T > > r e d u c t i o n s ; std : : set< s t d : : p a i r < S g I n i t i a l i z e d N a m e , Var ia nt T > > : : c o n s t i t e r a t o r S a g e I n t e r f a c e : : ReductionRecognition ( loop , r e d u c t i o n s ) ;

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34

iter ;

// Show t h e r e s u l t s cout< < R e d u c t i o n r e c o g n i t i o n r e s u l t s : < <e n d l ; f o r ( i t e r =r e d u c t i o n s . b e g i n ( ) ; i t e r != r e d u c t i o n s . end ( ) ; i t e r ++) { s t d : : p a i r < S g I n i t i a l i z e d N a m e , Var ia nt T > i t e m = i t e r ; cout< < \ t v a r i a b l e : < <i t e m . f i r s t >u n p a r s e T o S t r i n g ()<< \ t o p e r a t i o n : < <g e t V a r i a n t N a m e ( i t e m . s e c o n d)<< e n d l ; ; } return } backend ( p r o j e c t ) ;

Figure 53.1: Example source code showing reduction recognition. Using this translator we can compile the code shown in gure 53.2. The output is shown in gure 53.3.

397

398

CHAPTER 53. REDUCTION RECOGNITION

2 4 6 8 10 12 14 16

i n t a [ 1 0 0 ] , sum ; int main ( ) { i n t i , sum2 , yy , z z ; sum = 0 ; f o r ( i =0; i < 100; i ++) { i n t xx ; a [ i ]= i ; sum = a [ i ]+ sum ; xx++; yy =0; yy ; z z =a [ i ] ; } return 0 ; }

Figure 53.2: Example source code used as input to loop reduction recognition processor.

Reduction r e c o g n i t i o n r e s u l t s : v a r i a b l e : sum o p e r a t i o n : SgAddOp v a r i a b l e : zz o p e r a t i o n : SgMultAssignOp

Figure 53.3: Output of input to reduction recognition processor.

Part IX

Tutorial Summary

Summary of the ROSE tutorials.

399

Chapter 54

Tutorial Wrap-up
This tutorial has shown the construction and simple manipulation of the AST as part of the construction of the source-to-source translators using ROSE. Much more complex translators are possible using ROSE, but they are not such that they present well as part of a tutorial with short example programs. The remaining chapters of the tutorial include examples of translators built using ROSE as part of active collaborations with external research groups.

FIXME: Reference the User Manual, HTML Doxygen generated documentation, unresolved issues, etc. Reference other work currently using ROSE (ANL, Cornell in the future), academic collaborations.

401

402

CHAPTER 54. TUTORIAL WRAP-UP

Appendix
This appendix includes information useful in supporting the ROSE Tutorial.

54.1

Location of To Do List

This was an older location for the Tutorial Tod List. We now keep the Tod list in the ROSE/docs/testDoxygen/ProjectToDoList.docs in the section called: ROSE Tutorial Todo List.

54.2

Abstract Grammar

In this section we show an abstract grammar for the ROSE AST. The grammar generates the set of all ASTs. On the left hand side of a production we have a non-terminal that corresponds to an inner node of the class hierarchy. On the right hand side of a production we have either one non-terminal or one terminal. The terminal corresponds to a leaf-node where the children of the respective node are listed as double-colon separated pairs, consisting of an access name (= name for get function) and a name that directly corresponds to the class of the child. Details like pointers are hidden. The asterisk shows where lists of children (containers) are used in the ROSE AST. For each terminal, a name followed by ( and ), a variant exists in ROSE with the prex V_ that can be obtained by using the function variantT() on a node. Note, that concrete classes of AST nodes directly correspond to terminals and base classes to non-terminals. START:SgNode SgNode : SgSupport | SgLocatedNode | SgSymbol ; SgSupport : SgName() | SgSymbolTable() | SgInitializedName ( initptr:SgInitializer ) | SgFile ( root:SgGlobal ( declarations:SgDeclarationStatement* ) ) | SgProject ( fileList:SgFile ( root:SgGlobal ( declarations:SgDeclarationStatement* ) ) ) | SgOptions() | SgBaseClass ( base_class:SgClassDeclaration ) | SgTemplateParameter ( expression:SgExpression, defaultExpressionParameter:SgExpression, 403

404

APPENDIX templateDeclaration:SgTemplateDeclaration(), defaultTemplateDeclarationParameter:SgTemplateDeclaration() ) SgTemplateArgument ( expression:SgExpression, templateInstantiation:SgTemplateInstantiationDecl ( definition:SgClassDefinition ) ) SgFunctionParameterTypeList() SgAttribute SgModifier

| | | ;

SgAttribute : SgPragma() | SgBitAttribute ; SgBitAttribute : SgFuncDecl_attr() | SgClassDecl_attr() ; SgModifier : SgModifierNodes() | SgConstVolatileModifier() | SgStorageModifier() | SgAccessModifier() | SgFunctionModifier() | SgUPC_AccessModifier() | SgSpecialFunctionModifier() | SgElaboratedTypeModifier() | SgLinkageModifier() | SgBaseClassModifier() | SgDeclarationModifier() ; SgLocatedNode : SgStatement | SgExpression ; SgStatement : SgExprStatement ( expression_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgLabelStatement() | SgCaseOptionStmt ( key_root:SgExpressionRoot ( operand_i:SgExpression ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgTryStmt ( body:SgBasicBlock ( statements:SgStatement* ), catch_statement_seq_root:SgCatchStatementSeq ( catch_statement_seq:SgStatement* ) ) | SgDefaultOptionStmt ( body:SgBasicBlock ( statements:SgStatement* ) ) | SgBreakStmt() | SgContinueStmt() | SgReturnStmt ( expression_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgGotoStatement()

54.2. ABSTRACT GRAMMAR | | | | | | ; SgSpawnStmt ( the_func_root:SgExpressionRoot ( operand_i:SgExpression ) ) SgForInitStatement ( init_stmt:SgStatement* ) SgCatchStatementSeq ( catch_statement_seq:SgStatement* ) SgClinkageStartStatement() SgDeclarationStatement SgScopeStatement

405

SgDeclarationStatement : SgVariableDeclaration ( variables:SgInitializedName ( initptr:SgInitializer ) ) | SgVariableDefinition ( vardefn:SgInitializedName ( initptr:SgInitializer ), bitfield:SgUnsignedLongVal() ) | SgEnumDeclaration() | SgAsmStmt ( expr_root:SgExpressionRoot ( operand_i:SgExpression ) ) | SgTemplateDeclaration() | SgNamespaceDeclarationStatement ( definition:SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) ) | SgNamespaceAliasDeclarationStatement() | SgUsingDirectiveStatement() | SgUsingDeclarationStatement() | SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ) | SgCtorInitializerList ( ctors:SgInitializedName ( initptr:SgInitializer ) ) | SgPragmaDeclaration ( pragma:SgPragma() ) | SgClassDeclaration | SgFunctionDeclaration ; SgClassDeclaration : SgTemplateInstantiationDecl ( definition:SgClassDefinition ) ; SgFunctionDeclaration : SgTemplateInstantiationFunctionDecl ( parameterList:SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ), definition:SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ) ) | SgMemberFunctionDeclaration ; SgMemberFunctionDeclaration : SgTemplateInstantiationMemberFunctionDecl ( parameterList:SgFunctionParameterList ( args:SgInitializedName ( initptr:SgInitializer ) ), definition:SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ), CtorInitializerList:SgCtorInitializerList

406

APPENDIX

( ctors:SgInitializedName ( initptr:SgInitializer ) ) ; SgScopeStatement : SgGlobal ( declarations:SgDeclarationStatement* ) | SgBasicBlock ( statements:SgStatement* ) | SgIfStmt ( conditional:SgStatement, true_body:SgBasicBlock ( statements:SgStatement* ), false_body:SgBasicBlock ( statements:SgStatement* ) ) | SgForStatement ( for_init_stmt:SgForInitStatement ( init_stmt:SgStatement* ), test_expr_root:SgExpressionRoot ( operand_i:SgExpression ), increment_expr_root:SgExpressionRoot ( operand_i:SgExpression ), loop_body:SgBasicBlock ( statements:SgStatement* ) ) | SgFunctionDefinition ( body:SgBasicBlock ( statements:SgStatement* ) ) | SgWhileStmt ( condition:SgStatement, body:SgBasicBlock ( statements:SgStatement* ) ) | SgDoWhileStmt ( condition:SgStatement, body:SgBasicBlock ( statements:SgStatement* ) ) | SgSwitchStatement ( item_selector_root:SgExpressionRoot ( operand_i:SgExpression ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgCatchOptionStmt ( condition:SgVariableDeclaration ( variables:SgInitializedName ( initptr:SgInitializer ) ), body:SgBasicBlock ( statements:SgStatement* ) ) | SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) | SgClassDefinition ; SgClassDefinition : SgTemplateInstantiationDefn ( members:SgDeclarationStatement* ) ; SgExpression : SgExprListExp ( expressions:SgExpression* ) | SgVarRefExp() | SgClassNameRefExp() | SgFunctionRefExp() | SgMemberFunctionRefExp() | SgFunctionCallExp ( function:SgExpression, args:SgExprListExp ( expressions:SgExpression* ) ) | SgSizeOfOp ( operand_expr:SgExpression ) | SgConditionalExp ( conditional_exp:SgExpression, true_exp:SgExpression, false_exp:SgExpression ) | SgNewExp ( placement_args:SgExprListExp ( expressions:SgExpression* ), constructor_args:SgConstructorInitializer( args:SgExprListExp(expressions:SgExpression*) ), builtin_args:SgExpression ) | SgDeleteExp ( variable:SgExpression ) | SgThisExp()

54.2. ABSTRACT GRAMMAR | | | | | | | | | | ; SgRefExp() SgVarArgStartOp ( lhs_operand:SgExpression, rhs_operand:SgExpression ) SgVarArgOp ( operand_expr:SgExpression ) SgVarArgEndOp ( operand_expr:SgExpression ) SgVarArgCopyOp ( lhs_operand:SgExpression, rhs_operand:SgExpression ) SgVarArgStartOneOperandOp ( operand_expr:SgExpression ) SgInitializer SgValueExp SgBinaryOp SgUnaryOp

407

SgInitializer : SgAggregateInitializer ( initializers:SgExprListExp ( expressions:SgExpression* ) ) | SgConstructorInitializer ( args:SgExprListExp ( expressions:SgExpression* ) ) | SgAssignInitializer ( operand_i:SgExpression ) ; SgValueExp : SgBoolValExp() | SgStringVal() | SgShortVal() | SgCharVal() | SgUnsignedCharVal() | SgWcharVal() | SgUnsignedShortVal() | SgIntVal() | SgEnumVal() | SgUnsignedIntVal() | SgLongIntVal() | SgLongLongIntVal() | SgUnsignedLongLongIntVal() | SgUnsignedLongVal() | SgFloatVal() | SgDoubleVal() | SgLongDoubleVal() ; SgBinaryOp : SgArrowExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgDotExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgDotStarOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgArrowStarOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgEqualityOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgLessThanOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgGreaterThanOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgNotEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) | SgLessOrEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression )

408 | | | | | | | | | | | | | | | | | | | | | | | | | | | | ;

APPENDIX SgGreaterOrEqualOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAddOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgSubtractOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMultiplyOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgDivideOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgIntegerDivideOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgModOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAndOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgOrOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitXorOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitAndOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgBitOrOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgCommaOpExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgLshiftOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgRshiftOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgPntrArrRefExp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgScopeOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgPlusAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMinusAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgAndAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgIorAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgMultAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgDivAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgModAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgXorAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgLshiftAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression ) SgRshiftAssignOp ( lhs_operand_i:SgExpression, rhs_operand_i:SgExpression )

SgUnaryOp : SgExpressionRoot ( operand_i:SgExpression ) | SgMinusOp ( operand_i:SgExpression ) | SgUnaryAddOp ( operand_i:SgExpression ) | SgNotOp ( operand_i:SgExpression ) | SgPointerDerefExp ( operand_i:SgExpression ) | SgAddressOfOp ( operand_i:SgExpression ) | SgMinusMinusOp ( operand_i:SgExpression ) | SgPlusPlusOp ( operand_i:SgExpression ) | SgBitComplementOp ( operand_i:SgExpression ) | SgCastExp ( operand_i:SgExpression ) | SgThrowOp ( operand_i:SgExpression ) ; SgSymbol : SgVariableSymbol() | SgClassSymbol ( declaration:SgClassDeclaration ) | SgTemplateSymbol ( declaration:SgTemplateDeclaration() )

54.2. ABSTRACT GRAMMAR | | | | | SgEnumSymbol ( declaration:SgEnumDeclaration() ) SgEnumFieldSymbol() SgLabelSymbol ( declaration:SgLabelStatement() ) SgDefaultSymbol() SgNamespaceSymbol ( declaration:SgNamespaceDeclarationStatement ( definition:SgNamespaceDefinitionStatement ( declarations:SgDeclarationStatement* ) ) ) | SgFunctionSymbol ;

409

SgFunctionSymbol : SgMemberFunctionSymbol ( declaration:SgFunctionDeclaration ) ; SgPartialFunctionType : SgPartialFunctionModifierType ( ref_to:SgReferenceType, ptr_to:SgPointerType, modifiers:SgModifierNodes(), typedefs:SgTypedefSeq, return_type:SgType, orig_return_type:SgType ) ; This grammar was generated with GRATO, a grammar transformation tool, written by Markus Schordan. The input is a representation generated by ROSETTA. Several other versions of the grammar can be generated as well, such as eliminating nested tree nodes by introducing auxiliary non-terminals, introducing base types as non-terminals etc. Additionally from that grammar we can also generate grammars that can be used with yacc/bison, Coco, and other attribute grammar tools, as well as tree grammar based tools such as burg (requires a transformation to a binary tree).

410

APPENDIX

Glossary
We dene terms used in the ROSE manual which might otherwise be unclear. AST Abstract Syntax Tree. A very basic understanding of an AST is the entry level into ROSE. Attribute User dened information (objects) associated with IR nodes. Forms of attributes include: accumulator, inherited, persistent, and synthesized. Both inherited and synthesized attributes are managed automatically on the stack within a traversal. Accumulator attributes are typically something semantically equivalent to a global variable (often a static data member of a class). Persistent attributes are explicitly added to the AST and are managed directly by the user. As a result, they can persist across multiple traversals of the AST. Persistent attributes are also saved in the binary le I/O, but only if the user provides the attribute specic pack() and unpack() virtual member functions. See the ROSE User Manual for more information, and the ROSE Tutorial for examples. CFG As used in ROSE, this is the Control Flow Graph, not Context Free Grammar or anything else. EDG Edison Design Group (the commercial company that produces the C and C++ front-end that is used in ROSE). IR Intermediate Representation (IR). The IR is the set of classes dened within SAGE III that allow an AST to be built to dene any application in C, C++, and Fortran application. Query (as in AST Query) Operations on the AST that return answers to questions posed about the content or context in the AST. ROSE A project that covers both research in optimization and a specic infrastructure for handling large scale C, C++, and Fortran applications. Rosetta A tool (written by the ROSE team) used within ROSE to automate the generation of the SAGE III IR. SAGE++ and SAGE II An older object-oriented IR upon which the API of SAGE III IR is based. Semantic Information What abstractions mean (short answer). (This might be better as a description of what kind of semantic information ROSE could take advantage, not a denition.) 411
FIXME: Dene the following terms: IR node, Inherited Attribute, Synthesized Attribute, Accumulator Attribute, AST Traversal

412

GLOSSARY Telescoping Languages A research area that denes a process to generate domainspecic languages from a general purpose languages. Transformation The process of automating the editing (either reconguration, addition, or deletion; or some combination) of input application parts to build a new application. In the context of ROSE, all transformations are source-to-source. Translator An executable program (in our context built using ROSE) that performs source-to-source translation on an existing input application source to generate a second (generated) source code le. The second (generated) source code is then typically provided as input to a vendor provided compiler (which generates object code or an executable program). Traversal The process of operating on the AST in some order (usually pre-order, postorder, out of order [randomly], depending on the traversal that is used). The ROSE user builds a traversal from base classes that do the traversal and execute a function, or a number of functions, provided by the user.