AI

Week1_Tensorflow_Fundamentals_HW http://localhost:8888/notebooks/Desktop/Ai/Inte...
Class 01 - TensorFlow Fundamentals
Some Setup
In [ ]: import random
import tensorflow as tf
import numpy as np
Computation Graphs
The primary construct in the TensorFlow API is the "computation graph", or just "graph" as we'll be calling
it in this class. A computation graph is a useful way to illustrate the flow of data through multiple
operations. Here's a basic example showing the graph for adding two numbers together:
The arrows represent data flowing through through the graph (in this case the numbers 7, 3, and 10) while
the nodes represent some form of computation (in this case addition).
We can chain multiple of these computations together to form a more complex transformation of the data:
Here, the numbers 3 and 7 are sent to an addition operation and a multiplication operation. The outputs of
both of these are then sent to a subtraction operation. There are a few ways to illustrate the above
operations.
We could show this as a one-liner math equation, using parentheses to show the order of operations:
output = (3 + 7) − (3 × 7) = −11
We could also define the separate operations as mathematical functions:
a(x, y) = x + y
b(x, y) = x × y
c(x, y) = a(x, y) − b(x, y)
c(3, 7) = a(3, 7) − b(3, 7) = −11

Programmatically, the previous equations could be created with something like this:
1 of 23 3/10/18, 6:49 PM
In [ ]: def add(x, y):

return x + y
def multiply(x, y):

return x * ya=
def subtract(x, y):

return x - y
x = 3
y = 7
a = add(x, y)
b = multiply(x, y)
c = subtract(a, b)
The data isn't just one and done, either; it can be reused multiple times throughout the computation:
This graph is similar to the previous model, but now we're reusing the 3 by adding it back in at the end.
The code might look like this:
In [ ]: x = 3
y = 7
a = add(x, y)
b = multiply(x, y)
c = subtract(a, b)
d = add(c, x)
My First TensorFlow Graph
Fundamental TensorFlow work flow
This pattern will be used again and again throughout the course. When working with TensorFlow, your
code will effectively be divided into two parts:
1. Define a graph which contains your model

2. Run the graph. Two special cases are:
Train the model
Test/predict using the model
Step 1: Define a computation graph
2 of 23 3/10/18, 6:49 PM
In [ ]: # tf.placeholder creates an "input" node

# we MUST give it value when we run our model
# these can be data we want to learn from or
# values of hyper-parameters for our model
a = tf.placeholder(tf.int32, name="input_a")
b = tf.placeholder(tf.int32, name="input_b")
# tf.add creates an addition node

c = tf.add(a, b, name="add")
# tf.multiply creates a multiplication node

d = tf.multiply(a, b, name="multiply")
# Add up the results of the previous two nodes

out = tf.add(c, d, name="output")
Step 2: Run the graph
3 of 23 3/10/18, 6:49 PM
In [ ]: # Start a session

sess = tf.Session()
# Create a "feed_dict" dictionary to define input values

# Keys to dictionary are handles to our placeholders
# Values to dictionary are values we'd like to feed in
feed_dict = { a: 4, b: 3 }
# Execute the graph using `sess.run()`, which takes two parameters:

# - `fetches` lists which node(s) we'd like to receive as output
# - `feed_dict` feeds in key-value pairs to input to various nodes
# In this case, we pass in the Tensor òut` as our value for `fetches`,
# which causes the value for out to be computed and returned
result = sess.run(out, feed_dict=feed_dict)
# Print the value of òut`

print("({0}*{1}) + ({0}+{1}) = {2!s}".format(feed_dict[a], feed_dict[b], result
# Close the session

sess.close()
Jupyter Notebook
Week1_Tensorflow_Fundamentals_HW Current Kernel Logo
Python 3
File
Edit
View
Insert
Cell
Kernel
Widgets
Help

Some Setup
import random
import numpy as np
# Computation Graphs
The primary construct in the TensorFlow API is the "computation graph", or just
The arrows represent data flowing through through the graph (in this case the
We can chain multiple of these computations together to form a more complex
Here, the numbers 3 and 7 are sent to an addition operation and a multiplication
We could show this as a one-liner math equation, using parentheses to show
output=(3+7)−(3×7)=−11
output=(3+7)−(3×7)=−11
a(x,y)b(x,y)c(x,y)c(3,7)=x+y=x×y=a(x,y)−b(x,y)=a(3,7)−b(3,7)=−11
a(x,y)=x+yb(x,y)=x×yc(x,y)=a(x,y)−b(x,y)c(3,7)=a(3,7)−b(3,7)=−11
Programmatically, the previous equations could be created with something
def add(x, y):
return x + y
4 of 23 3/10/18, 6:49 PM
TensorFlow Core API

Our main TensorFlow objects are:
tf.Tensor
tf.Operation
tf.Graph
tf.Session
tf.Variable
We will introduce each of these below.
Tensor Objects
What is a Tensor?
Tensors, for our purposes, are n-dimensional matrices (or tables of numbers).
a 0-dimensional tensor is a single number (or scalar)

a 1-dimensional tensor is a vector, and
a 2-dimensional tensor is a standard matrix.
Higher dimensional tensors are simply referred to as an n-D tensor. Every value that is passed through a
TensorFlow model is a Tensor object - the TensorFlow representation of a tensor.
Defining tensors by hand
You can define Tensor object values in two main ways:
1. Native Python types

2. NumPy arrays (recommended)
Both of these can be automatically converted into TensorFlow Tensor objects.
Tensors from Native Python
5 of 23 3/10/18, 6:49 PM
In [1]: # 0-D tensor (scalar)

t_0d_py = 4
# 1-D tensor (vector)

t_1d_py = [1, 2, 3]
# 2-D tensor (matrix)

t_2d_py = [[1, 2],
[3, 4],
[5, 6]]
# 3-D tensor
t_3d_py = [[[0, 0], [0, 1], [0, 2]],
[[1, 0], [1, 1], [1, 2]],
[[2, 0], [2, 1], [2, 2]]]
python_defined_tensors = [t_0d_py, t_1d_py, t_2d_py, t_3d_py]
# tf.constant creates a tf.Tensor from a fixed value

# you can read more here:
# https://www.tensorflow.org/api_docs/python/tf/constant
for pdt in python_defined_tensors:
print(tf.constant(pdt))
--------------------------------------------------------------------------
-
NameError Traceback (most recent call last
)
<ipython-input-1-09fd3ae24959> in <module>()
21 # https://www.tensorflow.org/api_docs/python/tf/constant (https:/
/www.tensorflow.org/api_docs/python/tf/constant)
22 for pdt in python_defined_tensors:
---> 23 print(tf.constant(pdt))
NameError: name 'tf' is not defined
Interestingly, we haven't run this constant tensor yet. We've only defined it. And, it knows a little bit about
itself: its shape. We'll get into these details in just a bit.
NumPy Arrays
Pretty much the same as native Python, but with the numpy.array function wrapping it:
6 of 23 3/10/18, 6:49 PM
In [2]: # 0-D tensor (scalar)

t_0d_np = np.array(4, dtype=np.int32)
# 1-D tensor (vector)

t_1d_np = np.array([1, 2, 3], dtype=np.int64)
# 2-D tensor (matrix)

t_2d_np = np.array([[1, 2],
[3, 4],
[5, 6]],
dtype=np.float32)
# 3-D tensor
t_3d_np = np.array([[[0, 0], [0, 1], [0, 2]],
[[1, 0], [1, 1], [1, 2]],
[[2, 0], [2, 1], [2, 2]]],
dtype=np.int32)
numpy_defined_tensors = [t_0d_np, t_1d_np, t_2d_np, t_3d_np]
for ndt in numpy_defined_tensors:

print(tf.constant(ndt))
--------------------------------------------------------------------------
-
)
<ipython-input-2-ab99c7d90d7c> in <module>()
1 # 0-D tensor (scalar)
----> 2 t_0d_np = np.array(4, dtype=np.int32)
3
4 # 1-D tensor (vector)
5 t_1d_np = np.array([1, 2, 3], dtype=np.int64)
NameError: name 'np' is not defined
7 of 23 3/10/18, 6:49 PM
Data Types
In general, using np.array (or np.asarray ) is the recommended way of defining values for tensors
by hand in TensorFlow. The primary reason for this is that you can specify the exact data type ( dtype )
you'd like the values to be represented with. For example, there's no way to specify a 32-bit integer vs a
64-bit integer with native Python. TensorFlow is tightly integrated with NumPy, and most TensorFlow data
types have a corresponding NumPy dtype :
TensorFlow Equivalent NumPy

Description
type type
tf.float32 np.float32 32 bit floating point.
tf.float64 np.float64 64 bit floating point.
tf.int8 np.int8 8 bit signed integer.
tf.uint8 np.uint8 8 bit unsigned integer.
tf.string N/A String type, as byte array
tf.bool np.bool Boolean.
Complex number made of two 32 bit floating point numbers: real and
tf.complex64 np.complex64
imaginary parts.
tf.qint8 N/A 8 bit signed integer used in quantized Ops.
tf.qint32 N/A 32 bit signed integer used in quantized Ops.
tf.quint8 N/A 8 bit unsigned integer used in quantized Ops.
Slightly modified version of this table (https://www.tensorflow.org/versions/r0.12/resources

/dims_types#data_types).
In [ ]: # Just to show that they are equivalent

(tf.float32 == np.float32 and
tf.float64 == np.float64 and
tf.int8 == np.int8 and
tf.uint8 == np.uint8 and
tf.bool == np.bool and
tf.complex64 == np.complex64)
The primary exception to when you should not use np.array() is when defining a Tensor of
strings. When using strings, just use standard Python lists. It's best practice to include the b prefix in
front of strings to explicitly define the strings as byte-arrays:
In [ ]: tf_string_tensor = [b"first", b"second", b"third"]
Tensor Shapes
A common term in TensorFlow is a Tensor object's "shape". A shape value is a list or tuple containing
an ordered set of integers. The i-th element in the list describes the length of the i-th dimension in the
tensor, while the number of elements in the list defines the dimensionality of the tensor. Here are some
examples:
8 of 23 3/10/18, 6:49 PM
In [ ]: # Shapes corresponding to scalars

# Note that either lists or tuples can be used
s_0d_list = []
s_0d_tuple = ()
# Shape corresponding to a vector of length 3

s_1d = [3]
# Shape corresponding to a 2-by-3 matrix

s_2d = (2, 3)
# Shape corresponding to a 4-by-4-by-4 cube tensor

s_3d = [4, 4, 4]
You can use the tf.shape Operation to get the shape value of Tensor objects:
In [1]: # note, np.ndarray.reshape takes its args "flattened" out

# (not in a list or tuple)
arr = np.arange(24).reshape(2,3,4)
print("In NumPy:", arr.shape,arr,sep="\n")
# tf.shape creates an Operation that returns a Tensor

# the returned Tensor is the shape of arr
shape_op = tf.shape(arr)
print("In TensorFlow:", shape_op, sep="\n")
shape = tf.Session().run(shape_op)
print("Shape of tensor: " + str(shape))
--------------------------------------------------------------------------
-
)
<ipython-input-1-9bce9c209600> in <module>()
1 # note, np.ndarray.reshape takes its args "flattened" out
2 # (not in a list or tuple)
----> 3 arr = np.arange(24).reshape(2,3,4)
4 print("In NumPy:", arr.shape,arr,sep="\n")
5
NameError: name 'np' is not defined
As mentioned, defining an Operation doesn't execute it. To execute it, we have to run it. We do that
with a helper tf.Session . More to come below! Quick quiz: is shape_op a 3D tensor or 1D tensor?
TensorFlow Operation Objects
TensorFlow Operation objects (commonly abbreviated as "Ops" in the TensorFlow documentation)

are nodes that perform compuation on or with Tensor objects. They take as input zero or more Tensor
objects (or objects that can be converted into tensors- see the previous section), and output zero or more
tensors. These outputs can then either be returned to the client or passed on to further Operations.
Operations are the fundamental building blocks of any TensorFlow graph- their calculations represent
nodes, and data flowing from one to the next represents edges.
We've already seen a few Operation examples: tf.add and tf.multiply are classic
examples: they both take in two tensors and output one. When given non-scalar values, they do
addition/multiplication element-wise. Also, tf.constant and tf.shape are Operation s.
9 of 23 3/10/18, 6:49 PM
In [ ]: # Initialize some arrays to be fed into tf.add as Tensors

a = np.array([1, 2], dtype=np.int32)
b = np.array([3, 4], dtype=np.int32)
# tf.add creates an "add" Operation and places it in the graph

# The variable c will be a handle to the output of the operation
# This output can be passed on to other Operations!
c = tf.add(a, b)
The important thing to remember is that Operations do not execute when

created - that's the reason `tf.add([1, 2],[3, 4])` doesn't return the
value `[4, 6]` immediately. It must be passed into a `Session.run()`
method, which we'll cover in more detail below.
In [ ]: result = tf.Session().run(c)

print(result)
10 of 23 3/10/18, 6:49 PM
In [1]: # or, since we're in a notebook, we can just evaluate the result
# (without explicitly needing to print it out)
# note: that the returned value is a numpy array
tf.Session().run(c)
Jupyter Notebook
Week1_Tensorflow_Fundamentals_HW Current Kernel Logo
Python 3
File
Edit
View
Insert
Cell
Kernel
Widgets
Help

Some Setup
import random
import numpy as np
Computation Graphs
The primary construct in the TensorFlow API is the "computation graph", or just
The arrows represent data flowing through through the graph (in this case the
We can chain multiple of these computations together to form a more complex
Here, the numbers 3 and 7 are sent to an addition operation and a multiplication
We could show this as a one-liner math equation, using parentheses to show
output=(3+7)−(3×7)=−11
output=(3+7)−(3×7)=−11
a(x,y)b(x,y)c(x,y)c(3,7)=x+y=x×y=a(x,y)−b(x,y)=a(3,7)−b(3,7)=−11
a(x,y)=x+yb(x,y)=x×yc(x,y)=a(x,y)−b(x,y)c(3,7)=a(3,7)−b(3,7)=−11
Programmatically, the previous equations could be created with something
def add(x, y):
return x + y
def multiply(x, y):
return x * ya=
def subtract(x, y):
return x - y
x = 3
11 of 23 3/10/18, 6:49 PM
The majority of the TensorFlow API is Operations. In addition to Operation-specific inputs, each Operation
can take in a name parameter, which can help identify Operations in TensorBoard and other tools.
In [ ]: c = tf.add(a, b, name="my_add_operation")
Getting into the habit of adding names to your Operations now will save you headaches later on.
TensorFlow Graph Objects
When TensorFlow is imported into Python, it automatically creates a Graph object and makes it the
default graph. You can create more graphs as well:
In [ ]: # Create a new graph - constructor takes no parameters

new_graph = tf.Graph()
However, operations (such as tf.add and tf.multiply ) are added to the default graph when
created. To add operations to your new graph, use a with statement along with the graph's
as_default() method. This makes that graph the default while inside of the with block:
In [ ]: with new_graph.as_default():

a = tf.add(3, 4)
b = tf.multiply(a, 2)
The default graph, other than being set to the default, is no different than any other Graph . If you need
to get a handle to the default graph, use the tf.get_default_graph function:
default_graph = tf.get_default_graph()
Note: get_default_graph() will return whatever graph is set to the default, so if you are inside of a
with g.as_default() block, get_default_graph() will return g :
In [ ]: with new_graph.as_default():

print(new_graph is tf.get_default_graph())
Most TensorFlow models will not require more than one graph per script. So, often, you can simply use the
default, implicit graph and be done. However, you may find multiple Graph instances useful when
defining two independent models side-by-side. Additionally, there are mechanisms to export and import
external models and load them in as Graph objects, which can allow you to feed the output of existing
models into your new model (or vice versa). We won't be able to demonstrate these now, but see the
TensorFlow API for more info:
Graph.as_graph_def (https://www.tensorflow.org/versions/r0.12/api_docs/python/framework
/core_graph_data_structures#Graph.as_graph_def)
tf.import_graph_def (https://www.tensorflow.org/api_docs/python/tf/import_graph_def)
TensorFlow Session
12 of 23 3/10/18, 6:49 PM
Creating Sessions
As we saw earlier, Session objects are used to launch and execute graphs. Earlier, we created a
session using its default constructor, but it has three optional parameters:
target specifies the execution engine to use. By default it is the empty string, which causes the
Session to use the standard local execution context. Typically, this parameter is only used when using
TensorFlow in a distributed setting
graph specifies which Graph object the session should run. The default value is None , which
causes the Session to load in the default graph. Sessions only manage one graph at a time, so
executing more than one graph will require more than one session
config allows users to specify advanced options to configure the session. We won't cover this
today, but some things that are available are: limiting the number of CPUs/GPUs used, logging
options, and changing optimization of the graph
In [ ]: # A session with the default graph launched

# Equivalent to: tf.Session(graph=tf.get_default_graph())
sess_default = tf.Session()
# A session with new_graph launched

new_graph = tf.Graph()
sess_new = tf.Session(graph=new_graph)
Running Sessions
The most important method of a Session is its run() function. Earlier in this notebook, we saw basic
usage of the two primary parameters to run() : fetches and feed_dict .
Retrieving information: fetches
fetches expects a list of Tensor and/or Operation handles (or just a single
Tensor / Operation ). The list specifies what computations we would like TensorFlow to run, as well
as what we'd like run() to output:
In [ ]: # equiv: tf.Session().run(tf.add(3,2))

sess_default.run(tf.add(3,2))
TensorFlow will only perform calculations necessary to compute the values specified in fetches , so it
won't waste time if you only need to run a small part of a large, complicated graph.
Sending information: feed_dict
feed_dict is an optional parameter to run , but it becomes required when placeholder nodes are
included. We saw it used to feed input data to placeholders, but feed_dict can actually send values to
any node. The keys to the dictionary should be handles to Tensor objects (usually outputs of
Operations), and the values should be replacement data:
13 of 23 3/10/18, 6:49 PM
In [3]: import tensorflow as tf

import numpy as np
# Create Operations, Tensors, etc (using the default graph)
a = tf.add(3, 4)
b = tf.multiply(a, 5)
# Define a dictionary that says to replace the value of à` with 15

replacers = {a: 15}
# Run the session without feed_dict

# Prints (3 + 4) * 5 = 35
print(sess_default.run(b))
# Run the session, passing in `replace_dict` as the value to `feed_dict`

# Prints 15 * 5 = 75 instead of 7 * 5 = 35
print(sess_default.run(b, feed_dict=replacers))
--------------------------------------------------------------------------
-
)
<ipython-input-3-c8dad27a115f> in <module>()
10 # Run the session without feed_dict
11 # Prints (3 + 4) * 5 = 35
---> 12 print(sess_default.run(b))
13
14 # Run the session, passing in `replace_dict` as the value to `feed
_dict`
NameError: name 'sess_default' is not defined
When using placeholders,TensorFlow insists that any calls to Session.run() include feed_dict
values for all placeholders:
In [ ]: a = tf.placeholder(tf.int32, name="my_placeholder")

b = tf.add(a, 3)
# This raises an error (with a LONG error message, so we

# use try-except to catch it and print out just a portion)
try:
sess_default.run(b)
except tf.errors.InvalidArgumentError as e:
print("The error:\n", e.message)
In [ ]: a = tf.placeholder(tf.int32, name="my_placeholder")

b = tf.add(a, 3)
# Create feed dictionary

feed_dict = {a: 8}
# Now it works!
print(sess_default.run(b, feed_dict=feed_dict))
In [ ]: # Closing out the Sessions we opened up

sess_default.close()
sess_new.close()
TensorFlow Variable Objects

The last fundamental TensorFlow class is the Variable . A TensorFlow Variable has persistent
state across multiple calls to Session.run() , which means that learned parameters in machine
learning models are Variables. We can create a Variable with a starting value of 0 like so:
14 of 23 3/10/18, 6:49 PM
In [ ]: my_var = tf.Variable(0, name="my_var")
However, even though the object has been created, the value of the Variable has to be initialized
separately with either of the tf.variables_initializer() or, more commonly,
tf.global_variables_initializer() Operations. Remember that Operations must be passed
into Session.run() to be executed:
In [ ]: sess = tf.Session()

init = tf.global_variables_initializer()
sess.run(init)
Having value initialization separated from object creation allows us to reinitialize the variable later if we'd
like.
Note: tf.global_variables_initializer() used to be named

tf.initialize_all_variables() , and tf.variables_initializer() used to be called
tf.initialize_variables() . These were renamed just before version 1.0.0 of TensorFlow, so if
you follow older tutorials, you may need to update these functions.
Now that the Variable is initialized, we can tweak it's value! Let's do some basic incrementing with the
Variable.assign() method:
In [ ]: increment = my_var.assign(my_var + 1)
for i in range(10):
print(sess.run(increment), end=" ")
You may notice that if you run the previous code multiple times in the notebook (i.e., rerun the prior cell a
few times in a row), the value persists and continues to climb. The Variable's state is maintained by the
Session object, and the state will persist unless either the session is close, the Variable is re-initialized, or
a new value is assigned to the Variable.
In [ ]: # Re-initialize variables

sess.run(init)
# Start incrementing, beginning from 0 again

for i in range(10):
print(sess.run(increment), end=" ")
Trainable Variables
There are several optional parameters in the Variable constructor, but one to pay close attention to is
trainable . It takes in a boolean value, which defaults to True , and specifies to TensorFlow whether
the built-in optimization functions (which we will cover in a separate notebook) should affect this
Variable . If a Variable in you model should not be adjusted during gradient descent, make
sure to set its trainable parameter to False
Exercise 1
15 of 23 3/10/18, 6:49 PM
Part 1
Create a TensorFlow Graph that is based on the following image:
Use TensorFlow's API guide for math operations (https://www.tensorflow.org/api_guides/python

/math_ops) to find Operations you don't know. You'll want to look in the section called "Reduction". If you
are confused about how to give inputs to a reduction Operation, don't get too fancy. Remember, a Python
list (of Python lists/scalars or of Tensor s) can be used to create another input Tensor .
Part 2
Once you've created the graph, use a Session to run the graph and confirm the following input/output
pairs:
In Out
1, 2, 3 14
-1, -2, 3 2
123, 456, 789 44669304
Part 3
Finally, use tf.summary.FileWriter to output the Graph to disk and double check that your
model resembles the above image in TensorBoard (the image reads from left to right, TensorBoard
displays them from bottom to top.
Solution 1
Part 1
In [ ]:
Part 2
In [ ]:
TensorBoard
16 of 23 3/10/18, 6:49 PM
TensorBoard (https://www.tensorflow.org/versions/r0.11/how_tos/summaries_and_tensorboard/index.html)
is TensorFlow's built-in visualization software. With it, users can examine the error rate of models over
time, debug their graphs, examine summary statistics, and compare various models with one another.
Basic steps to use TensorBoard
Without context, graphs that use TensorBoard summaries can seem cluttered and confusing. However,
they are actually quite straightforward once you understand what each node is doing. Here are the basic
steps you'll most commonly see:
1. Pass the result of a Tensor you'd like to analyze to one of several summary operations
(https://www.tensorflow.org/api_guides/python/summary).
2. Group all of your summaries together using tf.summary.merge_all()
(https://www.tensorflow.org/api_docs/python/tf/summary/merge_all) for convenience.
3. Create a tf.summary.FileWriter (https://www.tensorflow.org/api_docs/python/tf/summary
/FileWriter), which is responsible for writing summaries to disk.
4. While training/evaluating, pass in the output of tf.summary.merge_all() to the FileWriter
5. Once finished, start up TensorBoard in the terminal by using the built-in tensorboard command
Let's start with a simple graph to get us going. The graph below simply takes in two placeholder
inputs and performs addition, subtraction, multiplication, and division with them:
In [ ]: # Explicitly create the graph we're going to use

graph = tf.Graph()
with graph.as_default():
# Input placeholders
a = tf.placeholder(tf.float32, name="a")
b = tf.placeholder(tf.float32, name="b")
# Perform +, -, *, and / operations

plus = tf.add(a, b, name="a_plus_b")
minus = tf.subtract(a, b, name="a_minus_b")
times = tf.multiply(a, b, name="a_times_b")
divided = tf.divide(a, b, name="a_divided_by_b")
# Global step counter and its increment operation

global_step = tf.Variable(0, trainable=False, dtype=tf.int32, name="global_step"
inc_step = tf.assign(global_step, global_step + 1, name="increment_step"
Pretty straightforward. We have two placeholder nodes, a and b , and then four Operations that take in
a and b as input. For this exercise, we don't really need to use a tf.Variable for
global_step , but it's good to get in the habit of using this pattern. All of these nodes are placed in a
tf.Graph object, graph .
Now that we've constructed our graph, we can start up a tf.Session , initialize global_step , and
run the graph. To keep things looking clean, we place all of the nodes we'd like to run into a list called
fetches ahead of time:
17 of 23 3/10/18, 6:49 PM
In [ ]: sess = tf.Session(graph=graph)

sess.run(init) # don't forget to run init for our Variable!
# Feed values for placeholders à` and `b`

a_val, b_val = 8.0, 4.0
feed_dict = {a: a_val, b: b_val}
fetches = [inc_step, plus, minus, times, divided]
results = sess.run(fetches, feed_dict)
# Print all results

print("a: {} b: {}".format(a_val, b_val))
for i, result in enumerate(results):
print("{}: {}".format(fetches[i].name[:-2], result))
sess.close() # clean up the session
Exploring a Graph with TensorBoard
Now, let's get our first taste of TensorBoard. We'll add in some summaries in just a moment, but it's
important to see that TensorBoard can be used to examine your graph with barely any extra effort. As
mentioned above, the key class used to utilize TensorBoard is tf.summary.FileWriter , which
takes in a string output directory as an input parameter. That is, you tell the FileWriter where to save
files locally on disk. When creating a FileWriter , you can optionally pass in a Graph object, which
will store a serialized version of the graph to disk, allowing you to explore your model graphically!
If all you want to do is check out your graph, all you need to do is include these two lines:
In [ ]: writer = tf.summary.FileWriter('tbout/basic_ops', graph=graph)

writer.close()
This creates a FileWriter and tells it to save information about our graph. Calling close() flushes
any summaries that still need to be written to disk and closes the FileWriter . Since we aren't adding
any summaries yet, we can immediately close the writer. Now, we should have a new directory inside the
folder this notebook is located in called basic_ops . Navigate to the notebook directory inside your
console and type the following:
$ tensorboard --logdir=tbout/basic_ops
At first, you won't see much: the initial screen loaded is the "Summaries" tab, and we don't have any
summaries (yet)! Click the "Graph" button at the top, and you should see a visual representation of our
graph. Neat!
Note: If tensorboard doesn't automatically start up a browser window or tab in your current browser,
simply create a new tab on your own and paste the URL to the right of the at from this line in the output:
Starting TensorBoard b'47' at http://0.0.0.0:6006 . In this case, we would paste
http://0.0.0.0:6006 (http://0.0.0.0:6006) into a new brower tab
Adding Summaries
Now we're going to modify our previous graph by adding in some summary operations. In this example,
we'll use tf.summary.scalar , which creates a summary for a single (scalar) number. We'll make a
new graph called summary_graph to store our new model:
18 of 23 3/10/18, 6:49 PM

summary_graph = tf.Graph()
with summary_graph.as_default():
# Input placeholders
# Summaries for inputs

a_summ = tf.summary.scalar("a_input_summary", a)
b_summ = tf.summary.scalar("b_input_summary", b)
# Perform +, -, *, and / operations

# Summaries for operations

plus_summ = tf.summary.scalar("plus_op_summary", plus)
minus_summ = tf.summary.scalar("minus_op_summary", minus)
times_summ = tf.summary.scalar("times_op_summary", times)
divided_summ = tf.summary.scalar("divided_op_summary", divided)
# Global step counter and its increment operation

global_step = tf.Variable(0, trainable=False, dtype=tf.int32, name="global_step"
# Group all summaries together

merged = tf.summary.merge_all()
As you can see, we've added summaries for our inputs as well as our four different operations. At the very
bottom (line 31 - hit Control-m followed by l (the letter L) in the code cell to see the line numbers), we've
created a tf.summary.merge_all() operation, which bundles all of our summaries together into
one operation. If we didn't use this, we'd have to run each of the summaries individually, which would get
cumbersome.
Now, as before, let's open a Session . We'll also open a FileWriter that points to the directory
'basic_summaries', but leave it open, as we're going to be writing summaries to disk with it.
In [ ]: # we use get_fresh_dir so that we can keep track of different

# runs of our graph; here, it will creates directories like:
# tbout/basic_summaries/1
from helpers_01 import get_fresh_dir
sess = tf.Session(graph=summary_graph)
writer = tf.summary.FileWriter(get_fresh_dir('tbout/basic_summaries'),
graph=summary_graph)
sess.run(init)
We're going to create 20 random samples to generate some toy data. For each sample, we get a random
number between 0-25 for both a and b , and then run our graph with those random numbers as input.
When we run merged , our merge_all op, sess.run() outputs a summary object which we can
pass into our SummaryWriter . We do this by calling writer.add_summary , which takes in a
summary object and (optionally) a global_step parameter.
Notice that instead of creating a fetches list as we did above, we're only calling the merge_all
operation. Because we don't need to use the output from the model and the summaries depend on the
output nodes in order to be calculated, we can simply call the merged op to implicitly run each of our
addition, subtraction, multiplication, and division nodes.
19 of 23 3/10/18, 6:49 PM
In [ ]: for i in range(20):

rand_a = random.uniform(0, 25)
rand_b = random.uniform(0, 25)
feed_dict = {a: rand_a, b: rand_b}
step, summaries = sess.run([inc_step, merged], feed_dict=feed_dict)
writer.add_summary(summaries, global_step=step)
writer.flush()
Once we're done running our graph, we can close out our Session and FileWriter :
In [ ]: sess.close()
writer.close()
Now let's check out our summaries! Start up TensorBoard, pointing to the "basic_summaries" directory:
$ tensorboard --logdir=tbout/basic_summaries
We'll be using and discussing TensorBoard throughout the class, but you can check out its documentation
here (https://www.tensorflow.org/get_started/summaries_and_tensorboard).
Exercise 2
Part 1
Examine the following code. Using pen/paper or a whiteboard, draw the computation graph you think is the
result of running this code in TensorFlow.
Remember that the Python variable handles (in this case, the variables a and b ) are simply pointers to
underlying Operation or Tensor objects, and assigning them a new Operation or Tensor
doesn't change/modify the original Tensor .
To check your answer, start up TensorBoard on tbout/01_lab21 and look at the displayed Graph .
In [ ]: graph_2 = tf.Graph()

with graph_2.as_default():
a = tf.placeholder(tf.float32, name='input')
b = a
for i in range(5):
b = tf.add(a, b)
writer = tf.summary.FileWriter('tbout/01_lab21', graph=graph_2)

writer.close()
Part 2
Revisit the graph you made in Exercise 1 (http://localhost:8888/notebooks

/01_tensorflow_fundamentals.ipynb#Exercise-1). Use a tf.summary.FileWriter to output the
Graph to disk (use the directory tbout/01_lab22 ). Double check that your model resembles the
image you used to design the graph. The image is aligned from left to right, TensorBoard displays the
graph from bottom to top.
Solution 2
Part 1
20 of 23 3/10/18, 6:49 PM
Run the code in the cell for Part 1 and then view the graph with:
shell: $ tensorboard --logdir=tbout/01_lab21

open a browser tab: http://0.0.0.0:6006 (http://0.0.0.0:6006) (or similar)
Click on the Graph tab at the top of the page
Part 2
In [ ]:
And view the graph with:
shell: $ tensorboard --logdir=tbout/01_lab22

browser tab: http://0.0.0.0:6006 (http://0.0.0.0:6006)
Click on the Graph tab at the top of the page
Organizing graphs with tf.name_scope()

You might notice that the graph is starting to get a little bit more complicated. As we start building more
complex models, it's going to be important that we keep our code organized, or else it will be impossible to
discern what is going on when we load up TensorFlow. Luckily, TensorFlow provides functionality to help
you keep your code organized cleanly- the most basic of which is tf.name_scope()
(https://www.tensorflow.org/api_docs/python/tf/name_scope). Simply use this function with a with
statement to break your code up into labeled "sections".
with tf.name_scope('my_section'):
# Place some ops indented here
...
with tf.name_scope('another_section'):
# Place ops related to something else here
...
Let's add some name scopes to our last Graph to see how they help us:
21 of 23 3/10/18, 6:49 PM

organized_graph = tf.Graph()
with organized_graph.as_default():
with tf.name_scope('inputs'):
with tf.name_scope('input_summaries'):
a_summ = tf.summary.scalar("a_input_summary", a)
b_summ = tf.summary.scalar("b_input_summary", b)
with tf.name_scope('operations'):
with tf.name_scope('op_summaries'):
plus_summ = tf.summary.scalar("plus_op_summary", plus)
minus_summ = tf.summary.scalar("minus_op_summary", minus)
times_summ = tf.summary.scalar("times_op_summary", times)
divided_summ = tf.summary.scalar("divided_op_summary", divided)
with tf.name_scope('global_step'):
global_step = tf.Variable(0, trainable=False, dtype=tf.int32, name="global_step
with tf.name_scope('helpers'):
merged = tf.summary.merge_all()
As you can see, we simply added different sections to our existing code, which provides a nice form of
self-documentation. Now we can run it, placing our summaries into a new directory,
tbout/organized_summaries .
In [ ]: sess = tf.Session(graph=organized_graph)

writer = tf.summary.FileWriter(get_fresh_dir('tbout/organized_summaries'),
graph=organized_graph)
sess.run(init)
for i in range(20):
rand_a = random.uniform(0, 25)
rand_b = random.uniform(0, 25)
feed_dict = {a: rand_a, b: rand_b}
step, summaries = sess.run([inc_step, merged], feed_dict=feed_dict)
writer.add_summary(summaries, global_step=step)
writer.flush()
writer.close()
sess.close()
As before, we can look at the results with:
tensorboard --logdir=tbout/organized_summaries`]
Exercise 3
22 of 23 3/10/18, 6:49 PM
Revisit the graph you made in Exercise 1 (http://localhost:8888/notebooks

/01_tensorflow_fundamentals.ipynb#Exercise-1). Modify it to include the following features:
1. Name scopes around the first, second, and third "layers"

These are the inputs, multiplication/product nodes, and sum nodes, respectively.
2. A global step Variable to keep track of number of runs
3. A Variable that keeps a running sum of the outputs. It should have an assign Op
(https://www.tensorflow.org/api_docs/python/tf/assign) that adds itself to the (current) sum of outputs.
4. Add Summaries for the input nodes, final output node, and the running sum Variable
Solution 3
Build the Model in a Graph
In [ ]:
Run the Model and Gather Summaries
In [ ]:
In [ ]:
23 of 23 3/10/18, 6:49 PM

AI

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AI

Uploaded by

Copyright:

Available Formats

Week1_Tensorﬂow_Fundamentals_HW http://localhost:8888/notebooks/Desktop/Ai/Inte...

Class 01 - TensorFlow Fundamentals

c(3, 7) = a(3, 7) − b(3, 7) = −11

In [ ]: def add(x, y):

def multiply(x, y):

def subtract(x, y):

My First TensorFlow Graph

Fundamental TensorFlow work flow

1. Define a graph which contains your model

Step 1: Define a computation graph

In [ ]: # tf.placeholder creates an "input" node

# tf.add creates an addition node

# tf.multiply creates a multiplication node

# Add up the results of the previous two nodes

Step 2: Run the graph

In [ ]: # Start a session

# Create a "feed_dict" dictionary to define input values

# Execute the graph using `sess.run()`, which takes two parameters:

# Print the value of `out`

# Close the session

Class 01 - TensorFlow Fundamentals

We can chain multiple of these computations together to form a more complex

We could show this as a one-liner math equation, using parentheses to show

We could also define the separate operations as mathematical functions:

Programmatically, the previous equations could be created with something

def add(x, y):

TensorFlow Core API

We will introduce each of these below.

a 0-dimensional tensor is a single number (or scalar)

Defining tensors by hand

You can define Tensor object values in two main ways:

1. Native Python types

Both of these can be automatically converted into TensorFlow Tensor objects.

Tensors from Native Python

In [1]: # 0-D tensor (scalar)

# 1-D tensor (vector)

# 2-D tensor (matrix)

python_defined_tensors = [t_0d_py, t_1d_py, t_2d_py, t_3d_py]

# tf.constant creates a tf.Tensor from a fixed value

NameError: name 'tf' is not defined

In [2]: # 0-D tensor (scalar)

# 1-D tensor (vector)

# 2-D tensor (matrix)

numpy_defined_tensors = [t_0d_np, t_1d_np, t_2d_np, t_3d_np]

for ndt in numpy_defined_tensors:

NameError: name 'np' is not defined

TensorFlow Equivalent NumPy

tf.float32 np.float32 32 bit floating point.

tf.float64 np.float64 64 bit floating point.

tf.int8 np.int8 8 bit signed integer.

tf.int16 np.int16 16 bit signed integer.

tf.int32 np.int32 32 bit signed integer.

tf.int64 np.int64 64 bit signed integer.

tf.uint8 np.uint8 8 bit unsigned integer.

tf.string N/A String type, as byte array

tf.bool np.bool Boolean.

tf.qint8 N/A 8 bit signed integer used in quantized Ops.

tf.qint32 N/A 32 bit signed integer used in quantized Ops.

tf.quint8 N/A 8 bit unsigned integer used in quantized Ops.

Slightly modified version of this table (https://www.tensorflow.org/versions/r0.12/resources

In [ ]: # Just to show that they are equivalent

In [ ]: tf_string_tensor = [b"first", b"second", b"third"]

In [ ]: # Shapes corresponding to scalars

# Shape corresponding to a vector of length 3