Tensorflow - part 1: Creating and manipulating tensors
When learning and working with machine learning, we have to get on well with tensors. In this tutorial, we will show some of the ways to create and manipulate tensors in Tensorflow.
Tensorflow is one of the parallel machine learning frameworks that allow us to work with tensors in an efficient way. Below are our instructions on how to construct a tensor and do operations on it. Tensorflow currently supports mostly Python and C++, especially Python. Therefore, in this series, we choose Python to have fun with Tensorflow.
First, supposing that you have installed Tensorflow, let's import it.
import numpy as np # Numpy array is often used together with tensors,
# so you need to include it as well.
import tensorflow as tf
print(tf.__version__) # You can also use this line to check its installed versionCreating tensors
Some basic commands that create a tensor. You can create a tensor with all zero values or all one values, or makes these values generated from uniform or normal distribution.
tensor_zeros = tf.zeros((3, 4)) # a zero-value tensor with 3 rows and 4 columns
tensor_ones = tf.ones((3, 4)) # a one-value tensor with 3 rows and 4 columns
tensor_uniform = tf.random.uniform(shape=(5, 3), minval=-1.0, maxval=1.0) # the values are sampled
# from a uniform distribution,
# each of them is from 'minval' to 'maxval'
tensor_normal = tf.random.normal(shape=(5, 3), mean=0.0, stddev=1.0) # the values are sampled
# from a normal distribution
# of mean = 'mean' and
# standard deviation = 'stddev',Output
tf.Tensor(
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
TensorShape([3, 4])
tf.Tensor(
[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]], shape=(3, 4), dtype=float32)
tf.Tensor(
[[-0.67 0.803 0.262]
[-0.131 -0.416 0.285]
[ 0.952 -0.13 0.32 ]
[ 0.21 0.273 0.229]
[ 0.779 0.256 0.064]], shape=(5, 3), dtype=float32)
tf.Tensor(
[[ 0.403 -1.088 -0.063]
[ 1.337 0.712 -0.489]
[-0.764 -1.037 -1.252]
[ 0.021 -0.551 -1.743]
[-0.335 -1.043 1.009]], shape=(5, 3), dtype=float32)Moreover, you can also create a tensor with specified values by constructing one from a list or a numpy array.
x = np.array([11, 12, 13], dtype=np.int32) # a numpy array
y = [14, 15 ,16] # a Python list
tf.convert_to_tensor(x)
tf.convert_to_tensor(y)Output
tf.Tensor([11 12 13], shape=(3,), dtype=int32)
tf.Tensor([14 15 16], shape=(3,), dtype=int32)And we can specify directly the values when instantiating a tensor by using tf.constant:
tf.constant([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])Output
tf.Tensor(
[[1 2 3]
[4 5 6]
[7 8 9]], shape=(3, 3), dtype=int32)Convert a tensor to numpy array
Sometimes, you may need to work with functions in numpy array for some calculations that Tensorflow does not support. That is when you need to convert a tensor to a numpy array. To do that, just simply call numpy():
t_ones.numpy()Output
array([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]], dtype=float32)Cast to a new type
The type of a tensor, which is the type of its values, is also very important. For example, when calculating gradients of a tensor, the required type is float. For casting to a new type, use tf.cast() with 2 arguments: a tensor that needs to be cast and a new type.
tf.cast(t_ones, tf.int64)Output
tf.Tensor(
[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]], shape=(3, 4), dtype=int64)Some attributes of a tensor
Each tensor has several attributes that are very useful to know.
First, we need to have an example tensor to see its attributes:
tf.ones(shape = [2, 3, 5], dtype = tf.float32) # shape is like numpy array
# each value has type 'float32'Output
tf.Tensor(
[[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]
[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]], shape=(2, 3, 5), dtype=float32)Some attributes like rank, dimension, shape, dtype that we usually have to work with:
print('Rank of a tensor: ', tf.rank(tensor))
print('The number of dimensions: ', tensor.ndim)
print('Shape - the size of each dimension: ', tensor.shape)
print('The size of a specific dimension: ', tensor.shape[1])
print('The size of the first dimension: ', tensor.shape[0])
print('The size of the last dimension: ', tensor.shape[-1])
print('Convert shape into python list: ', tensor.shape.as_list())
print('Type of each value in a tensor: ', tensor.dtype)
print('The total number of elements in a tensor: ', tf.size(tensor).numpy())Output
Rank of a tensor: tf.Tensor(3, shape=(), dtype=int32)
The number of dimensions: 3
Shape - the size of each dimension: (2, 3, 5)
The size of a specific dimension: 3
The size of the first dimension: 2
The size of the last dimension: 5
Convert shape into python list: [2, 3, 5]
Type of each value in a tensor: <dtype: 'float32'>
The total number of elements in a tensor: 30The rank of a tensor is its number of dimensions, so the first two outputs are equal.
Math operations on tensors
Tensorflow has built some basic math operations for tensors.
- Addition
- Multiplication
- Mean, sum, and standard deviation
- Matrix multiplication
- ...
Element-wise addition:
tf.add(tensor_uniform, tensor_normal)Output
tf.Tensor(
[[-0.267 -0.285 0.199]
[ 1.206 0.296 -0.204]
[ 0.187 -1.167 -0.932]
[ 0.231 -0.278 -1.514]
[ 0.443 -0.787 1.073]], shape=(5, 3), dtype=float32)Element-wise multiplication:
tf.multiply(tensor_uniform, tensor_normal)Output
tf.Tensor(
[[-0.27 -0.874 -0.017]
[-0.175 -0.296 -0.139]
[-0.727 0.135 -0.401]
[ 0.004 -0.151 -0.399]
[-0.261 -0.266 0.065]], shape=(5, 3), dtype=float32)Compute the mean, sum, and standard deviation along a specified axis of a tensor:
tf.math.reduce_mean(tensor_uniform, axis=0)
tf.math.reduce_sum(tensor_uniform, axis=1)
tf.math.reduce_std(tensor_uniform, axis=0)Output
tf.Tensor([0.228 0.157 0.232], shape=(3,), dtype=float32)
tf.Tensor([ 0.395 -0.262 1.142 0.712 1.098], shape=(5,), dtype=float32)
tf.Tensor([0.594 0.413 0.089], shape=(3,), dtype=float32)Matrix multiplication between two tensors:
tf.linalg.matmul(tensor_uniform, tensor_normal, transpose_b=True) # transpose_b results in 5x5 tensor
tf.linalg.matmul(tensor_uniform, tensor_normal, transpose_a=True) # transpose_a results in 3x3 tensorOutput
[[-1.16 -0.452 -0.649 -0.914 -0.348]
[ 0.382 -0.611 0.175 -0.27 0.765]
[ 0.505 1.023 -0.993 -0.466 0.139]
[-0.227 0.363 -0.73 -0.545 -0.124]
[ 0.032 1.191 -0.94 -0.236 -0.463]]
[[-1.429 -1.279 -0.665]
[-0.213 -1.452 0.097]
[ 0.225 -0.607 -0.891]]Concatenating and stacking
The two most used functions that join two or more tensors are concatenate and stack.
- Concatenate 2 tensors
tensor_a = tf.ones((3, 2))
tensor_b = tf.zeros((2, 2))
tensor_c = tf.concat([tensor_a, tensor_b], axis=0)
print(tensor_c)
print(tensor_c.shape)Output
tf.Tensor(
[[1. 1.]
[1. 1.]
[1. 1.]
[0. 0.]
[0. 0.]], shape=(5, 2), dtype=float32)
(5, 2)=> All the axes of 2 tensors have to be equal in size, except the one that is used to concatenate. The new tensor does not have a new axis.
- Stack 2 tensors
tensor_a = tf.ones((5,))
tensor_b = tf.zeros((5,))
tensor_c = tf.stack([tensor_a, tensor_b], axis=1) # Stack along the new axis which is at position 1.
# There will be one more axis of size 2 in the new tensor, because we concatenate 2 tensors.
print(tensor_c)
print(tensor_c.shape)Output
tf.Tensor(
[[1. 0.]
[1. 0.]
[1. 0.]
[1. 0.]
[1. 0.]], shape=(5, 2), dtype=float32)
(5, 2)=> All the axes of 2 tensors have to be equal in size. There will be a new axis in the new tensor.
Reshaping
Reshaping a tensor is to change its shape as long as the total number of elements in the new tensor is the same as that of the original one.
Supposing that we have a rank_2_tensor with values:
tf.Tensor(
[[ 0 1 1 2 3 5 8 13 21 34]
[ 5 6 9 7 11 10 0 4 3 2]
[ 20 30 40 50 60 100 90 80 70 10]], shape=(3, 10), dtype=int32)and an another_rank_2_tensor with values:
tf.Tensor(
[[ 22 40 42 31 10 -13 17 19 -24 46]
[ 17 -31 -3 -43 17 7 46 19 12 11]
[-27 -3 -20 34 -50 6 23 -12 -26 -50]], shape=(3, 10), dtype=int32)Stack them along the axis 1:
stack_axis_1_tensor = tf.stack([rank_2_tensor, another_rank_2_tensor], axis=1)Output
tf.Tensor(
[[[ 0 1 1 2 3 5 8 13 21 34]
[ 22 40 42 31 10 -13 17 19 -24 46]]
[[ 5 6 9 7 11 10 0 4 3 2]
[ 17 -31 -3 -43 17 7 46 19 12 11]]
[[ 20 30 40 50 60 100 90 80 70 10]
[-27 -3 -20 34 -50 6 23 -12 -26 -50]]], shape=(3, 2, 10), dtype=int32)Now, we will try doing reshaping on this tensor of shape (3, 2, 10). A "reshaped" new tensor from this tensor need to have the same number of elements, so when choosing the new dimensions and their sizes, guarantee that the multiplication of these new sizes must be equal to the total number of elements in the original one.
Case 1: Reshape into ((3*2), 10) tensor (a new dimension is constructed from two old dimensions). What is the order of elements of new tensor?
tf.reshape(stack_axis_1_tensor, [3*2, 10])Output
tf.Tensor(
[[ 0 1 1 2 3 5 8 13 21 34]
[ 22 40 42 31 10 -13 17 19 -24 46]
[ 5 6 9 7 11 10 0 4 3 2]
[ 17 -31 -3 -43 17 7 46 19 12 11]
[ 20 30 40 50 60 100 90 80 70 10]
[-27 -3 -20 34 -50 6 23 -12 -26 -50]], shape=(6, 10), dtype=int32)=> Elements in dim 10 are still kept with the same values and order. Its original nature is not destroyed.
Case 2: Reshape into a (3, (2*10)) tensor (a new dimension is constructed from two old dimensions). What is the order of elements of the new tensor?
tf.reshape(stack_axis_1_tensor, [3, 2*10])Output
tf.Tensor(
[[ 0 1 1 2 3 5 8 13 21 34 22 40 42 31 10 -13 17 19
-24 46]
[ 5 6 9 7 11 10 0 4 3 2 17 -31 -3 -43 17 7 46 19
12 11]
[ 20 30 40 50 60 100 90 80 70 10 -27 -3 -20 34 -50 6 23 -12
-26 -50]], shape=(3, 20), dtype=int32)=> Elements in dim 3 are still kept with the same values and order. Its original nature is not destroyed.
Case 3: Reshape into a ((3*10), 2) tensor (a new dimension is constructed from two old dimensions). Note that the positions of 2 and 10 have been exchanged.
tf.reshape(stack_axis_1_tensor, [3*10, 2])Output
tf.Tensor(
[[ 0 1]
[ 1 2]
[ 3 5]
[ 8 13]
[ 21 34]
[ 22 40]
[ 42 31]
[ 10 -13]
[ 17 19]
[-24 46]
[ 5 6]
[ 9 7]
[ 11 10]
[ 0 4]
[ 3 2]
[ 17 -31]
[ -3 -43]
[ 17 7]
[ 46 19]
[ 12 11]
[ 20 30]
[ 40 50]
[ 60 100]
[ 90 80]
[ 70 10]
[-27 -3]
[-20 34]
[-50 6]
[ 23 -12]
[-26 -50]], shape=(30, 2), dtype=int32)=> What is the theory behind it? It gets elements from the most inner dimension and also fills into the most inner dimension of the new tensor, then gradually does this in the outer dimensions. The process continues like this until filling up all the elements for the new tensor.
Case 4: Reshape into a (4, 5, 3) tensor. We can see that 453 = 3210 = 60, so it is totally possible to reshape into this new shape. One difference with the 3 cases above is this case is not created following the shape pattern of the original shape (a new dimension is not constructed from two old dimensions).
tf.reshape(stack_axis_1_tensor, [4, 5, 3])Output
tf.Tensor(
[[[ 0 1 1]
[ 2 3 5]
[ 8 13 21]
[ 34 22 40]
[ 42 31 10]]
[[-13 17 19]
[-24 46 5]
[ 6 9 7]
[ 11 10 0]
[ 4 3 2]]
[[ 17 -31 -3]
[-43 17 7]
[ 46 19 12]
[ 11 20 30]
[ 40 50 60]]
[[100 90 80]
[ 70 10 -27]
[ -3 -20 34]
[-50 6 23]
[-12 -26 -50]]], shape=(4, 5, 3), dtype=int32)The underlying principle is that it gets elements one by one in the old tensor from the deepest dimension to the shallowest dimension, which is to fill in the new tensor in the same manner. In other words, the reshape starts from left to right and from top to bottom (look at the Output!!) in both getting elements from the old tensor and filling elements in the new tensor.