# Tensorflow - part 1: Creating and manipulating tensors

When learning and working with machine learning, we have to get on well with tensors. In this tutorial, we will show some of the ways to create and manipulate tensors in Tensorflow.

Tensorflow is one of the parallel machine learning frameworks that allow us to work with tensors in an efficient way. Below are our instructions on how to construct a tensor and do operations on it. Tensorflow currently supports mostly Python and C++, especially Python. Therefore, in this series, we choose Python to have fun with Tensorflow.

First, supposing that you have installed Tensorflow, let's import it.

``````import numpy as np # Numpy array is often used together with tensors,
# so you need to include it as well.
import tensorflow as tf
print(tf.__version__) # You can also use this line to check its installed version``````

## Creating tensors

Some basic commands that create a tensor. You can create a tensor with all zero values or all one values, or makes these values generated from uniform or normal distribution.

``````tensor_zeros = tf.zeros((3, 4)) # a zero-value tensor with 3 rows and 4 columns
tensor_ones = tf.ones((3, 4)) # a one-value tensor with 3 rows and 4 columns
tensor_uniform = tf.random.uniform(shape=(5, 3), minval=-1.0, maxval=1.0) # the values are sampled
# from a uniform distribution,
# each of them is from 'minval' to 'maxval'
tensor_normal = tf.random.normal(shape=(5, 3), mean=0.0, stddev=1.0) # the values are sampled
# from a normal distribution
# of mean = 'mean' and
# standard deviation = 'stddev',``````

Output

``````tf.Tensor(
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]], shape=(3, 4), dtype=float32)
TensorShape([3, 4])

tf.Tensor(
[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]], shape=(3, 4), dtype=float32)

tf.Tensor(
[[-0.67   0.803  0.262]
[-0.131 -0.416  0.285]
[ 0.952 -0.13   0.32 ]
[ 0.21   0.273  0.229]
[ 0.779  0.256  0.064]], shape=(5, 3), dtype=float32)

tf.Tensor(
[[ 0.403 -1.088 -0.063]
[ 1.337  0.712 -0.489]
[-0.764 -1.037 -1.252]
[ 0.021 -0.551 -1.743]
[-0.335 -1.043  1.009]], shape=(5, 3), dtype=float32)``````

Moreover, you can also create a tensor with specified values by constructing one from a list or a numpy array.

``````x = np.array([11, 12, 13], dtype=np.int32) # a numpy array
y = [14, 15 ,16] # a Python list
tf.convert_to_tensor(x)
tf.convert_to_tensor(y)``````

Output

``````tf.Tensor([11 12 13], shape=(3,), dtype=int32)
tf.Tensor([14 15 16], shape=(3,), dtype=int32)``````

And we can specify directly the values when instantiating a tensor by using `tf.constant`:

``````tf.constant([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])``````

Output

``````tf.Tensor(
[[1 2 3]
[4 5 6]
[7 8 9]], shape=(3, 3), dtype=int32)``````

## Convert a tensor to numpy array

Sometimes, you may need to work with functions in numpy array for some calculations that Tensorflow does not support. That is when you need to convert a tensor to a numpy array. To do that, just simply call `numpy()`:

``t_ones.numpy()``

Output

``````array([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]], dtype=float32)``````

## Cast to a new type

The type of a tensor, which is the type of its values, is also very important. For example, when calculating gradients of a tensor, the required type is float. For casting to a new type, use `tf.cast()` with 2 arguments: a tensor that needs to be cast and a new type.

``tf.cast(t_ones, tf.int64)``

Output

``````tf.Tensor(
[[1. 1. 1. 1.]
[1. 1. 1. 1.]
[1. 1. 1. 1.]], shape=(3, 4), dtype=int64)``````

## Some attributes of a tensor

Each tensor has several attributes that are very useful to know.

First, we need to have an example tensor to see its attributes:

``````tf.ones(shape = [2, 3, 5], dtype = tf.float32) # shape is like numpy array
# each value has type 'float32'``````

Output

``````tf.Tensor(
[[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]

[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]], shape=(2, 3, 5), dtype=float32)``````

Some attributes like rank, dimension, shape, dtype that we usually have to work with:

``````print('Rank of a tensor: ', tf.rank(tensor))
print('The number of dimensions: ', tensor.ndim)
print('Shape - the size of each dimension: ', tensor.shape)
print('The size of a specific dimension: ', tensor.shape[1])
print('The size of the first dimension: ', tensor.shape[0])
print('The size of the last dimension: ', tensor.shape[-1])
print('Convert shape into python list: ', tensor.shape.as_list())
print('Type of each value in a tensor: ', tensor.dtype)
print('The total number of elements in a tensor: ', tf.size(tensor).numpy())``````

Output

``````Rank of a tensor:  tf.Tensor(3, shape=(), dtype=int32)
The number of dimensions:  3
Shape - the size of each dimension:  (2, 3, 5)
The size of a specific dimension:  3
The size of the first dimension:  2
The size of the last dimension:  5
Convert shape into python list:  [2, 3, 5]
Type of each value in a tensor:  <dtype: 'float32'>
The total number of elements in a tensor:  30``````

The rank of a tensor is its number of dimensions, so the first two outputs are equal.

## Math operations on tensors

Tensorflow has built some basic math operations for tensors.

• Multiplication
• Mean, sum, and standard deviation
• Matrix multiplication
• ...

``tf.add(tensor_uniform, tensor_normal)``

Output

``````tf.Tensor(
[[-0.267 -0.285  0.199]
[ 1.206  0.296 -0.204]
[ 0.187 -1.167 -0.932]
[ 0.231 -0.278 -1.514]
[ 0.443 -0.787  1.073]], shape=(5, 3), dtype=float32)``````

Element-wise multiplication:

``tf.multiply(tensor_uniform, tensor_normal)``

Output

``````tf.Tensor(
[[-0.27  -0.874 -0.017]
[-0.175 -0.296 -0.139]
[-0.727  0.135 -0.401]
[ 0.004 -0.151 -0.399]
[-0.261 -0.266  0.065]], shape=(5, 3), dtype=float32)``````

Compute the mean, sum, and standard deviation along a specified axis of a tensor:

``````tf.math.reduce_mean(tensor_uniform, axis=0)
tf.math.reduce_sum(tensor_uniform, axis=1)
tf.math.reduce_std(tensor_uniform, axis=0)``````

Output

``````tf.Tensor([0.228 0.157 0.232], shape=(3,), dtype=float32)
tf.Tensor([ 0.395 -0.262  1.142  0.712  1.098], shape=(5,), dtype=float32)
tf.Tensor([0.594 0.413 0.089], shape=(3,), dtype=float32)``````

Matrix multiplication between two tensors:

``````tf.linalg.matmul(tensor_uniform, tensor_normal, transpose_b=True) # transpose_b results in 5x5 tensor
tf.linalg.matmul(tensor_uniform, tensor_normal, transpose_a=True) # transpose_a results in 3x3 tensor``````

Output

``````[[-1.16  -0.452 -0.649 -0.914 -0.348]
[ 0.382 -0.611  0.175 -0.27   0.765]
[ 0.505  1.023 -0.993 -0.466  0.139]
[-0.227  0.363 -0.73  -0.545 -0.124]
[ 0.032  1.191 -0.94  -0.236 -0.463]]

[[-1.429 -1.279 -0.665]
[-0.213 -1.452  0.097]
[ 0.225 -0.607 -0.891]]``````

## Concatenating and stacking

The two most used functions that join two or more tensors are `concatenate` and `stack`.

• Concatenate 2 tensors
``````tensor_a = tf.ones((3, 2))
tensor_b = tf.zeros((2, 2))
tensor_c = tf.concat([tensor_a, tensor_b], axis=0)
print(tensor_c)
print(tensor_c.shape)``````

Output

``````tf.Tensor(
[[1. 1.]
[1. 1.]
[1. 1.]
[0. 0.]
[0. 0.]], shape=(5, 2), dtype=float32)

(5, 2)``````

=> All the axes of 2 tensors have to be equal in size, except the one that is used to concatenate. The new tensor does not have a new axis.

• Stack 2 tensors
``````tensor_a = tf.ones((5,))
tensor_b = tf.zeros((5,))
tensor_c = tf.stack([tensor_a, tensor_b], axis=1) # Stack along the new axis which is at position 1.
# There will be one more axis of size 2 in the new tensor, because we concatenate 2 tensors.
print(tensor_c)
print(tensor_c.shape)``````

Output

``````tf.Tensor(
[[1. 0.]
[1. 0.]
[1. 0.]
[1. 0.]
[1. 0.]], shape=(5, 2), dtype=float32)

(5, 2)``````

=> All the axes of 2 tensors have to be equal in size. There will be a new axis in the new tensor.

## Reshaping

Reshaping a tensor is to change its shape as long as the total number of elements in the new tensor is the same as that of the original one.

Supposing that we have a `rank_2_tensor` with values:

``````tf.Tensor(
[[  0   1   1   2   3   5   8  13  21  34]
[  5   6   9   7  11  10   0   4   3   2]
[ 20  30  40  50  60 100  90  80  70  10]], shape=(3, 10), dtype=int32)``````

and an `another_rank_2_tensor` with values:

``````tf.Tensor(
[[ 22  40  42  31  10 -13  17  19 -24  46]
[ 17 -31  -3 -43  17   7  46  19  12  11]
[-27  -3 -20  34 -50   6  23 -12 -26 -50]], shape=(3, 10), dtype=int32)``````

Stack them along the axis 1:

``stack_axis_1_tensor = tf.stack([rank_2_tensor, another_rank_2_tensor], axis=1)``

Output

``````tf.Tensor(
[[[  0   1   1   2   3   5   8  13  21  34]
[ 22  40  42  31  10 -13  17  19 -24  46]]

[[  5   6   9   7  11  10   0   4   3   2]
[ 17 -31  -3 -43  17   7  46  19  12  11]]

[[ 20  30  40  50  60 100  90  80  70  10]
[-27  -3 -20  34 -50   6  23 -12 -26 -50]]], shape=(3, 2, 10), dtype=int32)``````

Now, we will try doing reshaping on this tensor of shape (3, 2, 10). A "reshaped" new tensor from this tensor need to have the same number of elements, so when choosing the new dimensions and their sizes, guarantee that the multiplication of these new sizes must be equal to the total number of elements in the original one.

Case 1: Reshape into ((3*2), 10) tensor (a new dimension is constructed from two old dimensions). What is the order of elements of new tensor?

``tf.reshape(stack_axis_1_tensor, [3*2, 10])``

Output

``````tf.Tensor(
[[  0   1   1   2   3   5   8  13  21  34]
[ 22  40  42  31  10 -13  17  19 -24  46]
[  5   6   9   7  11  10   0   4   3   2]
[ 17 -31  -3 -43  17   7  46  19  12  11]
[ 20  30  40  50  60 100  90  80  70  10]
[-27  -3 -20  34 -50   6  23 -12 -26 -50]], shape=(6, 10), dtype=int32)``````

=> Elements in dim 10 are still kept with the same values and order. Its original nature is not destroyed.

Case 2: Reshape into a (3, (2*10)) tensor (a new dimension is constructed from two old dimensions). What is the order of elements of the new tensor?

``tf.reshape(stack_axis_1_tensor, [3, 2*10])``

Output

``````tf.Tensor(
[[  0   1   1   2   3   5   8  13  21  34  22  40  42  31  10 -13  17  19
-24  46]
[  5   6   9   7  11  10   0   4   3   2  17 -31  -3 -43  17   7  46  19
12  11]
[ 20  30  40  50  60 100  90  80  70  10 -27  -3 -20  34 -50   6  23 -12
-26 -50]], shape=(3, 20), dtype=int32)``````

=> Elements in dim 3 are still kept with the same values and order. Its original nature is not destroyed.

Case 3: Reshape into a ((3*10), 2) tensor (a new dimension is constructed from two old dimensions). Note that the positions of 2 and 10 have been exchanged.

``tf.reshape(stack_axis_1_tensor, [3*10, 2])``

Output

``````tf.Tensor(
[[  0   1]
[  1   2]
[  3   5]
[  8  13]
[ 21  34]
[ 22  40]
[ 42  31]
[ 10 -13]
[ 17  19]
[-24  46]
[  5   6]
[  9   7]
[ 11  10]
[  0   4]
[  3   2]
[ 17 -31]
[ -3 -43]
[ 17   7]
[ 46  19]
[ 12  11]
[ 20  30]
[ 40  50]
[ 60 100]
[ 90  80]
[ 70  10]
[-27  -3]
[-20  34]
[-50   6]
[ 23 -12]
[-26 -50]], shape=(30, 2), dtype=int32)``````

=> What is the theory behind it? It gets elements from the most inner dimension and also fills into the most inner dimension of the new tensor, then gradually does this in the outer dimensions. The process continues like this until filling up all the elements for the new tensor.

Case 4: Reshape into a (4, 5, 3) tensor. We can see that 453 = 3210 = 60, so it is totally possible to reshape into this new shape. One difference with the 3 cases above is this case is not created following the shape pattern of the original shape (a new dimension is not constructed from two old dimensions).

``tf.reshape(stack_axis_1_tensor, [4, 5, 3])``

Output

``````tf.Tensor(
[[[  0   1   1]
[  2   3   5]
[  8  13  21]
[ 34  22  40]
[ 42  31  10]]

[[-13  17  19]
[-24  46   5]
[  6   9   7]
[ 11  10   0]
[  4   3   2]]

[[ 17 -31  -3]
[-43  17   7]
[ 46  19  12]
[ 11  20  30]
[ 40  50  60]]

[[100  90  80]
[ 70  10 -27]
[ -3 -20  34]
[-50   6  23]
[-12 -26 -50]]], shape=(4, 5, 3), dtype=int32)``````

The underlying principle is that it gets elements one by one in the old tensor from the deepest dimension to the shallowest dimension, which is to fill in the new tensor in the same manner. In other words, the reshape starts from left to right and from top to bottom (look at the Output!!) in both getting elements from the old tensor and filling elements in the new tensor.