Tensorflow - part 1: Creating and manipulating tensors (continue)

Posted 16. August 2021. 8 min read.

In the previous part, we have shown how to create a tensor, convert a tensor to numpy array, some attributes of a tensor, math operations, concatenating and stacking, reshaping. Now, we will tell a litte more things about what we can do with tensors.

First, supposing that you have installed Tensorflow, let's import it.

import numpy as np # Numpy array is often used together with tensors,
                   # so you need to include it as well. 
import tensorflow as tf
print(tf.__version__) # You can also use this line to check its installed version

Calculating mean, standard deviation

In tf.math, there are functions that are used to compute the mean, sum, or standard deviation of values in a tensor. We can choose which axis to compute along. You should investigate the code below to see how axis affects the shape and values of the result tensor.

tensor_x = tf.random.uniform(shape=(5, 3), minval=-1.0, maxval=1.0) # create a tensor to work with
tensor_mean = tf.math.reduce_mean(tensor_x, axis=0) # mean
tensor_sum = tf.math.reduce_sum(tensor_x, axis=1) # sum
tensor_std = tf.math.reduce_std(tensor_x, axis=0) # standard deviation

Output

tf.Tensor(
[[ 0.47  -0.716 -0.873]
 [-0.622  0.596  0.408]
 [-0.182  0.029 -0.856]
 [-0.309  0.501 -0.693]
 [-0.473  0.25   0.309]], shape=(5, 3), dtype=float32)
tf.Tensor([-0.223  0.132 -0.341], shape=(3,), dtype=float32)
tf.Tensor([-1.119  0.383 -1.009 -0.502  0.087], shape=(5,), dtype=float32)
tf.Tensor([0.377 0.468 0.576], shape=(3,), dtype=float32)

When axis=0, the calculations happen along the vertical direction (column) of the tensor_x. There are 3 columns in the tensor_x, so the result tensor (tensor_mean and tensor_std) will have 3 values, each of which is the result of one column. When axis=1, in contrast, the calculations happen along the horizontal direction (row) of the tensor_x.

Slicing

If wanting to extract sub tensors from a tensor, we need to use slicing. Supposing we have a tensor

tensor = tf.range(0, 8)

Output

tf.Tensor([0 1 2 3 4 5 6 7], shape=(8,), dtype=int32)

Extract from the second element to the third element. It is also the same as in python slicing that the last element (at position 4) is excluded.

tensor[1:4]

Output

tf.Tensor([1 2 3], shape=(3,), dtype=int32)

Extract 3 last elements

tensor[-3:]

Output

tf.Tensor([5 6 7], shape=(3,), dtype=int32)

You can also use tf.slice for this with one difference. For the second argument size, it does not use the end index like above, but size instead. Get a tensor from previous post as example.

stack_axis_1_tensor # from previous post

Output

tf.Tensor(
[[[  0   1   1   2   3   5   8  13  21  34]
  [ 22  40  42  31  10 -13  17  19 -24  46]]

 [[  5   6   9   7  11  10   0   4   3   2]
  [ 17 -31  -3 -43  17   7  46  19  12  11]]

 [[ 20  30  40  50  60 100  90  80  70  10]
  [-27  -3 -20  34 -50   6  23 -12 -26 -50]]], shape=(3, 2, 10), dtype=int32)

Use tf.slice:

tf.slice(stack_axis_1_tensor, begin=[1, 0, 3], size=[2, 1, 4])

Output

tf.Tensor(
[[[  7  11  10   0]]

 [[ 50  60 100  90]]], shape=(2, 1, 4), dtype=int32)

We can also use step for strided slice. For example, if you want to extract one element in every two elements (step=2), you can use slicing as above. tf.gather is an another way to do this, but with this you need need to specify correctly which positions to get the value.

tensor[::2]
tf.gather(tensor, indices=[0, 2, 4, 6]) # tf.gather to get elements at even indices: 0, 2, 4, 6

Output

tf.Tensor([0 2 4 6], shape=(4,), dtype=int32)
tf.Tensor([0 2 4 6], shape=(4,), dtype=int32)

tf.gather can even get elements at indices that are not equally spaces and the new tensor can also be shuffled

string_tensor = tf.constant(list('Hello world'))
tf.gather(string_tensor, indices=[8, 1, 4]) # Get at positions 8, 1, 4 respectively

Output

tf.Tensor([b'r' b'e' b'o'], shape=(3,), dtype=string)

Here, the indices list does not contain positions in the increasing or decreasing order, but they are shuffled in an unordered style.

Expand dims

Sometimes you need to add a new axis to a tensor so it can be computed (or concatenated) together with another tensor. Some operations required the involved tensors have the same number of dimensions.

tensor_x = tf.zeros([50, 50, 4])
tf.expand_dims(tensor_x, axis=0).shape
tf.expand_dims(tensor_x, axis=1).shape

Output

(1, 50, 50, 4)
(50, 1, 50, 4)

The new axis has been added at the specified position (0 and 1).

Tiling

Repeat elements in tensor_a by a factor as specified for each dim in tensor_b

tensor_a = tf.constant([[1, 2], [3, 4]], tf.int32) 
tensor_b = tf.constant([1, 3], tf.int32) # repeat 1 time in dimension 0, repeat 3 times in dimension 1
print(tf.tile(tensor_a, tensor_b))
tensor_c = tf.constant([3, 1], tf.int32) # repeat 3 times in dimension 0, repeat 1 time in dimension 1
print(tf.tile(tensor_a, tensor_c))

Output

tf.Tensor(
[[1 2 1 2 1 2]
 [3 4 3 4 3 4]], shape=(2, 6), dtype=int32)

tf.Tensor(
[[1 2]
 [3 4]
 [1 2]
 [3 4]
 [1 2]
 [3 4]], shape=(6, 2), dtype=int32)

In the example above, the tensor_b is [1, 3]. This means the tensor_a is replicated one time along the 0-th dimension (row) and 3 times along the 1-th dimension (column).

Squeeze

Removing the unnecessary dimensions (dimensions that have size 1)

tensor_x = tf.zeros((1, 2, 1, 4, 1))
tensor_sqz = tf.squeeze(tensor_x)
print('old shape:', tensor_x.shape, '-->', 'new shape:', tensor_sqz.shape)
tensor_sqz = tf.squeeze(tensor_x, axis=(2, 4)) # We can also choose what axis to remove
print('old shape:', tensor_x.shape, '-->', 'new shape:', tensor_sqz.shape)

Output

old shape: (1, 2, 1, 4, 1) --> new shape: (2, 4)
old shape: (1, 2, 1, 4, 1) --> new shape: (1, 2, 4)

Split

There are 2 types of splits:

  • Split a tensor into multiple tensors of equal size
tf.random.set_seed(1)
tensor = tf.random.uniform((9,))
print(tensor.numpy())

tensor_splits = tf.split(tensor, num_or_size_splits=3)
[item.numpy() for item in tensor_splits]

Output

[0.165 0.901 0.631 0.435 0.292 0.643 0.976 0.435 0.66 ]

[array([0.165, 0.901, 0.631], dtype=float32),
 array([0.435, 0.292, 0.643], dtype=float32),
 array([0.976, 0.435, 0.66 ], dtype=float32)]
  • Split a tensor into multiple tensors of specified size for each new tensor
tensor = tf.random.uniform((7,))
print(tensor.numpy())

tensor_splits = tf.split(tensor, num_or_size_splits=[3, 4]) # size
[item.numpy() for item in tensor_splits]

Output

[0.829 0.634 0.515 0.391 0.581 0.048 0.178]

[array([0.829, 0.634, 0.515], dtype=float32),
 array([0.391, 0.581, 0.048, 0.178], dtype=float32)]

First tensor has size 3 and second tensor has size 4. That is because the numbers that we give to the paramerter num_or_size_splits.

Unique

Let's have a small exercise here. We have 2 flat tensors and want to check if all of their elements are equal.

Supposing we have 2 tensors: reshape_0 and reshape_1:

print(reshape_0)
print(reshape_1)

Output

tf.Tensor(
[  0   1   1   2   3   5   8  13  21  34   5   6   9   7  11  10   0   4
   3   2  20  30  40  50  60 100  90  80  70  10  22  40  42  31  10 -13
  17  19 -24  46  17 -31  -3 -43  17   7  46  19  12  11 -27  -3 -20  34
 -50   6  23 -12 -26 -50], shape=(60,), dtype=int32)

tf.Tensor(
[  0   1   1   2   3   5   8  13  21  34  22  40  42  31  10 -13  17  19
 -24  46   5   6   9   7  11  10   0   4   3   2  17 -31  -3 -43  17   7
  46  19  12  11  20  30  40  50  60 100  90  80  70  10 -27  -3 -20  34
 -50   6  23 -12 -26 -50], shape=(60,), dtype=int32)

To check if they are equal or not, use tf.math.equal: this method checks in an element-wise way and return a list of True, False values. If the list are all True values, it means the 2 tensors are equal.

print('Use tf.math.equal: ', tf.math.equal(reshape_0, reshape_1))

Output

Use tf.math.equal:  tf.Tensor(
[ True  True  True  True  True  True  True  True  True  True False False
 False False False False False False False False False False False False
 False False False False False False False False False False False False
 False  True False False False False False False False False False False
 False False  True  True  True  True  True  True  True  True  True  True], shape=(60,), dtype=bool)

In the case when there are too many elements, it would be better if we only show the unique elements.

print('Use tf.unique: ', tf.unique(tf.math.equal(reshape_0, reshape_1)))

Output

Use tf.unique:  Unique(y=<tf.Tensor: shape=(2,), dtype=bool, numpy=array([ True, False])>, idx=<tf.Tensor: shape=(60,), dtype=int32, numpy=
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)>)

Moreover, you can count the number of repetitions of each unique element.

print('Use tf.unique_with_count', tf.unique_with_counts(tf.math.equal(reshape_0, reshape_1)))

Output

Use tf.unique_with_count: UniqueWithCounts(y=<tf.Tensor: shape=(2,), dtype=bool, numpy=array([ True, False])>, idx=<tf.Tensor: shape=(60,), dtype=int32, numpy=
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)>, count=<tf.Tensor: shape=(2,), dtype=int32, numpy=array([21, 39], dtype=int32)>)

You can see in the end of the output there is [21, 39] which means there are 21 values 0 (True) and 39 values 1 (False). Therefore, not all elements are equal.

Broadcasting

When running computation on 2 tensors having different sizes, the smaller tensor will be automatically expanded to be matched to the larger one.

smaller = tf.constant(2) # is automatically expanded into [2, 2, 2]
larger = tf.constant([1, 2, 3])
print(smaller * larger)
print(tf.multiply(smaller, larger))

Output

tf.Tensor([2 4 6], shape=(3,), dtype=int32)
tf.Tensor([2 4 6], shape=(3,), dtype=int32)

Broadcasting is also useful when we consider the case of outer product of 2 tensors

x = tf.reshape(larger, [3, 1])
print(x)
y = tf.range(1, 6)
print(y)
print(tf.multiply(x, y)) # like outer product

Output

tf.Tensor(
[[1]
 [2]
 [3]], shape=(3, 1), dtype=int32)

tf.Tensor([1 2 3 4 5], shape=(5,), dtype=int32)

tf.Tensor(
[[ 1  2  3  4  5]
 [ 2  4  6  8 10]
 [ 3  6  9 12 15]], shape=(3, 5), dtype=int32)

The end

# tensorflow # tensor

AI and Software Development enthusiasts. Knowledge is common. Our insights and experience on it is unique