Tensors and Tensor Arithmetic

In deep learning it is common to see a lot of discussion around tensors as the cornerstone data structure. 

Tensor even appears in name of Google’s flagship machine learning library: TensorFlow. 

Tensors are a type of data structure used in linear algebra, and like vectors and matrices, you can calculate arithmetic operations with tensors.

This tutorial is divided into 3 parts; they are:

• What are Tensors
• Tensors in Python
• Tensor Arithmetic

A. What are Tensors

A tensor is a generalization of vectors and matrices and is easily understood as a multidimensional array.

A vector is a one-dimensional or first order tensor and a matrix is a two-dimensional or second order tensor.

For example, below defines a 3 × 3 × 3 three-dimensional tensor T with dimensions index as i,j,k.

B. Tensors in Python

Like vectors and matrices, tensors can be represented in Python using N-dimensional array (narray).

A tensor can be defined in-line to the constructor of array() as a list of lists. The example below defines a 3 × 3 × 3 tensor as a NumPy ndarray.

# create tensor
from numpy import array
T = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
T = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
T = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
print(T.shape)
print(T)

-----Result-----

(3, 3, 3)

[[[ 1 2 [ 4 5 [ 7 8

3] 6] 9]]

[[11 12 13]
[14 15 16]
[17 18 19]]
[[21 22 23]
[24 25 26]

C. Tensor Arithmetic

As with matrices, we can perform element-wise arithmetic between tensors.

1. Tensor Addition

# tensor addition
from numpy import array
# define first tensor
A = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# define second tensor
B = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# add tensors
C = A + B
print(C)

-----Result-----

[[[ 2 4 6]
[ 8 10 12]
[14 16 18]]
[[22 24 26]
[28 30 32]
[34 36 38]]
[[42 44 46]
[48 50 52]
[54 56 58]]]

2. Tensor Subtraction

# tensor subtraction
from numpy import array
# define first tensor
A = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# define second tensor
B = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# subtract tensors
C = A - B
print(C)

-----Result-----

[[[0 0 0]
[0 0 0]
[0 0 0]]
[[0 0 0]
[0 0 0]
[0 0 0]]
[[0 0 0]
[0 0 0]
[0 0 0]]]

3. Tensor Hadamard Product

# tensor Hadamard product
from numpy import array
# define first tensor
A = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# define second tensor
B = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# multiply tensors
C = A * B
print(C)

-----Result-----

[[[ 1 4 9]
[ 16 25 36]
[ 49 64 81]]
[[121 144 169]
[196 225 256]
[289 324 361]]
[[441 484 529]
[576 625 676]
[729 784 841]]]

4. Tensor Division

# tensor division
from numpy import array
# define first tensor
A = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# define second tensor
B = array([
[[1,2,3], [4,5,6], [7,8,9]],
[[11,12,13], [14,15,16], [17,18,19]],
[[21,22,23], [24,25,26], [27,28,29]]])
# divide tensors
C = A / B
print(C)

-----Result-----

[[[ 1. 1. [ 1. 1. [ 1. 1.

1.] 1.] 1.]]

[[ 1. 1. [ 1. 1. [ 1. 1.

1.] 1.] 1.]]

[[ 1. 1. [ 1. 1. [ 1. 1.

1.] 1.] 1.]]]

5. Tensor Product

# tensor product
from numpy import array
from numpy import tensordot
# define first vector
A = array([1,2])
# define second vector
B = array([3,4])
# calculate tensor product
C = tensordot(A, B, axes=0)
print(C)

-----Result-----

[[3 4]
[6 8]]