Fundamental Tensor Operations in PyTorch

Hello there! In today's lesson, we delve deeper into PyTorch Tensors and their fundamental operations such as addition and multiplication. We will also learn about broadcasting, a powerful feature in PyTorch that makes it possible to perform operations between tensors of different shapes. Let's get started!

In our previous lesson, we learned about PyTorch Tensors — the basic building blocks in PyTorch that hold data. They resemble arrays in NumPy but come with additional features that are essential for machine learning tasks like GPU acceleration. Now, we will learn the different operations that can be performed on tensors and better understand how these computations happen deep inside neural networks.

Fundamentally, tensor operations include arithmetical operations such as addition, subtraction, multiplication, and division. Two essential forms of multiplication in tensors are element-wise multiplication and matrix multiplication. We can also perform operations between a scalar and a tensor, or between tensors of different shapes — thanks to a technique called Broadcasting.

In tensor addition, we simply add the corresponding elements in the tensors. PyTorch provides the torch.add() function that takes two tensors as inputs and returns a new tensor which is the result of adding the input tensors. It is also as easy as using the + operator.

Let's go over an example:

import torch

# Creating two tensors
tensor_a = torch.tensor([[1, 2], [3, 4]], dtype=torch.int32)
tensor_b = torch.tensor([[5, 6], [7, 8]], dtype=torch.int32)

# Tensor addition
tensor_sum = torch.add(tensor_a, tensor_b)
print(f"Tensor Addition:\n{tensor_sum}")

The output of the above code will be:

Tensor Addition:
tensor([[ 6,  8],
        [10, 12]], dtype=torch.int32)

This output shows the result of adding two tensors element-wise. Each element from tensor_a is added to the corresponding element in tensor_b, resulting in a new tensor of the same shape.

Next, let's learn about tensor multiplication. One type is element-wise multiplication where each element in one tensor is multiplied with the corresponding element in the other tensor. We can use the torch.mul() function or the * operator for this.

The other type is matrix multiplication. This is similar to the matrix multiplication in linear algebra where the dot product of rows and columns is calculated. For this, we use the torch.matmul() function.

Below is the Python code demonstrative of these operations:

# Element-wise Multiplication
tensor_product = torch.mul(tensor_a, tensor_b)
print(f"Element-wise Multiplication:\n{tensor_product}")

# Matrix Multiplication
tensor_c = torch.tensor([[1], [2]], dtype=torch.int32) # 2x1 tensor
tensor_matmul = torch.matmul(tensor_a, tensor_c)
print(f"Matrix Multiplication:\n{tensor_matmul}")

The output of the above code will be:

Element-wise Multiplication:
tensor([[ 5, 12],
        [21, 32]], dtype=torch.int32)

Matrix Multiplication:
tensor([[ 5],
        [11]], dtype=torch.int32)

These outputs demonstrate two different multiplication techniques. In element-wise multiplication, each element of tensor_a is multiplied with the corresponding element of tensor_b. In matrix multiplication, the dot product between tensor_a and tensor_c results in a new tensor shape based on matrix product rules.