Writing and Reading Large R Matrices

Welcome to the next step in our journey of handling large datasets. In the last lesson, we explored managing data from compressed JSON files within a zip archive. Today, we will delve into how R's native capabilities for data handling can help us efficiently manage large data arrays by using the .Rdata file format.

R is widely used in data science and statistics for its ability to process large arrays and matrices swiftly. The .Rdata format allows us to store multiple objects, including matrices, in a single file, making it efficient for storing and accessing large datasets.

To begin handling large datasets, we need some large matrices as examples. Here’s how you can generate large matrices filled with random values using R.

First, let's create two large matrices of random numbers. We’ll use matrix() and runif() to generate matrices of size 1000x1000. This size is just for demonstration; you can adjust it based on your needs.

array1 <- matrix(runif(1000000), nrow = 1000, ncol = 1000)
array2 <- matrix(runif(1000000), nrow = 1000, ncol = 1000)

Here, array1 and array2 are two-dimensional matrices filled with random floats between 0 and 1, each with a shape of 1000x1000.

Now that we have our matrices, let's save them to an .Rdata file. This file format is efficient, as it can store multiple objects in a single file.

We use the save() function to achieve this:

rdata_file_path <- "large_data.Rdata"
save(list = c("array1", "array2"), file = rdata_file_path)
cat(sprintf("Arrays saved to '%s'.\n", rdata_file_path))

In this snippet:

• save() is used to save the matrices into an .Rdata file.
• rdata_file_path is the location where the file will be saved.
• list = c("array1", "array2") specifies which objects to save.

Next, let’s read the saved matrices from the .Rdata file. For this, use the load() function.

load(rdata_file_path)
cat("Data read from Rdata file.\n")

loaded_array1 <- array1
loaded_array2 <- array2

Explanation:

• load(rdata_file_path) opens the .Rdata file.
• After loading, we access the matrices simply by their names.

Once loaded, you can verify the matrices by checking their dimensions:

cat(sprintf("Array1 dimensions: %dx%d\n", dim(loaded_array1)[1], dim(loaded_array1)[2]))
cat(sprintf("Array2 dimensions: %dx%d\n", dim(loaded_array2)[1], dim(loaded_array2)[2]))

The output should confirm the dimensions are unchanged:

Array1 dimensions: 1000x1000
Array2 dimensions: 1000x1000