# Introduction to ragged tensors

## Introduction

We review Tensorflow’s concept of ragged tensors, which were introduced at the end of 2018. We explain their basic structure and why they are useful.

## Problem statement

With the standardization of traditional machine learning problems, many
models are very easy to implement: read a table of features from a
database, use pandas and numpy for preprocessing, build a model with one
of the well-known libraries, and type `model.fit()`

. In many cases -
that’s it! Done!

But wait - what if the data does not come in tabular form and is irregular by nature? What if there are instances with varying dimensions? Consider a scenario for time series classification and suppose we have a dataset consisting of four different short time series:

As you can see, the series differ in both the number and time of the measurements. Since machine learning models typically require a fixed input size, it’s a bit more complicated to fit such data into our models.

There are a number of possibilities to handle this type of input; for
example we could interpolate the series and take *virtual* measurements
at the same timestamps for each series:

Here we take the values from timestamps 0, 2, 4, 6, 8, and 10 such that every series consists of 6 values. However, at this stage we already have to choose hyperparameters such as the type of interpolation, how many values, etc. However, we cannot rely on the accuracy of the interpolation, especially for extrapolated values and values within large gaps between successive measurements (see the orange and green series at time 10).

From the technical side, when we feed the data into a TensorFlow Keras model and do not want to use interpolation techniques, a common practice is to pad the series, e.g. with zeros at the end. This is necessary because TensorFlow groups batches of data together which must have the same shape in every dimension. A batch of the 4 series above would have the shape (4, 6) with 4 being the number of series (=batch dimension) and 6 being the number of measurements per series.

However, the 6 arises from artificial data, either interpolated measurements or padding values. To overcome the uncertainty and the overhead of both these techniques, we can use ragged tensors to work with the original data.

## Concept of ragged tensors

The concept of ragged tensors is surprisingly easy after understanding the intention behind them. Let’s stick with our above example with 4 time series. As you can see, the minimum number of measurements per series is 3, while the maximum is 5. With padding we would have to fill every series with zeros at the end (or sometimes at the beginning) to achieve a common length of 5.

In contrast, a ragged tensor consists of the concatenation of all values
from all series together with metadata specifying where to split the
concatenation into the individual series. Let’s define our dataframe
`df`

and then our ragged tensor `rt`

:

time | value |
---|---|

0 | 3 |

3 | 1 |

6 | 8 |

8 | 0 |

10 | 9 |

0 | 15 |

5 | 11 |

8 | 7 |

0 | 12 |

2 | 7 |

4 | 8 |

9 | 2 |

0 | 9 |

4 | 0 |

6 | 13 |

10 | 4 |

```
row_splits = [0, 5, 8, 12, 16]
rt = tf.RaggedTensor.from_row_splits(values=df.values, row_splits=row_splits)
rt
<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
```

As we can see, the `row_splits`

array defines the individual series by
specifying their startrow (inclusive) and endrow (exclusive).

That’s it. This is the really simple structure of ragged tensors. As an
alternative to specifying the `row_splits`

we can also create the same
ragged tensor with one of the following methods:

`value_rowids`

: for every row in the concatenated series we specify an id number which indexes the individual series:

```
value_rowids = [0, 0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]
rt_1 = tf.RaggedTensor.from_value_rowids(values=df.values, value_rowids=value_rowids)
rt_1
<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
```

`row_lengths`

: we state the length of every individual series:

```
row_lengths = [5, 3, 4, 4]
rt_2 = tf.RaggedTensor.from_row_lengths(values=df.values, row_lengths=row_lengths)
rt_2
<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
```

`constant`

: we can define the ragged tensor as a “constant” by directly specifying a list of arrays:

```
rt_3 = tf.ragged.constant([df.loc[0:4, :].values, df.loc[5:7, :].values, df.loc[8:11, :].values, df.loc[12:15, :].values])
rt_3
<tf.RaggedTensor [[[0, 3], [3, 1], [6, 8], [8, 0], [10, 9]], [[0, 15], [5, 11], [8, 7]], [[0, 12], [2, 7], [4, 8], [9, 2]], [[0, 9], [4, 0], [6, 13], [10, 4]]]>
```

Internally, it does not matter which method we choose to create a ragged tensor, the results are all equivalent. Next we’ll see how to perform mathematical operations on ragged tensors.

## Working with ragged tensors

TensorFlow provides a very handy function to perform operations on
ragged tensors: `tf.ragged.map_flat_values(op, *args, **kwargs)`

. It
does what the function name says - every ragged tensor in `args`

is
substituted by its concatenated (=flat) version, omitting the batch
dimension. In our example, this is the same as if we operate on the
`df.values`

directly. The only difference is that the output of the
operation is again a ragged tensor with the same metadata information
about where to split. Let’s consider an example where we compute the
matrix product of the ragged tensor with a matrix `m`

of shape (2, 5).
Each individual series in our ragged tensor has shape (k, 2) where k
corresponds to the number of measurements in the given series. Taking
care to first casting to floats:

```
m = tf.random.uniform(shape=[2, 5])
print(m.shape)
(2, 5)
rt = tf.cast(rt, tf.float32)
result = tf.ragged.map_flat_values(tf.matmul, rt, m)
print(*(t.shape for t in result), sep='\n')
(5, 5)
(3, 5)
(4, 5)
(4, 5)
```

Perfect! The resulting ragged tensor has the same row splits as the
input, but the inner dimension changed from 2 to 5 because of the matrix
multiplication. We could do some more complicated operations, for
example if `m`

is not a 2-dimensional matrix, but a 3-dimensional
tensor:

```
m = tf.random.uniform(shape=[2, 5, 4])
print(m.shape)
(2, 5, 4)
rt = tf.cast(rt, tf.float32)
result = tf.ragged.map_flat_values(tf.einsum, "bi, ijk -> bjk", rt, m)
print(*(t.shape for t in result), sep='\n')
(5, 5, 4)
(3, 5, 4)
(4, 5, 4)
(4, 5, 4)
```

As expected, the batch dimension `b`

corresponds to the length of the
individual series, while the other dimensions originate from `m`

. By the
way, `tf.einsum`

refers to the Einstein summation convention, which is
extremely handy if we are working with higher dimensional tensors. Read
more about it here.

One last thing, it is also very easy to perform aggregations over ragged tensors. For example, if we want to know the columnwise sum, we can use reduction functions for this:

```
tf.reduce_sum(rt, axis=1)
<tf.Tensor: shape=(4, 2), dtype=float32, numpy=
array([[27., 21.],
[13., 33.],
[15., 29.],
[20., 26.]], dtype=float32)>
```

There exists many more operations for ragged tensors which are listed here.

## Conclusion

We learned about the structure of TensorFlow ragged tensors and how to perform basic mathematical operations on them. They make it unnecessary to apply unnatural preprocessing techniques like interpolation or padding. This is especially useful for irregular time series datasets, although there are many other applications. Imagine a dataset with images of various sizes - ragged tensors are even able to handle multiple ragged dimensions, perfect for that.

In a subsequent post I will dive a bit deeper into how to work with ragged tensors as input types for a Keras model by treating the individual time series as sets and performing attention directly on the ragged tensors. Stay tuned!