Dask for Parallel Computing in Python

In past lectures, we learned how to use numpy, pandas, and xarray to analyze various types of geoscience data. In this lecture, we address an incresingly common problem: what happens if the data we wish to analyze is "big data"

Aside: What is "Big Data"?

There is a lot of hype around the buzzword "big data" today. Some people may associate "big data" with specific sortware platforms (e.g. "Hadoop", "spark"), while, for others, "big data" means specific machine learning techniques. But I think wikipedia's definition is the most useful

Big data is data sets that are so voluminous and complex that traditional data processing application software are inadequate to deal with them.

By this definition, a great many datasets we regularly confront in Earth science are big data.

A good threshold for when data becomes difficult to deal with is when the volume of data exceeds your computer's RAM. Most modern laptops have between 2 and 16 GB of RAM. High-end workstations and servers can have 1 TB (1000 GB) or RAM or more. If the dataset you are trying to analyze can't fit in you computer's memory, some special care is required to carry out the analysis. Data that can't fit in RAM but can fit on your hard drive is sometimes called "medium data."

The next threshold of difficulty is when the data can't fit on your hard drive. Most modern laptops have between 100 GB and 4 TB of storage space on the hard drive. If you can't fit your dataset on your internal hard drive, you can buy an external hard drive. However, at that point you are better off using a high-end server, HPC system, or cloud-based storage for your dataset. Once you have many TB of data to analyze, you are definitely in the realm of "big data"

What is Dask?

Dask is a tool that helps us easily extend our familiar python data analysis tools to medium and big data, i.e. dataset that can't fit in our computer's RAM. In many cases, dask also allows us to speed up our analysis by using mutiple CPU cores. Dask can help us work more efficiently on our laptop, and it can also help us scale up our analysis on HPC and cloud platforms. Most importantly, dask is almost invisible to the user, meaning that you can focus on your science, rather than the details of parallel computing.

Dask was created by the brilliant Matt Rocklin. You can learn more about it on

Dask provides collections for big data and a scheduler for parallel computing. It is probably easiest to illustrate what these mean through examples, so we will jump right in.

Dask Arrays

A dask array looks and feels a lot like a numpy array. However, a dask array doesn't directly hold any data. Instead, it symbolically represents the computations needed to generate the data. Nothing is actually computed until the actual numerical values are needed. This mode of operation is called "lazy"; it allows one to build up complex, large calculations symbolically before turning them over the scheduler for execution.

If we want to create a numpy array of all ones, we do it like this:

In [1]:
import numpy as np
shape = (1000, 4000)
ones_np = np.ones(shape)
ones_np
Out[1]:
array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

This array contains exactly 32 MB of data:

In [2]:
ones_np.nbytes / 1e6
Out[2]:
32.0

Now let's create the same array using dask's array interface.

In [3]:
import dask.array as da
ones = da.ones(shape)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
 in ()
      1 import dask.array as da
----> 2 ones = da.ones(shape)

/opt/conda/lib/python3.6/site-packages/dask/array/wrap.py in wrap_func_shape_as_first_arg(func, *args, **kwargs)
     35     dtype = np.dtype(dtype)
     36 
---> 37     chunks = normalize_chunks(chunks, shape, dtype=dtype)
     38     name = kwargs.pop('name', None)
     39 

/opt/conda/lib/python3.6/site-packages/dask/array/core.py in normalize_chunks(chunks, shape, limit, dtype, previous_chunks)
   2094         dtype = np.dtype(dtype)
   2095     if chunks is None:
-> 2096         raise ValueError(CHUNKS_NONE_ERROR_MESSAGE)
   2097     if isinstance(chunks, list):
   2098         chunks = tuple(chunks)

ValueError: You must specify a chunks= keyword argument.
This specifies the chunksize of your array blocks.

See the following documentation page for details:
  http://dask.pydata.org/en/latest/array-creation.html#chunks

This did not work, because we didn't tell dask how to split up the array.

A crucal difference with dask is that we must specify the chunks argument. "Chunks" describes how the array is split up over many sub-arrays.

Dask Arrays source: Dask Array Documentation

There are several ways to specify chunks. In this lecture, we will use a block shape.

In [4]:
chunk_shape = (1000, 1000)
ones = da.ones(shape, chunks=chunk_shape)
ones
Out[4]:
dask.array

Notice that we just see a symbolic represetnation of the array, including its shape, dtype, and chunksize. No data has been generated yet. When we call .compute() on a dask array, the computation is trigger and the dask array becomes a numpy array.

In [5]:
ones.compute()
Out[5]:
array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

In order to understand what happened when we called .compute(), we can visualize the dask graph, the symbolic operations that make up the array

In [6]:
ones.visualize()
Out[6]:

Our array has four chunks. To generate it, dask calls np.ones four times and then concatenates this together into one array.

Rather than immediately loading a dask array (which puts all the data into RAM), it is more common to want to reduce the data somehow. For example

In [7]:
sum_of_ones = ones.sum()
sum_of_ones.visualize()
Out[7]:

Here we see dask's strategy for finding the sum. This simple example illustrates the beauty of dask: it automatically designs an algorithm appropriate for custom operations with big data.

If we make our operation more complex, the graph gets more complex.

In [8]:
fancy_calculation = (ones * ones[::-1, ::-1]).mean()
fancy_calculation.visualize()
Out[8]:

A Bigger Calculation

The examples above were toy examples; the data (32 MB) is nowhere nearly big enough to warrant the use of dask.

We can make it a lot bigger!

In [9]:
bigshape = (200000, 4000)
big_ones = da.ones(bigshape, chunks=chunk_shape)
big_ones
Out[9]:
dask.array
In [10]:
big_ones.nbytes / 1e6
Out[10]:
6400.0

This dataset is 3.2 GB, rather MB! This is probably close to or greater than the amount of available RAM than you have in your computer. Nevertheless, dask has no problem working on it.

Do not try to .visualize() this array!

When doing a big calculation, dask also has some tools to help us understand what is happening under the hood

In [11]:
from dask.diagnostics import ProgressBar

big_calc = (big_ones * big_ones[::-1, ::-1]).mean()

with ProgressBar():
    result = big_calc.compute()
result
[########################################] | 100% Completed |  8.4s
Out[11]:
1.0

Reduction

All the usual numpy methods work on dask arrays. You can also apply numpy function directly to a dask array, and it will stay lazy.

In [12]:
big_ones_reduce = (np.cos(big_ones)**2).mean(axis=0)
big_ones_reduce
Out[12]:
dask.array

Plotting also triggers computation, since we need the actual values

In [13]:
from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (12,8)
In [14]:
plt.plot(big_ones_reduce)
Out[14]:
[]

Distributed Cluster

For more fancy visualization of what dask is doing, we can use the distributed scheduler.

In [18]:
from dask_kubernetes import KubeCluster
cluster = KubeCluster(nworkers=4)
cluster
In [19]:
from dask.distributed import Client
client = Client(cluster)
client
Out[19]:

Client

Cluster

  • Workers: 0
  • Cores: 0
  • Memory: 0 B
In [16]:
big_calc.compute()
Out[16]:
1.0
In [17]:
random_values = da.random.normal(size=(2e8,), chunks=(1e6,))
hist, bins = da.histogram(random_values, bins=100, range=[-5, 5]) 
In [18]:
hist
Out[18]:
dask.array
In [19]:
x = 0.5 * (bins[1:] + bins[:-1])
width = np.diff(bins)
plt.bar(x, hist, width);

Dask + XArray

Xarray can automatically wrap its data in dask arrays. This capability turns xarray into an extremely powerful tool for Big Data earth science

To see this in action, we will download a fairly large dataset to analyze. This file contains 1 year of daily data from the AVISO sea-surface height satellite altimetry dataset.

In [24]:
! wget http://www.ldeo.columbia.edu/~rpa/aviso_madt_2015.tar.gz
--2018-11-08 15:00:20--  http://www.ldeo.columbia.edu/~rpa/aviso_madt_2015.tar.gz
Resolving www.ldeo.columbia.edu (www.ldeo.columbia.edu)... 129.236.14.15
Connecting to www.ldeo.columbia.edu (www.ldeo.columbia.edu)|129.236.14.15|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.ldeo.columbia.edu/~rpa/aviso_madt_2015.tar.gz [following]
--2018-11-08 15:00:20--  https://www.ldeo.columbia.edu/~rpa/aviso_madt_2015.tar.gz
Connecting to www.ldeo.columbia.edu (www.ldeo.columbia.edu)|129.236.14.15|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 494834680 (472M) [application/x-gzip]
Saving to: ‘aviso_madt_2015.tar.gz’

aviso_madt_2015.tar 100%[===================>] 471.91M  94.6MB/s    in 5.2s    

2018-11-08 15:00:25 (90.5 MB/s) - ‘aviso_madt_2015.tar.gz’ saved [494834680/494834680]

In [25]:
! tar -xvzf aviso_madt_2015.tar.gz
2015/
2015/dt_global_allsat_madt_h_20150409_20150914.nc
2015/dt_global_allsat_madt_h_20150406_20150914.nc
2015/dt_global_allsat_madt_h_20150704_20160128.nc
2015/dt_global_allsat_madt_h_20150223_20150914.nc
2015/dt_global_allsat_madt_h_20151220_20160610.nc
2015/dt_global_allsat_madt_h_20150912_20160610.nc
2015/dt_global_allsat_madt_h_20150607_20160128.nc
2015/dt_global_allsat_madt_h_20150320_20150914.nc
2015/dt_global_allsat_madt_h_20150608_20160128.nc
2015/dt_global_allsat_madt_h_20150512_20160128.nc
2015/dt_global_allsat_madt_h_20151029_20160610.nc
2015/dt_global_allsat_madt_h_20150618_20160128.nc
2015/dt_global_allsat_madt_h_20150502_20160128.nc
2015/dt_global_allsat_madt_h_20151026_20160610.nc
2015/dt_global_allsat_madt_h_20150330_20150914.nc
2015/dt_global_allsat_madt_h_20150617_20160128.nc
2015/dt_global_allsat_madt_h_20150826_20160128.nc
2015/dt_global_allsat_madt_h_20150416_20150914.nc
2015/dt_global_allsat_madt_h_20150829_20160128.nc
2015/dt_global_allsat_madt_h_20150419_20150914.nc
2015/dt_global_allsat_madt_h_20150126_20150914.nc
2015/dt_global_allsat_madt_h_20151230_20160610.nc
2015/dt_global_allsat_madt_h_20151125_20160610.nc
2015/dt_global_allsat_madt_h_20150129_20150914.nc
2015/dt_global_allsat_madt_h_20150714_20160128.nc
2015/dt_global_allsat_madt_h_20150310_20150914.nc
2015/dt_global_allsat_madt_h_20151006_20160610.nc
2015/dt_global_allsat_madt_h_20150522_20160128.nc
2015/dt_global_allsat_madt_h_20151009_20160610.nc
2015/dt_global_allsat_madt_h_20150922_20160610.nc
2015/dt_global_allsat_madt_h_20150905_20160128.nc
2015/dt_global_allsat_madt_h_20150213_20150914.nc
2015/dt_global_allsat_madt_h_20150109_20150914.nc
2015/dt_global_allsat_madt_h_20151105_20160610.nc
2015/dt_global_allsat_madt_h_20151210_20160610.nc
2015/dt_global_allsat_madt_h_20150106_20150914.nc
2015/dt_global_allsat_madt_h_20150809_20160128.nc
2015/dt_global_allsat_madt_h_20150806_20160128.nc
2015/dt_global_allsat_madt_h_20150116_20150914.nc
2015/dt_global_allsat_madt_h_20150203_20150914.nc
2015/dt_global_allsat_madt_h_20151115_20160610.nc
2015/dt_global_allsat_madt_h_20150119_20150914.nc
2015/dt_global_allsat_madt_h_20150724_20160128.nc
2015/dt_global_allsat_madt_h_20150816_20160128.nc
2015/dt_global_allsat_madt_h_20150819_20160128.nc
2015/dt_global_allsat_madt_h_20151019_20160610.nc
2015/dt_global_allsat_madt_h_20150628_20160128.nc
2015/dt_global_allsat_madt_h_20151016_20160610.nc
2015/dt_global_allsat_madt_h_20150627_20160128.nc
2015/dt_global_allsat_madt_h_20150615_20160128.nc
2015/dt_global_allsat_madt_h_20151024_20160610.nc
2015/dt_global_allsat_madt_h_20150824_20160128.nc
2015/dt_global_allsat_madt_h_20150414_20150914.nc
2015/dt_global_allsat_madt_h_20150716_20160128.nc
2015/dt_global_allsat_madt_h_20151127_20160610.nc
2015/dt_global_allsat_madt_h_20150719_20160128.nc
2015/dt_global_allsat_madt_h_20150124_20150914.nc
2015/dt_global_allsat_madt_h_20151128_20160610.nc
2015/dt_global_allsat_madt_h_20150404_20150914.nc
2015/dt_global_allsat_madt_h_20151222_20160610.nc
2015/dt_global_allsat_madt_h_20150709_20160128.nc
2015/dt_global_allsat_madt_h_20150221_20150914.nc
2015/dt_global_allsat_madt_h_20150706_20160128.nc
2015/dt_global_allsat_madt_h_20150510_20160128.nc
2015/dt_global_allsat_madt_h_20150322_20150914.nc
2015/dt_global_allsat_madt_h_20150605_20160128.nc
2015/dt_global_allsat_madt_h_20150726_20160128.nc
2015/dt_global_allsat_madt_h_20150201_20150914.nc
2015/dt_global_allsat_madt_h_20151117_20160610.nc
2015/dt_global_allsat_madt_h_20150729_20160128.nc
2015/dt_global_allsat_madt_h_20151202_20160610.nc
2015/dt_global_allsat_madt_h_20150114_20150914.nc
2015/dt_global_allsat_madt_h_20151118_20160610.nc
2015/dt_global_allsat_madt_h_20150814_20160128.nc
2015/dt_global_allsat_madt_h_20150625_20160128.nc
2015/dt_global_allsat_madt_h_20150302_20150914.nc
2015/dt_global_allsat_madt_h_20151014_20160610.nc
2015/dt_global_allsat_madt_h_20150530_20160128.nc
2015/dt_global_allsat_madt_h_20150930_20160610.nc
2015/dt_global_allsat_madt_h_20150520_20160128.nc
2015/dt_global_allsat_madt_h_20150312_20150914.nc
2015/dt_global_allsat_madt_h_20151004_20160610.nc
2015/dt_global_allsat_madt_h_20150907_20160128.nc
2015/dt_global_allsat_madt_h_20150908_20160128.nc
2015/dt_global_allsat_madt_h_20150920_20160610.nc
2015/dt_global_allsat_madt_h_20151212_20160610.nc
2015/dt_global_allsat_madt_h_20151108_20160610.nc
2015/dt_global_allsat_madt_h_20150104_20150914.nc
2015/dt_global_allsat_madt_h_20150211_20150914.nc
2015/dt_global_allsat_madt_h_20151107_20160610.nc
2015/dt_global_allsat_madt_h_20150804_20160128.nc
2015/dt_global_allsat_madt_h_20150929_20160610.nc
2015/dt_global_allsat_madt_h_20150901_20160128.nc
2015/dt_global_allsat_madt_h_20150926_20160610.nc
2015/dt_global_allsat_madt_h_20150526_20160128.nc
2015/dt_global_allsat_madt_h_20150529_20160128.nc
2015/dt_global_allsat_madt_h_20150314_20150914.nc
2015/dt_global_allsat_madt_h_20151002_20160610.nc
2015/dt_global_allsat_madt_h_20150802_20160128.nc
2015/dt_global_allsat_madt_h_20150425_20160128.nc
2015/dt_global_allsat_madt_h_20151214_20160610.nc
2015/dt_global_allsat_madt_h_20150218_20150914.nc
2015/dt_global_allsat_madt_h_20150102_20150914.nc
2015/dt_global_allsat_madt_h_20150730_20160128.nc
2015/dt_global_allsat_madt_h_20150217_20150914.nc
2015/dt_global_allsat_madt_h_20151101_20160610.nc
2015/dt_global_allsat_madt_h_20150422_20150914.nc
2015/dt_global_allsat_madt_h_20150812_20160128.nc
2015/dt_global_allsat_madt_h_20150207_20150914.nc
2015/dt_global_allsat_madt_h_20151111_20160610.nc
2015/dt_global_allsat_madt_h_20150720_20160128.nc
2015/dt_global_allsat_madt_h_20150208_20150914.nc
2015/dt_global_allsat_madt_h_20151204_20160610.nc
2015/dt_global_allsat_madt_h_20150112_20150914.nc
2015/dt_global_allsat_madt_h_20150911_20160128.nc
2015/dt_global_allsat_madt_h_20150304_20150914.nc
2015/dt_global_allsat_madt_h_20151012_20160610.nc
2015/dt_global_allsat_madt_h_20150623_20160128.nc
2015/dt_global_allsat_madt_h_20151224_20160610.nc
2015/dt_global_allsat_madt_h_20150228_20150914.nc
2015/dt_global_allsat_madt_h_20150227_20150914.nc
2015/dt_global_allsat_madt_h_20150402_20150914.nc
2015/dt_global_allsat_madt_h_20150516_20160128.nc
2015/dt_global_allsat_madt_h_20150603_20160128.nc
2015/dt_global_allsat_madt_h_20150519_20160128.nc
2015/dt_global_allsat_madt_h_20150324_20150914.nc
2015/dt_global_allsat_madt_h_20150919_20160610.nc
2015/dt_global_allsat_madt_h_20150916_20160610.nc
2015/dt_global_allsat_madt_h_20151022_20160610.nc
2015/dt_global_allsat_madt_h_20150613_20160128.nc
2015/dt_global_allsat_madt_h_20150509_20160128.nc
2015/dt_global_allsat_madt_h_20150506_20160128.nc
2015/dt_global_allsat_madt_h_20151121_20160610.nc
2015/dt_global_allsat_madt_h_20150710_20160128.nc
2015/dt_global_allsat_madt_h_20150122_20150914.nc
2015/dt_global_allsat_madt_h_20150822_20160128.nc
2015/dt_global_allsat_madt_h_20150412_20150914.nc
2015/dt_global_allsat_madt_h_20150420_20150914.nc
2015/dt_global_allsat_madt_h_20150810_20160128.nc
2015/dt_global_allsat_madt_h_20151206_20160610.nc
2015/dt_global_allsat_madt_h_20150110_20150914.nc
2015/dt_global_allsat_madt_h_20150722_20160128.nc
2015/dt_global_allsat_madt_h_20151209_20160610.nc
2015/dt_global_allsat_madt_h_20150205_20150914.nc
2015/dt_global_allsat_madt_h_20151113_20160610.nc
2015/dt_global_allsat_madt_h_20150309_20150914.nc
2015/dt_global_allsat_madt_h_20150621_20160128.nc
2015/dt_global_allsat_madt_h_20150306_20150914.nc
2015/dt_global_allsat_madt_h_20151010_20160610.nc
2015/dt_global_allsat_madt_h_20150924_20160610.nc
2015/dt_global_allsat_madt_h_20150903_20160128.nc
2015/dt_global_allsat_madt_h_20150316_20150914.nc
2015/dt_global_allsat_madt_h_20150319_20150914.nc
2015/dt_global_allsat_madt_h_20150524_20160128.nc
2015/dt_global_allsat_madt_h_20150215_20150914.nc
2015/dt_global_allsat_madt_h_20151219_20160610.nc
2015/dt_global_allsat_madt_h_20151103_20160610.nc
2015/dt_global_allsat_madt_h_20150428_20160128.nc
2015/dt_global_allsat_madt_h_20151216_20160610.nc
2015/dt_global_allsat_madt_h_20150427_20160128.nc
2015/dt_global_allsat_madt_h_20150504_20160128.nc
2015/dt_global_allsat_madt_h_20150611_20160128.nc
2015/dt_global_allsat_madt_h_20151020_20160610.nc
2015/dt_global_allsat_madt_h_20150120_20150914.nc
2015/dt_global_allsat_madt_h_20150712_20160128.nc
2015/dt_global_allsat_madt_h_20151123_20160610.nc
2015/dt_global_allsat_madt_h_20150820_20160128.nc
2015/dt_global_allsat_madt_h_20150410_20150914.nc
2015/dt_global_allsat_madt_h_20150225_20150914.nc
2015/dt_global_allsat_madt_h_20151229_20160610.nc
2015/dt_global_allsat_madt_h_20150702_20160128.nc
2015/dt_global_allsat_madt_h_20150130_20150914.nc
2015/dt_global_allsat_madt_h_20151226_20160610.nc
2015/dt_global_allsat_madt_h_20150830_20160128.nc
2015/dt_global_allsat_madt_h_20151030_20160610.nc
2015/dt_global_allsat_madt_h_20150326_20150914.nc
2015/dt_global_allsat_madt_h_20150601_20160128.nc
2015/dt_global_allsat_madt_h_20150329_20150914.nc
2015/dt_global_allsat_madt_h_20150514_20160128.nc
2015/dt_global_allsat_madt_h_20150914_20160610.nc
2015/dt_global_allsat_madt_h_20150123_20150914.nc
2015/dt_global_allsat_madt_h_20150711_20160128.nc
2015/dt_global_allsat_madt_h_20151120_20160610.nc
2015/dt_global_allsat_madt_h_20150413_20150914.nc
2015/dt_global_allsat_madt_h_20150823_20160128.nc
2015/dt_global_allsat_madt_h_20150507_20160128.nc
2015/dt_global_allsat_madt_h_20150508_20160128.nc
2015/dt_global_allsat_madt_h_20150612_20160128.nc
2015/dt_global_allsat_madt_h_20151023_20160610.nc
2015/dt_global_allsat_madt_h_20150325_20150914.nc
2015/dt_global_allsat_madt_h_20150518_20160128.nc
2015/dt_global_allsat_madt_h_20150602_20160128.nc
2015/dt_global_allsat_madt_h_20150517_20160128.nc
2015/dt_global_allsat_madt_h_20150917_20160610.nc
2015/dt_global_allsat_madt_h_20150918_20160610.nc
2015/dt_global_allsat_madt_h_20150226_20150914.nc
2015/dt_global_allsat_madt_h_20151130_20160610.nc
2015/dt_global_allsat_madt_h_20150701_20160128.nc
2015/dt_global_allsat_madt_h_20151225_20160610.nc
2015/dt_global_allsat_madt_h_20150403_20150914.nc
2015/dt_global_allsat_madt_h_20150910_20160128.nc
2015/dt_global_allsat_madt_h_20150622_20160128.nc
2015/dt_global_allsat_madt_h_20151013_20160610.nc
2015/dt_global_allsat_madt_h_20150305_20150914.nc
2015/dt_global_allsat_madt_h_20150813_20160128.nc
2015/dt_global_allsat_madt_h_20150423_20150914.nc
2015/dt_global_allsat_madt_h_20150113_20150914.nc
2015/dt_global_allsat_madt_h_20150209_20150914.nc
2015/dt_global_allsat_madt_h_20151205_20160610.nc
2015/dt_global_allsat_madt_h_20150721_20160128.nc
2015/dt_global_allsat_madt_h_20151110_20160610.nc
2015/dt_global_allsat_madt_h_20150206_20150914.nc
2015/dt_global_allsat_madt_h_20150803_20160128.nc
2015/dt_global_allsat_madt_h_20150216_20150914.nc
2015/dt_global_allsat_madt_h_20150731_20160128.nc
2015/dt_global_allsat_madt_h_20150103_20150914.nc
2015/dt_global_allsat_madt_h_20151215_20160610.nc
2015/dt_global_allsat_madt_h_20150219_20150914.nc
2015/dt_global_allsat_madt_h_20150424_20160128.nc
2015/dt_global_allsat_madt_h_20150927_20160610.nc
2015/dt_global_allsat_madt_h_20150928_20160610.nc
2015/dt_global_allsat_madt_h_20151003_20160610.nc
2015/dt_global_allsat_madt_h_20150315_20150914.nc
2015/dt_global_allsat_madt_h_20150528_20160128.nc
2015/dt_global_allsat_madt_h_20150527_20160128.nc
2015/dt_global_allsat_madt_h_20150515_20160128.nc
2015/dt_global_allsat_madt_h_20150328_20150914.nc
2015/dt_global_allsat_madt_h_20150327_20150914.nc
2015/dt_global_allsat_madt_h_20151031_20160610.nc
2015/dt_global_allsat_madt_h_20150915_20160610.nc
2015/dt_global_allsat_madt_h_20151227_20160610.nc
2015/dt_global_allsat_madt_h_20150131_20150914.nc
2015/dt_global_allsat_madt_h_20150703_20160128.nc
2015/dt_global_allsat_madt_h_20150224_20150914.nc
2015/dt_global_allsat_madt_h_20151228_20160610.nc
2015/dt_global_allsat_madt_h_20150401_20150914.nc
2015/dt_global_allsat_madt_h_20150831_20160128.nc
2015/dt_global_allsat_madt_h_20151122_20160610.nc
2015/dt_global_allsat_madt_h_20150713_20160128.nc
2015/dt_global_allsat_madt_h_20150121_20150914.nc
2015/dt_global_allsat_madt_h_20150411_20150914.nc
2015/dt_global_allsat_madt_h_20150821_20160128.nc
2015/.listing
2015/dt_global_allsat_madt_h_20151021_20160610.nc
2015/dt_global_allsat_madt_h_20150610_20160128.nc
2015/dt_global_allsat_madt_h_20150505_20160128.nc
2015/dt_global_allsat_madt_h_20150801_20160128.nc
2015/dt_global_allsat_madt_h_20150426_20160128.nc
2015/dt_global_allsat_madt_h_20150101_20150914.nc
2015/dt_global_allsat_madt_h_20151217_20160610.nc
2015/dt_global_allsat_madt_h_20150429_20160128.nc
2015/dt_global_allsat_madt_h_20151102_20160610.nc
2015/dt_global_allsat_madt_h_20150214_20150914.nc
2015/dt_global_allsat_madt_h_20151218_20160610.nc
2015/dt_global_allsat_madt_h_20150902_20160128.nc
2015/dt_global_allsat_madt_h_20150925_20160610.nc
2015/dt_global_allsat_madt_h_20150525_20160128.nc
2015/dt_global_allsat_madt_h_20150318_20150914.nc
2015/dt_global_allsat_madt_h_20150630_20160128.nc
2015/dt_global_allsat_madt_h_20151001_20160610.nc
2015/dt_global_allsat_madt_h_20150317_20150914.nc
2015/dt_global_allsat_madt_h_20151011_20160610.nc
2015/dt_global_allsat_madt_h_20150307_20150914.nc
2015/dt_global_allsat_madt_h_20150620_20160128.nc
2015/dt_global_allsat_madt_h_20150308_20150914.nc
2015/dt_global_allsat_madt_h_20150811_20160128.nc
2015/dt_global_allsat_madt_h_20150421_20150914.nc
2015/dt_global_allsat_madt_h_20151112_20160610.nc
2015/dt_global_allsat_madt_h_20151208_20160610.nc
2015/dt_global_allsat_madt_h_20150204_20150914.nc
2015/dt_global_allsat_madt_h_20150723_20160128.nc
2015/dt_global_allsat_madt_h_20150111_20150914.nc
2015/dt_global_allsat_madt_h_20151207_20160610.nc
2015/dt_global_allsat_madt_h_20150626_20160128.nc
2015/dt_global_allsat_madt_h_20151017_20160610.nc
2015/dt_global_allsat_madt_h_20150301_20150914.nc
2015/dt_global_allsat_madt_h_20150629_20160128.nc
2015/dt_global_allsat_madt_h_20151018_20160610.nc
2015/dt_global_allsat_madt_h_20150725_20160128.nc
2015/dt_global_allsat_madt_h_20151114_20160610.nc
2015/dt_global_allsat_madt_h_20150118_20150914.nc
2015/dt_global_allsat_madt_h_20150202_20150914.nc
2015/dt_global_allsat_madt_h_20150430_20160128.nc
2015/dt_global_allsat_madt_h_20150117_20150914.nc
2015/dt_global_allsat_madt_h_20151201_20160610.nc
2015/dt_global_allsat_madt_h_20150818_20160128.nc
2015/dt_global_allsat_madt_h_20150817_20160128.nc
2015/dt_global_allsat_madt_h_20150107_20150914.nc
2015/dt_global_allsat_madt_h_20151211_20160610.nc
2015/dt_global_allsat_madt_h_20150108_20150914.nc
2015/dt_global_allsat_madt_h_20151104_20160610.nc
2015/dt_global_allsat_madt_h_20150212_20150914.nc
2015/dt_global_allsat_madt_h_20150807_20160128.nc
2015/dt_global_allsat_madt_h_20150808_20160128.nc
2015/dt_global_allsat_madt_h_20151008_20160610.nc
2015/dt_global_allsat_madt_h_20150523_20160128.nc
2015/dt_global_allsat_madt_h_20151007_20160610.nc
2015/dt_global_allsat_madt_h_20150311_20150914.nc
2015/dt_global_allsat_madt_h_20150904_20160128.nc
2015/dt_global_allsat_madt_h_20150923_20160610.nc
2015/dt_global_allsat_madt_h_20150418_20150914.nc
2015/dt_global_allsat_madt_h_20150828_20160128.nc
2015/dt_global_allsat_madt_h_20150417_20150914.nc
2015/dt_global_allsat_madt_h_20150827_20160128.nc
2015/dt_global_allsat_madt_h_20150715_20160128.nc
2015/dt_global_allsat_madt_h_20151124_20160610.nc
2015/dt_global_allsat_madt_h_20150128_20150914.nc
2015/dt_global_allsat_madt_h_20151231_20160610.nc
2015/dt_global_allsat_madt_h_20150127_20150914.nc
2015/dt_global_allsat_madt_h_20150616_20160128.nc
2015/dt_global_allsat_madt_h_20150331_20150914.nc
2015/dt_global_allsat_madt_h_20151027_20160610.nc
2015/dt_global_allsat_madt_h_20150503_20160128.nc
2015/dt_global_allsat_madt_h_20150619_20160128.nc
2015/dt_global_allsat_madt_h_20151028_20160610.nc
2015/dt_global_allsat_madt_h_20150913_20160610.nc
2015/dt_global_allsat_madt_h_20150513_20160128.nc
2015/dt_global_allsat_madt_h_20150609_20160128.nc
2015/dt_global_allsat_madt_h_20150321_20150914.nc
2015/dt_global_allsat_madt_h_20150606_20160128.nc
2015/dt_global_allsat_madt_h_20150407_20150914.nc
2015/dt_global_allsat_madt_h_20150408_20150914.nc
2015/dt_global_allsat_madt_h_20151221_20160610.nc
2015/dt_global_allsat_madt_h_20150222_20150914.nc
2015/dt_global_allsat_madt_h_20150705_20160128.nc
2015/dt_global_allsat_madt_h_20151106_20160610.nc
2015/dt_global_allsat_madt_h_20150210_20150914.nc
2015/dt_global_allsat_madt_h_20151109_20160610.nc
2015/dt_global_allsat_madt_h_20150105_20150914.nc
2015/dt_global_allsat_madt_h_20151213_20160610.nc
2015/dt_global_allsat_madt_h_20150805_20160128.nc
2015/dt_global_allsat_madt_h_20151005_20160610.nc
2015/dt_global_allsat_madt_h_20150313_20150914.nc
2015/dt_global_allsat_madt_h_20150521_20160128.nc
2015/dt_global_allsat_madt_h_20150921_20160610.nc
2015/dt_global_allsat_madt_h_20150909_20160128.nc
2015/dt_global_allsat_madt_h_20150906_20160128.nc
2015/dt_global_allsat_madt_h_20150531_20160128.nc
2015/dt_global_allsat_madt_h_20151015_20160610.nc
2015/dt_global_allsat_madt_h_20150303_20150914.nc
2015/dt_global_allsat_madt_h_20150624_20160128.nc
2015/dt_global_allsat_madt_h_20150115_20150914.nc
2015/dt_global_allsat_madt_h_20151119_20160610.nc
2015/dt_global_allsat_madt_h_20151203_20160610.nc
2015/dt_global_allsat_madt_h_20150728_20160128.nc
2015/dt_global_allsat_madt_h_20151116_20160610.nc
2015/dt_global_allsat_madt_h_20150727_20160128.nc
2015/dt_global_allsat_madt_h_20150815_20160128.nc
2015/dt_global_allsat_madt_h_20150604_20160128.nc
2015/dt_global_allsat_madt_h_20150323_20150914.nc
2015/dt_global_allsat_madt_h_20150511_20160128.nc
2015/dt_global_allsat_madt_h_20150405_20150914.nc
2015/dt_global_allsat_madt_h_20150707_20160128.nc
2015/dt_global_allsat_madt_h_20150220_20150914.nc
2015/dt_global_allsat_madt_h_20150708_20160128.nc
2015/dt_global_allsat_madt_h_20151223_20160610.nc
2015/dt_global_allsat_madt_h_20150415_20150914.nc
2015/dt_global_allsat_madt_h_20150825_20160128.nc
2015/dt_global_allsat_madt_h_20150125_20150914.nc
2015/dt_global_allsat_madt_h_20151129_20160610.nc
2015/dt_global_allsat_madt_h_20150718_20160128.nc
2015/dt_global_allsat_madt_h_20151126_20160610.nc
2015/dt_global_allsat_madt_h_20150717_20160128.nc
2015/dt_global_allsat_madt_h_20150501_20160128.nc
2015/dt_global_allsat_madt_h_20151025_20160610.nc
2015/dt_global_allsat_madt_h_20150614_20160128.nc
In [22]:
! ls 2015 | wc -l
     365

Let's load the first file as a regular xarray dataset.

In [29]:
import xarray as xr
In [30]:
ds_first = xr.open_dataset('2015/dt_global_allsat_madt_h_20150101_20150914.nc')
ds_first
Out[30]:

Dimensions:   (lat: 720, lon: 1440, nv: 2, time: 1)
Coordinates:
  * time      (time) datetime64[ns] 2015-01-01
  * lat       (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ...
  * lon       (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ...
  * nv        (nv) int32 0 1
Data variables:
    lat_bnds  (lat, nv) float32 ...
    lon_bnds  (lon, nv) float32 ...
    crs       int32 ...
    adt       (time, lat, lon) float64 ...
Attributes:
    Conventions:                     CF-1.6
    Metadata_Conventions:            Unidata Dataset Discovery v1.0
    cdm_data_type:                   Grid
    comment:                         Absolute Dynamic Topography
    contact:                         aviso@altimetry.fr
    creator_email:                   aviso@altimetry.fr
    creator_name:                    SSALTO/DUACS
    creator_url:                     http://www.aviso.altimetry.fr
    date_created:                    2015-10-23T17:07:42Z
    date_issued:                     2015-09-14T00:00:00Z
    date_modified:                   2015-11-13T16:23:24Z
    geospatial_lat_max:              89.875
    geospatial_lat_min:              -89.875
    geospatial_lat_resolution:       0.25
    geospatial_lat_units:            degrees_north
    geospatial_lon_max:              359.875
    geospatial_lon_min:              0.125
    geospatial_lon_resolution:       0.25
    geospatial_lon_units:            degrees_east
    geospatial_vertical_max:         0.0
    geospatial_vertical_min:         0.0
    geospatial_vertical_positive:    down
    geospatial_vertical_resolution:  point
    geospatial_vertical_units:       m
    history:                         2015-10-23T17:07:42Z: created by DUACS D...
    institution:                     CNES, CLS
    keywords:                        Oceans > Ocean Topography > Sea Surface ...
    keywords_vocabulary:             NetCDF COARDS Climate and Forecast Stand...
    license:                         http://www.aviso.altimetry.fr/fileadmin/...
    platform:                        AltiKa, Cryosat-2, Haiyang-2A, OSTM/Jason-2
    processing_level:                L4
    product_version:                 5.7
    project:                         SSALTO/DUACS :  Data Unification and Alt...
    references:                      http://www.aviso.altimetry.fr
    source:                          Altimetry measurements
    ssalto_duacs_comment:            The reference mission used for the altim...
    standard_name_vocabulary:        NetCDF Climate and Forecast (CF) Metadat...
    summary:                         Delayed-Time Level-4 sea surface height ...
    time_coverage_duration:          P1D
    time_coverage_end:               2015-01-01T12:00:00Z
    time_coverage_resolution:        P1D
    time_coverage_start:             2014-12-31T12:00:00Z
    title:                           DT merged all satellites Global Ocean Gr...
In [31]:
ds_first.nbytes / 1e6
Out[31]:
8.32034

This one file is about 8 MB. So 365 of them will be nearly 3 GB. If we had downloaded all 25 years of data, it would be 73 GB. This is a good example of "medium data."

open_mfdataset

An incredibly useful function in xarray is open_mfdataset.

In [32]:
help(xr.open_mfdataset)
Help on function open_mfdataset in module xarray.backends.api:

open_mfdataset(paths, chunks=None, concat_dim='__infer_concat_dim__', compat='no_conflicts', preprocess=None, engine=None, lock=None, data_vars='all', coords='different', autoclose=False, parallel=False, **kwargs)
    Open multiple files as a single dataset.
    
    Requires dask to be installed. See documentation for details on dask [1].
    Attributes from the first dataset file are used for the combined dataset.
    
    Parameters
    ----------
    paths : str or sequence
        Either a string glob in the form "path/to/my/files/*.nc" or an explicit
        list of files to open.  Paths can be given as strings or as pathlib
        Paths.
    chunks : int or dict, optional
        Dictionary with keys given by dimension names and values given by chunk
        sizes. In general, these should divide the dimensions of each dataset.
        If int, chunk each dimension by ``chunks``.
        By default, chunks will be chosen to load entire input files into
        memory at once. This has a major impact on performance: please see the
        full documentation for more details [2].
    concat_dim : None, str, DataArray or Index, optional
        Dimension to concatenate files along. This argument is passed on to
        :py:func:`xarray.auto_combine` along with the dataset objects. You only
        need to provide this argument if the dimension along which you want to
        concatenate is not a dimension in the original datasets, e.g., if you
        want to stack a collection of 2D arrays along a third dimension.
        By default, xarray attempts to infer this argument by examining
        component files. Set ``concat_dim=None`` explicitly to disable
        concatenation.
    compat : {'identical', 'equals', 'broadcast_equals',
              'no_conflicts'}, optional
        String indicating how to compare variables of the same name for
        potential conflicts when merging:
    
        - 'broadcast_equals': all values must be equal when variables are
          broadcast against each other to ensure common dimensions.
        - 'equals': all values and dimensions must be the same.
        - 'identical': all values, dimensions and attributes must be the
          same.
        - 'no_conflicts': only values which are not null in both datasets
          must be equal. The returned dataset then contains the combination
          of all non-null values.
    preprocess : callable, optional
        If provided, call this function on each dataset prior to concatenation.
    engine : {'netcdf4', 'scipy', 'pydap', 'h5netcdf', 'pynio'}, optional
        Engine to use when reading files. If not provided, the default engine
        is chosen based on available dependencies, with a preference for
        'netcdf4'.
    autoclose : bool, optional
        If True, automatically close files to avoid OS Error of too many files
        being open.  However, this option doesn't work with streams, e.g.,
        BytesIO.
    lock : False, True or threading.Lock, optional
        This argument is passed on to :py:func:`dask.array.from_array`. By
        default, a per-variable lock is used when reading data from netCDF
        files with the netcdf4 and h5netcdf engines to avoid issues with
        concurrent access when using dask's multithreaded backend.
    data_vars : {'minimal', 'different', 'all' or list of str}, optional
        These data variables will be concatenated together:
          * 'minimal': Only data variables in which the dimension already
            appears are included.
          * 'different': Data variables which are not equal (ignoring
            attributes) across all datasets are also concatenated (as well as
            all for which dimension already appears). Beware: this option may
            load the data payload of data variables into memory if they are not
            already loaded.
          * 'all': All data variables will be concatenated.
          * list of str: The listed data variables will be concatenated, in
            addition to the 'minimal' data variables.
    coords : {'minimal', 'different', 'all' o list of str}, optional
        These coordinate variables will be concatenated together:
          * 'minimal': Only coordinates in which the dimension already appears
            are included.
          * 'different': Coordinates which are not equal (ignoring attributes)
            across all datasets are also concatenated (as well as all for which
            dimension already appears). Beware: this option may load the data
            payload of coordinate variables into memory if they are not already
            loaded.
          * 'all': All coordinate variables will be concatenated, except
            those corresponding to other dimensions.
          * list of str: The listed coordinate variables will be concatenated,
            in addition the 'minimal' coordinates.
    parallel : bool, optional
        If True, the open and preprocess steps of this function will be
        performed in parallel using ``dask.delayed``. Default is False.
    **kwargs : optional
        Additional arguments passed on to :py:func:`xarray.open_dataset`.
    
    Returns
    -------
    xarray.Dataset
    
    See Also
    --------
    auto_combine
    open_dataset
    
    References
    ----------
    .. [1] http://xarray.pydata.org/en/stable/dask.html
    .. [2] http://xarray.pydata.org/en/stable/dask.html#chunking-and-performance

Using open_mfdataset we can easily open all the netcdf files into one Dataset object.

In [33]:
# On I got a "Too many open files" OSError.
# It's only 365 files. That shouldn't be too many. 
# However, I discovered my ulimit was extremely low.
# One workaround is to call 
#  $ ulimit -S -n 4000
# from the command line before launching the notebook

ds = xr.open_mfdataset('2015/*.nc')
ds
Out[33]:

Dimensions:   (lat: 720, lon: 1440, nv: 2, time: 365)
Coordinates:
  * lat       (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ...
  * lon       (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ...
  * nv        (nv) int32 0 1
  * time      (time) datetime64[ns] 2015-01-01 2015-01-02 2015-01-03 ...
Data variables:
    lat_bnds  (time, lat, nv) float32 dask.array
    lon_bnds  (time, lon, nv) float32 dask.array
    crs       (time) int32 -2147483647 -2147483647 -2147483647 -2147483647 ...
    adt       (time, lat, lon) float64 dask.array
Attributes:
    Conventions:                     CF-1.6
    Metadata_Conventions:            Unidata Dataset Discovery v1.0
    cdm_data_type:                   Grid
    comment:                         Absolute Dynamic Topography
    contact:                         aviso@altimetry.fr
    creator_email:                   aviso@altimetry.fr
    creator_name:                    SSALTO/DUACS
    creator_url:                     http://www.aviso.altimetry.fr
    date_created:                    2015-10-23T17:07:42Z
    date_issued:                     2015-09-14T00:00:00Z
    date_modified:                   2015-11-13T16:23:24Z
    geospatial_lat_max:              89.875
    geospatial_lat_min:              -89.875
    geospatial_lat_resolution:       0.25
    geospatial_lat_units:            degrees_north
    geospatial_lon_max:              359.875
    geospatial_lon_min:              0.125
    geospatial_lon_resolution:       0.25
    geospatial_lon_units:            degrees_east
    geospatial_vertical_max:         0.0
    geospatial_vertical_min:         0.0
    geospatial_vertical_positive:    down
    geospatial_vertical_resolution:  point
    geospatial_vertical_units:       m
    history:                         2015-10-23T17:07:42Z: created by DUACS D...
    institution:                     CNES, CLS
    keywords:                        Oceans > Ocean Topography > Sea Surface ...
    keywords_vocabulary:             NetCDF COARDS Climate and Forecast Stand...
    license:                         http://www.aviso.altimetry.fr/fileadmin/...
    platform:                        AltiKa, Cryosat-2, Haiyang-2A, OSTM/Jason-2
    processing_level:                L4
    product_version:                 5.7
    project:                         SSALTO/DUACS :  Data Unification and Alt...
    references:                      http://www.aviso.altimetry.fr
    source:                          Altimetry measurements
    ssalto_duacs_comment:            The reference mission used for the altim...
    standard_name_vocabulary:        NetCDF Climate and Forecast (CF) Metadat...
    summary:                         Delayed-Time Level-4 sea surface height ...
    time_coverage_duration:          P1D
    time_coverage_end:               2015-01-01T12:00:00Z
    time_coverage_resolution:        P1D
    time_coverage_start:             2014-12-31T12:00:00Z
    title:                           DT merged all satellites Global Ocean Gr...

Note that the values are not displayed, since that would trigger computation.

In [5]:
ssh = ds.adt
ssh
Out[5]:

dask.array
Coordinates:
  * lat      (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ...
  * lon      (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ...
  * time     (time) datetime64[ns] 2015-01-01T00:01:44.499838976 ...
Attributes:
    grid_mapping:   crs
    long_name:      Absolute Dynamic Topography
    standard_name:  sea_surface_height_above_geoid
    units:          m
In [8]:
ssh[0].plot()
Out[8]:
In [9]:
ssh_2015_mean = ssh.mean(dim='time')
ssh_2015_mean.load()
Out[9]:

array([[ nan,  nan,  nan, ...,  nan,  nan,  nan],
       [ nan,  nan,  nan, ...,  nan,  nan,  nan],
       [ nan,  nan,  nan, ...,  nan,  nan,  nan],
       ..., 
       [  0.,   0.,   0., ...,   0.,   0.,   0.],
       [  0.,   0.,   0., ...,   0.,   0.,   0.],
       [  0.,   0.,   0., ...,   0.,   0.,   0.]])
Coordinates:
  * lat      (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ...
  * lon      (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ...
In [10]:
ssh_2015_mean.plot()
Out[10]:
In [11]:
ssh_anom = ssh - ssh_2015_mean
ssh_variance_lonmean = (ssh_anom**2).mean(dim=('lon', 'time'))
In [12]:
ssh_variance_lonmean.plot()
Out[12]:
[]
In [15]:
weight = np.cos(np.deg2rad(ds.lat))
weight /= weight.mean()
(ssh_anom * weight).mean(dim=('lon', 'lat')).plot()
Out[15]:
[]

Cloud Storage

In [35]:
import gcsfs
gcsmap = gcsfs.mapping.GCSMap('pangeo-data/dataset-duacs-rep-global-merged-allsat-phy-l4-v3-alt')
ds = xr.open_zarr(gcsmap)
ds
Out[35]:

Dimensions:    (latitude: 720, longitude: 1440, nv: 2, time: 8901)
Coordinates:
    crs        int32 ...
    lat_bnds   (time, latitude, nv) float32 dask.array
  * latitude   (latitude) float32 -89.875 -89.625 -89.375 -89.125 -88.875 ...
    lon_bnds   (longitude, nv) float32 dask.array
  * longitude  (longitude) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 ...
  * nv         (nv) int32 0 1
  * time       (time) datetime64[ns] 1993-01-01 1993-01-02 1993-01-03 ...
Data variables:
    adt        (time, latitude, longitude) float64 dask.array
    err        (time, latitude, longitude) float64 dask.array
    sla        (time, latitude, longitude) float64 dask.array
    ugos       (time, latitude, longitude) float64 dask.array
    ugosa      (time, latitude, longitude) float64 dask.array
    vgos       (time, latitude, longitude) float64 dask.array
    vgosa      (time, latitude, longitude) float64 dask.array
Attributes:
    Conventions:                     CF-1.6
    Metadata_Conventions:            Unidata Dataset Discovery v1.0
    cdm_data_type:                   Grid
    comment:                         Sea Surface Height measured by Altimetry...
    contact:                         servicedesk.cmems@mercator-ocean.eu
    creator_email:                   servicedesk.cmems@mercator-ocean.eu
    creator_name:                    CMEMS - Sea Level Thematic Assembly Center
    creator_url:                     http://marine.copernicus.eu
    date_created:                    2014-02-26T16:09:13Z
    date_issued:                     2014-01-06T00:00:00Z
    date_modified:                   2015-11-10T19:42:51Z
    geospatial_lat_max:              89.875
    geospatial_lat_min:              -89.875
    geospatial_lat_resolution:       0.25
    geospatial_lat_units:            degrees_north
    geospatial_lon_max:              359.875
    geospatial_lon_min:              0.125
    geospatial_lon_resolution:       0.25
    geospatial_lon_units:            degrees_east
    geospatial_vertical_max:         0.0
    geospatial_vertical_min:         0.0
    geospatial_vertical_positive:    down
    geospatial_vertical_resolution:  point
    geospatial_vertical_units:       m
    history:                         2014-02-26T16:09:13Z: created by DUACS D...
    institution:                     CLS, CNES
    keywords:                        Oceans > Ocean Topography > Sea Surface ...
    keywords_vocabulary:             NetCDF COARDS Climate and Forecast Stand...
    license:                         http://marine.copernicus.eu/web/27-servi...
    platform:                        ERS-1, Topex/Poseidon
    processing_level:                L4
    product_version:                 5.0
    project:                         COPERNICUS MARINE ENVIRONMENT MONITORING...
    references:                      http://marine.copernicus.eu
    source:                          Altimetry measurements
    ssalto_duacs_comment:            The reference mission used for the altim...
    standard_name_vocabulary:        NetCDF Climate and Forecast (CF) Metadat...
    summary:                         SSALTO/DUACS Delayed-Time Level-4 sea su...
    time_coverage_duration:          P1D
    time_coverage_end:               1993-01-01T12:00:00Z
    time_coverage_resolution:        P1D
    time_coverage_start:             1992-12-31T12:00:00Z
    title:                           DT merged all satellites Global Ocean Gr...
In [ ]: