From describing a sequence of numbers to estimating how often a storm returns. We build step by step, from simple to complex, using actual data from the Danube Delta and the Black Sea.
Three questions come back over and over with environmental data: what do the data look like (the distribution), how do they change over time (trend and season) and how rare is something extreme (the return period). The five parts below answer exactly these.
Before any statistics, ask yourself what shape the data have. The shape decides how hard it is — both in volume and in code.
Almost everything we use fits into a few structures, depending on how many dimensions they have. A time series is a single point tracked over time (a sequence of numbers) — the easiest to analyse. A grid / raster is a 2D spatial matrix of values (a map); when you track it over time it becomes a 3D cube (map × time) — much harder, because both volume and code complexity grow. If you add depth or atmospheric levels, you reach a 4D field — which we won't cover today.
numpy / pandas · easyxarray / rasterio · mediumxarray + NetCDF · harder (volume)xarray + dask · hard — not coveredThe practical trick for today: from a grid you can extract a single cell (the one nearest to the Danube Delta) and you're back to a simple time series — on which everything that follows applies directly. Analysis over full grids (pixel-by-pixel trend maps) uses the same methods, just repeated over millions of cells.
There are other structures you'll meet:
geopandas, scipy · require interpolation.geopandas · spatial operations, not pixel statistics.First reflex with any new sequence: summarise it in a few numbers and look at its shape.
A few numbers summarise the centre, the spread and the shape. For the centre: the mean (the "centre of mass") and the median (the middle value). For spread: the standard deviation and the variance (its square), the IQR (interquartile range, Q3−Q1) and the coefficient of variation (CV = std/mean, relative spread). For shape: skewness (how tilted the distribution is) and kurtosis (how "heavy" the tails are). And percentiles (e.g. P95 = the threshold below which 95% of values lie) describe any point of the distribution.
The median, the IQR and the percentiles are called robust statistics: they aren't pulled by a few huge values. The mean, the standard deviation and especially skewness/kurtosis are, on the contrary, sensitive to extremes — a single storm wave noticeably shifts them.
But where do these numbers come from? From a population of values. Environmental data usually appear as a time series: a variable measured at successive moments (e.g. wave height, daily, over a year).
If you count how many measurements fall in each value interval, you obtain a distribution — a map of how often each value appears. The histogram is exactly that: each measurement "falls" into a column, and the column height shows how frequent that value is.
The difference between mean and median is key for environmental data, because many are asymmetric. Wave height, discharge, turbidity — all have a long tail toward large values (storms, floods) and a low peak near the small values. For such distributions the mean is larger than the median and can mislead.
Three plots tell you almost everything: the histogram (shape of the distribution), the boxplot (median, quartiles, outliers) and the ECDF (cumulative proportion — you read directly "what fraction of the time the wave exceeds 2 m").
Many environmental variables follow known distributions, and recognising which is useful: it tells you which statistics make sense and is the basis of the extreme analysis in Part 4.
How do you check that a distribution "fits"? Visually with a QQ-plot (data quantiles vs. theoretical quantiles — if they lie on the diagonal, it fits), and numerically with a Kolmogorov-Smirnov or Anderson-Darling test.
import numpy as np import pandas as pd import matplotlib.pyplot as plt # vhm0 = pandas.Series with significant wave height (m), time-indexed print(vhm0.describe()) # count, mean, std, min, 25/50/75%, max print("median:", vhm0.median()) print("P95:", vhm0.quantile(0.95)) print("P99:", vhm0.quantile(0.99)) fig, ax = plt.subplots(1, 3, figsize=(12, 3.5)) vhm0.hist(bins=40, ax=ax[0]); ax[0].set_title("Histogram") ax[1].boxplot(vhm0.dropna()); ax[1].set_title("Boxplot") # ECDF: what fraction of the time the wave is below a given value x = np.sort(vhm0.dropna().values) y = np.arange(1, len(x)+1) / len(x) ax[2].plot(x, y); ax[2].set_title("ECDF") plt.tight_layout(); plt.show() # fit a distribution and check with a QQ-plot + KS test from scipy import stats par = stats.lognorm.fit(vhm0.dropna()) # e.g. log-normal stats.probplot(vhm0.dropna(), dist=stats.lognorm, sparams=par, plot=plt) # QQ-plot D, p = stats.kstest(vhm0.dropna(), "lognorm", par) print(f"KS: D={D:.3f}, p={p:.3g}") # large p = good fit
Environmental data almost always have an annual rhythm. You have to separate it from real change.
Three related concepts: climatology is the long-term average for each month (or day of year) — "what a typical February looks like". The seasonal cycle is exactly this recurring pattern. The anomaly is what remains after you subtract the climatology: anomaly = value − the average of that month. Anomalies are where you see trends and events, without the season hiding them.
Also useful: the rolling mean: it smooths day-to-day noise so you can see the pattern. Careful, though — smoothing doesn't "create" information, it just makes it more readable.
All of this boils down to a single idea: any environmental series can be decomposed into three pieces — the slow trend, the recurring seasonal cycle, and the rest (noise + events). That is what the STL method from the extended toolkit does.
# sst = pandas.Series monthly (°C) with a DatetimeIndex clim = sst.groupby(sst.index.month).transform("mean") # monthly climatology anom = sst - clim # SST anomaly anom.plot(title="SST anomaly relative to the monthly climatology") # 12-month rolling mean to see the smooth trend anom.rolling(12, center=True).mean().plot() # xarray equivalent (gridded data), using groupby on month: # clim = ds.tos.groupby("time.month").mean() # anom = ds.tos.groupby("time.month") - clim
Now we ask: is something growing over time? and do two things move together?
The simplest trend estimator is linear regression: you fit the best straight line through the data, and its slope is the change per unit of time (mm/yr, °C/decade). Easy to compute, but sensitive to outliers.
The slope tells you how much things change, but not how well the line describes the data. For that we use R² (the coefficient of determination): the fraction of the data's variance explained by the model, between 0 and 1. R² = 0.9 means the line explains 90% of the spread; R² = 0.1 means 90% remains noise. For simple linear regression, R² is just the square of the Pearson correlation.
Here lies the subtle and very common pitfall in environmental data: significance (p) and strength (R²) are different things. With long series you can obtain a "significant" trend (small p) that still explains only a few percent of variability (small R²). Both panels below have the same slope and can both be "significant" — but confidence in them is radically different.
The other half of the story is the p-value (or p-score): the probability of obtaining a slope at least as large if in fact there is no trend (the null hypothesis). In other words: how often would you see "accidentally" a slope like yours if you shuffled the data in time? Small p = hard to explain by chance (so the slope is probably real). The usual threshold is p < 0.05. Caution: small p ≠ important trend — over long series, even a tiny slope becomes "significant".
The third metric is RMSE (root-mean-square error): the typical prediction error, in the real units of the variable (cm, °C, m³/s). Computed as RMSE = √(mean of residuals²) — that is, an average of the vertical distances between the real data and the model line. Unlike R² (unitless, abstract), RMSE is directly interpretable: "my prediction is off by ±X cm on average". It's useful when you want to size the margin of error of a model.
For environmental data we often prefer a robust, non-parametric duo: the Mann-Kendall test says whether there is a monotonic trend (without assuming linearity or normality), and the Sen slope (Theil-Sen) estimates how large it is, resistant to outliers. It's the standard in hydrology and climatology.
When you want to see whether two variables move together, you use correlation: the Pearson coefficient measures the linear relationship, and Spearman (on ranks) measures any monotonic relationship — better for asymmetric data. Remember the golden rule: correlation does not imply causation.
from scipy import stats import numpy as np # --- linear regression (simple trend) --- t = np.arange(len(sla)) # sla = sea level (cm) res = stats.linregress(t, sla.values) print(f"slope = {res.slope*12:.2f} cm/yr, p = {res.pvalue:.3g}") print(f"R² = {res.rvalue**2:.2f}") # share of variance explained by the trend pred = res.intercept + res.slope*t rmse = np.sqrt(np.mean((sla.values-pred)**2)) print(f"RMSE = {rmse:.2f} cm") # typical error, in real units # --- Mann-Kendall + Sen slope (robust) --- import pymannkendall as mk # pip install pymannkendall out = mk.original_test(sla.values) print(out.trend, "| Sen slope:", out.slope, "| p:", out.p) # --- correlations --- r_p, _ = stats.pearsonr(discharge, turbidity) # linear r_s, _ = stats.spearmanr(discharge, turbidity) # monotonic, robust print(f"Pearson={r_p:.2f} Spearman={r_s:.2f}")
R² = 1 − SS_res/SS_tot.RMSE = √(mean of residuals²). Unlike R² (unitless), RMSE is directly interpretable.The pitfall to remember: small p does not mean an important trend. With decades of data, even a tiny slope becomes "statistically significant" (small p), but if R² is small and RMSE is large, the trend explains very little and the prediction remains weak. Always report all three.
For the coast, means matter less than extremes. The real question: how big is the wave that occurs once in 100 years?
We start with a bridge concept: the quantile. A percentile is just a quantile in percent — P95 is the value below which 95% of the data lie. On the cumulative plot (CDF/ECDF) you read it the other way around: pick a probability on the vertical axis, go across to the curve, and drop down to the value. This operation — "from probability to value" — is exactly what a return period does.
p = 1 − 1/T of the annual maxima: T=100 → quantile 0.99.Quantiles also have a role you've already met: quantile mapping, the bias-correction method that aligns a model's quantiles with the observed ones. Still percentiles, different use.
Now the extremes themselves. A return period of 100 years means an event with a 1% probability of being exceeded in any given year — not "exactly once every 100 years". The corresponding level is called the return level.
Two classical approaches. Block maxima: take the maximum of each year and fit a GEV distribution (Generalized Extreme Value). Peaks-over-threshold: take all the peaks above a high threshold and fit a GPD distribution (Generalized Pareto) — uses more data but requires choosing a threshold. For a quick start, the empirical return period (from the rank of the values) is enough to grasp the idea.
from scipy import stats import numpy as np # vhm0 = hourly series of wave height (m) maxima = vhm0.resample("YE").max().dropna() # one maximum per year # fit a GEV distribution (block maxima) shape, loc, scale = stats.genextreme.fit(maxima.values) def return_level(T): p = 1 - 1.0/T # non-exceedance probability return stats.genextreme.ppf(p, shape, loc, scale) for T in (10, 50, 100): print(f"H_s at {T} years = {return_level(T):.2f} m") # EMPIRICAL return period (quick intuition, no fit): x = np.sort(maxima.values)[::-1] rank = np.arange(1, len(x)+1) T_emp = (len(x)+1) / rank # Weibull formula # confidence interval via bootstrap (uncertainty matters!) rng = np.random.default_rng(0) boot = [] for _ in range(1000): s = rng.choice(maxima.values, len(maxima), replace=True) f, l, sc = stats.genextreme.fit(s) boot.append(stats.genextreme.ppf(0.99, f, l, sc)) lo, hi = np.percentile(boot, [2.5, 97.5]) print(f"H_s 100 yr: 95% CI = [{lo:.2f}, {hi:.2f}] m")
Almost all mistakes in environmental data analysis boil down to eight. Recognise them and you've won half the battle.
Consecutive values (daily, monthly) resemble each other, so the "effective" number of independent observations is smaller than it looks. That overestimates the significance of a trend: a small p can be illusory. Use the effective number of degrees of freedom or tests that account for autocorrelation (e.g. modified Mann-Kendall).
The mean, the standard deviation and the t-test assume symmetric distributions. Waves, discharge and turbidity are not. Prefer the median, percentiles and robust or non-parametric methods (Spearman, Sen).
If you compute the trend in 10,000 pixels, ~500 will look "significant" at p<0.05 by chance alone. Correct (False Discovery Rate / Benjamini-Hochberg) when making gridded trend maps.
A short series plus large natural variability can "look like" a trend that doesn't exist. You need long series — hence the value of multi-decadal reanalyses. Don't extrapolate a trend from 5 years.
In CMIP6, the spread between models is often larger than the difference between scenarios in the short term. Report the ensemble median + interval, never a single model as a certainty.
SST, NDVI and Lake Water Quality have gaps (clouds). Don't naively interpolate over long gaps and don't confuse a _FillValue with a real zero — always check the data mask.
Drifting sensors, wrong units, real outliers (a storm) vs. artefacts (an instrument spike). Don't run statistics "blind": check the distribution, look at min/max, exclude physically impossible values. A single -9999 left in the series wrecks the mean, standard deviation and trend. Plot before you compute.
Using statistics designed for the wrong kind of distribution: the mean (and standard deviation) when the distribution is asymmetric or heavy-tailed, instead of median + IQR. For discharge, waves, turbidity — extremes pull the mean up, and the "typical value" becomes a fiction. Choose your statistic after looking at the histogram, not before.
Beyond the five parts, here are the methods you'll most often encounter on delta data. You can tackle them in the AI-assisted coding session: you understand when they're used, and you ask the assistant for the code.
Resample the data with replacement thousands of times and recompute the statistic — you get confidence intervals without assuming a distribution.
scipy.stats.bootstrapCompare two groups: t-test if the data are ~normal, Mann-Whitney U otherwise. Is the observed difference real or by chance?
scipy.statsThe ACF shows how much the series "remembers" its past; the periodogram / FFT extracts hidden cycles (season, seiches, tides).
statsmodels.acf, numpy.fftSplits a series into trend + season + residual, robust to outliers. The formal version of Part 2.
statsmodels STLEstimates the trend not of the mean but of a percentile (e.g. P95). Climate change often affects extremes more than the mean.
statsmodels QuantRegReduce a gridded field to a few dominant modes — the spatial patterns that explain most of the variance.
sklearn.PCA, eofsGroup pixels or observations into similar regimes — k-means (hard membership) or fuzzy c-means (gradual membership).
sklearn, scikit-fuzzyFrom sparse measurements to a continuous map, with point-wise uncertainty — superior to simple interpolation.
pykrige, scipy.griddataThree tools that squeeze far more out of a time series than the mean and the trend.
Any hidden rhythm shows in frequency. The power spectrum (FFT / periodogram) shows which cycles dominate — annual, half-yearly, tidal. Wavelet goes further: it shows when each cycle appears, useful for non-stationary signals (a storm event, a shifting regime).
import numpy as np # synthetic example: three sinusoids + noise t = np.linspace(0, 1, 1000) x = (18*np.sin(2*np.pi*3*t) + 11*np.sin(2*np.pi*7*t) + 7*np.sin(2*np.pi*13*t) + np.random.normal(0, 3, t.size)) freq = np.fft.rfftfreq(t.size, d=t[1]-t[0]) power = np.abs(np.fft.rfft(x - x.mean()))**2 # the spectrum peaks fall at 3, 7 and 13 # continuous wavelet for non-stationary signals: import pywt coef, _ = pywt.cwt(x, np.arange(1,128), "morl")
Two series can be linked, but shifted. Cross-correlation (CCF) looks for the lag at which the correlation is maximum — that is, by how much one signal lags the other.
from scipy import signal ccf = signal.correlate(turbid - turbid.mean(), discharge - discharge.mean(), mode="full") lags = signal.correlation_lags(len(turbid), len(discharge)) lag_max = lags[ccf.argmax()] # lag with maximum correlation
Sometimes there is no smooth trend but a rupture: the mean or the slope changes abruptly at a given moment. Tests like Pettitt or CUSUM find that point.
import ruptures as rpt # pip install ruptures algo = rpt.Pelt(model="rbf").fit(series.values) points = algo.predict(pen=10) # indices of the breaks
Two methods tailored to marine data — exactly your terrain.
Direction is circular data: 359° and 1° are neighbours, so the arithmetic mean makes no sense (the vector mean is used). The wave rose shows which directions waves come from and with what intensity.
from windrose import WindroseAxes # pip install windrose ax = WindroseAxes.from_ax() ax.bar(direction, hs, normed=True, bins=[0,1,2,3]) # circular mean: from scipy.stats import circmean dir_mean = circmean(direction, high=360)
Wave climate isn't just height, it's the height–period pair. A 2D density (or scatter) shows which combinations occur most often — the basis for computing energy and loading on structures.
import matplotlib.pyplot as plt plt.hexbin(tp, hs, gridsize=30, cmap="viridis") # or seaborn.kdeplot(x=tp, y=hs)
When the data are gridded fields, statistics can extract the dominant spatial patterns.
A gridded field (SST, sea level) varies in time and space. EOF (PCA on geophysical data) decomposes it into a few dominant modes: each has a fixed spatial pattern and a time series that says how active that pattern is.
from eofs.xarray import Eof # pip install eofs solver = Eof(sst_anom) modes = solver.eofs(neofs=3) # the spatial patterns pcs = solver.pcs(npcs=3) # the time series