retostauffer.org

R/exams upgrades, features, and improvements

2025-05-02T00:00:00+02:00

Over the past weeks and months we have been updating both the R/exams package as well as it’s newest extension exams2forms to resolve known (and new) issues as well as extend the functionality of both packages. I would like to thank everyone involved in the latest additions and improvements!

I, myself, was mainly involved in (i) extending and refining the stresstest functionalilty to automatically stress-testing R/exams exercise questions, as well as (ii) refining and extending the functionality of the R-package exams2forms which now provides additional features for our users developing and testing R/exams exercise questions.

R/exams

Stresstesting exercises

For quite some time, the R/exams package provides automated stress-tests for (one or multiple) exercises using the stresstest_exercise() function (see e.g., our tutorial on Stress Testing Dynamic R/exams Exercises).

Whilst always working perfectly fine, one inconvenience was that the function may ‘stall’ if an exercise was taking an unexpectedly long time to be generated (or ended up in an infinite loop in the worst case).

The new release of R/exams now allows to set a timeout. This useful if one wants to identify exercises which take “too long” to be rendered, or exercises which may end up in infinite loops. It becomes especially useful if one want’s to check or test extensive libraries of exercises (hundreds or thousands) and stress-test them extensively (generate them hundreds or thousand times). stresstest_exercise() now allows to set timeouts which will result in an error if violated. In addition, this is reported back to the user (see below).

Besides a timeout option, stresstest_exercise() now reports all warnings and errors captured during execution. This feature is available no matter if you only test one exercise (you must set stop_on_error = FALSE) or stacks of exercises. This allows to stress-test bulks of exercises and not only get reported where errors occured and how often, but also where warnings have been raised and how often, even if–a the end–the questions was successfully rendered.

`num_to_schoice()`

The extension of the stresstest-functionality above was somewhat interlocked with the function num_to_schoice() which allows to generate single-choice questions from a numeric solution. If the data generation of an exercise is badly written, it can cause num_to_schoice() to search for a possible (valid) return forever. Besides the new timeout-feature of stresstest_exercise(), num_to_schoice() has also been updated, allowing to specify how often the function is allowed to try to find a valid solution (return) via it’s new maxit argument which can also be set global (via getOption). See ?num_to_schoice for more details.

`exams2forms`

Development/testing features

Besides exams2forms providing great functionality to use for stand-allone versions of R/exams, its functionalty also helps those writing new exercises or testing existing ones. With the new release, exams2forms() (called by exams2webquiz()) got some new features allowing to (i) display the name of the exercise (show_filename), show (ii) the required tolerance for correct answers of numeric questions (show_tolerance), as well as (iii) features to be used by developers/testers to quickly check if everything is set up correctly (see argument auto).

Check out ?exams2forms for more details (all available via exams2webquiz()).

############################################################################

Over the past weeks and months, we have been updating both the R/exams package and its newest extension, exams2forms, to resolve known (and new) issues and to expand the functionality of both packages. I’d like to thank everyone involved in the latest additions and improvements!

I was primarily involved in:

Extending and refining the stresstest functionality to automatically
stress-test R/exams exercise questions, and
Improving and extending the functionality of the R package exams2forms, which now offers additional features for users developing and testing R/exams exercises.

R/exams

Stress-testing Exercises

The R/exams package has long supported automated stress-testing of one or multiple exercises via the stresstest_exercise() function (see the tutorial on
Stress Testing Dynamic R/exams Exercises).

While this functionality has worked reliably and as originally intended, it had one limitation: if an exercise took too long to generate–or entered an infinite loop–the function possibly stalled.

The latest release introduces a timeout option. This is especially helpful when testing large libraries of exercises (hundreds or thousands), allowing you to flag exercises that take too long to render or never complete. If the timeout is exceeded, an error is raised and reported to the user.

In addition, stresstest_exercise() now captures and reports all warnings and errors during execution. This applies whether you’re testing a single exercise (set stop_on_error = FALSE) or a large batch. You’ll get detailed feedback on where and how often issues occurred—even if an exercise was ultimately rendered successfully.

`num_to_schoice()`

The improvements on stresstest_exercise() were closely related to updates in the num_to_schoice() function, which creates single-choice questions from numeric solutions. Poorly written generation code could previously cause num_to_schoice() to loop indefinitely in search of a valid return.

Alongside the timeout in stresstest_exercise(), num_to_schoice() now accepts a maxit argument, specifying the maximum number of attempts before giving up. This can also be set globally using setOpttion().

See ?num_to_schoice for full details.

exams2forms

Development and Testing Features

In addition to supporting stand-alone versions of R/exams exercises, exams2forms now includes additional features that are especially helpful when developing or testing exercises.

The updated exams2forms() function (typically called via exams2webquiz()) got the following new options:

show_filename: Displays the name of the exercise file.
show_tolerance: Shows the tolerance for numeric answers.
auto: Enables developer/tester-specific features for quick verification that everything is correctly set up.

Check out ?exams2forms or ?exams2webquiz for more details.

exams2forms: Embedding 'exams' exercises in HTML documents

2024-11-11T00:00:00+01:00

The goal is to enable users to easily re-use dynamic exercises written in one of R/exams’ formats in HTML documents, such as stand-alone HTML files, online books, or websites.

This allows to provide additional self-assessment resources without the need for a learning management systems (e.g., OpenOlat, Moodle, Canvas, …) and supports all R/exams exercise types: single-choice, multiple-choice, numeric and text questions, as well as cloze questions combining one or all of the previously mentioned types.

Exercises written for exams2forms can be easily used in (summative) assessments and vice versa, as they share the same format. exams2forms is a valuable addition to the features and capabilities of the exams package, offering even more flexibility.

Interested? Our online tutorial provides interactive examples and templates to get you started today!

R package: https://cran.r-project.org/package=exams2forms
Tutorial: https://www.r-exams.org/tutorials/exams2forms/

colorspace: A Python toolbox for colors and palettes

2024-07-30T00:00:00+02:00

Citation

Reto Stauffer, Achim Zeileis (2024). “colorspace: A Python Toolbox for Manipulating and Assessing Colors and Palettes.” arXiv.org E-Print Archive arXiv:2407.19921 [cs.GR]. doi:10.48550/arXiv.2407.19921

Abstract

The Python colorspace package provides a toolbox for mapping between different color spaces which can then be used to generate a wide range of perceptually-based color palettes for qualitative or quantitative (sequential or diverging) information. These palettes (as well as any other sets of colors) can be visualized, assessed, and manipulated in various ways, e.g., by color swatches, emulating the effects of color vision deficiencies, or depicting the perceptual properties. Finally, the color palettes generated by the package can be easily integrated into standard visualization workflows in Python, e.g., using matplotlib, seaborn, or plotly.

Read full paper ›

Software

Package (PyPI): https://pypi.org/project/colorspace/
Documentation: https://retostauffer.github.io/python-colorspace/
Interactive apps: https://hclwizard.org/
Repository (GitHub): https://github.com/retostauffer/python-colorspace/

Motivation

Color is an integral element of visualizations and graphics and is essential for communicating (scientific) information. However, colors need to be chosen carefully so that they support the information displayed for all viewers (see e.g., Tufte 1990; Ware 2004; Wilke 2019). Therefore, suitable color palettes have been proposed in the literature (e.g., Brewer 1999; Ihaka 2003; Crameri, Shephard, and Heron 2020) and many software packages transitioned to better color defaults over the last decade. A prominent example from the Python community is matplotlib 2.0 (Hunter, Dale, Firing, Droettboom, and the Matplotlib Development Team 2017) which replaced the classic “jet” palette (a variation of the infamous “rainbow”) by the perceptually-based “viridis” palette. Hence a wide range of useful palettes for different purposes is provided in a number of Python packages today, including cmcramery (Rollo 2024), colormap (Cokelaer 2024), colormaps (Patel 2024), matplotlib (Hunter 2007), palettable (Davis 2023), or seaborn (Waskom 2021).

However, in most graphics packages colors are provided as a fixed set. While this makes it easy to use them in different applications, it is usually not easy to modify the perceptual properties or to set up new palettes following the same principles. The colorspace package addresses this by supporting color descriptions using different color spaces (hence the package name), including some that are based on human color perception. One notable example is the Hue-Chroma-Luminance (HCL) model which represents colors by coordinates on three perceptually-based axes: Hue (type of color), chroma (colorfulness), and luminance (brightness). Selecting colors along paths along these axes allows for intuitive construction of palettes that closely match many of the palettes provided in the packages listed above.

In addition to functions and interactive apps for HCL-based colors, the colorspace package also offers functions and classes for handling, transforming, and visualizing color palettes (from any source). In particular, this includes the simulation of color vision deficiencies (Machado Oliviera, and Fernandes 2009) but also contrast ratios, desaturation, lightening/darkening, etc.

The colorspace Python package was inspired by the eponymous R package (Zeileis, Fisher, Hornik, Ihaka, McWhite, Murrell, Stauffer, and Wilke 2020). It comes with extensive documentation at https://retostauffer.github.io/python-colorspace/, including many practical examples. Selected highlights are presented in the following.

Key functionality

HCL-based color palettes

The key functions and classes for constructing color palettes using hue-chroma-luminance paths (and then mapping these to hex codes) are:

qualitative_hcl: For qualitative or unordered categorical information, where every color should receive a similar perceptual weight.
sequential_hcl: For ordered/numeric information from high to low (or vice versa).
diverging_hcl: For ordered/numeric information around a central neutral value, where colors diverge from neutral to two extremes.

These functions provide a range of named palettes inspired by well-established packages but actually implemented using HCL paths. Additionally, the HCL parameters can be modified or new palettes can be created from scratch.

As an example, the figure below depicts color swatches for four viridis variations. The first pal1 sets up the palette from its name. It is identical to the second pal2 which employes the HCL specification directly: The hue ranges from purple (300) to yellow (75), colorfulness (chroma) increases from 40 to 95, and luminance (brightness) from dark (15) to light (90). The power parameter chooses a linear change in chroma and a slightly nonlinear path for luminance.

In pal3 and pal4 the most HCL properties are kept the same but some are modified: pal3 uses a triangular chroma path from 40 via 90 to 20, yielding muted colors at the end of the palette. pal4 just changes the starting hue for the palette to green (200) instead of purple. All four palettes are visualized by the swatchplot function from the package.

The objects returned by the palette functions provide a series of methods, e.g., pal1.settings for displaying the HCL parameters, pal1(3) for obtaining a number of hex colors, or pal1.cmap() for setting up a matplotlib color map, among others.

from colorspace import palette, sequential_hcl, swatchplot

pal1 = sequential_hcl(palette = "viridis")
pal2 = sequential_hcl(h = [300, 75], c = [40, 95], l = [15, 90],
                      power = [1., 1.1])
pal3 = sequential_hcl(palette = "viridis", cmax = 90,  c2 = 20)
pal4 = sequential_hcl(palette = "viridis", h1 = 200)

swatchplot({"Viridis (and altered versions of it)": [
               palette(pal1(7), "By name"), 
               palette(pal2(7), "By hand"),
               palette(pal3(7), "With triangular chroma"),
               palette(pal4(7), "With smaller hue range")
           ]}, figsize = (8, 1.75));

An overview of the named HCL-based palettes in colorspace is depicted below.

from colorspace import hcl_palettes
hcl_palettes(plot = True, figsize = (20, 15))

Palette visualization and assessment

To better understand the properties of palette pal4, defined above, the following figure shows its HCL spectrum (left) and the corresponding path through the HCL space (right).

The spectrum in the first panel shows how the hue (right axis) changes from about 200 (green) to 75 (yellow), while chroma and luminance (left axis) increase from about 20 to 95. Note that the kink in the chroma curve for the greenish colors occurs because such dark greens cannot have higher chromas when represented through RGB-based hex codes. The same is visible in the second panel where the path moves along the outer edge of the HCL space.

pal4.specplot(figsize = (5, 5));
pal4.hclplot(n = 7, figsize = (5, 5));

Color vision deficiency

Another important assessment of a color palette is how well it works for viewers with color vision deficiencies. This is exemplified below by depicting a demo plot (heatmap) under “normal” vision (left), deuteranomaly (colloquially known as “red-green color blindness”, center), and desaturated (gray scale, right). The palette in the top row is the traditional fully-saturated RGB rainbow, deliberately selected here as a palette with poor perceptual properties. It is contrasted with a perceptually-based sequential blue-yellow HCL palette in the bottom row.

The sequential HCL palette is monotonic in luminance so that it is easy to distinguish high-density and low-density regions under deuteranomaly and desaturation. However, the rainbow is non-monotonic in luminance and parts of the red-green contrasts collapse under deuteranomaly, making it much harder to interpret correctly.

from colorspace import rainbow, sequential_hcl
col1 = rainbow(end = 2/3, rev = True)(7)
col2 = sequential_hcl("Blue-Yellow", rev = True)(7)

from colorspace import demoplot, deutan, desaturate
import matplotlib.pyplot as plt

fig, ax = plt.subplots(2, 3, figsize = (9, 4))
demoplot(col1, "Heatmap", ax = ax[0,0], ylabel = "Rainbow", title = "Original")
demoplot(col2, "Heatmap", ax = ax[1,0], ylabel = "HCL (Blue-Yellow)")
demoplot(deutan(col1), "Heatmap", ax = ax[0,1], title = "Deuteranope")
demoplot(deutan(col2), "Heatmap", ax = ax[1,1])
demoplot(desaturate(col1), "Heatmap", ax = ax[0,2], title = "Desaturated")
demoplot(desaturate(col2), "Heatmap", ax = ax[1,2])
plt.show()

Integration with Python graphics packages

To illustrate that colorspace can be easily combined with different graphics workflows in Python, the code below shows a heatmap (two-dimensional histogram) from matplotlib and multi-group density from seaborn. The code below employs an example data set from the package (using pandas) with daily maximum and minimum temperature. For matplotlib the colormap (.cmap(); LinearSegmentedColormap) is extracted from the adapted viridis palette pal3 defined above. For seaborn the hex codes from a custom qualitative palette are extracted via .colors(4).

from colorspace import dataset, qualitative_hcl
import matplotlib.pyplot as plt
import seaborn as sns

df = dataset("HarzTraffic")

fig = plt.hist2d(df.tempmin, df.tempmax, bins = 20,
                 cmap = pal3.cmap().reversed())
plt.title("Joint density daily min/max temperature")
plt.xlabel("minimum temperature [deg C]")
plt.ylabel("maximum temperature [deg C]")
plt.show()

pal = qualitative_hcl("Dark 3", h1 = -180, h2 = 100)
g = sns.displot(data = df, x = "tempmax", hue = "season", fill = "season",   
                kind = "kde", rug = True, height = 4, aspect = 1,
                palette = pal.colors(4))
g.set_axis_labels("temperature [deg C]")              
g.set(title = "Distribution of daily maximum temperature given season")
plt.show()

Dependencies and availability

The colorspace is available from PyPI at https://pypi.org/project/colorspace. It is designed to be lightweight, requiring only numpy (Harris et al. 2020) for the core functionality. Only a few features rely on matplotlib, imageio (Klein et al. 2024), and pandas (The Pandas Development Team 2024). More information and an interactive interface can be found on https://hclwizard.org/. Package development is hosted on GitHub at https://github.com/retostauffer/python-colorspace. Bug reports, code contributions, and feature requests are warmly welcome.

References

Brewer CA (1999). “Color Use Guidelines for Data Representation.” In Proceedings of the Section on Statistical Graphics, American Statistical Association, pp. 55–60. Alexandria, VA.
Cokelaer T (2024). Colormap. Version 1.1.0, Python Package Index (PyPI), URL https://pypi.org/project/colormap/.
Crameri F, Shephard GE, Heron PJ (2020). “The Misuse of Colour in Science Communication.” Nature Communications, 11(5444), 1–10. doi:10.1038/s41467-020-19160-7.
Davis M (2023). palettable: Color Palettes for Python. Version 3.3.3, Python Package Index (PyPI), URL https://pypi.org/project/palettable/.
Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Cournapeau D, Wieser E, Taylor J, Berg S, Smith NJ, Kern R, Picus M, Hoyer S, van Kerkwijk MH, Brett M, Haldane A, del Río JF, Wiebe M, Peterson P, Gérard-Marchant P, Sheppard K, Reddy T, Weckesser W, Abbasi H, Gohlke C, Oliphant TE (2020). “Array Programming with NumPy.” Nature, 585(7825), 357–362. doi:10.1038/s41586-020-2649-2.
Hunter JD (2007). “Matplotlib: A 2D Graphics Environment.” Computing in Science & Engineering, 9(3), 90–95. doi:10.1109/mcse.2007.55.
Hunter JD, Dale D, Firing E, Droettboom M, the Matplotlib Development Team (2017). “What’s New in Matplotlib 2.0 (Jan 17, 2017), Changes to the Default Style.” Accessed 2024-07-22, URL https://matplotlib.org/stable/users/prev_whats_new/dflt_style_changes.html.
Ihaka R (2003). “Colour for Presentation Graphics.” In K Hornik, F Leisch, A Zeileis (eds.), Proceedings of the 3rd International Workshop on Distributed Statistical Computing, Vienna, Austria. ISSN 1609-395X, URL https://www.R-project.org/conferences/DSC-2003/Proceedings/Ihaka.pdf.
Klein A, Wallkötter S, Silvester S, Rynes A, actions-user, Müller P, Nunez-Iglesias J, Harfouche M, Schrangl L, Dennis, Lee A, Pandede, McCormick M, OrganicIrradiation, Rai A, Ladegaard A, van Kemenade H, Smith TD, Vaillant G, jackwalker64, Nises J, Komarčevič M, rreilink, Barnes C, Zulko, Hsieh PC, Rosenstein N, Górny M, scivision, Singleton J (2024). Imageio/Imageio: V2.34.2. doi:10.5281/zenodo.12514964. Version 2.34.2, Zenodo.
Machado GM, Oliviera MM, Fernandes LAF (2009). “A Physiologically-Based Model for Simulation of Color Vision Deficiency.” IEEE Transactions on Visualization and Computer Graphics, 15(6), 1291–1298. doi:10.1109/tvcg.2009.113.
Patel P (2024). Colormaps. Version 0.4.2, Python Package Index (PyPI), URL https://pypi.org/project/colormaps/.
Rollo C (2024). cmcrameri: Python Wrapper around Fabio Crameri’s Perceptually Uniform Colormaps. Version 1.9, Python Package Index (PyPI), URL https://pypi.org/project/cmcrameri/.
The Pandas Development Team (2024). pandas-Dev/Pandas: Pandas. doi:10.5281/zenodo.10957263. Version 2.2.2, Zenodo.
Tufte E (1990). Envisioning Information. Graphics Press, Cheshire.
Ware C (2004). “Color.” In Information Visualization: Perception for Design, chapter 4, pp. 103–149. Morgan Kaufmann Publishers Inc.
Waskom ML (2021). “seaborn: Statistical Data Visualization.” Journal of Open Source Software, 6(60), 3021. doi:10.21105/joss.03021.
Wilke CO (2019). Fundamentals of Data Visualization. O’Reilly Media. ISBN 1492031089. URL https://clauswilke.com/dataviz/color-basics.html.
Zeileis A, Fisher JC, Hornik K, Ihaka R, McWhite CD, Murrell P, Stauffer R, Wilke CO (2020). “colorspace: A Toolbox for Manipulating and Assessing Colors and Palettes.” Journal of Statistical Software, 96(1), 1–49. doi:10.18637/jss.v096.i01.

gsdata: Interface to the GeoSphere Austria DataHub API (Data Access)

2024-01-26T00:00:00+01:00

In 2019, GeoSphere Austria (formerly ZAMG) began offering public access to a wide variety of weather, climate, and environmental data. Most of this data is available under the Creative Commons Public Domain Dedication (CC0) license, making it free to use for any application.

The data can be retrieved through the GeoSphere Austria Data Hub API, and the gsdata R package provides a convenient interface to easily access and process the data. Initially developed for personal use, this package may require future updates or extensions.

The example below demonstrates a basic use case. For more information and additional examples, please refer to the documentation..

Documentation: https://retostauffer.github.io/gsdata
Repository: https://github.com/retostauffer/gsdata
Issues and bug reports: https://github.com/retostauffer/gsdata/issues

Innsbruck weather data (example)

The following call to gs_stationdata() retrieves historical temperature, dewpoint, and wind speed observations (synop; hourly data) for the first two weeks of 2023 at Innsbruck Airport, station 11120. When requesting data for a single station, gs_stationdata() returns a time series object. For multiple stations, the function will return a named list of time series objects.

library("gsdata")
innsbruck <- gs_stationdata(mode        = "historical",
                            resource_id = "synop-v1-1h",
                            start       = "2023-01-01",
                            end         = "2023-01-14",
                            parameters  = c("T", "Td", "ff"),
                            station_ids = 11120)

The default plotting method allows to easily visualize the data:

# Generic default plot
plot(innsbruck)

# Slightly modified plot
plot(innsbruck,
     screen = c(1, 1, 2),
     col    = c(2, 3, 4),
     ylab   = c("air temperature [C]\nwelt bulb temperature [C]",
                "mean wind speed [m/s]"))

Available data sets

As shown above, gs_stationdata() requires a mode, resource_id, station_ids, and (optional) a list of parameters. The package provides a series of functions to list available data sets as well as stations and parameters.

As demonstrated above, gs_stationdata() requires a mode, resource_id, station_ids, and optionally a list of parameters when called. The package also offers several functions to list available datasets, stations, and parameters.

Data sets

gs_datasets() returns a data.frame of available datasets from the GeoSphere Austria API. By default, gs_dataset() uses type = "stations", limiting the results to station-related datasets.

ds <- gs_datasets()
head(ds, n = 3)

##      type       mode   resource_id data_format response_formats
## 1 station historical histalp-v1-1y     station      geojson|csv
## 2 station historical   klima-v1-1d     station      geojson|csv
## 3 station historical   klima-v1-1h     station      geojson|csv
##                                                                        url
## 1 https://dataset.api.hub.geosphere.at/v1/station/historical/histalp-v1-1y
## 2   https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v1-1d
## 3   https://dataset.api.hub.geosphere.at/v1/station/historical/klima-v1-1h

In the object ds we can find the dataset used in the Innsbruck example above, which provides the correct settings for mode and resource_id.

subset(ds, resource_id == "synop-v1-1h")

##       type       mode resource_id data_format response_formats
## 12 station historical synop-v1-1h     station      geojson|csv
##                                                                       url
## 12 https://dataset.api.hub.geosphere.at/v1/station/historical/synop-v1-1h

Station metadata

Once the appropriate mode and resource_id for the desired dataset are identified, additional metadata can be retrieved using gs_metadata(). This function returns a named list with all details provided by the API.

meta <- gs_metadata(mode = "historical", resource_id = "synop-v1-1h", type = "station")
names(meta)

##  [1] "title"            "parameters"       "frequency"        "type"             "mode"             "response_formats"
##  [7] "start_time"       "end_time"         "stations"         "id_type"

title: Title of the data set.
frequency: Temporal resolution (observation frequency).
start_time/end_time: Period for which this data set is available. Note that not all tations/parameters are available for the entire period!
stations: List of all available stations.
parameters: List of available parameters with description (German only).
…

Two of the most useful elements in the metadata are stations and parameters. The parameters element is a data frame that includes the name (used when retrieving data via the parameters argument in gs_stationdata()), a long name, a German description, and the corresponding measurement units.

subset(meta$parameters, name %in% c("T", "Td", "ff"))

##    name           long_name                                                                   desc unit
## 39    T      Lufttemperatur                                                         Lufttemperatur   °C
## 40   Td  Taupunkttemperatur                       Taupunkt (bis 2001/06/19 tw. mit rel beschickt!)   °C
## 57   ff Windgeschwindigkeit Windgeschwindigkeit in 1/10 m/s (wird umgerechnet, wenn Knoten: *5.14)  m/s

The stations meta information is returned as a simple feature data.frame looking as follows:

head(meta$stations)

## Simple feature collection with 6 features and 8 fields
## Geometry type: POINT
## Dimension:     XY
## Bounding box:  xmin: 9.848611 ymin: 47.1625 xmax: 15.36694 ymax: 48.69083
## Geodetic CRS:  WGS 84
##         type    id             name            state altitude valid_from   valid_to is_active
## 1 INDIVIDUAL 11330        MAYRHOFEN            Tirol      640 2007-08-28 2100-01-01      TRUE
## 2 INDIVIDUAL 11328       ACHENKIRCH            Tirol      904 1998-09-24 2016-10-04     FALSE
## 3 INDIVIDUAL 11375           AFLENZ       Steiermark      783 1992-10-08 2100-01-01      TRUE
## 4 INDIVIDUAL 11157 AIGEN IM ENNSTAL       Steiermark      641 1972-01-01 2100-01-01      TRUE
## 5 INDIVIDUAL 11301    ALBERSCHWENDE       Vorarlberg      715 1996-01-16 2100-01-01      TRUE
## 6 INDIVIDUAL 11019      ALLENTSTEIG Niederösterreich      599 1988-05-30 2100-01-01      TRUE
##                    geometry
## 1  POINT (11.85167 47.1625)
## 2 POINT (11.70528 47.53222)
## 3 POINT (15.24083 47.54583)
## 4 POINT (14.13833 47.53278)
## 5  POINT (9.848611 47.4575)
## 6 POINT (15.36694 48.69083)

Along with the id, the unique identifier used when calling gs_stationdata() via on the station_ids argument, the API provides additional information about each station. Since the stations are provided as a simple feature data frame, it is easy to visualize their locations or search for specific stations of interest.

# Plotting altitude
plot(meta$stations["altitude"], pch = 19)

For more examples and a more comprehensive overview of all features of the gsdata package please have a look at the documentation.

White Christmas in Innsbruck?

2022-12-16T00:00:00+01:00

At this specific time of the year, many of us are interested in the question: Will we have a white Christmas? To understand how this question can be answered one first needs to understand how weather can be forecasted. Our atmosphere is a very complex and chaotic system, but is based on physical principles which are - in large parts - well understood. Starting out from the current state of the atmosphere, i.e., the current weather around the globe, the physical principles can be used to calculate the state of the atmosphere in a few hours or days. This is known as numerical weather prediction (NWP).

Nowadays, this is done using high-performance supercomputers which run NWP models to forecast the weather up to several weeks ahead. As neither the current state of the atmosphere nor all involved physical processes are known exactly and certain technical or numerical simplifications are required, these NWPs are subject to some uncertainties. This is especially true for long forecast horizons (e.g., predicting the weather nine days ahead) as errors naturally grow over time.

However, these errors can often be corrected with expert knowledge, thus improving the accuracy of the forecast. Meteorologists practice interpreting NWP output and make corrections if needed. E.g., if the NWP always predicts slightly too cold temperatures in a specific weather situation, the expert can correct the most recent forecast if the same situation happens again based on their profound knowledge about previous events. An alternative way is to use machine-learning techniques following the same idea. Based on historical NWP forecasts and observations, the algorithm learns in which situations the NWP was wrong and allows to adjust incoming forecasts accordingly if needed.

As this work is a cooperation between different departments at the Universität Innsbruck, we made use of both worlds. While Georg J. Mayr (Professor at the Department of Atmospheric and Cryospheric Sciences) manually analyzed the latest forecasts using his expert knowledge in synoptics, Achim Zeileis (Professor at the Department of Statistics) and Reto Stauffer (Assistant Professor at the Digital Science Center and the Department of Statistics) implemented the machine-learning algorithm to tackle the question.

Defining the target

To have an objective target, we defined ‘white Christmas in Innsbruck’ as having a closed snow cover with a snow depth of one centimeter or more at Innsbruck Airport on December 24, 0600 UTC (seven o’clock local time).

Data

For training the machine-learning model, historical NWP forecasts and observations from the most recent seven years are used, plus/minus 20 days around Christmas (December 4 to January 13), resulting in a total sample size of N=259.

The observations come from a weather station at Innsbruck Airport, reporting snow depth at 0600 UTC on a daily basis which are turned into a binary variable. If a closed snow cover with a snow depth of one centimeter or more is observed, the snow observation is “white”, or “no” otherwise.

The NWP forecasts are taken from the European Centre for Medium-Range Weather Forecasts (ECMWF) ensemble prediction system (0000 UTC run). As the latest ensemble forecast run is the one from December 15 when this blog post goes online, forecasts up to +222 hours (nine days and six hours) are used to be able to predict December 24, 2022, 0600 UTC.

A wide range of different variables from the NWP have been extracted such as temperatures at ground level and the lower atmosphere, cloud cover, precipitation and snowfall, or humidity. In addition, aggregated variables have been calculated covering the full forecast horizon (up to +222 hours) or the three days before the forecast step of interest (+150 hours to +222 hours) including aggregated minima, maxima, averages, and/or sums, resulting in M=92 different input variables for the model.

Methodology: Random forests

As the method of choice, a random forest is used, a powerful and flexible, yet robust, machine-learning algorithm. To give a brief introduction to random forests: A random forest consists of a series of individual (but all slightly different) trees.

Each conditional inference tree is based on randomly selected variables from the NWP (15 out of 92). In each iteration it is tested which of the input variables shows the strongest dependence to the target variable. If there is at least one, a binary split is performed on the one with the highest dependency, splitting the data into two subsets resulting in two new nodes. This process is repeated until there is no more dependency between the target and the input variables, or other stopping criteria are met. The last node after each split is also called a leaf containing a series of observations which either fall into “white” or “no”. Thus, the relative frequency (or probability) observing “white” can be calculated in each leaf.

Many of these trees build a random forest. For this application, 5000 individual trees are used, whereof one tree is shown exemplarily below.

Once the random forest is estimated, variable importance can be calculated by performing a random permutation test. Without going into details: The more important a variable, the stronger the accuracy of the random forest decreases. The image below shows the mean decrease in accuracy of the 15 most important variables of the estimated random forest.

Once a new NWP forecast becomes available, we can predict the outcome by putting the new forecast into each of the 5000 trees and check in which leaf (final node) the new observation will fall into. Averaging over the outcome of all trees gives the final probability of the event happening (“white”) or not happening (“no”).

Prediction: Will there be a white Christmas in Innsbruck?

To finally answer the question, a prediction was made for Christmas (December 24, 2022, 0600 UTC) based on the latest NWP forecast (initialized December 15, 2022, 0000 UTC). The random forest returns a probability of 8.8%. Given the information we have today, nine days before Christmas, the chances of having a closed snow cover with a snow depth of one centimeter or more is not zero, but not large either.

Does the expert meteorologist agree with the algorithm? Independently from the machine-learning approach, Georg J. Mayr was going trough a vast amount of weather forecast maps on December 15, 2022. He came to the conclusion that in the days leading up to the event the notorious “Christmas thaw” weather will erase the remaining white and gave the probability of a white Christmas of less than 5%; close to the result from the random forest.

To know for sure whether December 24 will be a white Christmas and how good the forecasts are, we will have to wait for nine days (and watch the latest forecasts). To be able to give such a clear answer with our application is an exciting development. If you are interested in learning more about some of the topics covered but in much greater detail, please have a look at the suggested references below.

How to cite

Reto Stauffer (2022). “White Christmas in Innsbruck?”, Digital Science Center Blog, 10.48763/000004.

References

Schlosser L, Hothorn T, Stauffer R, Zeileis A (2019). “Distributional regression forests for probabilistic precipitation forecasting in complex terrain”. Annals of Applied Statistics, 13(3), 1564–1589. doi:10.1214/19-AOAS1247.
Stauffer R, Mayr GJ, Messner JW, Zeileis A (2018). “Hourly Probabilistic Snow Forecasts over Complex Terrain: A Hybrid Ensemble Postprocessing Approach”. Advances in Statistical Climatoloy, Meteorology and Oceanography, 4(1/2), 65-86. doi:10.5194/ascmo-4-65-2018.
Stauffer R, Umlauf N, Messner JW, Mayr GJ, Zeileis A (2017b). “Ensemble Postprocessing of Daily Precipitation Sums over Complex Terrain Using Censored High-Resolution Standardized Anomalies”. Monthly Weather Review, 145(3), 955–969. doi:10.1175/MWR-D-16-0260.1.

useR!2021 Contribution

2021-07-06T00:00:00+02:00

Today we have had the chance to give some insights about our teaching at the Digital Science Center of the Universität Innsbruck, to be more precise into how we teach an introductory programming course with R.

In case you have not had the chance to participate at the R conference feel free to watch our 5 minute elevators pitch on youtube. The slides to the presentation are also available as pdf.

Thanks to my coauthors Joanna, Miguel, and Achim.

colorspace: A Toolbox for Manipulating and Assessing Colors and Palettes

2019-03-18T00:00:00+01:00

Authors:

Achim Zeileis, Universität Innsbruck
Jason C. Fisher, U.S. Geological Survey
Kurt Hornik, WU Wirtschaftsuniversität Wien
Ross Ihaka, University of Auckland
Claire D. McWhite, The University of Texas at Austin
Paul Murrell, University of Auckland
Reto Stauffer, Universität Innsbruck
Claus O. Wilke, The University of Texas at Austin

Abstract:

The R package colorspace provides a flexible toolbox for selecting individual colors or color palettes, manipulating these colors, and employing them in statistical graphics and data visualizations. In particular, the package provides a broad range of color palettes based on the HCL (Hue-Chroma-Luminance) color space. The three HCL dimensions have been shown to match those of the human visual system very well, thus facilitating intuitive selection of color palettes through trajectories in this space. Using the HCL color model general strategies for three types of palettes are implemented: (1) Qualitative for coding categorical information, i.e., where no particular ordering of categories is available. (2) Sequential for coding ordered/numeric information, i.e., going from high to low (or vice versa). (3) Diverging for coding ordered/numeric information around a central neutral value, i.e., where colors diverge from neutral to two extremes. To aid selection and application of these palettes the package also contains scales for use with ggplot2, shiny (and tcltk) apps for interactive exploration, visualizations of palette properties, accompanying manipulation utilities (like desaturation and lighten/darken), and emulation of color vision deficiencies.

More information:

Workingpaper available on arXiv.org
The R package on CRAN
Detailed Package documentation (R-Forge.R-project.org)
The article has been submitted to the Journal of Statistical Software

Convenient GFS Reforecast V2 Downloader

2019-01-28T00:00:00+01:00

PyGFSV2 Package in Beta

Today I fixed some smaller issues in the PyGFSV2 code (thanks to Naschi) and made the code python 3 ready. However, have not yet had time for extensive testing!

Feel free to use it and do not hesitate to contact me if there are any problems with the code! Or report problems as issues on github.

Downloading ERA-5 Data

2018-12-15T00:00:00+01:00

This work by Reto Stauffer (2025) is licensed under the GNU General Public License v2. If you improve and/or extend the code it would be great to send me a message and the extensions such that I can include the changes in the original repository.

Short guide for downloading ERA5

Some time ago I have written a short introduction on how to download ERA INTERIM data from ecmwf.int. ERA INTERIM was the fourth version of ECMWFs global reanalysis data set. Just recently (mid 2018) the ECMWF started publishing the new ERA-5 data set with a spatial resolution of about 32 km times 32 kilometers.

Please note that this data set is still in a pre-release state. Every now and then some fields are getting updated as some problems have been identified in the data. The data set is currently going back to January 2000 but will be further extended into the past.

Access to ERA-5 is very similar to accessing ERA INTERIM, however, the new data set is redistributed over the Copernicus Climate Data Store. Even if access via the ECMWF public data access portal is still possible it is not recommended to do so as it will be shut down in the near future.

All we need to download data:

A valid CDS user account and create a local file containing the user token.
The cdsapi python package plus a few lines of python code.

Even if you are not yet familiar to python you’ll see that it is pretty simple!

Create a user account, get your API key

If you don’t own a user account yet create a new account on the CDS website. After activating your account use your new account to log in. Your user name will be show in the top right corner. You can now enter your user profile by clicking on your user name. On the profile you’ll find your user id (UID) and your personal API Key.

Create local key file

For batch script data downloads you’ll have to create a local ASCII file with your user information (UID, API key) which is used by the python package (cdsapi). To do so (linux) simply create a file called .cdsapirc in your home directory and add the following two lines:

url: https://cds.climate.copernicus.eu/api/v2
key: 1234:abcdefghij-134-abcdefgadf-82391b9d3f

where 1234 is your personal user ID (UID), the part behind the colon your personal API key. Line one simply contains the URL to the web API. More details can be found here.

Install cdsapi package

What you’ll need next is the python package cdsapi. As it is available on PyPI you can simply install it using your preferred python package installer, e.g., pip:

pip install cdsapi

For the sake of completeness: I am using cdsapi 2.18.4 at the moment.

Write your first data request

To get data you’ll have to write a small python script which sends requests to the CDS servers. Here is one simple example how the python scripts will look like:

# Import cdsapi
import cdsapi
# Open a new Client instance
c = cdsapi.Client()
# Send your request (download data)
c.retrieve('reanalysis-era5-single-levels', {
        'product_type': 'reanalysis',
        'variable':     'orography',
        'year':         '2016',
        'month':        '01',
        'day':          '01',
        'time':         '00:00'
    }, 'orography.grib')

Copy the lines above into a python script (e.g., orography.py) and execute it (python orography.py). In this simple case I am requesting orography (variable) reanalysis (product_type) data for January 1, 2016 00:00 UTC (year, month, day, time). Orography is a single-level variable and thus identified by the reanalysis-era5-single-levels product keyword. Once you executed this script you should now have a file called orography.grib just next to your orography.py script file.

Note: the CDS provides a nice web interface where you can create these requests by selecting the parameters and time periods you need. Under Show API Request you’ll see the request (as shown above) for your custom selection, but not all options are shown. Two different data sets are available:

Make a more useful request

Let’s assume we are interested in the 00 UTC temperature and geopotential height of the 900 and 700 hectopascal level for 2017 over Europe for some reason. The corresponding request looks like this:

# Import cdsapi and create a Client instance
import cdsapi
c = cdsapi.Client()
# More complex request
c.retrieve("reanalysis-era5-pressure-levels", {
        "product_type":   "reanalysis",
        "format":         "netcdf",
        "area":           "52.00/2.00/40.00/20.00",
        "variable":       ["geopotential","temperature"],
        "pressure_level": ["700","900"],
        "year":           "2017",
        "month":          ["01","02","03","04","05","06","07","08","09","10","11","12"],
        "day":            ["01","02","03","04","05","06","07","08","09","10","11",
                           "12","13","14","15","16","17","18","19","20","21","22",
                           "23","24","25","26","27","28","29","30","31"],
        "time":           "00"
    }, "output.nc")

Product identifier: reanalysis-era5-pressure-levels as we are interested in pressure level data
Product type: reanalysis (as before)
Format: instead of downloading grib1 data we would like to have NetCDF data. This is nice as the grib1 file format is a bit unhandy sometimes and can be specified using the format key.
Spatial extent: the keyword area allows to download a very specific subset. The definition is N/W/S/E in degrees longitude and latitude (see API manual). Negative values correspond to S and W. In the example above a domain over Europe.
Fields: we would like to get geopotential height and temperature (variable) on two levels, namely 900 and 700 hectopascal (pressure_level).
As we are interested in 00 UTC data for 2018 we simply specify "time":"00", "year":"2018", all months (01 to 12) and all days (01 to 31). Don’t worry that some combinations do not exist (e.g., February 31), the API will simply ignore these fields.

At the end you should get the output file output.nc. To drop some numbers: the output file is of about 10 MB in this case and the whole request took me less than five minutes.

Batch-scripting the requests

Rather than copy-pasting API requests you can, of course, also write some more fancy scripts. The script below is a script I’ve written few weeks ago to download a set of files for one of our current studies.

Reto’s ERA5 Downloader (download the script here)

Feel free to use, modify and redistribute the script (GPL2 license). Please note that the script is quite likely not free of bugs. If you find some please let me know :).

Some of the features:

Should be python3 and python2 ready.
Provides some help (call python ERA5_Downloader.py --help).
Downloads one parameter at a time.
Is downloading data in junks (month/year) stored in a sub-folder called era5_data.
Checks whether a file has already been downloaded.
Is subsetting Europe! The area is hard-coded in the script, needs to be adjusted depending on what you want to have.
This script has been written for one of my applications in only few quarter of minutes and is far away from beeing perfect or super flexible. However, it might be a good starter if you are looking for a script to start with.

Output of ERA5_Downloader.py –help

If you missed the link above: the ERA5 Downloader can be downloaded here (click).

Usage: Usage:

    ERA5_Downloader.py --years <years> --parameter <parameter> --level <level>
    ERA5_Downloader.py -y <years> -p <parameter> -l <level>

    Downloading ERA5 reanalysis data from Copernicus Climate Data
    Services. NOTE: Not yet made for ensemble ERA5!

    Requires `cdsapi` to be installed (python package to access
    Copernicus Climate Data Services, CDS).

    How to install cdsapi and the required API key:
    * https://cds.climate.copernicus.eu/api-how-to

    Available parameters for "single level" (-t/--type sf):
    * https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form

    Available parameters for "pressure level" (-t/--type pl):
    * https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=form

    Usage examples:

    10m wind speed for the year 2018
    >>> python ERA5_Downloader.py --years 2018 --parameter 10m_u_component_of_wind

    10m wind speed for 2010 and 2018
    >>> python ERA5_Downloader.py --years 2010,2018 --parameter 10m_u_component_of_wind

    10m wind speed for 2008, 2009, 2010, ..., 2018
    >>> python ERA5_Downloader.py --years 2008-2018 --parameter 10m_u_component_of_wind

    700hPa geopotential height, 2018
    >>> python ERA5_Downloader.py --years 2018 --parameter geopotential --level 700
    
    

Options:
  -h, --help            show this help message and exit
  -y YEARS, --years=YEARS
                        years for which the data should be downloaded. Can
                        either be a single year (e.g., 2010), a comma
                        separated list (e.g., 2010,2011), or a sequence of the
                        form <bgn>-<end> (e.g., 2010-2018; 2010 to 2018).
  -p PARAM, --parameter=PARAM
                        the parameter/variable to be downloaded. Please check
                        the cds website to see what's availabe.
  -l LEVEL, --level=LEVEL
                        level, only for pressure level data. For "single level
                        data" (e.g., surface level variables) do not set this
                        parameter!
  -t, --test            development flag. If set, only one day (January 1th of
                        the first year) will be downloaded before the script
                        stops. The output file will be labeled with "_test_".

Python colorspace package

2018-09-17T00:00:00+02:00

Some good news for python enthusiasts. I have had some “free time” and decided to port the R colorspace package to python. The result is the python colorspace package for Python 2.7 and Python 3.6+.

Today I decided to publish a first early beta release (v0.1.0). There might be some hilarious bugs (or features) in the current version and not all features are yet implemented, but should be a working tech demo! If you are interested you can give it a shot. Feel free to report bugs and/or contribute to this little project if you want to.

The documentation is available on readthedocs.io but needs some more attention from my side. Not all functions are documented, nothing is proof-read, and some pages are still empty (e.g., the usage examples). I’ll try to work on it as soon as possible.

Package: https://github.com/retostauffer/python-colorspace
Documentation: https://python-colorspace.readthedocs.io/en/latest/
More information: http://hclwizard.org/
The R package: https://cran.r-project.org/package=colorspace

What the package does

The colorspace provides some handy functions to create and use effective color maps for scientific data visualization based on the Hue-Chroma-Luminance color space. In contrast to the widely used Red-Green-Blue color space the HCL color space is based on how humans perceive colors. The three dimensions of the HCL color space allow to directly control the three dimensions of color perception and to avoid “bad” or inefficient color maps which can, in some cases, obscure the data presented in the figures and make them hard to interpret or even mislead the readers.

Choose Color Palettes

The choose_palette() function provides an interactive TclTk interface to customize color palettes. Some simple demo plots allow to directly access the look and feel of the palettes. Furthermore, the interface allows to check the selected color palettes for possible issues with respect to color vision deficiencies (color blindness).

The GUI returns a hclpalette object with a range of standardized methods to draw a set of colors:

As the returned object is an object it also allows you to dynamically draw a different number of colors from the palette, e.g,. 100:

Or simply return a vector with three HEX colors. There is even a method to directly get a matplotlib cmap (matplotlib.colors.LinearSegmentedColormap).

There are some more nice features such as swatchplot and specplot to access the palette, functions to simulate color vision deficiencies (deutan, protan, tritan, and desaturate), and more. The package also includes a colorlib module which allows to easily transform colors between a set of different color spaces including hex colors, CIE-XYZ, HCL, and others.

Again, it’s a beta! You can blame me, but it would be nice to get some feedback instead :). Bug-reports and issues welcome which help to fix existing errors and to improve the package.

Reto