Prerequisites
This part is only until November 22nd valid - see the information about the migration comming soon.
You have to be aware of the following when you plan to use scripts/programms to download data sets from data.mobilitaetsverbuende.at:
Data sets can be only downloaded by registered users
The authentication follows the OpenID Connect specification (implemented by a Keycloak server)
A user must be registered manually (cannot register per script)
Data sets can only be downloaded after the user agreed to the license
Accepting the license must be done manually
The preparation steps would then be:
Register to data.mobilitaetsverbuende.at and confirm the email address you gave by the registration
Take a look at the data sets page, click on the data set you want to download, show the license and accept it
Write the credentials (user and password) you used in the registration to some config file for your script
Use the credentials in the script to authenticate to Keycloak and obtain an access token
Use the access token to retrieve data sets and download data set files from the plattform
OpenID Connect Parameters
Use the following information to obtain an access token on behalf of your registered user:
...
Credentials: the ones you used for the registration
Example Script
This example script demonstrates how to get the access token from Keycloak and how to use the access token to automatically download a data set file. The OpenID Connect standard is well described on the internet. For all supported operations of the REST API interface of the DBP you can consult DBP Public Swagger UI.
| Code Block |
|---|
#!/bin/bash
# vim: ts=2 sw=2 et
#
# This example demonstrates the download of a dataset from Mobilitaetsverbuende Oesterreich OOG DBP per script.
# The user credentials must belong to an active user who registered to the DBP
# (data.mobilitaetsverbuende.at) and already accepted the corresponding licences.
#
# History:
# 11.11.2021 NIO: Some corrections & comments
# 29.10.2021 NIO: Example created
# Some constants:
# TODO: correct Keycloak URL
KEYCLOAK="https://user.mobilitaetsverbuende.at"
REALM="dbp-public"
# Typically you would store the credentials in a configuration file
MY_USERNAME="me@example.com"
MY_PASSWORD="abcdefghijkl"
# DBP access
# TODO: correct DBP URL
DBP_BASE=https://data.mobilitaetsverbuende.at
CLIENT_ID="dbp-script-download"
ENDPOINT_DATA_SETS=/api/public/v1/data-sets
echo Auth on $KEYCLOAK, DBP on $DBP_BASE
# Get the access token
function get_access_token() {
curl -sS -k -X POST \
-d "client_id=${CLIENT_ID}" \
-d "username=${MY_USERNAME}" \
-d "password=${MY_PASSWORD}" \
-d "grant_type=password" \
"${KEYCLOAK}/auth/realms/${REALM}/protocol/openid-connect/token" | jq -r '.access_token'
}
# List the existing datasets
# Needs access token
function get_dataset_list() {
local token=$1
curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS} \
-H "Accept: application/json" \
-H "Authorization: Bearer $token"
}
# Take the given dataset
# Needs access token and dataset id
function get_dataset() {
local token=$1
local id=$2
curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS}/${id} \
-H "Accept: application/json" \
-H "Authorization: Bearer $token"
}
# Download the given dataset
# Needs access token and dataset id
function download_dataset() {
local token=$1
local id=$2
curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS}/${id}/file \
-H "Accept: application/zip" \
-H "Authorization: Bearer $token"
}
# Get the acces token
echo Take access token
token=$(get_access_token)
echo $token
# Let's say we are interested in only 1 Dataset with the name "Liniennetz (JSON) 2021"
# We list all datasets and for the wanted one we get the id
# Note that here we make all these transformations with jq, but in another programming language this can be done with
# language specific means
WANTED='Liniennetz (JSON) 2021'
echo Take dataset id
dsid=$(get_dataset_list $token | jq -r "map({ name, id } | select( .name == \"$WANTED\") | .id)[]")
# Now we can get further information about the wanted dataset, for example the active dataset version,
# and we could further check if we already have that file (by name or by creation date/time)
# Here we just display that information and then proceed to download
echo Latest version of Dataset $dsid
get_dataset $token $dsid | jq '.activeVersion'
# Now we download the file
echo Download the file
ZIPFILE=my-download-$(date +%Y%m%d%H%M%S).zip
download_dataset $token $dsid> $ZIPFILE
echo Everything done, see $ZIPFILE
|
...