Page Comparison

...

You have to be aware of the following when you plan to use scripts/programms to download data sets from data.arge-oevvmobilitaetsverbuende.at:

Data sets can be only downloaded by registered users
The authentication follows the OpenID Connect specification (implemented by a Keycloak server)
A user must be registered manually (cannot register per script)
Data sets can only be downloaded after the user agreed to the license
Accepting the license must be done manually

The preparation steps would then be:

Register to data.arge-oevvmobilitaetsverbuende.at and confirm the email address you gave by the registration
Take a look at the data sets page, click on the data set you want to download, show the license and accept it
Write the credentials (user and password) you used in the registration to some config file for your script
Use the credentials in the script to authenticate to Keycloak and obtain an access token
Use the access token to retrieve data sets and download data set files from the plattform

...

Use the following information to obtain an access token on behalf of your registered user:

Access Token Endpoint: https://user.arge-oevvmobilitaetsverbuende.at/auth/realms/dbp-public/protocol/openid-connect/token

Client ID: dbp-script-download

...

Code Block

#!/bin/bash
# vim: ts=2 sw=2 et

#
# This example demonstrates the download of a dataset from ARGEMobilitaetsverbuende Oesterreich OEVVOOG DBP per script.
# The user credentials must belong to an active user who registered to the DBP
# (data.arge-oevvmobilitaetsverbuende.at) and already accepted the corresponding licences.
#
# History:
# 11.11.2021 NIO: Some corrections & comments
# 29.10.2021 NIO: Example created

# Some constants:
# TODO: correct Keycloak URL
KEYCLOAK="https://user.arge-oevvmobilitaetsverbuende.at"
REALM="dbp-public"

# Typically you would store the credentials in a configuration file
MY_USERNAME="me@example.com"
MY_PASSWORD="abcdefghijkl"

# DBP access
# TODO: correct DBP URL
DBP_BASE=https://data.arge-oevvmobilitaetsverbuende.at
CLIENT_ID="dbp-script-download"
ENDPOINT_DATA_SETS=/api/public/v1/data-sets

echo Auth on $KEYCLOAK, DBP on $DBP_BASE

# Get the access token
function get_access_token() {
  curl -sS -k -X POST \
    -d "client_id=${CLIENT_ID}" \
    -d "username=${MY_USERNAME}" \
    -d "password=${MY_PASSWORD}" \
    -d "grant_type=password" \
    "${KEYCLOAK}/auth/realms/${REALM}/protocol/openid-connect/token" | jq -r '.access_token'
}

# List the existing datasets
# Needs access token
function get_dataset_list() {
  local token=$1
  curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS} \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $token"
}

# Take the given dataset
# Needs access token and dataset id
function get_dataset() {
  local token=$1
  local id=$2
  curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS}/${id} \
    -H "Accept: application/json" \
    -H "Authorization: Bearer $token"
}

# Download the given dataset
# Needs access token and dataset id
function download_dataset() {
  local token=$1
  local id=$2
  curl -sS -k ${DBP_BASE}${ENDPOINT_DATA_SETS}/${id}/file \
    -H "Accept: application/zip" \
    -H "Authorization: Bearer $token"
}

# Get the acces token
echo Take access token
token=$(get_access_token)
echo $token

# Let's say we are interested in only 1 Dataset with the name "Liniennetz (JSON) 2021"
# We list all datasets and for the wanted one we get the id
# Note that here we make all these transformations with jq, but in another programming language this can be done with
# language specific means
WANTED='Liniennetz (JSON) 2021'
echo Take dataset id
dsid=$(get_dataset_list $token | jq -r "map({ name, id } | select( .name == \"$WANTED\") | .id)[]")

# Now we can get further information about the wanted dataset, for example the active dataset version,
# and we could further check if we already have that file (by name or by creation date/time)
# Here we just display that information and then proceed to download
echo Latest version of Dataset $dsid
get_dataset $token $dsid | jq '.activeVersion'

# Now we download the file
echo Download the file
ZIPFILE=my-download-$(date +%Y%m%d%H%M%S).zip
download_dataset $token $dsid> $ZIPFILE
echo Everything done, see $ZIPFILE

...

Versions Compared

Old Version 1

New Version 2

Key