EDAMIS SHORT USER GUIDE
For service support, please contact: |
| Last update : 25-07-2023 |
| EDAMIS web portal |
Key concepts
Country: The usual country definition, with the exception that the Commission is considered as a country with the code ‘EU’.
Organisation: An organisation belongs to a Country. Eurostat is one organisation belonging to the EU country. Organisation can have sub-organisations.
User: A person that can log in in EDAMIS. A user has one organisation. A user can request rights inside EDAMIS (action he/she can perform in the application). A user with the appropriate rights can grant/give rights to other users. A user is able to set how he/she wants to be notified: by email on each event, by daily or weekly emails, by notifications or not at all.
SDMX, code list, DSD/MSD, Registry: The standard widely adopted to exchange data and metadata. SDMX is the standard, DSD and MSD defines the structure of the data/metadata, both can reference code lists. The Registry is storing and exposing all this information centrally.
Dataset/Metadataset: A dataset is a key concept of EDAMIS. It represents a table of data that should be sent one or more time. A dataset or metadataset can be linked to one or more DSD or MSD. One DSD/MSD described that structure of the data that are expected. This structure is linked to a period of the data (i.e. use this structure for data of 2013-2016 and this one for data 2017-onward). Dataset have a periodicity (how often is it expected, i.e. daily, weekly, monthly, yearly, every 2 years…), if it is linked to a legal act or an agreement, if it is confidential or not, and a description of when the data are expected (for example, the data are expected at least one month after the end their period and at the latest two month after their period).
Dataset/Metadataset period: A dataset is linked to one or more period. The set of periods defines what data were expected (start and end date, for example this dataset). Periods cannot overlap. A period can be linked to a DSD. This link is pointing to which DSD (and so, the structure of the data) should be used to send data of that period.
Domain: A domain is simply a set of datasets/metadatasets that are more or less related. For example, all the datasets that are related to Transport can be regrouped under the Transport domain. EDAMIS is also offering 2 other ways to regroups datasets/metadatasets: dataset groups and statistical Domains. The first one is just a set of datasets/metadatasets that can be created when needed. The second one is a hierarchical partition of the all the datasets.
Provider: A provider is one Organisation that is expected/allowed to send data.
Consumer: A consumer is one Organisation that will receive data.
Delegate: Provider and Consumer can be of 2 types: Responsible and Delegate. A provider responsible can have one or more delegates that are able to send data in its name. A consumer responsible can have one or more delegate that will also receive the data provided to the responsible. A delegation is performed at the level of the dataset, i.e. Organisation B can be the delegate provider for Organisation A on the dataset A, and Organisation C is the delegate of Organisation A on the dataset B.
Routing: A routing is a set of providers and consumers (responsible) and a mapping between them, i.e. a routing is giving the list of providers for a given consumer. Routings are defined per dataset/metadataset. For example, a routing is Organisation A from County A and Organisation B from County B are delivering data of Dataset A to Organisation C of Country C and Organisation C of Country C is delivering data to Organisation D of Country D.
Web Forms: An on-line excel like way to deliver data. The manager is creating the tables, some validation rules and the providers fill in on-line the tables before launching the validation and sending the data.
Constraints: A generic way to manage restriction on code lists. These restrictions are published in the registry and eventually used by Web Forms.
Encryption: Data (and feedback) can be encrypted to ensure maximum security and that only the destination can read the data. Encryption can be performed outside EDAMIS or can be done by the browser before sending the file. Encryption is done with all the public keys of all the consumers. Encryption done by the browser is limited as it has to be perform in the browser memory. Encryption is done using OpenPGP.
1. General view
The Search bar (1) is use for quickly find information in the following sections:
- Dataset;
- Domain;
- Users;
Only four results of each section are displayed, however it is possible to click on 'more' to display more results. The results only show the items the user is entitled to see.
The user menu (2) is composed of four items:
- My profile (link to go to the detail page of my profile);
- My emails;
- My Rights;
- Logout (Link to disconnect from EDAMIS).
The Location Bar (3) indicates what the current view is. The menu Bar (4) allows accessing the different functionalities of the application. It changes according to the user rights.
When the user log in, the default page is the Dashboard. The user can also go back to his/her Dashboard by clicking on the 
icon. Its content is composed of widgets. The displayed widgets and their locations are configurable per user.
One buttons is available at the top of the page 
. This button provides access to the Dashboard configuration.
1.1 Dashboard configuration
The widgets are organised in an array of 2 columns. A widget can be extended to occupy the 2 columns, or only one.
The user can press the Add button to add a new widget to the page. Each widget has its own remove button.
For each widget, three options are available 
:
- The first allows the user to choose the width of the widget (one or two columns);
- The second and third moves the widget on the page (swap with the previous widget, swap with the next widget);
- The third is used to remove the widget.
All widgets have buttons on their title bar. The buttons allow, eventually, changing the displayed page of the widget, refreshing the content of the widget and configuring the content of the widget. Most widgets don’t have a default configuration. So, the user should open
The currently available widgets are:
- Last sent files: this widget displays the last file sends on the datasets attached to the user;
- Last received files: this widget displays the last file received on the datasets attached to the user;
- Tasks list: This widget displays the task (i.e. an action in the application that is required from the user) list. The tasks can be:
- No Public Key: when a dataset is confidential or the feedback channel is confidential and no public is defined for all consumers or all providers;
- No consumers on dataset;
- Application role requested by user;
- Country role requested by user;
- Organisation role requested by user;
- Domain role requested by user;
- Dataset role requested by user;
- Data provider responsible role requested by user;
- Data consumer responsible role requested by user;
- Log book: this widget displays all the log of the application to the user;
- Send datafile: this widget displays all the data the user is expected to send in a near future (and missed in the near past) for a given dataset;
- Notifications: this widget displays the past events that are linked to the objects the user has rights on;
- News: this widget displays the latest news;
- Welcome: the default widget that is displaying some useful tips for the new comers.
2. Transmission
2.1 Transmission of files
In EDAMIS 4 Web Portal, you can send data files to Eurostat using the widget “Send datafile or by clicking on the main menu item Transmissions" and then on the sub-menu item "Send datafiles”. From this option, you will be able to send one or multiple files. If the file respects the Dataset Structure Naming Convention (DSNC), the dataset field will be automatically filled:
2.2 Encryption
Some datasets can be marked as ‘confidential’, which means that their associated data are sensitives and that they cannot be sent directly in a clear format over the network. Thus, when sending a datafile, the user can ask to automatically encrypt the data (if it has not already been done manually by the user) when the file is sent.
The encryption process takes place directly in the browser of the user: no data is sent over the network to encrypt them.
2.2.1 Library
The only way to encrypt data in a browser is to use the JavaScript language. Therefore, we are using a library called ‘OpenPGP.js’, which is a JavaScript implementation of the OpenPGP standard. Using OpenPGP is a good choice because it is used widely in the world and it has been hardly tested to prevent security issues.
The current version of OpenPGP.js is the 2.4.0 and it is currently hosted on Github.
2.2.2 Algorithm
PGP uses both a symmetric key and a public key to encrypt the data. The data are encrypted using a 128-bit generated symmetric key (generated with the IDEA algorithm) that is used only once. This key is then encrypted using the public keys of the receivers (only the receivers will be able to decrypt it as he/she is the only owner of the associated private key). Finally, the encrypted data and the encrypted key are then sent to the server as an encrypted message.
To decrypt a message, the receiver will first decrypt the encrypted key using its private key. This will give the symmetric key that has been used to encrypt the data. The receivers can now decrypt the data using this key and he will be able to retrieve the original data.
2.2.3 Limitations
As the encryption takes place in the browser of the user, everything happens in the memory of its computed. Therefore, there is a limitation on the size of the files that can be encrypted. For example, if the computer of the users has 2GB of memory, it will not be possible to encrypt files bigger then 2GB (actually it will be even lower then 2GB as the user might have other applications running).
3. Dataset/Metadataset
A dataset is defined by its identifier. It is linked to a domain.
The properties of a dataset/metadataset are:
- Basis for transmission: It can be: “Legal Act”, “Agreement”, “Voluntary”;
- Transmission modes: A mix of “Send Data File” and “Web Form”;
- Data confidential: Checked if the dataset is confidential, and so should be encrypted;
- Feedback confidential: Checked if the feedback is confidential and so, should be encrypted;
- Transmission Network: The list of Transmission network where the dataset is available (for example TESTA only or Internet and TESTA).
- Timeline : described in chapter 1;
- Routing:
3.1 Timeline
Figure 2: Timeline for a monthly Dataset
In order to define when the data are expected, the data manager needs to define one period as reference and its corresponding starting date. For example, a yearly dataset can have 2015 as reference period and a start date as 01/01/2015. But a yearly dataset can also start on the 01/03/2015. These two values define one reference point in time. The end of the data period is computed by adding the periodicity (for example, 01/03/2016 in our last example). Then, there is the Collect Delay, which represents the delta, in days, months and years between the end of one period and the expected start date to send the data. This delay can be negative (so, data can be send before the end of the period of the data) The delay can be, as example, 1 month and 1 day or 1 month minus 1 day.
The last values (Transmission Delay in days, months and years) give the duration of the data collection.
Figure 2 is showing a monthly dataset with a reference period chosen as 2017/1 and a reference date as 01/01/2017 (so, the data period always starts on the 1st of the month), with a 14 days delay to transmit the data and a collection period of 1 month.
The configuration is performed with the following fields:
- Reference period: Defines one period as reference. It can be 2015 for a yearly dataset or 2015/1 for a quarterly or monthly dataset;
- Start of the reference period: Defines the start date of the data defines in the reference period;
- Collect delay: Defines the delta, in days, months, years between the end of the period of the data of the beginning of the data collection;
- Transmission delay: defines the duration of the data collection;
- Reminders: Defines what reminders should be sent (depending on the user preferences) and when (please, note that reminders are not sent of a data file has already been sent by the provider):
- On the start of transmission period: Checked if a reminder is send when the transmission period starts;
- Before the deadline: Checked if a reminder should be send a given number of days before the transmission period end;
- Number of days before the deadline: Defines the number of days before the end of the collection to send a reminder;
- On the day of the deadline: Checked if a reminder should be send on the end of the collection period;
- After the deadline: Checked if reminders should be send a given number of days after the end of the collection period;
- Max time(s) after the deadline: The number of days after the end of the collection period to send the reminders;
- Frequency of reminder in days: The number of days between two reminders after the end of the collection period.
3.2 Reporting Periods
EDAMIS is maintaining a list of which DSD should be used to validate data of which period. For example, it is possible to use DSD TRAIN_A1_V1.1 to validate data from 2014 and 2015, TRAIN_A1_V2.0 for data of 2016 and TRAIN_A1_V3.5 for data of 2017 onward. Any two reference period cannot overlap (so that only one DSD can be applied for a given period of the data). Web Forms are associated to a reporting period and Web Forms Templates
The reporting period fields are:
- Valid from (editable field): Defines when the period starts. This value can be NULL in order to state that the picked DSD can be used from the beginning of times until the Valid to period;
- Valid to (editable field): Defines when the period ends. This values can be NULL in order to state that the picked DSD can be used to defines the data to be send until the data manager says otherwise;
- Dataflow version/DSD version: the name and version of the DSD/dataflow to be used;
- Web Forms Template: This field links the web form to the reporting period, pointing or allowing to create a web form template for a given period.
3.3 Data Providers / Consumers
EDAMIS is managing the flow of data. For each dataset/metadataset, there is a list of data providers (the entity that should send the data) and a list of data consumer (the entity that should receive the data). Each provider is able to have one or more delegate: an entity that is allowed to send data in its name. Each consumer is able to have one or more delegates, that will receive the data too, and that are able to send a validation report in the name of the of its responsible. Finally, some providers of one dataset/metadataset can be linked to some consumer while other providers are linked to other consumers. For example, All Member states send data to Belgium, while Belgium sends data to Eurostat.
It is possible to add a data provider/consumer by clicking on the “Add new Provider” or “Add new Consumer” combo and pick a predefined organisation list or select “Select organisation” to look for a given organisation.
By clicking on the Country of the provider, it is possible exceptions (different values for the basis for Transmission and confidentiality flag and different values for the timelines) for the providers of the pointed country.
By clicking on the organisation of a provider/consumer, it is possible to set the public key of this provider/consumer for this dataset for confidential datasets and the managed the delegates. The provider public key will be used for feedback while the consumer public key will be used to send data. When multiple consumers are defined, each should have a public key.
4. User and Rights
A role is a set of rights on a given objects. For example, a Data Manager can create/edit datasets as well as grants/give rights to users on the datasets he/she managed.
User can request roles by using the top right menu: “My account->My rights”. Roles can be granted or refused by using the menu “Inventory->Users”, then clicking on a user and finally pressing the “Rights” button at the top. Granting rights also appear as an action in the Task List widget.
4.1 Rights
In the page “My Rights”, the user can see all the roles he/she have on objects (domain, dataset, provider/consumer, country, and organisation). .Below, the user can request new roles. Only roles that will bring him/her new rights are displayed. First, the user must select if he/she is looking for roles on domain/dataset/provider-consumer or Country/Organisation. Then, he/she can select the filter (only objects linked to his/her county or linked to his/her organisation or all objects). Bellow, the user sees a hierarchical list of objects (domains, then datasets of this domain, then providers and consumers of this dataset or countries, then organisation of this country, then sub organisations of this organisation). Each object of this hierarchical list has a ‘+’ button which allows to ask for a given role on this object. The available roles depend on the role the user already has and the king of object.
Users that have the rights to grants roles can see a task that is generated for them in the task list widgets. By clicking on the corresponding item, they are redirected to the corresponding action page or they can go to the User rights page. There, they can see the requested roles of the user, and the corresponding object (for example, “Sender Responsible” for dataset A). They can Grant of refuse the requested role. Bellow, they can see the already granted roles and the corresponding objects. Finally, there is a hierarchical list identical to the one where user can request role. This list allows to directly give roles on the corresponding object for the selected user.
4.2 Notifications
User can define the way he/she wants to be notified of events on objects he/she has rights on.
There is two ways to be notified: by emails or by the notification widgets. The user can select between on event email, daily email, weekly email and no email at all. The user can also select witch events will be in the notification widgets. The configuration depends on the user rights and can be accessed by the top right menu: “My account->My emails”. There, for each kind of events, the user can select the email notification type and the widget notification or not.
5. Navigation Map
Figure 3: Navigation Map