Book page

Technical specifications for file export and import to ESS-MH

Fernando MORENTE-ORIA
Fernando MORENTE-ORIA • 17 February 2025

TECHNICAL SPECIFICATIONS FOR FILE EXPORT AND IMPORT TO ESS-MH

For service support, please contact:

ESTAT-DATA-METADATA-SERVICES@ec.europa.eu

Last reviewed: 18-02-2025
ESSMH web portal

 

1. Introduction

This document provides explanations about the

  • Export options for retrieving of metadata files and MSDs from various tools and databases (in SDMX-ML 3.0 format)
  • And particularly import features in ESS-MH for SDMX-ML 3.0 files  and Excel files to create or edit metadata files and Guidelines, along with extra annotations used to represent functionalities specific to the ESS-MH application.

These annotations cover the following subjects:

  • Importation of concept values of various representation types
  • Images
  • Annexes – at the concepts level & global
  • Publication flag (for the whole metadata file)
  • Restricted for publication flag (at the concepts level)
  • Editability of concepts

A metadata file can be exchanged by:

Using only an SDMX-ML 3.0 file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to annexes or images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to annexes or images). Such an "Excel" zip file should have a specific structure and contain an xlsx file.

Until the end of 2025 the ESS-MH will also allow the import and export of files in SDMX-ML 2.0 format. Instructions for exports and imports of Metadata files in format SDMX 2.0 can be retrieved from CircaBC

 

2. Export options: MSD and Metadata files

Each metadata and quality report is described by an SDMX Metadata Structure Definition (MSD), and all Metadata Structure Definitions are publicly available in the Euro-SDMX Registry. The MSDs detail which concept IDs must be used and the structure the SDMX-ML file must have to be considered acceptable.

If you are not sure about the underlying MSD in use for a metadata or quality report, you may go to the ESS-MH and make sure that you select the “MSD” to be displayed in the metadata file overview or consult the Metadata flows report (accessible to Super Providers only).  

Displaying MSD information in the ESSMH

and

Metadata flow report

 

You may receive information on the MSD used for a specific metadata or quality reporting by exporting/downloading. To download the MSD, you have three options:

  1. from the Euro-SDMX Registry, where you can download the MSD specifically (and you can find further referenced artefacts there, such as Concept Scheme and Code Lists);
  2. from the ESS-MH, where you can download the metadata file, compressed in a zip file containing the MSD (and information on the Concepts and Code lists used), the metadata file itself and referenced documents, if any;
  3. from the Eurostat website - Data Navigation Tree, Data Browser, Data Explorer and bulk down facility, where you can retrieve published zip files containing the metadata file itself and the MSD and Eurostat's new data navigation tree using the Data Browser

1. In the new Euro-SDMX Registry, at Metadata > Metadata Structures, search for the concerned MSD and click on the button “Export SDMX-ML 3.0”.

New Euro-SDMX Registry example

Alternatively, go to Export Structures > Metadata Structures, search for the concerned MSD(s) and click on the button “Download Selected Structures”.

New Euro-SDMX Registry example

Further selections are possible, such as the desired SDMX-ML format v3.0, or v2.1 and whether to include Descendants (referenced structures) etc.

Download structures window

2. You can go to the ESS-MH, go to the specific report you are interested in and click on Download. A pop-up will appear where you may select the desired SDMX format; by default it is set to 3.0, but you may also opt for 2.0. The download will include, among others, the MSD and the SDMX-ML version of the metadata or quality report in the selected SDMX format. 

Selecting download button from the Actions columnA pop-up window opens with all the dowlnoaded files

 

3. From the Data Navigation Tree, you can retrieve Published files (currently in SDMX format2.0)

  • via the HTML file accessible through the M-icon or  
The M icon on the Eurostat navigation tree

Or the Data Browser

  • directly from the published dataset 
Metadata access from a dataset example

 

Download button on a metadata file example

 

 

3. Import options: Import Metadata files to ESS-MH

At present, the most common way how to import metadata files to the ESS-MH is "manually" by logging in and going to the "Import" section:

Import feature on the ESSMH

"Browse" the file you would like to import, and click "Import".
The application will automatically detect the SDMX format of the imported metadata file, that is either 3.0 or 2.0, hence no further specification is required from the user for importing a metadata file. 
A green success message, or a red error message with some indication will be returned consequently. 
Furthermore, efforts have been made to facilitate automated metadata file transmissions via EDAMIS without the need to login to the ESS-MH user interface. If you are interested in such transmission method, please reach out to ESTAT-DATA-METADATA-SERVICES@ec.europa.eu

 

3.1 "SDMX" zip file

It should be used when the metadata file refers to annexes or images.
In ESS-MH the files used to be called like xx.sdmx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.
Example: DMBB_SIMS_A_4D_2020_0000.sdmx.zip.
 

SDMX zip file example

            

This zip file contains:

  • One SDMX-ML file named xx.sdmx.xml (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.sdmx.xml
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_an_nn.ext where:
      •  xx is the metadata file name,
      • nn is a sequential number starting from 1 or a customised name suffix,
      • ext is the extension of the annex according to the type of document.
        Example: DMBB_SIMS_A_4D_2020_0000_an_1.xlsx, DMBB_SIMS_A_4D_2020_0000_an_2.docx
    • Each annex should have a reference in the SDMX-ML file (like "Annexes/xx_an_nn.ext").
    • Note that you can put your own name (instead of xx_an_nn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the SDMX-ML file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the SDMX-ML file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images/YourOwnName.ext".
  • Not actually required, but can be inserted optionally: 
    • one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
      Example: SIMS_MSD.msd.xml
    • DSD file(s) (in SDMX-ML format) named as zz.xml. The DSDs are the basis for validating the structure of tabular files of type FILE_DSD, and they are stored in the ESS-MH, hence do not need to, but may be imported along with the FILE_DSD / structure .xlsx file.

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific application to open them. Therefore, we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

3.2 "Excel" zip file

It should be used when the metadata file refers to annexes or images.

In ESS-MH we are used to call the file like xx.xlsx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.

Example: DMBB_SIMS_A_4D_2020_0000.xlsx.zip

SDMX zip file example

This zip file contains:

  • One xlsx file named xx.xlsx (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.xlsx
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_an_nn.ext, where:
      • xx is the metadata file name,
      • nn is a number starting from 1 or a customised name suffix,
      • ext is the extension of the annex according to the type of document).
        Example: DMBB_SIMS_A_4D_2020_0000_an1.xlsx, DMBB_SIMS_A_4D_2020_0000_an2.docx
    • Each annex should have a reference in the xlsx file (like "Annexes/xx_an_nn.ext").
    • Note that you can put your own name (instead of xx_an_nn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the xlsx file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the xlsx file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • Not actually required, but can be inserted optionally:  
    • one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
      Example: SIMS_MSD.msd.xml
    • Folder DSDs including the DSD file(s) (in SDMX-ML format) named as zz.xml. The DSDs are the basis for validating the structure of tabular files of type FILE_DSD, and they are stored in ESS-MH, hence do not need to, but may be imported along with the FILE_DSD / structured .xlsx file.

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific program to open them. Therefore, we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

4. SDMX-ML file

In order to retrieve the correct template for the SDMX-ML used for the import, it is advised to download first an example file from ESS-MH, conveniently from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.sdmx.xml (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.sdmx.xml

 

4.1 <Header> tag

The <Header> is mandatory and should contain all the tags presented in the example.

xml file example

 

Some of the tags in the <Header> are mandatory to contain a value,

 

Item

Usage status

Specification

Example

<ID>

optional

No value needed here.

DMBB_SIMS_A

<Test>

optional

No value needed here.

false

<Prepared>

optional

No value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss (like 2020-01-01T00:00:00)

2020-11-20T11:34:00

<Sender id>

mandatory

The <Sender> should have the same value as in <generic:Target> in <MetadataSet> tag below.

4D0

<Name>

mandatory

The name of the metadata file is required here. The name is basically a composition of the Target values that need to be provided in the <MetadataSet> tag. It follows the structure 

DataFlow_DataProvider_ TimeDimension

DMBB_SIMS_

A_4D_2020_0000

<Structure id>

mandatory

Identifies the metadata flow the metadata file is provided for, including the version of the metadata flow, appended with “_” as METADATAFLOW_n_n.

DMBB_SIMS_A_1_0

<DataSet

ID>

optional

Can take any value, but may optimally be the same as in <Name>. Upon file creation, the DataSetID is generated automatically by the application ESS-MH and takes the following form: DataFlow_n where n is an application specific identifier automatically generated for storing the file in the ESS-MH database.

DMBB_SIMS_A

_1733755023224

<DataSet

Action>

optional

To edit an existing file, normally the Import is used by checking the "overwrite" box in ESS-MH UI. The content of the new file replaces the old one completely.

With the <DataSetAction> tag, an incremental import is facilitated: The application is able to import partial files of the same report while keeping the existing content. Based on the value in the (optional) <DataSetAction> tag in the <Header> the following actions can be carried out:

  • Append: New data (i.e. concepts that are not empty) may be added, but cannot overwrite or delete existing ones.
  • Replace: New data (i.e. concepts that are not empty) may be added and also replace existing one. 
  • Delete: New data (both, concepts that are empty or contain a value) replace existing ones and old ones are deleted if the new ones are missing values.
  • [missing value in the<DataSetAction> tag]: the new file replaces the existing one entirely (= “Delete”, complete overwrite of the file).

Note:

  • The "overwrite" checkbox in ESS-MH UI is taken into account only when there is no action tag in the SDMX-ML file. The action tag has priority over the checkbox.
  • A downloaded file does not contain the <DataSetAction> tag.

 

 

4.2 <MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD it refers to, including as identifier the MSD id, the agency, the version.

 

4.2.1 <Annotations>

 

Annotations example

The first part of the MetadataSet comprises the Annotations made to the metadata file. Those are the Annotation Types

 

  • REPORT_STRUCTURE with AnnotationText e.g. SIMS_REPORT_FULL. This item is not necessary in SDMX 3.0 however remains in the metadata file to ensure compatibility with SDMX 2.0 for the transition period. This item does not need to be included, filled in or altered.

  • REFERENCE_PERIOD which identifies the periodicity of the metadata file replacing the former Target Value for the TimeDimension. This item is not necessary in SDMX 3.0 however remains in the metadata file to ensure compatibility with SDMX 2.0 for the transition period. This item needs to be included in the import file in for the time being.

    PeriodicityDescriptionPossible periodExample
    2(every 2 years)020
    3(every 3 years)030
    4(every 4 years)040
    5(every 5 years)050
    A(Annual)0A0
    M(Monthly)1-12M1 to M12
    Q(Quarterly)1-4Q1 to Q4
  • FOR_PUBLICATION which indicates whether the imported metadata file is targeted for Publication or not – see below under 4.2.4 Publication flags and Restriction flags.

  • The global annexes for the Metadataset of types URL, FILE, FILE_DSD – see below under 4.2.3 Annexes.

Tag

<generic:Metadataflow>

Metadataflow contains the urn pointing to the respective metadataflow following the pattern urn:sdmx:org.sdmx.infomodel.metadatastructure.Metadataflow=AGENCY: METADATAFLOW ID (VERSION)

  • E.g. “urn:sdmx:org.sdmx.infomodel.metadatastructure.Metadataflow=ESTAT: DMBB_SIMS_A (1.0)”The code for the Domain (Category), here 'DMBB'.
  • An additional label indicating also the report structure used, as specified in the metadata flow. Here 'SIMS'.
  • The periodicity, as specified in the metadata flow. Here 'A' for annual. 
  • The version of the metadata flow in parenthesis, here ':1.0'
  • The Metadataflow= must be the same as for the <generic:Target> pointing to the datastructure.Dataflow – see below.

<generic:Target> for object "DataProvider"

  • Contains the urn pointing to the Data Provider scheme targeted and the specific Organisation code that applies for the metadata file. It should be the same as the Sender ID as provided in the <Header> tag.
  • Following the patern urn:sdmx:org.sdmx.infomodel.base.DataProvider=AGENCY:DATA PROVIDER SCHEME(VERSION).ORGANISATION ID, e.g. urn:sdmx:org.sdmx.infomodel.base.DataProvider=ESTAT:DATA_PROVIDERS(1.0).AL1

<generic:Target> for object="Dataflow"

  • Contains the Dataflow (Metadataflow) reference
  • Following the pattern urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=AGENCY: METADATAFLOW ID (VERSION), e.g. urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ESTAT: DMBB_SIMS_A (1.0)
4.2.2 Metadata Attributes - The concepts and their values
  • Are reported in a tag <generic:Attribute id=”…”>
  • Their value should be put in the tag <generic:Value>.
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts are specified in the MSD. They can have String format, Date, reference to a Code List (CL_...) or be of type Boolean. In the table below are described some examples for each representation type.

Metadata attribute / concept

Specification

Example

String

Simple plain text

Plain text can be imported.

HTML entities, allowing to introduce HTML tags for special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)

Images

  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • <generic:Value>
    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images
  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

&lt;p&gt;&lt;img src= 

"Images /xx_imgnn.ext

alt="Alternative text describing what is shown on the image" name="xx_imgnn.ext" /&gt;&lt;/p&gt;

</generic:Value>

/YourOwnName.ext".

Simple plain text

<generic:

Value>Here is some plain text.

</generic:Value>

HTML entities

<generic:Value>

&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;

ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;

abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;

&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p 

style="padding-left: 30px;"&gt;Text

on level 2&amp;nbsp;without bullets 

or  numbering&lt;/p&gt;

</generic:Value>

Images

<generic:Value> 

&lt;p&gt;Eurostat&lt;

/p&gt;&lt;p&gt;

&lt;img        src=

"Images/DMBB_SIMS_A_

4D_2020_0000_img1.png" 

alt="text what the image shows" name="DMBB_SIMS_A_

4D_2020_0000_img1.png" 

/&gt;&lt;/p&gt;

</generic:Value>

Date

The dates contained e.g. in the concepts

META_CERTIFIED

META_POSTED

META_LAST_UPDATE

should be inserted as plain text in format YYYY-MM-DD. Alternatively, also accepted, dd/mm/yyyy

<generic:Value>

2020-11-20</generic:Value>

<generic:Value>

20/11/2020</generic:Value>

Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accomodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.

Single value code list

<generic:Value>

codeID

</generic:Value>

Multiple value code list

<generic:Value>

codeID1

</generic:Value>

<generic:Value>

codeID2

</generic:Value>

<generic:Value>

codeID3

</generic:Value>

<generic:Value>

codeID4

</generic:Value>

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1

 

Single value code list

</generic:Attribute id="...">

<generic:Value>

HIGH</generic:Value>

</generic:Attribute>

Multiple value code list

<generic:

Attribute id="...">

<generic:Value>

HIGH</generic:Value>

<generic:Value>

LOW</generic:Value>

<generic:Value>

MEDIUM</generic:Value>

<generic:Value>

NOT_APPLICABLE

</generic:Value>

<generic:Value>

NOT_AVAILABLE</generic:

Value>

</generic:

Attribute>

Boolean

A concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE. The application ESS-MH is not case sensitive in that respect and accepts values also in lowercase true or false.

<generic: Attribute id="...">

<generic:<generic:Value>

TRUE</generic:</generic:Value>

</generic:Attribute>

 

</generic:

Attribute id="...">

<generic:Value>

FALSE</generic:Value>

</generic:Attribute>

 

4.2.3 Annexes

Annexes can be

  • Global to the whole metadata file
  • Or specific to one concept.

Global annexes should be added

  • in the tag <MetadataSet>,
  • in a tag <common:Annotations>
    and <common:Annotation> for each annex.
Annexes example

 

Concept's annexes should be added

  • in the related tag <generic:Attribute id=…>
  • in a tag <common:Annotations>
    and <common:Annotation> for each annex.
Concept annexes examples

 

The different types of an annex are:

  • FILE - any kind of file formats such as .doc, xlsx, pdf, jpg, etc...
  • File_DSD [1] - SDMX compatible tabular data files that need to be provided in a pair of a DSD and underlying Excel file in .xlsx format. Tabular data files are being validated with STRUVAL.
  • URL - a link to a webpage

Type

Specification

Example

FILE

  • A tag <common:AnnotationType> containing "FILE"
  • A title put in the tag <common:AnnotationTitle>
  • Refer to a file put in the tag <common:AnnotationText>
  • And the file should be present in the subdirectory Annexes of the "SDMX" zip file with the specified name.

<common:Annotation>

        <common:AnnotationType>

FILE</common:AnnotationType>

        <common:AnnotationTitle>

Annex 1 Excel file</common:AnnotationTitle>

<common:AnnotationText>Annexes/

DMBB_SIMS_A_4D_2020_0000_an1.xlsx

</common:AnnotationText>

</common:Annotation>

FILE_DSD

  • A tag <common:AnnotationType> containing "FILE_DSD"
  • A title put in the tag <common:AnnotationTitle> and info on the referenced DSD in brackets
  • Refer to a file put in the tag <common:AnnotationText>
  • And the file should be present in the subdirectory Annexes of the "SDMX" zip file with the specified name.

            <common:Annotation>

               <common:AnnotationType>FILE_DSD

</common:AnnotationType>

               <common:AnnotationTitle>LFS ANNEX [LFS_QR_Multiple+1.0]

</common:AnnotationTitle>

               <common:AnnotationText>Annexes

/LFS_SIMS_A_

BE_2023_0000_an_1.xlsx</common:AnnotationText>

            </common:Annotation>

            <common:Annotation>

               <common:AnnotationType>FOR_PUBLICATION

</common:AnnotationType>

               <common:AnnotationText>YES<

/common:AnnotationText>

            </common:Annotation>

URL

  • A tag <common:AnnotationType> containing "URL"
  • A title put in the tag <common:AnnotationTitle>
  • And an URL put in the tag <common:AnnotationURL>

<common:Annotation>

        <common:AnnotationType>URL

</common:AnnotationType>

        <common:AnnotationTitle>

Eurostat Website</common:AnnotationTitle>

        <common:AnnotationURL>

https://ec.europa.eu/eurostat/

web/main/home

</common:AnnotationURL>

</common:Annotation>


4.2.4 Publication flags and Restriction flags, non editable concepts

 

Specification

Example

Publication 

flag for 

whole 

metadata file

If the metadata file should be published, an annotation for the "Publication flag"

  • should be present
  • and be set to "YES".
  • in the tag 
  • after 
  • in a tag 

<generic:

MetadataSet>,

</generic:

AttributeValueSet>,

<generic:

Annotations>
<generic:

Annotation>
<common:AnnotationType> 

containing "FOR_PUBLICATION"
<common:AnnotationText> 

containing "YES".

<common:Annotation>

<common:AnnotationType>

FOR_PUBLICATION<

/common:

AnnotationType>

<common:AnnotationText

>YES<

/common:AnnotationText>

</common:Annotation>

Restricted for 

publication flag

at the 

concepts level

For each concept which values should not be published, an annotation for the "Restricted for publication flag"

  • should be present
  • and set to "YES".

The value "Restricted for publication" will appear as content of the concept when the file is published.

Such flag should be added

  • in the related tag 
  • in a tag 

<generic:

ReportedAttribute conceptID=…>

<generic:Annotations>
<generic:Annotation>
<common:AnnotationType> 

containing 

"RESTRICTED_FOR_

PUBLICATION"
<common:AnnotationText> 

containing "YES".

<common:Annotation>

<common:AnnotationType>

RESTRICTED_FOR_

PUBLICATION<

/common:AnnotationType>

<common:AnnotationText>

YES<

/common:AnnotationText>

</common:Annotation>

Restricted from

publication flag

at the Annexes

Each Annex, that is being attached to a metadata file but should not be published, can be flagged as «Restricted from publication» by adding the property «restricted» under Annotation Type 

        <common:

AnnotationType>FILE-restricted<

/common:AnnotationType>

<common:AnnotationType>FILE_DSD-restricted</common:AnnotationType>       

 <common:

AnnotationType>URL-restricted<

/common:AnnotationType>

Concept marked as not editable

Concept may be “locked” for editing, so that any values imported by data providers will be ignored or not be possible in order to avoid contents in the concept at all, or to “freeze” values provided by Eurostat. Such concepts contain an annotation indicating that they are not editable. 

 

The annotation is included in the exported metadata file for information purposes and is not required on import of metadata files.

<common:

AnnotationType>EDITABLE<

/common:AnnotationType>

<common:

AnnotationText>NO<

/common:AnnotationText>

 

 

5. xlsx file 

To retrieve the correct template for the xlsx used for the import, it is advised to download first an example file from ESS-MH, conveniently a comparable file from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.xlsx (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.xlsx
The xlsx file comprises three sheets that serve different purposes and require different degree of filling.
•    Metadata
•    Parameters
•    Annexes
 

 

5.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B4:B8.

Header components example

 

Items

Usage status

Specification

Example

Filename

optional

equivalent to SDMX-ML file

DMBB_SIMS_A_4D_

2020_0000

Published name

optional

The download of a metadata file includes also the publication name of it, specified by the code of the node in the DataNavigationTree, the file has been attached to, the typology used and, in case of national files, the country code of the file. 

scitech_sims

(or in case of a national file, e.g. scitech_sims_lu)

Data flow ID

mandatory

equivalent to SDMX-ML file

DMBB_SIMS_A

Data flow version

mandatory

equivalent to SDMX-ML file

1.0

Organization code

mandatory

equivalent to SDMX-ML file

4D0

Time dimension

mandatory

equivalent to SDMX-ML file

2020-A0

For publication

mandatory

If the metadata file should be published, the value "For Publication" should be set to YES, otherwise to NO.

 

YES

Data set action

optional

equivalent to SDMX-ML file

 

 

5.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with the corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 

 

5.2.1 Concept values

The concept values should be put in cells in column B "Concepts". According to the specifications in the MSD, concept values can be expressed in different representation types:

Metadata attribute / concept

Specification

Example

String

Simple plain text

Plain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.

Note: ESS-MH also supports the import of values derived from a formula.

HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

Images

  • Should be referenced in one concept as an <img> tag
  • <img src=
  • "Images/xx_imgnn.ext" alt="Alternative text describing what the image shows"name="xx_imgnn.ext"
    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

Simple plain text

Here is some plain text.

HTML tags

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;

&amp;&nbsp;

&zeta;&nbsp;&phi;</p>

<p><a href=

"https://ec.europa.eu

/eurostat/

web/main/home"

</a></p

Images

<p><img

src="Images/

DMBB_SIMS_A_4D_

2020_0000_img1.png"alt="Brief, descriptive text what the image shows"

"name="

DMBB_SIMS_A_4D_

2020_0000_img1.png" /></p>

Date

The dates contained e.g. in the concepts

2.1. Metadata last certified (META_CERTIFIED)
2.2. Metadata last posted (META_POSTED)
2.3. Metadata last update (META_LAST_UPDATE)

may be inserted as YYYY-MM-DD. Alternatively, also accepted, plain text with apostrophe in format 'dd/mm/yyyy or formatted as Date, e.g. dd-mm-yyyy.

2020-11-20

'20/11/2020

20-11-2020

Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accommodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.

 

Single value code list

codeID in uppercase

Multiple value code list

codeID1,codeID2,

codeID3,codeID4,...

 

 

 

 

 

 

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1

Single value code list

Single value code list example

Multiple value code list

Multiple value code list example

Boolean

A concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE

 

Boolean example

 

5.2.2 Restricted for publication flag (at the concepts level)

For each concept which should not be published, the corresponding cell in column C headed by "Restricted from publication" should be set to YES. Otherwise NO.

Then the value "Restricted for publication" will appear as content of the concept when the file is published.

 

5.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the Metadata files in ESS-MH. Generally, when you work with a downloaded template of a comparable Metadata file from ESS-MH, in sheet "Parameters" editing is neither expected nor advised.

Parameters example

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the Metadata file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Lastly, column E identifies whether a concept is configured as being "Editable" or not. In case of "NO", any value inserted in the respective concept in Metadata tab / Annexes tab will not be taken into account on import of the Metadata file. Only, if you plan to set up your file import per Excel file from scratch with a blank file, or you plan to import only a subset of concepts to ESS-MH, you should ensure to align the Parameters sheet with the Metadata sheet respectively. 

 

5.4 Annexes

Equivalent to SDMX-ML files, Annexes can be attached

  • Specifically to one concept,
  • Globally to the whole metadata file.

The different types of an annex are:

  • FILE - any kind of file formats such as .doc, .xlsx, pdf, jpg, etc..
  • FILE_DSD [2] - SDMX compatible tabular data files that need to be provided in a pair of a DSD and underlying Excel file in .xlsx format. Tabular data files are being validated with STRUVAL.
  • URL - a link to a webpage

Note: You can attach as many Annexes as necessary, provided you keep the maximum size of the entire Excel.zip file of 10MB.

Annexes need to be added in the corresponding "Annexes" sheet, both for concepts and globally for the whole metadata file.

Example of Annexes

 

In Column A "Concept", you specify whether an Annex is supposed to be attached

  • To a concept by providing here the corresponding concept ID (can be looked up from the Parameters sheet),
  • To the metadata file globally by leaving the cell blank.

In column B "Annex Type" you specify whether an Annex is of the type

  • FILE, or
  • FILE_DSD
  • URL,
  • adding «restricted» in case you want to flag the Annex not to be published.

In column C "Annex Name", you reference the Annex by providing

  • Which file needs to be present from the subdirectory Annexes of the "Excel" zip file with the specified name.
  • The concrete URL.

In column D "Annex Description", you specify the custom name of the Annex, as you wish it to appear in the HTML view of the metadata file later. In case of FILE_DSD, you must specify the referenced DSD + version in brackets. 

 

6. Guidelines

Users with role VALIDATOR may also create and edit Guidelines by file import. The importation mechanism is equivalent to the one for Metadata files, you can choose between 

Using only an SDMX-ML file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to images). Such an "Excel" zip file should have a specific structure and contain an xlsx file. 

Also for the guidelines, it is advised to download first an example file from ESS-MH, conveniently from a report based on the same MSD in order to retrieve the correct template for the SDMX-ML or Excel files used for the import.

Note: Annexes and Publication flags or 'Restricted from publication' flags are not part of the guidelines.

 

6.1 SDMX-ML file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.sdmx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 

The imported file should be archived in an sdmx.zip file when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.sdmx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.sdmx.xml (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.sdmx.xml
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

6.1.1<Header> tag
  • The <Header> is mandatory and should contain all the tags presented in the example, of which the following items need to contain a value
    • <ExportType>EXPORT_GUIDELINES</ExportType>
    • <Sender id> should have the same value as Data Provider <componentValues component="DATA_PROVIDER"> (see below). The <Sender> is generally "ESTAT".
  • A value in <DataSetAction> is optional, but might turn out useful to edit an existing file. Normally the Import is used by checking the "overwrite" box in ESS-MH UI. The content of the new file replaces the old one completely. With the <DataSetAction> tag, an incremental import is facilitated: The application is able to import partial files of the same report while keeping the existing content. Based on the value in the (optional) <DataSetAction> tag in the <Header> the following actions can be carried out:
    • Append: New data (i.e. concepts that are not empty) may be added, but cannot overwrite or delete existing ones.
      • Replace: New data (i.e. concepts that are not empty) may be added and also replace existing one. 
      • Delete: New data (both, concepts that are empty or contain a value) replace existing ones and old ones are deleted if the new ones are missing values.
      • [missing value in the<DataSetAction> tag]: the new file replaces the existing one entirely (= “Delete”, complete overwrite of the file).
      • Note: The "overwrite" checkbox in ESS-MH UI is taken into account only when there is no data set action tag in the SDMX-ML file. The data set action tag has priority over the checkbox.
Header tags example

 

6.1.2<MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD it refers to, including the MSD id, the agency, the version.

An Annotation are included in the Export of guidelines, however are not needed for importing guidelines files, pointing to the Report Structure of the referenced MSD. This information is being kept for the time of transition from SDMX 2.0 to 3.0.

 

6.1.2.1<TargetValues>

The name of the guidelines file in ESS-MH (on an import) is defined based on the mandatory Component Values, as hereunder

Target value example

Tag

Example

<componentValues component="DATA_PROVIDER"> is always "ESTAT"

ESTAT

<componentValues component="DATAFLOW">

DATAFLOW contains:

  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the metadata flow. Here ESMS.
  • The periodicity, as specified in the metadata flow. Here 'A' for annual. 

DMAA_ESMS_A

<componentValues component="TYPOLOGY"> contains the referenced typology in ESS-MH.

 

esms

<componentValues component="AGENCY"> is always "ESTAT"

 

ESTAT

<componentValues component="START_YEAR"> is the reference year from which on the guidelines are applicable.

 

2019

<componentValues component="END_YEAR"> is the reference year until which the guidelines are applicable.

 

2025

<componentValues component="MDFLOW_VERSION"> is the version of the metadata flow concerned.

 

1.1

 

6.1.2.2 Metadata Attributes - The concepts and their values

  • Are reported in a tag <generic:Attribute id=…>
  • Their value should be put in the tag <generic:Value> (without any '<' or '>').
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts values can have String format and can be inserted as

 

Specification 

Example

Simple plain text

Plain text can be imported.

<generic:Value>Here is some plain text.</generic:Value>

HTML entities

HTML entities allowing HTML tags to accomode special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)

<generic:Value>&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p style="padding-left: 30px;"&gt;Text on level 2&amp;nbsp;without bullets or  numbering&lt;/p&gt;</generic:Value>

Images

  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

<generic:Value>  &lt;p&gt;Eurostat&lt;/p&gt;&lt;p&gt;&lt;img        src=

"Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt=

"Brief descriptive text what the image shows" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /&gt;&lt;/p&gt;</generic:Value>

6.1.2.3 Annotations

For concepts / Attributes, that are configured to be not editable by the Metadata providers, an annotation is added to the concerned concept. This annotation has no relevance for the import of guidelines, only serves as information upon export of guidelines from ESS-MH.

Annotations example

 

6.2 Excel file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.xlsx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 

The imported file should be archived in an xlsx.zip file, when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.xlsx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.xlsx (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.xlsx
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

The xlsx file comprises two sheets that serve different purposes and require different degree of filling.

  • Metadata
  • Parameters

 

6.2.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B3:B7.

Header section example

Items

Usage status

Specification

Example

Filename

optional

Equivalent to SDMX-ML file

esms-DMAA_ESMS_A-2019

Data flow ID

mandatory

equivalent to SDMX-ML file, contains:

  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the metadata flow. Here ESMS.
  • The periodicity, as specified in the metadata flow. Here 'A' for annual. 

DMAA_ESMS_A

Data flow version

mandatory

Equivalent to SDMX-ML file

1.1

Typology code

mandatory

Specifies the ESS-MH specific typology referred to, based on the MSD.

esms

Start year

mandatory

Equivalent to SDMX-ML file: The reference year from which on the guidelines are applicable.

2019

End Year

mandatory

Equivalent to SDMX-ML file: The reference year until which the guidelines are applicable.

2020

Data set action

optional

equivalent to SDMX-ML file

 

 

6.2.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with the corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 

The concept values should be put in cells in column B "Concepts". Concept values can be expressed as string.

 

Specification

Example

Simple plain text

Plain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.

Here is some plain text.

HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;&amp;&nbsp;&zeta;&nbsp;&phi;</p>

<p><a href=

"https://ec.europa.eu/eurostat/web/main/home"

</a></p>

Images

  • Should be referenced in one concept as an <img> tag 
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

<p><img src=

"Images/DMBB_SIMS_A_4D_2020_0000_img1.png"

alt="Brief, descriptive text what the image shows" name="DMBB_SIMS_A_4D_2020_0000_img1.png"

/></p>

 

6.2.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the metadata files in ESS-MH. Generally, when you work with a downloaded template of a comparable guideline file from ESS-MH, in sheet "Parameters" editing is neither expected nor advised.

Example of Parameters

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the guidelines file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Lastly, column E identifies whether a concept of the Metadata file is configured as being "Editable" or not.

Only, if you plan to set up your file import per Excel file from scratch with a blank file, you should ensure to align the Parameters sheet with the Metadata sheet respectively.

 

[1] The FILE_DSD always needs a reference to the DSD it is built upon. To retrieve the information on the DSD (ID and version), it is advisable to download a Metadata file from ESS-MH, in which the DSD is provided separately. On import of the Metadata file incl. a FILE_DSD, the DSD does not need to be physically provided, there reference on the FILE_DSD is sufficient.

[2] [1] The FILE_DSD always needs a reference to the DSD it is built upon. To retrieve the information on the DSD (ID and version), it is advisable to download a Metadata file from ESS-MH, in which the DSD is provided separately. On import of the Metadata file incl. a FILE_DSD, the DSD does not need to be physically provided, there reference on the FILE_DSD is sufficient.