Book page

Technical specifications for file export and import to ESS metadata handler

TECHNICAL SPECIFICATIONS FOR FILE EXPORT AND IMPORT TO ESS-MH

For service support, please contact:

ESTAT-DATA-METADATA-SERVICES@ec.europa.eu

Last reviewed: 09-09-2024
ESSMH web portal

 

1. Introduction

This document provides explanations about the

  • export options for retrieving of Metadata files and MSDs from various tools and databases
  • and particularly import features in ESS MH for SDMX-ML files and Excel files to create or edit Metadata files and Guidelines, along with extra annotations used to be more accurate and compliant with the application ESS-MH.

These annotations cover the following subjects:

  • Importation of concept values of various representation types
  • Images
  • Annexes – at the concepts level & global
  • Publication flag (for the whole Metadata file)
  • Restricted for publication flag (at the concepts level)

 

A metadata file can be exchanged by:

Using only an SDMX-ML file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts and its structure).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to annexes or images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to annexes or images). Such an "Excel" zip file should have a specific structure and contain an xlsx file.

 

2. Export options: MSD and Metadata files

Each metadata and quality report are described by an SDMX Metadata Structure Definition (MSD), and all Metadata Structure Definitions are publicly available in the Euro-SDMX Registry. The MSDs detail which concept IDs must be used and the structure the SDMX-ML file must have to be considered acceptable.

If you are not sure about the underlying MSD in use for a metadata or quality report, you may go to the ESS MH and make sure that you select the “MSD” to be displayed in the Metadata file overview or consult the Metadata flows report (accessible to Super Providers only). 

Displaying MSD information in the ESSMH

and

Metadata flow report

 

You may receive information on the MSD used for a specific metadata or quality reporting by exporting/downloading. To download the MSD, you have three options:

  1. from the Euro-SDMX Registry, where you can download the MSD specifically (and you can find further referenced artefacts there, such as Concept Scheme and Code Lists);
  2. from the ESS-MH, where you can download the metadata file, compressed in a zip file containing the MSD (and information on the Concepts and Code lists used), the metadata file itself and referenced documents, if any;
  3. from the Eurostat website - Data Navigation Tree, Data Browser, Data Explorer and bulk down facility, where you can retrieve published zip files containing metadata file itself and the MSD and Eurostat's new data navigation tree using the Data Browser

 

1. In the Euro-SDMX Registry, search for the concerned MSD and click on the download button. You may opt for different formats v2.1 or v2.0, with or without References included.

Displaying MSD information in the ESSMHDownloading MSDs from the Euro SDMX Registry

 

2. You can go to the ESS-MH, go to the specific report you are interested in and click on Download. The download will include, among others, the MSD and the SDMX-ML version of the metadata or quality report. 

Selecting download button from the Actions columnA pop-up window opens with all the dowlnoaded files

 

3. From the Data Navigation Tree, you can retrieve Published files

  • via the HTML file accessible through the M-icon or  
The M icon on the Eurostat navigation tree

Or the Data Browser

  • directly from the published dataset 
Metadata access from a dataset example

 

Download button on a metadata file example

 

Or 

Data browser bulk downloads photo

 

3. Import options: Import Metadata files to ESS MH

At present, the most common way how to import Metadata files to the ESS MH is "manually" by logging in and going to the "Import" section:

Import feature on the ESSMH

"Browse" the file you would like to import, and click "Import".

A green success message, or a red error message with some indication will be returned consequently.

Furthermore, efforts have been made to facilitate automated Metadata file transmissions via EDAMIS without the need to login to the ESS-MH user interface. If you are interested in such transmission method, please reach out to ESTAT-DATA-METADATA-SERVICES@ec.europa.eu

 

3.1 "SDMX" zip file

It should be used when the metadata file refers to annexes or images.

In ESSMH the files used to be called like xx.sdmx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.

Example: DMBB_SIMS_A_4D_2020_0000.sdmx.zip.

SDMX zip file example

            

This zip file contains:

  • One SDMX-ML file named xx.sdmx.xml (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.sdmx.xml
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_an_nn.ext where:
      •  xx is the metadata file name,
      • nn is a sequential number starting from 1 or a customised name suffix,
      • ext is the extension of the annex according to the type of document.
        Example:DMBB_SIMS_A_4D_2020_0000_an_1.xlsx, DMBB_SIMS_A_4D_2020_0000_an_2.docx
    • Each annex should have a reference in the SDMX-ML file (like "Annexes/xx_an_nn.ext").
    • Note that you can put your own name (instead of xx_an_nn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the SDMX-ML file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the SDMX-ML file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images/YourOwnName.ext".
  • Not actually required, but can be inserted optionally: 
    • one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
      Example: SIMS_MSD.msd.xml
    • DSD file(s) (in SDMX-ML format) named as zz.xml. The DSDs are the basis for validating the structure of tabular files of type FILE_DSD, and they are stored in the ESS-MH, hence do not need to, but may be imported along with the FILE_DSD / structure .xlsx file.

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific application to open them. Therefore, we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

3.2 "Excel" zip file

It should be used when the metadata file refers to annexes or images.

In ESSMH we are used to call the file like xx.xlsx.zip (where xx is the name of the metadata file). Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.

Example: DMBB_SIMS_A_4D_2020_0000.xlsx.zip

SDMX zip file example

This zip file contains:

  • One xlsx file named xx.xlsx (where xx is preferably the metadata file name)
    In our case: DMBB_SIMS_A_4D_2020_0000.xlsx
  • If there are annexes, one directory Annexes/ containing one file per annex:
    • Each annex is named like xx_an_nn.ext, where:
      • xx is the metadata file name,
      • nn is a number starting from 1 or a customised name suffix,
      • ext is the extension of the annex according to the type of document).
        Example: DMBB_SIMS_A_4D_2020_0000_an1.xlsx, DMBB_SIMS_A_4D_2020_0000_an2.docx
    • Each annex should have a reference in the xlsx file (like "Annexes/xx_an_nn.ext").
    • Note that you can put your own name (instead of xx_an_nn), but you should keep the extension, put the file in the "Annexes" directory, and refer to it in the xlsx file by using "Annexes/YourOwnName.ext".
  • If there are images, one directory Images/ containing one file per image:
    • Each image is named like xx_imgnn.ext (where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image).
      Example: DMBB_SIMS_A_4D_2020_0000_img1.png
    • Each image should have a reference in the xlsx file.
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • Not actually required, but can be inserted optionally:  
  • If there are images, one directory Images/ containing one file per image:
    • one MSD file (in SDMX-ML format) named yy.msd.xml (where yy is the name of the MSD in the SDMX registry).
      Example: SIMS_MSD.msd.xml
    • Folder DSDs including the DSD file(s) (in SDMX-ML format) named as zz.xml. The DSDs are the basis for validating the structure of tabular files of type FILE_DSD, and they are stored in ESS-MH, hence do not need to, but may be imported along with the FILE_DSD / structured .xlsx file.

For annexes, all formats are available but keep in mind that every person should be able to open it and sometimes might need a specific program to open them. Therefore, we propose to use "standard" files like PDF, XLSX, DOCX, jpg, png for example.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

4. SDMX-ML file

In order to retrieve the correct template for the SDMX-ML used for the import, it is advised to download first an example file from ESS-MH, conveniently from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.sdmx.xml (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.sdmx.xml

MSD in xml format example part 1MSD in xml format example part 2MSD in xml format example part 3

 

4.1 <Header> tag

The <Header> is mandatory and should contain all the tags presented in the example.

Header in xml format example

 

Some of the tags in the <Header> are mandatory to contain a value,

 
 
ItemUsage statusSpecificationExample
<ID>optionalNo value needed here.DMBB_SIMS_A
<Test>optionalNo value needed here.false
<Name>mandatory

The name of the metadata file is required here. The name is basically a composition of the Target values that need to be provided in the <MetadataSet> tag. It follows the structure 

DataFlow_DataProvider_ TimeDimension

DMBB_SIMS_

A_4D_2020_0000

<Prepared>optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss (like 2020-01-01T00:00:00)2020-11-20T11:34:00

<Sender

ID>

mandatory

The <Sender> should have the same value as the DataProvider in the <TargetValues> in <MetadataSet> tag (<genericmetadata

:ComponentValue component="DATA_PROVIDER" object="DataProvider">).

4D0

<DataSet

ID>

optionalCan take any value, but may optimally be the same as in <Name>. Upon file creation, the DataSetID is generated automatically by the application ESS MH and takes the following form: DataFlow_n where n is an application specific identifier automatically generated for storing the file in the ESS MH database.

DMBB_SIMS_A

_1605868455406

<DataSet

Action>

optional

To edit an existing file, normally the Import is used by checking the "overwrite" box in ESS-MH UI. The content of the new file replaces the old one completely.

With the <DataSetAction> tag, an incremental import is facilitated: The application is able to import partial files of the same report while keeping the existing content. Based on the value in the (optional) <DataSetAction> tag in the <Header> the following actions can be carried out:

  • Append: New data (i.e. concepts that are not empty) may be added, but cannot overwrite or delete existing ones.
  • Replace: New data (i.e. concepts that are not empty) may be added and also replace existing one. 
  • Delete: New data (both, concepts that are empty or contain a value) replace existing ones and old ones are deleted if the new ones are missing values.
  • [missing value in the<DataSetAction> tag]: the new file replaces the existing one entirely (= “Delete”, complete overwrite of the file).

 

Note:

  • The "overwrite" checkbox in ESS-MH UI is taken into account only when there is no action tag in the SDMX-ML file. The action tag has priority over the checkbox.
  • A downloaded file contains by default an empty <DataSetAction> tag.
 
<Extracted>optionalNo value expected here. 

<Reporting

Begin>

optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss.2020-01-01T00:00:00

<Reporting

End>

optionalNo value needed here, but may contain a date in the format yyyy-nn-ddThh:mn:ss.2020-12-31T00:00:00

<DataSet

Agency>

optionalThe <DataSetAgency> tag contains the Agency, which should be defined in the ESS-MH Application. A downloaded file contains by default "ESTAT".ESTAT

4.2 <MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD. EDAMIS pre-validation

 

4.2.1 <TargetValues>

The name of the file in ESS-MH (on an import) is defined based on the Full Target identifiers / the tags. (Only the Full target identifiers are taken into account, the Partial Target identifiers are ignored)

Target values example
 
Tag

<genericmetadata:ComponentValue component="DATAFLOW" object="DataFlow">

DATAFLOW contains:

  • The code for the Domain (Category), here 'DMBB'.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here 'SIMS'.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
  • The version of the Metadata flow is optional, here ':1.0'

<genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider">

  • Contains the specific Organisation code that applies for the Metadata file. It should be the same as the Sender ID as provided in the <Header> tag.

<genericmetadata:ComponentValue component="TIME_PERIOD" object="TimeDimension">

TIME_PERIOD can contain:

  • The year for which we are reporting (here 2020),
  • The periodicity (here 'A' for annual) which is the same as the last character of the DATAFLOW,
  • The value for the period depending on the periodicity (here '0' because there is no period for annual flows). The following periodicities are currently in use in ESS-MH:
PeriodicityDescriptionPossible period2020-A0
2(every 2 years)0
3(every 3 years)0
4(every 4 years)0
5(every 5 years)0
A(Annual)0
M(Monthly)1-12
Q(Quarterly)1-4

 

4.2.2 Reported Metadata Attributes - The concepts and their values
  • Are reported in a tag <genericmetadata:ReportedAttribute conceptID=…>
  • Their value should be put in the tag <genericmetadata:Value> (without any '<' or '>').
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts are specified in the MSD. They can have String format, Date, reference to a Code List (CL_...) or be of type Boolean. In the table below are described some examples for each representation type.
 
 
Metadata attribute / conceptSpecificationExample
String

Simple plain text

Plain text can be imported.

HTML entities, allowing to introduce HTML tags for special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)

Images

  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • <genericmetadata:Value>

    &lt;p&gt;&lt;img src= 

    "Images /xx_imgnn.ext

    alt="Alternative text describing what is shown on the image" name="xx_imgnn.ext" /&gt;&lt;/p&gt;

    </genericmetadata:Value>

    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the SDMX-ML file by using "Images

      /YourOwnName.ext".

  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

Simple plain text

<genericmetadata:

Value>Here is some plain text.

</genericmetadata:Value>

HTML entities

<genericmetadata:Value>

&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;

ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;

abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;

&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p 

style="padding-left: 30px;"&gt;Text

on level 2&amp;nbsp;without bullets 

or  numbering&lt;/p&gt;

</genericmetadata:Value>

Images

<genericmetadata:Value> 

&lt;p&gt;Eurostat&lt;

/p&gt;&lt;p&gt;

&lt;img        src=

"Images/DMBB_SIMS_A_

4D_2020_0000_img1.png" 

alt="text what the image shows" name="DMBB_SIMS_A_

4D_2020_0000_img1.png" 

/&gt;&lt;/p&gt;

</genericmetadata:Value>

Date

The dates contained e.g. in the concepts

META_CERTIFIED

META_POSTED

META_LAST_UPDATE

should be inserted as plain text in format YYYY-MM-DD. Alternatively, also accepted, dd/mm/yyyy

<genericmetadata:Value>

2020-11-20</genericmetadata:Value>

<genericmetadata:Value>

20/11/2020</genericmetadata:Value>

Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accomodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.

Single value code list

<genericmetadata:Value>

codeID

</genericmetadata:Value>

Multiple value code list

<genericmetadata:Value>

codeID1

</genericmetadata:Value>

<genericmetadata:Value>

codeID2

</genericmetadata:Value>

<genericmetadata:Value>

codeID3

</genericmetadata:Value>

<genericmetadata:Value>

codeID4

</genericmetadata:Value>

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1

 

Single value code list

</genericmetadata:

ReportedAttribute conceptID="...">

<genericmetadata:Value>

HIGH</genericmetadata:Value>

</genericmetadata:

ReportedAttribute>

Multiple value code list

<genericmetadata:

ReportedAttribute conceptID="...">

<genericmetadata:Value>

HIGH</genericmetadata:Value>

<genericmetadata:Value>

LOW</genericmetadata:Value>

<genericmetadata:Value>

MEDIUM</genericmetadata:Value>

<genericmetadata:Value>

NOT_APPLICABLE

</genericmetadata:Value>

<genericmetadata:Value>

NOT_AVAILABLE</genericmetadata:

Value>

</genericmetadata:

ReportedAttribute>

BooleanA concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE. The application ESS-MH is not case sensitive in that respect and accepts values also in lowercase true or false.

</genericmetadata:ReportedAttribute conceptID="...">

<genericmetadata:Value>

TRUE</genericmetadata:Value>

</genericmetadata:

ReportedAttribute>

 

</genericmetadata:

ReportedAttribute conceptID="...">

<genericmetadata:Value>

FALSE</genericmetadata:Value>

</genericmetadata:

ReportedAttribute>

 

4.2.3 Annexes

Annexes can be

  • Global to the whole metadata file
  • Or specific to one concept.

Global annexes should be added

  • in the tag <genericmetadata:MetadataSet>,
  • after </genericmetadata:AttributeValueSet>,
  • in a tag <genericmetadata:Annotations>
    and <genericmetadata:Annotation> for each annex.
XML format file example

 

Concept's annexes should be added

  • in the related tag <genericmetadata:ReportedAttribute conceptID=…>
  • in a tag <genericmetadata:Annotations>
    and <genericmetadata:Annotation> for each annex.
ConceptID example

 

The different types of an annex are:

  • FILE - any kind of file formats such as .doc, xlsx, pdf, jpg, etc...
  • File_DSD [1] - SDMX compatible tabular data files that need to be provided in a pair of a DSD and underlying Excel file in .xlsx format. Tabular data files are being validated with STRUVAL.
  • URL - a link to a webpage
Type SpecificationExample
FILE
  • A tag <common:AnnotationType> containing "FILE"
  • A title put in the tag <common:AnnotationTitle>
  • Refer to a file put in the tag <common:AnnotationText>
  • And the file should be present in the subdirectory Annexes of the "SDMX" zip file with the specified name.

<common:Annotation>

        <common:AnnotationType>

FILE</common:AnnotationType>

        <common:AnnotationTitle>

Annex 1 Excel file</common:AnnotationTitle>

<common:AnnotationText>Annexes/

DMBB_SIMS_A_4D_2020_0000_an1.xlsx

</common:AnnotationText>

</common:Annotation>

FILE_DSD
  • A tag <common:AnnotationType> containing "FILE_DSD"
  • A title put in the tag <common:AnnotationTitle> and info on the referenced DSD in brackets
  • Refer to a file put in the tag <common:AnnotationText>
  • And the file should be present in the subdirectory Annexes of the "SDMX" zip file with the specified name.

            <common:Annotation>

               <common:AnnotationType>FILE_DSD

</common:AnnotationType>

               <common:AnnotationTitle>LFS ANNEX [LFS_QR_Multiple+1.0]

</common:AnnotationTitle>

               <common:AnnotationText>Annexes/LFS_SIMS_A_

BE_2023_0000_an_1.xlsx</common:AnnotationText>

            </common:Annotation>

            <common:Annotation>

               <common:AnnotationType>FOR_PUBLICATION

</common:AnnotationType>

               <common:AnnotationText>YES</common:AnnotationText>

            </common:Annotation>

URL
  • A tag <common:AnnotationType> containing "URL"
  • A title put in the tag <common:AnnotationTitle>
  • And an URL put in the tag <common:AnnotationURL>

<common:Annotation>

        <common:AnnotationType>URL

</common:AnnotationType>

        <common:AnnotationTitle>

Eurostat Website</common:AnnotationTitle>

        <common:AnnotationURL>

https://ec.europa.eu/eurostat/

web/main/home

</common:AnnotationURL>

</common:Annotation>


4.2.4 Publication flags and Restriction flags
 SpecificationExample

Publication 

flag for 

whole 

Metadata file

If the Metadata file should be published, an annotation for the "Publication flag"

  • should be present
  • and be set to "YES".
  • in the tag 

    <genericmetadata:

    MetadataSet>,

  • after 

    </genericmetadata:

    AttributeValueSet>,

  • in a tag 

    <genericmetadata:

    Annotations>
    <genericmetadata:

    Annotation>
    <common:AnnotationType> 

    containing "FOR_PUBLICATION"
    <common:AnnotationText> 

    containing "YES".

<common:Annotation>

<common:AnnotationType>

FOR_PUBLICATION<

/common:

AnnotationType>

<common:AnnotationText

>YES<

/common:AnnotationText>

</common:Annotation>

Restricted for 

publication flag

at the 

concepts level

For each concept which values should not be published, an annotation for the "Restricted for publication flag"

  • should be present
  • and set to "YES".

The value "Restricted for publication" will appear as content of the concept when the file is published.

Such flag should be added

  • in the related tag 

    <genericmetadata:

    ReportedAttribute conceptID=…>

  • in a tag 

    <genericmetadata:Annotations>
    <genericmetadata:Annotation>
    <common:AnnotationType> 

    containing 

    "RESTRICTED_FOR_

    PUBLICATION"
    <common:AnnotationText> 

    containing "YES".

<common:Annotation>

<common:AnnotationType>

RESTRICTED_FOR_

PUBLICATION<

/common:AnnotationType>

<common:AnnotationText>

YES<

/common:AnnotationText>

</common:Annotation>

Restricted from

publication flag

at the Annexes

Each Annex, that is being attached to a Metadata file but should not be published, can be flagged as «Restricted from publication» by adding the property «restricted» under Annotation Type 

        <common:

AnnotationType>FILE-restricted<

/common:AnnotationType>

<common:AnnotationType>FILE_DSD-restricted</common:AnnotationType>       

 <common:

AnnotationType>URL-restricted<

/common:AnnotationType>

 

5. xlsx file 

In order to retrieve the correct template for the xlsx used for the import, it is advised to download first an example file from ESS-MH, conveniently a comparable file from a report based on the same MSD/typology.

The name of the file to be prepared for the import is like xx.xlsx (where xx is preferably the metadata file name)
Example: DMBB_SIMS_A_4D_2020_0000.xlsx

The xlsx file comprises three sheets that serve different purposes and require different degree of filling.

  • Metadata
  • Parameters
  • Annexes

 

5.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B4:B8.

Header components example

 

ItemsUsage statusSpecificationExample
Filenameoptionalequivalent to SDMX-ML file

DMBB_SIMS_A_4D_

2020_0000

Published nameoptionalThe download of a metadata file includes also the published name of it, specified by the code of the node in the DataNavigationTree, the file has been attached to, the typology used and, in case of national files, the country code of the file. 

scitech_sims

(or in case of a national file, e.g. scitech_sims_lu)

Data flow IDmandatoryequivalent to SDMX-ML fileDMBB_SIMS_A
Data flow versionmandatoryequivalent to SDMX-ML file1.0
Organization codemandatoryequivalent to SDMX-ML file4D0
Time dimensionmandatoryequivalent to SDMX-ML file2020-A0
For publicationmandatory

If the Metadata file should be published, the value "For Publication" should be set to YES, otherwise to NO.

 

YES
Data set actionoptionalequivalent to SDMX-ML file 

 

5.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with the corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 

 

5.2.1 Concept values

The concept values should be put in cells in column B "Concepts". According to the specifications in the MSD, concept values can be expressed in different representation types:

Metadata attribute / conceptSpecificationExample
String

Simple plain text

Plain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.

Note: ESS-MH also supports the import of values derived from a formula.

HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

Images

  • Should be referenced in one concept as an <img> tag
  • <img src=
  • "Images/xx_imgnn.ext" alt="Alternative text describing what the image shows"name="xx_imgnn.ext"
    • where xx is the metadata file name, nn is a number starting from 1, ext is the extension of the image
    • src contains the reference to the image file in the accompanied Images/ directory
    • name gives the name of the file 
    • Note that you can put your own name (instead of xx_imgnn), but you should keep the extension, put the file in the "Images" directory, and refer to it in the xlsx file by using "Images/YourOwnName.ext".
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

Simple plain text

Here is some plain text.

HTML tags

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;

&amp;&nbsp;

&zeta;&nbsp;&phi;</p>

<p><a href=

"https://ec.europa.eu

/eurostat/

web/main/home"

</a></p

Images

<p><img

src="Images/

DMBB_SIMS_A_4D_

2020_0000_img1.png"alt="Brief, descriptive text what the image shows"

"name="

DMBB_SIMS_A_4D_

2020_0000_img1.png" /></p>

Date

The dates contained e.g. in the concepts

2.1. Metadata last certified (META_CERTIFIED)
2.2. Metadata last posted (META_POSTED)
2.3. Metadata last update (META_LAST_UPDATE)

may be inserted as YYYY-MM-DD. Alternatively, also accepted, plain text with apostrophe in format 'dd/mm/yyyy or formatted as Date, e.g. dd-mm-yyyy.

2020-11-20

'20/11/2020

20-11-2020

Code list

A concept can refer to a specific Code List from which values are to be chosen. The underlying code list is given in the MSD at the respective Metadata Attribute. Details on the Code list with its different Code values can be retrieved from the MSD from the tag <CodeLists>

ESS-MH accomodates single selection of values from a Code list as well as multiple selection. This definition is made in the application exclusively and is not retrievable from the MSD. For such cases, it needs to be clarified with the Metadata Support or the Production Domain, if single or multiple selection mode applies. In case, they have been made available, tailored Guidelines may also provide useful information.

 

Single value code list

codeID in uppercase

Multiple value code list

codeID1,codeID2,

codeID3,codeID4,...

 

 

 

 

 

 

Example of values for concepts, referenced to CL_ESMSIP_GRADE_1

Single value code list

Single value code list example

Multiple value code list

Multiple value code list example

Boolean
A concept can be of type Boolean, which is specified in the MSD at the respective Metadata Attribute. The concept values can be TRUE or FALSE
Boolean example

 

5.2.2 Restricted for publication flag (at the concepts level)

For each concept which should not be published, the corresponding cell in column C headed by "Restricted from publication" should be set to YES. Otherwise NO.

Then the value "Restricted for publication" will appear as content of the concept when the file is published.

 

5.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the Metadata files in ESS MH. Generally, when you work with a downloaded template of a comparable Metadata file from ESS MH, in sheet "Parameters" editing is neither expected nor advised.

Parameters example

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the Metadata file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Lastly, column E identifies whether a concept is configured as being "Editable" or not. In case of "NO", any value inserted in the respective concept in Metadata tab / Annexes tab will not be taken into account on import of the Metadata file.E Only, if you plan to set up your file import per Excel file from scratch with a blank file, or you plan to import only a subset of concepts to ESS-MH, you should ensure to align the Parameters sheet with the Metadata sheet respectively. 

 

5.4 Annexes

Equivalent to SDMX-ML files, Annexes can be attached

  • specifically to one concept,
  • globally to the whole metadata file.

The different types of an annex are:

  • FILE - any kind of file formats such as .doc, .xlsx, pdf, jpg, etc..
  • FILE_DSD [2] - SDMX compatible tabular data files that need to be provided in a pair of a DSD and underlying Excel file in .xlsx format. Tabular data files are being validated with STRUVAL.
  • URL - a link to a webpage

Note: You can attach as many Annexes as necessary, provided you keep the maximum size of the entire Excel.zip file of 10MB.

Annexes need to be added in the corresponding "Annexes" sheet, both for concepts and globally for the whole metadata file.

Example of Annexes

 

In Column A "Concept", you specify whether an Annex is supposed to be attached

  • to a concept by providing here the corresponding concept ID (can be looked up from the Parameters sheet),
  • to the metadata file globally by leaving the cell blank.

In column B "Annex Type" you specify whether an Annex is of the type

  • FILE, or
  • FILE_DSD
  • URL,
  • adding «restricted» in case you want to flag the Annex not to be published.

In column C "Annex Name", you reference the Annex by providing

  • which file needs to be present from the subdirectory Annexes of the "Excel" zip file with the specified name.
  • the concrete URL.

In column D "Annex Description", you specify the custom name of the Annex, as you wish it to appear in the HTML view of the metadata file later. In case of FILE_DSD, you must specify the referenced DSD + version in brackets. 

 

6. Guidelines

Users with role VALIDATOR may also create and edit Guidelines by file import. The importation mechanism is equivalent to the one for Metadata files, you can choose between 

Using only an SDMX-ML file if there is no annex and no image,

  • Should follow the SDMX standard (with a <header> and specific tags…)
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using only an xlsx file if there is no annex and no image,

  • Should follow a certain template complying with the SDMX standard
  • Including the content itself following a valid MSD (Metadata Structure Definition, defining especially the reported concepts).

Using a zip file

  • "SDMX" zip file (necessary if the file refers to images). Such an "SDMX" zip file should have a specific structure and contain an SDMX-ML file.
  • "Excel" zip file (necessary if the file refers to images). Such an "Excel" zip file should have a specific structure and contain an xlsx file. 

Also for the guidelines, it is advised to download first an example file from ESS-MH, conveniently from a report based on the same MSD in order to retrieve the correct template for the SDMX-ML or Excel files used for the import.

Note: Annexes and Publication flags or 'Restricted from publication' flags are not part of the guidelines.

 

6.1 SDMX-ML file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.sdmx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 

The imported file should be archived in an sdmx.zip file when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .sdmx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.sdmx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.sdmx.xml (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.sdmx.xml
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

 

6.1.1<Header> tag
  • The <Header> is mandatory and should contain all the tags presented in the example, of which the following items need to contain a value<ExportType>EXPORT_GUIDELINES</ExportType>
  • <Sender id> should have the same value as Data Provider in the Full target Identifier / <genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider">. The <sender> is generally "ESTAT". <ReportingBegin> and <ReportingEnd> should contain a reference year in the format yyyy (like 2019 and 2020)
Header tags example

 

6.1.2<MetadataSet> tag

The <MetadataSet> tag depends completely on the MSD.

 

6.1.2.1<TargetValues>

The name of the file in ESS-MH (on an import) is defined based on the Full Target Identifiers / the tags. (Only the Full target Identifiers are taken into account, the Partial Target Identifiers are ignored)

Target values tags

 

TagExample
<genericmetadata:ComponentValue component="DATA_PROVIDER" object="DataProvider"> is always "ESTAT"ESTAT

<genericmetadata:ComponentValue component="DATAFLOW" object="DataFlow">

DATAFLOW contains:

  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here ESMS.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
DMAA_ESMS_A
<genericmetadata:ComponentValue component="AGENCY" object="Agency"> is always "ESTAT"ESTAT

<genericmetadata:ComponentValue component="START_YEAR" object="StartYear">

  • The reference year from which on the guidelines are applicable.
2019

<genericmetadata:ComponentValue component="END_YEAR" object="EndYear">

  • The reference year until which the guidelines are applicable.
2020

<genericmetadata:ComponentValue component="MDFLOW_VERSION" object="MdFlowVersion">

  • The version of the Metadata flow concerned.
1.1

 

6.1.2.2 Reported Metadata Attributes - The concepts and their values

  • Are reported in a tag <genericmetadata:ReportedAttribute conceptID=…>
  • Their value should be put in the tag <genericmetadata:Value> (without any '<' or '>').
  • The order and hierarchy should follow the MSD. 
  • The representation types of the concepts values can have String format and can be inserted as

 

 Specification Example
Simple plain textPlain text can be imported.<genericmetadata:Value>Here is some plain text.</genericmetadata:Value>
HTML entities

HTML entities allowing HTML tags to accomode special formatting

  • &amp; becomes & (ampersand)
  • &lt; becomes < (less than)
  • &gt; becomes > (greater than)
<genericmetadata:Value>&lt;p&gt;This is a list with different items&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;ABCDEFGHIJKLMN&lt;/li&gt;
&lt;ul&gt;
&lt;li&gt;abcdefghijklmn&lt;/li&gt;
&lt;ul&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;p&gt;Text on level 1 without bullets or numbering&lt;/p&gt;
&lt;p style="padding-left: 30px;"&gt;Text on level 2&amp;nbsp;without bullets or  numbering&lt;/p&gt;</genericmetadata:Value>
Images
  • Should be referenced in one concept as an <img> tag (where '<' and '>' should be encoded as '&lt;' and '&gt;')
  • And the file should be present in the subdirectory Images of the "SDMX" zip file with the specified name.

<genericmetadata:Value>  &lt;p&gt;Eurostat&lt;/p&gt;&lt;p&gt;&lt;img        src=

"Images/DMBB_SIMS_A_4D_2020_0000_img1.png" alt=

"Brief descriptive text what the image shows" name="DMBB_SIMS_A_4D_2020_0000_img1.png" /&gt;&lt;/p&gt;</genericmetadata:Value>

 

6.2 Excel file - specifics for "Guidelines"

In ESS-MH the guidelines used to be called like esms-xx-YYYY.xlsx (where xx is the name of the metadata flow and YYYY the reference year from which on the guidelines is valid). 

The imported file should be archived in an xlsx.zip file, when the guidelines file refers to images. Note that the file to be imported can have a different name, as long as the extension .xlsx.zip is kept.

Example for guidelines containing Images: esms-DMAA_ESMS_A-2019.xlsx.zip.
The zip file contains:

  • One SDMX-ML file named esms-xx-YYYY.xlsx (where xx is preferably the metadata flow name)
    In our case: esms-DMAA_ESMS_A-2019.xlsx
  • If there are images, one directory Images/ containing one file per image.

For images, JPEG (.jpg) is probably the lightest format but .png, .gif and .bmp are also accepted.

Currently there is a limit for the zip file of 10 MB.

The xlsx file comprises two sheets that serve different purposes and require different degree of filling.

  • Metadata
  • Parameters

 

6.2.1 Metadata - "Header" section

Most parts in the Header section are mandatory and should contain the items presented in the example.

The Metadata file will be assembled based on the values provided in cells B3:B7.

Header section example

 

ItemsUsage statusSpecificationExample
FilenameoptionalEquivalent to SDMX-ML fileesms-DMAA_ESMS_A-2019
Data flow IDmandatory

equivalent to SDMX-ML file, contains:

  • The code for the Domain (Category), here DMAA.
  • An additional label indicating also the report structure used, as specified in the Metadata flow. Here ESMS.
  • The periodicity, as specified in the Metadata flow. Here 'A' for annual. 
DMAA_ESMS_A
Data flow versionmandatoryEquivalent to SDMX-ML file1.1
typology codemandatorySpecifies the ESS-MH specific typology referred to, based on the MSD.esms
Start yearmandatoryEquivalent to SDMX-ML file: The reference year from which on the guidelines are applicable.2019
End YearmandatoryEquivalent to SDMX-ML file: The reference year until which the guidelines are applicable.2020

 

6.2.2 Metadata "Concepts" section 

The Concepts section depends completely on the MSD. The order and hierarchy should follow the MSD. 

Column A contains the "Concept name" with thee corresponding numbering based on the typology in ESS-MH. This information is retrievable from the underlying MSD to a certain extent, while the numbering is autogenerated by the application ESS-MH upon building the typology. Otherwise, this information can be obtained directly from the ESS-MH, or can be clarified with the Metadata Support or the Production Domain. 

The concept values should be put in cells in column B "Concepts". Concept values can be expressed as string.

 SpecificationExample
Simple plain textPlain text can be imported. Also carriage returns are accepted and turned into </br> tags upon import.Here is some plain text.
HTML tags

A downloaded metadata file in xlsx will contain values  with HTML tags, hence they are accepted respectively on import.

  • <p>, </br>, ...
  • <ol>, <ul>, <li>, ...
  • <a>, ...
  • <table>. <tbody>, <tr>, <td>, ...
  • <strong>, <span>, <em>...
  • <sub>, <sup>, ...
  • ...

<p><strong>This is text in bold.</strong></p>
<p><em>This is text in italics.</em></p>
<p><span style="text-decoration: underline;">This is text with underline.</span></p>
<p><span style="color: #ff0000;">This is text in red.</span></p>
<p><span style="background-color: #ffff00;">This is text with background color yellow.</span></p>
<p>This is text in <sub>subscript</sub>.</p>
<p>This is text in <sup>superscript</sup>.</p>
<p>These are symbols/special character:&nbsp;&amp;&nbsp;&zeta;&nbsp;&phi;</p>

<p><a href=

"https://ec.europa.eu/eurostat/web/main/home"

</a></p>

Images
  • Should be referenced in one concept as an <img> tag 
  • And the file should be present in the subdirectory Images of the "Excel" zip file with the specified name.

<p><img src=

"Images/DMBB_SIMS_A_4D_2020_0000_img1.png"

alt="Brief, descriptive text what the image shows" name="DMBB_SIMS_A_4D_2020_0000_img1.png"

/></p>

 

6.2.3 Parameters

The second sheet is a Parameter sheet that specifies how the items and data in sheet "Metadata" should be coded and implemented in the Metadata files in ESS MH. Generally, when you work with a downloaded template of a comparable guideline file from ESS MH, in sheet "Parameters" editing is neither expected nor advised.

Example of Parameters

In column A "Element name" you can find all items as per column A in the Metadata sheet.

Column B "Element code" lists the related IDs for the items and concepts.

Column C "Type" specifies the items further in terms of function and representation in the guidelines file while column D "Position" indicates the rows of the items of Metadata sheet, i.e. references the elements from the two sheets Metadata and Parameters and defines, which values will be taken into account for the importation. 

Lastly, column E identifies whether a concept of the Metadata file is configured as being "Editable" or not.

Only, if you plan to set up your file import per Excel file from scratch with a blank file, you should ensure to align the Parameters sheet with the Metadata sheet respectively.

 

[1] The FILE_DSD always needs a reference to the DSD it is built upon. To retrieve the information on the DSD (ID and version), it is advisable to download a Metadata file from ESS-MH, in which the DSD is provided separately. On import of the Metadata file incl. a FILE_DSD, the DSD does not need to be physically provided, there reference on the FILE_DSD is sufficient.

[2] [1] The FILE_DSD always needs a reference to the DSD it is built upon. To retrieve the information on the DSD (ID and version), it is advisable to download a Metadata file from ESS-MH, in which the DSD is provided separately. On import of the Metadata file incl. a FILE_DSD, the DSD does not need to be physically provided, there reference on the FILE_DSD is sufficient.