The emeScheme - an Introduction

class: center, middle, inverse, title-slide

# The emeScheme - an Introduction
## What is it?<br/>What is it for?<br/>Why do we need it?
<html>
<div style="float:left">

</div>
<hr color='#EB811B' size=1px width=796px>
</html>
### Rainer M Krug
### Group Seminar 18/12/2018<br/>(updated: 2018-12-17)

---

class: inverse, center, middle, animated, fadeIn

.title-slide.remark-slide-content:before{
  position: absolute;
  bottom: 0;
  left: 0;
  content:url(https://zenodo.org/badge/DOI/10.5281/zenodo.2244999.svg);
  height:60px;
}

.left-code {
  color: #777;
  width: 38%;
  height: 92%;
  float: left;
}
.right-plot {
  width: 60%;
  float: right;
  padding-left: 1%;
}
.plot-callout {
  height: 225px;
  width: 450px;
  bottom: 5%;
  right: 5%;
  position: absolute;
  padding: 0px;
  z-index: 100;
}
.plot-callout img {
  width: 100%;
  border: 4px solid #23373B;
}
.middle-text {
  color: #777;
  width: 30%;
  height: 100%;
  float: middle;
}
.left-text {
  color: #777;
  width: 30%;
  height: 100%;
  float: left;
}
.right-text {
  color: #777;
  width: 30%;
  height: 100%;
  float: right;
}

.scroll {
  background: #FFBB33;
  width: 100%;
  height: 80%;
  overflow-x: scroll;
  overflow-y: scroll;
  padding-right: 1px;
  padding-bottom: 1px;
}

.dt {
  font-size: 11px;
}

</style>

# What is the emeScheme?

---
class: left, top, animated, fadeIn

## What is it?

- metadata **Scheme**
- for **E**cological **M**icrocosm **E**xperiments

--
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

## What does this metada scheme do?

- standardised structure
- standardised terminology / property names
- to describe the an Ecological Microcosm Experiment

--
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

## What is the metadata?

- Materials
- Conditions / Manipulations
- Measuring
- Data conversions

---
class: inverse, center, middle, animated, fadeIn

# Why do we need it?

---
class: animated, fadeIn
## Data Life Cylce
.center[
<img src="figs/data_life_cycle_1.svg" width="600">
]
Adapted from https://www.dataone.org/data-life-cycle

???
- **Plan**: description of the data that will becompiled, and how the data will be managed and made accessible throughout its lifetime

- **Collect**: observations are made either by hand  or with sensors or other instruments  and the data are placed a into digital form

- **Assure**: the quality of the data are assured through checks and inspections

- **Analyze**: data are analyzed

- **Preserve**: data are submitted to an appropriate long-term archive (i.e. data center)

- **Discover**: potentially useful data are located and obtained, along with the relevant information about the data

- **Integrate**: data from disparate sources are combined to form one homogeneous set of data that can be readily analyzed

- **Describe**: data are accurately and thoroughly described using the appropriate metadata standards

(metadata)

---
class: animated, fadeIn

## Data Life Cylce

.center[
<img src="figs/data_life_cycle_2.svg" width="600">
]

Adapted from https://www.dataone.org/data-life-cycle

???
- **Plan**: description of the data that will becompiled, and how the data will be managed and made accessible throughout its lifetime

- **Collect**: observations are made either by hand  or with sensors or other instruments  and the data are placed a into digital form

- **Assure**: the quality of the data are assured through checks and inspections

- **Describe**: data are accurately and thoroughly described using the appropriate metadata standards

- **Preserve**: data are submitted to an appropriate long-term archive (i.e. data center)

- **Discover**: potentially useful data are located and obtained, along with the relevant information about the data (metadata)

- **Integrate**: data from disparate sources are combined to form one homogeneous set of data that can be readily analyzed

- **Analyze**: data are analyzed
---
class: left, top, animated, fadeIn

## Why do we want this?

- Data should be archived

- Data should be findable

- Data is Re-usable

- We can re-use other data

- Data should not disappear

- Data should be FAIR

---
class: animated, fadeIn

## What is FAIR?
- **F**indable
- **A**ccessible
- **I**nteroperable
- **R**e-usable

See [https://www.force11.org/group/fairgroup/fairprinciples](https://www.force11.org/group/fairgroup/fairprinciples) for a detailed description.

???
- To be Findable:

- F1. (meta)data are assigned a globally unique and eternally persistent identifier.
  - F2. data are described with rich metadata.
  - F3. (meta)data are registered or indexed in a searchable resource.
  - F4. metadata specify the data identifier.

- TO BE ACCESSIBLE:

- A1  (meta)data are retrievable by their identifier using a standardized communications protocol.
  - A1.1 the protocol is open, free, and universally implementable.
  - A1.2 the protocol allows for an authentication and authorization procedure, where necessary.
  - A2 metadata are accessible, even when the data are no longer available.

- TO BE INTEROPERABLE:

- I1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation.
  - I2. (meta)data use vocabularies that follow FAIR principles.
  - I3. (meta)data include qualified references to other (meta)data.

- TO BE RE-USABLE:

- R1. meta(data) have a plurality of accurate and relevant attributes.
  - R1.1. (meta)data are released with a clear and accessible data usage license.
  - R1.2. (meta)data are associated with their provenance.
  - R1.3. (meta)data meet domain-relevant community standards.
---
class: inverse, center, middle, animated, fadeIn

# We need the emeScheme!

---
class: animated, fadeIn
background-image: url(figs/data_layout_1.svg)
background-position: center
background-size: contain

.left-text[
## Data Levels

- Raw Data
  - videos
  - images
  - count data from manual count

- Extracted data
  - bemovi
  - extracted size distributions
  - extracted count / density data
]
---
class: animated, fadeIn
background-image: url(figs/data_layout_2.svg)
background-position: center
background-size: contain

.right-text[
## Metadata levels

- Experiment
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_3.svg)
background-position: center
background-size: contain

.right-text[
## Metadata levels

- Experiment

- Treatment
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_4.svg)
background-position: center
background-size: contain

.right-text[
## Metadata levels

- Experiment

- Treatment

- Sampling 
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_5.svg)
background-position: center
background-size: contain

.right-text[
## Metadata levels

- Experiment

- Treatment

- Sampling

- Measurement <span>&#8658;</span> **Raw Data**
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_6.svg)
background-position: center
background-size: contain

.right-text[
## Metadata levels

- Experiment

- Treatment

- Sampling

- Measurement <span>&#8658;</span> **Raw Data**

- Data DataExtraction <span>&#8658;</span> **Extracted Data**
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_6.svg)
background-position: center
background-size: contain

.left-text[
## Data Levels

- Raw Data
  - videos
  - images
  - count data from manual count

- Extracted data
  - bemovi
  - extracted size distributions
  - extracted count / density data
]

.right-text[
## Metadata levels

- Experiment

- Treatment

- Sampling

- Measurement <span>&#8658;</span> **Raw Data**

- Data DataExtraction <span>&#8658;</span> **Extracted Data**
]

---
class: animated, fadeIn

.dt[
<div id="htmlwidget-c43c346b008b15c325e5" style="width:100%;height:600px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-c43c346b008b15c325e5">{"x":{"filter":"none","data":[["1","2","3","4","5"],["Experiment","Treatment","Sampling","Measuring","DataExtraction"]],"container":"<table class=\"cell-border stripe\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Property_Level_1<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":18,"dom":"tip","ordering":false,"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}],"lengthMenu":[10,18,25,50,100]}},"evals":[],"jsHooks":[]}</script>
]

---
class: animated, fadeIn

.dt[
<div id="htmlwidget-52a8044d6d816e1d38da" style="width:100%;height:600px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-52a8044d6d816e1d38da">{"x":{"filter":"none","data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39"],["Experiment",null,null,null,"Treatment",null,null,null,null,null,null,null,null,null,null,"Sampling",null,null,null,"Measuring",null,null,null,null,null,null,null,"DataExtraction",null,null,null,null,null,null,null,null,null,null,null],[null,"ExperimentalDesign","Materials","Initialization",null,"DuringExperimentMaintenance","SpeciesTreatment","BacteriaFlagellatesTreatment","TemperatureTreatment","LightTreatment","Dispersal","NutrientTreatment","Viscosity","BioticEnvironment","Disturbance",null,"frequency (daysBetweenSamples) [numeric]","volume (ml) [numeric]","replacementMediumType",null,"Microscopy","VideoRecording","FlowCam","BacterialDensityFlowcytometry","DNABarcoding","Respirometry","NutrientsDecomposition",null,"MicroscopyAnalysis","VideoAnalysis","FlowCamAnalysis","BacterialDensityFlowcytometryAnalysis","ParticleCounterAnalysis","RamanMicrospectroscopyAnalysis","DNABarcodingAnalysis","GenProteoEpigenomicsAnalysis","RespirometryAnalysis","NutrientsDecompositionAnalysis","InteractionsAnalysis"]],"container":"<table class=\"cell-border stripe\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Property_Level_1<\/th>\n      <th>Property_Level_2<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":18,"dom":"tip","ordering":false,"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}],"lengthMenu":[10,18,25,50,100]}},"evals":[],"jsHooks":[]}</script>
]

---
class: animated, fadeIn

.dt[
<div id="htmlwidget-ebff619fb20629630e93" style="width:100%;height:600px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-ebff619fb20629630e93">{"x":{"filter":"none","data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136"],["Experiment",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Treatment",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Sampling",null,null,null,"Measuring",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"DataExtraction",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,"ExperimentalDesign",null,null,null,"Materials",null,null,null,null,null,null,null,null,null,"Initialization",null,null,null,null,null,"DuringExperimentMaintenance",null,null,"SpeciesTreatment",null,null,null,null,"BacteriaFlagellatesTreatment",null,null,null,"TemperatureTreatment",null,null,null,"LightTreatment",null,null,null,"Dispersal",null,null,"NutrientTreatment",null,null,"Viscosity",null,null,"BioticEnvironment",null,null,null,"Disturbance",null,null,null,"frequency (daysBetweenSamples) [numeric]","volume (ml) [numeric]","replacementMediumType",null,"Microscopy",null,null,null,null,null,"VideoRecording",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"FlowCam",null,null,null,null,"BacterialDensityFlowcytometry",null,null,null,null,"DNABarcoding",null,null,"Respirometry",null,null,"NutrientsDecomposition",null,null,null,"MicroscopyAnalysis",null,null,"VideoAnalysis",null,null,"FlowCamAnalysis",null,null,"BacterialDensityFlowcytometryAnalysis",null,null,"ParticleCounterAnalysis",null,null,"RamanMicrospectroscopyAnalysis",null,null,"DNABarcodingAnalysis",null,null,"GenProteoEpigenomicsAnalysis",null,null,"RespirometryAnalysis",null,null,"NutrientsDecompositionAnalysis",null,null,"InteractionsAnalysis",null,null],[null,null,"noTreatments (count) [integer]","noReplicatesPerTreatment (count) [integer]","description",null,"Incubators","Container","Shakers","Species","CultureMedium","BacteriaFlagellates","LabPractices","LongTermMaintenance","LongTermPreservation",null,"InitialConditions","Temperature","Light","FinalConditions",null,null,"description","comment",null,"species","initialDensity (cells/ml) [numeric]","densityChanges","ecologicalRoleInTreatment",null,"species","initial density","densityChanges",null,"temperature (C) [numeric]","temperatureRange (C)","temperatureChangeRate",null,"intensity (lx) [numeric]","colour (nm) [numeric]","photoperiod (h) [numeric]",null,"spatialStructure","change",null,"changeInTreatment","comment",null,"description","comment",null,"type","value","change",null,"description","comment",null,null,null,null,null,null,"volume (ml) [numeric]","preparation","typeOfMicroscope","microscopyMethod","comment (e.g.Species)",null,"volume (ml) [numeric]","preparation","microscope","magnification","camera","microscopyMethod","resolution","colourSpace","lengthOfRecording (sec) [numeric]","frameRate (perSecond) [integer]","lightIntensity (lx) [numeric]","lightColour (nm) [numeric]","videoFormat","comment (e.g.species)",null,"makeTypeVersion","settings Parameter","volume (ml) [numeric]","treatment",null,"typeVersion","volume (ml) [numeric]","settingsParameter","treatment",null,"description","comment",null,"description","moreDetailsNeeded",null,"description (Yves)","comment",null,null,"description","comment",null,"analysisMethod","parameterUsed",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment"]],"container":"<table class=\"cell-border stripe\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Property_Level_1<\/th>\n      <th>Property_Level_2<\/th>\n      <th>Property_Level_3<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":18,"dom":"tip","ordering":false,"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}],"lengthMenu":[10,18,25,50,100]}},"evals":[],"jsHooks":[]}</script>
]

---
class: animated, fadeIn

.dt[
<div id="htmlwidget-af7fb01150237ac51481" style="width:100%;height:600px;" class="datatables html-widget"></div>
<script>HTMLWidgets.pymChild = new pym.Child();HTMLWidgets.addPostRenderHandler(function(){
                                setTimeout(function(){HTMLWidgets.pymChild.sendHeight();},100);
                            });</script>
<script type="application/json" data-for="htmlwidget-af7fb01150237ac51481">{"x":{"filter":"none","data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168"],["Experiment",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Treatment",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Sampling",null,null,null,"Measuring",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"DataExtraction",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,"ExperimentalDesign",null,null,null,"Materials",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"Initialization",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"DuringExperimentMaintenance",null,null,"SpeciesTreatment",null,null,null,null,"BacteriaFlagellatesTreatment",null,null,null,"TemperatureTreatment",null,null,null,"LightTreatment",null,null,null,"Dispersal",null,null,"NutrientTreatment",null,null,"Viscosity",null,null,"BioticEnvironment",null,null,null,"Disturbance",null,null,null,"frequency (daysBetweenSamples) [numeric]","volume (ml) [numeric]","replacementMediumType",null,"Microscopy",null,null,null,null,null,"VideoRecording",null,null,null,null,null,null,null,null,null,null,null,null,null,null,"FlowCam",null,null,null,null,"BacterialDensityFlowcytometry",null,null,null,null,"DNABarcoding",null,null,"Respirometry",null,null,"NutrientsDecomposition",null,null,null,"MicroscopyAnalysis",null,null,"VideoAnalysis",null,null,"FlowCamAnalysis",null,null,"BacterialDensityFlowcytometryAnalysis",null,null,"ParticleCounterAnalysis",null,null,"RamanMicrospectroscopyAnalysis",null,null,"DNABarcodingAnalysis",null,null,"GenProteoEpigenomicsAnalysis",null,null,"RespirometryAnalysis",null,null,"NutrientsDecompositionAnalysis",null,null,"InteractionsAnalysis",null,null],[null,null,"noTreatments (count) [integer]","noReplicatesPerTreatment (count) [integer]","description",null,"Incubators",null,null,"Container",null,null,null,"Shakers",null,null,"Species",null,null,null,null,"CultureMedium",null,null,"BacteriaFlagellates",null,null,null,"LabPractices",null,null,"LongTermMaintenance",null,null,"LongTermPreservation",null,null,null,"InitialConditions",null,null,"Temperature",null,null,null,"Light",null,null,null,"FinalConditions",null,null,null,null,"description","comment",null,"species","initialDensity (cells/ml) [numeric]","densityChanges","ecologicalRoleInTreatment",null,"species","initial density","densityChanges",null,"temperature (C) [numeric]","temperatureRange (C)","temperatureChangeRate",null,"intensity (lx) [numeric]","colour (nm) [numeric]","photoperiod (h) [numeric]",null,"spatialStructure","change",null,"changeInTreatment","comment",null,"description","comment",null,"type","value","change",null,"description","comment",null,null,null,null,null,null,"volume (ml) [numeric]","preparation","typeOfMicroscope","microscopyMethod","comment (e.g.Species)",null,"volume (ml) [numeric]","preparation","microscope","magnification","camera","microscopyMethod","resolution","colourSpace","lengthOfRecording (sec) [numeric]","frameRate (perSecond) [integer]","lightIntensity (lx) [numeric]","lightColour (nm) [numeric]","videoFormat","comment (e.g.species)",null,"makeTypeVersion","settings Parameter","volume (ml) [numeric]","treatment",null,"typeVersion","volume (ml) [numeric]","settingsParameter","treatment",null,"description","comment",null,"description","moreDetailsNeeded",null,"description (Yves)","comment",null,null,"description","comment",null,"analysisMethod","parameterUsed",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment",null,"description","comment"],[null,null,null,null,null,null,null,"makeTypeVersion","comment",null,"type","volume (ml) [numeric]","covers",null,"makeTypeVersion","speed (rpm) [integer]",null,"species","strain","source","density (cells/ml) [numeric]",null,"volume (ml) [numeric]","type",null,"species","strain","sourcePellet",null,"description","comment",null,"description","comment",null,"description","comment",null,null,"description","runInTime (hours) [integer]",null,"temperature (C) [numeric]","temperatureRange (C)","temperatureChangeRate",null,"intensity (lx) [numeric]","colour (nm) [integer]","photoperiod (h) [numeric]",null,"description","comment",null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]],"container":"<table class=\"cell-border stripe\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Property_Level_1<\/th>\n      <th>Property_Level_2<\/th>\n      <th>Property_Level_3<\/th>\n      <th>Property_Level_4<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":18,"dom":"tip","ordering":false,"order":[],"autoWidth":false,"orderClasses":false,"columnDefs":[{"orderable":false,"targets":0}],"lengthMenu":[10,18,25,50,100]}},"evals":[],"jsHooks":[]}</script>
]

---
class: animated, fadeIn
background-image: url(figs/data_layout_6.svg)
background-position: center
background-size: contain

.left-text[
## Data Levels

- Raw Data
  - videos
  - images
  - count data from manual count

- Extracted data
  - bemovi
  - extracted size distributions
  - extracted count / density data
]

.right-text[
## Metadata levels

- Experiment

- Treatment

- Sampling

- Measurement <span>&#8658;</span> **Raw Data**

- Data DataExtraction <span>&#8658;</span> **Extracted Data**
]

---
class: animated, fadeIn, center, middle

**Website** of the **emeScheme**:

[https://exp-micro-ecol-hub.github.io/emeScheme/](https://exp-micro-ecol-hub.github.io/emeScheme/)

![](The_emeScheme_files/figure-html/qr_emeShemeSite-1.png)

**Presentation** at:

[https://rkrug.github.io/emeScheme_Introduction/The_emeScheme.html](https://rkrug.github.io/emeScheme_Introduction/The_emeScheme.html)

![](The_emeScheme_files/figure-html/qr_presentation-1.png)

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.2244999.svg)](https://doi.org/10.5281/zenodo.2244999)