trylkstopocket - Tumblr blog

trylkstopocket · 2 years ago

Text

RO-crate spec (w\o code)

This document specifies a method, known as RO-Crate (Research Object Crate), of aggregating and describing research data with associated metadata. RO-Crates can aggregate and describe any resource including files, URI-addressable resources, or use other addressing schemes to locate digital or physical data. RO-Crates can describe data in aggregate and at the individual resource level, with metadata to aid in discovery, re-use and long term management of data. Metadata includes the ability to describe the context of data and the entities involved in its production, use and reuse. For example: who created it, using which equipment, software and workflows, under what licenses can it be re-used, where was it collected, and/or where is it about.

RO-Crate uses JSON-LD to express this metadata using linked data, describing data resources as well as contextual entities such as people, organizations, software and equipment as a series of linked JSON-LD objects - using common published vocabularies, chiefly schema.org.

The core of RO-Crate is a JSON-LD file, the RO-Crate Metadata File, named ro-crate-metadata.json. This file contains structured metadata about the dataset as a whole (the Root Data Entity) and, optionally, about some or all of its files. This provides a simple way to, for example, assert the authors (e.g. people, organizations) of the RO-Crate or one its files, or to capture more complex provenance for files, such as how they were created using software and equipment.

While providing the formal specification for RO-Crate, this document also aims to be a practical guide for software authors to create tools for generating and consuming research data packages, with explanation by examples.

3 Terminology

RO-Crate: A directory structure that contains a dataset, which is described in an RO-Crate Metadata File.

RO-Crate Root: The top-level directory of the RO-Crate, indicated by the presence of the RO-Crate Metadata File ro-crate-metadata.json (or ro-crate-metadata.jsonld for crates that comply with versions before v1.1 of this specification)

RO-Crate Metadata File: A JSON-LD file stored as ro-crate-metadata.json in the RO-Crate Root. The metadata file describes the RO-Crate with structured data in form of RO-Crate JSON-LD. (In version 1.0 this file was named ro-crate-metadata.jsonld but has been renamed to improve the usability of crates.)

RO-Crate Website: Human-readable HTML pages which describe the RO-Crate (i.e. the Root Data Entity, its Data Entities and Context Entities), with a home-page at ro-crate-preview.html (any additional files reside in ro-crate-preview_files/)

Entity: An identified object, which have a given type and may be described using a set of properties.

Type: A classification of objects or their descriptions. The type (or class) is identified by a URI, mapped to a key by JSON-LD.

Property: A relationship from one entity to another entity, or to a value. The type of relationship is identified by a URI, mapped to a key by JSON-LD.

Data Entity: A JSON-LD representation, in the RO-Crate Metadata File, of a directory, file or other resource contained or described by the RO-Crate.

Root Data Entity: A Data Entity of type Dataset, representing the RO-Crate as a whole.

RO-Crate Metadata File Descriptor: A Contextual Entity of type CreativeWork, which describes the RO-Crate Metadata File and links it to the Root Data Entity.

JSON-LD: A JSON-based file format for storing Linked Data. This document assumes JSON-LD 1.0. JSON-LD use a context to map from JSON keys to URIs.

JSON: The JavaScript Object Notation (JSON) Data Interchange Format as defined by RFC 7159; a structured text file format that can be programmatically consumed and generated in a wide range of programming languages. The main JSON structures are objects ({}) indexed by keys, sequential arrays ([]) and literal values ("").

Contextual Entity: A JSON-LD representation of an entity associated with a Data Entity, needed to adequately describe that Data Entity. For example, a Person, Organization (including research projects), item of equipment (IndividualProduct), license or any other thing or event that forms part of the metadata for a Data Entity or supporting information.

Linked Data: A data structure where properties, types and resources are identified with URIs, which if retrieved over the Web, further describe or provide the identified property/type/resource.

URI: A Uniform Resource Identifier as defined in RFC 3986, for example http://example.com/path/file.html - commonly known as URL. In this document the term URI includes IRI, which also permit international Unicode characters.

URI Path: The relative path element of an URI as defined in RFC3986 section 3.3, e.g. path/file.html

RO-Crate JSON-LD Context: A JSON-LD context that provides Linked Data mapping for RO-Crate metadata to vocabularies like Schema.org.

RO-Crate JSON-LD: JSON-LD structure using the RO-Crate JSON-LD Context and containing RO-Crate metadata, written as if flattened and then compacted according to the rules in JSON-LD 1.0. The RO-Crate JSON-LD for an RO-Crate is stored in the RO-Crate Metadata File.

3.1 Linked Data conventions

Throughout this specification, RDF terms (properties, types) are referred to using the keys defined in the RO-Crate JSON-LD Context.

Following Schema.org practice, property names start with lowercase letters and Type names start with uppercase letters.

In the RO-Crate Metadata File the RDF terms use their RO-Crate JSON-LD names as defined in the RO-Crate JSON-LD Context, which is available at https://w3id.org/ro/crate/1.1/context

4 RO-Crate Structure

The structure an RO-Crate MUST follow is:

The name of the RO-Crate root directory is not defined, but a root directory is identifiable by the presence of the RO-Crate Metadata File, ro-crate-metadata.json. For instance, if an RO-Crate is archived in a ZIP-file, the ZIP root directory is an RO-Crate root directory if it contains ro-crate-metadata.json.

Data Entities in the RO-Crate MUST either be payload files/directories present within the RO-Crate root directory or its subdirectories, or be Web-based Data Entities.

4.1 RO-Crate Metadata File (ro-crate-metadata.json)

In new RO-Crates the RO-Crate Metadata File MUST be named ro-crate-metadata.json and appear in the RO-Crate Root

The RO-Crate Metadata File MUST contain RO-Crate JSON-LD; a valid JSON-LD 1.0 document in flattened and compacted form

The RO-Crate JSON-LD SHOULD use the RO-Crate JSON-LD Context https://w3id.org/ro/crate/1.1/context by reference.

If an RO-Crate conforming to version 1.0 or earlier contains a file named ro-crate-metadata.jsonld instead of ro-crate-metadata.json then processing software should treat this as the RO-Crate Metadata File. If the crate is updated then the file SHOULD be renamed to ro-crate-metadata.json and the RO-Crate Metadata File Descriptor SHOULD be updated to reference it, with an up to date conformsTo property naming an appropriate version of this specification.

JSON-LD is a structured form of JSON that can represent a Linked Data graph.

A valid RO-Crate JSON-LD graph MUST describe:

The RO-Crate Metadata File Descriptor

The Root Data Entity

Zero or more Data Entities

Zero or more Contextual Entities

It is RECOMMENDED that any referenced contextual entities are also described in the RO-Crate Metadata File with the same identifier. Similarly it is RECOMMENDED that any contextual entity in the RO-Crate Metadata file is linked to from at least one of the other entities using the same identifier.

The appendix RO-Crate JSON-LD details the general structure of the JSON-LD that is expected in the RO-Crate Metadata File. In short, the rest of this specification describe the different types of entities that can be added as {} objects to the RO-Crate JSON-LD @graph array below:

4.2 RO-Crate Website (ro-crate-preview.html and ro-crate-preview_files/)

In addition to the machine-oriented RO-Crate Metadata File, the RO-Crate MAY include a human-readable HTML rendering of the same information, known as the RO-Crate Website.

If present in the root directory, ro-crate-preview.html MUST:

Be a valid HTML 5 document

Be useful to users of the RO-Crate - this will vary by community and intended use, but in general the aim to assist users in reusing data by explaining what it is, how it was created how it can be used and how to cite it. One simple approach to this is to expose all the metadata in the RO-Crate Metadata File.

Contain a copy of the RO-Crate JSON-LD in a script element of the head element of the HTML, for example:

ro-crate-preview.html SHOULD:

Display at least the metadata relating to the Root Data Enity as static HTML without the need for scripting. It MAY contain extra features enabled by JavaScript.

When a Data Entity or Contextual Entity is referenced by its ID:

If it has a name property, provide a link to its HTML version.

If it does not have a name (e.g. a GeoCoordinates location), show it embedded in the HTML for the entity.

For external URI values, provide a link.

For keys that resolve in the RO-Crate JSON-LD Context to a URI, indicate this (the simplest way is to link the key to its definition).

If there is sufficient metadata, contain a prominent "Cite-as" text with a natural language data citation (see for example the FORCE11 Data Citation Principles).

If there are additional resources necessary to render the preview (e.g. CSS, JSON, HTML), link to them in a subdirectory ro-crate-preview-files/

4.3 Payload files and directories

These are the actual files and directories that make up the dataset being described.

The base RO-Crate specification makes no assumptions about the presence of any specific files or folders beyond the reserved RO-Crate files described above. Payload files may appear directly in the RO-Crate Root alongside the RO-Crate Metadata File, and/or appear in sub-directories of the RO-Crate Root. Each file and directory MAY be represented as Data Entities in the RO-Crate Metadata File.

4.4 Self-describing and self-contained

RO-Crates SHOULD be self-describing and self-contained.

A minimal RO-Crate is a directory containing a single RO-Crate Metadata File ro-crate-metadata.json.

At the basic level, an RO-Crate is a collection of files and resources represented as a Schema.org Dataset, that together form a meaningful unit for the purposes of communication, citation, distribution, preservation, etc. The RO-Crate Metadata File describes the RO-Crate, and MUST be stored in the RO-Crate Root.

While RO-Crate is well catered for describing a Dataset as files and relevant metadata that are contained by the RO-Crate in the sense of living within the same root directory, RO-Crates can also reference external resources which are stored or accessed separately, via absolute URIs. This is particularly recommended where some resources cannot be co-hosted for practical or legal reasons, or if the RO-Crate itself is primarily web-based.

It is important to note that the RO-Crate Metadata File is not an exhaustive manifest or inventory, that is, it does not necessarily list or describe all files in the package. Rather it is focused on providing sufficient amount of metadata to understand and use the content, and is designed to be compatible with existing and future approaches that do have full inventories / manifest and integrity checks, e.g. by using checksums, such as BagIt and Oxford Common File Layout OCFL Objects.

The intention is that RO-Crates can work well with a variety of archive file formats, e.g. tar, zip, etc., and approaches to capturing file manifests and file fixity, such as BagIt, OCFL and git (see also appendix Combining with other packaging schemes). An RO-Crate can also be hosted on the web or mainly refer to web resources, although extra care to ensure persistence and consistency should be taken for archiving such RO-Crates.

5 RO-Crate Metadata

RO-Crate aims to capture and describe the Research Object using structured metadata.

The RO-Crate Metadata File Descriptor contains the metadata that describes the RO-Crate and its content, in particular:

Root Data Entity - the RO-Crate Dataset itself, a gathering of data

Data Entities - the data payload, in the form of files and folders

Contextual Entities - related things in the world (e.g. people, organizations, places), providing provenance for the data entities and the RO-Crate.

This machine-readable metadata can also be represented for human consumption in the RO-Crate Website, linking to data and Web resources.

5.1 RO-Crate uses Linked Data principles

RO-Crate makes use of the Linked Data principles for its description. In particular:

(Meta)data should be made available as Open Data on the web.

(Meta)data should be machine-readable in a structured format.

(Meta)data should not require proprietary software packages.

(Meta)data should use open standards from W3C, such as RDF and SPARQL.

(Meta)data should link to other people's data to provide context, using URIs as global identifiers

RO-Crate realizes these principles using a particular set of technologies and best practices:

The RO-Crate Metadata File and RO-Crate Website can be directly published on the web together with the RO-Crate payload. In addition, a data package (e.g. BagIt Zip archive) that contain the RO-Crate can also be published on the web.

The RO-Crate Metadata File is based on the structured data format JSON-LD.

Multiple open source tools/libraries are available for JSON and for JSON-LD.

The RO-Crate Website is HTML 5, and the RO-Crate Metadata File is JSON-LD, one of the W3C RDF 1.1 formats.

The RO-Crate Metadata File reuse common vocabularies like Schema.org, and this specification recommend identifiers it should link to.

5.2 Base metadata standard: Schema.org

Schema.org is the base metadata standard for RO-Crate. Schema.org was chosen because it is widely used on the World Wide Web and supported by search engines, on the assumption that discovery is likely to be maximized if search engines index the content.

Note: As far as we know there is no alternative, well-maintained linked-data schema for research data with the coverage needed for this project - i.e. a single standard for expressing all the examples presented in this specification.

RO-Crate relies heavily on Schema.org, using a constrained subset of JSON-LD, and this document gives opinionated recommendations on how to represent the metadata using existing linked data best practices.

5.2.1 Differences from Schema.org

Generally, the standard type and property names (terms) from Schema.org should be used. However, RO-Crate uses variant names for some elements, specifically:

File is mapped to http://schema.org/MediaObject which was chosen as a compromise as it has many of the properties that are needed to describe a generic file. Future versions of Schema.org or a research data extension may re-define File.

Journal is mapped to http://schema.org/Periodical.

Warning: JSON-LD examples given on the Schema.org website may not be in flattened form; any nested entities in RO-Crate JSON-LD SHOULD be described as separate contextual entities in the flat @graph list.

To simplify processing and avoid confusion with string values, the RO-Crate JSON-LD Context requires URIs and entity references to be given in the form "author": {"@id": "http://example.com/alice"}, even where Schema.org for some properties otherwise permit shorter forms like "author": "http://example.com/alice".

See the appendix RO-Crate JSON-LD for details.

5.3 Additional metadata standards

RO-Crate also uses the Portland Common Data Model (PCDM version https://pcdm.org/2016/04/18/models) to describe repositories or collections of digital objects and imports these terms:

RepositoryObject mapped to http://pcdm.org/models#Object

RepositoryCollection mapped to http://pcdm.org/models#Collection

RepositoryFile mapped to http://pcdm.org/models#File

hasMember mapped to http://pcdm.org/models#hasMember

hasFile mapped to http://pcdm.org/models#hasFile

Note: The terms RepositoryObject and RepositoryCollection are renamed to avoid collision between other vocabularies and the PCDM terms Collection and Object. The term RepositoryFile is renamed to avoid clash with RO-Crate's File mapping to http://schema.org/MediaObject.

From Dublin Core Terms RO-Crate use:

conformsTo mapped to http://purl.org/dc/terms/conformsTo

These terms are being proposed by Bioschemas profile ComputationalWorkflow 0.5-DRAFT and FormalParameter 0.1-DRAFT to be integrated into Schema.org:

ComputationalWorkflow mapped to https://bioschemas.org/ComputationalWorkflow

FormalParameter mapped to https://bioschemas.org/FormalParameter

input mapped to https://bioschemas.org/ComputationalWorkflow#input

output mapped to https://bioschemas.org/ComputationalWorkflow#output

funding mapped to http://schema.org/funding (schemaorg #383)

Note: In this specification the proposed Bioschemas terms use the temporary https://bioschemas.org/ namespace; future releases of RO-Crate may reflect mapping to the http://schema.org/ namespace.

5.4 Summary of Coverage

RO-Crate is simply a way to make metadata assertions about a set of files and folders that make up a Dataset. These assertions can be made at two levels:

Assertions at the RO-Crate level: for an RO-Crate to be useful, some metadata should be provided about the dataset as a whole (see minimum requirements for different use-cases below). In the RO-Crate Metadata File, we distinguish the Root Data Entity which represents the RO-Crate as a whole, from other Data Entities (files and folders contained in the RO-Crate) and Contextual Entities, e.g. a person, organisation, place related to an RO-Crate Data Entity

Assertions about files and folders contained in the RO-Crate: in addition to providing metadata about the RO-Crate as a whole, RO-Crate allows metadata assertions to be made about any other Data Entity

This document has guidelines for ways to represent common requirements for describing data in a research context, e.g.:

Contact information for a data set.

Descriptive information for a dataset and the files within it and their contexts such as an abstract, spatial and temporal coverage.

Associated publications.

Funding relationships.

Provenance information of various kinds; who (people and organizations) and what (instruments and computer programs) created or contributed to the data set and individual files within it.

Workflows that operate on the data using standard workflow descriptions including 'single step workflows'; executable files or environments such as singularity containers or Jupyter notebooks.

However, as RO-Crate uses the Linked Data principles, adopters of RO-Crate are free to supplement RO-Crate using Schema.org metadata and/or assertions using other Linked Data vocabularies.

5.5 Future coverage

A future version of this specification aim to cater for variable-level assertions: In some cases, e.g. for tabular data, additional metadata may be provided about the structure and variables within a given file. See the use case Describe a tabular data file directly in RO-Crate metadata for work-in-progress.

5.6 Recommended Identifiers

RO-Crate JSON-LD SHOULD use the following IDs where possible:

For a Root Data Entity, an identifier which is RECOMMENDED to be a https://doi.org/ URI.

For a Person participating in the research process: ORCID identifiers, e.g. https://orcid.org/0000-0002-1825-0097

For Organizations including funders, Research Organization Registry URIs, e.g. https://ror.org/0384j8v12

For entities of type Place, a geonames URL, e.g. http://sws.geonames.org/8152662/

For file formats, a Pronom URL, for example https://www.nationalarchives.gov.uk/PRONOM/fmt/831.

In the absence of the above, RO-Crates SHOULD contain stable persistent URIs to identify all entities wherever possible.

6 Root Data Entity

The Root Data Entity is a Dataset that represent the RO-Crate as a whole; a Research Object that includes the Data Entities and the related Contextual Entities.

As explained in section RO-Crate structure, the RO-Crate description is stored as JSON-LD in the RO-Crate Metadata File ro-crate-metadata.json in the RO-Crate root directory.

6.1 RO-Crate Metadata File Descriptor

The RO-Crate JSON-LD MUST contain a self-describing RO-Crate Metadata File Descriptor with the @id value ro-crate-metadata.json (or ro-crate-metadata.jsonld in legacy crates) and @type CreativeWork. This descriptor MUST have an about property referencing the Root Data Entity, which SHOULD have an @id of ./.

The conformsTo of the RO-Crate Metadata File Descriptor SHOULD be a versioned permalink URI of the RO-Crate specification that the RO-Crate JSON-LD conforms to. The URI SHOULD start with https://w3id.org/ro/crate/.

6.1.1 Finding the Root Data Entity

Consumers processing the RO-Crate as an JSON-LD graph can thus reliably find the Root Data Entity by following this algorithm:

For each entity in @graph array

..if the conformsTo property is a URI that starts with https://w3id.org/ro/crate/

....from this entity's about object keep the @id URI as variable root

For each entity in @graph array

.. if the entity has an @id URI that matches root return it

See also the appendix on finding RO-Crate Root in RDF triple stores.

6.1.2 Purpose of Metadata File

To ensure a base-line interoperability between RO-Crates, and for an RO-Crate to be considered a Valid RO-Crate, a minimum set of metadata is required for the Root Data Entity. As stated earlier the RO-Crate Metadata File is not an exhaustive manifest or inventory, that is, it does not necessarily list or describe all files in the package. For this reason, there are no minimum metadata requirements in terms of describing Data Entities (files and folders) other than the Root Data Entity. Extensions of RO-Crate dealing with specific types of dataset may put further constraints or requirements of metadata beyond the Root Data Entity (see the appendix Extending RO-Crate).

The RO-Crate Metadata File Descriptor MAY contain information such as licensing for the RO-Crate Metadata File so metadata can be licensed separately from Data.

The table below outlines the properties that the Root Data Entity MUST have to be minimally valid and additionally highlights properties required to meet other common use-cases:

6.2 Direct properties of the Root Data Entity

The Root Data Entity MUST have the following properties:

@type: MUST be Dataset

@id: MUST end with / and SHOULD be the string ./

name: SHOULD identify the dataset to humans well enough to disambiguate it from other RO-Crates

description: SHOULD further elaborate on the name to provide a summary of the context in which the dataset is important.

datePublished: MUST be a string in ISO 8601 date format and SHOULD be specified to at least the precision of a day, MAY be a timestamp down to the millisecond.

license: SHOULD link to a Contextual Entity in the RO-Crate Metadata File with a name and description. MAY have a URI (eg for Creative Commons or Open Source licenses). MAY, if necessary be a textual description of how the RO-Crate may be used.

Note: These requirements are stricter than those published > for Google Dataset Search which > requires a Dataset to have a name and description,

Warning: The properties above are not sufficient to generate a DataCite citation. Advice on integrating with DataCite will be provided in a future version of this specification, or as an implementation guide.

6.3 Minimal example of RO-Crate

The following RO-Crate Metadata File represents a minimal description of an RO-Crate.

7 Data Entities

The primary purpose for RO-Crate is to gather and describe a set of Data entities in the form of:

Files

Directories

Web resources

The data entities can be further described by referencing contextual entities such as persons, organizations and publications.

7.1 Referencing files and folders from the Root Data Entity

Where files and folders are represented as Data Entities in the RO-Crate JSON-LD, these MUST be linked to, either directly or indirectly, from the Root Data Entity using the hasPart property. Directory hierarchies MAY be represented with nested Dataset Data Entities, or the Root Dataset MAY refer to files anywhere in the hierarchy using hasPart.

Data Entities representing files MUST have "File" as a value for @type. File is an RO-Crate alias for http://schema.org/MediaObject. The term File here is liberal, and includes "downloadable" resources where @id is an absolute URI.

Data Entities representing directories MUST be of "@type": "Dataset". The term directory here includes HTTP file listings where @id is an absolute URI, however "external" directories SHOULD have a programmatic listing of their content (e.g. another RO-Crate).

Data Entities can also be other types, for instance an online database. These SHOULD be of "@type": "CreativeWork" and typically have a @id which is an absolute URI.

In all cases, @type MAY be an array in order to also specify a more specific type, e.g. "@type": ["File", "ComputationalWorkflow"]

Tip: There is no requirement to represent every file and folder in an RO-Crate as Data Entities in the RO-Crate JSON-LD.

7.1.1 Example linking to a file and folders

An example RO-Crate JSON-LD for the above would be as follows:

7.1.2 Adding detailed descriptions of encodings

The above example provides a media type for the file cp7glop.ai - which is useful as it may not be apparent that the file is readable as a PDF file from the extension alone. To add more detail, encodings SHOULD be linked using a PRONOM identifier to a Contextual Entity of @type WebSite.

If there is no PRONOM identifier, then a contextual entity with a URL as an @id MAY be used:

If there is no web-accessible description for a file format it SHOULD be described locally in the dataset.

7.2 Core Metadata for Data Entities

The table below outlines the properties that Data Entities, when present, MUST have to be minimally valid.

7.2.1 Encoding file paths

Note that all @id identifiers must be valid URI references, care must be taken to express any relative paths using / separator, correct casing, and escape special characters like space (%20) and percent (%25), for instance a File Data Entity from the Windows path Results and Diagrams\almost-50%.png becomes "@id": "Results%20and%20Diagrams/almost-50%25.png" in the RO-Crate JSON-LD.

In this document the term URI includes international IRIs; the RO-Crate Metadata File is always UTF-8 and international characters in identifiers SHOULD be written using native UTF-8 characters (IRIs), however traditional URL encoding of Unicode characters with % MAY appear in @id strings. Example: "@id": "面试.mp4" is preferred over the equivalent "@id": "%E9%9D%A2%E8%AF%95.mp4"

7.2.2 File Data Entity

A File Data Entity MUST have the following properties:

@type: MUST be File, or an array where File is one of the values.

@id MUST be either a URI Path relative to the RO Crate root, or an absolute URI.

7.2.3 Directory File Entity

A Dataset (directory) Data Entity MUST have the following properties:

@type MUST be Dataset or an array where Dataset is one of the values.

@id MUST be either a URI Path relative to the RO Crate root, or an absolute URI. The id SHOULD end with /

7.3 Web-based Data Entities

While one use-case of RO-Crates is to describe files contained within the RO-Crate root directory, RO-Crates can also gather resources from the web identified by absolute URIs instead of relative URI paths, i.e. Web-based data entities.

Using Web-based data entities can be important particularly where a file can't be included in the RO-Crate root because of licensing concerns, large data sizes, privacy, or where it is desirable to link to the latest online version.

Example of an RO-Crate including a File Data Entity external to the RO-Crate root (file entity https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf):

Additional care SHOULD be taken to improve persistence and long-term preservation of web resources included in an RO-Crate as they can be more difficult to archive or move along with the RO-Crate root, and may change intentionally or unintentionally leaving the RO-Crate with incomplete or outdated information.

File Data Entries with an @id URI outside the RO-Crate Root SHOULD at the time of RO-Crate creation be directly downloadable by a simple retrieval (e.g. HTTP GET), permitting redirections and HTTP/HTTPS authentication. For instance, in the example above, https://zenodo.org/record/3541888 and https://doi.org/10.5281/zenodo.3541888 cannot be used as @id above as retrieving these URLs give a HTML landing page rather than the desired PDF as indicated by encodingFormat.

As files on the web may change, the timestamp property sdDatePublished SHOULD be included to indicate when the absolute URL was accessed, and derived metadata like encodingFormat and contentSize were considered to be representative.

7.3.1 Embedded data entities that are also on the web

File Data Entities may already have a corresponding web presence, for instance a landing page that describes the file, including persistent identifiers (e.g. DOI) resolving to an intermediate HTML page instead of the downloadable file directly.

These can be included for File Data Entities as additional metadata, regardless of whether the File is included in the RO-Crate Root directory or exists on the Web, by using the properties:

identifier for formal identifier strings such as DOIs

url with a string URL corresponding to a download link (if not available, a download landing page) for this file

subjectOf to a CreativeWork (or WebPage) that mentions this file or its content (but also other resources)

mainEntityOfPage to a CreativeWork (or WebPage) that primarily describes this file (or its content)

7.3.2 Directories on the web; dataset distributions

A Directory File Entry or Dataset identifier expressed as an absolute URL on the web can be harder to download than a File because it consists of multiple resources. It is RECOMMENDED that such directories have a complete listing of their content in hasPart, enabling download traversal.

Alternatively, a common mechanism to provide downloads of a reasonably sized directory is as an archive file in formats such as .zip or .tar.gz, described as a DataDownload.

Similarly, the RO-Crate root entity may also provide a distribution URL, in which case the download SHOULD be an archive that contains the RO-Crate Metadata file.

In all cases, consumers should be aware that a DataDownload is a snapshot that may not reflect the current state of the Dataset or RO-Crate.

8 Representing Contextual Entities

The RO-Crate SHOULD contain additional information about Contextual Entities for the use of both humans (in ro-crate-preview.html) and machines (in ro-crate-metadata.json). This also helps to maximize the extent to which an RO-Crate is self-contained and self-describing, in that it reduces the need for the consumer of an RO-Crate to refer to external information which may change or become unavailable over time.

8.1 Contextual vs Data entities

RO-Crate distinguishes between Contextual entities and Data entities.

Data entities primarily exist in their own right as a file or directory (which may be in the RO-Crate Root directory or downloadable by URL).

Contextual entities however primarily exist outside the digital sphere (e.g. People, Places) or are conceptual descriptions that primarily exists as metadata, like GeoCoordinates and ContactPoint.

Some contextual entities can also be considered data entities - for instance the license property refers to a CreativeWork that can reasonably be downloaded, however a license document is not usually considered as part of research outputs and would therefore typically not be included in hasPart on the root data entity.

Likewise, some data entities may also be described as contextual entities, for instance a File that is also a ScholarlyArticle. In such cases the Contextual Data Entity MUST be described as a single JSON object in the RO-Crate Metadata JSON @graph and SHOULD list both relevant data and contextual types in a @type array.

The RO-Crate Metadata JSON @graph MUST NOT list multiple entities with the same @id; behaviour of consumers of an RO-Crate encountering multiple entities with the same @id is undefined.

8.2 Identifiers for contextual entities

A challenge can be how to assign identifiers for contextual entities, that is deciding on their @id value.

RO-Crate recommend that if an existing permalink (e.g. https://orcid.org/0000-0002-1825-0097) or other absolute URI (e.g. https://en.wikipedia.org/wiki/Josiah_S._Carberry) is reasonably unique for that entity, that URI should be used as identifier for the contextual entity in preference of an identifier local to the RO-Crate (e.g. #josiah or #0fa587c6-4580-4ece-a5df-69af3c5590e3).

Care should be taken to not describe two conceptually different contextual entities with the same identifier - e.g. if https://en.wikipedia.org/wiki/Josiah_S._Carberry is a Person it should not also be a CreativeWork (although this example is a fictional person!).

Where a related URL exist that may not be unique enough to serve as identifier, it can instead be added to a contextual entity using the property url.

See the appendix on JSON-LD identifiers for details.

8.3 People

A core principle of Linked data is to use URIs to identify important entities such as people. The following is the minimum recommended way of representing a author of a RO-Crate. The author property MAY also be applied to a directory (Dataset), a File or other CreativeWork entities.

This uses an ORCID to unambiguously identify an author, represented as a Contextual Entity of type Person.

Note the string value for the organizational affiliation. This SHOULD be improved by also providing a Contextual Entity for the organization (see example below).

8.4 Organizations as values

An Organization SHOULD be the value for the publisher property of a Dataset or ScholarlyArticle or affiliation property of a Person.

An Organization SHOULD also be used for a Person's affiliation property.

8.5 Contact information

A RO-Crate SHOULD have contact information, using a contextual entity of type ContactPoint. Note that in Schema.org Dataset does not currently have the corresponding contactPoint property, so the contact point would need to be given through a Person or Organization contextual entity which are related to the Dataset via a author or publisher property.

8.6 Publications via citation property

To associate a publication with a dataset the RO-Crate JSON-LD MUST include a URL (for example a DOI URL) as the @id of a publication using the citation property.

The publication SHOULD be described further as an additional contextual entity of type ScholarlyArticle or CreativeWork.

citation MAY also be used with other data and contextual entities.

A data entity MAY provide a published DOI identifier that, compared with any related publication in citation, primarily captures that file or dataset.

8.7 Publisher

The Root Data Entity SHOULD have a publisher property. This SHOULD be an Organization though it MAY be a Person.

8.8 Funding and grants

To associate a research project with a Dataset, the RO-Crate JSON-LD SHOULD contain an entity for the project using type Organization, referenced by a funder property. The project Organization SHOULD in turn reference any external funder, either by using its URL as an @id or via a Contextual Entity describing the funder.

Tip: To make it very clear where funding is coming from, the Root Data Entity SHOULD also reference funders directly, as well as via a chain of references.

8.9 Licensing, Access control and copyright

If a Data Entity has a license that is different from the license on the Root Data Entity, the entity SHOULD have a license property referencing a Contextual Entity with a type CreativeWork to describe the license. The @id of the license SHOULD be its URL (e.g. a Creative Commons License URL) and, when possible, a summary of the license included using the description property.

The below Data Entity has a copyrightHolder which is different from its author. There is a reference to an Organization describing the copyright holder and, to give credit, a sameAs relation to a web page. The license property here refers to https://creativecommons.org/licenses/by/4.0/ which is expanded in a separate contextual entity.

8.9.1 Metadata license

In some cases the license of the RO-Crate metadata (the JSON-LD statements in the RO-Crate Metadata File Descriptor) is different from the license on the Root Data Entity and its content (data entities indicated by hasPart).

For instance, a common pattern for repositories is to license metadata as CC0 Public Domain Dedication, while data is licensed as CC-BY or similar. This pattern allow metadata to be combined freely (e.g. the DataCite knowledge graph), while redistribution of data files would require explicit attribution and statement of their license.

To express the metadata license is different from the Root Data Entity, expand the RO-Crate Metadata File Descriptor to include license.

If no explicit license is expressed on the RO-Crate Metadata File Descriptor, the license expressed on the Root Data Entity apply also on the RO-Crate metadata.

9 Detailing provenance of entities

9.1 Equipment used to create files

To specify which equipment was used to create or update a Data Entity, the RO-Crate JSON-LD SHOULD have a Context Entity for each item of equipment which SHOULD be of @type IndividualProduct. The entity SHOULD have a serial number, manufacturer that identifies it as completely as possible. In this case the equipment is a bespoke machine. The equipment SHOULD be described on a web page, and the address of the description SHOULD be used as its @id.

Uses CreateAction and UpdateAction type to model the contributions of Context Entities of type Person or Organization in the creation of files.

In this example the CreateAction has a human agent, the object is a Place (a cave) and the Hovermap drone is the instrument used in the file creation event.

9.2 Software used to create files

To specify which software was used to create or update a file, the software application SHOULD be represented with an entity of type SoftwareApplication, with a version property, e.g. from tool --version.

The software SHOULD be associated with the File (or other data entities) it created as an instrument of a CreateAction, with the File referenced by a result property. Any input files SHOULD be referenced by the object property.

In the below example, an image with the @id of pics/2017-06-11%2012.56.14.jpg was transformed into an new image pics/sepia_fence.jpg using the ImageMagick software application as "instrument". Actions MAY have human-readable names, which MAY be machine generated for use at scale.

Tip: If representing command lines, double escape \\ so that JSON preserves the \ character.

If multiple SoftwareApplications have been used in composition, such as from a script or workflow, then the CreateAction's instrument SHOULD rather reference a SoftwareSourceCode which can be further described as explained in the Workflows and scripts section.

9.3 Recording changes to RO-Crates

To record an action which changes an entity's metadata, or changes its state in a publication or other workflow, a CreateAction or UpdateAction SHOULD be associated with a Data Entity or, for the RO-Crate itself, with the root data entity.

A curation Action MUST have at least one object which associates it with either the root data entity Dataset or one of its components.

An Action which creates new Data entities - for example, the creation of a new metadata file - SHOULD have these as results.

An Action SHOULD have a name and MAY have a description.

An Action SHOULD have an endTime, which MUST be in ISO 8601 date format and SHOULD be specified to at least the precision of a day. An Action MAY have a startTime meeting the same specifications.

An Action SHOULD have a human agent who was responsible for authorizing the action, and MAY have an instrument which associates the action with a particular piece of software (for example, the content management system or data catalogue through which an update was approved) which SHOULD be of @type SoftwareApplication.

An Action's status MAY be recorded in an actionStatus property. The status must be one of the values enumerated by ActionStatusType: ActiveActionStatus, CompletedActionStatus, FailedActionStatus or PotentialActionStatus.

An Action which has failed MAY record any error information in an error property.

UpdateAction SHOULD only be used for actions which affect the Dataset as a whole, such as movement through a workflow.

To record curation actions which modify a File within a Dataset - for example, by correcting or enhancing metadata - the old version of the File SHOULD be retained, and a CreateAction added which has the original version as its object and the new version as its result.

9.4 Digital Library and Repository content

To describe an export from a Digital Library or repository system, RO-Crate uses the Portland Common Data Model (PCDM).

A Contextual Entity from a repository, representing an abstract entity such as a person, or a work, or a place SHOULD have a @type of RepositoryObject, in addition to any other types.

Objects MAY be grouped together in RepositoryCollections with hasMember pointing to the RepositoryObject.

Note: The terms RepositoryObject and RepositoryCollection are renamed in RO-Crate to avoid collision between other vocabularies and the PCDM terms Collection and Object. The term RepositoryFile is renamed to avoid clash with RO-Crate's File mapping to http://schema.org/MediaObject.

Warning: PCDM specifies that files should have only technical metadata, not descriptive metadata, which is not a restriction in RO-Crate. If the RO-Crate is to be imported into a strict PCDM repository, modeling of object/file relationships will be necessary.

For example, this data is exported from an Omeka repository.

10 Workflows and Scripts

Scientific workflows and scripts that were used (or can be used) to analyze or generate files contained in an RO-Crate MAY be embedded in an RO-Crate. See also the Provenance section on Software Used to Create Files.

Workflows and scripts SHOULD be described using data entities of type SoftwareSourceCode.

The distinction between SoftwareSourceCode and SoftwareApplication for software is fluid, and comes down to availability and understandability. For instance, office spreadsheet applications are generally available and do not need further explanation (SoftwareApplication); while a Python script that is customized for a particular data analysis might be important to understand deeper and should therefore be included as SoftwareSourceCode in the RO-Crate dataset.

10.1 Describing scripts and workflows

A script is a Data Entity which MUST have the following properties:

@type is an array with at least File and SoftwareSourceCode as values

@id is a File URI linking to the executable script

A workflow is a Data Entity which MUST have the following properties:

@type is an array with at least File, SoftwareSourceCode and ComputationalWorkflow as values

@id is a File URI linking to the workflow entry-point.

There is no strong distinction between a script and a workflow; many computational workflows are written in script-like languages, and many scripts perform a pipeline of steps.

Here are some indicators for when a script should be considered a workflow:

It performs a series of steps (pipeline)

The executed steps are mainly external tools or services

The main work is performed by the steps (script is not algorithmic)

The steps exchange data in a dataflow, typically file inputs/outputs

The script has well-defined inputs and outputs, e.g. file arguments

Here are some counter-indicators for when a script might not be a workflow:

The script contains mainly algorithms or logic

Data is exchanged out of bands, e.g. a SQL database

The script relies on a particular state of the system (e.g. appends existing files)

An interactive user interface that controls the actions

10.2 Workflow Runtime and Programming Language

Scripts written in a programming language, as well as workflows, generally need a runtime; in RO-Crate the runtime SHOULD be indicated using a liberal interpretation of programmingLanguage.

Note that the language and its runtime MAY differ (e.g. different C++ compilers), but for scripts and workflows, frequently the language and runtime are essentially the same, and thus the programmingLanguage, implied to be a ComputerLanguage, can also be described as an executable SoftwareApplication:

A contextual entity representing a ComputerLanguage and/or SoftwareApplication MUST have a name, url and version, which should indicate a known version the workflow/script was developed or tested with. alternateName MAY be provided if there is a shorter colloquial name, for instance "R" instead of "The R Project for Statistical Computing".

It is possible to indicate steps that are executed as part of an ComputationalWorkflow or Script, by using hasPart to relate additional SoftwareApplication or nested SoftwareSourceCode contextual entities.

10.3 Workflow diagram/sketch

It can be beneficial to show a diagram or sketch to explain the script/workflow. This may have been generated from a workflow management system, or drawn manually as a diagram. This diagram MAY be included from the SoftwareSourceCode data entity by using image, pointing to an ImageObject data entity which is about the SoftwareSourceCode.

The image file format SHOULD be indicated with encodingFormat using an IANA registered media type like image/svg+xml or image/png. Additionally a reference to a Pronom identifier SHOULD be provided, which MAY be described as an additional contextual entity to give human-readable name to the format.

A workflow diagram may still be provided even if there is no programmatic SoftwareSourceCode that can be executed (e.g. because the workflow was done by hand). In this case the sketch itself is a proxy for the workflow and SHOULD have an about property referring to the RO-Crate dataset as a whole (assuming the RO-Crate represents the outcome of a single workflow), or to other Data Entities otherwise.

10.4 Complying with Bioschemas Computational Workflow profile

Data entities representing workflows (@type: ComputationalWorkflow) SHOULD comply with the Bioschemas ComputationalWorkflow profile, where possible.

When complying with this profile, the workflow data entities MUST describe these properties and their related contextual entities: name, programmingLanguage, creator, dateCreated, license, sdPublisher, url, version.

The ComputationalWorkflow profile explains the above and list additional properties that a compliant ComputationalWorkflow data entity SHOULD include: citation, contributor, creativeWorkStatus, description, funding, hasPart, isBasedOn, keywords, maintainer, producer, publisher, runtimePlatform, softwareRequirements, targetProduct

A data entity conforming to the ComputationalWorkflow profile SHOULD declare the versioned profile URI using conformsTo.

10.4.1 Describing inputs and outputs

The input and output parameters for a workflow or script can be given with input and output to FormalParameter contextual entities. Note that this entity usually represent a potential input/output value in a reusable workflow, much like function parameter definitions in general programming.

If complying with the Bioschemas FormalParameter profile, the contextual entities for FormalParameter, referenced by input or output, MUST describe: name, additionalType, encodingFormat

The Bioschemas FormalParameter profile explains the above and lists additional properties that can be used, including description, valueRequired, defaultValue and identifier.

A contextual entity conforming to the FormalParameter profile SHOULD declare the versioned profile URI using conformsTo.

Note: input, output and FormalParameter are at time of writing proposed by Bioschemas and not yet integrated in Schema.org

10.5 Complete Workflow Example

The below is an example of an RO-Crate complying with the Bioschemas ComputationalWorkflow profile 0.5:

0 notes

trylkstopocket · 2 years ago

Text

RO-crate spec (with code)