Link Search Menu Expand Document
Open Trusted Data Initiative
AI Alliance Banner
Join Our Initiative   Browse the Datasets   Contribute a New Dataset

Dataset Specification

Note: The specification documented here is the “V0.1” version of what we think will be required for cataloged datasets. We need and welcome your feedback! Either contact us or consider using pull requests with your suggestions. See the AI Alliance community page on contributing for more details.

Also contact us if you are interested in contributing a dataset, but you have any questions or concerns about meeting the following specification.

Table of contents
  1. Dataset Specification
    1. About This Specification
    2. The Data Must Be Yours to Contribute
    3. Dataset Hosting
    4. Dataset Card
    5. Quick Steps
      1. Details
    6. Required Metadata
      1. YAML Metadata Block
    7. The Markdown Content in the Dataset Card
    8. Other Considerations for the Data Itself
      1. Formats
      2. Diverse Datasets
        1. Science and Industrial
        2. Other Domains
        3. Modalities
    9. Derived Dataset Specification
      1. Categories of Dataset Transformations

About This Specification

The specification attempts to be minimally sufficient, to impose just enough constraints to meet our goals for cataloged datasets.

The specification is adapted from the Hugging Face Dataset Card, with a few extensions for clearer provenance and governance.

In addition, we are exploring incorporation of the following sources:

Most of the details are captured in the dataset card that every version of a dataset carries (e.g., after various stages of processing).

Let’s begin.

The Data Must Be Yours to Contribute

To promote fully-traceable provenance and governance, for all data within the dataset, you must affirm that you are either (a) the owner of the dataset or (b) you have rights from the owner of the data that enables you to provide it to anyone under the CDLA Permissive 2.0 license; for example, you have been granted permission by the owner to act on their behalf with respect to the data and enable others to use it without restriction.

WARNING: Do not contribute any data that was obtained by crawling or scraping public data from the Internet or other public places. At this time, we are not accepting such data because we are seeking to build datasets with a heightened level of clarity around ownership, provenance, and quality.

Dataset Hosting

You can either retain your current hosting location or you can have the AI Alliance host it for you.

Dataset Card

All useful datasets include metadata about their provenance, license(s), target uses, known limitations and risks, etc. To provide a uniform, standardized way of expressing this metadata, we ask you to provide a dataset card (or data card) when you contribute the dataset.

Since your dataset is already likely to be available on the Hugging Face Hub, we ask you create a “complete” Hugging Face Dataset Card with the metadata fields they support. The project README.md file functions as the card. We provide a list below of the fields we consider necessary to be sufficient for our purposes, including a few additional items we need to know that you should add to the README.md you create.

TIP: For a general introduction to Hugging Face datasets, see here.

Quick Steps

Here are the steps to create your dataset card, summarized. Read the rest of this page for details:

  1. Download our version of the Hugging Face dataset card template, datasetcard_otdi_template.md. (If you already have a card in Hugging Face, i.e., the README.md, compare our template to your card and add the new fields.)
  2. Edit the Markdown in the template file to provide the details, as described below.
  3. Create the card in the Hugging Face UI (or edit your existing card.)
  4. Fill in the metadata fields shown in their editor UI. (See Table 1 below.)
  5. Paste the rest of your prepared Markdown into the file, after the YAML block delimited by ---.
  6. Commit your changes!

Details

Refer to the datasetcard.md for details about the metadata fields Hugging Face (and we!) recommend for inclusion in a YAML block at the top of the README.md. We comment on these fields below, see Table 1.

The templates/README_guide.md provides additional information about the template fields in their Markdown template file, datasetcard_template.md in the huggingface-hub GitHub repo. However, we recommend that you use our extended version: datasetcard_otdi_template.md. (You might need to right click on the link…)

If you want to contribute a dataset that isn’t currently hosted in Hugging Face, use the template above to create a dataset card yourself. Manually add the YAML header block, too. Finally, follow the Hugging Face convention of using the README.md in the top-level directory as the dataset card.

Required Metadata

This section describes the minimum set of metadata we expect, including some optional elements of the Hugging Face dataset card that believe are essential.

All of the fields apply to synthesized data as well as real data, but of course details will be different.

YAML Metadata Block

TIP: The following tables are long, but starting with the datasetcard_template.md and the dataset card process will handle most of the details. Then you can add the additional fields requested in Table 2, those marked with “OTDI”.

Our first table describes the metadata encoded in the YAML header block at the beginning of the Hugging Face README format. See datasetcard.md for details.

For completeness, the optional fields in that block are also shown. The Required? column uses ☑ to indicate the field is required, empty for optional fields (but often recommended), and ☒ for fields that we don’t allow, because they are incompatible with this project.

Field Name Description Required?
license We strongly recommend cdla-permissive-2.0 for the Community Data License Agreement – Permissive, Version 2.0 and may require it in the future 1. Use these names for licenses. Also covers the OTDI License to use. ☑
license_name e.g, Community Data License Agreement – Permissive, Version 2.0. ☑
license_link e.g, LICENSE or LICENSE.md in the same repo or a URL to another location. ☑
license_details Not needed if you use a standard license.  
tags Useful for indicating target areas for searches, like chemistry, synthetic, etc. See also task_categories. Where applicable, we recommend that you use the categories described below in Diverse Datasets….  
annotations_creators If appropriate. Examples: crowdsourced, found, expert-generated, machine-generated (e.g., using LLMs as judges).  
language_creators If appropriate. Examples: crowdsourced, found, expert-generated, machine-generated (i.e., synthetic data).  
language_details One or more of, for example, en-US, fr-FR, etc. ☑
pretty_name E.g., Common Chemistry. This is equivalent to the Dataset title/name field in the Data Provenance Standard (OTDI). ☑
size_categories E.g., n<1K, 100K<n<1M.  
source_datasets A YAML list; zero or more. Recall our emphasis on provenance. This list is very important, as each source must meet our provenance standards. See also the discussions below. ☑
task_categories A YAML list; one or more from the list in this code. ☑
task_ids A YAML list; “a unique identifier in the format lbpp/{idx}, consistent with HumanEval and MBPP” from here. See also examples here.  
paperswithcode_id Dataset id on PapersWithCode (from the URL).  
configs Can be used to pass additional parameters to the dataset loader, such as data_files, data_dir, and any builder-specific parameters.  
config_name One or more dataset subsets, if applicable. See the example in datasetcard.md and the discussions here and here.  
dataset_info Can be used to store the feature types and sizes of the dataset to be used in Python. See the discussion in datasetcard.md. Also covers the OTDI Data format field.  
extra_gated_fields Used for protected datasets and hence incompatible with the goals of OTDI. ☒
train-eval-index Add this if you want to encode a train and evaluation info in a structured way for AutoTrain or Evaluation on the Hub. See the discussion in datasetcard.md.  

Table 1: Hugging Face Datacard Metadata

The Markdown Content in the Dataset Card

Our second table lists content that we require or recommend in the Markdown body of the dataset card, below the YAML header block. The Source column in the table contains the following:

  • “HF” for fields in the Hugging Face datasetcard_template.md. See the README_guide.md for descriptions of many of these fields.
  • “OTDI” for additional fields we believe are necessary.

Field Name Description Required? Source
standards_version_used A schema version. Standard schemas are not currently specified and TBD.   OTDI
unique_metadata_identifier A UUID that is globally unique. Derived datasets must have their own UUIDs. The UUID is very useful for unambiguous lineage tracking, which is why we require it. ☑ OTDI
dataset_summary A concise summary of the dataset and its purpose. ☑ HF
dataset_description Describe the contents, scope, and purpose of the dataset, which helps users understand what the data represents, how it was collected, and any limitations or recommended uses. However, this field should not include redundant information covered elsewhere. ☑ HF
curated_by One or more legal entities responsible for creating the dataset, providing accountability and a point of contact for inquiries. See also dataset_card_authors below. ☑ HF
dataset_sources HF template section (from datasetcard_template.md). Complements the information provided above for source_datasets. The Repository URL “subfield” is required for each source dataset, unless it was provided by source_datasets in Table 1. The Paper and Demo subfields are optional. See also source_data and source_metadata_for_dataset next. ☑ HF
source_data HF template section. Use the subsections, described next, data_collection_and_processing_section and source_data_producers_section to describe important provenance information. Is the data synthetic or not? Note our specification above that you can only submit datasets where you have the necessary rights (see also consent_documentation_location below). ☑ HF
data_collection_and_processing_section HF template section. Describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. ☑ HF
source_data_producers_section HF template section. This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. ☑ HF
source_metadata_for_dataset Additional content for source_data; if the corresponding metadata for any dataset is not part of that dataset, then it must be explicitly linked here. This information is necessary for lineage tracking, part of our provenance objectives. Marked required, but if all metadata is part of all datasets (e.g., in README.md dataset cards), then this field can be omitted. ☑ OTDI
consent_documentation_location “Specifies where consent documentation or agreements related to the data can be found, which help enable legal compliance and regulatory use.” Required for third-party datasets you are contributing. ☑ OTDI
data_origin_geography “The geographical location where the data was originally collected, which can be important for compliance with regional laws and understanding the data’s context.” Required if restrictions apply.   OTDI
data_processing_geography_inclusion_exclusion “Defines the geographical boundaries within which the data can or cannot be processed, often for legal or regulatory reasons.” Required if restrictions apply.   OTDI
data_storage_geography_inclusion_exclusion “Specifies where the data is stored and any geographical restrictions on storage locations, crucial for compliance with data sovereignty laws.” Required if restrictions apply.   OTDI
uses See the HF template section. Optional, but useful for describing Direct Use (field name: direct_use) and Out-of-Scope Use (field name: out_of_scope_use) for the dataset. Consider structuring the Direct Use as described in the Supported Tasks and Leaderboards section in the templates/README_guide.md.   HF
annotations HF template section. Add any additional information for the annotations_creators above, if any. Subsections are annotation_process_section and who_are_annotators_section.   HF
annotation_process_section HF template section. Describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, inter-annotator statistics, annotation validation, etc.   HF
who_are_annotators_section HF template section. Describes the people or systems who created the annotations.   HF
bias_risks_limitations HF template section. While provenance and governance are the top priorities for OTDI, we also want to communicate to potential users what risks they need to understand about our cataloged datasets. Therefore, we require any information you can provide in this section, along with the Recommendations subsection for mitigations, if known. ☑ HF
personal_and_sensitive_information State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). Consider using one or more of the values listed below, after this table. If efforts were made to anonymize the data, describe the anonymization process and also fill in use_of_privacy_enhancing_technologies_pets. ☑ HF, OTDI
use_of_privacy_enhancing_technologies_pets “Indicates whether techniques were used to protect personally identifiable information (PII) or sensitive personal information (SPI), highlighting the dataset’s privacy considerations.” ☑ OTDI
citation HF template section. A place to add BibTeX (field name: citation_bibtex) and APA (field name: citation_apa) citations.   HF
glossary HF template section. Define useful terms.   HF
dataset_card_authors HF template section. We need to know the authors. ☑ HF
dataset_card_contact HF template section. We need to know whom to contact when needed. Okay to leave blank if the authors’ contact information is provided.   HF
dataset_issue_date When the dataset was compiled or created. (New versions require new dataset cards.) Recommended format: YYYY-mm-dd:THH:MM:SS ☑ OTDI
date_previously_issued_version_dataset Timestamp for previous releases, if applicable. Redundant with other traceability tools, so could be omitted.   OTDI
range_dates_data_generation The span of time during which the data within the dataset was collected or generated, offering insight into the dataset’s timeliness and relevance. ☑ OTDI

Table 2: Additional Content for the Dataset Card (`README.md`)

For the personal_and_sensitive_information field, consider using one or more of the following values:

  • Personal Information (PI)/Demographic
  • Payment Card Industry (PCI)
  • Personal Financial Information (PFI)
  • Personally Identifiable Information (PII)
  • Personal Health Information (PHI)
  • Sensitive Personal Information (SPI)
  • Other (please specify)
  • None

Other Considerations for the Data Itself

The dataset card template has sections for all the required and optional information. Here we discuss a few points.

Formats

We endeavor to be flexible on dataset file formats and how they are organized. For text, we recommend formats like CSV, JSON, Parquet, ORC, AVRO. Supporting PDFs, where extraction will be necessary, can be difficult, but not impossible.

NOTE: Using Parquet has the benefit that MLCommons Croissant can be used to automatically extract some metadata. See this Hugging Face page and the mlcroissant library, which supports loading a dataset using the Croissant metadata.

Diverse Datasets

Diverse datasets are desired for creating a variety of AI models and applications with special capabilities.

We are particularly interested in new datasets that can be used to train and tune models to excel in particular domains, although general-purpose datasets are also welcome, of course.

These are the current domains of particular interest. Use the tags metadata field discussed above to indicate domains, when applicable.

Science and Industrial

  • Climate: Supporting research in climate change, modeling vegetation and water cover, studying agriculture, etc.
  • Marine: Supporting research on and applications targeted towards marine environments.
  • Materials: Known chemical and mechanical properties of chemicals useful for research into potential new and existing materials.
  • Semiconductors: Specific area of materials research focused on improving the state of the art for semiconductor performance and manufacturing.

Other science and industrial domains are welcome, too.

Other Domains

  • Finance: Historical market activity and behaviors. Connections to influences like climate, weather events, political events, etc.
  • Healthcare: Everything from synthetic patient data for modeling outcomes, to public literature on known diseases and conditions, to diagnostics results and their analysis.
  • Legal: Jurisdiction-specific data about case law, etc. specific applications.
  • Social Sciences: Social dynamics, political activity and sentiments, etc.
  • Timeseries: Data for training, tuning, and testing time series models, including specific applications.

Modalities

In addition, we welcome datasets with different modalities. Hugging Face attempts to determine the modalities of datasets, but you can also use the tags to indicate modalities, such as the following:

  • Text:
  • Image: i.e., still images
  • Audio:
  • Video: including optional audio

Derived Dataset Specification

Every dataset that is derived via a processing pipeline from one or more other datasets requires its own dataset card, which must reference all upstream datasets that feed into it (and by extension, their dataset cards of metadata). Similarly, each new version of an existing dataset, whre only additional (or removed) data is involved, also needs an updated card, but more of the metadata will be unchanged.

Note: We are considering a way to allow a derived dataset card to just specify what’s new or changed and inherit unchanged metadata from its ancestors. Also, when automated pipelines are used to create derived datasets, our processing pipelines will automatically generate some updated metadata, such as timestamps, processing tools and steps, etc.

When the derived dataset is the filtered output of one or more raw datasets (defined below), where duplication and offensive content removal was performed, the new dataset may support different uses, have different bias_risks_limitations, and it will need to identify the upstream (ancestor) source_datasets, for example.

Table 3 lists the fields that must change (with some exceptions), to avoid ambiguities:

Field Name Possible Updates Required?
pretty_name A modified name is strongly recommended to avoid potential confusion. It might just embed a version string.  
unique_metadata_identifer Must be new! ☑
dataset_issue_date The date for this new card. ☑

Table 4: Required Dataset Card Changes for a Derived Dataset

Categories of Dataset Transformations

At this time, we have the following concepts for original and derived datasets, concerning levels of quality and cleanliness. This list corresponds to stages in our ingestion process and subsequent possible derivations of datasets. This list is subject to change.

  • Raw: The dataset as submitted, which could already be in “good shape”. Our most important concern at this stage is unambiguous provenance. Raw datasets may go through filtering and analysis to remove potential objectionable content. However, the presence of some content in the raw data could have legal implications, such as some forms of PII and company confidential information, which may force us to reject the contribution. (Should this happen, we will discuss mitigation options with you.)
  • Filtered: A raw dataset that has gone through a processing pipeline to remove duplicates, filter for objectional content, etc.
  • Structured: A filtered dataset that has been reformatted to be most suitable for model training (LLMs, time series, etc.), RAG patterns, and similar purposes. For example, PDFs converted to JSON.

See How We Process Datasets for more details on these levels and how we process datasets.

After you have prepared or updated the dataset card as required, it’s time to contribute your dataset!

  1. For source code, e.g., the code used for the data processing pipelines, the AI Alliance standard code license is Apache 2.0. For documentation, it is The Creative Commons License, Version 4.0, CC BY 4.0. See the Alliance community/CONTRIBUTING page for more details about licenses. ↩

Child Pages