path: root/docs/YamlIO.rst

=====================
YAML I/O
=====================

.. contents::
   :local:

Introduction to YAML
====================

YAML is a human readable data serialization language.  The full YAML language 
spec can be read at `yaml.org 
<http://www.yaml.org/spec/1.2/spec.html#Introduction>`_.  The simplest form of
yaml is just "scalars", "mappings", and "sequences".  A scalar is any number
or string.  The pound/hash symbol (#) begins a comment line.   A mapping is 
a set of key-value pairs where the key ends with a colon.  For example:

.. code-block:: yaml

     # a mapping
     name:      Tom
     hat-size:  7
     
A sequence is a list of items where each item starts with a leading dash ('-'). 
For example:

.. code-block:: yaml

     # a sequence
     - x86
     - x86_64
     - PowerPC

You can combine mappings and sequences by indenting.  For example a sequence
of mappings in which one of the mapping values is itself a sequence:

.. code-block:: yaml

     # a sequence of mappings with one key's value being a sequence
     - name:      Tom
       cpus:
        - x86
        - x86_64
     - name:      Bob
       cpus:
        - x86
     - name:      Dan
       cpus:
        - PowerPC
        - x86

Sometime sequences are known to be short and the one entry per line is too
verbose, so YAML offers an alternate syntax for sequences called a "Flow
Sequence" in which you put comma separated sequence elements into square 
brackets.  The above example could then be simplified to :


.. code-block:: yaml

     # a sequence of mappings with one key's value being a flow sequence
     - name:      Tom
       cpus:      [ x86, x86_64 ]
     - name:      Bob
       cpus:      [ x86 ]
     - name:      Dan
       cpus:      [ PowerPC, x86 ]


Introduction to YAML I/O
========================

The use of indenting makes the YAML easy for a human to read and understand,
but having a program read and write YAML involves a lot of tedious details.
The YAML I/O library structures and simplifies reading and writing YAML 
documents.

YAML I/O assumes you have some "native" data structures which you want to be
able to dump as YAML and recreate from YAML.  The first step is to try 
writing example YAML for your data structures. You may find after looking at 
possible YAML representations that a direct mapping of your data structures
to YAML is not very readable.  Often the fields are not in the order that
a human would find readable.  Or the same information is replicated in multiple
locations, making it hard for a human to write such YAML correctly.  

In relational database theory there is a design step called normalization in 
which you reorganize fields and tables.  The same considerations need to 
go into the design of your YAML encoding.  But, you may not want to change
your existing native data structures.  Therefore, when writing out YAML
there may be a normalization step, and when reading YAML there would be a
corresponding denormalization step.  

YAML I/O uses a non-invasive, traits based design.  YAML I/O defines some 
abstract base templates.  You specialize those templates on your data types.
For instance, if you have an enumerated type FooBar you could specialize 
ScalarEnumerationTraits on that type and define the enumeration() method:

.. code-block:: c++

    using llvm::yaml::ScalarEnumerationTraits;
    using llvm::yaml::IO;

    template <>
    struct ScalarEnumerationTraits<FooBar> {
      static void enumeration(IO &io, FooBar &value) {
      ...
      }
    };


As with all YAML I/O template specializations, the ScalarEnumerationTraits is used for 
both reading and writing YAML. That is, the mapping between in-memory enum
values and the YAML string representation is only in place.
This assures that the code for writing and parsing of YAML stays in sync.

To specify a YAML mappings, you define a specialization on 
llvm::yaml::MappingTraits.
If your native data structure happens to be a struct that is already normalized,
then the specialization is simple.  For example:

.. code-block:: c++
   
    using llvm::yaml::MappingTraits;
    using llvm::yaml::IO;
    
    template <>
    struct MappingTraits<Person> {
      static void mapping(IO &io, Person &info) {
        io.mapRequired("name",         info.name);
        io.mapOptional("hat-size",     info.hatSize);
      }
    };


A YAML sequence is automatically inferred if you data type has begin()/end()
iterators and a push_back() method.  Therefore any of the STL containers
(such as std::vector<>) will automatically translate to YAML sequences.

Once you have defined specializations for your data types, you can 
programmatically use YAML I/O to write a YAML document:

.. code-block:: c++
   
    using llvm::yaml::Output;

    Person tom;
    tom.name = "Tom";
    tom.hatSize = 8;
    Person dan;
    dan.name = "Dan";
    dan.hatSize = 7;
    std::vector<Person> persons;
    persons.push_back(tom);
    persons.push_back(dan);
    
    Output yout(llvm::outs());
    yout << persons;
   
This would write the following:

.. code-block:: yaml

     - name:      Tom
       hat-size:  8
     - name:      Dan
       hat-size:  7

And you can also read such YAML documents with the following code:

.. code-block:: c++

    using llvm::yaml::Input;

    typedef std::vector<Person> PersonList;
    std::vector<PersonList> docs;
    
    Input yin(document.getBuffer());
    yin >> docs;
    
    if ( yin.error() )
      return;
    
    // Process read document
    for ( PersonList &pl : docs ) {
      for ( Person &person : pl ) {
        cout << "name=" << person.name;
      }
    }
  
One other feature of YAML is the ability to define multiple documents in a 
single file.  That is why reading YAML produces a vector of your document type.



Error Handling
==============

When parsing a YAML document, if the input does not match your schema (as 
expressed in your XxxTraits<> specializations).  YAML I/O 
will print out an error message and your Input object's error() method will 
return true. For instance the following document:

.. code-block:: yaml

     - name:      Tom
       shoe-size: 12
     - name:      Dan
       hat-size:  7

Has a key (shoe-size) that is not defined in the schema.  YAML I/O will 
automatically generate this error:

.. code-block:: yaml

    YAML:2:2: error: unknown key 'shoe-size'
      shoe-size:       12
      ^~~~~~~~~

Similar errors are produced for other input not conforming to the schema.


Scalars
=======

YAML scalars are just strings (i.e. not a sequence or mapping).  The YAML I/O
library provides support for translating between YAML scalars and specific
C++ types.


Built-in types
--------------
The following types have built-in support in YAML I/O:

* bool
* float
* double
* StringRef
* int64_t
* int32_t
* int16_t
* int8_t
* uint64_t
* uint32_t
* uint16_t
* uint8_t

That is, you can use those types in fields of MappingTraits or as element type
in sequence.  When reading, YAML I/O will validate that the string found
is convertible to that type and error out if not.


Unique types
------------
Given that YAML I/O is trait based, the selection of how to convert your data
to YAML is based on the type of your data.  But in C++ type matching, typedefs
do not generate unique type names.  That means if you have two typedefs of
unsigned int, to YAML I/O both types look exactly like unsigned int.  To
facilitate make unique type names, YAML I/O provides a macro which is used
like a typedef on built-in types, but expands to create a class with conversion
operators to and from the base type.  For example:

.. code-block:: c++

    LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyFooFlags)
    LLVM_YAML_STRONG_TYPEDEF(uint32_t, MyBarFlags)

This generates two classes MyFooFlags and MyBarFlags which you can use in your
native data structures instead of uint32_t. They are implicitly 
converted to and from uint32_t.  The point of creating these unique types
is that you can now specify traits on them to get different YAML conversions.

Hex types
---------
An example use of a unique type is that YAML I/O provides fixed sized unsigned
integers that are written with YAML I/O as hexadecimal instead of the decimal
format used by the built-in integer types:

* Hex64
* Hex32
* Hex16
* Hex8

You can use llvm::yaml::Hex32 instead of uint32_t and the only different will
be that when YAML I/O writes out that type it will be formatted in hexadecimal.


ScalarEnumerationTraits
-----------------------
YAML I/O supports translating between in-memory enumerations and a set of string
values in YAML documents. This is done by specializing ScalarEnumerationTraits<>
on your enumeration type and define a enumeration() method. 
For instance, suppose you had an enumeration of CPUs and a struct with it as 
a field:

.. code-block:: c++

    enum CPUs {
      cpu_x86_64  = 5,
      cpu_x86     = 7,
      cpu_PowerPC = 8
    };
    
    struct Info {
      CPUs      cpu;
      uint32_t  flags;
    };
    
To support reading and writing of this enumeration, you can define a 
ScalarEnumerationTraits specialization on CPUs, which can then be used 
as a field type: 

.. code-block:: c++

    using llvm::yaml::ScalarEnumerationTraits;
    using llvm::yaml::MappingTraits;
    using llvm::yaml::IO;

    template <>
    struct ScalarEnumerationTraits<CPUs> {
      static void enumeration(IO &io, CPUs &value) {
        io.enumCase(value, "x86_64",  cpu_x86_64);
        io.enumCase(value, "x86"