# Guide to CPAN::Meta::YAML testing

CPAN::Meta::YAML tests use several components:

* .t files
* Test libraries in t/lib
* YAML data files in t/data
* TestML data files in t/tml-*

The use of each is described below.

## .t files

The .t files are standard Perl test files.  They may use one or more of the
test libraries in t/lib.  They may only use modules available in Perl 5.8.1 or
later (and not subsequently deprecated), but they may use newer non-XS versions
of those modules as necessary to avoid known bugs.

Some .t files have complete inputs/outputs for their tests.  Others iterate
over .tml files in the t/tml-* directories.

A .t file should load Test::More and either use `done_testing` or provide a
test plan.  If tests iterate over external data, the use of `done_testing` is
preferred so that external data can be updated with new tests without needing
to also update a test plan.

Currently, the convention is to name .t files matching the pattern
qr/^\d\d_\w+\.t$/

## Test libraries

There are currently three test libraries in t/lib.  A .t file that uses one or
more of them should put `use lib 't/lib';` at the top of the .t file.  Test
libraries can assume that if they were loaded, that 't/lib' is already in @INC.

The test libraries are:

* TestML::Tiny
* TestBridge
* TestUtils

The TestML::Tiny library contains functions for parsing and executing TestML
tests with callbacks.  TestML is a data-driven testing language; TestML::Tiny
implements a small subset of its features. See the section on TestML, below,
for an example. Generally, bugs should be patched upstream on CPAN and then
a new Test::Tiny CPAN release can be copied here and pod-stripped.

The TestBridge library contains testing functions for use in .t files or to
be passed to TestML::Tiny functions as callbacks.  Test functions should not
include `done_testing`.  They should use `subtest` for any repetitive testing
that loops over test cases.  Callback should check for the expected test
points (see below) and skip a TML block if those points are not available.

The TestUtils library contains utility functions.  Testing functions should
not be added here (i.e. nothing that uses Test::More).

## YAML data files in t/data

Files in the t/data directory are intended to test how YAML files are loaded
and decoded and typically need some custom test code to load the file and see
if the result matches expectations (successful or not).

If a real-world YAML file cannot be loaded due to character set encoding
issues, it should be placed in this directory for testing.  If a real-world
YAML file is ASCII or UTF-8 encoded, can be decoded successfully, but has
problems in parsing, it should be reduced to the smallest sample of YAML that
demonstrates the parsing problem and added to a .tml file for testing.  See
below for more details.

## TestML quick intro

TestML data files are UTF-8 encoded files with a .tml suffix that contain one
or more test "blocks".  Each block has a test label, and one or more 'test
points', usually representing input and expected output, and possibly
additional annotations or flags.

Here is an example of a .tml file with a single block:

    # This is a TestML block:    (this line is a comment)
    === This is the test label
    Lines until the first point are ignored

    # This is a "block" style point. All non-comment lines until next point
    # are the data for the 'yaml' point. The data ends with newline, and
    # trailing blank lines are trimmed.
    --- yaml
    ---
    foo: bar
    # a comment
    \# not a comment

    # This is the second point; "inline" style. The data after the colon goes
    # to end of line. Leading/trailing whitespace is trimmed.
    --- perl: [ { foo => 'bar' } ]

    # This is a point whose value is the empty string
    --- a_flag

    # This is the next block:
    === Another test case

The test label is provided on a line beginning with qr/^===/.  Test "points"
are provided in sections beginning with qr/^--- +\w+/.  All flush-left comment
lines are stripped.  Lines beginning with '\' are escaped.

Different tests expect different test points in a .tml file, based on the
specific test callback being used.

Many .tml files have the points 'yaml' and 'perl' as in the example above.  The
'yaml' point is a YAML document and the 'perl' point is a Data::Dumper Perl
data structure.  The test checks whether the YAML parses into a data structure
identical to the Perl one.  The 'a_flag' point is an annotation that the
testing callback can use to affect the run of a given test.

The semantics of points (including annotations) is specific to the callback
functions used to process test blocks.

# TestML data files in t/tml-*

TestML data files are organized into three directories:

* t/tml-spec — these test files are provided by the YAML spec maintainers and
should not be modified except to skip testing features that CPAN::Meta::YAML does not
support

* t/tml-local — these test files are CPAN::Meta::YAML's own unit tests; generally new
test cases for coverage or correctness should be added here; these are
broken into subdirectories, described later

* t/tml-world — these test files represent "real world" YAML and their
corresponding expected Perl output

Generally, if a "real world" problem can be isolated to a particular snippet of
YAML, it's best to add it to a t/tml-local file (or create a new one).  If the
problem can only be seen in the context of the entire YAML document, include it
in t/tml-world.  If the problem relates to encoding, it should be put into
t/data instead.

# t/tml-local subdirectories

The subdirectories in t/tml-local define four types of tests:

* perl-to-yaml: test that perl data dump to an expected YAML string

* yaml-roundtrip: test that a YAML string loads to an expected perl data
  structure; also tests that the perl data can be dumped and loaded back;

* dump-error: test that certain perl data trigger expected errors

* load-error: test that certain YAML strings trigger expected errors

All .tml files in a t/tml-local directory must have the TestML
test points required by the corresponding test functions defined
in the TestBridge library.

Generally, files should be grouped by data type or feature so that
related tests are kept together.
