Define a test suite

See the folder example:

EXAMPLES_FOLDER = "../examples"

… where you can find the following files:

import os, pprint
[f for f in os.listdir(EXAMPLES_FOLDER) if not f.startswith('.')]

['change_case',
 'fluxomics_stationary',
 'multivariate',
 'sacurine',
 'workflow-test-suite-full.yml',
 'workflow-test-suite-min.yml',
 'workflow-test-suite.yml',
 'workflows.json']

Consider the definition file workflow-test-suite-min.yml (steps[3-4]), which contains the two workflow tests named change_case and multivariate respectively:

suite_conf_filename = os.path.join(EXAMPLES_FOLDER, "workflow-test-suite-min.yml")

import yaml, json
with open(suite_conf_filename, "r") as fp:
    data = yaml.load(fp)
    print(json.dumps(data, indent=4))

{
    "enable_logger": false,
    "workflows": {
        "change_case": {
            "expected": {
                "OutputText": "change_case/expected_output"
            },
            "inputs": {
                "InputText": "change_case/input"
            },
            "file": "change_case/workflow.ga"
        },
        "multivariate": {
            "expected": {
                "variableMetadata_out": "multivariate/variableMetadata_out",
                "sampleMetadata_out": "multivariate/sampleMetadata_out"
            },
            "inputs": {
                "DataMatrix": "multivariate/dataMatrix.tsv",
                "SampleMetadata": "multivariate/sampleMetadata.tsv",
                "VariableMetadata": "multivariate/variableMetadata.tsv"
            },
            "params": {
                "3": {
                    "predI": "1",
                    "respC": "gender",
                    "orthoI": "NA",
                    "testL": "FALSE"
                }
            },
            "file": "multivariate/workflow.ga"
        }
    }
}

Suppose that you have two WorkflowTestCase instances (we load them in steps[5-6] from a definition file, but you can define them programmatically as described in “Define a single test case”):

from wft4galaxy.core import WorkflowTestCase

wft1 = WorkflowTestCase.load(suite_conf_filename, "change_case")
wft2 = WorkflowTestCase.load(suite_conf_filename, "multivariate")

To programmatically create test suite you need to create an instance of the WorkflowTestSuite class:

from wft4galaxy.core import WorkflowTestSuite
suite = WorkflowTestSuite()

and register the WorkflowTestSuite instances to the suite instance:

suite.add_workflow_test(wft1)
suite.add_workflow_test(wft2)

You can now run your test suite and inspect the results of its execution (steps [9-10]), as described in “Run a test suite”:

test_results = suite.run(enable_logger=True)

Workflow Test: 'change_case' ... 2017-03-30 15:08:02,477 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_e79f068a-1549-11e7-823b-a45e60c4fc6b' (id: u'c8a1d1385ed48f8d')
2017-03-30 15:08:03,756 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: c8a1d1385ed48f8d) running ...
2017-03-30 15:08:06,367 [wft4galaxy] [ INFO]  waiting for datasets
2017-03-30 15:08:06,559 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:07,502 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:08,239 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:08,900 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:09,830 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:10,476 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: new
2017-03-30 15:08:11,386 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: queued
2017-03-30 15:08:12,103 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running
2017-03-30 15:08:12,807 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running
2017-03-30 15:08:13,638 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running
2017-03-30 15:08:14,357 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: running
2017-03-30 15:08:15,365 [wft4galaxy] [ INFO]  53fb5c5113d9beb7: ok
2017-03-30 15:08:15,872 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: c8a1d1385ed48f8d) executed
2017-03-30 15:08:15,875 [wft4galaxy] [ INFO]  Checking test output: ...
2017-03-30 15:08:16,054 [wft4galaxy] [ INFO]  Checking test output: DONE
ok
Workflow Test: 'multivariate' ... 2017-03-30 15:08:17,738 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_f08acdb0-1549-11e7-bf2e-a45e60c4fc6b' (id: u'2d190d7d262b60f4')
2017-03-30 15:08:30,657 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Multivariate (imported from API)' (id: 2d190d7d262b60f4) running ...
2017-03-30 15:08:39,075 [wft4galaxy] [ INFO]  waiting for datasets
2017-03-30 15:08:39,483 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued
2017-03-30 15:08:39,970 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued
2017-03-30 15:08:40,455 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued
2017-03-30 15:08:40,781 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued
2017-03-30 15:08:41,453 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued
2017-03-30 15:08:41,952 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued
2017-03-30 15:08:42,266 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued
2017-03-30 15:08:42,443 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued
2017-03-30 15:08:43,290 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued
2017-03-30 15:08:43,526 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued
2017-03-30 15:08:43,788 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued
2017-03-30 15:08:44,030 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued
2017-03-30 15:08:44,678 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued
2017-03-30 15:08:45,228 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued
2017-03-30 15:08:45,440 [wft4galaxy] [ INFO]  77efbec71a7e1a47: queued
2017-03-30 15:08:45,629 [wft4galaxy] [ INFO]  ea3d97b57366b47f: queued
2017-03-30 15:08:46,418 [wft4galaxy] [ INFO]  b425b2361a02d64e: queued
2017-03-30 15:08:46,848 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: queued
2017-03-30 15:08:47,146 [wft4galaxy] [ INFO]  77efbec71a7e1a47: ok
2017-03-30 15:08:47,662 [wft4galaxy] [ INFO]  ea3d97b57366b47f: ok
2017-03-30 15:08:48,312 [wft4galaxy] [ INFO]  b425b2361a02d64e: ok
2017-03-30 15:08:48,485 [wft4galaxy] [ INFO]  eff20c5ba6bf6f87: ok
2017-03-30 15:08:48,986 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Multivariate (imported from API)' (id: 2d190d7d262b60f4) executed
2017-03-30 15:08:48,989 [wft4galaxy] [ INFO]  Checking test output: ...
2017-03-30 15:08:49,187 [wft4galaxy] [ INFO]  Checking test output: DONE
ok

----------------------------------------------------------------------
Ran 2 tests in 48.039s

OK

for r in test_results:
    print("Test %s:\n\t - workflow: [%s] \n\t - results: %r" % (r.test_id, r.workflow.name, r.results))

Test e79f068a-1549-11e7-823b-a45e60c4fc6b:
         - workflow: [_WorkflowTest_Change Case (imported from API)]
         - results: {u'OutputText': True}
Test f08acdb0-1549-11e7-bf2e-a45e60c4fc6b:
         - workflow: [_WorkflowTest_Multivariate (imported from API)]
         - results: {u'variableMetadata_out': True, u'sampleMetadata_out': True}