Run a test suite

Given wft4galaxy installed as native Python library (see Installation) and a well formed workflow test suite definition file (we use “examples/change_case/workflow-test.yml” as example; see Define a test suite from a Workflow and “Test Definition file reference” for more details) you can run your tests by simply typing:

In [2]:

wft4galaxy -f ../examples/change_case/workflow-test.yml

Workflow Test: 'change_case' ... ok

----------------------------------------------------------------------
Ran 1 test in 12.294s

OK

Notice that if the name of your test-definition-file is workflow-test-suite.yml and it is in your current working directory, you can ommit the option -f <...> and run the test above by simply typing wft4galaxy from your current path.

As you can see in the call [1] above, the default behaviour of the wft4galaxy tool is to run tests with a minimal output, showing the executed tests and which of them succeeded or failed. To display more information about the test execution you can enable logs, using one of the following options:

--enable-logger           Enable INFO level logs
--debug                   Show more details using DEBUG level logs

In [3]:

wft4galaxy -f ../examples/change_case/workflow-test.yml --enable-logger

2017-03-29 16:54:57,414 [wft4galaxy] [ INFO]  Configuration: <wft4galaxy.core.WorkflowTestSuite object at 0x10d4e01d0>
Workflow Test: 'change_case' ... 2017-03-29 16:54:58,434 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_ad44de7a-148f-11e7-a76d-a8667f33b7fc' (id: '84ffe6fca6c4fbda')
2017-03-29 16:54:59,798 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: f3a63b25ef201245) running ...
2017-03-29 16:55:04,081 [wft4galaxy] [ INFO]  waiting for datasets
2017-03-29 16:55:04,304 [wft4galaxy] [ INFO]  6e7233e069aad1a7: new
2017-03-29 16:55:06,861 [wft4galaxy] [ INFO]  6e7233e069aad1a7: queued
2017-03-29 16:55:07,608 [wft4galaxy] [ INFO]  6e7233e069aad1a7: ok
2017-03-29 16:55:08,113 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: f3a63b25ef201245) executed
2017-03-29 16:55:08,114 [wft4galaxy] [ INFO]  Checking test output: ...
2017-03-29 16:55:08,309 [wft4galaxy] [ INFO]  Checking test output: DONE
ok

----------------------------------------------------------------------
Ran 1 test in 11.289s

OK

Another useful option to deeply inspect the output of a test execution (especially when failed) is --disable-cleanup, which tells wft4galaxy to not clean all intermediate files produced during its execution. All of them (logs and workflow output datasets) are stored in the ./results output folder, but you can set your preferred target folder with the option -o <YOUR_FOLDER> (short form of --output <YOUR_FOLDER>).

In [7]:

wft4galaxy -f ../examples/change_case/workflow-test.yml --enable-logger -o change_case_results --disable-cleanup

2017-03-29 17:38:56,728 [wft4galaxy] [ INFO]  Configuration: <wft4galaxy.core.WorkflowTestSuite object at 0x105075c90>
Workflow Test: 'change_case' ... 2017-03-29 17:38:57,567 [wft4galaxy] [ INFO]  Create a history '_WorkflowTestHistory_d26878f8-1495-11e7-95a7-a8667f33b7fc' (id: u'6ee95371d25d237b')
2017-03-29 17:38:58,910 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: 7ca8f1b7f24e5a2d) running ...
2017-03-29 17:39:03,114 [wft4galaxy] [ INFO]  waiting for datasets
2017-03-29 17:39:03,501 [wft4galaxy] [ INFO]  c389501033f7c311: new
2017-03-29 17:39:04,420 [wft4galaxy] [ INFO]  c389501033f7c311: queued
2017-03-29 17:39:05,506 [wft4galaxy] [ INFO]  c389501033f7c311: queued
2017-03-29 17:39:06,229 [wft4galaxy] [ INFO]  c389501033f7c311: running
2017-03-29 17:39:06,864 [wft4galaxy] [ INFO]  c389501033f7c311: running
2017-03-29 17:39:07,806 [wft4galaxy] [ INFO]  c389501033f7c311: ok
2017-03-29 17:39:08,309 [wft4galaxy] [ INFO]  Workflow '_WorkflowTest_Change Case (imported from API)' (id: 7ca8f1b7f24e5a2d) executed
2017-03-29 17:39:08,309 [wft4galaxy] [ INFO]  Checking test output: ...
2017-03-29 17:39:08,442 [wft4galaxy] [ INFO]  Checking test output: DONE
ok

----------------------------------------------------------------------
Ran 1 test in 11.714s

OK

The resulting change_case_results folder will contain:

  1. test-suite level logs (i.e., a global log file for the test suite):
In [18]:

ls change_case_results/*.log

change_case_results/20170329@171841.log change_case_results/20170329@173856.log
  1. test-case level logs and output datasets (i.e., a log file specific for the test case and the datasets generated as output during the workflow execution):
In [13]:

ls change_case_results/change_case

20170329@173857.log     OutputText