Runner exits with failures while browser tests report none

[ncoden]

Bug

For unknown reasons, browserstack-runner sometimes exits with failures while all tests run without errors.

For exemple see the following Travis job with mocha: all tests passed (or are skipped) but browserstack-runner exists with failures: 3.

Using config: /home/travis/build/ncoden/foundation-sites/browserstack.json
Launching 18 worker(s) for 18 run(s).
[OS X Sierra, Safari 10.1] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 6546ms
[Windows 10, Internet Explorer 11.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 14034ms
[Windows 8, Chrome 65.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 6260ms
[Windows 10, Chrome 66.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 7288ms
[Windows 7, Internet Explorer 10.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 15411ms
[Windows 10, Internet Explorer 11.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 16303ms
[Windows 8.1, Firefox 60.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 10193ms
[Windows 7, Opera 53.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 5517ms
[ios 7.0, Mobile Safari null] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 18048ms
[Windows 10, Edge 17.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 9909ms
[Windows 7, Opera 52.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 5855ms
[OS X High Sierra, Safari 11.1] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 9277ms
[Windows 10, Firefox 61.0 beta] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 10925ms
[Windows 10, Edge 16.0] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 11188ms
[android 8.0, Android Browser null] Passed: 260 tests, 252 passed, 0 failed, 8 skipped; ran for 7173ms
All tests done, failures: 3.
Exit with fail due to some tests failure.

I found in the following in trace logs, I don't know why android_browser_4_4 is not recognized, but I don't think this is related.

[android 4.4, Android Browser null] client.createWorker | response: undefined Error: Validation Failed - `device` invalid
    at IncomingMessage.<anonymous> (/home/travis/build/ncoden/foundation-sites/node_modules/browserstack/lib/client.js:73:9)
    at IncomingMessage.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1090:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Error from BrowserStack:  Error: Validation Failed - `device` invalid
    at IncomingMessage.<anonymous> (/home/travis/build/ncoden/foundation-sites/node_modules/browserstack/lib/client.js:73:9)
    at IncomingMessage.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1090:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

See also:

• Browserstack config file: https://github.com/zurb/foundation-sites/blob/develop/browserstack.json
• Full log: https://travis-ci.org/ncoden/foundation-sites/jobs/387169582
• Full log with trace log level: https://travis-ci.org/ncoden/foundation-sites/jobs/387194396

Possible solution

I checked at the source code. I have no problem with Node, but the code around the worker execution and results processing is very confusing. I am unable to propose a solution for now.

[ncoden]

Poke @mohitmun @pulkitsharma07 👋

[DanielRuf]

Bump. Would be good to have a solution for testing Android 4.4.