Testing automation

[jzaefferer]

We want to test this in all supported browsers automatically. I'll look into creating a configuration file for browserstack-runner to see if that works here. Otherwise we can try it with TestSwarm.

[jzaefferer]

@rxaviers can you follow the usage instructions on here? https://github.com/browserstack/browserstack-runner

It should be really easy to get running. Just make sure you've got BROWSERSTACK_USER and BROWSERSTACK_KEY set as env vars.

Would be good to get some feedback from someone who hasn't used the tool before. Please let me know about any issues, no matter how tiny.

[rxaviers]

I assume you have used it before then. Do you have any project with this setup?

I've come up with the below browserstack.json:

{
  "username": "<username>",
  "key": "<key>",
  "test_framework": "qunit",
  "test_path": ["test"],
  "browsers": [
    "chrome_previous",
    "chrome_latest",
    "firefox_previous",
    "firefox_latest",
    "ie_8",
    "ie_9",
    "ie_10",
    "ie_11",
    "ie_latest",
    "opera_12_1",
    "opera_previous",
    "opera_latest",
    "safari_5_1",
    "safari_previous",
    "safari_latest"
  ]
}

Questions:
1)

  "test_framework": "qunit",
  "test_path": ["test"],

Tests should be executed using grunt test. Is the above equivalent?

  "browsers": [...]

Globalize supports http://jquery.com/browser-support/, except IE6 and IE7 right? What are the keywords for mobile? Is config missing any browser entry?

[jzaefferer]

1. Let's get this running first using the browserstack-runner CLI, we can worry about grunt integration later.

2. You don't need both ie_11 and ie_latest. As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

[rxaviers]

Note I am not worried about grunt integration. My concern is whether or not executing browserstack-runner CLI will run the same tests we do via grunt test.

[rxaviers]

You don't need both ie_11 and ie_latest.

Why?

As for mobile browsers, we can add those later, once this is working. Meanwhile I've asked them about the lacks of docs for mobile browser config.

Excellent!

[jzaefferer]

I lost track of our IRC conversation. What was your last result after changing the test_path?

[rxaviers]

http://irc.jquery.org/%23jqueryui-dev/default_%23jqueryui-dev_20140423.log.html#t11:54:32 :)

[jzaefferer]

I just pulled and ran grunt and got a timeout from PhantomJS:

Running "qunit:unit" (qunit) task
Testing http://localhost:9001/unit.html
>> PhantomJS timed out, possibly due to a missing QUnit start() call.
Warning: 1/1 assertions failed (0ms) Use --force to continue.

Seems pointless to run tests through browserstack when they don't even run through PhantomJS.

Once that works, could you retry with browserstack-runner? I suggest to start with just one browser, that will speed things up a lot.

[rxaviers]

It works just fine to me... I have double checked by cloning Globalize into a brand new directory, npm and bower installed dependencies, ran grunt and it worked again. This is my environment https://gist.github.com/rxaviers/11368480.

Could you double check yours please?

PS: What command does browserstack-runner actually run?

[jzaefferer]

I've tracked down the problem within browserstack-runner and sent a PR: browserstack/browserstack-runner#83

Once that is merged and released it should work fine. I had a green run locally without timeouts.

[rxaviers]

👍

[rxaviers]

@jzaefferer your PR is merged. Are we good to proceed?

[jzaefferer]

Sure. It was working for me with the patch, so can you give it another try?