Add plugin for cpu/memory intensive tests

[saimn]

Related to astropy/astropy#12350

With this plugin, cpu/memory intensive tests

• are not run by default
• can be run with the other tests with --run-slow and --run-hugemem
• can be run separately with -m slow and -n hugemem

If you have better ideas for the marker names ...

[pllim]

I think these two new options need to be documented on README.

• When should one mark a test as slow or hugemem or both?
• How do you use these features?
• What if another plugin already registered these markers? Then what is the intended behavior? (Possible since "slow" is actually in official pytest example.)

[pllim]

Needs a change log too. Thanks!

[pllim]

Maybe tests too or is that overkill? 🤷

[saimn]

When should one mark a test as slow or hugemem or both?

• slow: for tests that take a lot of time and that you want to run only in one job, or in a specific job

What if another plugin already registered these markers? Then what is the intended behavior? (Possible since "slow" is actually in official pytest example.)

No idea, but not it's an issue for all plugins I guess.

Maybe tests too or is that overkill? shrug

Ideally yes but I'm not sure how to do that. It works on the astropy PR so that's a good test for now :)

[adrn]

I agree with @pllim that any new functionality here should be documented!

I'm also unsure on what makes the most sense: having a --run-slow option (where by default slow tests are skipped), or having a --skip-slow option (where by default all tests are run). Do you have thoughts on that?

[saimn]

Sure I can add some docs, just wanted to get the feelings before since names etc could change.
The idea of --run-slow is mostly about defaults. We could also not have this option and use marker selection (-m not slow or something like that). But with the little plugin and --run-slow at least we can be sure that slow tests are not run by default which I think is what we want.

[pllim]

we can be sure that slow tests are not run by default

I agree. I wouldn't want to run them by default myself.

[pllim]

Let's get this in so we can use it at astropy/astropy#12764 . Thanks!

[eerovaher]

This pull request is missing its change log entry.

[pllim]

Ops, thanks for catching that. Added in 2d4ea39 .