Retry Mechanism for Package, DNS and Command Tests

[j1shnu]

Checklist

• make test-all (UNIX) passes. CI will also test this
• unit and/or integration tests are included (if applicable)
• documentation is changed or added (if applicable)

Description of change

Added retry mechanism for package/DNS/command testing. Details are mentioned in issue #1034. The goss file syntax will be the following:

package:
  httpd:
    retry_count: 1      # Enables retry mechanism
    retry_delay: 10     # Delay (in seconds) before retrying the check
    installed: true
    name: httpd
    versions:
      - 2.2.15
    skip: false
dns:
  CNAME:c.dnstest.io:
    resolvable: true
    server: 208.67.222.222
    addrs:
    - "a.dnstest.io."
    retry_count: 2   # Enables retry mechanism
    retry_delay: 5.  # Delay (in seconds) before retrying the check
command:
  'go version':
    exit-status: 0
    exec: "go version"
    stdout:
    - go version go1.6 linux/amd64
    stderr: []
    timeout: 10000
    skip: false
    retry_count: 1   # Enables retry mechanism
    retry_delay: 5   # Delay (in seconds) before each retry attempt

---

📚 Documentation preview 📚: https://goss--1037.org.readthedocs.build/en/1037/

[ns-mjames]

@aelsabbahy Could you help merge this pr? This is imperative for our testing env

[petemounce]

I've hit the workflow-approve button for the github actions, but Travis is still a required check and I cannot address its not running.

Please would you add some test-coverage to exercise this code-path? Retries as a feature can become quite fiddly because the feature tends to grow as people decide they want also "exponential back-off" "... with jitter" "... conditional on x" and so on - is there a best-of-breed golang library that might be used instead of a bespoke implementation that this contribution begins? My thought is longer-tail support would be reduced if so, and the testing burden I'm asking for would probably be reduced were a library used instead of something bespoke.

Also, what do you think about making the delay interval units be milliseconds to support finer-grained intervals than seconds - or is this not a concern as the input will accept floats like 0.5? My thought is that being limited to 1s as the finest grained interval would be limiting in cases where goss tests are used as a health-check expecting to evaluate several times a minute.