TestDirectory reports number of failures and failed files

[fingolfin]

With this, you may get output like this, with the
relevant change in the "total" line near the end.

gap> TestDirectory("misc");
Architecture: x86_64-apple-darwin15.6.0-default64-kv5

testing: misc/err.tst
       0 ms (0 ms GC) and 46.6KB allocated for err.tst
testing: misc/foo.tst
########> Diff in misc/foo.tst:3
# Input is:
1+1;
# Expected output:
3
# But found:
2
########
       0 ms (0 ms GC) and 6.62KB allocated for foo.tst
-----------------------------------
total         1 ms (0 ms GC) and 53.8KB allocated
              1 failures in 1 of 2 files

#I  Errors detected while testing

false
gap>

Resolves #2861

[olexandr-konovalov]

This will be useful @fingolfin - let's see how this will be seen in Travis.

[codecov]

Codecov Report

Merging #2899 into master will decrease coverage by <.01%.
The diff coverage is 86.95%.

@@            Coverage Diff             @@
##           master    #2899      +/-   ##
==========================================
- Coverage   83.74%   83.74%   -0.01%     
==========================================
  Files         680      680              
  Lines      346265   346280      +15     
==========================================
+ Hits       289971   289979       +8     
- Misses      56294    56301       +7

Impacted Files	Coverage Δ
lib/test.gi	65.05% <86.95%> (+0.73%)	⬆️
src/iostream.c	62.35% <0%> (-1.15%)	⬇️
hpcgap/lib/hpc/stdtasks.g	71.77% <0%> (-0.41%)	⬇️
lib/read4.g	100% <0%> (ø)	⬆️

[olexandr-konovalov]

@fingolfin The problem with building manuals is not present in the master branch. Please rebase.

[fingolfin]

What problem are you talking about? Travis and AppVeyor passed now. Also, what do you mean with "please rebase" -- this is already based on 94e61ab, the current master branch.

[fingolfin]

(There simply was a stupid XML error in the GAPDoc, which is fixed now)

[olexandr-konovalov]

I've been looking at the earlier test of this PR on Travis - that happened at https://travis-ci.org/gap-system/gap/builds/437551821. Indeed, fine now. Seen XML error in two separate PRs but not in the master branch, hence suggested to rebase...

[fingolfin]

Ah I see: was confused because I saw your earlier comment pointing out broken XML; but after fixing that, the comment was gone?! Anyway, tests pass now.

[olexandr-konovalov]

Thanks @fingolfin - the code+docs look good, and I've fetched the PR and tested locally with both outcomes:

total     81576 ms (24411 ms GC) and 9.39GB allocated, 10 failures in 2 of 
220 files

and

total     78906 ms (23648 ms GC) and 9.34GB allocated, 0 failures in 0 of 
220 files

• the total line becomes longer, maybe print the X failures in Y of Z files from a new line?
• maybe 0 failures in 220 files rather than 0 failures in 0 of 220 files?

[fingolfin]

@alex-konovalov Done. One could next argue that 1 failures should be 1 failure, but I really don't want to spend more time on such minor details right now. If somebody is annoyed by that, I'll be happy to review their PR, though :-).

[olexandr-konovalov]

@fingolfin should this be backported to stable-4.10?

[fingolfin]

I don't see why.

[olexandr-konovalov]

Because it's useful and should not break things... Or, actually, because I don't want to maintain two different versions of scripts for jenkins which look up for the line with the word total at the end of the file with tail -5 | head -1 and tail -4 | head -1respectively :) (but I can).

[fingolfin]

I don't find the first argument compelling, but the final one kinda is ;-)

[fingolfin]

Backported via 9179000