diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 0000000..b5e880e --- /dev/null +++ b/.editorconfig @@ -0,0 +1,12 @@ +.editorconfigroot = true + +[*] +end_of_line = lf +trim_trailing_whitespace = true +insert_final_newline = true +indent_style = space +indent_size = 2 + +[Makefile] +indent_style = tab +indent_size = 4 diff --git a/.gitignore b/.gitignore index ce7a652..79f500d 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,7 @@ /test/dummy/log/*.log /test/dummy/tmp/ Gemfile.lock + +# Claude +**/.claude/settings.local.json +**/CLAUDE.local.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b6b832..0faa78d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +- Add report-only mode for accessibility violations + + Introduces the ability to collect and report accessibility violations without + failing tests. Configure `config.capybara_accessibility_audit.audit_enabled` + with `:stdout` to log violations to stdout, or `{ file: 'path' }` to export + structured JSON reports with full violation details including severity, WCAG + tags, affected elements, and remediation guidance. + + Maintains full backwards compatibility with existing `true`/`false` values. + + *Daniel Gasienica* + ## 0.2.0 (April 08, 2024) - Remove unused `require "axe-capybara"` statement diff --git a/README.md b/README.md index c2b06b2..d980b6b 100644 --- a/README.md +++ b/README.md @@ -30,23 +30,23 @@ Invocation: axe.run({:exclude=>[]}, {}, callback); Installing the gem will automatically configure your System Tests to audit for accessibility violations after common actions, including: -* [`visit`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session:visit) -* [`click_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_button-instance_method) -* [`click_link`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link-instance_method) -* [`click_link_or_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link_or_button-instance_method) -* [`click_on`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions:click_on) +- [`visit`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session:visit) +- [`click_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_button-instance_method) +- [`click_link`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link-instance_method) +- [`click_link_or_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link_or_button-instance_method) +- [`click_on`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions:click_on) Under the hood, `capybara_accessibility_audit` relies on [axe-core-rspec][], which uses [aXe][] to audit for accessibility violations. To configure which options are passed to the `be_axe_clean` matcher, override the class-level `accessibility_audit_options`. Supported keys include: -* [`according_to:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#according_to---accessibility-standard-tag-clause) -* [`checking_only:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking_only---exclusive-rules-clause) -* [`checking:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking---checking-rules-clause) -* [`excluding:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#excluding---exclusion-clause) -* [`skipping:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#skipping---skipping-rules-clause) -* [`within:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#within---inclusion-clause) +- [`according_to:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#according_to---accessibility-standard-tag-clause) +- [`checking_only:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking_only---exclusive-rules-clause) +- [`checking:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking---checking-rules-clause) +- [`excluding:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#excluding---exclusion-clause) +- [`skipping:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#skipping---skipping-rules-clause) +- [`within:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#within---inclusion-clause) To override the class-level setting, wrap code in calls to the `with_accessibility_audit_options` method: @@ -62,8 +62,7 @@ end ## Frequently Asked Questions -My application already exists, automated accessibility audits are uncovering violations left and right. Do I have to fix them all at once? ---- +## My application already exists, automated accessibility audits are uncovering violations left and right. Do I have to fix them all at once? Your suite has control over which rules are skipped and which rules are enforced through the `accessibility_audit_options` configuration. @@ -114,8 +113,7 @@ end As you resolve the violations, you can remove entries from the list of skipped rules. -I've implemented a custom Capybara action to toggle a disclosure element. How can I automatically audit for violations after it's called? ---- +## I've implemented a custom Capybara action to toggle a disclosure element. How can I automatically audit for violations after it's called? You can add the method to the list of methods that will initiate an automated audit: @@ -130,8 +128,34 @@ class ApplicationSystemTestCase < ActionDispatch::SystemTestCase end ``` -How can I turn off auditing for the entire suite? ---- +## How can I report violations without failing my tests? + +You can configure the gem to collect and report violations without failing tests +by setting `accessibility_audit_enabled` to `:stdout` or `{ file: 'path' }`: + +**Report to stdout:** + +```ruby +# config/environments/test.rb +config.capybara_accessibility_audit.audit_enabled = :stdout +``` + +**Report to JSON file:** + +```ruby +# config/environments/test.rb +config.capybara_accessibility_audit.audit_enabled = { file: 'tmp/accessibility_violations.json' } +``` + +The JSON report includes structured violation data with severity levels, WCAG tags, +affected elements, and remediation guidance. Violations are collected throughout +the test run and output at the end with a summary grouped by page and by rule. + +This is useful when introducing accessibility auditing to an existing application +with many violations - you can run your suite without failures to get a complete +report of all issues to address. + +## How can I turn off auditing for the entire suite? You can disable automated auditing within your `ApplicationSystemTestCase`: @@ -141,8 +165,7 @@ class ApplicationSystemTestCase < ActionDispatch::SystemTestCase end ``` -How can I turn off auditing for a block of code? ---- +## How can I turn off auditing for a block of code? You can disable automated auditing temporarily by wrapping code in a `skip_accessibility_audits` block: @@ -161,8 +184,7 @@ class MySystemTest < ApplicationSystemTestCase end ``` -How can I turn off auditing hooks for a method? ---- +## How can I turn off auditing hooks for a method? You can remove the method from the test's [Set][] of `accessibility_audit_after_methods` configuration by calling @@ -176,8 +198,7 @@ end [Set]: https://ruby-doc.org/stdlib-3.0.1/libdoc/set/rdoc/Set.html -How can I turn off auditing for a test file? ---- +## How can I turn off auditing for a test file? You can disable automated auditing at the class-level: @@ -205,6 +226,7 @@ end ``` ## Installation + Add this line to your application's Gemfile: ```ruby @@ -212,6 +234,7 @@ gem "capybara_accessibility_audit" ``` And then execute: + ```bash $ bundle ``` @@ -221,9 +244,11 @@ $ bundle Please read [CONTRIBUTING.md](./CONTRIBUTING.md). ## License + The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT). + ## About thoughtbot ![thoughtbot](https://thoughtbot.com/thoughtbot-logo-for-readmes.svg) @@ -238,5 +263,4 @@ We are [available for hire][hire]. [community]: https://thoughtbot.com/community?utm_source=github [hire]: https://thoughtbot.com/hire-us?utm_source=github - diff --git a/capybara_accessibility_audit.gemspec b/capybara_accessibility_audit.gemspec index edb104a..47c1103 100644 --- a/capybara_accessibility_audit.gemspec +++ b/capybara_accessibility_audit.gemspec @@ -3,8 +3,8 @@ require_relative "lib/capybara_accessibility_audit/version" Gem::Specification.new do |spec| spec.name = "capybara_accessibility_audit" spec.version = CapybaraAccessibilityAudit::VERSION - spec.authors = ["Sean Doyle"] - spec.email = ["sean.p.doyle24@gmail.com"] + spec.authors = ["Sean Doyle", "Daniel Gasienica"] + spec.email = ["sean.p.doyle24@gmail.com", "daniel@gasienica.ch"] spec.homepage = "https://github.com/thoughtbot/capybara_accessibility_audit" spec.summary = "Accessibility tooling for Capybara" spec.description = "Accessibility tooling for Capybara" diff --git a/lib/capybara_accessibility_audit/audit_system_test_extensions.rb b/lib/capybara_accessibility_audit/audit_system_test_extensions.rb index 9624e59..5cbf9dd 100644 --- a/lib/capybara_accessibility_audit/audit_system_test_extensions.rb +++ b/lib/capybara_accessibility_audit/audit_system_test_extensions.rb @@ -1,4 +1,6 @@ require "axe/matchers/be_axe_clean" +require_relative "report_mode" +require_relative "reporter" module CapybaraAccessibilityAudit module AuditSystemTestExtensions @@ -13,9 +15,46 @@ module AuditSystemTestExtensions included do class_attribute :accessibility_audit_after_methods, default: Set.new - class_attribute :accessibility_audit_enabled, default: true class_attribute :accessibility_audit_options, default: ActiveSupport::OrderedOptions.new + # Store the actual value internally + class_attribute :_accessibility_audit_enabled_value, default: true + + # Internal accessor for report mode - converts accessibility_audit_enabled to ReportMode + def self.accessibility_audit_report_mode + @accessibility_audit_report_mode ||= ReportMode.from_config(_accessibility_audit_enabled_value) + end + + def self.accessibility_audit_report_mode=(mode) + @accessibility_audit_report_mode = mode.is_a?(ReportMode) ? mode : ReportMode.from_config(mode) + end + + def accessibility_audit_report_mode + self.class.accessibility_audit_report_mode + end + + def accessibility_audit_report_mode=(mode) + self.class.accessibility_audit_report_mode = mode + end + + # Public accessors for backwards compatibility + def self.accessibility_audit_enabled=(value) + @accessibility_audit_report_mode = nil # Clear cache + self._accessibility_audit_enabled_value = value + end + + def self.accessibility_audit_enabled + accessibility_audit_report_mode.enabled? + end + + def accessibility_audit_enabled=(value) + self.class.accessibility_audit_enabled = value + end + + def accessibility_audit_enabled + self.class.accessibility_audit_enabled + end + MODAL_METHODS.each do |method| define_method method do |*arguments, **options, &block| result = super(*arguments, **options) { skip_accessibility_audits(&block) } @@ -103,7 +142,52 @@ def assert_no_accessibility_violations(**options) axe_matcher = Axe::Matchers::BeAxeClean.new axe_matcher = options.inject(axe_matcher) { |matcher, option| matcher.public_send(*option) } - assert axe_matcher.matches?(page), axe_matcher.failure_message + # Run the audit to get structured results + audit = axe_matcher.audit(page) + + # If audit passed, nothing to do + return if audit.passed? + + # When assert_no_accessibility_violations is called explicitly, always assert + # (ignore the global report mode setting) + # This ensures manual assertions always fail tests, even if auto-audits are disabled + failure_message = ReportMode::Assert.new.handle_violations( + audit: audit, + url: page.current_url + ) + + assert false, failure_message + end + + # Used by Auditor for automatic audits - respects the report mode setting + def audit_with_report_mode(**options) + options.assert_valid_keys( + :according_to, + :checking, + :checking_only, + :excluding, + :skipping, + :within + ) + options.compact_blank! + + axe_matcher = Axe::Matchers::BeAxeClean.new + axe_matcher = options.inject(axe_matcher) { |matcher, option| matcher.public_send(*option) } + + # Run the audit to get structured results + audit = axe_matcher.audit(page) + + # If audit passed, nothing to do + return if audit.passed? + + # Use the configured report mode for automatic audits + failure_message = accessibility_audit_report_mode.handle_violations( + audit: audit, + url: page.current_url + ) + + # Only assert if we're in assert mode (failure_message will be nil for report modes) + assert false, failure_message if failure_message end end end diff --git a/lib/capybara_accessibility_audit/auditor.rb b/lib/capybara_accessibility_audit/auditor.rb index 131fd5a..8ba5fc9 100644 --- a/lib/capybara_accessibility_audit/auditor.rb +++ b/lib/capybara_accessibility_audit/auditor.rb @@ -8,7 +8,7 @@ def initialize(test) def audit!(method) if accessibility_audit_enabled && method.in?(accessibility_audit_after_methods) && javascript_enabled? - assert_no_accessibility_violations(**accessibility_audit_options) + audit_with_report_mode(**accessibility_audit_options) end end diff --git a/lib/capybara_accessibility_audit/engine.rb b/lib/capybara_accessibility_audit/engine.rb index 8e02629..a39c52c 100644 --- a/lib/capybara_accessibility_audit/engine.rb +++ b/lib/capybara_accessibility_audit/engine.rb @@ -8,18 +8,22 @@ class Engine < ::Rails::Engine click_link_or_button click_on ] + # audit_enabled accepts: false (disabled), true (assert mode), :assert, :stdout, or { file: 'path' } config.capybara_accessibility_audit.audit_enabled = true + # Minitest initializer "capybara_accessibility_audit.minitest" do |app| ActiveSupport.on_load :action_dispatch_system_test_case do include CapybaraAccessibilityAudit::AuditSystemTestExtensions + # Use the backwards-compatible accessor which handles conversion self.accessibility_audit_enabled = app.config.capybara_accessibility_audit.audit_enabled accessibility_audit_after app.config.capybara_accessibility_audit.audit_after end end + # RSpec initializer "capybara_accessibility_audit.rspec" do |app| if defined?(RSpec) require "rspec/core" @@ -36,8 +40,25 @@ class Engine < ::Rails::Engine config.before(type: :system, &configure) config.before(type: :feature, &configure) + + config.after(:suite) do + Reporter.report! + end + end + end + end + + # Minitest + config.after_initialize do + if defined?(Minitest) + Minitest.after_run do + Reporter.report! end end end + + rake_tasks do + load File.expand_path("../tasks/capybara_accessibility_audit.rake", __dir__) + end end end diff --git a/lib/capybara_accessibility_audit/report_merger.rb b/lib/capybara_accessibility_audit/report_merger.rb new file mode 100644 index 0000000..6c6c288 --- /dev/null +++ b/lib/capybara_accessibility_audit/report_merger.rb @@ -0,0 +1,213 @@ +# frozen_string_literal: true + +require "json" +require "fileutils" + +module CapybaraAccessibilityAudit + class ReportMerger + def self.merge(report_paths:, output_path:) + new(report_paths: report_paths, output_path: output_path).merge + end + + def self.merge_data(reports) + new.merge_data(reports) + end + + def initialize(report_paths: [], output_path: nil) + @report_paths = Array(report_paths) + @output_path = output_path + end + + def merge + validate_inputs! + reports = load_reports + merged = merge_data(reports) + write_output(merged) if @output_path + merged + end + + def merge_data(reports) + return empty_report if reports.empty? + + violations_by_page = merge_violations_by_page(reports) + violations_by_rule = merge_violations_by_rule(reports) + + { + summary: calculate_summary(violations_by_page, reports), + violations_by_rule: violations_by_rule, + violations_by_page: violations_by_page + } + end + + private + + IMPACT_PRIORITY = Reporter::IMPACT_PRIORITY + + def validate_inputs! + raise ArgumentError, "No report paths provided" if @report_paths.empty? + + @report_paths.each do |path| + raise ArgumentError, "File not found: #{path}" unless File.exist?(path) + end + end + + def load_reports + @report_paths.map do |path| + JSON.parse(File.read(path), symbolize_names: true) + rescue JSON::ParserError => e + raise JSON::ParserError, "Invalid JSON in #{path}: #{e.message}" + end + end + + def write_output(merged_data) + FileUtils.mkdir_p(File.dirname(@output_path)) + File.write(@output_path, JSON.pretty_generate(merged_data)) + puts "\nMerged report written to: #{@output_path}" + end + + def empty_report + { + summary: { + total_violations: 0, + num_pages_with_violations: 0, + num_violations_by_impact: IMPACT_PRIORITY.keys.sort_by { |k| -IMPACT_PRIORITY[k] }.each_with_object({}) { |k, h| h[k] = 0 }, + generated_at: Time.now.iso8601 + }, + violations_by_rule: {}, + violations_by_page: [] + } + end + + def merge_violations_by_page(reports) + pages_by_url = {} + + reports.each do |report| + report[:violations_by_page].each do |page_data| + url = page_data[:url] + + pages_by_url[url] = if pages_by_url[url] + merge_page_data(pages_by_url[url], page_data) + else + deep_dup(page_data) + end + end + end + + pages_by_url.values.sort_by { |page| page[:url] } + end + + def merge_page_data(existing_page, new_page) + { + url: existing_page[:url], + timestamp: [existing_page[:timestamp], new_page[:timestamp]].max, + violations: merge_violations(existing_page[:violations], new_page[:violations]), + test_engine: new_page[:test_engine] || existing_page[:test_engine], + test_environment: new_page[:test_environment] || existing_page[:test_environment] + } + end + + def merge_violations(violations1, violations2) + violations_by_id = {} + + (violations1 + violations2).each do |violation| + rule_id = violation[:id] + + if violations_by_id[rule_id] + violations_by_id[rule_id][:nodes] += violation[:nodes] + violations_by_id[rule_id][:nodes].uniq! { |node| node[:target] } + else + violations_by_id[rule_id] = deep_dup(violation) + end + end + + violations_by_id.values.sort_by do |v| + [-IMPACT_PRIORITY.fetch(v[:impact].to_s, 0), v[:id]] + end + end + + def merge_violations_by_rule(reports) + rules = {} + + reports.each do |report| + report[:violations_by_rule].each do |rule_id, rule_data| + if rules[rule_id] + rules[rule_id][:num_occurrences] += rule_data[:num_occurrences] + rules[rule_id][:pages] = (rules[rule_id][:pages] + rule_data[:pages]).uniq.sort + else + rules[rule_id] = { + impact: rule_data[:impact], + description: rule_data[:description], + help: rule_data[:help], + helpUrl: rule_data[:helpUrl], + tags: rule_data[:tags]&.sort || [], + num_occurrences: rule_data[:num_occurrences], + pages: rule_data[:pages].dup.sort + } + end + end + end + + rules.sort_by do |_rule_id, data| + [ + -IMPACT_PRIORITY.fetch(data[:impact].to_s, 0), + -data[:num_occurrences] + ] + end.to_h + end + + def calculate_summary(violations_by_page, reports) + { + total_violations: calculate_total_violations(violations_by_page), + num_pages_with_violations: count_pages_with_violations(violations_by_page), + num_violations_by_impact: calculate_violations_by_impact(violations_by_page), + generated_at: latest_timestamp(reports) + } + end + + def calculate_total_violations(violations_by_page) + violations_by_page.sum do |page_data| + page_data[:violations].sum { |v| v[:nodes].count } + end + end + + def count_pages_with_violations(violations_by_page) + violations_by_page.count { |page| page[:violations].any? } + end + + def calculate_violations_by_impact(violations_by_page) + impact_counts = Hash.new(0) + + violations_by_page.each do |page_data| + page_data[:violations].each do |violation| + impact = violation[:impact].to_s + impact_counts[impact] += violation[:nodes].count + end + end + + IMPACT_PRIORITY.keys.sort_by { |impact| -IMPACT_PRIORITY[impact] }.each_with_object({}) do |impact, result| + result[impact] = impact_counts[impact] + end + end + + def latest_timestamp(reports) + reports.map { |r| r.dig(:summary, :generated_at) } + .compact + .max || Time.now.iso8601 + end + + def deep_dup(obj) + case obj + when Hash + obj.each_with_object({}) { |(k, v), h| h[k] = deep_dup(v) } + when Array + obj.map { |v| deep_dup(v) } + else + begin + obj.dup + rescue + obj + end + end + end + end +end diff --git a/lib/capybara_accessibility_audit/report_mode.rb b/lib/capybara_accessibility_audit/report_mode.rb new file mode 100644 index 0000000..d4c2c7b --- /dev/null +++ b/lib/capybara_accessibility_audit/report_mode.rb @@ -0,0 +1,105 @@ +# frozen_string_literal: true + +module CapybaraAccessibilityAudit + class ReportMode + # Base class for report modes + def enabled? + true + end + + def report? + false + end + + def assert? + !report? + end + + def handle_violations(audit:, url:) + raise NotImplementedError + end + + # Factory method to create mode from config + # Supports backwards compatibility with accessibility_audit_enabled: + # false -> Disabled (backwards compatible with accessibility_audit_enabled = false) + # true -> Assert (backwards compatible with accessibility_audit_enabled = true) + # :assert -> Assert (new: explicit assert mode) + # :stdout -> StdoutReporter (new: report to stdout) + # { file: 'path' } -> FileReporter (new: report to JSON file) + def self.from_config(mode_config) + case mode_config + when false # Backwards compatibility: accessibility_audit_enabled = false + Disabled.new + when true # Backwards compatibility: accessibility_audit_enabled = true + Assert.new + when :assert + Assert.new + when :stdout + StdoutReporter.new + when Hash + if mode_config[:file] + FileReporter.new(mode_config[:file]) + else + raise ArgumentError, "Invalid report mode configuration: #{mode_config.inspect}" + end + else + raise ArgumentError, "Invalid report mode: #{mode_config.inspect}. Expected false, true, :assert, :stdout, or { file: 'path' }" + end + end + + # Disabled mode - audits don't run at all + # Used for backwards compatibility when accessibility_audit_enabled = false + class Disabled < ReportMode + def enabled? + false + end + + def handle_violations(audit:, url:) + raise "Impossible state: handle_violations called on Disabled mode. Audits should not run when disabled." + end + end + + # Assert mode - fails tests on violations (default behavior) + class Assert < ReportMode + def assert? + true + end + + def handle_violations(audit:, url:) + # Return the failure message to be used in assert + audit.failure_message + end + end + + # Stdout mode - logs violations to stdout, doesn't fail tests + class StdoutReporter < ReportMode + def report? + true + end + + def handle_violations(audit:, url:) + Reporter.add_violation(url: url, audit: audit) + nil # Don't fail the test + end + end + + # File mode - logs violations to JSON file, doesn't fail tests + class FileReporter < ReportMode + attr_reader :file_path + + def initialize(file_path) + @file_path = file_path + Reporter.report_file_path = file_path + end + + def report? + true + end + + def handle_violations(audit:, url:) + Reporter.add_violation(url: url, audit: audit) + nil # Don't fail the test + end + end + end +end diff --git a/lib/capybara_accessibility_audit/reporter.rb b/lib/capybara_accessibility_audit/reporter.rb new file mode 100644 index 0000000..4969424 --- /dev/null +++ b/lib/capybara_accessibility_audit/reporter.rb @@ -0,0 +1,168 @@ +# frozen_string_literal: true + +require "json" +require "fileutils" + +module CapybaraAccessibilityAudit + class Reporter + IMPACT_PRIORITY = { + "critical" => 4, + "serious" => 3, + "moderate" => 2, + "minor" => 1 + } + + def self.violations + @violations ||= [] + end + + def self.add_violation(url:, audit:, timestamp: Time.now) + violation_data = { + url: url, + timestamp: timestamp.iso8601, + violations: audit.results.violations.map { |v| violation_to_hash(v) }, + test_engine: audit.results.testEngine, + test_environment: audit.results.testEnvironment + } + + violations << violation_data + end + + def self.clear! + @violations = [] + end + + def self.report_to_stdout! + puts "\n" + "=" * 80 + puts "ACCESSIBILITY AUDIT REPORT" + puts "=" * 80 + puts "Total violations found: #{total_violation_count}" + puts "Pages with violations: #{violations.count}" + puts "=" * 80 + + violations.each_with_index do |page_data, idx| + puts "\n#{idx + 1}. URL: #{page_data[:url]}" + puts " Time: #{page_data[:timestamp]}" + puts " Violations: #{page_data[:violations].count}" + + page_data[:violations].each do |violation| + puts "\n - [#{violation[:impact].upcase}] #{violation[:id]}" + puts " #{violation[:help]}" + puts " #{violation[:helpUrl]}" + puts " Affected elements: #{violation[:nodes].count}" + puts " Tags: #{violation[:tags].join(", ")}" + end + puts " " + "-" * 76 + end + + puts "=" * 80 + "\n" + end + + def self.report_to_json!(file_path) + FileUtils.mkdir_p(File.dirname(file_path)) + + File.write(file_path, JSON.pretty_generate(summary_data)) + + puts "\n======> Accessibility audit report written to: #{file_path}" + end + + def self.report! + if report_file_path + report_to_json!(report_file_path) + else + report_to_stdout! + end + end + + class << self + attr_accessor :report_file_path + end + + private_class_method def self.violation_to_hash(rule) + { + id: rule.id, + impact: rule.impact, + description: rule.description, + help: rule.help, + helpUrl: rule.helpUrl, + tags: rule.tags, + nodes: rule.nodes.map { |node| node_to_hash(node) } + } + end + + private_class_method def self.node_to_hash(node) + { + html: node.html, + target: node.target, + failureSummary: node.failureSummary, + impact: node.impact + } + end + + private_class_method def self.total_violation_count + violations.sum { |page_data| page_data[:violations].count } + end + + private_class_method def self.summary_data + { + summary: { + total_violations: total_violation_count, + num_pages_with_violations: violations.count, + num_violations_by_impact: num_violations_by_impact, + generated_at: Time.now.iso8601 + }, + violations_by_rule: group_violations_by_rule, + violations_by_page: violations + } + end + + private_class_method def self.num_violations_by_impact + impact_counts = Hash.new(0) + + violations.each do |page_data| + page_data[:violations].each do |violation| + impact = violation[:impact].to_s + impact_counts[impact] += violation[:nodes].count + end + end + + IMPACT_PRIORITY.keys.sort_by { |impact| -IMPACT_PRIORITY[impact] }.each_with_object({}) do |impact, result| + result[impact] = impact_counts[impact] + end + end + + private_class_method def self.group_violations_by_rule + rule_counts = Hash.new(0) + rule_details = {} + + violations.each do |page_data| + page_data[:violations].each do |violation| + rule_id = violation[:id] + rule_counts[rule_id] += violation[:nodes].count + rule_details[rule_id] ||= { + impact: violation[:impact], + description: violation[:description], + help: violation[:help], + helpUrl: violation[:helpUrl], + tags: violation[:tags]&.sort || [], + num_occurrences: 0, + pages: [] + } + rule_details[rule_id][:num_occurrences] += violation[:nodes].count + rule_details[rule_id][:pages] << page_data[:url] unless rule_details[rule_id][:pages].include?(page_data[:url]) + end + end + + rule_details.each do |_rule_id, data| + data[:pages].sort! + end + + rule_details.sort_by do |_id, data| + [ + -IMPACT_PRIORITY.fetch(data[:impact].to_s, 0), + -data[:num_occurrences] + ] + end.to_h + end + end +end diff --git a/lib/tasks/capybara_accessibility_audit.rake b/lib/tasks/capybara_accessibility_audit.rake index d95b6b7..1fab62c 100644 --- a/lib/tasks/capybara_accessibility_audit.rake +++ b/lib/tasks/capybara_accessibility_audit.rake @@ -1,4 +1,42 @@ -# desc "Explaining what the task does" -# task :capybara_accessibility_audit do -# # Task goes here -# end +namespace :capybara_accessibility_audit do + desc "Merge multiple JSON accessibility audit reports into a single report" + task :merge, [:output, :pattern] => :environment do |_t, args| + require "capybara_accessibility_audit/report_merger" + + unless args[:output] + puts "Error: output path is required" + puts "Usage: rake capybara_accessibility_audit:merge[output.json,reports/*.json]" + exit 1 + end + + pattern = args[:pattern] || "tmp/accessibility_reports/*.json" + report_files = Dir.glob(pattern) + + if report_files.empty? + puts "Error: No report files found matching pattern: #{pattern}" + exit 1 + end + + puts "Merging #{report_files.count} report#{"s" if report_files.count != 1}:" + report_files.each { |f| puts " - #{f}" } + + begin + merged = CapybaraAccessibilityAudit::ReportMerger.merge( + report_paths: report_files, + output_path: args[:output] + ) + + puts "\nMerge complete!" + puts " Total violations: #{merged[:summary][:total_violations]}" + puts " Pages with violations: #{merged[:summary][:num_pages_with_violations]}" + puts " Violations by impact:" + merged[:summary][:num_violations_by_impact].each do |impact, count| + puts " #{impact}: #{count}" + end + rescue => e + puts "\nError merging reports: #{e.message}" + puts e.backtrace.first(5).map { |line| " #{line}" } if ENV["DEBUG"] + exit 1 + end + end +end diff --git a/test/fixtures/golden_report.json b/test/fixtures/golden_report.json new file mode 100644 index 0000000..721975f --- /dev/null +++ b/test/fixtures/golden_report.json @@ -0,0 +1,89 @@ +{ + "summary": { + "total_violations": 1, + "num_pages_with_violations": 1, + "num_violations_by_impact": { + "critical": 1, + "serious": 0, + "moderate": 0, + "minor": 0 + }, + "generated_at": "" + }, + "violations_by_rule": { + "label": { + "impact": "critical", + "description": "Ensure every form element has a label", + "help": "Form elements must have labels", + "helpUrl": "https://dequeuniversity.com/rules/axe/4.11/label?application=axeAPI", + "tags": [ + "ACT", + "EN-301-549", + "EN-9.4.1.2", + "RGAA-11.1.1", + "RGAAv4", + "TT5.c", + "TTv5", + "cat.forms", + "section508", + "section508.22.n", + "wcag2a", + "wcag412" + ], + "num_occurrences": 1, + "pages": [ + "http://example.com/violations?rules%5B%5D=label" + ] + } + }, + "violations_by_page": [ + { + "url": "http://example.com/violations?rules%5B%5D=label", + "timestamp": "", + "violations": [ + { + "id": "label", + "impact": "critical", + "description": "Ensure every form element has a label", + "help": "Form elements must have labels", + "helpUrl": "https://dequeuniversity.com/rules/axe/4.11/label?application=axeAPI", + "tags": [ + "cat.forms", + "wcag2a", + "wcag412", + "section508", + "section508.22.n", + "TTv5", + "TT5.c", + "EN-301-549", + "EN-9.4.1.2", + "ACT", + "RGAAv4", + "RGAA-11.1.1" + ], + "nodes": [ + { + "html": "", + "target": [ + "input" + ], + "failureSummary": "Fix any of the following:\n Element does not have an implicit (wrapped)