Add report-only mode for accessibility violations

.editorconfig

@@ -0,0 +1,12 @@
+.editorconfigroot = true
+
+[*]
+end_of_line = lf
+trim_trailing_whitespace = true
+insert_final_newline = true
+indent_style = space
+indent_size = 2
+
+[Makefile]
+indent_style = tab
+indent_size = 4

.gitignore

@@ -6,3 +6,7 @@
 /test/dummy/log/*.log
 /test/dummy/tmp/
 Gemfile.lock
+
+# Claude
+**/.claude/settings.local.json
+**/CLAUDE.local.md

CHANGELOG.md

@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+- Add report-only mode for accessibility violations
+
+  Introduces the ability to collect and report accessibility violations without
+  failing tests. Configure `config.capybara_accessibility_audit.audit_enabled`
+  with `:stdout` to log violations to stdout, or `{ file: 'path' }` to export
+  structured JSON reports with full violation details including severity, WCAG
+  tags, affected elements, and remediation guidance.
+
+  Maintains full backwards compatibility with existing `true`/`false` values.
+
+  *Daniel Gasienica*
+
 ## 0.2.0 (April 08, 2024)
 
 - Remove unused `require "axe-capybara"` statement

README.md

@@ -30,23 +30,23 @@ Invocation: axe.run({:exclude=>[]}, {}, callback);
 Installing the gem will automatically configure your System Tests to audit for
 accessibility violations after common actions, including:
 
-* [`visit`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session:visit)
-* [`click_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_button-instance_method)
-* [`click_link`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link-instance_method)
-* [`click_link_or_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link_or_button-instance_method)
-* [`click_on`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions:click_on)
+- [`visit`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session:visit)
+- [`click_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_button-instance_method)
+- [`click_link`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link-instance_method)
+- [`click_link_or_button`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions#click_link_or_button-instance_method)
+- [`click_on`](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions:click_on)
 
 Under the hood, `capybara_accessibility_audit` relies on [axe-core-rspec][], which uses [aXe][]
 to audit for accessibility violations. To configure which options are passed to
 the `be_axe_clean` matcher, override the class-level
 `accessibility_audit_options`. Supported keys include:
 
-* [`according_to:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#according_to---accessibility-standard-tag-clause)
-* [`checking_only:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking_only---exclusive-rules-clause)
-* [`checking:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking---checking-rules-clause)
-* [`excluding:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#excluding---exclusion-clause)
-* [`skipping:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#skipping---skipping-rules-clause)
-* [`within:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#within---inclusion-clause)
+- [`according_to:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#according_to---accessibility-standard-tag-clause)
+- [`checking_only:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking_only---exclusive-rules-clause)
+- [`checking:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#checking---checking-rules-clause)
+- [`excluding:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#excluding---exclusion-clause)
+- [`skipping:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#skipping---skipping-rules-clause)
+- [`within:`](https://github.com/dequelabs/axe-core-gems/blob/develop/packages/axe-core-rspec/README.md#within---inclusion-clause)
 
 To override the class-level setting, wrap code in calls to the
 `with_accessibility_audit_options` method:
@@ -62,8 +62,7 @@ end
 
 ## Frequently Asked Questions
 
-My application already exists, automated accessibility audits are uncovering violations left and right. Do I have to fix them all at once?
----
+## My application already exists, automated accessibility audits are uncovering violations left and right. Do I have to fix them all at once?
 
 Your suite has control over which rules are skipped and which rules are
 enforced through the `accessibility_audit_options` configuration.
@@ -114,8 +113,7 @@ end
 As you resolve the violations, you can remove entries from the list of skipped
 rules.
 
-I've implemented a custom Capybara action to toggle a disclosure element. How can I automatically audit for violations after it's called?
----
+## I've implemented a custom Capybara action to toggle a disclosure element. How can I automatically audit for violations after it's called?
 
 You can add the method to the list of methods that will initiate an automated
 audit:
@@ -130,8 +128,34 @@ class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
 end
 ```
 
-How can I turn off auditing for the entire suite?
----
+## How can I report violations without failing my tests?
+
+You can configure the gem to collect and report violations without failing tests
+by setting `accessibility_audit_enabled` to `:stdout` or `{ file: 'path' }`:
+
+**Report to stdout:**
+
+```ruby
+# config/environments/test.rb
+config.capybara_accessibility_audit.audit_enabled = :stdout
+```
+
+**Report to JSON file:**
+
+```ruby
+# config/environments/test.rb
+config.capybara_accessibility_audit.audit_enabled = { file: 'tmp/accessibility_violations.json' }
+```
+
+The JSON report includes structured violation data with severity levels, WCAG tags,
+affected elements, and remediation guidance. Violations are collected throughout
+the test run and output at the end with a summary grouped by page and by rule.
+
+This is useful when introducing accessibility auditing to an existing application
+with many violations - you can run your suite without failures to get a complete
+report of all issues to address.
+
+## How can I turn off auditing for the entire suite?
 
 You can disable automated auditing within your `ApplicationSystemTestCase`:
 
@@ -141,8 +165,7 @@ class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
 end
 ```
 
-How can I turn off auditing for a block of code?
----
+## How can I turn off auditing for a block of code?
 
 You can disable automated auditing temporarily by wrapping code in a
 `skip_accessibility_audits` block:
@@ -161,8 +184,7 @@ class MySystemTest < ApplicationSystemTestCase
 end
 ```
 
-How can I turn off auditing hooks for a method?
----
+## How can I turn off auditing hooks for a method?
 
 You can remove the method from the test's [Set][] of
 `accessibility_audit_after_methods` configuration by calling
@@ -176,8 +198,7 @@ end
 
 [Set]: https://ruby-doc.org/stdlib-3.0.1/libdoc/set/rdoc/Set.html
 
-How can I turn off auditing for a test file?
----
+## How can I turn off auditing for a test file?
 
 You can disable automated auditing at the class-level:
 
@@ -205,13 +226,15 @@ end
 ```
 
 ## Installation
+
 Add this line to your application's Gemfile:
 
 ```ruby
 gem "capybara_accessibility_audit"
 ```
 
 And then execute:
+
 ```bash
 $ bundle
 ```
@@ -221,9 +244,11 @@ $ bundle
 Please read [CONTRIBUTING.md](./CONTRIBUTING.md).
 
 ## License
+
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
 <!-- START /templates/footer.md -->
+
 ## About thoughtbot
 
 ![thoughtbot](https://thoughtbot.com/thoughtbot-logo-for-readmes.svg)
@@ -238,5 +263,4 @@ We are [available for hire][hire].
 [community]: https://thoughtbot.com/community?utm_source=github
 [hire]: https://thoughtbot.com/hire-us?utm_source=github
 
-
 <!-- END /templates/footer.md -->

capybara_accessibility_audit.gemspec

@@ -3,8 +3,8 @@ require_relative "lib/capybara_accessibility_audit/version"
 Gem::Specification.new do |spec|
   spec.name = "capybara_accessibility_audit"
   spec.version = CapybaraAccessibilityAudit::VERSION
-  spec.authors = ["Sean Doyle"]
-  spec.email = ["[email protected]"]
+  spec.authors = ["Sean Doyle", "Daniel Gasienica"]
+  spec.email = ["[email protected]", "[email protected]"]
   spec.homepage = "https://github.com/thoughtbot/capybara_accessibility_audit"
   spec.summary = "Accessibility tooling for Capybara"
   spec.description = "Accessibility tooling for Capybara"

lib/capybara_accessibility_audit/audit_system_test_extensions.rb

@@ -1,4 +1,6 @@
 require "axe/matchers/be_axe_clean"
+require_relative "report_mode"
+require_relative "reporter"
 
 module CapybaraAccessibilityAudit
   module AuditSystemTestExtensions
@@ -13,9 +15,46 @@ module AuditSystemTestExtensions
 
     included do
       class_attribute :accessibility_audit_after_methods, default: Set.new
-      class_attribute :accessibility_audit_enabled, default: true
       class_attribute :accessibility_audit_options, default: ActiveSupport::OrderedOptions.new
 
+      # Store the actual value internally
+      class_attribute :_accessibility_audit_enabled_value, default: true
+
+      # Internal accessor for report mode - converts accessibility_audit_enabled to ReportMode
+      def self.accessibility_audit_report_mode
+        @accessibility_audit_report_mode ||= ReportMode.from_config(_accessibility_audit_enabled_value)
+      end
+
+      def self.accessibility_audit_report_mode=(mode)
+        @accessibility_audit_report_mode = mode.is_a?(ReportMode) ? mode : ReportMode.from_config(mode)
+      end
+
+      def accessibility_audit_report_mode
+        self.class.accessibility_audit_report_mode
+      end
+
+      def accessibility_audit_report_mode=(mode)
+        self.class.accessibility_audit_report_mode = mode
+      end
+
+      # Public accessors for backwards compatibility
+      def self.accessibility_audit_enabled=(value)
+        @accessibility_audit_report_mode = nil  # Clear cache
+        self._accessibility_audit_enabled_value = value
+      end
+
+      def self.accessibility_audit_enabled
+        accessibility_audit_report_mode.enabled?
+      end
+
+      def accessibility_audit_enabled=(value)
+        self.class.accessibility_audit_enabled = value
+      end
+
+      def accessibility_audit_enabled
+        self.class.accessibility_audit_enabled
+      end
+
       MODAL_METHODS.each do |method|
         define_method method do |*arguments, **options, &block|
           result = super(*arguments, **options) { skip_accessibility_audits(&block) }
@@ -103,7 +142,52 @@ def assert_no_accessibility_violations(**options)
       axe_matcher = Axe::Matchers::BeAxeClean.new
       axe_matcher = options.inject(axe_matcher) { |matcher, option| matcher.public_send(*option) }
 
-      assert axe_matcher.matches?(page), axe_matcher.failure_message
+      # Run the audit to get structured results
+      audit = axe_matcher.audit(page)
+
+      # If audit passed, nothing to do
+      return if audit.passed?
+
+      # When assert_no_accessibility_violations is called explicitly, always assert
+      # (ignore the global report mode setting)
+      # This ensures manual assertions always fail tests, even if auto-audits are disabled
+      failure_message = ReportMode::Assert.new.handle_violations(
+        audit: audit,
+        url: page.current_url
+      )
+
+      assert false, failure_message
+    end
+
+    # Used by Auditor for automatic audits - respects the report mode setting
+    def audit_with_report_mode(**options)
+      options.assert_valid_keys(
+        :according_to,
+        :checking,
+        :checking_only,
+        :excluding,
+        :skipping,
+        :within
+      )
+      options.compact_blank!
+
+      axe_matcher = Axe::Matchers::BeAxeClean.new
+      axe_matcher = options.inject(axe_matcher) { |matcher, option| matcher.public_send(*option) }
+
+      # Run the audit to get structured results
+      audit = axe_matcher.audit(page)
+
+      # If audit passed, nothing to do
+      return if audit.passed?
+
+      # Use the configured report mode for automatic audits
+      failure_message = accessibility_audit_report_mode.handle_violations(
+        audit: audit,
+        url: page.current_url
+      )
+
+      # Only assert if we're in assert mode (failure_message will be nil for report modes)
+      assert false, failure_message if failure_message
     end
   end
 end

lib/capybara_accessibility_audit/auditor.rb

@@ -8,7 +8,7 @@ def initialize(test)
 
     def audit!(method)
       if accessibility_audit_enabled && method.in?(accessibility_audit_after_methods) && javascript_enabled?
-        assert_no_accessibility_violations(**accessibility_audit_options)
+        audit_with_report_mode(**accessibility_audit_options)
       end
     end
 

lib/capybara_accessibility_audit/engine.rb

@@ -8,18 +8,22 @@ class Engine < ::Rails::Engine
       click_link_or_button
       click_on
     ]
+    # audit_enabled accepts: false (disabled), true (assert mode), :assert, :stdout, or { file: 'path' }
     config.capybara_accessibility_audit.audit_enabled = true
 
+    # Minitest
     initializer "capybara_accessibility_audit.minitest" do |app|
       ActiveSupport.on_load :action_dispatch_system_test_case do
         include CapybaraAccessibilityAudit::AuditSystemTestExtensions
 
+        # Use the backwards-compatible accessor which handles conversion
         self.accessibility_audit_enabled = app.config.capybara_accessibility_audit.audit_enabled
 
         accessibility_audit_after app.config.capybara_accessibility_audit.audit_after
       end
     end
 
+    # RSpec
     initializer "capybara_accessibility_audit.rspec" do |app|
       if defined?(RSpec)
         require "rspec/core"
@@ -36,8 +40,25 @@ class Engine < ::Rails::Engine
 
           config.before(type: :system, &configure)
           config.before(type: :feature, &configure)
+
+          config.after(:suite) do
+            Reporter.report!
+          end
+        end
+      end
+    end
+
+    # Minitest
+    config.after_initialize do
+      if defined?(Minitest)
+        Minitest.after_run do
+          Reporter.report!
         end
       end
     end
+
+    rake_tasks do
+      load File.expand_path("../tasks/capybara_accessibility_audit.rake", __dir__)
+    end
   end
 end