[formats] allow highlights to be limited to a field

Related to #1620

Author: tstack

NEWS.md

@@ -29,6 +29,12 @@ Features:
   has been translated to Python3.  If the remote host
   does not have Python, lnav will fall back to the APE
   binary.
+* Highlights can now be applied to a particular field
+  instead of the whole line.  For log formats, a
+  `value` entry can now have a `highlights` object
+  that contains definitions for the pattern to match
+  and the style to apply.  For interactive use, the
+  `:highlight-field` command was added.
 * The "Files" panel now shows a progress bar for each
   file as it is being indexed and finishes with a
   check-mark if indexing was successful, a warning

docs/schemas/format-v1.schema.json

@@ -471,6 +471,52 @@
                                     "title": "/<format_name>/value/<value_name>/description",
                                     "description": "A description of the field",
                                     "type": "string"
+                                },
+                                "highlights": {
+                                    "description": "The set of highlight definitions",
+                                    "title": "/<format_name>/value/<value_name>/highlights",
+                                    "type": "object",
+                                    "patternProperties": {
+                                        "^(.+)$": {
+                                            "description": "The definition of a highlight",
+                                            "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>",
+                                            "type": "object",
+                                            "properties": {
+                                                "pattern": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/pattern",
+                                                    "description": "A regular expression to highlight in logs of this format.",
+                                                    "type": "string"
+                                                },
+                                                "color": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/color",
+                                                    "description": "The color to use when highlighting this pattern.",
+                                                    "type": "string"
+                                                },
+                                                "background-color": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/background-color",
+                                                    "description": "The background color to use when highlighting this pattern.",
+                                                    "type": "string"
+                                                },
+                                                "underline": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/underline",
+                                                    "description": "Highlight this pattern with an underline.",
+                                                    "type": "boolean"
+                                                },
+                                                "blink": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/blink",
+                                                    "description": "Highlight this pattern by blinking.",
+                                                    "type": "boolean"
+                                                },
+                                                "nestable": {
+                                                    "title": "/<format_name>/value/<value_name>/highlights/<highlight_name>/nestable",
+                                                    "description": "This highlight can be nested in another highlight.",
+                                                    "type": "boolean"
+                                                }
+                                            },
+                                            "additionalProperties": false
+                                        }
+                                    },
+                                    "additionalProperties": false
                                 }
                             },
                             "additionalProperties": false

docs/source/cookbook.rst

@@ -7,6 +7,19 @@ This chapter contains recipes for common tasks that can be done in **lnav**.
 These recipes can be used as a starting point for your own needs after some
 adaptation.
 
+Custom Web Access Log Highlights
+--------------------------------
+
+To highlight log message fields with custom colors based on their values, you
+can install a file that patches the format with the desired highlights.
+The following patch updates the :code:`access_log` format.  A
+:code:`highlights` entry is added to the :code:`value/sc_status` object that
+applies a color for codes of interest.
+
+.. literalinclude:: ../../example-scripts/access_log_highlights.json
+   :language: json
+   :caption: access_log_highlights.json
+   :linenos:
 
 Annotating Logs
 ---------------

src/cmd.parser.cc

@@ -342,6 +342,7 @@ parse_for(mode_t mode,
                     case help_parameter_format_t::HPF_LOADED_FILE:
                     case help_parameter_format_t::HPF_FORMAT_FIELD:
                     case help_parameter_format_t::HPF_NUMERIC_FIELD:
+                    case help_parameter_format_t::HPF_HIGHLIGHTED_FIELD:
                     case help_parameter_format_t::HPF_TIMEZONE:
                     case help_parameter_format_t::HPF_FILE_WITH_ZONE:
                     case help_parameter_format_t::HPF_VISIBLE_FILES:

src/command_executor.cc

@@ -384,10 +384,12 @@ execute_sql(exec_context& ec, const std::string& sql, std::string& alt_msg)
 
     require(!ec.ec_source.empty());
     const auto& source = ec.ec_source.back();
-    sql_progress_guard progress_guard(sql_progress,
-                                      sql_progress_finished,
-                                      source.s_location,
-                                      source.s_content);
+    sql_progress_guard progress_guard(
+        sql_progress,
+        sql_progress_finished,
+        source.s_location,
+        source.s_content,
+        !ec.get_provenance<exec_context::format_rewrite>().has_value());
     gettimeofday(&start_tv, nullptr);
 
     const auto* curr_stmt = stmt_str.c_str();

src/command_executor.hh

@@ -145,6 +145,7 @@ struct exec_context {
 
     struct mouse_input {};
     struct keyboard_input {};
+    struct format_rewrite {};
     struct user {};
     struct file_open {
         std::string fo_name;
@@ -153,8 +154,12 @@ struct exec_context {
         std::string ea_src;
     };
 
-    using provenance_t = mapbox::util::
-        variant<user, keyboard_input, mouse_input, file_open, external_access>;
+    using provenance_t = mapbox::util::variant<user,
+                                               format_rewrite,
+                                               keyboard_input,
+                                               mouse_input,
+                                               file_open,
+                                               external_access>;
 
     struct provenance_guard {
         explicit provenance_guard(exec_context* context, provenance_t prov)

src/help_text.hh

@@ -86,6 +86,7 @@ enum class help_parameter_format_t {
     HPF_ENABLED_FILTERS,
     HPF_DISABLED_FILTERS,
     HPF_HIGHLIGHTS,
+    HPF_HIGHLIGHTED_FIELD,
     HPF_TIMEZONE,
     HPF_FILE_WITH_ZONE,
     HPF_CONFIG_PATH,

src/highlighter.cc

@@ -66,11 +66,13 @@ highlighter::annotate_capture(attr_line_t& al, const line_range& lr) const
     }
 }
 
-void
+bool
 highlighter::annotate(attr_line_t& al, const line_range& lr) const
 {
+    auto retval = false;
+
     if (!this->h_regex) {
-        return;
+        return retval;
     }
 
     auto& vc = view_colors::singleton();
@@ -82,11 +84,12 @@ highlighter::annotate(attr_line_t& al, const line_range& lr) const
         std::min(size_t{8192}, std::min((size_t) lr.lr_end, str.size())));
 
     if (!sf.is_valid()) {
-        return;
+        return retval;
     }
 
     this->h_regex->capture_from(sf).for_each<PCRE2_NO_UTF_CHECK>(
         [&](lnav::pcre2pp::match_data& md) {
+            retval = true;
             if (md.get_count() == 1) {
                 this->annotate_capture(al, to_line_range(md[0].value()));
             } else {
@@ -114,4 +117,6 @@ highlighter::annotate(attr_line_t& al, const line_range& lr) const
                 }
             }
         });
+
+    return retval;
 }

src/highlighter.hh

@@ -34,14 +34,15 @@
 
 #include <memory>
 #include <string>
+#include <utility>
 
 #include "base/attr_line.hh"
 #include "base/intern_string.hh"
+#include "base/line_range.hh"
 #include "base/map_util.hh"
 #include "base/string_attr_type.hh"
 #include "base/text_format_enum.hh"
 #include "pcrepp/pcre2pp_fwd.hh"
-#include "styling.hh"
 
 struct highlighter {
     highlighter() = default;
@@ -53,6 +54,12 @@ struct highlighter {
 
     virtual ~highlighter() = default;
 
+    highlighter& with_field(intern_string_t field)
+    {
+        this->h_field = field;
+        return *this;
+    }
+
     highlighter& with_role(role_t role)
     {
         this->h_role = role;
@@ -74,13 +81,6 @@ struct highlighter {
         return *this;
     }
 
-    highlighter& with_format_name(intern_string_t name)
-    {
-        this->h_format_name = name;
-
-        return *this;
-    }
-
     highlighter& with_name(std::string name)
     {
         this->h_name = std::move(name);
@@ -93,9 +93,15 @@ struct highlighter {
         return *this;
     }
 
+    highlighter& with_preview(bool val)
+    {
+        this->h_preview = val;
+        return *this;
+    }
+
     text_attrs get_attrs() const { return this->h_attrs; }
 
-    void annotate(attr_line_t& al, const line_range& lr) const;
+    bool annotate(attr_line_t& al, const line_range& lr) const;
 
     void annotate_capture(attr_line_t& al, const line_range& lr) const;
 
@@ -106,12 +112,13 @@ struct highlighter {
     }
 
     std::string h_name;
+    intern_string_t h_field;
     role_t h_role{role_t::VCR_NONE};
     std::shared_ptr<lnav::pcre2pp::code> h_regex;
     text_attrs h_attrs;
     lnav::set::small<text_format_t> h_text_formats;
-    intern_string_t h_format_name;
     bool h_nestable{true};
+    bool h_preview{false};
 };
 
 #endif

src/internals/cmd-ref.rst

@@ -233,6 +233,29 @@
 ----
 
 
+.. _clear_highlight_field:
+
+:clear-highlight-field *field*
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Remove a field highlight
+
+  **Parameters**
+    * **field\*** --- The name of highlighted field
+
+  **Examples**
+    To clear the highlights for the 'sc_status' field:
+
+    .. code-block::  lnav
+
+      :clear-highlight-field sc_status
+
+  **See Also**
+    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`highlight_field`, :ref:`set_text_view_mode`
+
+----
+
+
 .. _clear_mark_expr:
 
 :clear-mark-expr
@@ -523,7 +546,7 @@
   Disable word-wrapping for the current view
 
   **See Also**
-    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`highlight`, :ref:`set_text_view_mode`
+    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`set_text_view_mode`
 
 ----
 
@@ -583,7 +606,7 @@
   Enable word-wrapping for the current view
 
   **See Also**
-    :ref:`disable_word_wrap`, :ref:`hide_fields`, :ref:`highlight`, :ref:`set_text_view_mode`
+    :ref:`disable_word_wrap`, :ref:`hide_fields`, :ref:`set_text_view_mode`
 
 ----
 
@@ -811,7 +834,7 @@
       :hide-fields syslog_log.log_procname
 
   **See Also**
-    :ref:`enable_word_wrap`, :ref:`highlight`, :ref:`set_text_view_mode`, :ref:`show_fields`
+    :ref:`enable_word_wrap`, :ref:`set_text_view_mode`, :ref:`show_fields`
 
 ----
 
@@ -924,6 +947,36 @@
 ----
 
 
+.. _highlight_field:
+
+:highlight-field *\[--color\]* *\[--bold\]* *\[--underline\]* *\[--italic\]* *\[--strike\]* *\[--blink\]* *field* *pattern*
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+  Highlight a field that matches the given pattern
+
+  **Parameters**
+    * **--color** --- The foreground color to apply
+    * **--bold** --- Make the text bold
+    * **--underline** --- Underline the text
+    * **--italic** --- Italicize the text
+    * **--strike** --- Strikethrough the text
+    * **--blink** --- Make the text blink
+    * **field\*** --- The name of the field to highlight
+    * **pattern\*** --- The regular expression to match
+
+  **Examples**
+    To color status values that start with '2' green:
+
+    .. code-block::  lnav
+
+      :highlight-field --color=green sc_status ^2.*
+
+  **See Also**
+    :ref:`clear_highlight_field`, :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`set_text_view_mode`
+
+----
+
+
 .. _load_session:
 
 :load-session
@@ -1401,7 +1454,7 @@
     * **mode\*** --- The display mode
 
   **See Also**
-    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`highlight`
+    :ref:`enable_word_wrap`, :ref:`hide_fields`
 
 ----
 
@@ -1441,7 +1494,7 @@
       :show-fields log_procname
 
   **See Also**
-    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`highlight`, :ref:`set_text_view_mode`
+    :ref:`enable_word_wrap`, :ref:`hide_fields`, :ref:`set_text_view_mode`
 
 ----