From 2d051d809cd8a73d4a929af263be9d9af73c6752 Mon Sep 17 00:00:00 2001 From: shannonbux <32467162+shannonbux@users.noreply.github.com> Date: Thu, 25 Apr 2019 15:52:07 -0700 Subject: [PATCH 1/4] created rfc error survey draft @KyleAMathews helped a lot with this, will appreciate feedback from @jamo and many more from the OSS community! --- text/000-error-survey.md | 82 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 text/000-error-survey.md diff --git a/text/000-error-survey.md b/text/000-error-survey.md new file mode 100644 index 0000000..13fe36d --- /dev/null +++ b/text/000-error-survey.md @@ -0,0 +1,82 @@ +- Start Date: 2019-04-25 +- RFC PR: (leave this empty) +- Gatsby Issue: (leave this empty) + +# Summary + +We want to survey error messages and see where we can improve them. + +# Basic example + +If the proposal involves a new or changed API, include a basic code example. +Omit this section if it's not applicable. + +# Motivation + +Part of starting, developing, and producing a Gatsby site is resolving errors. Gatsby’s error messages have inconsistent formatting and don’t always help people resolve the errors. We can improve them! + +# Detailed design + +## Constraints in error message surveying +We probably cannot create and maintain helpful solutions to all error messages; however, we can track the most common errors and improve those that will have the biggest impact on both user's goals and Gatsby's goals. + +1. Survey +- Identify all errors +- Number the errors +- Name each error +- Give each error a grade +2. Generate a daily report +- Most common errors and their grades +- Most common errors that don’t have a number, name, or grade +3. Improve the errors +- Prioritize tasks by most common errors with lowest grades +- For the list of prioritized tasks + - Fix the reason the error is occurring, if possible + - For errors whose messages don’t point to the root cause, identify the root cause of the error and warn or error at the root cause + - Improve error design, including consistent formatting + +## Special note about data model +Beware of paying too much attention to the following numbers. This kind of data could seem negative and could be neutral because they could be part of learning: +- Increase in error messages +- No activity in a site +- Abandoning a project + +This kind of data could seem positive but is actually neutral: +- Lots of activity in a site (it could be a problematic site or be multiple people working on the same site? +- Lots of hot reloading + +# Drawbacks + +[I need help with this section!] +Why should we *not* do this? Please consider: + +- implementation cost, both in term of code size and complexity +- whether the proposed feature can be implemented in user space +- the impact on teaching people Gatsby +- integration of this feature with other existing and planned features +- cost of migrating existing Gatsby applications (is it a breaking change?) + +There are tradeoffs to choosing any path. Attempt to identify them here. + +# Alternatives + +We have looked through quite a few documents on error design and surveyed other CLI’s to report on their error messaging. [RFC CLI Redesign here]() + +The impact of not doing this is that Gatsby users continue to struggle to resolve errors with unhelpful messages and the Github repo serves as a sorting ground for issues dealing with known errors that could be fixed closer to their source. + +# Adoption strategy + +Gatsby users will not need to do anything differently to adopt this except update their version of Gatsby. + +Is this a breaking change? Can we write a codemod? Should we coordinate with +other projects or libraries? + +# How we teach this + +We would publish an error design style guide and an error survey and improvement plan. +The rest will be taught on the fly as developers run into errors and read the new messages. The goal is that they won’t have to do anything differently and will only be served with more helpful error messages (and sometimes, they won’t even run into an error if we fixed a bug). + +# Unresolved questions + +We aren’t sure how many errors are being generated, so we don’t know how many of the most common we can improve each day. We don’t have a baseline velocity to measure our performance against. +Naming conventions for errors are TBD From 7086f92b5efd2e209c1f1f72e89dfd0864b9162a Mon Sep 17 00:00:00 2001 From: shannonbux <32467162+shannonbux@users.noreply.github.com> Date: Fri, 21 Jun 2019 11:59:01 +0200 Subject: [PATCH 2/4] Update 000-error-survey.md --- text/000-error-survey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/000-error-survey.md b/text/000-error-survey.md index 13fe36d..db940ed 100644 --- a/text/000-error-survey.md +++ b/text/000-error-survey.md @@ -60,7 +60,7 @@ There are tradeoffs to choosing any path. Attempt to identify them here. # Alternatives -We have looked through quite a few documents on error design and surveyed other CLI’s to report on their error messaging. [RFC CLI Redesign here]() +We have looked through quite a few documents on error design and surveyed other CLI’s to report on their error messaging. [RFC CLI Redesign here](https://github.com/gatsbyjs/rfcs/pull/38/files) The impact of not doing this is that Gatsby users continue to struggle to resolve errors with unhelpful messages and the Github repo serves as a sorting ground for issues dealing with known errors that could be fixed closer to their source. From 831e1005c3396ae7c03e1a0b83868b86c8aa20e6 Mon Sep 17 00:00:00 2001 From: shannonbux <32467162+shannonbux@users.noreply.github.com> Date: Tue, 18 Feb 2020 14:31:36 -0800 Subject: [PATCH 3/4] Update 000-error-survey.md --- text/000-error-survey.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/text/000-error-survey.md b/text/000-error-survey.md index db940ed..e28f879 100644 --- a/text/000-error-survey.md +++ b/text/000-error-survey.md @@ -6,11 +6,6 @@ We want to survey error messages and see where we can improve them. -# Basic example - -If the proposal involves a new or changed API, include a basic code example. -Omit this section if it's not applicable. - # Motivation Part of starting, developing, and producing a Gatsby site is resolving errors. Gatsby’s error messages have inconsistent formatting and don’t always help people resolve the errors. We can improve them! From ca35b9cf7f26cfe1de8d728570eeb56a19856624 Mon Sep 17 00:00:00 2001 From: shannonbux <32467162+shannonbux@users.noreply.github.com> Date: Tue, 18 Feb 2020 14:42:00 -0800 Subject: [PATCH 4/4] Update 000-error-survey.md --- text/000-error-survey.md | 47 ++++++++++++++++------------------------ 1 file changed, 19 insertions(+), 28 deletions(-) diff --git a/text/000-error-survey.md b/text/000-error-survey.md index e28f879..968b114 100644 --- a/text/000-error-survey.md +++ b/text/000-error-survey.md @@ -15,43 +15,32 @@ Part of starting, developing, and producing a Gatsby site is resolving errors. G ## Constraints in error message surveying We probably cannot create and maintain helpful solutions to all error messages; however, we can track the most common errors and improve those that will have the biggest impact on both user's goals and Gatsby's goals. -1. Survey +### Step 1: Survey errors - Identify all errors - Number the errors - Name each error - Give each error a grade -2. Generate a daily report + +### Step 2: How to give an error a grade +0: Silent error +1: Throw Error and show changed state when it is resolved +2: Error message with id, name, and description and overwrite error listing when it is resolved +3: Error message with id, name, and description + Prompt for fix (can be URL to docs page, code example, example site, etc) and overwrite error listing when it is resolved +4: Error message with id, name, and description + Prompt for fix + Adaptive Prompting and overwrite error listing when it is resolved +5: Same as 4 except error also shows in both terminal AND browser + +### Step 3: Generate a daily report - Most common errors and their grades - Most common errors that don’t have a number, name, or grade -3. Improve the errors + +### Step 4: Improve the errors - Prioritize tasks by most common errors with lowest grades - For the list of prioritized tasks - - Fix the reason the error is occurring, if possible - - For errors whose messages don’t point to the root cause, identify the root cause of the error and warn or error at the root cause - - Improve error design, including consistent formatting - -## Special note about data model -Beware of paying too much attention to the following numbers. This kind of data could seem negative and could be neutral because they could be part of learning: -- Increase in error messages -- No activity in a site -- Abandoning a project - -This kind of data could seem positive but is actually neutral: -- Lots of activity in a site (it could be a problematic site or be multiple people working on the same site? -- Lots of hot reloading + - Fix the reason the error is occurring so it happens less often or not at all, if possible + - For errors that can't be eliminated from happening, improve the grade of the error by at least 1 grade (take a grade 0 error to grade 1, for example) # Drawbacks - -[I need help with this section!] -Why should we *not* do this? Please consider: - - implementation cost, both in term of code size and complexity -- whether the proposed feature can be implemented in user space -- the impact on teaching people Gatsby -- integration of this feature with other existing and planned features -- cost of migrating existing Gatsby applications (is it a breaking change?) - -There are tradeoffs to choosing any path. Attempt to identify them here. # Alternatives @@ -68,10 +57,12 @@ other projects or libraries? # How we teach this -We would publish an error design style guide and an error survey and improvement plan. +We would publish an error design style guide and an error survey and improvement plan. + The rest will be taught on the fly as developers run into errors and read the new messages. The goal is that they won’t have to do anything differently and will only be served with more helpful error messages (and sometimes, they won’t even run into an error if we fixed a bug). # Unresolved questions We aren’t sure how many errors are being generated, so we don’t know how many of the most common we can improve each day. We don’t have a baseline velocity to measure our performance against. -Naming conventions for errors are TBD + +Naming conventions for errors are TBD.