Lexer: Bugfixes & Enhancements

[Clipi-12]

Changes in the first commit

Changes to stb_c_lexer.h

• Removed "STB_C_LEX_DEFINE_ALL_TOKEN_NAMES", since tokens have been always defined since this commit.
• Defined "STB__clex_identifiers" when "STB_C_LEX_C_IDENTIFIERS==Y". I then used #ifdef directives to avoid creating CLEX_id tokens when users don't want to. In that case, the user will receive a raw character, as if it was a symbol. Unicode characters will be split, but I think that's expected in this case.
• "STB_C_LEX_OCTAL_SUFFIXES" was wrongly being used instead of "STB_C_LEX_DECIMAL_SUFFIXES".
• Re-implemented warning suppression for "stb__clex_parse_suffixes" (and added it to argc and argv of the main() in "STB_C_LEXER_SELF_TEST") via casting to void (i.e. replaced the previous warning suppression with (void) suffixes;).
• Fixed some compilation bugs with "STB_C_LEX_ISWHITE". I later found out that they were also mentioned in I updated stale code in STB_C_LEX_ISWHITE #1426.
• Freed memory in the main() of "STB_C_LEXER_SELF_TEST" so that valgrind would stop complaining.
• Change exit-code -> 1 if any CLEX_parse_error is found during the "STB_C_LEXER_SELF_TEST".
• Some comments were wrong/outdated. For example the macro "STB_C_LEX_C_SHIFTS" was wronly referenced as "STB_C_LEX_SHIFTS", while "STB_C_LEX_USE_STDLIB" was referenced as "STB_C_LEX_STDLIB".
• Added comments at the end of some directives to improve readability (e.g. #endif // STB_C_LEXER_SELF_TEST)
• Added comments (TODOs) in the definition of "STB_C_LEX_MULTILINE_DSTRINGS" and "STB_C_LEX_MULTILINE_SSTRINGS" to show users that these are not implemented yet (as stated in the "Status" section).

Changes to tests/c_lexer_test.c

• Changed #define STB_C_LEX_FLOAT_SUFFIXES "uulL" to #define STB_C_LEX_FLOAT_SUFFIXES "uUlL" (this was probably a bug).
• Added the same comment-changes and definition-removals as in stb_c_lexer.h.
• Added "STB_C_LEX_ISWHITE" comment to display to have all the configurations displayed.

Changes to tests/stb_c_lexer_fuzzer.cpp

• Same definition-removal change.
• Same "uulL" -> "uUlL" change.
• Removed empty whitespaces at the end of some lines.

Changes to tests/test_c_lexer.c

• Same comment-changes and definition-removals as in stb_c_lexer.h.

Changes to tools/make_readme.c

• #include "../stb.h" -> #include "../deprecated/stb.h"

Changes in other commits

These other commits are not that important, so feel free to remove them from the PR if you feel so inclined.

• Reordering of "STB_C_LEXER_IMPLEMENTATION" definitions (e.g. "STB__clex_decimal_ints" before "STB__clex_hex_ints", the same way it is defined in "STB_C_LEXER_DEFINITIONS").

Testing

Tested the code with the built-in test of stb_c_lexer.h. I ran it with multiple compilers, on Windows 10 and WSL2-Ubuntu. This is the testing script i used (in Windows10 i used the Git Bash):

#! /bin/bash
set -euo pipefail

if [[ $@ == "-v" || $@ == "--verbose" ]]; then set -x; fi;

if which valgrind >/dev/null 2>&1; then
	function run() {
		# Just some standard flags I normally use with valgrind
		valgrind \
			--leak-check=full --show-leak-kinds=all \
			--track-origins=yes --track-fds=yes \
			--quiet "$@"
	}
else
	function run() {
		"$@"
	}
fi

function test_lexer() {
	if which "$1" >/dev/null 2>&1; then
		# -O3 because clang may find undefined behaviour when optimizing (-O2 before for MSCV)
		# Allow deprecations (fopen on Windows. only used in the main() of STB_C_LEXER_SELF_TEST)
		"$1" -Wall -Wextra -O2 -O3 \
			-Wno-deprecated-declarations -Wno-unused-command-line-argument \
			-DSTB_C_LEXER_SELF_TEST -DSTB_C_LEXER_IMPLEMENTATION \
			-x c stb_c_lexer.h -o ./out
		echo "Successful compilation with $1"
		run ./out >/dev/null
		echo "Successful execution with $1"
	else
		echo "Command $1 not found. Ignoring..."
	fi
	echo
}
test_lexer gcc
test_lexer g++
test_lexer clang
test_lexer clang++
test_lexer clang-cl
test_lexer cl # Found on the PATH if GitBash is launched from the Developer Comand Prompt

The PR allows edits by maintainers, so feel free to modify anything.