[TRACKING] sys/shell refactoring.

[jcarrano]

Description

This is an issue to track enhancements and bugfixes and refactoring of the shell. First bugs must be fixed, then the refactoring should take place.

Bugs:

• sys/shell: correctly detect and handle long lines #10635 : fix bug caused by long lines

Enhancements:

• sys/shell: cancel current line on CTRL-C. #11004 : cancel current line with ctrl-C
• sys/shell: Make echoing and prompt runtime-configurable. #12087 : allow the prompt and echoing to be configure at run-time using special commands.

Refactoring

• sys/shell: refactor tokenizer code. #12085 : rewrite the tokenizer (the part that splits the words in the command line). This is also a bug fix but the original code was way too messy so I just rewrote it.
• sys/shell: refactor readline function. #12099 : similar change to the readline (the part that reads from stdin into an array).

Other:

• sys/shell: Exit the shell on ctrl-D #10788 : Exit the shell on control-D.This one is contested and I'm not sure it is desirable. Actually, ctrl-D is normally handled at another OS layer (which does not exist in RIOT).

Related:

• [TRACKING] buildsystem: introduce "make rawterm" #11099 : The raw-terminal is needed if we want to properly test the shell.
• sys/optparse: Add command line parser. #9538: argument parser code. This goes after the shell, it would make shell handlers easier to write and less bug-prone.