Refactor Parser

[blattm]

Hi!
This PR aims to improve the readability of main.c by separating parsing logic from message handling.
Although the code is already separated, it's not perfect, and there is still work to be done. There are, for example, too many variables being passed in function calls for my taste.

I'm planning to write a read_byte function that the parser can call. If would fetch data from the network when the buffer is empty. This would allow us to eliminate the switch / case and make many variables locally scoped. I would use error codes to detect whether a reset (corresponding to the outermost while loop of the ´http_get_task`) is necessary.

It might also make sense to further modularize the message-handling side (e.g., splitting out time sync, codecs, etc.), but that can easily be done in a later PR.

[CarlosDerSeher]

You could put all those variables passed to the functions in a struct/union combination.

[blattm]

Yes! I will do this, once the parser is simplified. I think I will be able to remove a lot of parameters corresponding to internal parser state first.

[CarlosDerSeher]

Before you continue your effort maybe we should check if netconn api is really the best thing to use. There have been some packet delivery issues on my last tests and I am uncertain where this comes from. Also espressif says netconn is less resource intensive but when asking chatGPT I get this for an answer

Why does espressif say in it's documentation netconn is less resource intensive than bsd sockets as those are built on top of netconn in lwip

**ChatGPT:**
Great question — and the confusion is completely justified.
The apparent contradiction comes from how lwIP was originally designed versus how ESP-IDF actually implements BSD sockets today.

Below is the precise reason why Espressif says “netconn is less resource-intensive than BSD sockets”, and why this statement is both true in theory and false in practice depending on context.

✅ 1. In upstream lwIP, BSD sockets are layered on top of netconn
In vanilla lwIP (the version without Espressif modifications):

sockets API → netconn API → raw lwIP (PCB)
Therefore:

A BSD socket contains a struct lwip_sock

That struct contains a pointer to a struct netconn

The netconn contains mailboxes, semaphores, and state

→ So sockets add one extra layer of abstraction on top of netconn

In pure lwIP, this means:

BSD sockets = netconn + extra wrapper
→ So yes, sockets use more RAM than netconn.

This is the context Espressif is referencing.

❗ 2. BUT ESP-IDF modifies lwIP — and the layering is not identical anymore
Espressif has patched lwIP to integrate:

esp_netif

TCP/IP task changes

Optimized memory pools

mbedTLS handling

Event loop integration

In ESP-IDF:

✔ The sockets layer still uses netconn
BUT
✔ Espressif optimized out some netconn overhead for the socket layer
✔ And the socket layer has its own optimizations

In other words:

BSD sockets in ESP-IDF do NOT incur all of the overhead that upstream lwIP documented.

This is why users notice that:

BSD sockets behave more predictably

Netconn often uses more RAM for real-world workloads

Netconn copies data more often

Socket recv() can reuse a single buffer while netconn buffers queue up pbufs/netbufs

Espressif’s documentation is talking about theoretical upstream lwIP architecture, not actual ESP-IDF runtime memory usage.

So I am abit confused now and I think a comparison of the two should be done by logging max heap usage while reception. Maybe we can integrate this into http_task and just drop all the data and do the same using sockets

[blattm]

I’ve significantly reduced the number of parameters passed to process_data. Many of the previous parameters were only used to forward specific parsing results to the handling functions.

These values now live in much smaller scopes: either directly inside process_data or within the parsing functions where they logically belong.

Strongly reducing the parameter count further would likely mean grouping values with little semantic connection into a shared struct. I’m not sure that would improve readability or maintainability. But I may also be missing what kind of refactoring you have in mind. If you’d like me to continue in this direction, a concrete example of the struct or grouping you’re thinking of would really help.

Let me know if the current state works for you. If so, I’ll resolve the conflicts and get things wrapped up 🙂

[CarlosDerSeher]

Thank you. Hopefully I'll find some time to review soon. Maybe someone else will jump in sooner?

[blattm]

I just had a look at the conflicts and it's ... a lot. main.c on the develop branch had been edited in the meantime, including some refactoring. It looks like it could take multiple hours to fix everything and I'm not quite sure about the best strategy yet.

Maybe it makes sense to postpone the review until the conflicts are resolved? I'm not sure. But I guess I'll already start working toward resolving the conflicts, because this will only get worse the more develop changes over time.

[blattm]

I finally managed to resolve all of the conflicts! 🎉
To achieve this, I had to merge develop's new commits one by one. This makes the commit history a little cluttered, but I guess we are going to squash this anyway.
I'd say it's ready for review now! I hope we can get this done before develop changes again 😅

[luar123]

I had no time to look into details, but I like the structure a lot and a quick test was running fine.

@CarlosDerSeher I think it would be great if we could move forward because all the PRs or planned PRs would benefit from not modifying the same main.c file. #174 wants to wait for this, I also wait with further work. And #195 again adds a lot of modifications, that would otherwise go the the connection_manager I guess.

[CarlosDerSeher]

Oh didn't realize this was ready :) nice

[CarlosDerSeher]

Great, and just today I merged some other stuff :( hopefully this isn't so much work to resolve

[blattm]

Great, and just today I merged some other stuff :( hopefully this isn't so much work to resolve

It's resolved again :)