feat: OpenAPI integration #984

NexVeridian · 2024-11-13T04:06:49Z

related #855

cargo check -F all_openapi
cargo nextest run --test-threads 1 -F all_openapi

docker run -d \
  -e POSTGRES_DB=postgres_test \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  --health-cmd="pg_isready" \
  --health-interval=10s \
  --health-timeout=5s \
  --health-retries=5 \
  postgres

export DATABASE_URL="postgresql://postgres:postgres@localhost:5432/postgres_test"

cargo nextest run --all-features --workspace --exclude loco-gen --exclude loco --test-threads 1

TODO:

cc @DenuxPlays

DenuxPlays · 2024-11-13T08:24:55Z

Looks like a good start.

I think the yaml also should contain a few other things:

openapi and ui endpoints (I am not sure if this is want you mean in your description)
whether it serves json, yaml or both (Perhaps this can be controlled by using features to exclude unnecessary dependencies)

cfg for tests

DenuxPlays · 2024-11-26T15:17:44Z

Hey @NexVeridian

Can I help you finish this pr?

NexVeridian · 2024-11-27T04:37:54Z

@DenuxPlays yeah of course

DenuxPlays · 2024-11-27T07:09:48Z

How can I help?

Also just one Note to your pr description.
I wouldn't wrap the utoipa path macro.
Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

NexVeridian · 2024-11-27T08:24:27Z

Also just one Note to your pr description. I wouldn't wrap the utoipa path macro. Its working very well just how it is and wrapping it would make upgrading and maintaining very difficult

thanks!

this would be nice if possible:

impl Routes {
/// .add_openapi(routes!(get_action, post_action))
pub fn add_openapi(mut self, method: UtoipaMethodRouter<AppContext>) -> Self {}
}

any of the unchecked ones in the pr description would be great:
maybe cargo loco generate controller --openapi and docs first

…ms)" This reverts commit 9a74966. Revert "some derive(ToSchema) and derive(IntoParams)" This reverts commit fb9e237.

NexVeridian · 2025-01-30T17:46:37Z

5f4c63f still has some errors that I don't know how to fix, one of you two should take a look maybe

DenuxPlays · 2025-01-30T20:07:28Z

I think the Solution would be something like cfg_of and we have to define the struct two times.

SorenEdwards · 2025-01-31T01:14:32Z

I think this would work, I tested it on your branch. The cfg_attr for the trait bounds is pretty ugly so I think @DenuxPlays has a point with using cfg_of and having two structs.

Pragnation.rs


#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::ToSchema)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct Pager<
#[cfg(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
))] T: utoipa::ToSchema,
#[cfg(not(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
),))] T,
> {
    #[serde(rename(serialize = "results"))]
    pub results: T,

    #[serde(rename(serialize = "pagination"))]
    pub info: PagerMeta,
}


#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::ToSchema)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct PagerMeta {
    #[serde(rename(serialize = "page"))]
    pub page: u64,
    #[serde(rename(serialize = "page_size"))]
    pub page_size: u64,
    #[serde(rename(serialize = "total_pages"))]
    pub total_pages: u64,
    #[serde(rename(serialize = "total_items"))]
    pub total_items: u64,
}

impl<
#[cfg(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
))] T: utoipa::ToSchema,
#[cfg(not(any(
    feature = "openapi_swagger",
    feature = "openapi_redoc",
    feature = "openapi_scalar"
),))] T,
> Pager<T> {
    #[must_use]
    pub const fn new(results: T, meta: PagerMeta) -> Self {
        Self {
            results,
            info: meta,
        }
    }
}

pragnate/mod.rs

#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::IntoParams)
)]
#[derive(Debug, Deserialize, Serialize)]
pub struct PaginationQuery {
    #[serde(
        default = "default_page_size",
        rename = "page_size",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page_size: u64,
    #[serde(
        default = "default_page",
        rename = "page",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page: u64,
}

DenuxPlays · 2025-01-31T06:32:36Z

I had something like the following in mind.
Also this isn't tested I do not have time for this atm.

pagination.rs

use cfg_if::cfg_if;

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        #[derive(Debug, serde::Deserialize, serde::Serialize, utoipa::ToSchema)]
        pub struct Pager<T: utoipa::ToSchema> {
            #[serde(rename(serialize = "results"))]
            pub results: T,

            #[serde(rename(serialize = "pagination"))]
            pub info: PagerMeta,
        }
    } else {
        #[derive(Debug, serde::Deserialize, serde::Serialize)]
        pub struct Pager<T> {
            #[serde(rename(serialize = "results"))]
            pub results: T,

            #[serde(rename(serialize = "pagination"))]
            pub info: PagerMeta,
        }
    }
}

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        #[derive(Debug, serde::Deserialize, serde::Serialize, utoipa::ToSchema)]
        pub struct PagerMeta {
            #[serde(rename(serialize = "page"))]
            pub page: u64,
            #[serde(rename(serialize = "page_size"))]
            pub page_size: u64,
            #[serde(rename(serialize = "total_pages"))]
            pub total_pages: u64,
            #[serde(rename(serialize = "total_items"))]
            pub total_items: u64,
        }
    } else {
        #[derive(Debug, serde::Deserialize, serde::Serialize)]
        pub struct PagerMeta {
            #[serde(rename(serialize = "page"))]
            pub page: u64,
            #[serde(rename(serialize = "page_size"))]
            pub page_size: u64,
            #[serde(rename(serialize = "total_pages"))]
            pub total_pages: u64,
            #[serde(rename(serialize = "total_items"))]
            pub total_items: u64,
        }
    }
}

cfg_if! {
    if #[cfg(any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ))] {
        impl<T: utoipa::ToSchema> Pager<T> {
            #[must_use]
            pub const fn new(results: T, meta: PagerMeta) -> Self {
                Self {
                    results,
                    info: meta,
                }
            }
        }
    } else {
        impl<T> Pager<T> {
            #[must_use]
            pub const fn new(results: T, meta: PagerMeta) -> Self {
                Self {
                    results,
                    info: meta,
                }
            }
        }
    }
}

I think that @SorenEdwards implementation for the query is good but we should also add ToSchema to it.
It may be useful for some use cases.
So this is what I would do:

#[cfg_attr(
    any(
        feature = "openapi_swagger",
        feature = "openapi_redoc",
        feature = "openapi_scalar"
    ),
    derive(utoipa::IntoParams, utoipa::ToSchema)
)]
#[derive(Debug, serde::Deserialize, serde::Serialize)]
pub struct PaginationQuery {
    #[serde(
        default = "default_page_size",
        rename = "page_size",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page_size: u64,
    #[serde(
        default = "default_page",
        rename = "page",
        deserialize_with = "deserialize_pagination_filter"
    )]
    pub page: u64,
}

kaplanelad · 2025-02-03T13:33:23Z

After reviewing this implementation again, I feel we should consider moving it into an initializer maintained in a separate repository. For example:

Could you outline the changes needed in Loco to allow managing OpenAPI as an initializer?

DenuxPlays · 2025-02-03T13:41:27Z

Note:

I don't think thats possible.
we need loco to implement certain traits for this to work.
examples are the above mentioned Pager and PagerMeta types

kaplanelad · 2025-02-03T18:04:49Z

Note:

I don't think thats possible. we need loco to implement certain traits for this to work. examples are the above mentioned Pager and PagerMeta types

Everything is possible. When initializing is public create, you need to load Loco (see the other implementation), which allows us to make the dependent struct public.
I want to count the changes that we need to make in the framework to make it happen

SorenEdwards · 2025-02-04T03:58:12Z

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

DenuxPlays · 2025-02-04T05:47:04Z

Note:

I don't think thats possible. we need loco to implement certain traits for this to work. examples are the above mentioned Pager and PagerMeta types

Everything is possible. When initializing is public create, you need to load Loco (see the other implementation), which allows us to make the dependent struct public.

I want to count the changes that we need to make in the framework to make it happen

I am not sure what you mean.
AFAIK traits can only be implemented on internal, your own types or if you created the trait.

So unless you copy the structs it isnt possible right?

DenuxPlays · 2025-02-04T05:52:10Z

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

Yeah I also use it as an initializer.
But I don't think that Code Generation is my Main aspect.
Currently I use utoipauto to collect all the paths, schemas etc. But this would actually use the utoipa-Axum Router which is far cleaner then using utoipauto.
Also you do not have to copy types like Pager.

It is just a Smoother experience.

Just a small note you should Never Serve entities over your api.
Just use DTOs/Views for this.
Ofcourse I do Not know your project and it may be not the best Solution for your I just wanted to mention it.

kaplanelad · 2025-02-04T06:37:59Z

@kaplanelad @DenuxPlays @NexVeridian I am actually using it in the initializer right now on my project/work. If you are interested I can add a post about it.

I think the real upside of adding a feature in loco would be the code generation aspect as I do a lot of this by hand currently and it can get pretty tedious. If you have any suggestions I would love to help out making the initializer approach work.

That being said seaorms --extra-model-derive could fix ToSchema not being added to the generated db entities.

Yes, it can be great. can you also share how you implement it with an example repo?

DenuxPlays · 2025-02-04T06:51:09Z

You can take look here:
https://github.com/financrr/financrr-app/blob/backend/feature/sync-bank-data/backend/src/initializers/openapi.rs

kaplanelad · 2025-03-10T11:13:33Z

After reviewing this implementation again, I feel we should consider moving it into an initializer maintained in a separate repository. For example:

loco-oauth2

rhai-loco

loco-extras

Could you outline the changes needed in Loco to allow managing OpenAPI as an initializer?

@NexVeridian ?

NexVeridian · 2025-03-10T16:51:04Z

@kaplanelad

There are some issues with some of the options, like creating code generation in the future might be harder, here are some options for converting this pr to a initializer:

option 1 - probably not possible

Get the original function from a axum::routing::MethodRouter or use axum::routing::MethodRouter in utopia

option 2

have the user manually add the data to the initializer, similar to #855

# controllers::auth
pub fn api_routes() -> OpenApiRouter<AppContext> {
    OpenApiRouter::new().routes(routes!(register, verify, login, forgot, reset))
}

then pass controllers::auth::api_routes and others to the initializer

 let (_, api) = OpenApiRouter::with_openapi(ApiDoc::openapi())
            .merge(controllers::auth::api_routes())
            .merge(controllers::responses::api_routes())
            .split_for_parts();

option 3

Merge these two into loco:

controller/routes.rs - LocoMethodRouter
- so the user would call Routes::new().add("/_ping", routes!(ping));

loco/src/controller/app_routes.rs

Lines 218 to 237 in 5f4c63f

    
                   let mut api_router: OpenApiRouter<AppContext> = 
        
                       OpenApiRouter::with_openapi(H::inital_openapi_spec(&ctx)); 
        
                   for router in self.collect() { 
        
                       tracing::info!("{}", router.to_string()); 
        
                       match router.method { 
        
                           LocoMethodRouter::Axum(method) => { 
        
                               app = app.route(&router.uri, method); 
        
                           } 
        
                           #[cfg(any( 
        
                               feature = "openapi_swagger", 
        
                               feature = "openapi_redoc", 
        
                               feature = "openapi_scalar" 
        
                           ))] 
        
                           LocoMethodRouter::Utoipa(method) => { 
        
                               app = app.route(&router.uri, method.2.clone()); 
        
                               api_router = api_router.routes(method.with_state(ctx.clone())); 
        
                           } 
        
                       } 
        
                   }

then here you would pass api_router or vec!method.with_state(ctx.clone()) to the initializer

option 4

This comment is also a good option but has these drawbacks, as stated in the comment:

Yeah I also use it as an initializer.
But I don't think that Code Generation is my Main aspect.
Currently I use utoipauto to collect all the paths, schemas etc. But this would actually use the utoipa-Axum Router which is far cleaner then using utoipauto.

kaplanelad · 2025-03-17T08:43:41Z

All the options look legitimate to me when we move them to the initializer.
I can create a project under Loco, and we can start working on it together.

WDYT?

DenuxPlays · 2025-03-17T09:19:12Z

What do you mean?

We cannot convert this feature to an initializer at least not without dropping 50% of the features.

kaplanelad · 2025-03-17T09:28:34Z

Could you please clarify which features are going to be supported and which ones won’t,and why?

I’d prefer to support this feature as an initializer (that is supported by Loco teams) and keep them separate from the base Loco codebase.

DenuxPlays · 2025-03-17T10:01:24Z

automatic Schema collection (includign routes, responses etc.)
Because internally Loco would still use the axum router instead of the utoipa-axum router.
Pagination etc. cannot work with utoipa as they do not derive the Schema, Response traits etc.
There is a work around that you just copy the code but tbh. copy code is always bad

There are probably some other things I forgot but the integration wouldn't feel like an integration

kaplanelad · 2025-03-17T10:09:42Z

automatic Schema collection (includign routes, responses etc.)
Because internally Loco would still use the axum router instead of the utoipa-axum router.

Why not make the internal Loco route responses public so they can be accessed externally? This way, the initializer can retrieve the app context, which provides access to everything required by OpenAPI.

Pagination etc. cannot work with utoipa as they do not derive the Schema, Response traits etc.
There is a work around that you just copy the code but tbh. copy code is always bad

There are probably some other things I forgot but the integration wouldn't feel like an integration

Same for here, we can expose it

DenuxPlays · 2025-03-17T10:39:30Z

I am not sure what you mean.

How can you "expose" something that needs to replaced when you want to use it with utoipa?

Also the struct is already exposed but the derive(Schema) is missing.
How is this fixable?

NexVeridian · 2025-03-17T21:07:24Z

keep them separate from the base Loco codebase

option 2:

this option removes automatic schema collection
and is the easiest to implement
and has almost no code that needs to merged into loco, just src/controller/format.rs and some derive(utoipa::ToSchema)

option 3:

this option has automatic schema collection
but is harder to implement
and a lot of the code in this pr will need to be merged

the only things that can be extracted is: the config, the tests, and this section app_routes.rs#L239-L290
and the LocoMethodRouter and app_routes.rs would need to stay

option 1:

Why not make the internal Loco route responses public so they can be accessed externally? This way, the initializer can retrieve the app context, which provides access to everything required by OpenAPI.

this is talking about option 1, which is probably not possible or I can't figure it out, maybe if you do something that looks bad like this:Routes::new().add("/_ping", "get", ping);

jcorrie · 2025-04-01T06:23:53Z

I was very hopefully following this thread a couple of weeks ago - I think this would be an incredible feature for a loco to implement out of the box. (Auto /docs was always a driving reason for me to choose fastapi in the python ecosystem.)

Not sure how qualified I am, but would be more than happy to give this a stab if needed.

NexVeridian · 2025-04-11T20:02:53Z

@kaplanelad
I've decided to go with option 3

The initializer is working, with automatic schema collection!
you can find the pr for the initializer here

some things from this pr still need to get merged into loco
you can find the tracking issue for those prs here #1363

NexVeridian force-pushed the OpenAPI-integration branch 4 times, most recently from 9055d99 to 3e106d0 Compare November 22, 2024 09:21

NexVeridian added 17 commits November 23, 2024 04:38

inti

8ed9348

hook initial_openapi_spec

6deca2b

config.yaml and merge hosted doc with router

def3dd5

update existing tests

8db4443

test: OpenAPI config from file

34d10e6

snapshot tests

c887042

fix snapshot path

4ed2668

match title, upload snapshots

2aa3d66

OpenAPI json snapshot test

0a304c1

Another snapshot

8c0fd96

LocoMethodRouter

5f63b08

fix AppContext

1c67cc6

missing cfg for tests

c1e953e

cfg for tests

clippy

74f78cc

openapi.josn and openapi.yaml endpoints for all types

ac0f1ce

SecurityAddon

2082e12

fix cfg flag for import

ea48919

NexVeridian force-pushed the OpenAPI-integration branch from 3e106d0 to ea48919 Compare November 23, 2024 14:13

NexVeridian added 2 commits November 26, 2024 19:50

Merge branch 'upstream' into OpenAPI-integration

c849c73

fix: snapshots and imports

022909d

Merge branch 'master' into OpenAPI-integration

165d162

NexVeridian added 2 commits January 30, 2025 09:24

more ToSchema

9a74966

Revert "more ToSchema" and "some derive(ToSchema) and derive(IntoPara…

5f4c63f

…ms)" This reverts commit 9a74966. Revert "some derive(ToSchema) and derive(IntoParams)" This reverts commit fb9e237.

kaplanelad mentioned this pull request Mar 9, 2025

Cool suggestion Strong openapi support #1292

Closed

NexVeridian mentioned this pull request Apr 10, 2025

feat: yaml responses #1360

Merged

NexVeridian closed this Apr 11, 2025

NexVeridian deleted the OpenAPI-integration branch April 28, 2025 03:40

NexVeridian mentioned this pull request Oct 22, 2025

Add automatic creation of open api documentation through the aide crate #1672

Open

Uh oh!

feat: OpenAPI integration #984

feat: OpenAPI integration #984

Uh oh!

Conversation

NexVeridian commented Nov 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO:

Uh oh!

DenuxPlays commented Nov 13, 2024

Uh oh!

DenuxPlays commented Nov 26, 2024

Uh oh!

NexVeridian commented Nov 27, 2024

Uh oh!

DenuxPlays commented Nov 27, 2024

Uh oh!

NexVeridian commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NexVeridian commented Jan 30, 2025

Uh oh!

DenuxPlays commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SorenEdwards commented Jan 31, 2025

Pragnation.rs

pragnate/mod.rs

Uh oh!

DenuxPlays commented Jan 31, 2025

Uh oh!

kaplanelad commented Feb 3, 2025

Uh oh!

DenuxPlays commented Feb 3, 2025

Uh oh!

kaplanelad commented Feb 3, 2025

Uh oh!

SorenEdwards commented Feb 4, 2025

Uh oh!

DenuxPlays commented Feb 4, 2025

Uh oh!

DenuxPlays commented Feb 4, 2025

Uh oh!

kaplanelad commented Feb 4, 2025

Uh oh!

DenuxPlays commented Feb 4, 2025

Uh oh!

kaplanelad commented Mar 10, 2025

Uh oh!

NexVeridian commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

option 1 - probably not possible

option 2

option 3

option 4

Uh oh!

kaplanelad commented Mar 17, 2025

Uh oh!

DenuxPlays commented Mar 17, 2025

Uh oh!

kaplanelad commented Mar 17, 2025

Uh oh!

DenuxPlays commented Mar 17, 2025

Uh oh!

kaplanelad commented Mar 17, 2025

Uh oh!

DenuxPlays commented Mar 17, 2025

Uh oh!

NexVeridian commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

option 2:

option 3:

option 1:

Uh oh!

jcorrie commented Apr 1, 2025

Uh oh!

NexVeridian commented Apr 11, 2025

Uh oh!

Reviewers

Assignees

NexVeridian commented Nov 13, 2024 •

edited

Loading

NexVeridian commented Nov 27, 2024 •

edited

Loading

DenuxPlays commented Jan 30, 2025 •

edited

Loading

NexVeridian commented Mar 10, 2025 •

edited

Loading

NexVeridian commented Mar 17, 2025 •

edited

Loading