Taking API surface seriously: Proposing Syntax for declaring API

[Seelengrab]

The general goal of this proposal is to formalize what it means to be considered API in julia, to give a framework to talk about what even is a "breaking" change to make it easier for developers to provide more documentation for their packages as well as avoiding breakage in the first place. The reasoning in this document comes from the current practices in use in the ecosystem, as well as type theoretic requirements of the type system, though having a background in type theory is not required to participate in the discussion.

Please do reply if you have something to add/comment on!

Motivation

The current stance of Base and much of the package ecosystem of what is considered to be API under semver is "If it's in the manual, it's API". Other approaches are "If it has a docstring it's API", "If it's exported, it's API", "If it's in the Internal submodule it's API" or "If we explicitly mention it, it's API". These approaches are inconsistent with each other and have a number of issues:

• Discoverability
  • As a user writing code, it is nigh impossible to figure out whether a given function they found being used in some other code is stable under semver or not, unnecessarily increasing the chances for accidental breakage later on.
  • Tooling can't reliably query/show whether some symbol/object is stable API or not, because the inconsistencies between the approaches only lend themselves to bespoke solutions on a package by package basis.
• Maintainability
  • The fuzziness of some of these approaches disincentivizes writing documentation for internal functionality, for fear of others taking the existence of a docstring as guarantor for the function in question being API.
  • This fuzziness and lack of internal documentation increases the "time to first contribution" by newcomers and returning developers as well, due to having to reverse engineer the package to contribute in the first place.
  • "Unofficial" API surfaces grow that are using internals (often because the developers in question wrote the internal in the first place), leading to a sort-of tacit implication that it is ok to use internal functionality of Base/a package.
  • Accidental exposure of an internal as API (even just perceived API!) generally means that removing that internal would necessitate a breaking change - this doesn't scale well, and would quickly lead to not being able to change any code.
  • Due to the fuzziness of what exactly is considered API, deprecations are difficult - @deprecate is hard to use correctly and often not used at all.
• Reviewability
  • It is all too easy to accidentally rely on an internal of another package due to the bad discoverability of what actually is API. This increases the burden on reviewers to check contributions for whether or not the functionality a PR is introducing is relying on/bringing in internal uses, even if they may not be familiar with the dependency in question.
  • The lax stance around what exactly is considered API and the fact that internals rarely get docstrings or even comments explaining what the code does means that contributions don't usually get checked for whether they (accidentally or not) increase the API surface or whether a particular aspect of a contribution or new feature ought to have a documented API in the first place.

This proposal has three parts - first, presenting a full-stack solution to both the Discoverability and Maintainability issues described above, and second, proposing a small list of things that could be done with the proposed feature to make Reviewability easier. Finally, the third part is focused on a (preliminary, nonexhaustive) "How do we get there" list of things required to be implemented for this proposal.

There is also a FAQ list at the end, hoping to anticipate some questions about this proposal that already came up in previous discussions and thoughts about this design.

The api keyword

The main proposal for user-facing interactions with declaring whether an object is the new api keyword. The keyword can be placed in front of definitions in global scope, like struct, abstract type, module, function and const/type annotated global variables. Using api is declaring the intent of a developer, about which parts of the accessible symbols they consider API under semver and plan to support in newer versions.

When a project/environment importing a package wants to access a symbol not marked as API (if it is not the same project/environment originally defining the symbol), a warning is displayed, making the user aware of the unsupported access but doesn't otherwise hinder it. This behavior should be, to an extent, configurable, to support legitimate accesses of internals (insofar those exist). There is the caveat that silencing these warnings makes no difference to whether or not the access is supported by a developer. This is intended to provide an incentive to either use the supported subset of the API, or encourage a user to start a discussion with the developer to provide an API for what they would like to do. The result of such a discussion can be a "No, we won't support this"! This, however, is a far more desirable outcome to accessing internals and fearing breakage later on, if it would have been avoided by such a discussion.

The following sections explain how api interacts with various uses, what the interactions ought to mean semantically as well as the reasoning for choosing the semantics to be so.

function

Consider this example:

api function foo(arg1, arg2)
   arg1 + arg2
end

# equivalently:
# api foo(arg1, arg2) = arg1 + arg2

This declares the function foo, with a single method taking two arguments of any type. The api keyword, when written in front of such a method definition, only declares the given method as API. If we were to later on define a new method, taking different arguments like so

function foo(arg1::Int, arg2::Float64)
  arg1 * arg2
end

the new method would not be considered API under semver. The reasoning for this is as simple - once a method (any object, really) is included in a new release as API, removing it is a breaking change, even if that inclusion as API was accidental. As such, being conservative with what is considered API is a boon to maintainability.

Being declared on a per-method case means the following:

• A method annotated with api MAY NOT be removed in a non-breaking release, without splitting the existing method into multiple definitions that are able to fully take on the existing dispatches of the previous single method. In type parlance, this means that the type union of the signatures of the replacement methods MUST be at least as specific as the original method, but MAY be less specific. This is to prevent introducing accidental MethodErrors where there were none before.
• A function/method annotated with api MAY NOT introduce an error where there was none before, without that being done in a breaking release.
• A function/method annotated with api MAY change the return type of a given set of input arguments, even in a non-breaking release. Developers are free to strengthen this to MAY NOT if they feel it appropriate for their function/method.
• A function/method annotated with api MAY remove an error and introduce a non-throwing result.
• A function/method annotated with api MAY change one error type to another, but developers are free to strengthen this to MAY NOT if they feel it appropriate for their function.

This is not enforced in the compiler (it can't do so without versioned consistency checking between executions/compilations, though some third party tooling could implement such a mechanism for CI checks or similar), but serves as a semantic guideline to be able to anticipate breaking changes and allow developers to plan around and test for them easier. The exact semantics a method that is marked as API must obey to be considered API, apart of its signature and the above points, are up to the developers of a function.

Depending on usecase (for example, some interface packages), it is desirable to mark all methods of a function as API. As a shorthand, the syntax

api function bar end

declares ALL methods of bar to be public API as an escape hatch - i.e., the above syntax declares the function bar to be API, not just individual methods. In Base-internal parlance, the api keyword on a single method only marks an entry in the method table as API, while the use on a zero-arg definition marks the whole method table as API. An API mark on the whole method table trumps a nonexistent mark on a single method - it effectively acts as if there were a method taking a sole ::Vararg{Any} argument and marking that as API.

struct

The cases elaborated on here are (effectively) already the case today, and are only mentioned here for clarity. They are (mostly) a consequence of subtyping relationships and dispatch.

Consider this example:

abstract type AbstractFoo end

api struct MyStruct{T, S} <: AbstractFoo
    a::T
    b::S
end

A struct like the above annotated with api guarantees that the default constructor methods are marked as api. The subtyping relationship is considered API under semver, up to and including Any. The existence of the fields a and b are considered API, as well as their relationship to the type parameters T and S.

In the example above, the full chain MyStruct{T,S} <: AbstractFoo <: Any is considered API under semver, which means that methods declared as taking either an AbstractFoo or an Any argument must continue to also take objects of type MyStruct. This means that changing a definition like the one above into one like this

abstract type AbstractBar end
abstract type AbstractFoo end

api struct MyStruct{T,S} <: AbstractBar
    a::T
    b::S
end

is a breaking change under semver. It is however legal to do the following:

abstract type AbstractFoo end
abstract type AbstractBar <: AbstractFoo end

api struct MyStruct{T,S} <: AbstractBar
    a::T
    b::S
end

because the old subtyping chain MyStruct{T,S} <: AbstractFoo <: Any is a subchain of the new chain MyStruct{T,S} <: AbstractBar <: AbstractFoo <: Any. That is, it is legal to grow the subtyping chain downwards.

Notably, making MyStruct API does not mean that AbstractFoo itself is API, i.e. adding new subtypes to AbstractBar is not supported and is not considered API purely by annotating a subtype as API.

Since the new type in a changing release must be useable in all places where the old type was used, the only additional restriction placed on MyStruct as defined above is that no type parameters may be removed. Due to the way dispatch is lazy in terms of matching type parameters, it is legal to add more type parameters without making a breaking change (even if this makes uses of things like MyStruct{S,T} in structs containing objects of this type type unstable).

In regards to whether field access is considered API or not, it is possible to annotate individual fields as api:

api struct MyStruct{T,S}
    a::T
    api b::S
end

This requires the main struct to be annotated as api as well - annotating a field as API without also annotating the struct as API is illegal. This means that accessing an object of type MyStruct via getfield(::MyStruct, :b) or getproperty(::MyStruct, :b) is covered under semver and considered API. The same is not true of the field a, its type or the connection to the first type parameter, the layout of MyStruct or the internal padding bytes that may be inserted into instances of MyStruct.

abstract type

abstract type behaves similarly to struct, in that it is illegal to remove a type from a subtype chain while it being legal to extend the chain downwards or to introduce new supertypes in the supertype chain.

Consider this example:

abstract type AbstractBar end
api abstract type MyAbstract <: AbstractBar end
# MyAbstract <: AbstractBar <: Any

The following changes do not require a breaking version:

# introducing a supertype
abstract type AbstractBar end
abstract type AbstractFoo <: AbstractBar end
api abstract type MyAbstract <: AbstractFoo end
# MyAbstract <: AbstractFoo <: AbstractBar <: Any

The following changes require a new breaking version:

# removing a supertype
api abstract type MyAbstract <: AbstractFoo end
# MyAbstract <: <: Any

# removing the `api` keyword
abstract type AbstractBar end
abstract type MyAbstract <: AbstractFoo end
# MyAbstract <: AbstractBar <: Any

What the api keyword used on abstract types effectively means for the users of a package is that it is considered API to subtype the abstract type, to opt into some behavior/set of API methods/dispatches the package provides, as long as the semantics of the type (usually detailed in its docstring) are followed. In particular, this means that methods like api function foo(a::MyAbstract) are expected to work with new objects MyConcrete <: MyAbstract defined by a user, but methods like function bar(a::MyAbstract) (note the lack of api) are not.

At the same time, a lack of api can be considered an indicator that it is not generally expected nor supported to subtype the abstract type in question.

type annotated/const global variables

api MyFoo::Foo = ...
api const MyBar = ...

Marking a global const or type annotated variable as api means that the variable binding is considered API under semver and is guaranteed to exist in new versions as well. For a type annotated global variable, both reading from and writing to the variable by users of a package is considered API, while for const global variables only reading is considered API, writing is not API.

The type of a type annotated variable is allowed to be narrowed to a subtype of the original type (i.e. a type giving more guarantees), since all uses of the old assumption (the weaker supertype giving less guarantees) are expected to continue to work.

Non-type-annotated global variables can never be considered API, as the variable can make no guarantees about the object in question and any implicit assumptions of the object that should hold ought to be encoded in an abstract type representing those assumptions/invariants. It is legal to explicitly write api Bar::Any = ....

It should be noted that it is the variable binding that is considered API, not the the variable refers to itself. It is legal to document additional guarantees or requirements of an object being referred to through a binding marked as api.

module

Annotating an entire module expression with api means that all first-level symbols defined in that module are considered API under semver. This means that they cannot be removed from the module and accessing them should return an object compatible with the type that same binding had in the previous version.

Consider this example:

api module Foo
    module Bar
      api const baz = 1
      const bak = 42
    end

    f() = "hello"
end

In this example, Foo, Foo.f, Foo.Bar, Foo.Bar.baz are considered API, while Foo.Bar.bak is not.

Consider this other example:

module Foo
    api module Bar
      const baz = 1
      const bak = 42
    end

    f() = "hello"
    api g() = 
end

In this example, Foo.g, Foo.Bar, Foo.Bar.baz and Foo.Bar.bak are considered API, while Foo and Foo.f are not.

Consider this third example:

module Foo
    module Bar
      api const baz = 1
      const bak = 42
    end

    f() = "hello"
end

Only Foo.Bar.baz is considered API, the other names in and from Foo and Foo.Bar are not.

Uses

This is a list of imagined uses of this functionality:

• Linting hints in LSP.jl, to make users aware of accidentally using internals of a package/Base.
• API surface tracking over time, especially in regards to test coverage and breaking changes.
• Tree-shaking for Pkg images/static compilation, to only make api bindings available in the final image/binary/shared object
  • This is primarily thought of in the context of compilation to a shared object - the api marker could be used for not mangling the names of julia functions when compiling an .so, as currently all names are mangled by default (unless marked as @ccallable, if I'm not mistaken, which is limited to taking C-compatible types in a C-style ABI/calling convention).
• Automatic generation of documentation hints about API surface in Documenter.jl
• Easier tracking of deprecated functionality before it is ultimately removed in a breaking change
• Have a PR template mentioning "Does this PR introduce new API?"
• CI Check enforcing that new API bindings have a docstring associated with them

Do you have ideas? Mention them and I'll edit them in here!

Required Implementation Steps

• Parse the api keyword in the correct places and produce an expression the compiler can use later on
• Add api marker handling to methods and the method table implementation, as well as to binding lookup in modules
• Make the REPL help> mode aware of api tags.
• Go through all of Base and the stdlibs and mark the bindings currently residing in the manual as api

Known Difficulties with the proposal

• Expr(:function) does not have space in its first two arguments for additional metadata, so this would need to be added to either a third argument, or create a new Expr(:api_function). Analogous issues exist for Expr(:struct), Expr(:=), Expr(:module) etc. Both approaches potentially require macros to be changed, to be aware of api in their expansion.
• There is a lot of doc churn, but sadly there's no way around that. My hope is that this can make it easier to just write more docstrings, by virtue of not "promoting" a function to API status just by virtue of having a docstring.
• This proposal requires quite a bit of julia-internals expertise, touching the parser, probably lowering as well, lots of internal objects/state, the REPL and external packages. It's a very large amount of work, and there's likely no chance of any one person being able to implement all of it - this will be a group effort.

FAQ

• Why not a macro instead of new syntax?
  • A macro has the disadvantage of not composing nicely with existing macros. Additionally, since this
    requires quite deep changes to Method and other (internal?) objects of Base, exposing this
    as a macro would also mean exposing this as an API to the runtime, even though this api distinction is
    not about dynamicness - the api surface of a package really ought to be fixed in a given version,
    and not change dynamically at runtime.
• What about annotating bindings as private instead?
  • There is no intention of preventing access to an internal object or otherwise introduce access modifiers
    into the language. Additionally, marking things as private, internal or similar instead of api
    means that any time a developer accidentally forgets to add that modifier means a technically breaking
    change in a release by adding that. The whole point of this proposal is to avoid this kind of breakage.
• What about naming the keyword public?
  • As there is no intention to provide access modifiers, I feel like naming this public overloads this
    already overloaded term in the wider programming community too much. public/private are commonly
    associated with access modifiers, which is decidedly not what this proposal is about.
• What about naming the keyword ?
  • Bikeshedding the name is always welcome, though I think it hard to compete with the short conciseness of
    api, which makes its intent very clear. It would also be prudent to have that discussion after we've
    come to a compromise about the desired semantics.
• How does this proposal interact with export?
  • export is a bit tricky, since it doesn't distinguish between methods the way api does. I think
    it could work to mark all exported symbols with api as well (this is certainly not without its
    own pitfallse..), though I also think that export
    is a bit of an orthogonal concept to api, due to the former being about namespacing and the latter
    being exclusively about what is considered to be actually supported. I think a good example is
    the way save/load are implemented with FileIO.jl. While the parent interface package exports save
    and load, packages wishing to register a new file format define new, private functions for these
    and register those on loading with FileIO (or FileIO calls into them if they're in the environment).
    This means that MyPkg.save is not exported from MyPkg, but is nevertheless a supported API
    provided by MyPkg. The intention is to support these kinds of usecases, where export is
    undesirable for various reasons, while still wishing to provide a documented/supported API surface
    to a package.
• Why not prototype this in a package?
  • There are various prototypes of similar things in some packages, none of which has been widely adopted as far as I know and I think this is something Base itself could really use as well. Not to mention that starting this in a package is IMO going to splinter the "this is how we define API" discussion further.

---

I hope this proposal leads to at least some discussion around the issues we face or, failing to get implemented directly, hopefully some other version of more formalized API semantics being merged at some point.

[c42f]

I think the motivation here is solid! Well said :-)

I don't so much like the feel like the proposed api keyword directly in front of syntactic constructs. Basically for similar reasons that I'm against the same thing for export, as expressed here #8005 (comment)

Also I don't understand why api would make sense as a per-method thing rather than a per-function thing. It's generic functions which are named/called by the user's code, not individual methods. So the entire generic function should have the same API status.

Did you consider the alternative of having the API declared separately, in a similar way to how the export list currently is? Something more with the flavor of the proposal in #30204?

For example, we could have scoped export:

scoped export A, B

The idea is to export A and B as public API, but keep their names "scoped" within the namespace of the exporting module.

A nice thing about this is that it could be implemented as a contextual keyword: the scoped would only be a keyword when followed by export in the same way that mutable is not a keyword unless followed by struct. Thus not breaking any existing code. It would also be easy to support in Compat.jl as @scoped export A, B (which would just return export A, B on older Julia versions, and would retain the usual special case parsing rules for export lists).

Regardless of my quibbles, I think this is an important issue and it's great that you've put the effort into writing up your thoughts. Thank you!

[Seelengrab]

Did you consider the alternative of having the API declared separately, in a similar way to how the export list currently is?

I explicitly haven't mentioned other proposals because I want to see how people react to this taken at face value - not to mention that I think most other ideas in this space are a bit underspecified in what exactly the desired semantics are. I think that's supported by the amount of discussion around what the various PRs & issues mean & want, which is what I attempted to give here directly. Most of the proposal follows from the requirements imposed by "what is a breaking change" when looking at functions, methods, variables etc from the POV of a user upgrading a version of a package.

My reasoning for placing api directly in front, rather than as an explicit list, is that it gives a small incentive/reminder to be careful when working on that code. The whole point is to be more explicit & thoughtful with what could be breaking changes, and I think having the API marker at the place of definition helps with that (it also makes it obvious that there's an undocumented API thing, which makes it extremely easy for newcomers to come in with a docstring PR, since they then "only" have to figure out what a function is doing to be able to document it, instead of having to figure out that there's an undocumented thing in the first place).

Another reason is that api is a relatively weak marker, giving only the most necessary semantics required from the type system (in case of methods/dispatch/structs/abstract types). For any more details what api means in the context of a given package, a docstring giving the exact surface is required - otherwise, every single bit of observable behavior could be taken as supported API, meaning you suddenly couldn't change the time complexity of a function anymore, for example.

And a third reason - if the whole function is always considered API, what about new methods added from outside of the package defining the function? Are they also automatically considered API, and who supports them? That's not something you can do with export - you can't export new names from a foreign module (without eval, but that breaks everything anyway). I think it's better to be a bit more conservative here - while the symbol is important in terms of what is directly accessible at all, often not the entire method surface is considered API, and while there could be a an argument made for subtly enforcing forwards to actual internal functions, I think that's quite an unnecessary indirection, requiring possible duplication of docs to devdocs, duplication of (future) additions in permitted dispatch, taking even more care to double check that you don't accidentally violate an invariant guaranteed by the parent caller etc. This, to me, feels like too much additional boilerplate, when we already have dispatch to distinguish these cases from each other - so why not expose that in API as well?

Finally, I don't think it's difficult per se to add something like

api foo, bar, baz

as a syntax for an API list, marking all of these symbols as API. The difficulty lies in the semantics - what the semantics of that api marker are depends strongly on what foo, bar and baz are, and so using such a list requires re-inspection into what foo etc are, at a time when they (potentially) aren't even defined yet (and requires users to jump around in the source code while writing docs - something I'd like to avoid for dev-UX reasons). So I guess that's a fourth reason why I prefer local api - unlike namespacing of symbols, which is a global property due to the symbol being unique, api really has local differences in meaning & requirements. Not to mention that Jeff already expressed dislike for having "more than one way" of writing the same thing when talking about export - maybe he feels differently about api, but we'll have to wait for him to chime in to know that :)

Apologies, this got longer than expected - thanks for asking about it!

A nice thing about this is that it could be implemented as a contextual keyword:

This proposal can also be done like that, no? I've given an explicit list of places where writing api would be legal, after all :) All of them are currently syntax errors, so adding this as a contextual keyword shouldn't break any existing code:

julia> api function foo end
ERROR: syntax: extra token "function" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

julia> api module Foo end
ERROR: syntax: extra token "module" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

julia> api const bar = 1
ERROR: syntax: extra token "const" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

julia> api abstract type MyAbstract end
ERROR: syntax: extra token "abstract" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

julia> api struct Foo end
ERROR: syntax: extra token "struct" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

julia> api foo::Int = 1
ERROR: syntax: extra token "foo" after end of expression
Stacktrace:
 [1] top-level scope
   @ none:1

---

Thank you for taking the time to read through the proposal! I really appreciate your thoughts, especially because of your work on JuliaSyntax. If the scheme parser weren't so hard to jump into and hack on, this might have already been a draft PR instead of a proposal :)

[martinholters]

I'm a bit skeptical about the per-method thing. At least it needs some good thinking about what would be breaking. Coming from above example

api function foo(arg1, arg2)
   arg1 + arg2
end

A user can now happily do foo(1, 2.0). But the package in a new version now adds

function foo(arg1::Int, arg2::Float64)
  arg1 * arg2
end

The same call as before is now accessing an internal method.

[Seelengrab]

The same call as before is now accessing an internal method.

Yes, that is a good example for an accidental breaking change, of the sort described in the proposal that really must not happen. I realize now that I've only written about the case of removing foo(arg1, arg2), but of course this applies to adding new dispatches as well. I can add something like this to the list of "MUST"s, if you think that's correct:

• Adding new methods to a function MUST NOT reduce the available API surface. Dispatches that were api before the addition must also be marked api after the addition, such that code relying on the old dispatch doesn't have to access a non-api method.

I think there is room for having this testable as part of a testsuite - for example, one could take the declared signature of all api methods of foo in the last commit, and compare that against the dispatches of those signatures in all methods of foo in the new commit. If any signature would dispatch to a non-api marked method in the new commit, you have a testfailure due to an accidental breaking change. This does have the downside of requiring the testsuite to have awareness of being a testsuite and being able to check against a known-good version, but IMO that's something we should do anyway - we currently just do it (badly) in our heads when writing the code.

Another way this could be done without checking past versions is have something of the sort of

@test_api foo(::Number, ::Number)

to check whether the given signature (and its subtypes) dispatches to an api method or not. I feel like there's also room for improvement there, but I haven't thought about that part too deeply yet (compared to the other parts of the proposal anyway).

[rfourquet]

I don't have much insignt into the merits of marking api per-method vs per-function, but I fear that per-method would be too complex and lead many devs to just don't bother with api because of cognitive overhead. On the other hand, the rule for per-function seems straightforward enough: "an api function must not break use cases allowed by the attached docstring". And for those motivated, it seems the code can always be re-organized to make "non api methods" unavailable by renaming them with e.g. a leading underscore.

[Seelengrab]

Well conversely, the api function foo end functionality would already give that exact behavior 🤷 I think a per-method annotation could be a boon, as described above, but I'm willing to drop that if others feel that it's too much mental overhead (though I'll gripe about this, since it'll inevitably lead to more accidental breakages down the road 😛 the complexity of API surface doesn't vanish by marking the whole function as api after all, it just hides it away until it becomes an issue).

In either case, I think it's good that this discussion is happening - keep the feedback coming!

[dalum]

Thanks for a well-written proposal! I 100% agree that there is an issue with a lack of consensus around what makes an API of a package, and that it would be nice to have a rigid way to specify this.

However, first, in the bikeshed department, I don't like the keyword api. Unlike all other keywords, it's a lowercase form of an abbreviation, and I'd much rather see a keyword that isn't pronounced letter-by-letter. Or maybe this would be the motivation to start pronouncing it "ay-pai". I'm also not even sure if we need a new keyword. It feels like something that could be added to the docstring for special-handling. That would also nudge developers to write docstrings for things considered public API. Something like, if the docstring starts with @API:

"""@API
    my_function(args...)
    
This is a nicely documented function that does what you expect.
"""
function my_function(args...) end

Second, I'm a bit worried about how the notion of an API surface and the genericity of Julia code composes. It is often very convenient to define a function like my_public_function(::Any, ::Any), which then, in principle, has an API surface covering anything the user throws at it that doesn't error, but in practice, and as perhaps written in its docstring, it requires some trait-like behaviour. I can definitely foresee situations where my_public_function would work for some arguments by accident in one version, and through a non-breaking change, according to the docstring of the function, suddenly wouldn't work in a later version, because the internal implementation changed. While it is, in a sense, problematic that the user code breaks in this case, I think a strength of Julia comes from actually being a bit fuzzy here. However, if I understand the semantics you proposed correctly, this would be considered an accidental breaking change that should "never happen", which I guess I then disagree with.

Third, I think if something like this is implemented, having the notion of an "experimental API" or something similar will be invaluable as a way to experiment with new features, while declaring the intent that an interface could become stable in a later, non-breaking release.

Finally, like with export, I do not think it is useful to think of the API at a method level. I think there has been a fairly strong push in Julia to make sure that a given function has just one meaning, which means that methods are simply implementation details on how to implement that functionality for a given set of types. I wouldn't want to propose a convention where only specific methods are considered part of the public API. Something feels off to me to think about my_function being public if you call it with types X and Y, but internal if you call it with Z. This does clash a bit with declaring the API in the docstring, but that might not be important.

[Seelengrab]

Thank you for your time and discussing the proposal!

However, first, in the bikeshed department, I don't like the keyword api.

The bikeshed is noted, but barring an alternative.. 🤷 I'm only partial to api myself because it's a ridicolously well-known concept that fits the intended meaning precisely, as well as being quick to type and unambiguous with other keywords (tab completion can then work its magic wonders). I do agree that it being an abbreviation is kind of bad though. Suggestions are welcome! :)

I'm also not even sure if we need a new keyword. It feels like something that could be added to the docstring for special-handling. That would also nudge developers to write docstrings for things considered public API. Something like, if the docstring starts with @API:

I originally considered proposing an API-string macro, used like so:

api"""
    my_function(args...)
    
This is a nicely documented function that does what you expect.
"""
function my_function(args...) end

and implemented like

struct APIMarker
    doc::String
end

macro api_str(s::String)
     APIMarker(s)
end

which would give any documentation rendering the ability to distinguish API from non-API. The issue with that or with placing the marker in a docstring is that you now need to parse & process the docstring to be able to say "this is API" for doing code analysis. It's no longer a static property of the source code as written, and is something that can change at runtime - Base.Docs allows a developer mutating documentation at runtime as they see fit, which isn't really something that should be done for what is considered the API surface of a package, in a given version - that should be static information, hence the proposal for syntax. I really do think this needs to be a language feature.

Second, I'm a bit worried about how the notion of an API surface and the genericity of Julia code composes.

That whole paragraph is a very good point, and I have thought about that! There's also a subtlety that I forgot to put into my original proposal, so let me take this opportunity to reply and add that subtlety here as well, by lieu of an example.

Consider a function with a single method like

function foo(arg1, arg2)
   arg1 + arg2
end

that we say is API of some package. It's signature is Tuple{typeof(foo), Any, Any}, and taken at face value, we might think that we can call foo with literally any two objects. This is of course wrong - there's an implicit constraint on the types of arg1 and arg2 to also have a method on +, taking their types - Tuple{typeof(+), typeof(arg1), typeof(arg2)}. Neither exporting a whole function name, nor marking a single method as api can represent that constraint accurately, and in fact we don't even currently have ways of discovering those kinds of constrains (I did play around with abstract interpretation for this, but quickly ran into dynamic dispatch ruining my day, so I postponed that idea). This is quite a bit more powerful a requirement than even Haskell does with its typeclasses, I think, though I don't have a proof of that. While typing arguments as Any is tempting and can often help with interoperability, it's really unfortunate that these kinds of implicit constraints in reality constrain Any further, so it wasn't ever going to work with truly Anything anyway. I guess the "hot take" here is that any API function that claims to take Any is a lie, and shouldn't be trusted (well, other than the identity function - at least, I haven't seen any object that couldn't be returned from a function. Sounds like a fun challenge..).

So, barring being able to express those kinds of constraints, the next best thing is to say "well, I support this signature, provided you implement functions X, Y, Z with arguments W, V, S on your type T" in a docstring. The foo(::Any, ::Any) example is just the most extreme case of this; the use case I had in mind when writing this was something along the lines of

api function foo(x::MyAbstract, y)
    2x+3y
end

and being able to express "I only support foo with a first argument that's obeying the constraints of MyAbstract", effectively transferring the constraint of *(::Int, ::MyAbstract) to the contract/api that MyAbstract provides/that subtypes of MyAbstract are required to provide. From my POV, an abstract type is already what we'd call an informal "interface" in other languages - they just don't compose at all, due to only being able to subtype a single abstract type, leading to the proliferation of Any in signatures (which ends up hiding the true required surface).

I can definitely foresee situations where my_public_function would work for some arguments by accident in one version, and through a non-breaking change, according to the docstring of the function, suddenly wouldn't work in a later version, because the internal implementation changed.

Well, I don't per se see an issue with that - not every implementation detail is part of the API of a function. While some implicit constraints on the arguments can be, in practice those may be considered an implementation detail, and if they are, it's fine if non-breaking change (according to the docstring/additional documentation of the method) the breaks some user code that relied on them. The correct solution here is to lift the actual requirements to the type domain, by using a dispatch type for those kinds of invariants/requirements, not to make the API more restrictive by lifting the implicit constraints to Any.

Third, I think if something like this is implemented, having the notion of an "experimental API" or something similar will be invaluable as a way to experiment with new features, while declaring the intent that an interface could become stable in a later, non-breaking release.

I'd implement that with this proposal by having an inner module Experimental, like Base.Experimental, which isn't marked as api (perhaps individual functions/methods inside of that module are marked API though, to make clear which part of the Experimental module is considered part of the experimental API and which parts are just implementation details of the experimental functionality). The reasoning for this is that experimental features, by definition, can't be relied upon under semver, as they may be removed at any time, so they must not be marked as part of the API surface of the package. Mixing experimental API into the same namespace as stable API seems to me like a way to more accidental breakages, through people assuming that something being marked as soft-API will be stable at some point, so it's fine to rely on it (which is not the case - usage of experimental functionality needs to be guarded by version checks, just like any other use of internal functionality).

Something feels off to me to think about my_function being public if you call it with types X and Y, but internal if you call it with Z.

Well.. from my POV, that's kind of what we're doing all the time though already, by having surface-level API functions like foo that forward their Any typed arguments to _foo or _foo_impl where the actual disambiguation through dispatch (with MethodError to boot if a combination is not supported) happens. All I'm proposing is removing that indirection, by making the API method-specific. If your _foo_impl has a different signature entirely to gather some more dispatchable data you don't need your users to give themselves (see an example here), it's of course still fine to forward to that internal function.

In part, the "API per method" section of the proposal stems from my (perhaps irrational) fear that people are going to slap api on everything to make the warnings go away, and then being undisciplined with actually providing the guarantees they claim to give with that marker. I think this issue already exists today, but it's just masked by noone being able to consistently tell whether something is part of the API of a package or not - my gut says that a surprising amount of code runs on the "happy path" and always uses the latest version of things anyway, but the few times a big upgrade in a version from 1-2 year old code and subsequent breakage of that code came up on slack just got stuck in my mind. If people were able to consistently rely on some given methods being API and developers were able to actually accurately test that their API surface on a method-by-method basis doesn't change, there'd be less of a risk of such breakage occuring. But again, that may be an irrational fear of mine 🤷

[dalum]

It's signature is Tuple{typeof(foo), Any, Any}, and taken at face value, we might think that we can call foo with literally any two objects. This is of course wrong - there's an implicit constraint on the types of arg1 and arg2 to also have a method on +, taking their types - Tuple{typeof(+), typeof(arg1), typeof(arg2)}. Neither exporting a whole function name, nor marking a single method as api can represent that constraint accurately, and in fact we don't even currently have ways of discovering those kinds of constrains (I did play around with abstract interpretation for this, but quickly ran into dynamic dispatch ruining my day, so I postponed that idea).

I appreciate your point, but I disagree with the analysis. I could define a function:

"""
    multiply_by_two(x)
    
Return `x` multiplied by two.
""""
multiply_by_two(x) = x

This function will never error. It is the identity function. However, it is a buggy implementation, as it does in fact not do what it states. An API is not something that could or, I would argue, even should be analyzed from a code perspective. It's a slightly more fuzzy human-to-human contract written in prose.

It's no longer a static property of the source code as written, and is something that can change at runtime - Base.Docs allows a developer mutating documentation at runtime as they see fit, which isn't really something that should be done for what is considered the API surface of a package, in a given version - that should be static information, hence the proposal for syntax. I really do think this needs to be a language feature.

As my point above, I don't see any compelling argument for why this shouldn't be allowed to change at runtime, though. As I see it, having API declarations in the code is a feature that is purely intended to help users (and potentially linters). Intentionally making it less flexible than the rest of Julia's documentation system seems to be in stark contrast to Julia's dynamic nature, and I think analysers will be able to overcome the docstring-parsing requirement.

The correct solution here is to lift the actual requirements to the type domain, by using a dispatch type for those kinds of invariants/requirements, not to make the API more restrictive by lifting the implicit constraints to Any.

While I agree in principle that it would be nice to use the type system to restrict these cases, based on my own experience, I don't think the type system is the correct level to do this at in Julia. And there are several cases where it would be impractical to require something to subtype a particular abstract type.

Well.. from my POV, that's kind of what we're doing all the time though already, by having surface-level API functions like foo that forward their Any typed arguments to _foo or _foo_impl where the actual disambiguation through dispatch (with MethodError to boot if a combination is not supported) happens. All I'm proposing is removing that indirection, by making the API method-specific. If your _foo_impl has a different signature entirely to gather some more dispatchable data you don't need your users to give themselves (see an example here), it's of course still fine to forward to that internal function.

I also think it would be really nice to have all methods exposed at the surface-level of the dispatch tree, but there are a lot of practical reasons for having function indirection. Often it's with an args... capture (Any-typed), etc., and it'd be very inconvenient to have to explicitly type those in every method.

[Seelengrab]

This function will never error. It is the identity function. However, it is a buggy implementation, as it does in fact not do what it states.

Right, and I'm not claiming that api (or the compiler) could or should be a total, machine readable description of all the invariants a function requires to be called - that would indeed be impractical and against the dynamic nature of julia. What I am saying/proposing is that the ability for a developer to declare "multiply_by_two is part of the API of my package" for a given version on a method-by-method basis is valuable; whether the actual implementation matches the desired behavior for a function is a different matter. Checking that is the job of a testsuite, not some declaration about what a developers sees as public API for their package (which is the only thing I'm concerned about here - giving developers the ability to say "this is what I support", in a consistent, Base supported manner. As such, it IMO needs to be flexible enough to accomodate granularity).

An API is not something that could or, I would argue, even should be analyzed from a code perspective. It's a slightly more fuzzy human-to-human contract written in prose.

I disagree; that's exactly what a testsuite is. A docstring is the prose describing the desired behavior, and a testsuite is the code checking that the functional implementation matches the prose.

While I agree in principle that it would be nice to use the type system to restrict these cases, based on my own experience, I don't think the type system is the correct level to do this at in Julia. And there are several cases where it would be impractical to require something to subtype a particular abstract type.

I agree with you here, this is not currently a desirable thing to do because there's only one possible direct abstract supertype. That's still an orthogonal concern to whether or not a single method is considered API or not though. Maybe that's an argument for holding off on method-level-API until such a time when we can express these kinds of constraints, of saying "I require the invariants these 2+ abstract types provide", better?

[adienes]

personally, I would not prefer the syntax be a modifier to existing definitions like api function ... because then the declarations are still scattered throughout the code. the primary problem for me is discoverability, which is maximized when the interface(s) are more or less centralized, or in some very clear hierarchical structure

an API is kind of like an "interface for a package," so what if the declaration format were similar to an interface proposal I have seen being discussed recently? https://github.com/rafaqz/Interfaces.jl

Mockup below with a new syntax interface block operating kind of like a header file, with arbitrary "tags" for API. the only information given about the functions is the names and signatures. Hopefully this should provide enough information for good tooling ? anything more substantive should probably just go in a docstring. interface could also support the where keyword so that methods can be made api with as much specificity as desired. also multiple declarations could be made so the entire interface doesn't have to fit in a single format if that pattern is desired. Another benefit of declaring it this way is that things can be made api conditionally via package extensions

module Animals
    export speak, Dogs

    abstract type Animal end
    speak(::Animal) = "Hello, World!"
    speak(::Animal, s <: AbstractString) = "Hello, $s !"

    module Dogs
        using ..Animals

        struct Dog <: Animals.Animal end
        Animals.speak(::Dog) = "woof"

        bark(d::Dog) = Animals.speak(d)
    end

    module Cats
        using ..Animals

        struct Cat <: Animals.Animal end

        Animals.speak(::Cat) = "meow"
        fight(::Cat, ::Cat) = String["hiss", "screech"]
        getalong(::Cat, ::Cat) = nothing
    end
end


interface Animals where T <: AbstractString
    :exported => (
        speak => Tuple{Animal} => String,
        speak => Tuple{Animal, T} => String,
        Dogs => bark => Tuple{Dog} => String
    ),
    :stable => (
        Animals => Cats => speak => Tuple{Cat} => String
    )
end

interface Animals
    :experimental => (
        Animals => Cats => fight => Tuple{Cat, Cat} => Vector{String}
    ),
    :deprecated => (
        Animals => Cats => getalong => Tuple{Cat, Cat} => Nothing
    )
end

[adienes]

also, possibly this discussion should be folded together with #48819 ?

[Seelengrab]

an API is kind of like an "interface for a package," so what if the declaration format were similar to an interface proposal I have seen being discussed recently? https://github.com/rafaqz/Interfaces.jl

I don't think that should be tackled in this proposal. I don't aim to modify the type system here at all - this is purely for allowing developers to declare their existing API, without having to migrate to new concepts in terms of what they want to develop.

Also, while I can see a path for something like interfaces to be a thing, I don't think the approach in Interfaces.jl is right, in particular because it's a big disruption to the ecosystem due to the amount of "legacy" code we have. I have a different idea for how to tackle that (mostly based on type theoretic consequences/additions to the current type system), which does integrate with this proposal nicely without special cases while keeping code churn as small as possible (not larger than the code churn produced by this proposal, in fact), but that's for a different discussion.

also, possibly this discussion should be folded together with #48819 ?

While that discussion is the first time I wrote down parts of the ideas contained in this proposal, it's not a concrete discussion about a concrete proposal. Thus, I think this issue is more appropriate to talk about this proposal at hand, instead of the larger discussion whether we want something like access modifiers at all. So far, the only major gripe with this proposal seems to be the per-method-API and some minor bikeshedding on the name/syntax to expose this (which I think I have argued sufficiently now why it ought to be syntax, and not dynamic information).