Document transaction fees

docs/conceptual/runtime/weight.md

@@ -0,0 +1,163 @@
+---
+title: Transaction Weight
+---
+
+A number of resources in any chain can be limited by definition, such as time, memory and
+computation. A mechanism should exist to prevent individual resource-consuming components of the
+chain, including but not limited to dispatchable functions, from consume too much of any of the
+mentioned resources. This concept is represented in substrate by weights. Furthermore, consuming
+some weights could optionally incur some fee.
+
+The fee implications of the weight system is covered in the [Fee Developer document](). This
+document mostly covers the concept of weights.
+
+## Transaction Weight
+
+As mentioned, weights can technically refer to, or represent, numerous _limited_ resources. A custom
+implementation may use complex structures to demonstrate this. At the time of this writing,
+substrate weights are simply a [numeric
+value](/rustdocs/master/sr_primitives/weights/type.Weight.html). Each dispatchable function
+is given a weight using the `#[weight = $x]` annotation, where `$x` is some function capable of
+determining the weight of the dispatch. `$x` can access and examine the arguments of the dispatch.
+Nonetheless, a weight calculation should always:
+- Be computable __ahead of dispatch__. A block producer, or a validator, should be able to examine
+  the weight of a dispatchable before actually deciding to accept it or not.
+- While not enforced, calculating the weight should not consume much resource itself. It does not
+  make sense to consume similar resources computing a transaction's weight as would be spent while
+  executing it. Thus, weight computation should typically be very fast, much faster than
+  dispatching.
+- Finally, if a transaction's resource consumption varies over a wide range, it should manually
+  prevent restrict certain dispatches through any means that makes sense to that particular chain.
+  An example of such dispatches in substrate is the call to a smart contract (todo: link when this
+  is implemented).
+
+The system module is responsible for accumulating the weight of each block as it gets executed and
+making sure that it does not exceed the limit. The transaction-payment module is responsible for
+interpreting these weights and deducting fees based upon them.
+
+
+## Block Weight and Length Limit
+
+Aside from affecting the fees, the main purpose of the weight system is to prevent a block from
+being too filled with transactions. The system module, while processing transactions within a block,
+accumulates both the total length of the block (sum of encoded transactions in number of bytes) and
+the total weight of the blocks. At any points, if these numbers surpass the limits, no further
+transactions are accepted to the block. These limits are defined in
+[`MaximumBlockLength`](/rustdocs/master/srml_system/trait.Trait.html#associatedtype.MaximumBlockLength)
+and
+[`MaximumBlockWeight`](/rustdocs/master/srml_system/trait.Trait.html#associatedtype.MaximumBlockLength)
+respectively.
+
+One important note about these limits is that a ratio of them are reserved for the `Operational`
+dispatch class. This rule applies to both of the limits and the ratio can be found in
+[`AvailableBlockRatio`](/rustdocs/master/srml_system/trait.Trait.html#associatedtype.AvailableBlockRatio).
+For example, if the block length limit is 1 mega bytes and the ratio is set the 80%, all
+transactions can fill the first 800 kilo bytes of the block while the last 200 can only be filled by
+the operation class.
+
+## Custom Weight Implementation
+
+Implementation a custom weight calculation function can vary from _very easy_ to _super
+complicated_. The `SimpleDispatchInfo` struct provided by substrate is fairly one of the easiest
+appraoches. Anything more sophisticated would require a bit more work.
+
+Any weight calculation function must provide two trait implementations:
+  - [`WeightData<T>`]: To determine the weight of the dispatch.
+  - [`ClassifyDispatch<T>`]: To determine the class of the dispatch. See the enum definition for
+    more information on dispatch classes. This struct can then be used with `#[weight]` annotation.
+
+Substrate then bundles then output information of the two traits into [`DispatchInfo`] struct and
+provides it by implementing the [`GetDispatchInfo`] for all `Call` variants, and opaque extrinsic
+types. This is used internally by the system and executive module; you won't use it most likely.
+
+Both `ClassifyDispatch` and `WeightData` are generic over `T` which gets resolved into the tuple of
+all dispatch arguments except for the origin. To demonstrate, we will craft a struct that calculates
+the weight as `m * len(args)` where `m` is a given multiplier and `args` is the concatenated tuple
+of all dispatch arguments. Furthermore, the dispatch class is Operational if the transaction has
+more than 100 bytes of length in arguments.
+
+```rust
+use coded::Encode;
+use sr_primitives::weights::{DispatchClass, ClassifyDispatch, WeightData}
+
+// self.0 is the multiplier, `m`
+struct LenWeight(u32);
+
+// We don't quite know what T is. After all, different dispatches have different arguments, hence
+// `T` will be different. All that we care about is that `T` is encodable. That is always true by
+// definition. All dispatch arguments are encodable.
+impl<T: Encode> WeighData<T> for LenWeight {
+    fn weigh_data(&self, target: T) -> Weight {
+        let multiplier = self.0;
+        let encoded_len = target.encode().len() as u32;
+        multiplier * encoded_len
+    }
+}
+
+impl<T: Encode> ClassifyDispatch<T> for LenWeight {
+    fn classify_dispatch(&self, target: T) -> DispatchClass {
+        let encoded_len = target.encode().len() as u32;
+        if encoded_len > 100 {
+            DispatchClass::Operational
+        } else {
+            DispatchClass::Normal
+        }
+    }
+}
+```
+
+A weight calculator function can also be entirely coerced to the final type of the argument, instead
+of defining it as a vague type that is encodable. This is also possible and `srml-example` contains
+an example of how to do this. Just note that in that case your code would roughly look like:
+
+```rust
+
+struct CustomWeight;
+impl WeighData<(&u32, &u64)> for CustomWeight {
+    fn weigh_data(&self, target: (&u32, &u64)) -> Weight {
+        ...
+    }
+}
+
+// given dispatch:
+decl_module! {
+    fn foo(a: u32, b: u64) { ... }
+}
+```
+
+which means **`CustomWeight` can only be used in conjunction with a dispatch with a particular
+signature (u32, u64)**, as opposed to `LenWeight` which can be used with literally anything because
+they don't make any strict assumptions about `<T>`.
+
+
+## Frequent Questions
+
+> _Some operations get more expensive as storage grows and shrinks. How can the weight system
+> represent that?_
+
+As mentioned, weight should be known _readily_ and _ahead of dispatch_. Peeking the storage and
+analyzing it just to determine how much weight might be consumed does not fit this definition quite
+well. In this case the implementation of the dispatch should take the state of the change into
+account and manually take extra fees, bonds or take any other measures to make sure that the
+transaction is safe.
+
+> _Is the weighting function open to change through governance or any sort of a runtime upgrade?_
+
+Yes, it is part of a runtime itself, hence any runtime upgrade is an upgrade to the weight system as
+well. A upgrade can just update weight functions if needed.
+
+## Next Steps
+
+### Learn More
+
+- srml-examples
+- recipes on weights
+- fee module
+TODO: links
+
+
+### References
+
+- weights.rs
+TODO: links
+

docs/development/module/fees.md

@@ -0,0 +1,210 @@
+---
+title: Transaction Fees
+---
+
+When transactions are submitted to a blockchain, they are executed by the nodes in the network. To
+be economically sustainable, nodes charge a fee to execute a transaction. This fee must be covered
+by the submitter of the transaction. The cost to execute transactions could vary over orders of
+magnitude, and thus Substrate provides a flexible mechanism for characterizing the total, presumably
+minimum, cost of a transaction in order to be included into a block.
+
+The fee system is heavily linked to the [weight system](conceptual/runtime/weight.md). Make sure to read and understand weights
+section before continuing this document.
+
+## Transaction Fees
+
+The fee to include a transaction consists of three parts:
+
+* `base_fee` a fixed fee that is applied to every single transaction. See
+  [`TransactionBaseFee`](/rustdocs/master/srml_transaction_payment/trait.Trait.html#associatedtype.TransactionBaseFee)
+* `length_fee` a per-byte fee that is multiplied by the length, in bytes, of the encoded
+  transaction. See
+  [`TransactionByteFee`](/rustdocs/master/srml_transaction_payment/trait.Trait.html#associatedtype.TransactionByteFee)
+* `weight_fee` a per-weight-unit fee that is multiplied by the weight of the transaction. As
+  mentioned, weight of each dispatch is denoted via the flexible `#[weight]` annotation. Knowing the
+  weight, it must be converted to a deductible `balance` type (typically denoted by a module that
+  implements `Currency`, `srml-balances` in substrate node). For this, each runtime must define a
+  [`WeightToFee`](/rustdocs/master/srml_transaction_payment/trait.Trait.html#associatedtype.WeightToFee)
+  type that makes the conversion. `WeightToFee` must be a struct that implements a [`Convert<Weight,
+  Balance>`](/rustdocs/master/sr_primitives/traits/trait.Convert.html).
+
+based on the above, the final fee of a dispatchable is:
+
+```
+fee =
+  base_fee +
+  len(tx) * length_fee +
+  WeightToFee(weight)
+```
+
+Customizing the above would be as simple as configuring the appropriate associated type in the
+respective module.
+
+```rust
+use sr_primitives::{traits::Convert, weights::Weight}
+// Assume this is the balance type
+type Balance = u64;
+
+// Assume we want all the weights to have a `100 + 2 * w` conversion to fees
+struct CustomWeightToFee;
+impl Convert<Weight, Balance> for CustomWeightToFee {
+    fn convert(w: Weight) -> Balance {
+        let a = Balance::from(100);
+        let b = Balance::from(2);
+        let w = Balance::from(w);
+        a + b * w
+    }
+}
+
+parameter_types! {
+    TransactionBaseFee: Balance = 10;
+    TransactionByteFee: Balance = 10;
+}
+
+impl transaction_payment::Trait {
+    TransactionBaseFee = TransactionBaseFee;
+    TransactionByteFee = TransactionByteFee;
+    WeightToFee = CustomWeightToFee;
+    // we will get to this one soon enough
+    FeeMultiplierUpdate = ();
+}
+
+```
+
+Two further questions need to be answered, given the above snippet.
+
+1. What is the purpose of `FeeMultiplierUpdate`?
+2. Knowing how we are capable of defining the conversion of weights to fees (via `WeightToFee`), the
+   question remains, how can we define and customize the weights in the first place?
+
+We will answer each question in the following sections, respectively.
+
+## Adjusting Multiplier
+The above formula gives a fee which is __logically constant through time__. Of course, the weight
+can be dynamic and based on what `WeightToFee` is defined to be, the final fee can have some degree
+of variability. As for the length fee, the inputs of the transaction could change the length and
+hence affecting the length fee. Nonetheless, these changes are independent and a _general update
+logic to the **entire fee** cannot be composed out of them trivially_. In other words, for any
+dispatchable, given the same inputs, _it will always incur the same cost_. This might not always be
+desirable. Chains might need to increase or decrease fees based on some condition.  To fulfill this
+requirement, Substrate provides:
+  - a multiplier stored in the transaction-payment module that is applied to the outcome of the
+    above formula by default (needless to say, the default value of which is _multiplication
+    identity_, meaning that it has no effect). This is stored in
+    [`NextFeeMultiplier`](/rustdocs/master/srml_transaction_payment/struct.Module.html#method.next_fee_multiplier)
+    storage and can be configured through the genesis spec of the module.
+  - a configurable parameter for a runtime to describe how this multiplier can change. This is
+    expressed via
+    [`FeeMultiplierUpdate`](/rustdocs/master/srml_transaction_payment/trait.Trait.html#associatedtype.FeeMultiplierUpdate).
+
+`NextFeeMultiplier` has the type `Fixed64`, which can represent a fixed point number with a billion
+points accuracy. So, given the final fee formula above, the final version would be:
+
+```
+fee =
+  base_fee +
+  len(tx) * length_fee +
+  WeightToFee(weight)
+
+final_fee = fee * NextFeeMultiplier
+```
+
+Updating the `NextFeeMultiplier` has a similar manner as `WeightToFee`. The
+[`FeeMultiplierUpdate`](/rustdocs/master/srml_transaction_payment/trait.Trait.html#associatedtype.FeeMultiplierUpdate)
+associated type in `transaction-payment` is defined as a `Convert<Fixed64, Fixed64>`, which should
+be read as: it receives the previous multiplier and spits out the next one.
+
+The default update function is inspired by the Polkadot network and implements a targeted adjustment
+in which a target saturation level of block weight is defined. If the previous block is more
+saturated, then the fees are slightly increases. Similarly, if less transaction than the target are
+in the previous block, fees are decreased by a small amount. More information about this can be
+found in the [web3 research
+page](https://research.web3.foundation/en/latest/polkadot/Token%20Economics/#relay-chain-transaction-fees).
+
+## Substrate Weight System
+
+> This section assumes that you have already read the `Weight` section
+
+The entire substrate runtime module library is already annotated with a simple and fixed weight
+system. A user can decide to use the same system or implement a new one from scratch. The latter is
+outside the scope of this document and is explained int he dedicated [`Weight`]() conceptual
+document.
+
+### Using the Default Weight
+
+The default weight annotation is beyond _simple_. Substrate, by default, uses _fixed_ weights and
+the struct representing them is as follows:
+
+```rust
+pub enum SimpleDispatchInfo {
+    /// A normal dispatch with fixed weight.
+    FixedNormal(Weight),
+    /// A normal dispatch with the maximum weight.
+    MaxNormal,
+    /// A normal dispatch with no weight.
+    FreeNormal,
+    /// An operational dispatch with fixed weight.
+    FixedOperational(Weight),
+    /// An operational dispatch with the maximum weight.
+    MaxOperational,
+    /// An operational dispatch with no weight.
+    FreeOperational,
+}
+```
+
+This struct simple groups all dispatches into _normal_ and _operational_ (which makes the
+implementation of `ClassifyDispatch` pretty trivial) and gives them a fixed Weight. Fixed in this
+context means that the arguments of the dispatch do not play any role in the weight; the weight of a
+dispatch is always fixed.
+
+A simple example of using this simple struct in your runtime would be:
+
+```rust
+use sr_primitives::weights::{SimpleDispatchInfo};
+
+decl_module! {
+    // This means that this function has no weight. It will not contribute to block fullness at all,
+    // and no weight-fee is applied.
+    #[weight = SimpleDispatchInfo::FreeNormal]
+    pub fn some_normal_function_light() { noop(); }
+
+    // This function will always have a weight `10`.
+    #[weight = SimpleDispatchInfo::FixedNormal(10)]
+    pub fn some_normal_function_heavy() { some_computation(); }
+
+    // This function will have a fixed weight but can consume the reserved operational portion as well.
+    #[weight = SimpleDispatchInfo::FixedOperational(20)]
+    pub fn mission_critical_function() { some_sudo_op(); }
+
+    // This will automatically get `#[weight = SimpleDispatchInfo::default()]`
+    pub fn something_else
+}
+```
+
+**Be careful!** The default implementation of `SimpleDispatchInfo` resolves to
+`FixedNormal(10_000)`. This is entire due to how things work in `substrate-node` and the desired
+granularity of substrate. Even if you want to use the `SimpleDispatchInfo`, it is very likely that
+you would want it to have a different `Default`.
+
+
+
+## Next Steps
+
+One important note is that the entire logic of fees is encapsulated in `srml-transaction-payment`
+via a `SignedExtension`. While this module provides a high degree of flexibility, a user can opt to
+build their custom payment module while inspiring from transaction-payment.
+
+### Learn More
+
+- Dedicated weight document
+- srml-example
+- SignedExtensions
+
+### Examples
+
+TODO
+
+### References
+
+TODO
+