feat(node): file-based observability configuration by karlem · Pull Request #1078 · consensus-shipyard/ipc

karlem · 2024-07-17T21:17:33Z

Close #1071

raulk

Could you post an example of what the traces look like with the console and file outputs?

ipc/observability/src/lib.rs

ipc/observability/src/traces_settings.rs

ipc/observability/src/traces.rs

raulk

Just some nits.

fendermint/app/src/cmd/rpc.rs

fendermint/vm/topdown/src/proxy.rs

karlem · 2024-07-22T16:27:18Z

Output from file:

{"timestamp":"2024-07-22T16:23:58.163639Z","level":"INFO","fields":{"message":"Connecting to Tendermint at http://127.0.0.1:26657/"},"filename":"fendermint/app/src/cmd/run.rs","line_number":70}
{"timestamp":"2024-07-22T16:23:58.164816Z","level":"DEBUG","fields":{"message":"validator key not configured"},"filename":"fendermint/app/src/cmd/run.rs","line_number":104}
{"timestamp":"2024-07-22T16:23:58.166078Z","level":"INFO","fields":{"message":"opening database","path":"/Users/karlem/.fendermint/data/rocksdb"},"filename":"fendermint/app/src/cmd/run.rs","line_number":395}
{"timestamp":"2024-07-22T16:23:58.212059Z","level":"INFO","fields":{"message":"IPLD Resolver disabled."},"filename":"fendermint/app/src/cmd/run.rs","line_number":241}
{"timestamp":"2024-07-22T16:23:58.212092Z","level":"INFO","fields":{"message":"topdown finality disabled"},"filename":"fendermint/app/src/cmd/run.rs","line_number":272}
{"timestamp":"2024-07-22T16:23:58.212098Z","level":"INFO","fields":{"message":"snapshots disabled"},"filename":"fendermint/app/src/cmd/run.rs","line_number":298}
{"timestamp":"2024-07-22T16:23:58.212450Z","level":"INFO","fields":{"message":"serving metrics","listen_addr":"127.0.0.1:9184"},"filename":"fendermint/app/src/cmd/run.rs","line_number":343}
{"timestamp":"2024-07-22T16:23:58.213254Z","level":"INFO","fields":{"message":"starting ABCI server","addr":"\"127.0.0.1:26658\""},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs","line_number":127}
{"timestamp":"2024-07-22T16:23:58.213260Z","level":"TRACE","fields":{"message":"worker polling for next message"},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/buffer/worker.rs","line_number":108}
{"timestamp":"2024-07-22T16:23:58.213302Z","level":"INFO","fields":{"message":"bound tcp listener","local_addr":"127.0.0.1:26658"},"filename":"/Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs","line_number":130}

Output from console:

2024-07-22T16:23:58.154879Z  INFO fendermint/app/src/cmd/mod.rs:88: reading configuration path="/Users/karlem/.fendermint/config"
2024-07-22T16:23:58.163563Z  INFO fendermint/app/src/cmd/run.rs:70: Connecting to Tendermint at http://127.0.0.1:26657/
2024-07-22T16:23:58.164808Z DEBUG fendermint/app/src/cmd/run.rs:104: validator key not configured
2024-07-22T16:23:58.166069Z  INFO fendermint/app/src/cmd/run.rs:395: opening database path="/Users/karlem/.fendermint/data/rocksdb"
2024-07-22T16:23:58.212041Z  INFO fendermint/app/src/cmd/run.rs:241: IPLD Resolver disabled.
2024-07-22T16:23:58.212089Z  INFO fendermint/app/src/cmd/run.rs:272: topdown finality disabled
2024-07-22T16:23:58.212096Z  INFO fendermint/app/src/cmd/run.rs:298: snapshots disabled
2024-07-22T16:23:58.212444Z  INFO fendermint/app/src/cmd/run.rs:343: serving metrics listen_addr="127.0.0.1:9184"
2024-07-22T16:23:58.213176Z  INFO /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs:127: starting ABCI server addr="127.0.0.1:26658"
2024-07-22T16:23:58.213219Z TRACE /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/buffer/worker.rs:108: worker polling for next message
2024-07-22T16:23:58.213297Z  INFO /Users/karlem/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-abci-0.7.0/src/v037/server.rs:130: bound tcp listener local_addr=127.0.0.1:26658

raulk · 2024-07-22T16:28:43Z

@karlem do we have logs/outputs showing journal event traces? 🙏

raulk · 2024-07-22T16:43:37Z

A metrics dump would also be very welcome, to ensure everything is working as expected 🙏

karlem · 2024-07-22T16:57:39Z

Metrics

# HELP bottomup_checkpoint_created_confignum Configuration number of the checkpoint created
# TYPE bottomup_checkpoint_created_confignum gauge
bottomup_checkpoint_created_confignum 0
# HELP bottomup_checkpoint_created_height Height of the checkpoint created
# TYPE bottomup_checkpoint_created_height gauge
bottomup_checkpoint_created_height 0
# HELP bottomup_checkpoint_created_msgcount Number of messages in the checkpoint created
# TYPE bottomup_checkpoint_created_msgcount gauge
bottomup_checkpoint_created_msgcount 0
# HELP bottomup_checkpoint_created_total Bottom-up checkpoint produced
# TYPE bottomup_checkpoint_created_total counter
bottomup_checkpoint_created_total 0
# HELP bottomup_checkpoint_finalized_height Height of the checkpoint finalized
# TYPE bottomup_checkpoint_finalized_height gauge
bottomup_checkpoint_finalized_height 0
# HELP exec_fvm_apply_execution_time_secs Execution time of FVM apply in seconds
# TYPE exec_fvm_apply_execution_time_secs histogram
exec_fvm_apply_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_apply_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="1"} 0
exec_fvm_apply_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="5"} 0
exec_fvm_apply_execution_time_secs_bucket{le="10"} 0
exec_fvm_apply_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_apply_execution_time_secs_sum 0
exec_fvm_apply_execution_time_secs_count 0
# HELP exec_fvm_call_execution_time_secs Execution time of FVM call in seconds
# TYPE exec_fvm_call_execution_time_secs histogram
exec_fvm_call_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_call_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_call_execution_time_secs_bucket{le="1"} 0
exec_fvm_call_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_call_execution_time_secs_bucket{le="5"} 0
exec_fvm_call_execution_time_secs_bucket{le="10"} 0
exec_fvm_call_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_call_execution_time_secs_sum 0
exec_fvm_call_execution_time_secs_count 0
# HELP exec_fvm_check_execution_time_secs Execution time of FVM check in seconds
# TYPE exec_fvm_check_execution_time_secs histogram
exec_fvm_check_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_check_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_check_execution_time_secs_bucket{le="1"} 0
exec_fvm_check_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_check_execution_time_secs_bucket{le="5"} 0
exec_fvm_check_execution_time_secs_bucket{le="10"} 0
exec_fvm_check_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_check_execution_time_secs_sum 0
exec_fvm_check_execution_time_secs_count 0
# HELP exec_fvm_estimate_execution_time_secs Execution time of FVM estimate in seconds
# TYPE exec_fvm_estimate_execution_time_secs histogram
exec_fvm_estimate_execution_time_secs_bucket{le="0.005"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.01"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.025"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.05"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.1"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.25"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="0.5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="1"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="2.5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="5"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="10"} 0
exec_fvm_estimate_execution_time_secs_bucket{le="+Inf"} 0
exec_fvm_estimate_execution_time_secs_sum 0
exec_fvm_estimate_execution_time_secs_count 0
# HELP proposals_block_committed Block committed
# TYPE proposals_block_committed counter
proposals_block_committed{height="1"} 1
proposals_block_committed{height="10"} 1
proposals_block_committed{height="11"} 1
proposals_block_committed{height="12"} 1
proposals_block_committed{height="13"} 1
proposals_block_committed{height="14"} 1
proposals_block_committed{height="15"} 1
proposals_block_committed{height="16"} 1
proposals_block_committed{height="17"} 1
proposals_block_committed{height="18"} 1
proposals_block_committed{height="19"} 1
proposals_block_committed{height="2"} 1
proposals_block_committed{height="20"} 1
proposals_block_committed{height="21"} 1
proposals_block_committed{height="22"} 1
# HELP proposals_block_proposal_accepted Block proposal accepted
# TYPE proposals_block_proposal_accepted counter

proposals_block_proposal_accepted{height="55"} 1
proposals_block_proposal_accepted{height="56"} 1
proposals_block_proposal_accepted{height="57"} 1
proposals_block_proposal_accepted{height="58"} 1
proposals_block_proposal_accepted{height="59"} 1
proposals_block_proposal_accepted{height="6"} 1
proposals_block_proposal_accepted{height="60"} 1
proposals_block_proposal_accepted{height="61"} 1
proposals_block_proposal_accepted{height="62"} 1
proposals_block_proposal_accepted{height="63"} 1
proposals_block_proposal_accepted{height="64"} 1
proposals_block_proposal_accepted{height="65"} 1
proposals_block_proposal_accepted{height="66"} 1
proposals_block_proposal_accepted{height="67"} 1
proposals_block_proposal_accepted{height="68"} 1
proposals_block_proposal_accepted{height="69"} 1
proposals_block_proposal_accepted{height="7"} 1
proposals_block_proposal_accepted{height="70"} 1
proposals_block_proposal_accepted{height="71"} 1
proposals_block_proposal_accepted{height="72"} 1
proposals_block_proposal_accepted{height="73"} 1
proposals_block_proposal_accepted{height="74"} 1
proposals_block_proposal_accepted{height="75"} 1
proposals_block_proposal_accepted{height="76"} 1
proposals_block_proposal_accepted{height="77"} 1
proposals_block_proposal_accepted{height="78"} 1
proposals_block_proposal_accepted{height="79"} 1
proposals_block_proposal_accepted{height="8"} 1
proposals_block_proposal_accepted{height="80"} 1
proposals_block_proposal_accepted{height="81"} 1
proposals_block_proposal_accepted{height="82"} 1
proposals_block_proposal_accepted{height="9"} 1
# HELP proposals_block_proposal_received Block proposal received
# TYPE proposals_block_proposal_received counter

proposals_block_proposal_received{height="70"} 1
proposals_block_proposal_received{height="71"} 1
proposals_block_proposal_received{height="72"} 1
proposals_block_proposal_received{height="73"} 1
proposals_block_proposal_received{height="74"} 1
proposals_block_proposal_received{height="75"} 1
proposals_block_proposal_received{height="76"} 1
proposals_block_proposal_received{height="77"} 1
proposals_block_proposal_received{height="78"} 1
proposals_block_proposal_received{height="79"} 1
proposals_block_proposal_received{height="8"} 1
proposals_block_proposal_received{height="80"} 1
proposals_block_proposal_received{height="81"} 1
proposals_block_proposal_received{height="82"} 1
proposals_block_proposal_received{height="9"} 1
# HELP proposals_block_proposal_sent Block proposal sent
# TYPE proposals_block_proposal_sent counter
proposals_block_proposal_sent{height="61"} 1
proposals_block_proposal_sent{height="62"} 1
proposals_block_proposal_sent{height="63"} 1
proposals_block_proposal_sent{height="64"} 1
proposals_block_proposal_sent{height="65"} 1
proposals_block_proposal_sent{height="66"} 1
proposals_block_proposal_sent{height="67"} 1
proposals_block_proposal_sent{height="68"} 1
proposals_block_proposal_sent{height="69"} 1
proposals_block_proposal_sent{height="7"} 1
proposals_block_proposal_sent{height="70"} 1
proposals_block_proposal_sent{height="71"} 1
proposals_block_proposal_sent{height="72"} 1
proposals_block_proposal_sent{height="73"} 1
proposals_block_proposal_sent{height="74"} 1
proposals_block_proposal_sent{height="75"} 1
proposals_block_proposal_sent{height="76"} 1
proposals_block_proposal_sent{height="77"} 1
proposals_block_proposal_sent{height="78"} 1
proposals_block_proposal_sent{height="79"} 1
proposals_block_proposal_sent{height="8"} 1
proposals_block_proposal_sent{height="80"} 1
proposals_block_proposal_sent{height="81"} 1
proposals_block_proposal_sent{height="82"} 1
proposals_block_proposal_sent{height="9"} 1
# HELP topdown_parent_finality_committed_height Parent finality committed on chain
# TYPE topdown_parent_finality_committed_height gauge
topdown_parent_finality_committed_height 0
# HELP topdown_parent_finality_voting_latest_sent_height Parent finality peer
# TYPE topdown_parent_finality_voting_latest_sent_height gauge
topdown_parent_finality_voting_latest_sent_height 0
# HELP topdown_parent_finality_voting_quorum_height Parent finality vote tally new agreement; recorded whenever the latest epoch with quorum
# TYPE topdown_parent_finality_voting_quorum_height gauge
topdown_parent_finality_voting_quorum_height 0
# HELP topdown_parent_finality_voting_quorum_weight Parent finality vote tally new agreement; recorded whenever the latest epoch with quorum
# TYPE topdown_parent_finality_voting_quorum_weight gauge
topdown_parent_finality_voting_quorum_weight 0

karlem · 2024-07-22T17:02:46Z

Some consensus related traces:

Console:

2024-07-22T16:58:48.490982Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 173, size: 0, tx_count: 0 }
2024-07-22T16:58:48.508255Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:48.508269Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:48.544056Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 173, app_hash: 0171a0e40220d36a47bfd38c81b0acd8a87a5d473875dc31cdbba81b5d3157b2ae4c009a81f4 }
2024-07-22T16:58:49.526968Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 174, size: 0, tx_count: 0 }
2024-07-22T16:58:49.547365Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:49.547385Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:49.582128Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 174, app_hash: 0171a0e402206e947f59c0f0746fd8dacd2d4c8eb30055233ce89f93bd68e86194e8d83a761d }
2024-07-22T16:58:50.565149Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 175, size: 0, tx_count: 0 }
2024-07-22T16:58:50.582157Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalReceived { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }
2024-07-22T16:58:50.582189Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockProposalEvaluated { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }
2024-07-22T16:58:50.616325Z  INFO ipc/observability/src/lib.rs:42: domain="Proposals" event=BlockCommitted { height: 175, app_hash: 0171a0e40220093a9e7b9d7c6bb3e540710b560f6f5d9baccc51b5dc257be127c8926fc99f66 }

File:

{"timestamp":"2024-07-22T16:58:47.453943Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 172, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.473650Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 172, hash: ac51519dbb9241bd40ff1ffea0cd89f2a0c97e123f47a26545e253e17fe85439, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.473673Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 172, hash: ac51519dbb9241bd40ff1ffea0cd89f2a0c97e123f47a26545e253e17fe85439, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:47.512034Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 172, app_hash: 0171a0e40220ef7f37a1c3ea1c58f78fecc94f591551222c2a4bd9e3876346268e0f53a9d7b3 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.490986Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 173, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.508260Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.508535Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 173, hash: f87b67959f9c2f4119cfa2921ad8d121f0c23d7aa57fe0e363cf42a4f42fb36c, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:48.544060Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 173, app_hash: 0171a0e40220d36a47bfd38c81b0acd8a87a5d473875dc31cdbba81b5d3157b2ae4c009a81f4 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.527006Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 174, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.547372Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.547420Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 174, hash: 466a6d4efbd7cd78553e493844c69a661c917d30a25dd56a42e3a02539c24014, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:49.582133Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 174, app_hash: 0171a0e402206e947f59c0f0746fd8dacd2d4c8eb30055233ce89f93bd68e86194e8d83a761d }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.565155Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalSent { validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), height: 175, size: 0, tx_count: 0 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.582163Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalReceived { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41) }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.582304Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockProposalEvaluated { height: 175, hash: 4696b5a37bf003d1f4468059bf96c34a7d79a0c93203601aadadf921c17f782a, size: 0, tx_count: 0, validator: account::Id(44D6E9254B04951164143C6D688C2036FA593F41), accept: true, reason: None }"},"filename":"ipc/observability/src/lib.rs","line_number":42}
{"timestamp":"2024-07-22T16:58:50.616333Z","level":"INFO","fields":{"domain":"Proposals","event":"BlockCommitted { height: 175, app_hash: 0171a0e40220093a9e7b9d7c6bb3e540710b560f6f5d9baccc51b5dc257be127c8926fc99f66 }"},"filename":"ipc/observability/src/lib.rs","line_number":42}

karlem · 2024-07-22T17:11:01Z

@raulk Please see the output above. I have only set up a simple node without a connection to a parent, so it only includes consensus metrics/traces. Additionally, I took a random sample, so don't worry if they don't look coherent (for instance, they all refer to different block heights, etc.).

karlem changed the base branch from main to new-observability July 17, 2024 21:17

karlem mentioned this pull request Jul 18, 2024

feat(node): new observability architecture + events #1053

Merged

karlem force-pushed the new-observability-configuration branch from 77adbfc to 9f8f27a Compare July 18, 2024 22:21

raulk changed the title ~~New observability configuration~~ feat(node): file-based observability configuration Jul 19, 2024

karlem force-pushed the new-observability-configuration branch from 9f8f27a to 378d72c Compare July 19, 2024 13:41

Base automatically changed from new-observability to main July 19, 2024 13:41

raulk reviewed Jul 19, 2024

View reviewed changes

karlem and others added 23 commits July 22, 2024 15:45

feat: introduce metrics crate

ccde386

feat: improve library with macros, add top down events

6facf63

feat: emit metrics

1dcf517

feat: add remaining top down events

d34590d

feat: remove unused code & fix build

75e45e4

feat: address comments

c2465bc

feat: use latency wrapper instead

599bcc3

feat: add config filters

e80f3df

feat: add config as a cmd flags

79de523

feat: fix comments

2f40eef

feat: add consensus traces

a97b16d

feat: add execution metrics

f16f8c5

feat: add proposals metrics & reason

a164472

feat: add mempool event

1f5a38a

feat: remove

da84485

feat: address comments

c1ded52

feat: address comments

59f3ea7

feat: fix clippy

13bb9c6

minor cleanup.

d3b22d6

feat: fix typo

094e9b6

feat: standartize mpool trace

27e7e36

feat: address comments

e6e79f3

lint: clippy

01843f0

karlem added 5 commits July 22, 2024 16:01

feat: revert risky changes & small cleanup

6f67fe1

feat: use tracing config

4295027

feat: add tracing to each command

b77bc36

feat: introduce cleanups

edb7dc2

feat: adress comments

4e7d1c6

karlem force-pushed the new-observability-configuration branch from cfbeab6 to 4e7d1c6 Compare July 22, 2024 14:04

karlem added 2 commits July 22, 2024 16:25

feat: fix after rebase

ad1bd14

fix: rebase && comments

c544c46

karlem marked this pull request as ready for review July 22, 2024 15:59

raulk approved these changes Jul 22, 2024

View reviewed changes

fendermint/app/src/cmd/rpc.rs Outdated Show resolved Hide resolved

fendermint/vm/topdown/src/proxy.rs Show resolved Hide resolved

fix: select proper default

22a065a

feat: setup tracing in single place

68b2dbc

fix: do no register tracing twice in eth

73a4f4d

karlem merged commit c5e6b07 into main Jul 23, 2024

karlem deleted the new-observability-configuration branch July 23, 2024 10:01

Conversation

karlem commented Jul 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

karlem commented Jul 22, 2024

Uh oh!

raulk commented Jul 22, 2024

Uh oh!

raulk commented Jul 22, 2024

Uh oh!

karlem commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karlem commented Jul 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karlem commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

karlem commented Jul 17, 2024 •

edited

Loading

karlem commented Jul 22, 2024 •

edited

Loading

karlem commented Jul 22, 2024 •

edited

Loading