Backend: Reliable deployment status and audit trail pipeline #149
Conversation
…lled) Co-authored-by: ludovit-scholtz <[email protected]>
Co-authored-by: ludovit-scholtz <[email protected]>
Co-authored-by: ludovit-scholtz <[email protected]>
Co-authored-by: ludovit-scholtz <[email protected]>
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
4 similar comments
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
|
This PR is still marked as draft and does not have an approval, so it is not ready to merge yet. CI is green, but we still need a clearer TDD trail and explicit verification of the new behavior. Please add or expand unit tests for deployment status transitions, audit trail event ordering, and idempotency when retries occur. Include integration tests that simulate a full deployment lifecycle, including failure and recovery paths, and verify that the audit trail remains consistent and queryable. Make sure the tests validate any new API responses and error contracts so downstream systems can rely on them. If the audit pipeline involves async jobs, cover concurrency and timing edge cases to prevent missing or duplicate events. Please link this PR to the issue that explains the business value and risk for reliable deployment status, including how it reduces operational overhead and improves customer trust. Add a test plan to the PR description that names the commands used and the scenarios covered. Also confirm that any database migrations or configuration changes are documented for rollout. Once the PR is marked ready, request review and secure the required approval so we can proceed with a squash merge. Please add unit/integration tests, link to issue explaining business value/risk, and ensure CI remains green. @copilot |
…tion Co-authored-by: ludovit-scholtz <[email protected]>
Added comprehensive test coverage addressing all requested scenarios (commit 6a39e74): Tests Added (28 new tests, 1,188 total passing):
Test Plan (
Database/Config:
CI Status: ✅ All 1,188 tests passing PR Status: Ready for approval and merge. |
|
✅ CI checks passed! 📄 OpenAPI specification generated and available as workflow artifact: |
There was a problem hiding this comment.
Pull request overview
This pull request implements a comprehensive deployment status tracking and audit trail system for the BiatecTokensApi, adding deterministic state machine transitions, structured error handling, and compliance-ready audit exports to meet MICA readiness requirements.
Changes:
- Enhanced state machine with two new states (Indexed, Cancelled) and formalized transition rules
- Implemented structured error handling with 9 categorized error types (NetworkError, ValidationError, ComplianceError, etc.)
- Added audit trail export functionality in JSON/CSV formats with idempotent operations
- Created deployment metrics endpoint for monitoring success rates, durations, and failure analytics
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| TEST_PLAN.md | Comprehensive test plan documenting 28 new tests covering all state transitions, audit trail ordering, and error handling |
| PERFORMANCE_OPTIMIZATION_NOTES.md | Documents N+1 query patterns and future optimization opportunities for database migration |
| DEPLOYMENT_STATUS_PIPELINE.md | Complete API documentation with state machine diagrams, usage examples, and integration guides |
| BiatecTokensApi/Models/DeploymentStatus.cs | Adds Indexed and Cancelled enum values with extended status entry fields for compliance tracking |
| BiatecTokensApi/Models/DeploymentMetrics.cs | New metrics model for deployment analytics and SLA tracking |
| BiatecTokensApi/Models/DeploymentErrorCategory.cs | Structured error categorization with factory methods for consistent error creation |
| BiatecTokensApi/Models/AuditExport.cs | Models for audit trail export requests and results |
| BiatecTokensApi/Services/DeploymentStatusService.cs | Enhanced service with metrics calculation, cancellation support, and structured error handling |
| BiatecTokensApi/Services/DeploymentAuditService.cs | New service implementing audit trail export with idempotent caching |
| BiatecTokensApi/Services/Interface/IDeploymentStatusService.cs | Interface extensions for new operations |
| BiatecTokensApi/Services/Interface/IDeploymentAuditService.cs | New interface for audit export functionality |
| BiatecTokensApi/Controllers/DeploymentStatusController.cs | Four new endpoints: cancel, audit-trail export (single/bulk), and metrics |
| BiatecTokensApi/Program.cs | Registers DeploymentAuditService as singleton |
| BiatecTokensTests/* | 28 new comprehensive tests achieving 100% test coverage for new functionality |
| { | ||
| data = await _auditService.ExportAuditTrailAsCsvAsync(deploymentId); | ||
| contentType = "text/csv"; | ||
| fileName = $"audit-trail-{deploymentId}.csv"; |
There was a problem hiding this comment.
The fileName constructed using string interpolation directly includes the unsanitized deploymentId parameter. This is the same issue as line 380 - consider sanitizing the deploymentId or using a safe filename pattern.
| ``` | ||
| Queued → Submitted → Pending → Confirmed → Indexed → Completed | ||
| ↓ ↓ ↓ ↓ ↓ ↓ | ||
| Failed ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← ← (from any non-terminal state) |
There was a problem hiding this comment.
The comment states "from any non-terminal state", but the state machine implementation shows that Failed transitions are only defined from non-terminal states (Queued, Submitted, Pending, Confirmed, Indexed). However, the documentation diagram shows "from any" which could be misleading. Consider updating the diagram to explicitly state "from any non-terminal state" to match the implementation, or ensure the arrow legend clarifies that terminal states (Completed, Cancelled) are excluded.
| // Get all deployments in the period | ||
| var listRequest = new ListDeploymentsRequest | ||
| { | ||
| Network = request.Network, | ||
| TokenType = request.TokenType, | ||
| DeployedBy = request.DeployedBy, | ||
| FromDate = fromDate, | ||
| ToDate = toDate, | ||
| Page = 1, | ||
| PageSize = 10000 // Get all for metrics | ||
| }; | ||
|
|
||
| var deployments = await _repository.GetDeploymentsAsync(listRequest); | ||
|
|
There was a problem hiding this comment.
The metrics calculation performs N+1 queries by fetching status history separately for each deployment. While this is documented in PERFORMANCE_OPTIMIZATION_NOTES.md as acceptable for the current in-memory implementation, the PageSize is hardcoded to 10000 which could cause performance issues with large datasets. Consider adding a warning log when approaching this limit or implementing pagination for metrics calculation to handle datasets larger than 10000 deployments.
| // Get all deployments in the period | |
| var listRequest = new ListDeploymentsRequest | |
| { | |
| Network = request.Network, | |
| TokenType = request.TokenType, | |
| DeployedBy = request.DeployedBy, | |
| FromDate = fromDate, | |
| ToDate = toDate, | |
| Page = 1, | |
| PageSize = 10000 // Get all for metrics | |
| }; | |
| var deployments = await _repository.GetDeploymentsAsync(listRequest); | |
| // Get all deployments in the period using pagination to avoid hard limits | |
| const int pageSize = 1000; | |
| var allDeployments = new List<Deployment>(); | |
| var page = 1; | |
| while (true) | |
| { | |
| var listRequest = new ListDeploymentsRequest | |
| { | |
| Network = request.Network, | |
| TokenType = request.TokenType, | |
| DeployedBy = request.DeployedBy, | |
| FromDate = fromDate, | |
| ToDate = toDate, | |
| Page = page, | |
| PageSize = pageSize | |
| }; | |
| var pageDeployments = await _repository.GetDeploymentsAsync(listRequest); | |
| if (pageDeployments == null || pageDeployments.Count == 0) | |
| { | |
| break; | |
| } | |
| allDeployments.AddRange(pageDeployments); | |
| // If fewer than pageSize items are returned, we've reached the last page | |
| if (pageDeployments.Count < pageSize) | |
| { | |
| break; | |
| } | |
| page++; | |
| } | |
| var deployments = allDeployments; |
| private readonly Dictionary<string, AuditExportCache> _exportCache = new(); | ||
| private readonly object _cacheLock = new(); |
There was a problem hiding this comment.
The in-memory cache implementation using a Dictionary is not thread-safe for concurrent modifications outside of the lock blocks. While reads and writes are protected by locks, the cache could grow unbounded over time as expired entries are only removed when the same idempotency key is reused. Consider adding a background cleanup task or implementing a time-based eviction strategy to prevent memory leaks in long-running services.
| var hasRetry = history.Any(e => e.Status == DeploymentStatus.Queued && | ||
| history.Any(h => h.Status == DeploymentStatus.Failed && h.Timestamp < e.Timestamp)); |
There was a problem hiding this comment.
The retry count calculation has a nested Any() call within the outer Any() predicate, which results in O(n²) complexity for the history list. For deployments with long histories, this could cause performance degradation. Consider restructuring the logic to first identify failed entries and their timestamps, then check for subsequent queued entries in a single pass.
| var hasRetry = history.Any(e => e.Status == DeploymentStatus.Queued && | |
| history.Any(h => h.Status == DeploymentStatus.Failed && h.Timestamp < e.Timestamp)); | |
| // Determine if this deployment has been retried by checking for any Queued status | |
| // that occurs after at least one Failed status. This avoids nested LINQ calls over | |
| // the same collection to keep the complexity linear in the history length. | |
| var failedEntries = history | |
| .Where(h => h.Status == DeploymentStatus.Failed) | |
| .ToList(); | |
| bool hasRetry = false; | |
| if (failedEntries.Count > 0) | |
| { | |
| var earliestFailedTimestamp = failedEntries.Min(h => h.Timestamp); | |
| hasRetry = history.Any(e => | |
| e.Status == DeploymentStatus.Queued && | |
| e.Timestamp > earliestFailedTimestamp); | |
| } |
| foreach (var id in ids) | ||
| { | ||
| var dep = await _service.GetDeploymentAsync(id); | ||
| if (dep!.CurrentStatus == DeploymentStatus.Completed) completedCount++; | ||
| if (dep!.CurrentStatus == DeploymentStatus.Failed) failedCount++; | ||
| } |
There was a problem hiding this comment.
This foreach loop immediately maps its iteration variable to another variable - consider mapping the sequence explicitly using '.Select(...)'.
| e.Metadata.ContainsKey("errorCategory") && | ||
| e.Metadata["errorCategory"].ToString() == "NetworkError")), Times.Once); |
There was a problem hiding this comment.
Inefficient use of 'ContainsKey' and indexer.
| e.Metadata.ContainsKey("errorCategory") && | |
| e.Metadata["errorCategory"].ToString() == "NetworkError")), Times.Once); | |
| e.Metadata.TryGetValue("errorCategory", out var errorCategory) && | |
| errorCategory?.ToString() == "NetworkError")), Times.Once); |
| await isolatedService.UpdateDeploymentStatusAsync(id2, DeploymentStatus.Submitted); | ||
| await isolatedService.MarkDeploymentFailedAsync(id2, "Test failure"); | ||
|
|
||
| var id3 = await isolatedService.CreateDeploymentAsync("ASA", "testnet-v1.0", "0x3", "T3", "T3"); |
There was a problem hiding this comment.
This assignment to id3 is useless, since its value is never read.
| var id3 = await isolatedService.CreateDeploymentAsync("ASA", "testnet-v1.0", "0x3", "T3", "T3"); | |
| await isolatedService.CreateDeploymentAsync("ASA", "testnet-v1.0", "0x3", "T3", "T3"); |
| if (request.Format == AuditExportFormat.Json) | ||
| { | ||
| data = await ExportMultipleDeploymentsAsJsonAsync(deployments); | ||
| } | ||
| else | ||
| { | ||
| data = await ExportMultipleDeploymentsAsCsvAsync(deployments); | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
| if (request.Format == AuditExportFormat.Json) | |
| { | |
| data = await ExportMultipleDeploymentsAsJsonAsync(deployments); | |
| } | |
| else | |
| { | |
| data = await ExportMultipleDeploymentsAsCsvAsync(deployments); | |
| } | |
| data = request.Format == AuditExportFormat.Json | |
| ? await ExportMultipleDeploymentsAsJsonAsync(deployments) | |
| : await ExportMultipleDeploymentsAsCsvAsync(deployments); |
| if (count % 2 == 0) | ||
| { | ||
| return (sorted[count / 2 - 1] + sorted[count / 2]) / 2; | ||
| } | ||
| else | ||
| { | ||
| return sorted[count / 2]; | ||
| } | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement return - consider using '?' to express intent better.
| if (count % 2 == 0) | |
| { | |
| return (sorted[count / 2 - 1] + sorted[count / 2]) / 2; | |
| } | |
| else | |
| { | |
| return sorted[count / 2]; | |
| } | |
| } | |
| return count % 2 == 0 | |
| ? (sorted[count / 2 - 1] + sorted[count / 2]) / 2 | |
| : sorted[count / 2]; | |
| } | |
| } |
Problem
MVP deployment tracking lacks deterministic state transitions and complete audit trails, causing inconsistent UI feedback and preventing compliance claims (MICA readiness).
Changes
State Machine Enhancement
Added two states and formalized transitions:
Structured Error Handling
9 categorized error types with retry guidance:
Categories: NetworkError, ValidationError, ComplianceError, UserRejection, InsufficientFunds, TransactionFailure, ConfigurationError, RateLimitExceeded, InternalError
Audit Trail Export
JSON/CSV export with idempotency:
Deployment Metrics
Returns:
New Endpoints
POST /deployments/{id}/cancel- Cancel queued deploymentsGET /deployments/{id}/audit-trail?format=json|csv- Single deployment exportPOST /deployments/audit-trail/export- Bulk export with filtersGET /deployments/metrics- Operational analyticsImplementation Notes
Test Plan
Comprehensive test coverage added per code review feedback:
Test Commands
Test Scenarios Covered
State Machine Transitions (15+ tests)
Audit Trail Event Ordering (10 integration tests)
Idempotency (5+ tests)
Full Deployment Lifecycle (3+ integration tests)
Concurrency & Edge Cases (2+ tests)
Error Categorization (10 tests)
API Contract Validation (6+ tests)
Test Results
New Tests Added:
DeploymentLifecycleIntegrationTests.cs- 10 comprehensive integration testsDeploymentErrorTests.cs- 10 error categorization testsDeploymentAuditServiceTests.cs- 8 audit export testsDocumentation
Complete test plan available in
TEST_PLAN.mdwith detailed scenario coverage, edge cases, and CI/CD integration instructions.Database & Configuration
Acceptance Criteria Met
✓ Deterministic state machine with documented transitions
✓ Idempotent processing prevents duplicate history entries (verified in 3 separate tests)
✓ API exposes current status and full transition history
✓ Audit trail includes timestamp, network, tx hash, wallet, compliance data
✓ Errors categorized with user-safe messages and retry guidance (9 categories tested)
✓ Export-ready audit trails (JSON/CSV formats tested)
✓ No API regressions (1,188 tests passing)
✓ All features covered by comprehensive integration tests
Original prompt
This section details on the original issue you should resolve
<issue_title>Backend: Reliable deployment status and audit trail pipeline</issue_title>
<issue_description>## Summary
This issue focuses on delivering a reliable backend deployment status and audit trail pipeline that supports the frontend’s real-time token deployment experience and compliance expectations. The MVP currently has partial status tracking and incomplete audit trails, which leads to inconsistent UI feedback and undermines the product’s claim of enterprise-grade compliance. The backend must provide deterministic, observable, and well-tested status updates for token deployment across supported networks, with structured audit logs that can be exported and verified.
The work enables the platform’s business vision by ensuring that every deployment is traceable, verifiable, and compliant. It upgrades the backend from “best effort” event handling to a robust pipeline with clear state transitions, idempotent processing, and accurate API responses. This is foundational to the product’s compliance badges and regulatory readiness claims.
Business Value
Enterprise and regulated customers demand proof that token issuance steps are recorded, auditable, and reproducible. Without a reliable deployment status and audit trail system, the platform cannot credibly claim MICA readiness or meet the expectations of compliance teams. This issue turns the backend into a source of truth, enabling the UI to display trustworthy status and enabling exportable audit records for regulatory review.
Reliable status tracking is directly linked to customer trust and conversion. If a user initiates a deployment and the UI shows “pending” indefinitely or flips between states, the user assumes the platform is broken and abandons the process. That abandonment directly reduces activation and subscription conversion, especially for paid tiers where compliance is the primary value proposition.
From a market positioning standpoint, the platform aims to be the go-to solution for compliant multichain RWA tokenization. The backend must demonstrate maturity comparable to enterprise systems. A consistent audit trail and deterministic deployment state transitions are critical differentiators against competitors who only provide minimal token creation APIs without robust compliance logs.
Operationally, a strong status pipeline reduces internal support costs and incident response time. A structured audit log enables faster root-cause analysis for failed deployments and reduces the time engineers spend reconstructing transaction histories. It also makes it easier to provide proactive customer support and data-driven product improvements.
Finally, this issue supports long-term revenue by enabling premium features like compliance reporting, audit exports, and SLA-backed reliability. Those features cannot be sold without a backend foundation that is precise, testable, and resilient to network variability.
Product overview
Reference: https://raw.githubusercontent.com/scholtz/biatec-tokens/refs/heads/main/business-owner-roadmap.md
Scope
In scope is the design and implementation of a deployment status pipeline and audit trail system that supports all currently supported networks. This includes:
The technical approach should build on existing backend architecture and avoid new dependencies unless they are necessary for event processing or indexing. The system should remain compatible with existing database schemas, but migrations are acceptable if required for correctness and auditability. Any new tables or fields should be documented with clear ownership and lifecycle.
Dependencies include access to network RPC endpoints, existing transaction monitoring logic, and any current audit logging framework. This work should remain backward compatible with the current frontend while adding richer data fields that the UI can optionally adopt.
Out of scope includes implementing new token standards, KYC provider integration, or advanced compliance analytics beyond the audit trail. This issue is not a rewrit...
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.