Skip to content

Commit b35c605

Browse files
prkhrkatprkhrkat
committed
corrected specs
1 parent cdd8421 commit b35c605

File tree

16 files changed

+5213
-0
lines changed

16 files changed

+5213
-0
lines changed

‎tests/api-spec-validation/Makefile‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
.PHONY: build test clean help
2+
3+
# Default target
4+
help:
5+
@echo "Available targets:"
6+
@echo " build - Build the API spec validator"
7+
@echo " test - Run API spec validation tests"
8+
@echo " compare - Compare specs with REST handlers"
9+
@echo " clean - Clean build artifacts"
10+
@echo " help - Show this help message"
11+
12+
# Build the validator
13+
build:
14+
@echo "Building API spec validator..."
15+
go build -o bin/validator cmd/validator/main.go
16+
go build -o bin/live-test cmd/live-test/main.go
17+
18+
# Run API spec validation tests
19+
test: build
20+
@echo "Running API spec validation tests..."
21+
@mkdir -p reports
22+
./bin/validator \
23+
--server=http://localhost:8080 \
24+
--specs=../../../specs \
25+
--output=./reports \
26+
--verbose
27+
28+
# Run with custom server URL
29+
test-server: build
30+
@echo "Running API spec validation tests against custom server..."
31+
@mkdir -p reports
32+
./bin/validator \
33+
--server=$(SERVER_URL) \
34+
--specs=../../../specs \
35+
--output=./reports \
36+
--verbose
37+
38+
# Test against live Devtron server
39+
test-live: build
40+
@echo "Running API spec validation tests against live Devtron server..."
41+
@mkdir -p reports
42+
./bin/live-test \
43+
--server=https://devtron-ent-2.devtron.info \
44+
--specs=../../specs \
45+
--output=./reports \
46+
--argocd-token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE3NTQwNTE5NjMsImp0aSI6ImQwZjU0OGYyLWIzNDItNGUxNy05MzRhLWU0MzY3ZTE2ZTRlZCIsImlhdCI6MTc1Mzk2NTU2MywiaXNzIjoiYXJnb2NkIiwibmJmIjoxNzUzOTY1NTYzLCJzdWIiOiJhZG1pbiJ9.dbLq_5lnKnUKI55bg3dIkcIdLj5hVUKSwfU95Aajm7g \
47+
--verbose
48+
49+
# Compare specs with REST handlers
50+
compare:
51+
@echo "Comparing API specs with REST handlers..."
52+
go run cmd/compare/main.go \
53+
--specs=../../../specs \
54+
--handlers=../../../api \
55+
--output=./reports
56+
57+
# Clean build artifacts
58+
clean:
59+
@echo "Cleaning build artifacts..."
60+
rm -rf bin/
61+
rm -rf reports/
62+
63+
# Install dependencies
64+
deps:
65+
@echo "Installing dependencies..."
66+
go mod tidy
67+
go get github.com/getkin/kin-openapi/openapi3
68+
go get go.uber.org/zap
69+
70+
# Run all validations
71+
all: deps build test compare
72+
@echo "All validations completed. Check reports/ directory for results."

‎tests/api-spec-validation/PLAN.md‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Comprehensive API Spec Testing and Validation Plan
2+
3+
## Executive Summary
4+
5+
This plan outlines a comprehensive approach to test and validate OpenAPI specifications against the actual Devtron server implementation. The goal is to ensure API specs are accurate, up-to-date, and provide excellent documentation for users.
6+
7+
## Objectives
8+
9+
1. **Validate API Specs**: Ensure OpenAPI specs match actual server behavior
10+
2. **Compare with Code**: Verify specs align with REST handler implementations
11+
3. **Enhance Documentation**: Add realistic examples to improve user experience
12+
4. **Automate Testing**: Integrate validation into CI/CD pipelines
13+
5. **Maintain Quality**: Establish processes for ongoing spec maintenance
14+
15+
## Phase 1: Infrastructure Setup ✅
16+
17+
### 1.1 Testing Framework Creation
18+
- [x] Created `APISpecValidator` for live server testing
19+
- [x] Created `SpecComparator` for code-spec comparison
20+
- [x] Created `SpecEnhancer` for example generation
21+
- [x] Built command-line interface with comprehensive options
22+
- [x] Implemented detailed reporting system
23+
24+
### 1.2 Project Structure
25+
```
26+
tests/api-spec-validation/
27+
├── framework.go # Core validation engine
28+
├── spec_comparator.go # Spec-code comparison
29+
├── spec_enhancer.go # Example generation
30+
├── cmd/
31+
│ └── validator/
32+
│ └── main.go # Main test runner
33+
├── Makefile # Build and test automation
34+
├── README.md # User documentation
35+
└── PLAN.md # This plan document
36+
```
37+
38+
## Phase 2: Spec Discovery and Inventory
39+
40+
### 2.1 Spec Analysis
41+
**Status**: 🔄 In Progress
42+
43+
**Tasks**:
44+
- [ ] Scan all YAML files in `/specs/` directory
45+
- [ ] Create comprehensive inventory of endpoints
46+
- [ ] Map specs to corresponding REST handlers
47+
- [ ] Identify missing specs or handlers
48+
- [ ] Document authentication requirements
49+
50+
**Expected Output**:
51+
```
52+
Spec Inventory Report:
53+
- Total Specs: 25
54+
- Total Endpoints: 150
55+
- Missing Handlers: 5
56+
- Missing Specs: 3
57+
- Authentication Required: 80%
58+
```
59+
60+
### 2.2 Handler Mapping
61+
**Status**: 🔄 In Progress
62+
63+
**Tasks**:
64+
- [ ] Parse all REST handler files in `/api/` directory
65+
- [ ] Extract endpoint paths and methods
66+
- [ ] Map handlers to spec endpoints
67+
- [ ] Identify routing patterns
68+
- [ ] Document handler-spec relationships
69+
70+
## Phase 3: Automated Validation
71+
72+
### 3.1 Live Server Testing
73+
**Status**: ✅ Complete
74+
75+
**Implementation**:
76+
- Validates specs against running Devtron server
77+
- Tests all endpoints with realistic parameters
78+
- Compares actual responses with spec expectations
79+
- Generates detailed validation reports
80+
81+
**Usage**:
82+
```bash
83+
cd tests/api-spec-validation
84+
make test
85+
```
86+
87+
### 3.2 Parameter Validation
88+
**Status**: ✅ Complete
89+
90+
**Features**:
91+
- Required vs optional parameter validation
92+
- Parameter type and format checking
93+
- Query, path, and header parameter validation
94+
- Request body structure validation
95+
96+
### 3.3 Response Validation
97+
**Status**: ✅ Complete
98+
99+
**Features**:
100+
- Status code validation
101+
- Response body structure validation
102+
- Content-Type validation
103+
- Error response validation
104+
105+
## Phase 4: Spec-Code Comparison
106+
107+
### 4.1 Handler Analysis
108+
**Status**: ✅ Complete
109+
110+
**Implementation**:
111+
- Parses Go handler files using AST
112+
- Extracts function signatures and parameters
113+
- Identifies REST handler patterns
114+
- Maps handlers to spec endpoints
115+
116+
### 4.2 Comparison Logic
117+
**Status**: ✅ Complete
118+
119+
**Checks**:
120+
- Missing handler implementations
121+
- Parameter mismatches
122+
- Request/response body handling
123+
- Authentication requirements
124+
125+
## Phase 5: Spec Enhancement
126+
127+
### 5.1 Example Generation
128+
**Status**: ✅ Complete
129+
130+
**Features**:
131+
- Generates realistic request/response examples
132+
- Uses actual server responses when available
133+
- Creates synthetic examples for testing
134+
- Adds comprehensive example scenarios
135+
136+
### 5.2 Documentation Enhancement
137+
**Status**: 🔄 In Progress
138+
139+
**Tasks**:
140+
- [ ] Add detailed descriptions for all endpoints
141+
- [ ] Include usage examples and best practices
142+
- [ ] Document error scenarios and responses
143+
- [ ] Add authentication examples
144+
- [ ] Create troubleshooting guides
145+
146+
## Phase 6: Integration and Automation
147+
148+
### 6.1 CI/CD Integration
149+
**Status**: 🔄 In Progress
150+
151+
**Tasks**:
152+
- [ ] Create GitHub Actions workflow
153+
- [ ] Set up Jenkins pipeline integration
154+
- [ ] Configure automated testing on PRs
155+
- [ ] Implement failure notifications
156+
- [ ] Add test result reporting
157+
158+
**Example GitHub Actions**:
159+
```yaml
160+
name: API Spec Validation
161+
on: [push, pull_request]
162+
jobs:
163+
validate-specs:
164+
runs-on: ubuntu-latest
165+
steps:
166+
- uses: actions/checkout@v3
167+
- name: Run API Spec Validation
168+
run: |
169+
cd tests/api-spec-validation
170+
make test
171+
```
172+
173+
### 6.2 Reporting and Monitoring
174+
**Status**: ✅ Complete
175+
176+
**Features**:
177+
- Detailed Markdown reports
178+
- Success/failure metrics
179+
- Performance analysis
180+
- Issue categorization
181+
- Trend analysis over time
182+
183+
## Phase 7: Maintenance and Improvement
184+
185+
### 7.1 Ongoing Maintenance
186+
**Status**: 📋 Planned
187+
188+
**Processes**:
189+
- [ ] Weekly validation runs
190+
- [ ] Monthly spec reviews
191+
- [ ] Quarterly code-spec alignment checks
192+
- [ ] Continuous improvement based on feedback
193+
194+
### 7.2 Quality Metrics
195+
**Status**: 📋 Planned
196+
197+
**Metrics**:
198+
- Spec accuracy rate
199+
- Handler coverage
200+
- Example completeness
201+
- Documentation quality score
202+
- User satisfaction metrics
203+
204+
## Implementation Timeline
205+
206+
### Week 1-2: Infrastructure ✅
207+
- [x] Set up testing framework
208+
- [x] Create basic validation logic
209+
- [x] Implement reporting system
210+
211+
### Week 3-4: Core Validation ✅
212+
- [x] Live server testing
213+
- [x] Parameter validation
214+
- [x] Response validation
215+
- [x] Spec-code comparison
216+
217+
### Week 5-6: Enhancement
218+
- [ ] Example generation
219+
- [ ] Documentation improvement
220+
- [ ] Advanced validation rules
221+
222+
### Week 7-8: Integration
223+
- [ ] CI/CD integration
224+
- [ ] Automated testing
225+
- [ ] Monitoring setup
226+
227+
### Week 9-10: Optimization
228+
- [ ] Performance optimization
229+
- [ ] Advanced features
230+
- [ ] User feedback integration
231+
232+
## Success Criteria
233+
234+
### Quantitative Metrics
235+
- **Spec Accuracy**: >95% of endpoints pass validation
236+
- **Handler Coverage**: >98% of handlers have corresponding specs
237+
- **Example Coverage**: >90% of endpoints have realistic examples
238+
- **Documentation Quality**: >85% user satisfaction score
239+
240+
### Qualitative Goals
241+
- Improved developer experience
242+
- Reduced API integration issues
243+
- Better user documentation
244+
- Consistent API behavior
245+
246+
## Risk Mitigation
247+
248+
### Technical Risks
249+
1. **Server Availability**: Use multiple test environments
250+
2. **Authentication Issues**: Support multiple auth methods
251+
3. **Performance Impact**: Implement caching and optimization
252+
4. **False Positives**: Fine-tune validation rules
253+
254+
### Process Risks
255+
1. **Maintenance Overhead**: Automate where possible
256+
2. **Spec Drift**: Regular validation runs
257+
3. **User Adoption**: Provide clear documentation and examples
258+
259+
## Resource Requirements
260+
261+
### Development Team
262+
- 1 Backend Developer (2 weeks)
263+
- 1 DevOps Engineer (1 week)
264+
- 1 Technical Writer (1 week)
265+
266+
### Infrastructure
267+
- Test server environment
268+
- CI/CD pipeline access
269+
- Monitoring and alerting tools
270+
271+
### Tools and Dependencies
272+
- Go 1.19+
273+
- OpenAPI 3.0 libraries
274+
- HTTP testing libraries
275+
- Reporting tools
276+
277+
## Next Steps
278+
279+
### Immediate Actions (This Week)
280+
1. [ ] Run initial validation on existing specs
281+
2. [ ] Identify critical gaps and issues
282+
3. [ ] Prioritize fixes based on impact
283+
4. [ ] Set up basic CI/CD integration
284+
285+
### Short-term Goals (Next 2 Weeks)
286+
1. [ ] Complete spec enhancement
287+
2. [ ] Implement advanced validation rules
288+
3. [ ] Create comprehensive documentation
289+
4. [ ] Set up monitoring and alerting
290+
291+
### Long-term Vision (Next Month)
292+
1. [ ] Full automation of spec maintenance
293+
2. [ ] Integration with API documentation tools
294+
3. [ ] User feedback collection system
295+
4. [ ] Continuous improvement process
296+
297+
## Conclusion
298+
299+
This comprehensive plan provides a roadmap for establishing robust API spec validation and enhancement. The framework will ensure that Devtron's API documentation remains accurate, helpful, and up-to-date, significantly improving the developer experience and reducing integration issues.
300+
301+
The modular approach allows for incremental implementation and continuous improvement, ensuring that the system evolves with the codebase and user needs.

0 commit comments

Comments
 (0)