feat: Infrastructure and CI/CD for test environment

[BenGWeeks]

Summary

• Set up test environment (TMinus15Agents-Test) with separate infrastructure
• Implement CI/CD workflow with GitHub Actions
• Add Spot VM support for cost savings (60-80% cheaper)
• Fix VM deployment script issues with sequential command approach

Issues Addressed

Fixes #60 - CI/CD workflow with Git tag releases

• Push to main → Deploy to test (when AZURE_DEPLOYMENT_ENABLED=true)
• Tag v*.*.* → Auto-deploy to production
• workflow_dispatch → Manual deploy to test or prod

Fixes #59 - Set up test environment (TMinus15Agents-Test)

• Test resource group with separate infrastructure
• Test Teams manifest (manifest.test.json) with purple accent (#9C27B0)
• Test parameters file (main.parameters.test.json)
• GitHub environments configured (test, prod) with scoped secrets
• Spot VM support for cost savings

Does NOT fix (out of scope)

• feat: Set up local development workflow for Teams bot #61 - Local development workflow (Dev Tunnel, Bot Emulator setup)
• Expand unit test coverage for Teams bot audio and meeting handling #33 - Expand unit test coverage
• Unit & Integration Test Suite #6 - Unit & Integration Test Suite (some infrastructure added but not comprehensive)

Still TODO before merge

• Verify test bot health endpoint responds
• Configure appsettings.json on test VM (Teams credentials, etc.)
• Test Teams bot messaging end-to-end in test environment
• Verify tag-based production deployment works

Key Changes

• .github/workflows/deploy.yml - Full CI/CD pipeline with 6-step VM deployment
• infra/main.parameters.test.json - Test environment with Spot VM
• infra/main.bicep - Optional AI services and VM deployment
• bot/teams-manifest/manifest.test.json - Test Teams app manifest
• Various deployment script fixes for Azure VM run-command

Test Plan

• Manual workflow_dispatch to test environment
• Infrastructure deployment succeeds
• Bot build and publish succeeds
• VM deployment extracts files correctly
• Health endpoint responds at https://pennie-test-vgn7kzlubtavo.uksouth.cloudapp.azure.com/health
• Production tag deployment (after merge)

🤖 Generated with Claude Code

[BenGWeeks]

🔍 Test Bot Investigation Status (Dec 5, 2025)

Current Issue

The test bot pennie-bot-test returns InternalServerError when messaged via Azure Portal "Test in Web Chat" or Direct Line API.

Key Discovery: Production vs Test Credential Mystery

Production Bot (pennie-bot in TMinus15Agents):

• Azure Bot Service App ID: 9707c142-2583-4e56-9983-4d913338afb0
• VM appsettings.json: ALL credentials are EMPTY ("")
• No appsettings.local.json, no env vars, no NSSM parameters
• Yet it WORKS perfectly! 🤔

Test Bot (pennie-bot-test in TMinus15Agents-Test):

• Azure Bot Service App ID: 131b79ec-a659-4b35-aaf8-92185d97e457
• VM appsettings.json: Tried both WITH credentials (fresh secret) and WITHOUT (empty)
• Both fail with authentication errors

Error Progression

1. With credentials → 401 Unauthorized when bot tries to REPLY to Bot Framework

   • Error at ConversationsExtensions.ReplyToActivityAsync
   • Bot receives message fine, but fails when sending response

2. Without credentials (matching production) → Invalid AppId passed on token

   • Error at ChannelValidation.AuthenticateChannelToken
   • Bot rejects INCOMING requests because it can't validate JWT token

What We've Tried

• ✅ Changed test app from SingleTenant → MultiTenant → back to SingleTenant
• ✅ Created fresh client secret for test app
• ✅ Updated test VM appsettings.json with fresh credentials
• ✅ Cleared all credentials to match production's empty config
• ✅ Verified NSSM service config matches production
• ✅ Confirmed both bots are SingleTenant in Azure Bot Service

Configuration Verified

Setting	Production	Test
Bot Service App Type	SingleTenant	SingleTenant
Tenant ID	f36f6414-cb7d-4545-9cf2-7574f7b5c584	f36f6414-cb7d-4545-9cf2-7574f7b5c584
App Registration signInAudience	AzureADMyOrg	AzureADMyOrg
VM Credentials	Empty	Empty (now)
NSSM AppParameters	None	None

The Mystery

Production bot works with empty credentials, meaning it somehow:

1. Accepts incoming JWT tokens without validation
2. Sends replies without authentication headers

But the test bot with identical empty credentials fails on incoming token validation.

What Still Needs Investigation

1. How does production authenticate? We need to understand why production works with no visible credentials
2. Binary differences? Was production deployed with a different build that has credentials embedded?
3. Azure Bot Service configuration? Is there some setting that disables authentication validation?
4. Process memory? Was production started manually with credentials that remain in memory but aren't saved to disk?

Recommended Next Steps

1. Check if production bot was originally deployed with credentials, then appsettings.json was manually cleared later (bot keeps credentials in memory until restart)
2. Restart production bot and verify it still works (would prove credentials are in memory)
3. Create a fresh credential for test app and properly configure it in both:
   • Test VM appsettings.json
   • GitHub Secret TEAMS_APP_PASSWORD_TEST
4. Ensure the credential provider in the bot code recognizes the app ID correctly

Files Modified

• Test VM C:\Pennie\bot\appsettings.json (credentials cleared)
• GitHub Secret TEAMS_APP_PASSWORD_TEST (updated with fresh secret)

Commands for Continuation

# Check test bot logs
AZURE_RESOURCE_GROUP=TMinus15Agents-Test VM_NAME=pennie-vm-test ./scripts/bot-logs.sh

# Test bot via Direct Line
BOT_NAME=pennie-bot-test AZURE_RESOURCE_GROUP=TMinus15Agents-Test ./tests/bot-direct-line.sh "Hello" 20

# Update test VM credentials
az vm run-command invoke --resource-group TMinus15Agents-Test --name pennie-vm-test \
  --command-id RunPowerShellScript --scripts '<PowerShell to update appsettings.json>'

[claude]

PR Review: Infrastructure and CI/CD for Test Environment

This is a substantial PR that sets up a complete test environment and CI/CD pipeline. Overall, the implementation is solid, but there are several security concerns, potential bugs, and areas for improvement.

---

Critical Issues

1. Secret Exposure Risk in Deployment Script (.github/workflows/deploy.yml:345-355)

Severity: High

The Base64-encoded secrets are being logged. While you've added add-mask for the original values, the Base64-encoded versions are NOT masked. An attacker could potentially decode these from logs.

Recommendation: Remove these debug logging statements or ensure Base64 values are also masked.

2. PFX Password Stored in Plain Text (scripts/configure-ssl.ps1:34-35)

Severity: High

The PFX password is stored in a plain text file at C:\Pennie\certs\pfx-password.txt. Any user with access to the VM can read this password and extract the private key.

Recommendation: Use Windows Data Protection API (DPAPI) to encrypt the password file, or store the password in Azure Key Vault, or use certificate stores without exporting PFX files.

3. Anonymous Backend Authentication (CLAUDE.md:39)

Per CLAUDE.md: "Anonymous authentication (no API keys)" for the Azure Functions backend. This means anyone who discovers the backend URL can call your Azure DevOps integration functions.

Recommendation: Add function-level authentication using function keys or managed identity.

---

Medium Priority Issues

4. Error Handling in Deployment Steps (.github/workflows/deploy.yml:338-445)

The 7-step VM deployment process doesn't have robust error handling. If step 3 fails, steps 4-7 will still run with potentially corrupt state.

Recommendation: Add error checking between steps.

5. Certificate Renewal Task (scripts/configure-ssl.ps1:194)

The scheduled task runs renewal daily, but doesn't restart the bot service after renewal. The bot will continue using the old certificate until manually restarted.

Recommendation: Add a post-renewal script that restarts the PennieBot service.

6. Hardcoded Timeout Values (bot/Helpers/MeetingHelpers.cs:11)

100ms regex timeout might be too aggressive for complex but legitimate inputs. Consider making this configurable or increasing to 500ms-1s.

7. VM FQDN Resolution Inconsistency (.github/workflows/deploy.yml:387-392)

Good error handling in SSL step, but the deploy-bot step uses echo empty string and continues. Inconsistent behavior.

---

Positive Highlights

Excellent Code Organization

• Extracting MeetingHelpers from MediaBot.cs is great refactoring
• Separation of concerns is clean
• Using InternalsVisibleTo for testing is the right approach

Comprehensive Unit Tests (tests/Helpers/MeetingHelpersTests.cs)

• 167 lines of well-structured tests
• Good use of Theory/InlineData for parameterized tests
• Edge cases covered (null, empty, whitespace, regex timeouts)
• FluentAssertions makes tests readable

Security-Conscious Regex Usage

• All regex operations have timeouts to prevent ReDoS
• Catching RegexMatchTimeoutException and returning safe defaults
• This is production-ready security practice

Spot VM for Cost Savings

• 60-80% cost savings for test environment is smart
• Properly documented in CLAUDE.md

Environment Separation (.github/workflows/deploy.yml:30-73)

• Clean logic for determining environment (manual/tag/push)
• Separate resource groups prevent test/prod collision
• Good use of GitHub environment-scoped secrets

Comprehensive Deployment Steps

• 7-step VM deployment is thorough
• SSL certificate setup with Let's Encrypt
• Automatic renewal configured
• Bot configuration management

---

Recommendations & Best Practices

9. Add Rollback Capability

The notification step mentions rollback but doesn't implement it. Consider storing previous deployment artifact in Azure Storage or adding a manual rollback workflow.

10. Improve Health Check Robustness

The health check should also verify: Bot can authenticate with Azure OpenAI, Bot can reach Azure Functions backend, Certificate is valid and not self-signed.

11. Add Integration Tests

Currently only endpoint connectivity tests exist. Consider adding: Test Teams message sending/receiving, Test meeting join flow, Test Azure DevOps work item creation.

12. Documentation: Spot VM Limitations

Spot VMs can be evicted with 30 seconds notice. Document this risk in DEPLOYMENT.adoc.

13. Consider Using Managed Certificates

Instead of Let's Encrypt + win-acme, consider Azure App Service Managed Certificates or Azure Key Vault certificates with auto-rotation.

---

Test Coverage Assessment

Good Coverage:

• MeetingHelpers: 100% coverage (all methods tested)
• Regex edge cases well covered
• Null safety tested

Missing Coverage:

• Integration tests for deployment scripts
• End-to-end Teams bot messaging tests
• SSL certificate configuration tests
• Azure Functions backend tests (mentioned in issues 33 and 6)

---

Minor Issues

15. Magic Numbers (bot/Helpers/MeetingHelpers.cs:56)

Add a comment explaining why 9-29: "Teams meeting IDs are 10-30 characters including spaces"

16. Inconsistent Error Handling (scripts/configure-bot-settings.ps1)

Lines 48-65 have good null/empty checks, but lines 104-112 assume Azure OpenAI values are valid without validation.

---

Summary

Category	Count
Critical	3
Medium	5
Recommendations	6
Positives	6

Overall Assessment: This is a well-structured PR with excellent code organization and comprehensive testing for the helpers. However, the security concerns around secret handling and certificate storage must be addressed before merging to production. The test environment deployment is safe to merge after addressing the secret logging issue.

Recommendation:

1. Fix critical security issues (secrets logging, PFX password storage)
2. Add backend authentication
3. Improve error handling in deployment steps
4. Merge to test environment first
5. Validate end-to-end before production tag

---

Actionable Next Steps

1. Before Merge:

   • Mask Base64-encoded secrets in logs
   • Encrypt or move PFX password to Key Vault
   • Add error handling between deployment steps

2. After Merge (can be follow-up PRs):

   • Add backend authentication (new issue)
   • Implement rollback mechanism (new issue)
   • Add integration tests (ties into issues 33 and 6)
   • Document Spot VM limitations

---

Great work on this comprehensive infrastructure setup! The code quality is high, especially the refactoring and testing. Address the security concerns and this will be production-ready.