Having guided multiple enterprises through both upgrade paths, I can provide a comprehensive comparison based on your specific concerns.
Test Plan Migration - On-Premises vs Cloud:
For on-premises ado-2024 upgrades, test plan migration is more straightforward because you maintain the same database structure. Your 15,000 test cases will migrate with full fidelity, and the upgrade process preserves test suite hierarchies automatically. Cloud migration requires a data export/import process that can introduce subtle changes to test plan organization.
Custom Fields Mapping:
This is where the paths diverge significantly. On-premises upgrades preserve custom field configurations almost entirely - you’ll need to verify field types and update any custom rules, but the fields themselves transfer cleanly. Cloud migration requires explicit field mapping because Azure DevOps Services has different field name restrictions and type limitations. Calculated fields and complex validation rules need to be recreated using the cloud API.
For your regulatory compliance custom fields, document the exact field definitions before migration. Both paths support the fields, but cloud migration may require renaming fields that conflict with reserved names in Azure DevOps Services.
Historical Data Preservation:
Your 8 years of execution history is the most critical asset. On-premises upgrades maintain this data in place with no risk of loss. The database upgrade scripts in ado-2024 specifically preserve test result history and execution attachments. Cloud migration uses the data migration tool which successfully transfers test execution history, but with two caveats:
- Very large attachments (>100MB per test result) may need separate handling
- Execution history older than 5 years may require additional migration time and validation
Both paths preserve the history, but on-premises is lower risk for large historical datasets.
Hybrid Upgrade Strategy:
Your staged approach is the most prudent path for your situation. Here’s the recommended timeline:
Phase 1 - On-Premises Upgrade (Months 1-2):
- Upgrade to ado-2024 on-premises
- Run comprehensive validation reports on test case integrity
- Verify custom field functionality and traceability links
- Allow your QA team to adapt to any UI or workflow changes
Phase 2 - Validation Period (Months 3-5):
- Execute full test cycles using the upgraded system
- Generate audit reports to confirm compliance requirements are met
- Document any custom field or workflow adjustments needed
- Build cloud migration scripts for any custom configurations
Phase 3 - Cloud Migration (Month 6+):
- Use Azure DevOps data migration tool with your validated configuration
- Migrate in stages (test plans by product area or team)
- Run parallel operations for 2-4 weeks to ensure no data gaps
Validation Reports:
Azure DevOps 2024 includes enhanced validation reporting specifically for test management upgrades. The most valuable reports for your scenario:
- Test Case Integrity Report - Validates all test steps, expected results, and attachments migrated correctly
- Traceability Link Validation - Confirms all requirement-to-test-case links are preserved
- Custom Field Mapping Report - Shows which fields migrated cleanly vs. which need manual review
- Historical Data Completeness - Verifies execution history coverage across date ranges
- Test Suite Hierarchy Report - Ensures parent-child test suite relationships are intact
Run these reports immediately after each phase and again after 30 days of active use.
Test Traceability Considerations:
Your audit process requirements make traceability preservation critical. Both upgrade paths maintain work item links, but validate these specifically:
- Requirement-to-test-case links
- Test-case-to-bug links
- Test-result-to-build links
- Custom link types if you’ve defined any
The on-premises upgrade preserves these automatically. Cloud migration requires explicit verification because the work item IDs change during migration (though the tool maintains link relationships).
Recommendation:
Given your 15,000 test cases, 8 years of history, regulatory compliance needs, and complex custom fields, the hybrid approach is optimal. The on-premises upgrade to ado-2024 gives you a stable validation platform with minimal migration risk. After 3-6 months of validated operation, you can migrate to cloud with confidence that your test management data integrity is proven.
The additional 3-6 months may seem like a delay, but it significantly reduces the risk of data loss or compliance gaps that could impact your audit processes. The validation period also allows your team to identify any custom field or workflow issues in the lower-risk on-premises environment before introducing cloud-specific constraints.