Jenkins vs Azure DevOps pipeline integration for Rally autom

ALM Rally (Broadcom Agile Central)
, , , , , , , ,
1

Our team is evaluating CI/CD pipeline options for Rally automation and we’re debating between Jenkins and Azure DevOps. We currently have about 15 teams using Rally, with plans to scale to 30+ teams over the next year.

The primary use case is automated test result publishing to Rally, along with automated defect creation from failed builds. We’re also interested in bidirectional sync for build status updates.

I’d love to hear from teams that have implemented either solution at scale. Key questions:

  • What Jenkins plugins or Azure connectors have you found most reliable?
  • How does cost analysis compare for enterprise-scale deployments?
  • What POC strategy would you recommend to validate before full rollout?
  • Where are the scaling limits for each platform when dealing with high-frequency Rally API calls?

Any real-world experiences would be helpful as we make this decision.

2

Azure DevOps has native extensions for Rally in the marketplace. The setup is simpler than Jenkins, and if you’re already in the Microsoft ecosystem, the authentication and permissions integrate seamlessly.

Cost-wise, Azure DevOps pricing scales with parallel jobs and users. For 30 teams, you’re probably looking at several thousand per month depending on your build concurrency needs. But you save on infrastructure maintenance overhead.

3

Thanks both. Ryan - do the Azure Rally connectors support real-time bidirectional sync, or is it more of a one-way push from pipeline to Rally? We need Rally work item status changes to trigger pipeline actions in some scenarios.

Priya - when you mention custom scripts, what percentage of your Rally integration is custom versus using the plugin directly?

4

We’ve been running Jenkins with Rally integration for 3 years across 20 teams. The Jenkins Rally plugin works well for basic test result publishing, but we had to build custom scripts for more complex workflows.

Main advantage: Jenkins is free and highly customizable. Downside: you’re maintaining the infrastructure and custom code yourself.

5

For us, about 60% is custom scripting. The Jenkins Rally plugin handles basic test case result publishing, but we wrote Python scripts for defect creation, build-to-user-story linking, and status synchronization. The plugin is a good foundation but limited for advanced scenarios.

For POC strategy, I’d recommend starting with a single team doing just test result publishing. Get that working reliably for a month, then expand to defect automation with a second team. Don’t try to build everything at once.

6

The Azure marketplace extensions are primarily one-way (pipeline to Rally). For bidirectional sync, you’d need to set up Rally webhooks or scheduled polling jobs that trigger Azure pipeline runs. It’s doable but requires custom development.

One pattern we use: Rally webhook posts to Azure Function, which triggers pipeline via REST API. Works well but adds complexity to your architecture.

7

I’ll share our experience evaluating both platforms for a similar scale (we’re at 35 teams now).

Jenkins Plugins Analysis: The Rally plugin ecosystem for Jenkins is mature but fragmented. You’ll use the core Rally Plugin for basic integration, but expect to write wrapper scripts for anything beyond simple result publishing. The advantage is complete control and no licensing costs. The disadvantage is maintenance burden - every Rally API change or Jenkins update requires testing your custom code.

Azure DevOps Connectors Assessment: Azure’s Rally extensions are more polished out-of-the-box. The “Rally Integration” extension handles most common scenarios with configuration rather than code. However, the licensing model can get expensive at scale - factor in both Azure DevOps parallel job costs and potential Rally API rate limiting fees if you’re hitting the API heavily.

Cost Analysis Reality: For 30 teams, Jenkins infrastructure costs (servers, maintenance, DevOps engineer time) often exceed Azure’s licensing fees when you calculate total cost of ownership. We estimated Jenkins at $8K/month in infrastructure plus 0.5 FTE for maintenance, versus Azure at $12K/month but near-zero maintenance.

POC Strategy Recommendation: Run parallel POCs with 2 teams - one on each platform - for 6-8 weeks. Measure:

  • Time to implement standard workflows
  • API call efficiency (Rally has rate limits)
  • Developer satisfaction
  • Maintenance incidents

Don’t just test happy paths. Simulate Rally API outages, network issues, and high-volume scenarios.

Scaling Limits: Jenkins scales horizontally well - add more agents as needed. Rally API rate limiting becomes your bottleneck around 300-400 API calls per minute. Azure DevOps has similar scaling characteristics but better built-in retry logic for API failures.

Both platforms can handle 30+ teams, but Azure requires less operational overhead. If you have strong Jenkins expertise in-house and want maximum flexibility, go Jenkins. If you want faster time-to-value and predictable costs, Azure DevOps is the safer bet.

One critical factor: consider your existing CI/CD toolchain. If you’re already heavily invested in Jenkins for builds, adding Rally integration is incremental. If you’re starting fresh or already using Azure for other services, the ecosystem integration tips the scale toward Azure DevOps.