Automated GitOps-based promotion process streamlines hotfix deployment across environments

SCM Manhattan Active Supply Chain
, , , , , , , ,
1

Sharing our implementation of a GitOps-driven promotion pipeline that reduced our hotfix deployment time from 4 hours to 30 minutes across dev, test, and production environments. We manage a complex Manhattan distribution management setup with frequent configuration updates and bug fixes that previously required manual promotion steps and extensive validation.

The traditional process involved exporting configurations, manually merging changes, running validation scripts, and coordinating deployment windows. With our GitOps approach, we’ve automated the entire workflow using version-tagged branches and automated promotion pipelines that handle conflict resolution and rollback automatically.

Key benefits we’ve seen: 85% reduction in deployment errors, complete audit trail for compliance, and the ability to roll back to any previous configuration state within minutes. The system now handles version tagging automatically and maintains a complete history of all configuration changes across environments.

2

How are you handling Manhattan-specific configurations that can’t be version controlled through standard Git? Things like workflow definitions, allocation rules, and runtime parameters that are stored in the database rather than config files? We’ve tried similar approaches but always hit roadblocks with the non-file-based configurations.

3

Complete GitOps Implementation for Manhattan Hotfix Automation

Our GitOps workflow transformed Manhattan configuration management from a manual, error-prone process into a fully automated pipeline. Here’s the complete architecture:

GitOps Workflow Structure:

We use a trunk-based branching model with environment-specific promotion branches:

  • main: Production-ready configurations
  • staging: Pre-production validation
  • develop: Active development and testing
  • hotfix/*: Emergency fixes that bypass normal flow

Each hotfix branch follows the naming convention hotfix/HF-2025-03-###-description and gets automatically tagged as it progresses through environments.

Automated Promotion Pipeline:

The pipeline orchestrates the entire promotion lifecycle:

  1. Version Tagging and Validation: When a hotfix is committed, the pipeline automatically tags it with semantic version (e.g., v1.2.3-hotfix.1), runs Manhattan-specific validation scripts, and checks configuration compatibility against target environment baseline versions.

  2. Configuration Export and Packaging: All Manhattan configurations - including database-stored workflow definitions, allocation rules, and runtime parameters - are exported as XML artifacts and packaged with version metadata. We maintain a manifest file that tracks which configurations changed in each hotfix.

  3. Automated Testing: The pipeline deploys to our test environment first, runs automated integration tests against Manhattan APIs, validates workflow execution, and generates a test report. Only after all tests pass does it proceed to staging.

  4. Approval Gates: For production promotion, we implemented approval gates that require sign-off from both technical leads and business owners. These approvals are tracked in Git pull requests and logged to our audit database.

  5. Rollback Capability: Every deployment creates an automatic rollback tag. If issues are detected post-deployment, the pipeline can revert to the previous stable version within 5 minutes by applying the rollback tag and re-importing the previous configuration state.

Version Tagging and Rollback Strategy:

Our tagging strategy ensures complete traceability:

  • Base tags: v{major}.{minor}.{patch} for regular releases
  • Hotfix tags: v{major}.{minor}.{patch}-hotfix.{number}
  • Environment tags: {version}-{env}-{timestamp}
  • Rollback tags: {version}-rollback-{reason}

The rollback process is fully automated. If a deployment fails validation checks or if manual rollback is triggered, the pipeline:

  1. Identifies the last successful deployment tag for that environment
  2. Retrieves the configuration artifacts from that tag
  3. Executes Manhattan import utilities to restore previous state
  4. Validates the rollback was successful
  5. Notifies the team via Slack with rollback details

Key Technical Implementation Details:

We built custom scripts that integrate Manhattan’s export/import utilities with Git operations. For database-stored configurations, we schedule periodic exports that commit changes to Git, ensuring even manual configuration changes through Manhattan UI are captured in version control.

The pipeline uses Jenkins for orchestration, with Manhattan-specific plugins we developed for configuration validation. We also implemented health checks that run post-deployment to verify Manhattan services are functioning correctly with the new configuration.

Results After Six Months:

  • Deployment time reduced from 4 hours to 30 minutes average
  • Configuration errors dropped 85% (from ~12 per month to 1-2)
  • Zero failed deployments in production (test environment catches issues)
  • Complete audit trail covering 100% of production changes
  • Rollback capability tested successfully 8 times during the period
  • Team confidence in making configuration changes increased dramatically

The most valuable outcome has been the ability to promote hotfixes with confidence. Previously, we’d delay critical fixes because of deployment risk. Now we can push emergency fixes to production within an hour of validation, knowing we have automated rollback if needed.

For teams considering this approach: Start with a pilot environment, build confidence with the automation, and gradually expand. The upfront investment in pipeline development pays off quickly in reduced deployment overhead and improved reliability.

4

Great question. We export those database-stored configurations as XML using Manhattan’s export utilities and treat them as code artifacts in Git. Our pipeline includes pre-promotion and post-promotion hooks that execute Manhattan’s import/export scripts. For workflow definitions specifically, we maintain them as versioned XML files and use the Manhattan API to deploy them during the promotion process. Runtime parameters are managed through property files that get injected during deployment. It’s not perfect, but it gives us version control over about 90% of our configuration surface area.

5

This is exactly what we’ve been looking to implement. Can you share more details about how you handle the version tagging? We’re struggling with maintaining consistency across our six Manhattan instances when promoting hotfixes, especially when different environments are on slightly different baseline versions.