Skip to main content

Rollback Release

Overview​

Emergency procedure to revert a production deployment when critical issues are discovered post-release.

Prerequisites​

When to Use​

  • Critical bug discovered in production
  • Major functionality broken after deployment
  • Security vulnerability identified
  • Performance degradation affecting users

Procedure​

Step 1: Assess the Situation​

Purpose: Determine the scope and urgency of the rollback

  1. Identify what is broken (web, Electron, or both)
  2. Determine the last known good version
  3. Notify stakeholders that a rollback is in progress

Step 2: Rollback Web Deployment​

Purpose: Restore the previous web version

  1. Navigate to the Production pipeline
  2. Find the last successful production deployment
  3. Click on the build and select Re-run or manually trigger with the previous commit/tag

Verification: Production URL shows the previous version (check version number or specific fix that was reverted)

Step 3: Rollback MSIX/Electron (if needed)​

Purpose: Restore the previous Electron version for new installations

  1. In Accedo Control, update the integrity configuration:

    • Ensure the checksum matches the previous MSIX

  2. If a new MSIX build is needed, and it’s not available in Sharepoint:

    • Checkout the previous release tag
    • Run the MSIX build pipeline
    • Upload to SharePoint

Verification: New installations use the previous stable version

Step 4: Verify Rollback Success​

Purpose: Confirm the system is stable

  • Production application loads correctly
  • Previously broken functionality now works
  • User reports confirm resolution (if applicable)

Step 5: Post-Incident Communication​

Purpose: Inform stakeholders of resolution

  1. Update the incident channel/thread
  2. Send resolution notice:
[RESOLVED] Production rollback complete
Issue: [Brief description]
Resolution: Rolled back to version [X.X.X]
Next steps: [Investigation/fix timeline]

Post-Procedure Verification​

  • All critical user flows working
  • Monitoring shows normal error rates
  • No user complaints about previously reported issue

Troubleshooting​

ProblemPossible CauseSolution
Cannot find previous buildBuild artifacts expiredRebuild from previous git tag
Rollback doesn't fix issueIssue existed in previous versionInvestigate further, may need older version
Users still see broken versionCDN/browser cacheWait for cache expiry or purge CDN
Electron app still brokenApp cached locallyUsers need to reinstall or clear app cache

Escalation​

  1. Immediate: Alert team lead and on-call
  2. If unresolved in 30 min: Escalate to engineering manager
  3. If customer-facing for 1+ hour: Escalate to stakeholders