Summary
Build creation was blocked due to hitting Azure's role assignment limit (4000 per subscription). Users attempting to create builds received the error:
RestError: No more role assignments can be created.
Root Cause
The ToDesktop build system creates ephemeral Azure AD applications with Key Vault role assignments for each build. A cleanup cron job exists to remove stale role assignments, but it was misconfigured to always run in dry-run mode.
Resolution
Updated our cleanup script to run with `dry_run=false` for scheduled runs
Manually triggered the workflow with `dry_run=false` to clear the backlog of role assignments.
Lessons Learned
Cleanup jobs should be tested to verify they actually perform deletions
Consider adding monitoring/alerting for role assignment count approaching limits
Resolved
Summary
Build creation was blocked due to hitting Azure's role assignment limit (4000 per subscription). Users attempting to create builds received the error:
RestError: No more role assignments can be created.
Root Cause
The ToDesktop build system creates ephemeral Azure AD applications with Key Vault role assignments for each build. A cleanup cron job exists to remove stale role assignments, but it was misconfigured to always run in dry-run mode.
Resolution
Updated our cleanup script to run with `dry_run=false` for scheduled runs
Manually triggered the workflow with `dry_run=false` to clear the backlog of role assignments.
Lessons Learned
Cleanup jobs should be tested to verify they actually perform deletions
Consider adding monitoring/alerting for role assignment count approaching limits
Investigating
Builds failing