df1ddb94f3
Improved this significantly based on some learnings from trying to get this in better shape * Reduce folder enumerations significantly * Address issue when cleanup gets backlogged and cannot enumerate all items to clean in a single job * Remove the need to launch one job per prefix or folder name * The only way to clean /Audits reliably seems to be emptying (with hard delete) of /Revoverable Items (parent of Audits) * Eliminated need to control concurrency and overhead of job spin up * Attempt to set explicit network timeout for SOAP calls to prevent long hangs being observed NOTE: This change affect both cleanup script and the corresponding GHA workflow --- #### Does this PR need a docs update or release note? - [ ] ✅ Yes, it's included - [ ] 🕐 Yes, but in a later PR - [x] ⛔ No #### Type of change <!--- Please check the type of change your PR introduces: ---> - [ ] 🌻 Feature - [ ] 🐛 Bugfix - [ ] 🗺️ Documentation - [ ] 🤖 Test - [x] 💻 CI/Deployment - [ ] 🧹 Tech Debt/Cleanup #### Issue(s) #### Test Plan <!-- How will this be tested prior to merging.--> - [x] 💪 Manual - [ ] ⚡ Unit test - [ ] 💚 E2E
71 lines
2.7 KiB
YAML
71 lines
2.7 KiB
YAML
name: Purge M365 User Data
|
|
|
|
# Hard deletion of an m365 user's data. Our CI processes create a lot
|
|
# of data churn (creation and immediate deletion) of files, the likes
|
|
# of which wouldn't otherwise be seen by users of the system. Standard
|
|
# APIs don't have the tooling to gut out all the cruft which we accrue
|
|
# in microsoft's hidden nooks and secret crannies. A manual, SOAPy
|
|
# exorcism is the only way.
|
|
#
|
|
# The script focuses on the cleaning up the following:
|
|
# * All folders, descending from the exchange root, of a given prefix.
|
|
# * All folders in PersonMetadata
|
|
# * All already soft-deleted items
|
|
# * All folders under recoverable items
|
|
|
|
inputs:
|
|
user:
|
|
description: User whose data is to be purged.
|
|
folder-prefix:
|
|
description: Name of the folder to be purged. If falsy, will purge the set of static, well known folders instead.
|
|
older-than:
|
|
description: Minimum-age of folders to be deleted.
|
|
azure-client-id:
|
|
description: Secret value of for AZURE_CLIENT_ID
|
|
azure-client-secret:
|
|
description: Secret value of for AZURE_CLIENT_SECRET
|
|
azure-tenant-id:
|
|
description: Secret value of for AZURE_TENANT_ID
|
|
m365-admin-user:
|
|
description: Secret value of for M365TENANT_ADMIN_USER
|
|
m365-admin-password:
|
|
description: Secret value of for M365TENANT_ADMIN_PASSWORD
|
|
|
|
runs:
|
|
using: composite
|
|
steps:
|
|
- name: Run the all purge scripts for user
|
|
if: ${{ inputs.user != '' }}
|
|
shell: pwsh
|
|
working-directory: ./src/cmd/purge/scripts
|
|
env:
|
|
AZURE_CLIENT_ID: ${{ inputs.azure-client-id }}
|
|
AZURE_CLIENT_SECRET: ${{ inputs.azure-client-secret }}
|
|
AZURE_TENANT_ID: ${{ inputs.azure-tenant-id }}
|
|
run: |
|
|
./foldersAndItems.ps1 -User ${{ inputs.user }} -FolderNamePurgeList PersonMetadata --FolderPrefixPurgeList ${{ inputs.folder-prefix }} -PurgeBeforeTimestamp ${{ inputs.older-than }}
|
|
|
|
- name: Reset retention for all mailboxes to 0
|
|
if: ${{ inputs.user == '' }}
|
|
shell: pwsh
|
|
working-directory: ./src/cmd/purge/scripts
|
|
env:
|
|
M365TENANT_ADMIN_USER: ${{ inputs.m365-admin-user }}
|
|
M365TENANT_ADMIN_PASSWORD: ${{ inputs.m365-admin-password }}
|
|
run: |
|
|
./setRetention.ps1
|
|
|
|
- name: Run the old purge script to clear out onedrive buildup
|
|
working-directory: ./src
|
|
if: ${{ inputs.folder-prefix != '' && inputs.user != ''}}
|
|
shell: sh
|
|
env:
|
|
AZURE_CLIENT_ID: ${{ inputs.azure-client-id }}
|
|
AZURE_CLIENT_SECRET: ${{ inputs.azure-client-secret }}
|
|
AZURE_TENANT_ID: ${{ inputs.azure-tenant-id }}
|
|
run: >
|
|
go run ./cmd/purge/purge.go onedrive
|
|
--user ${{ inputs.user }}
|
|
--prefix ${{ inputs.folder-prefix }}
|
|
--before ${{ inputs.older-than }}
|