c58240ea3d
## Description This improves the initial backup speed for OneDrive. OneDrive backup was mostly slow when we had a lot of tiny files. Case where we had mostly large files was pretty much the best case scenario and we were throttled by purely how fast we can get the files from MS and how fast kopia can process and upload it. But of small files, we were slowed by the loop which was taking quite a bit of time to fetch the download urls. We have now parallelized the query for getting the download URL. Under best case scenarios, I was able to speed it up to under 20s from ~4-5m starting point. That said, MS graph api still seems to throttle us and when that happen we still go back to around ~2m for worst case scenario. I've added 3 retries as some requests were failing when we continuously making many requests. This should also take care of the issue of url expiring mentioned in https://github.com/alcionai/corso/issues/581 as we are only prefetching a few urls ahed of time. ## Type of change <!--- Please check the type of change your PR introduces: ---> - [ ] 🌻 Feature - [ ] 🐛 Bugfix - [ ] 🗺️ Documentation - [ ] 🤖 Test - [ ] 💻 CI/Deployment - [ ] 🐹 Trivial/Minor ## Issue(s) <!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. --> * https://github.com/alcionai/corso/issues/1595 * fixes https://github.com/alcionai/corso/issues/581 ## Test Plan <!-- How will this be tested prior to merging.--> - [ ] 💪 Manual - [x] ⚡ Unit test - [x] 💚 E2E
Corso
Corso is the first open-source tool that aims to assist IT admins with the critical task of protecting their Microsoft 365 data. It provides a reliable, secure, and efficient data protection engine. Admins decide where to store the backup data and have the flexibility to perform backups of their desired service through an intuitive interface. As Corso evolves, it can become a great building block for more complex data protection workflows.
Corso is currently in ALPHA and should NOT be used in production.
Corso supports M365 Exchange and OneDrive with SharePoint and Teams support in active development. Coverage for more services, possibly beyond M365, will expand based on the interest and needs of the community.
Getting Started
See the Corso Documentation for more information.
Building Corso
To learn more about working with the project source core and building Corso, see the Developer section of the Corso Documentation.
Roadmap
You can learn more about the Corso roadmap and how to interpret it here.
If you have feature requests, please file a GitHub issue
and attach the enhancement label to the issue.
Contribution Guidelines
Code of Conduct
It's important that our community is inclusive and respectful of everyone. We ask that all Corso users and contributors take a few minutes to review our Code of Conduct.
License
Corso is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.