760 Commits

Author SHA1 Message Date
jules
bb86610488
Getlicenseinfo (#4655)
Adds a call to get user total count of licenses. Also corrects one test ID which i hope doesn't break E2E tests (although it should if it does, ha)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-10 23:13:47 +00:00
Hitesh Pattanayak
e9118abe9f
abstract kopia setup code (#4637)
Refactors the repository code to abstract code responsible for kopia related setup.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 21:03:36 +00:00
ashmrtn
c85d364bcd
Allow limiting or cancelling pager enumeration (#4625)
Add the ability to cancel a pager partway through enumeration. This
will allow us to later request a specific number of items from the
pager. In those cases we should ideally fetch one page beyond what's
required to get the requested number of items (due to the pager
running in another goroutine)

Does not change the result set or add the ability to request a certain
number of items right now

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 20:33:27 +00:00
Hitesh Pattanayak
52c788eac5
check path equivalency (#4635)
<!-- PR description-->

This change is to address the path related issue that comes up while `corso repo connect filesystem`.
Below are the steps to reciprocate:
1. Initialize repo with filesystem provider (Notice that the path has a trailing slash): 
command: `./corso repo init filesystem --path ~/Documents/Backups/`
output:
Logging to file: ~/.cache/corso/logs/2023-11-09T05-21-57Z.log
Connecting to M365:         done 
Initializing repository   
Initialized a repository at path ~/Documents/Backups/

2. Connect repo with filesystem provider:
command: `./corso repo connect filesystem --path ~/Documents/Backups/`
output:
Logging to file: ~/.cache/corso/logs/2023-11-09T05-23-30Z.log
Error: retrieving storage provider details: verifying storage configs in corso config file: verifying configs in corso config file: value of path (~/Documents/Backups) does not match corso configuration value (~/Documents/Backups/)

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-10 19:59:12 +00:00
Keepers
5362137116
expose graph options to sdk clients (#4653)
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-11-10 19:26:59 +00:00
Keepers
85d13b382a
some minor logging touchups (#4617)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-10 18:40:57 +00:00
Keepers
1f756ce34f
support debug reading out delta files (#4629)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
2023-11-10 16:19:16 +00:00
ashmrtn
d173551d25
Check if delta link is populated in tests (#4624)
Update new test framework to populate page delta links and check if
a delta value is returned when expected. The exact value is not
compared but we do check if the value is non-empty

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 03:24:17 +00:00
ashmrtn
7662cfca68
Consolidate pager tests while adding more coverage (#4622)
Rewrite the pager tests to more uniformly test the different outcomes
when various pagers/configurations are used

This PR splits tests into two main categories: those that deal with
getting results from a single pager with no resets etc and those that
deal with resets, fallbacks, and other similar things

All tests now check for correctness when the pager does and doesn't
support returning mod times

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 01:51:10 +00:00
ashmrtn
3901087b60
Cleanup and consolidate test code for pagers (#4616)
Mostly logic rearrangement with a small test addition to
BatchEnumerateItems

Allow test pagers to return multiple pages and consolidate logic by
using the non-delta pager to source most of the logic for the delta
pager

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-10 00:18:46 +00:00
Keepers
49bbdfc096
add getPostIDs in conversations pager (#4578)
adds the getPostIDs func to ensure conversations
complies with standard data paging patterns

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-07 21:57:55 +00:00
Abhishek Pandey
ec1afa8c84
Add $top param for drive pagers (#4619)
<!-- PR description-->

Closes https://github.com/alcionai/corso/issues/4614 


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-07 07:22:29 +00:00
Abhishek Pandey
ea6a57b2a5
Share count bus between graph clients and backup/restore operations (#4605)
<!-- PR description-->

With #4497 changes, any counters that we log in graph middlewares ( e.g. `APICallTokensConsumed` or `ThrottledAPICalls`) will no longer show up in `BackupOperation.Results.Counts`. 

This is because graph adapters and backup operation will be initialized with different count busses. Since the backup result counts are captured from the operation bus, we will lose out on any graph metrics captured in middleware layers. This PR unifies the busses. 

Ill also add a regression check for graph metrics shortly

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-07 06:41:11 +00:00
Keepers
3261eefda2
boilerplate introduction of the conversations api (#4570)
adds basic get requests for the conversations api set to the api package.  Plust some other minor change for formatting, correctness, or necessity.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536 

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-06 20:05:52 +00:00
ashmrtn
e46cf645e5
Create and tag preview backups (#4595)
Add an option to request a preview backup and tag the resulting backup
as a preview if the flag is set. Preview backups must complete
successfully with no errors in order to be tagged

This does not update the item selection logic, so right now preview
backups will contain all items that normal backups do. Item selection
will be refined in upcoming PRs

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-11-01 17:22:49 +00:00
ashmrtn
74cf0ab737
Create service specific handlers that know how to run an export (#4491)
First step in reducing the number of places we have to check the service type manually. Create a way to get a handle to a service specific handler and implement exports for those handlers

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-01 15:55:30 +00:00
ashmrtn
66eb789d39
Helper function to get backup type (#4591)
Allow the backup model to tell us what type it was tagged with. This
will make logic for base selection easier since it won't have to worry
about extracting the type anymore.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-01 01:06:50 +00:00
Keepers
5e0014307c
remove ctx embedding for counter (#4497)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-31 16:15:05 +00:00
Keepers
d6047d3f63
add real world test examples (#4558)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4546

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-27 15:10:02 +00:00
Keepers
8b612c43d9
make channel message html human readable (#4556)
Adds two-step processing to the html previews for channel messages and replies.  First, all inline attachments are replaced with the string `[attachment:name]`.
Second, remaining html is stripped out, leaving only plaintext.

This transformation is applied to both the exported content and the preview content in details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4546

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-26 23:05:35 +00:00
Keepers
1470776f3c
expose additional channel metadata (#4539)
builds out more details for channel messages and replies

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-26 18:03:44 +00:00
ashmrtn
a4d039069d
Refactor site exclusion logic (#4548)
Pull the logic for excluding personal and search
sites into a separate function instead of overloading
the verify function for a site

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4540

merge after:
* #4547

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-26 16:01:24 +00:00
ashmrtn
8102523dc6
Unexport functions and fields in api package (#4547)
Unexport a bunch of stuff that isn't used outside the
package. Also use the same package for tests

Mostly renaming with some minor dead code cleanup

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-26 15:25:34 +00:00
ashmrtn
e1f0794185
Remove site name validation check (#4541)
It's possible to have a site without a display name and this is causing unrelated backups to fail.

Need to do some testing for upstream consumers prior to merging

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4540

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-26 02:16:01 +00:00
Keepers
2bc3c89885
fix api/mock circular ref (#4544)
No logic changes.  Just code movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-10-26 01:27:49 +00:00
jules
73202dad63
[Bugfix] Fixing getting owner email (#4538)
Owner data might have email and ID, or only one of them, or none. Fixing the code to handle this and adding a unit test. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-25 22:38:35 +00:00
Keepers
7f8becb8f8
extract container transforms from the api layer (#4478)
removes the cache container transformation funcs
from the exchange api container enumeration params. This ensures a cleaner separation of ownership where the api is only responsible for enumerating containers, and the caller is responsible for transforming them.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-24 23:44:08 +00:00
jules
4d0be25020
Adding email fallback to Groups getByID api (#4534)
As the title suggests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-24 21:00:55 +00:00
Keepers
9e40d88265
expose sitesInGroup to sdk consumers (#4532)
exposes the m365 api call to get all sites in a group to the sdk service interface.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-24 00:09:48 +00:00
Keepers
5cc68e27dc
use count bus in kopia backups (#4482)
uses the count bus in the kopia backup package.
This currently duplicates counts that we're getting
from the kopia stats.  A later pr will remove the old
stats entirely in favor of the counter.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-23 23:29:56 +00:00
Abin Simon
08e8b18afc
Fix backup of Groups without an associated Team (#4520)
Previously we were trying to fetch channels of Groups without channels.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/4519

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-10-17 17:34:11 +00:00
Abin Simon
5d90483bfa
Enforce site filter for Groups restore (#4496)
This PR reworks the groups restore to and makes the site to restore to mandatory. This also updates some missing filtering capabilities in groups export.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/4462

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-16 12:05:55 +00:00
Keepers
03c7667679
iterate pager results with channels (#4414)
corrects the prior PRs full-scale batching of driveItems by adding a channel to the api layer's enumerateItems and driveEnumerateItems funcs.  To avoid changes
in most parts of code, also adds batch wrappers
that hide these changes from existing pager usage.

DriveItem collection processing now ranges over
a channel of pager results, allowing it to process pages in a stream rather than batching all items.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-14 03:22:12 +00:00
Keepers
1452e8c0c9
add extra catches for resource locked (#4499)
Adds two layers of extra catches for resoruceLocked errors.  First, in pkg/errs, adds error check funcs to ensure the source comparators are checked even if
they somehow got skipped in the lower layer packages.

Second, adds checks for resource locked throughout sites, users, and groups in the api layer.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 23:27:43 +00:00
Keepers
7419faab23
refactor drive restore & export to use sanitree (#4425)
refactors the common drive sanity checks to use the sanitree data container.  Also expands the sanitree in two ways:
1. adds leaves (individual items) to nodes for granular data comparison
2. adds multi-type support for comparing nodes of different types.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-10-13 22:55:35 +00:00
ashmrtn
800170787f
Log the categories participating in an operation (#4490)
To assist in debugging via logs, add the set of
categories an operation is acting on to the
clues set in the context

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-13 21:17:16 +00:00
Keepers
d2e35a9ea8
filter out system messages (#4481)
adds a filter that removes system messages from
channel message backups.

Also extends the preview size from 16 to 128 characters.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 02:48:08 +00:00
Keepers
5b8b371976
fix message size report (#4493)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 02:19:59 +00:00
Keepers
7b6484663b
ensure api layer provides non-empty modtime (#4486)
enforce the following:
* no mod times returned from the api layer are zero-valued
* dttm.OrNow() provides a zero value time check
* ptr.OrNow() calls dttmOrNow on non-nil time values.
* itemPagers for getAddedAndRemoved call ptr.OrNow()
* callers to getAddedAndRemoved provide a transformer func which produces the mod time from the paged item, instead of assuming all items use the same interface for mod time.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-12 23:42:35 +00:00
Keepers
a9e0b4f95f
fix failing item replace test (#4488)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-12 22:29:33 +00:00
ashmrtn
c07b14c859
Consolidate Reasoner implementations (#4469)
Remove other implementation of Reasoner in the
selector package since it's a duplicate of the
one that's in the identity package

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-11 22:04:16 +00:00
ashmrtn
84b9de96ef
Move Reasoner implementation to identity package (#4468)
Move the Reasoner implementation from the kopia package to the identity
package. This will help avoid import cycles if we want to start
persisting Reason information in the backup model.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-11 21:24:08 +00:00
Abin Simon
040257f8be
Add stats to export operations (#4461)
Have a way to gather stats about the exported data.

Users can now call `ExportOperation.GetStats()` at the end of the run to get the stats for the operations. The data will be in the format `map[path.CategoryType]data.KindStats` whre `KindStats` is:

```go
type KindStats struct {
	BytesRead     int64
	ResourceCount int64
}
```
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4311

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-11 05:54:17 +00:00
Keepers
107b6883d5
modify onenote skip, clues add collection scope (#4472)
some additional logging context for collection scope.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-11 02:28:17 +00:00
Abhishek Pandey
3656e04676
Allow delta enum callers to specify $select properties (#4460)
<!-- PR description-->

This fixes a perf regression in #4456 

Context: 
URL cache only needs a subset of drive item properties while doing delta queries. See https://github.com/alcionai/corso/pull/4074 for details.  Changes in #4456 were applying the default item property set for all delta enumerator consumers including URL cache. This PR fixes the memory regression. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-10 20:07:42 +00:00
Keepers
7196b5d278
catch and sentinel resource locked error case (#4465)
handle cases where a resource is found, but is not accessible due to being locked out by an administrator or msoft process.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4464

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-10 19:24:48 +00:00
Abhishek Pandey
eb0299d316
Push item download requests through drive rate limiter (#4393)
<!-- PR description-->

**Changes:** 
1. Count item download requests under drive rate limit quota . Currently this is under exchange limiter.
2. Use 1 rate limit token instead of 2 for drive calls by default. 
3. Use 2 tokens instead of 1 for initial delta query ( which has no token). 

Sharing internal docs separately to go along with the review. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-10 16:45:37 +00:00
Keepers
aa66675b83
remove local bus struct (#4435)
removes the local bus from fault in favor of a single Bus that operates both within a local and global instance, and can be passed downstream independent of context.  Also includes some code separation in the fault package for readability.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-10 01:25:14 +00:00
Keepers
ff6c1eaf8d
adds purely informational alerts to the fault bus (#4434)
Adds a new type of entry to the fault bus: Alerts. These values are for non-erroneous messages that we want 1/ clearly displayed to the user at the end of an operation and 2/ persisted with the backup for later review.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4264

#### Test Plan

- [x]  Unit test
2023-10-09 21:57:36 +00:00
Keepers
5215e907b0
Revert "Revert "move drive pagers to pager pattern (#4316)" (#4412)" (#4456)
This reverts commit 3d78183651289e2051b8690850069c9b41df6bd0.

Replacement for https://github.com/alcionai/corso/pull/4316 after revert in https://github.com/alcionai/corso/pull/4412.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-10-09 19:46:18 +00:00