137 Commits

Author SHA1 Message Date
jules
bb86610488
Getlicenseinfo (#4655)
Adds a call to get user total count of licenses. Also corrects one test ID which i hope doesn't break E2E tests (although it should if it does, ha)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-10 23:13:47 +00:00
ashmrtn
c85d364bcd
Allow limiting or cancelling pager enumeration (#4625)
Add the ability to cancel a pager partway through enumeration. This
will allow us to later request a specific number of items from the
pager. In those cases we should ideally fetch one page beyond what's
required to get the requested number of items (due to the pager
running in another goroutine)

Does not change the result set or add the ability to request a certain
number of items right now

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 20:33:27 +00:00
Keepers
5362137116
expose graph options to sdk clients (#4653)
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-11-10 19:26:59 +00:00
Keepers
85d13b382a
some minor logging touchups (#4617)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-10 18:40:57 +00:00
ashmrtn
d173551d25
Check if delta link is populated in tests (#4624)
Update new test framework to populate page delta links and check if
a delta value is returned when expected. The exact value is not
compared but we do check if the value is non-empty

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 03:24:17 +00:00
ashmrtn
7662cfca68
Consolidate pager tests while adding more coverage (#4622)
Rewrite the pager tests to more uniformly test the different outcomes
when various pagers/configurations are used

This PR splits tests into two main categories: those that deal with
getting results from a single pager with no resets etc and those that
deal with resets, fallbacks, and other similar things

All tests now check for correctness when the pager does and doesn't
support returning mod times

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 01:51:10 +00:00
ashmrtn
3901087b60
Cleanup and consolidate test code for pagers (#4616)
Mostly logic rearrangement with a small test addition to
BatchEnumerateItems

Allow test pagers to return multiple pages and consolidate logic by
using the non-delta pager to source most of the logic for the delta
pager

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-10 00:18:46 +00:00
Keepers
49bbdfc096
add getPostIDs in conversations pager (#4578)
adds the getPostIDs func to ensure conversations
complies with standard data paging patterns

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-07 21:57:55 +00:00
Abhishek Pandey
ec1afa8c84
Add $top param for drive pagers (#4619)
<!-- PR description-->

Closes https://github.com/alcionai/corso/issues/4614 


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-07 07:22:29 +00:00
Keepers
3261eefda2
boilerplate introduction of the conversations api (#4570)
adds basic get requests for the conversations api set to the api package.  Plust some other minor change for formatting, correctness, or necessity.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536 

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-06 20:05:52 +00:00
Keepers
5e0014307c
remove ctx embedding for counter (#4497)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-31 16:15:05 +00:00
Keepers
d6047d3f63
add real world test examples (#4558)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4546

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-27 15:10:02 +00:00
Keepers
8b612c43d9
make channel message html human readable (#4556)
Adds two-step processing to the html previews for channel messages and replies.  First, all inline attachments are replaced with the string `[attachment:name]`.
Second, remaining html is stripped out, leaving only plaintext.

This transformation is applied to both the exported content and the preview content in details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4546

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-26 23:05:35 +00:00
Keepers
1470776f3c
expose additional channel metadata (#4539)
builds out more details for channel messages and replies

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-26 18:03:44 +00:00
ashmrtn
a4d039069d
Refactor site exclusion logic (#4548)
Pull the logic for excluding personal and search
sites into a separate function instead of overloading
the verify function for a site

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4540

merge after:
* #4547

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-26 16:01:24 +00:00
ashmrtn
8102523dc6
Unexport functions and fields in api package (#4547)
Unexport a bunch of stuff that isn't used outside the
package. Also use the same package for tests

Mostly renaming with some minor dead code cleanup

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-26 15:25:34 +00:00
ashmrtn
e1f0794185
Remove site name validation check (#4541)
It's possible to have a site without a display name and this is causing unrelated backups to fail.

Need to do some testing for upstream consumers prior to merging

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4540

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-26 02:16:01 +00:00
Keepers
2bc3c89885
fix api/mock circular ref (#4544)
No logic changes.  Just code movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-10-26 01:27:49 +00:00
jules
73202dad63
[Bugfix] Fixing getting owner email (#4538)
Owner data might have email and ID, or only one of them, or none. Fixing the code to handle this and adding a unit test. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-10-25 22:38:35 +00:00
Keepers
7f8becb8f8
extract container transforms from the api layer (#4478)
removes the cache container transformation funcs
from the exchange api container enumeration params. This ensures a cleaner separation of ownership where the api is only responsible for enumerating containers, and the caller is responsible for transforming them.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-24 23:44:08 +00:00
jules
4d0be25020
Adding email fallback to Groups getByID api (#4534)
As the title suggests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-24 21:00:55 +00:00
Abin Simon
08e8b18afc
Fix backup of Groups without an associated Team (#4520)
Previously we were trying to fetch channels of Groups without channels.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/4519

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-10-17 17:34:11 +00:00
Keepers
03c7667679
iterate pager results with channels (#4414)
corrects the prior PRs full-scale batching of driveItems by adding a channel to the api layer's enumerateItems and driveEnumerateItems funcs.  To avoid changes
in most parts of code, also adds batch wrappers
that hide these changes from existing pager usage.

DriveItem collection processing now ranges over
a channel of pager results, allowing it to process pages in a stream rather than batching all items.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-14 03:22:12 +00:00
Keepers
1452e8c0c9
add extra catches for resource locked (#4499)
Adds two layers of extra catches for resoruceLocked errors.  First, in pkg/errs, adds error check funcs to ensure the source comparators are checked even if
they somehow got skipped in the lower layer packages.

Second, adds checks for resource locked throughout sites, users, and groups in the api layer.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 23:27:43 +00:00
Keepers
7419faab23
refactor drive restore & export to use sanitree (#4425)
refactors the common drive sanity checks to use the sanitree data container.  Also expands the sanitree in two ways:
1. adds leaves (individual items) to nodes for granular data comparison
2. adds multi-type support for comparing nodes of different types.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-10-13 22:55:35 +00:00
Keepers
d2e35a9ea8
filter out system messages (#4481)
adds a filter that removes system messages from
channel message backups.

Also extends the preview size from 16 to 128 characters.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 02:48:08 +00:00
Keepers
5b8b371976
fix message size report (#4493)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-13 02:19:59 +00:00
Keepers
7b6484663b
ensure api layer provides non-empty modtime (#4486)
enforce the following:
* no mod times returned from the api layer are zero-valued
* dttm.OrNow() provides a zero value time check
* ptr.OrNow() calls dttmOrNow on non-nil time values.
* itemPagers for getAddedAndRemoved call ptr.OrNow()
* callers to getAddedAndRemoved provide a transformer func which produces the mod time from the paged item, instead of assuming all items use the same interface for mod time.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-12 23:42:35 +00:00
Keepers
a9e0b4f95f
fix failing item replace test (#4488)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-12 22:29:33 +00:00
Abhishek Pandey
3656e04676
Allow delta enum callers to specify $select properties (#4460)
<!-- PR description-->

This fixes a perf regression in #4456 

Context: 
URL cache only needs a subset of drive item properties while doing delta queries. See https://github.com/alcionai/corso/pull/4074 for details.  Changes in #4456 were applying the default item property set for all delta enumerator consumers including URL cache. This PR fixes the memory regression. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-10 20:07:42 +00:00
Keepers
7196b5d278
catch and sentinel resource locked error case (#4465)
handle cases where a resource is found, but is not accessible due to being locked out by an administrator or msoft process.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4464

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-10 19:24:48 +00:00
Abhishek Pandey
eb0299d316
Push item download requests through drive rate limiter (#4393)
<!-- PR description-->

**Changes:** 
1. Count item download requests under drive rate limit quota . Currently this is under exchange limiter.
2. Use 1 rate limit token instead of 2 for drive calls by default. 
3. Use 2 tokens instead of 1 for initial delta query ( which has no token). 

Sharing internal docs separately to go along with the review. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-10 16:45:37 +00:00
Keepers
5215e907b0
Revert "Revert "move drive pagers to pager pattern (#4316)" (#4412)" (#4456)
This reverts commit 3d78183651289e2051b8690850069c9b41df6bd0.

Replacement for https://github.com/alcionai/corso/pull/4316 after revert in https://github.com/alcionai/corso/pull/4412.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-10-09 19:46:18 +00:00
Keepers
3d78183651
Revert "move drive pagers to pager pattern (#4316)" (#4412)
This reverts commit c3f94fd7f76f377e4728c715abbb8c7846e9fb25.

The specified commit is working fine for CI and development, but contains performance
degredation (solved in a follow-up pr) that we want to avoid for the next release.  This
rever is temporary, and the changes will be re-instated after release.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-02 16:19:39 +00:00
Abin Simon
363cbca86f
Backup/Restore/Export multiples sites in Groups (#4344)
I've updated the Team used in the CI to include private and shared channels. Sanity tests should ideally do the e2e tests for multi site backups.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-02 13:17:15 +00:00
Keepers
b15f8a6fcd
add generic details command (#4352)
centralizes details command processing in the cli

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2025
2023-09-30 16:56:13 +00:00
Keepers
9e0d464854
sanity test refactor (#4370)
refactoring the sanity tests with three goals:
1. move from env vars to cli commands so that unsupported commands fail loudly.
2. set up support for groups restore and export testing.
3. introduce some code re-use throughout.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3988

#### Test Plan

- [x] 💚 E2E
2023-09-28 23:53:15 +00:00
Keepers
e5647a809d
add GroupByID to services (#4396)
Adds a groupByID call to services, and
adds CallConfig to the group by id api fn.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988

#### Test Plan

- [x] 💚 E2E
2023-09-28 22:36:38 +00:00
Keepers
c3f94fd7f7
move drive pagers to pager pattern (#4316)
Drive pager usage currently showcases strong coupling between two layers: drive collection logic processing and drive api.  This PR separates that coupling by moving the full item enumeration process into the
API, and letting the collection logic process the
results. This acs as both a simplification of complex code, and a clearer separation of ownership between the two layers.

A detrimental side effect of this change is that drive item enumeration has moved from page-streaming
(ie: each page is fully processed before moving on to the next) and onto batch processing (ie: all items are stored in memory and processed in a single pass). Acknowledging that this is an unacceptable regression, a follow-up PR will appear shortly with better handling for stream-processing enumeration from the API layer as a standard part of the pattern for all pager
implementations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-28 21:23:26 +00:00
ashmrtn
db9596d223
Use non-zero time as fallback (#4390)
Use a non-zero time as the fallback for the mod time of an item during enumeration. This should avoid an issue where the mod time in details differs from the mod time in kopia due to a serialization issue.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-27 23:07:59 +00:00
Keepers
a8a2aee99d
remove expand drive from root site (#4377)
missed the root site call usage of expand drive
on the last fix.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4337

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 22:32:15 +00:00
Keepers
0474e5c957
separate delta not supported from invalid (#4375)
groups backup has a failing case where delta
queries are not supported, but we aren't distinguishing
that case from the invalid delta case as expected. This
results in backup failures, instead of success with no data.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 18:37:15 +00:00
Keepers
01f61a44ed
move drive expand to config property (#4362)
the addition of the drive expand requires onedrive permissions in order to retrieve sites by ID.  This causes some users to violate customer contract by requiring unwanted permissions.  This change allows configuration of the getbyID call to optionally include the drive extend, which enables the backup process to return to the permission set as expected from v0.12

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4337

#### Test Plan

- [ ] 💪 Manual
- [ ] 💚 E2E
2023-09-25 19:18:16 +00:00
Keepers
0545365b12
move container enum to pager pattern (#4298)
moves exchange container enumeration funcs in
the api to the pager pattern established with item enumeration.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 17:34:06 +00:00
Keepers
dcffc61bf5
do not mark not-found items as skipped (#4342)
This was an accidental mis-use of the skipped item pattern.  Items deleted during in flight during backup due to race conditions do not count as permanent
skips in the way that other skipped items do.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #4044

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 16:39:16 +00:00
Abin Simon
ccf8df3548
Bump msgraph sdk to 1.19.0 (#4314)
Needed some changes in the func names.

``` bash
$ sed -i "s/By\([a-zA-Z]*\)IdString/By\1Id/g" **/*.go
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-21 07:01:07 +00:00
ashmrtn
4a9951b876
Fetch mod time when getting added and removed items (#4266)
Also return mod time when available if
getting the set of added and removed
items. This will be leveraged in later
PRs to implement kopia assisted
incrementals for exchange

Does not change any logic in
collections right now, just adds the
fields to be returned

Also adds an additional return value
denoting if the mod times are expected
to be valid. This is required because
events delta cannot return mod time

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2023

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-18 22:30:56 +00:00
Keepers
f7042129f4
minimize channel messages exports (#4245)
reduces channel message export data to the minimal set of valuable info: message content, creator,
creation and modification time, and replies (each
reply has the same data, sans other replies).

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3991  

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-09-15 21:16:06 +00:00
Keepers
265a77f1cd
match item type in groups selectors info filter (#4255)
adds item type comparisons to the info filter during groups selector reduction.  This ensures cross-
contamination of item types on shared info properties does not occur.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-15 20:29:14 +00:00
Abin Simon
ce422f0b1d
Add linter for checking with empty string (#4246)
Finding failing cases:

``` bash
tree-grepper -q go '((binary_expression (identifier) ["==" "!="] (interpreted_string_literal) @ri) (#eq? @ri "\"\""))'
```

Fixing failing cases:

``` bash
comby 'if :[1~[^ ]*] == ""' 'if len(:[1]) == 0' -matcher .go -in-place
comby 'if :[1~[^ ]*] != ""' 'if len(:[1]) > 0' -matcher .go -in-place
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/3654

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-15 06:45:00 +00:00