We were not checking for the error returned by the Get method before trying to use the result to get contact info.
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* We have started seeing `io.ErrUnexpectedEOF` and `read: connection timed out` in the last 2 days for exchange backups. Retry those.
* Also increase retry count from 3 to 6. This is more of a hail mary to retry `InvalidAuthenticationToken` errors. We have observed that retries do help. But for a small set of requests, we end up exhausting retries and eventually fail with InvalidAuthenticationToken error. Hoping that bumping this to 6 will get us some relief. This fix may be removed if we find the rootcause/pattern behind this.
* The event list test was taking > 150 secs. Thought I'd push this change as its a small fix. I first thought my PR https://github.com/alcionai/corso/pull/5202 broke the test. So I investigated this.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
If the delta pager with a smaller page size also falls then attempt to
use the regular events endpoint to enumerate items
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Update the events item enumerator to switch to a smaller page size for
the delta pager if we fail enumeration with a 503 error and no content.
We've found that that situation is indicative of the Graph server
being slow and a smaller page size allows us to make progress still
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test (in another PR)
- [ ] 💚 E2E
corso has a thousand bespoke approaches to setting the same info in all of its tests. This is a first step towards minimizing and standardizing the lift around that work. Future PRs will distribute these packages through the repo.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* We are seeing 500 errors from graph during exchange(email) backup. * Error is `:{"code":"ErrorCorruptData","message":"Data is corrupt., Invalid global object ID: some ID`.
* Catch the error and add a skip since these items cannot be downloaded from graph.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Start catching graph responses by broad response code, in addition to the response body details.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Now that graph errors are always transformed as part of the graph client wrapper or http_wrapper, we don't need to call graph.Stack or graph.Wrap outside of those inner helpers any longer. This PR swaps all those graph calls with equivalent clues funcs as a cleanup.
Should not contain any logical changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4685
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* Add the serialized `Postable` to EML converter code for group mailboxes and associated tests.
* We should converge this with `Messageable` code at some point with a small translator function which converts `Postable` to `Messageable`. Reason being that posts are a subset of messages. Although I feel it's a bit premature right now. Once we have tested this in prod a bit, and we know for a fact that posts are indeed a true subset of messages, we can do this convergence.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
seems like a lot of code, but this is 95% boilerplate additions copied from groups collections packages, with a find-replace for names.
Some noteworthy differences:
* teamsChats does not handle metadata, so all metadata, delta, and previous path handling was removed
* teamsChats does not produce tombstones
* chats are never deleted, so no "removed" items are tracked
* all chats gets stored at the prefix root, so no "containers" are iterated, and therefore only one collection is ever produced.
This means that, overall, while the boilerplate here is still the same, it's much reduced compared to similar packages.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5062
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* Each post now has a `.data` and `.meta` file.
* I've only made changes to groups lazy reader to keep this PR short. Note that channels don't have meta/data file concepts, we don't want to accidentally enable this for channels. Given channels doesn't use lazy reader, this is safe to do short term.
* I'll be adding small follow up PRs to 1) Make sure channels files don't get assigned `.data` suffix. 2) Support for data and meta files for prefetch conversations backup.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
handles single & multiple values for metadata columns
Similar to `Hyperlink` and `Column` columns, `Metadata` column too is unrecognizable from GRAPH API response. Hence identifying from the field column names.
`Metadata` fields are like tags. A `Metadata` fields can be configured to hold multiple values/tags
**Original List with `Metadata` column (Department) with single value/tag**:
**Restored List with `Metadata` column with single value/tag**:
**Original List with `Metadata` column (Department) with multiple value/tag**:
**Restored List with `Metadata` column with multiple value/tag**:
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#5084#5108
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Disable sending requests to microsoft servers to get API tokens when
using the gock client. This is accomplished by passing in a mock that
always returns no error for auth requests.
**This PR does not go through and make existing tests using gock unit
tests instead of integration tests. That will need to be done
separately**
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #5124
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
introduces the Chats service and Chats category
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5061
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
provisions to identify fields with multiple value.
Some fields can hold multiple values and single value based on a property called `allowMultipleValues`.
For example, `Lookup` column:
```
"lookup": {
"allowMultipleValues": true,
"allowUnlimitedLength": false,
"columnName": "Title",
"listId": "21b45bf2-e495-4582-b114-839577ff8e4f"
}
```
But `choice` columns, even though allows to set multiple choices/value, does not have that particular field `allowMultipleValues` to indicate.
So in this PR we are trying determine the same by the stored values while restoring.
**Original list with `choice` column in site**:
**Restored list with `choice` column in site**
The color does not come through though
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#5108
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Centralizing the suffixes so that group mailbox and future services can also use them.
* Move `.meta`, `.data`, `.dirmeta` definitions from `internal/m365/collection/drive/metadata` pkg to `pkg/services/m365/api/graph/metadata`.
* Adjust package names in misc places.
* No logic change.
* We should probably keep all metadata definitions and code in one place, i.e. relocate drive metadata definitions to `pkg/services/m365/api/graph/metadata`. This is a bigger change, we can do it later.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* Expand inReplyTo property to additionally get the parent post contents. This will be persisted as part of post data.
* This additional data will be useful for building a reply tree if we decide to do in-order restore/export in future.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
- includes GetList in list restore handler
- includes GetListsByCollisionKey in list restore handler
- updates mock list restore handler
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
we already select these properties when paging drive items, we just missed the configuration on per-item GETs.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
make the NotFound error comparison more lenient so that it catches the broader set of not-found error responses from graph.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This value was present in the control.Options in the struct and
passed as a parameter. This PR removes the parameter so it's clear
the control.Options one is used
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #5012
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Similar to https://github.com/alcionai/corso/pull/4990
Change due to https://github.com/alcionai/corso/pull/5017
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
Hyperlink column of a list item is not identifiable from GRAPH's
response.
Hence this is a workaround to handle such fields until a column
definition is introduced for `Hyperlink`.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Location column of a list item is not identifiable from GRAPH's
response.
Hence this is a maneouver to handle such fields until a column
definition is introduced for `Location`.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
---------
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
relax conditions on groups by allowing them to have an empty display name.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* Fixes#4918
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
gets lists with collision key
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3890
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
make a canonical 'not found' error for our core sentinels. This replaces the graph.DeletedInFlight error, which acted as an interpretation of "not found".
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4685
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
refactors post list to accept deseriallized stored list
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Another retry related optimization. This PR reduces `TestGraphIntgSuite/TestAdapterWrap_retriesConnectionClose` runtime from 150sec to 0.7sec.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
now that we have a core error library, we can start swapping out the percolation of graph api errors with core error sentinels. This PR begins that process by adding a centralized transformer to the graph service handlers that ensures all calls produce the same set of core errors according to some standard categorizations. This is just an initial step, and does not transfer all graph errors into core sentinels.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #4685
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* Attachment download needed an expand operation. Added that.
* Added test coverage with gock. Tested with manual backup of posts which contain attachments(embedded/non-embedded). We can't add e2e tests with attachments, since API to create new conversations requires delegated access.
* Note that https://github.com/alcionai/corso/issues/4991 is still an open item. We don't have a resolution for it right now, since attachment endpoint requires delegated token. Defaulting to let the backup fail for "too many attachments" error case. We don't know yet if we'd see that with group mailboxes, and whether it'd be the same error that we saw with exchange (recurring 503s).
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* Adding a retry for 400 "invalidRequest" errors. Restricting it to GET requests to limit the scope of the fix.
* Retrying at adapter wrapper `Send` level instead of doing it in retry middleware. If we do it in middleware, we'll have to resort to a string search of "invalidRequest" in the http response, which is not the best way. kiota-http nicely parses the http response into an odata error after the request has finished. So doing retry after that in adapter wrapper `Send` is better as we have better guarantees on it being an odata error type.
* Also added a small test optimization to reduce test run time.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Follow up from https://github.com/alcionai/corso/pull/4955#discussion_r1440922835
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
sets list item fields based on columns created while list creation
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
first step towards having a centralized set of error sentinels that can be passed around corso instead of re-using low level error sentinels like those found in the graph/errors.go file.
This PR works as a standalone, and only handles the lowest hanging fruit. SDK consumers will need to change their error enum references, but all api behavior should remain the same.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #4685
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
ignores accessRequest list templates from backup and restore
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
refactors sharepoint lists collection by:
- renaming `Collection` interface as `preFetchCollection` interface
- making `finishPopulation` method as independent function
- seralizing list using `serializeContent` function
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
skips documentLibrary templated lists from:
- restoration
- details
but the lists with 'documentLibrary' are backed-up.
this is in addition to 'webTemplateExtensionsList' templated lists which were previously skipped from restore.
the `skipping` of lists with template 'documentLibrary' is temporary and to be enabled with correct handling.
**update:27/12/2023**
skips `documentLibrary`, `sharingLinks`, `webTemplateExtensionsList` templated lists from being backed-up altogether
site contents of `https://10rqc2.sharepoint.com/sites/CorsoCI`:
backup details
```
./corso backup details sharepoint --backup 5e0a13b0-bff0-471b-8860-2aec869717b2
Logging to file: /home/hitesh/.cache/corso/logs/2023-12-23T14-52-37Z.log
Connecting to M365 done
Connecting to repository 30s done
ID List Items Created Modified
c7ffe2f0a275 Test List 0 2023-05-12T12:28:51Z 2023-05-12T12:28:51Z
5345df87b565 Meh 0 2023-05-12T12:59:50Z 2023-05-12T12:59:50Z
cd2854591ab4 corso-restore-list_135b9067-0e19-4a5c-9475-ced51730c211 0 2023-12-11T10:25:27Z 2023-12-23T13:19:24Z
0f56b7f450fa corsorestorelist_newlist 0 2023-12-11T10:31:54Z 2023-12-11T10:31:54Z
3a7be2fb9d50 new list 0 2023-11-29T18:29:07Z 2023-11-29T18:29:07Z
edab0c775e5e New List Name 0 2023-05-12T12:39:06Z 2023-12-23T13:25:39Z
4d7ad9b1df98 MockListing 0 2023-05-12T12:15:14Z 2023-12-14T06:22:36Z
d2bdca596f75 integration-test-list 6 2023-11-30T06:57:01Z 2023-12-14T06:09:45Z
c9eccc9771a3 Sharing Links 74 2023-06-26T07:21:31Z 2023-12-14T06:06:31Z
5e79dcd93c92 New List Name 2 0 2023-05-12T12:40:15Z 2023-05-12T12:40:15Z
6a6acbaa0d10 corsorestorelist_7f99d792-cae0-4a87-9e68-fd930f5b123e 0 2023-12-11T10:28:25Z 2023-12-23T13:19:57Z
```
restore details
```
./corso restore sharepoint --backup 5e0a13b0-bff0-471b-8860-2aec869717b2
Logging to file: /home/hitesh/.cache/corso/logs/2023-12-23T14-53-37Z.log
Connecting to M365 done
Connecting to repository 26s done
Restoring to folder Corso_Restore_23-Dec-2023_14-54-05
Restoring ∙ https://10rqc2.sharepoint.com/sites/CorsoCI
Discovered 11 items in backup 5e0a13b0-bff0-471b-8860-2aec869717b2 to restore
Enumerating items in repository 1m2s done
Restoring data 50s done
Restore Complete
Restored 11 items
ID List Items Created Modified
edab0c775e5e Corso_Restore_23-Dec-2023_14-54-05_New List Name 0 2023-12-23T14:54:23Z 2023-12-23T14:54:23Z
cd2854591ab4 Corso_Restore_23-Dec-2023_14-54-05_corso-restore-list_135b9067-0e19-4a5c-9475-ced51730c211 0 2023-12-23T14:54:25Z 2023-12-23T14:54:25Z
5345df87b565 Corso_Restore_23-Dec-2023_14-54-05_Meh 0 2023-12-23T14:54:26Z 2023-12-23T14:54:26Z
4d7ad9b1df98 Corso_Restore_23-Dec-2023_14-54-05_MockListing 0 2023-12-23T14:54:28Z 2023-12-23T14:54:28Z
d2bdca596f75 Corso_Restore_23-Dec-2023_14-54-05_integration-test-list 6 2023-12-23T14:54:29Z 2023-12-23T14:54:31Z
c7ffe2f0a275 Corso_Restore_23-Dec-2023_14-54-05_Test List 0 2023-12-23T14:54:34Z 2023-12-23T14:54:34Z
c9eccc9771a3 Corso_Restore_23-Dec-2023_14-54-05_Sharing Links 74 2023-12-23T14:54:36Z 2023-12-23T14:54:36Z
5e79dcd93c92 Corso_Restore_23-Dec-2023_14-54-05_New List Name 2 0 2023-12-23T14:55:09Z 2023-12-23T14:55:09Z
6a6acbaa0d10 Corso_Restore_23-Dec-2023_14-54-05_corsorestorelist_7f99d792-cae0-4a87-9e68-fd930f5b123e 0 2023-12-23T14:55:10Z 2023-12-23T14:55:10Z
0f56b7f450fa Corso_Restore_23-Dec-2023_14-54-05_corsorestorelist_newlist 0 2023-12-23T14:55:11Z 2023-12-23T14:55:11Z
```
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Skips backup of non-recoverable list templates.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Add code and tests for sanitizing emails
Also adds code for sanitizing events and contacts but we can remove
that if we'd like to address it if/when needed. We haven't yet seen
or tried to create instances of those
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
replaces list deletion, on creation of list items failure, with fault bus. Thereby restores list with list items that could be restored.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
We've found that it's possible to get malformed JSON back from
the graph server and graph SDK. This results in errors while trying
to deserialize objects that are malformed JSON
This adds a simple JSON sanitizer and tests to make sure that even if
graph and the graph SDK happen to generate malformed JSON we can get
back to a state where we can deserialize it again. The tests also
print info when the received content differs from the original
This PR does not change any of the logic that corso uses during
backups or restores, it just adds sanitization code and tests
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
enables sharepoint to use lists restore handler for lists ops
Changes previously approved in:
- https://github.com/alcionai/corso/pull/4854
- https://github.com/alcionai/corso/pull/4910
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
