849 Commits

Author SHA1 Message Date
ashmrtn
cd41d2fbce
Reduce test to just a single site to avoid failure (#5274)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-03-25 20:16:43 +00:00
Abhishek Pandey
f28e79c098
Log token lifetimes on 401 errors (#5239)
<!-- PR description-->

* When we encounter 401s, process the JWT token present in the [`Authorization` header](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow#use-a-token) of request. 
* Dump the issued at time and expires at time for the token. 
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-02-18 00:20:55 +00:00
Keepers
8bdf86bbad
apply missing opts to api client (#5233)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-02-15 17:25:49 +00:00
Abin Simon
bf52fdbe6a
Log every 1000 items when exporting (#5227)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-02-15 06:45:16 +00:00
ashmrtn
90d6db486b
Log if the context expired during retry (#5229)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-02-15 01:24:42 +00:00
Keepers
f10730cf98
add new control opt for skipping event 503s (#5223)
adds a new control option for skipping certain event item 503 failures.
Also adds a skip cause for that case. And exports the skipCause value
for future preparation.
2024-02-14 16:55:52 -07:00
Keepers
bb2bd6df3f
add authentication to requester (#5198)
the graph requester for large item downloads now includes the option to authenticate requests.  The option is configured at the time of creating the requester, therefore all requests using that servier are either authenticatd or not. In our case, we're opting to authenticate all requests, since we do not use this requester for non-graph api calls, and even if we did the addition of auth headers is likely benign.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💚 E2E
2024-02-14 17:50:36 +00:00
ashmrtn
28aba60cc5
Check for and retry 404s with no content (#5217)
We've started seeing 404 errors with no content being returned. Check for these in the http wrapper we use and retry them.

While graph SDK returns an error for this sort of situation it's a very basic error since it normally expects to parse info out of the response body. Therefore it should be safe to inject our own error that we can check for.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-02-13 02:26:04 +00:00
Abin Simon
f0b8041c3f
Fix possible panic in contacts fetch (#5205)
We were not checking for the error returned by the Get method before trying to use the result to get contact info.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-02-10 07:57:57 +00:00
Abhishek Pandey
f92f811559
Retry more errors in graph adapter wrapper (#5203)
<!-- PR description-->

* We have started seeing `io.ErrUnexpectedEOF` and `read: connection timed out` in the last 2 days for exchange backups. Retry those.
* Also increase retry count from 3 to 6. This is more of a hail mary to retry `InvalidAuthenticationToken` errors. We have observed that retries do help. But for a small set of requests, we end up exhausting retries and eventually fail with InvalidAuthenticationToken error. Hoping that bumping this to 6 will get us some relief. This fix may be removed if we find the rootcause/pattern behind this.
* The event list test was taking > 150 secs. Thought I'd push this change as its a small fix. I first thought my PR https://github.com/alcionai/corso/pull/5202 broke the test. So I investigated this.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-02-10 05:11:05 +00:00
ashmrtn
71a9087e4d
Add a fallback to non-delta pager during event item enumeration (#5201)
If the delta pager with a smaller page size also falls then attempt to
use the regular events endpoint to enumerate items

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-02-10 00:45:43 +00:00
ashmrtn
2ab3c890b4
Retry events item enumeration with smaller page size for some cases (#5194)
Update the events item enumerator to switch to a smaller page size for
the delta pager if we fail enumeration with a 503 error and no content.
We've found that that situation is indicative of the Graph server
being slow and a smaller page size allows us to make progress still

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test (in another PR)
- [ ] 💚 E2E
2024-02-08 07:43:20 +00:00
Keepers
f00dd0f88a
introduce intg and unit test setup packages (#5140)
corso has a thousand bespoke approaches to setting the same info in all of its tests.  This is a first step towards minimizing and standardizing the lift around that work. Future PRs will distribute these packages through the repo.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-02-07 18:01:35 +00:00
Abhishek Pandey
c1ec1585a2
Skip exchange items which fail to download due to ErrorCorruptData error (#5191)
<!-- PR description-->

* We are seeing 500 errors from graph during exchange(email) backup. * Error is `:{"code":"ErrorCorruptData","message":"Data is corrupt., Invalid global object ID: some ID`. 
* Catch the error and add a skip since these items cannot be downloaded from graph. 

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-02-07 02:40:14 +00:00
Keepers
a680f13f84
add by-resp-code catcher to err transformer (#5184)
Start catching graph responses by broad response code, in addition to the response body details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-02-06 18:02:34 +00:00
Keepers
e86592f51e
replace graph Wrap and Stack with clues (#5018)
Now that graph errors are always transformed as part of the graph client wrapper or http_wrapper, we don't need to call graph.Stack or graph.Wrap outside of those inner helpers any longer. This PR swaps all those graph calls with equivalent clues funcs as a cleanup.

Should not contain any logical changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4685 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-02-06 01:08:26 +00:00
Abhishek Pandey
7e2b9dab62
Add group mailbox export (#5153)
<!-- PR description-->

* Add EML exports for group mailbox.
* Tested E2E manually along with unit tests added in this PR.
* Will follow it up with a sanity test PR.
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-31 05:14:59 +00:00
Abhishek Pandey
1537db59c4
Convert serialized posts to eml (#5152)
<!-- PR description-->

* Add the serialized `Postable` to EML converter code for group mailboxes and associated tests.
* We should converge this with `Messageable` code at some point with a small translator function which converts `Postable` to `Messageable`. Reason being that posts are a subset of messages. Although I feel it's a bit premature right now. Once we have tested this in prod a bit, and we know for a fact that posts are indeed a true subset of messages, we can do this convergence. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-30 21:00:50 +00:00
Keepers
80d7d5c63d
boilerplate teamschat collection package (#5087)
seems like a lot of code, but this is 95% boilerplate additions copied from groups collections packages, with a find-replace for names.

Some noteworthy differences:
* teamsChats does not handle metadata, so all metadata, delta, and previous path handling was removed
* teamsChats does not produce tombstones
* chats are never deleted, so no "removed" items are tracked
* all chats gets stored at the prefix root, so no "containers" are iterated, and therefore only one collection is ever produced.

This means that, overall, while the boilerplate here is still the same, it's much reduced compared to similar packages.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #5062

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-30 20:04:49 +00:00
Abhishek Pandey
50ba30539a
Persist metadata files for group mailbox posts (#5135)
<!-- PR description-->

* Each post now has a `.data` and `.meta` file.
* I've only made changes to groups lazy reader to keep this PR short. Note that channels don't have meta/data file concepts, we don't want to accidentally enable this for channels. Given channels doesn't use lazy reader, this is safe to do short term.
* I'll be adding small follow up PRs to 1) Make sure channels files don't get assigned `.data` suffix. 2) Support for data and meta files for prefetch conversations backup. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-30 11:10:07 +00:00
Hitesh Pattanayak
576c9f6b53
handles single & multiple values for metadata columns (#5121)
handles single & multiple values for metadata columns

Similar to `Hyperlink` and `Column` columns, `Metadata` column too is unrecognizable from GRAPH API response. Hence identifying from the field column names.

`Metadata` fields are like tags. A `Metadata` fields can be configured to hold multiple values/tags

**Original List with `Metadata` column (Department) with single value/tag**:
![Metadata-List](https://github.com/alcionai/corso/assets/48874082/0b913a2a-46d5-4d9c-83f9-69a5236b1024)

**Restored List with `Metadata` column with single value/tag**:
![Restored-Metadata-List](https://github.com/alcionai/corso/assets/48874082/9420012b-345c-4fac-90c3-c0d421b2edfb)

**Original List with `Metadata` column (Department) with multiple value/tag**:
![Metadata-List-Multi](https://github.com/alcionai/corso/assets/48874082/054ef4a1-c46e-48ba-b410-a95b540cde33)

**Restored List with `Metadata` column with multiple value/tag**:
![Restored-Multi-Metadata-List](https://github.com/alcionai/corso/assets/48874082/ef6c904b-e431-4a85-9ef2-f08bcf8e21e4)


#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)
#5084 
#5108 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-30 06:35:33 +00:00
Hitesh Pattanayak
08d4803ebe
handles multiple lookup field values (#5112)
handles multiple lookup field values.

**Original `Lookup` list with single value**:
![Lookup-List-Single](https://github.com/alcionai/corso/assets/48874082/6a6b68cf-8fb9-4dfb-985e-702c4d74d3f0)

**Restored `Lookup` list with single value**:
![Restored-Lookup-List](https://github.com/alcionai/corso/assets/48874082/f97ac974-6a3b-4dd2-82c5-9f3596f9adaa)

**Original `Lookup` list with multiple values**:
![Lookup-List-Multi](https://github.com/alcionai/corso/assets/48874082/5f8b1b92-297f-4a66-b0b6-b5007d430690)

**Restored `Lookup` list with multiple values**:
![Restored-Lookup-List-Multi](https://github.com/alcionai/corso/assets/48874082/6c6d79ca-775d-4f50-abee-8090a28f3871)


#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)
#5108 
#5084 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-30 05:49:00 +00:00
Hitesh Pattanayak
734fd7239e
handles multiple persons list items (#5111)
handles multiple persons list items.

**Original `person` list with single value**:
![Person-List](https://github.com/alcionai/corso/assets/48874082/a4a87cde-f907-4fc7-94da-f9ddda0f5a18)
 
**Restored `person` list with single value**:
![Restored-Person-List](https://github.com/alcionai/corso/assets/48874082/6b5c2a8b-743c-4020-9393-356d28948bf0)

**Original `person` list with multi value**:
![Person-List-Multi](https://github.com/alcionai/corso/assets/48874082/18d2c536-67ac-4b28-87be-2352764f2c95)

**Restored `person` list with multi value**:
![Restored-Person-List-Multi](https://github.com/alcionai/corso/assets/48874082/f9694e0d-d2cc-48d9-94f2-16b61c5b7cdb)

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)
#5108 
#5084 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-30 04:55:03 +00:00
ashmrtn
d2f1bbb5c7
Skip ms auth checks for gock client (#5163)
Disable sending requests to microsoft servers to get API tokens when
using the gock client. This is accomplished by passing in a mock that
always returns no error for auth requests.

**This PR does not go through and make existing tests using gock unit
tests instead of integration tests. That will need to be done
separately**

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #5124

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-30 02:41:53 +00:00
Keepers
8e6a47b103
add chats service and category to paths (#5065)
introduces the Chats service and Chats category

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #5061

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-29 22:03:31 +00:00
Hitesh Pattanayak
8ac7e6caa2
provisions to identify fields with multiple value (#5109)
provisions to identify fields with multiple value.
Some fields can hold multiple values and single value based on a property called `allowMultipleValues`.
For example, `Lookup` column:
```
"lookup": {
    "allowMultipleValues": true,
    "allowUnlimitedLength": false,
    "columnName": "Title",
    "listId": "21b45bf2-e495-4582-b114-839577ff8e4f"
}
```
But `choice` columns, even though allows to set multiple choices/value, does not have that particular field `allowMultipleValues` to indicate.
So in this PR we are trying determine the same by the stored values while restoring.

**Original list with `choice` column in site**:
![Choice-List-Multi](https://github.com/alcionai/corso/assets/48874082/d4457b3c-0230-4a69-8467-f64b7d7d4f04)

**Restored list with `choice` column in site**
![Restored-Choice-List-Multi](https://github.com/alcionai/corso/assets/48874082/78478055-0e84-43d0-ac83-262b564ce778)
The color does not come through though


#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)
#5108 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-29 21:27:35 +00:00
ashmrtn
e1cb5b6313
Test closing repo without connect works (#5115)
Just make sure we can call Close() on the repository struct even if we have made a connection to kopia or initialized all the internal fields.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-29 17:52:03 +00:00
Abhishek Pandey
8133da3087
Centralize metadata suffix definitions (#5134)
<!-- PR description-->

Centralizing the suffixes so that group mailbox and future services can also use them.

* Move `.meta`, `.data`, `.dirmeta` definitions from `internal/m365/collection/drive/metadata` pkg to `pkg/services/m365/api/graph/metadata`.
* Adjust package names in misc places.
* No logic change. 
* We should probably keep all metadata definitions and code in one place, i.e. relocate drive metadata definitions to `pkg/services/m365/api/graph/metadata`. This is a bigger change, we can do it later.
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-26 22:06:04 +00:00
Keepers
e1457c23bf
show data category in backup list (#5023)
Adds the backup's data category to the cli output.

before:
```
  ID                                    Started At            Duration    Status     Resource Owner
  09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9  2024-01-12T17:54:33Z  16.103569s  Completed  rfinders@10rqc2.onmicrosoft.com
  893ef934-9403-4775-b667-09f77cd26e83  2024-01-12T17:55:22Z  5.66253s    Completed  rfinders@10rqc2.onmicrosoft.com
```

after:
```
  ID                                    Started at            Duration    Status     Protected resource               Data
  09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9  2024-01-12T17:54:33Z  16.103569s  Completed  rfinders@10rqc2.onmicrosoft.com  Contacts,Emails,Events
  893ef934-9403-4775-b667-09f77cd26e83  2024-01-12T17:55:22Z  5.66253s    Completed  rfinders@10rqc2.onmicrosoft.com  Emails
```

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3748

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2024-01-24 20:22:08 +00:00
Abhishek Pandey
b073c204c8
Store parent post content (#5096)
<!-- PR description-->

* Expand inReplyTo property to additionally get the parent post contents. This will be persisted as part of post data.
* This additional data will be useful for building a reply tree if we decide to do in-order restore/export in future.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-24 06:36:44 +00:00
ashmrtn
5a05223e90
Use more unique data in the kopia SourceInfo for snapshots (#5093)
Use unique info for the kopia snapshot source information to help
isolate snapshots for different resources from each other. This will
come in handy both for checkpoint snapshots and regular snapshots if
kopia's retention policy happens to look for things.

Integration tests for this probably won't pass until the next PR in
the series is merged into this PR

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

merge after:
* #5091
* #5092

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-24 02:46:57 +00:00
ashmrtn
e45b3565ad
Create a helper to go from a reason to a reason with the metadata service of the original reason (#5092)
Will be used in later PRs to help make long-running streamstore
snapshots safer to create if they ever go past the kopia checkpoint
interval

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

merge after:
* #5091

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-23 20:47:04 +00:00
ashmrtn
dd9bba6caa
Create helper function to go from service type to metadata service type (#5091)
Will also be used more in later PRs

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-23 19:32:25 +00:00
Keepers
d501a30761
use the tree-based drive backup by standard (#4980)
reverses the drive backup process toggle by making the tree-based backup standard, and the old, linear enumeration the toggleable fallback.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4690

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-23 18:57:11 +00:00
Hitesh Pattanayak
dfb4a73f56
modifies list restore handler and handles lists advanced restore (#5024)
- includes GetList in list restore handler
- includes GetListsByCollisionKey in list restore handler
- updates mock list restore handler


#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-23 16:18:47 +05:30
Abhishek Pandey
90648ef564
Augment post details with recipient information (#5027)
<!-- PR description-->
Posts are always sent to the group email address, along with additional
recipients if any. Currently, we don't have a way to get the unique
recipient list for individual posts due to graph api limitations.

Store the group mail address so that we can use it to populate the 'To'
field during Post -> EML exports.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-23 05:18:37 +00:00
Abin Simon
6d2d9c0099
Log more clues for export operations (#5068)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-22 04:36:52 +00:00
Keepers
89a6a76630
fix item property selection when getting drive items (#5077)
we already select these properties when paging drive items, we just missed the configuration on per-item GETs.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-21 06:43:27 +00:00
Keepers
232ebba13a
catch basic notFound graph errors (#5052)
make the NotFound error comparison more lenient so that it catches the broader set of not-found error responses from graph.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-17 22:57:15 +00:00
ashmrtn
b8b1299514
Cleanup immutable ID config passing (#5013)
This value was present in the control.Options in the struct and
passed as a parameter. This PR removes the parameter so it's clear
the control.Options one is used

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #5012

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-17 00:51:11 +00:00
Keepers
6e50d5216e
remove builder.toServicePath from paths (#5028)
path.Build produces the same result with more standard and centralized behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4025

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-16 16:42:49 +00:00
Abin Simon
d0d661398c
Fix nightly user not found tests (#5034)
Similar to https://github.com/alcionai/corso/pull/4990
Change due to https://github.com/alcionai/corso/pull/5017

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2024-01-15 10:29:56 +00:00
Hitesh Pattanayak
8989fcd7cd
handles hyperlink columns and fields (#4971)
Hyperlink column of a list item is not identifiable from GRAPH's
response.
Hence this is a workaround to handle such fields until a column
definition is introduced for `Hyperlink`.

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-13 13:07:28 +05:30
Hitesh Pattanayak
06afd53660
handles location columns and fields (#4970)
Location column of a list item is not identifiable from GRAPH's
response.
Hence this is a maneouver to handle such fields until a column
definition is introduced for `Location`.

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E

---------

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2024-01-13 11:35:28 +05:30
Keepers
dab53502ae
minor logging and cli output tweaks (#5016)
just touchups for supportability and tracking, plus a ux tweak here and there.  No real logic changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-12 22:25:07 +00:00
Keepers
8c2d7c3c18
log api call start-end times (#5025)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
2024-01-12 20:53:25 +00:00
Keepers
9aa32d2755
allow groups to have no display name (#5020)
relax conditions on groups by allowing them to have an empty display name.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* Fixes #4918

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2024-01-12 19:16:35 +00:00
Hitesh Pattanayak
c43ad6c517
get lists with collision key (#5009)
gets lists with collision key

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-12 17:50:54 +00:00
Hitesh Pattanayak
e16d4c5bd9
select list by name while export and restore (#4999)
selects list by name while export and restore

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)
#4754 
#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-12 15:54:58 +00:00
Abin Simon
303b8c31ce
Cancelled and modified events in ics export (#4996)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3890

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-12 10:41:10 +00:00