63 Commits

Author SHA1 Message Date
Abhishek Pandey
09ca1b4ae9 Share count bus between graph clients and backup operations 2023-11-08 16:29:55 -08:00
Abhishek Pandey
110b477346 Var renaming & tests 2023-11-06 15:41:02 -08:00
Abhishek Pandey
07740938fb Add sliding window rate limiter 2023-11-02 03:08:55 -07:00
Keepers
5e0014307c
remove ctx embedding for counter (#4497)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-31 16:15:05 +00:00
Keepers
7196b5d278
catch and sentinel resource locked error case (#4465)
handle cases where a resource is found, but is not accessible due to being locked out by an administrator or msoft process.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4464

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-10 19:24:48 +00:00
Abhishek Pandey
eb0299d316
Push item download requests through drive rate limiter (#4393)
<!-- PR description-->

**Changes:** 
1. Count item download requests under drive rate limit quota . Currently this is under exchange limiter.
2. Use 1 rate limit token instead of 2 for drive calls by default. 
3. Use 2 tokens instead of 1 for initial delta query ( which has no token). 

Sharing internal docs separately to go along with the review. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-10 16:45:37 +00:00
ashmrtn
6f25be4ad2
Rename generic item structs and functions (#4421)
`unindexedPrefetchedItem` -> `prefetchedItem`
`prefetchedItem` -> `prefetchedItemWithInfo`
`unindexedLazyItem` -> `lazyItem`
`lazyItem` -> `lazyItemWithInfo`

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4328

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-09 19:12:35 +00:00
Abhishek Pandey
757007e027
Skip graph call if the download url has expired (#4419)
<!-- PR description-->

Builds on top of earlier PR #4417 to skip graph API call if the token has already expired. This is a performance optimization.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
- [x] Performance Opt

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* internal

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-09 13:42:54 +00:00
Abin Simon
363cbca86f
Backup/Restore/Export multiples sites in Groups (#4344)
I've updated the Team used in the CI to include private and shared channels. Sanity tests should ideally do the e2e tests for multi site backups.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-10-02 13:17:15 +00:00
ashmrtn
f0ccf35b5c
Inject versions in generic item (#4409)
Leverage the generic item struct to inject
serialization format information for all
items

Unwires the old code that injected versions
in kopia wrapper but leaves some code in
the wrapper to strip out the serialization
format during restore

Future PRs should move the process of
pulling out serialization format to
individual services

Viewing by commit may make review
easier

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4328

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-29 20:16:01 +00:00
ashmrtn
a806ab59bf
Use generic unindex item struct (#4366)
Switch all metadata files (of all types) to use the generic unindexed
item type. Transitioned items include:
  * previous paths and deltas for all services
  * site list for groups
  * drive .meta files

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-29 04:15:45 +00:00
Keepers
f37e58ee5c
graph debug logging improvements (#4368)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
- [x] 🧹 Tech Debt/Cleanup
2023-09-26 22:01:00 +00:00
Keepers
0474e5c957
separate delta not supported from invalid (#4375)
groups backup has a failing case where delta
queries are not supported, but we aren't distinguishing
that case from the invalid delta case as expected. This
results in backup failures, instead of success with no data.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 18:37:15 +00:00
Keepers
0545365b12
move container enum to pager pattern (#4298)
moves exchange container enumeration funcs in
the api to the pager pattern established with item enumeration.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 17:34:06 +00:00
Keepers
6bc40ea2aa
add throttling, token counters to operation (#4318)
Adds context embedding to pkg/count, and embeds
the backup, restore, and export operation ctx with the operations count.bus.  The bus is extracted from the ctx in api middleware to track throttling response counts, as well as token consumption counts.  Counts are globally tracked, not on a per-time or per resource basis.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 00:43:16 +00:00
Abin Simon
ccf8df3548
Bump msgraph sdk to 1.19.0 (#4314)
Needed some changes in the func names.

``` bash
$ sed -i "s/By\([a-zA-Z]*\)IdString/By\1Id/g" **/*.go
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-21 07:01:07 +00:00
Keepers
c1ec3c6648
formalizes ErrApplicationThrottled (#4317)
adds graph error identification for application throttled and adds an ApplicationThrottled sentinel to pkg/errs for sdk consumer identification.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-20 21:27:01 +00:00
Abin Simon
ce422f0b1d
Add linter for checking with empty string (#4246)
Finding failing cases:

``` bash
tree-grepper -q go '((binary_expression (identifier) ["==" "!="] (interpreted_string_literal) @ri) (#eq? @ri "\"\""))'
```

Fixing failing cases:

``` bash
comby 'if :[1~[^ ]*] == ""' 'if len(:[1]) == 0' -matcher .go -in-place
comby 'if :[1~[^ ]*] != ""' 'if len(:[1]) > 0' -matcher .go -in-place
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/3654

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-15 06:45:00 +00:00
Keepers
cb319bb2ae
use display name to look up groups (#4242)
Allows lookup of groups using their display name in addition to their ID.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-14 17:03:14 +00:00
Keepers
a2e80a178a
path pkg rename resouceOwner to protectedResource (#4193)
updaing the path package to the current naming convention. No logic changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-12 22:05:23 +00:00
neha_gupta
75c5cb3582
fix sanity test for groups (#4226)
<!-- PR description-->

Handle-
- empty channel
-  do to add item if deleted

#### Does this PR need a docs update or release note?


- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
2023-09-12 18:30:55 +00:00
Abin Simon
b9c8bcbd30
Ignore permissions restore failure with a log (#4220)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-11 22:00:24 +00:00
Keepers
d3eda28989
enable groups integration tests (#4161)
enables nightly cli e2e tests, operations layer integration tests, and sanity tests for groups and teams.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-08 20:18:59 +00:00
Abin Simon
be59928f98
Fix cases where we had a trailing comma (#4208)
Not sure if we wanna merge it as it might generate way too many conflicts, but this should help us add a linter in CI. If we are good, I'll add something that can do lints for this in a follow up PR.

Super hacky, but this fix was created using `while true ; do tree-grepper -q go '(argument_list "," @nope .)' | tail -n1| awk -F: "{print \$1,\"+\"\$2\" -c ':norm \$xJZZ'\"}" | xargs vim ; done`.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3654

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-08 17:10:29 +00:00
Keepers
bf02f57527
add a BuildMetadata func to path package (#4192)
just a QoL shorthand to match the Build func.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [=x] 💚 E2E
2023-09-07 19:05:58 +00:00
Abin Simon
755db5ae92
Handle groups when setting currency middleware defaults (#4190)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-07 08:07:07 +00:00
Abin Simon
eb8f3131f9
Misc fixes for handling groups (#4195)
A bunch of places where we missed handling groups
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-07 06:36:18 +00:00
dependabot[bot]
ccdb672026
⬆️ Bump github.com/microsoftgraph/msgraph-sdk-go from 1.16.0 to 1.17.0 in /src (#4167)
Bumps [github.com/microsoftgraph/msgraph-sdk-go](https://github.com/microsoftgraph/msgraph-sdk-go) from 1.16.0 to 1.17.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/microsoftgraph/msgraph-sdk-go/blob/main/CHANGELOG.md">github.com/microsoftgraph/msgraph-sdk-go's changelog</a>.</em></p>
<blockquote>
<h2>[1.17.0]- 2023-08-30</h2>
<h3>Changed</h3>
<ul>
<li>Weekly generation.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="bd9689c25d"><code>bd9689c</code></a> Generated  models and request builders (<a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/563">#563</a>)</li>
<li>See full diff in <a href="https://github.com/microsoftgraph/msgraph-sdk-go/compare/v1.16.0...v1.17.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/microsoftgraph/msgraph-sdk-go&package-manager=go_modules&previous-version=1.16.0&new-version=1.17.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-09-05 16:45:27 +00:00
Abhishek Pandey
c1c4218994
Add common code to check if services are enabled (#4127)
<!-- PR description-->

Currently corso CLI & SDK have different checks for services enabled. These checks have diverged which can lead to bugs. For e.g. SDK users use [code](3e43028a88/src/pkg/services/m365/users.go (L92)) to check if OD is enabled, while corso uses [this code](https://github.com/alcionai/corso/blob/main/src/pkg/services/m365/api/users.go#L174). These funcs have different checks. This PR introduces common helpers to consolidate these checks in one place.


- Note: I decided against absorbing these helpers into api/users.go, because separation of concerns -  getters shouldn't arbitrate on return vals. 
- Note that this code is not yet wired up to SDK/CLI. It'll be added in a following [PR](https://github.com/alcionai/corso/pull/4096). 


**Changes**
1. `Is*ServiceEnabled` helpers in internal/m365/common.go. 

2. Add `GetMailboxInfo` to common code. This is currently a duplicate of code in `GetInfo`. That function will be removed once we port SDK users to `GetMailboxInfo`. 

3. Unit tests for common code. Integration tests will be added in a later PR
- Note:`TestIsOneDriveServiceEnabled` is copied from `TestCheckUserHasDrives`. The older test will be removed in a later PR.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3844

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-04 18:59:03 +00:00
Abin Simon
7c281d3fa0
Metadata related path changes (#4156)
This adds a `MetadataPathPrefix` func to any drive related handlers. And in case of exchange, we will create a path. These generated paths, in both scenario is what now gets passed to `MakeMetadataCollection` instead of passing the different values.
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4154

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-08-31 08:20:37 +00:00
Keepers
1f22b94fc1
small paths code rearrangement (#4007)
small cleanup in paths, primarily splitting files
so that file contents are more clearly owned, which should be a little better for readability and code placement.

Also renames `ServicePrefix` to `BuildPrefix` in
anticipation of multi-service prefixes.

no logic changes, just movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3993

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-11 23:02:45 +00:00
Keepers
0a947386e6
rename data pkg Stream to Item (#3966)
A Stream is a continuous transmission of data.
An item is a single structure.  Crossing the two
definitions generates confusion.

Primarily code movement/renaming.  Though there
is also some reduction/replacement of structs
where we'd made a variety of testable Item implementations
instead of re-using the generic mock.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-11 19:33:33 +00:00
Abin Simon
8c939c0f0d
Split services and collections for OneDrive & SharePoint (#4002)
Only code movement, no code changes.

Moved services to `/internal/m365/services/{onedrive,sharepoint,exchange}`
Moved collections to `/internal/m365/collection/{drive,site}`

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-08-10 06:08:34 +00:00
Keepers
7905adb8e1
test http2 stream errors, update client options (#3994)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
2023-08-09 19:17:38 +00:00
Keepers
99be32686d
centralize graph api panic and conn retry (#3884)
Adds a middleware to the graph api adapter which
watches all http request responses for both panics and connection reset issues.  Panics are
transformed into standard errors.  Connection
resets get retried up to three times.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
2023-08-04 01:19:16 +00:00
dependabot[bot]
c59c1d2408
⬆️ Bump github.com/microsoftgraph/msgraph-sdk-go from 1.12.0 to 1.13.0 in /src (#3931)
Bumps [github.com/microsoftgraph/msgraph-sdk-go](https://github.com/microsoftgraph/msgraph-sdk-go) from 1.12.0 to 1.13.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/microsoftgraph/msgraph-sdk-go/blob/main/CHANGELOG.md">github.com/microsoftgraph/msgraph-sdk-go's changelog</a>.</em></p>
<blockquote>
<h2>[1.13.0]- 2023-07-28</h2>
<h3>Changed</h3>
<ul>
<li>Weekly generation.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="e276af0eaf"><code>e276af0</code></a> Generated  models and request builders (<a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/542">#542</a>)</li>
<li>See full diff in <a href="https://github.com/microsoftgraph/msgraph-sdk-go/compare/v1.12.0...v1.13.0">compare view</a></li>
</ul>
</details>
<br />
[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/microsoftgraph/msgraph-sdk-go&package-manager=go_modules&previous-version=1.12.0&new-version=1.13.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
2023-08-03 02:04:46 +00:00
Keepers
fc6119064b
introduce backupProducerConfig (#3904)
Adds another inject container-of-things to hold
common properties used by backup collection producers.

No logic changes, just code movement, renames, and placing things into structs.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-02 00:38:19 +00:00
Keepers
d2dda00195
update all graph packages (#3879)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-07-24 21:45:57 +00:00
ashmrtn
c073ccc5ea
Add a couple more linters (#3886)
Lint for kv-pairs in log calls and things
that can be replaced with constants from
the standard library

Also add the linters that are enabled by
default just so things are more consistent
and clear going forward

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3654

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-22 01:34:38 +00:00
Keepers
d555501093
add integration tests for missing drives (#3847)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3652

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-21 01:13:09 +00:00
Keepers
5a78f478a1
add operations test for adv rest (#3783)
ads operations level tests for advanced restore
configuration on all three services.
Code is largely boilerplate between each service, but with just enough quirks that full consolidation would require excess jumping through hoops.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3562

#### Test Plan

- [x] 💚 E2E
2023-07-17 03:56:08 +00:00
Keepers
09e5e9464a
add test for 503 resp body (#3812)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* closes #3811

#### Test Plan

- [x]  Unit test
2023-07-14 23:40:30 +00:00
Keepers
fd6dff3270
retry on http/2 stream errors (#3802)
http/2 stream error failures don't allow
us to leverage client middleware.  As a
result we're lacking a retry handler.  This
change adds retries for http_wrapper
in the explicit case of a http/2 stream
error.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-07-14 22:57:51 +00:00
Keepers
5ea194dc87
tweaks and fixes found while testing (#3735)
a variety of small updates that came
from manual testing of restore with various
collision and destination combinations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3562

#### Test Plan

- [x] 💪 Manual
2023-07-06 22:24:26 +00:00
Keepers
2a150cc610
add counter for skip-collision counting (#3722)
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 19:22:02 +00:00
Keepers
8ba79709a6
split tester into separate files (#3762)
This is primarily an exercise in reducing the number of circular imports we get from adding the tester package to other packages.

No logic changes.  Purely movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 15:43:57 +00:00
Keepers
cf495a6c8f
add authErr to users mailbox check (#3744)
license issues can produce an authenticationError when retrieving a user's inbox.  This catches that error and treats it as a "mailbox not availble" condition.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3743

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-03 19:31:35 +00:00
Keepers
56151a82eb
remove discovery pkg (#3677)
Currently unused, and functionality is largely duplicated by the services/m365 package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-01 13:52:57 +00:00
Keepers
8b81728488
fix concurrency limiter init race (#3715)
initializes the concurrency limiter as part of CLI pre-run initialization to ensure we don't accidentally skip out on including the limiter in middleware.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3695

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-30 23:54:54 +00:00
Keepers
b74f65e301
properly declare resourceNotFound for sites (#3728)
Missing sites return an ItemNotFound error code,
which we aren't currently handling when doing
queries for individual sites.  This change catches that error code and stacks the error with a
resourceOwnerNotFound sentinel.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-06-30 18:14:06 +00:00