17 Commits

Author SHA1 Message Date
Keepers
680ec2b751
adding fault/clues to non-service connector pkgs (#2380)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-13 16:54:09 +00:00
Keepers
dad6776861
replace multierror with fault in setSites (#2377)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-09 18:11:59 +00:00
Keepers
faad5d35a4
refactor structured errors/logs in pkg (#2323)
## Description

Refactors errors and logs to support structured
data throughout the smaller packages within
pkg/...  Larger packages will come later.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
2023-02-06 20:45:09 +00:00
Keepers
24911a358b
quick graph client refactor (#2301)
## Description

A couple minor refactors.  First, combines the
graph/service_helper.go and graph/service.go
files into a single file (just lift'n'shift, no logic
changes).  Second, renames CreateHTTPClient
to just HTTPClient.  Third, replaces the new
LargeItemClient constructor with extensible
options for the http client, starting with a
NoTimeout option.  This will provide longer
term maintenance over multiple constructors.

This change sets up QoL for building multiple services
and clients within api layer handlers, which will appear
in an immediately following PR.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2299

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-27 19:34:00 +00:00
Vaibhav Kamra
bdb7f2b109
Re-use client for OneDrive item downloads (#2276)
## Description

This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client. 

Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success

Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-01-26 15:05:12 +00:00
Vaibhav Kamra
e9354a8429
Revert "add itemClient for re-usable od item downloads" (#2273)
Reverts alcionai/corso#2266. Am seeing issues on data restore that need to be debugged.
2023-01-26 07:52:56 +00:00
Keepers
bda8a5c60c
add itemClient for re-usable od item downloads (#2266)
## Description

onedrive currently constructs a new http client
for every file it downloads.  This causes the OS
to generate extra sockets, and hang onto them
after the download is complete.  Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* closes #2262

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-26 04:27:52 +00:00
Keepers
b3d4b4687b
discovery api, filter guest and external users (#2188)
## Description

Adds the api client pkg pattern to the connector/
discovery package.  Most code changes are plain
lift-n-shift, with minor clean-ups along the way.

User retrieval is now filtered to only include
member and on-premise accounts.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2094

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-01-20 20:09:59 +00:00
Keepers
edc4426b9c
Fix --users "*' not finding user in tenant (#2033)
## Description

DataCollections validation step was still using the full resourceOwner list in the selector to validate every backup, rather than checking only the DiscreteOwner.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
2023-01-05 00:29:21 +00:00
Keepers
8a29c52cdc
move graphService to graph as Service (#1790)
## Description

Relocates the graphService struct to graph as
the Service struct.  Replaces GC's embedded
graphService with a graph.Servicer reference.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-14 01:39:00 +00:00
Keepers
a6aa86ce5c
add a weburl to siteid reducer in gc (#1671)
## Description

Adds a func in graphConnector that reduces
siteIDs and webURLs into a set of siteIDs.  This
will be used by callers such as the CLI to
generate id-based selectors for sites even if
they handle webURLs as an alternative id.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-08 15:38:24 +00:00
Vaibhav Kamra
5533ada9be
Refactor user query and return user display name (#1679)
## Description

Move user discovery into a `discovery` package. Also use that to return user display name from `m365.Users()` call.

This will also allow us to remove `Users` from the GC struct going forward.

If this pattern works - will follow up with a similar change for `Sites()`.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-06 01:45:51 +00:00
ashmrtn
27d0980526
Split backups by (resource owner, service) at CLI layer (#1609)
## Description

Sketch of how to split backups by (resource owner, service) for exchange. Needs more support from selectors package to be complete.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1505

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:50:49 +00:00
Keepers
07a8d13d1e
adds tenant site lookup on new graph conn (#1540)
## Description

Adds a lookup for all tenant site ids when a new
graph connector is created.  Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-18 00:31:41 +00:00
Eno Thereska
fb46f5c7f3
Export call to get user ids (#1069)
## Description

Exposing user IDs in addition to user emails.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X]  Unit test
- [ ] 💚 E2E
2022-10-17 18:17:39 +00:00
Danny
3f3c8217a3
GC: Component: Version 40.0 (#917)
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.  
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-22 18:59:51 +00:00
Vaibhav Kamra
16ad25f3a1
Implement wrapper around M365 helper methods (#825)
## Description

Exports functionality useful for external services/validators

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-13 10:32:02 +00:00