187 Commits

Author SHA1 Message Date
Georgi Matev
ce755ecf3c
Cleanup birthday events for old contacts (#3019)
Purge events birthdays corresponding to old contacts
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-04 11:53:28 +00:00
Keepers
9eb070c0bc
clean up exchangePurge output (#3009)
Get contact counts isn't working. Just comments out the create for now.

---

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-04-04 00:23:29 +00:00
neha_gupta
4087b69a39
Enhancement Sanity Test: verify permissions and defined data (#2938)
<!-- PR description-->
onedrive -  
- verify permissions - grantedToV2 and roles 
- recursively check subfolder's data and permission

Exchange -
- since the data in email is huge, create custom data and check if that data is getting restored properly
 
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 10:43:48 +00:00
Keepers
7e04cefa2c
replace graphConnector with interface (#2932)
Replaces the operations graphConnector reference
with an interface.  Restore and Backups have
separate, unique interfaces.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-31 15:56:19 +00:00
Georgi Matev
b23faa0fac
CI Cleanup improvements (#2992)
* Allow exchange timestamps from item/folder names based with create time as backup
* Parametrize folder prefixes and move to var from secrets for ease of management. There is nothing sensitive
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-30 23:31:15 +00:00
ashmrtn
79b6c0957f
Re-enable import order linting (#2928)
Fixes current errors as well

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2927

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-28 16:45:28 +00:00
Keepers
9f0e259764
metrics PoC for github.com/armon/go-metrics (#2884)
Adds a logger-oriented metrics tracker to aggregate statistics like requests-per-minute and average
http request time.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2212

#### Test Plan

- [x] 💪 Manual
2023-03-28 15:57:55 +00:00
Keepers
9d73d0c8e2
replace errors.* with clues.* (#2924)
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.

Might follow up with a linter to enforce this change.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-28 04:01:54 +00:00
Keepers
ddeaaf2ff2
Sanity test cleanups and additional info (#2816)
updating sanity tests: fix the permissions check by not using reflect.DeepEquals, and various code cleanup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-03-27 16:25:08 +00:00
Georgi Matev
a28871c4c0
Parametrize libraries (#2942)
Add a secret for configuring SP libraries to purge
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-24 23:36:24 +00:00
Keepers
328d75945b
PoC for centralized throttling control (#2887)
PoC for using a centralized middleware to control
1k-per-minute get requests to graph api.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💪 Manual
2023-03-24 00:40:08 +00:00
neha_gupta
55be2f4e2e
check backup ouptut json (#2926)
<!-- PR description-->

with update in CLI output, have updated the sanity checks

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-23 17:39:39 +00:00
neha_gupta
a9c4479a7b
check subfolders email counts in exchange (#2900)
<!-- PR description-->

check that no of emails in all subfolders is equivalent in restores. 

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
2023-03-23 17:10:05 +05:30
Keepers
532c5c017b
update backup cli output (#2903)
changes a couple things in the backup cli output:
* appends the backup id after completion
* prints an error if the backup fails
* replaces multierr with a clues error
* logs each backup failure individually

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x] 💪 Manual
2023-03-22 16:01:02 +00:00
Georgi Matev
6c989dbd78
Adding new OneDrive and Sharepoint CI cleanup (#2870)
New script for OneDrive and Sharepoint cleanup 

Commit by commit review is encouraged for readability since this involves renames of older scripts.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)


#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E

Tested this with `nektos/act `so action logic should be mostly sound. There is one issue with the script installed for exchange that I believe is just a difference the container image used by GHA and the local I could use for `act`
2023-03-20 20:36:27 +00:00
Georgi Matev
df1ddb94f3
Improved CI cleanup script (#2851)
Improved this significantly based on some learnings from trying to get this in better shape
* Reduce folder enumerations significantly
* Address issue when cleanup gets backlogged and cannot enumerate all items to clean in a single job
* Remove the need to launch one job per prefix or folder name
* The only way to clean /Audits reliably seems to be emptying (with hard delete) of /Revoverable Items (parent of Audits)
* Eliminated need to control concurrency and overhead of job spin up
* Attempt to set explicit network timeout for SOAP calls to prevent long hangs being observed

NOTE: This change affect both cleanup script and the corresponding GHA workflow
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-18 02:13:45 +00:00
Keepers
968f51616f
add outputs to script (#2834)
Some quick QoL
2023-03-17 05:07:01 +00:00
Keepers
8dfb00f308
add fault errors streamstore (#2731)
Adds a new streamstore controller for fault.Errors. This provides large scale, extensible file storage for fault errors to be persisted, much like we do for backup details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x] 💚 E2E
2023-03-17 01:07:07 +00:00
Georgi Matev
41f0b86073
Script to set retention for all mailboxes in a tenant (#2798)
The script requires authenticating with Exchange admin user name and password and will set retention to 0 for all tenant mailboxes. This helps with CI cleanup of test data

The script can be integrated with some low frequency invocation with the CI pipeline to make sure the retention is properly set. Would require setting the credentials as Git secrets

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)


#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-15 16:07:39 +00:00
Keepers
dc21f4ce07
add a QoL path.Build func (#2763)
Adds a path.Build() func that can arbitrarily build a path according to expected standards without first initializing a path.Builder{}.
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-13 16:56:31 +00:00
Keepers
a83f16c6e4
add logger CtxErr() qol func (#2599)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-11 01:41:13 +00:00
neha_gupta
11677fdefb
Release Sanity test (#2622)
## Description

Github workflow - Release sanity test which can run pre and post
release.
Testing- 
- version command
- repo init and connect
- backup and restore of exchange and onedrive
-  verify correct no. of emails restored
- verify file sizes restored and roles on folder are correct.
- incremental backup and verification of the same

**Further enhancement-**
- check actual email data instead of just no. of emails
- verify contacts and calendar
- permission in onedrive
- slack notification on failure 
- if we recieve an email in between a backup and restore, we will fail
the test as the counts won't match

**issues for handling enhancement-** 
https://github.com/alcionai/corso/issues/2743
https://github.com/alcionai/corso/issues/2742

## Does this PR need a docs update or release note?
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🤖 Test

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/2622/

## Test Plan
- [ ] 💪 Manual
2023-03-10 17:46:40 +05:30
ashmrtn
aef90f8b86
Remove GraphConnector user population (#2582)
## Description

Don't populate users when GraphConnector instance is created

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1994

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-28 21:26:12 +00:00
Keepers
9e783efe3a
fault package funcs rename (#2583)
## Description

Renaming the funcs in the fault
package to be more clear about
their purpose and behavior.  Largely
just find&replace changes, except
for fault.go and the fault examples.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-25 03:29:02 +00:00
Keepers
8ee8d265b9
add clues & fault to onedrive collections (#2512)
## Does this PR need a docs update or release note?

- [ ]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-24 22:18:53 +00:00
Keepers
cb12329d0c
don't handle meta files in details (#2606)
## Description

Remove the display and interaction with .meta
files from details entries.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x]  Unit test
2023-02-23 04:14:17 +00:00
Keepers
26d286138b
add fault & clues to exchange api (#2492)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-15 17:01:04 +00:00
Keepers
570ce85656
add clues/fault to exchange restore (#2491)
## Does this PR need a docs update or release note?

- [ ]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-14 19:33:07 +00:00
Keepers
52455356e5
add locationRef to details (#2427)
## Description

Adds a new reference to the details ent: location-
ref.  The location holds the human-readable
version of the item's location in whatever m365
service sourced the item.  Hookup is incomplete,
following PRs will fill out functionality.

Also adds a LocationPather interface to data_
collections to pass this data back and forth
between producers and consumers.

Should be safe to merge into main.

## Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

## Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix

## Issue(s)

* #2423

## Test Plan

- [x]  Unit test
2023-02-13 20:19:04 +00:00
Keepers
dad6776861
replace multierror with fault in setSites (#2377)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-09 18:11:59 +00:00
ashmrtn
129d6b0b0c
Add Fetch() to RestoreCollection (#2434)
## Description

Add a function to fetch a file from the collection synchronously. This will help avoid data dependencies on the restore path created by splitting item information across multiple kopia files

Fetch function is currently unoptimized, though deeper analysis of memory footprint should be done before changing

Viewing by commit will help reduce chaff from updating tests to comply with the new interface

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-08 22:05:38 +00:00
ashmrtn
373f0458a7
Split collection interface (#2415)
## Description

Split the collection interface into stuff used during backup and stuff used during restore. Does not change other code beyond fixing types

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1944

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-02-07 22:15:48 +00:00
Abin Simon
9da0a7878b
Backup versioning (#2324)
## Description

Add backup format version information to the backups so that we can distinguish between backups which use a single file vs the ones that use both .data and .meta files.

Overrides https://github.com/alcionai/corso/pull/2297. I've also set it against `main` so that the diff shows up properly.
Ref: https://github.com/alcionai/corso/pull/2324#issuecomment-1409709118

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2230
* https://github.com/alcionai/corso/issues/2253

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual (Tested manually, will add an e2e test in a followup)
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-03 04:03:14 +00:00
Abin Simon
b8bc85deba
Metadata backup for OneDrive (#2148)
## Description

This PR adds option to backup and restore additional metadata(currently permissions) for OneDrive.

**This PR also adds the ability to restore empty folders for OneDrive along with their permissions.**

Breaking change: Any old backups will not work as we expect both `.data` and `.meta`/`.dirmeta` files to be available for OneDrive backups.

## Does this PR need a docs update or release note?

*Added changelog, docs pending.*

- [x]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1774

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-02-03 03:25:51 +00:00
ashmrtn
57accfc9c4
Use mocks to test getting drive info from Graph API (#2306)
## Description

Make a separate interface for fetching drive information from Graph API. Interface allows for better testing via mocks

Merges SharePoint and OneDrive code for getting drives.

Also fixes potential bug where not all drives would be fetched. This could have occurred because the previous implementation for both SharePoint and OneDrive were not checking for paginated results

Viewing by commit is recommended

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2264

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-27 22:48:30 +00:00
Keepers
24911a358b
quick graph client refactor (#2301)
## Description

A couple minor refactors.  First, combines the
graph/service_helper.go and graph/service.go
files into a single file (just lift'n'shift, no logic
changes).  Second, renames CreateHTTPClient
to just HTTPClient.  Third, replaces the new
LargeItemClient constructor with extensible
options for the http client, starting with a
NoTimeout option.  This will provide longer
term maintenance over multiple constructors.

This change sets up QoL for building multiple services
and clients within api layer handlers, which will appear
in an immediately following PR.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2299

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-27 19:34:00 +00:00
Vaibhav Kamra
bdb7f2b109
Re-use client for OneDrive item downloads (#2276)
## Description

This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client. 

Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success

Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-01-26 15:05:12 +00:00
Vaibhav Kamra
e9354a8429
Revert "add itemClient for re-usable od item downloads" (#2273)
Reverts alcionai/corso#2266. Am seeing issues on data restore that need to be debugged.
2023-01-26 07:52:56 +00:00
Keepers
bda8a5c60c
add itemClient for re-usable od item downloads (#2266)
## Description

onedrive currently constructs a new http client
for every file it downloads.  This causes the OS
to generate extra sockets, and hang onto them
after the download is complete.  Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* closes #2262

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-26 04:27:52 +00:00
ashmrtn
bb73d142c5
Remove duplicate string->path.CategoryType helper function (#2260)
## Description

Tackles a few things:
* normalize case when going from string->path.Service or path.Category
* removes duplicate helper function for string->path.Category in favor of path package one

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2259

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-25 20:13:50 +00:00
Keepers
1ef300d6c2
add item fetching and serialization to api (#2150)
## Description

Adds the per item collection streaming calls to
the api interface.  Primarily migrates a "getItem"
and a "serializeItem" acton into the api pkg, out
from exchange_data_collection.  Building an
ExchangeInfo is now also housed in api.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1996

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-17 20:27:37 +00:00
ashmrtn
6967f1bf6e
Fix compile error in factory script (#2092)
## Description

Go does not allow importing code from a file marked as package main into another file.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2026 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-11 01:04:41 +00:00
Keepers
978f9304bf
methodize all api funcs to the Client (#2008)
## Description

In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces.  This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.

No logic changes have occurred here.  The only
modifications are what is required to utilize the
api client.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 17:19:44 +00:00
Keepers
e5edbfd77c
first pass migrate graph api to subfolder (#2003)
## Description

In order to establish a standard api around our
graph client usage, and thus be able to mock
for testing, we need to migrate graph client
usage into another pacakge.  This is the first
step in that process of refactoring.

All changes are code relocation importing and
exporting may change as needed.  No logic
was altered.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 01:35:32 +00:00
Keepers
23a3cd6e16
remove optionIdentifier (#2001)
## Description

the exchange optionIdentifier performs almost
all the same functions as the path category.  In
many cases, it is actually a translation of the
other value.  This cleanup will cascade into
the creation of an api layer for the graph client.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-02 21:18:52 +00:00
Keepers
752ff20c6b
require resource owner set on selector create (#1887)
## Description

selector creation now includes a parameter for
a slice of resource owners (users or sites).  This
is step one in migrating resource owner lists out
of scopes and into the selector.  next step is to
have the selector utilize the primary list instead
of the per-scope list.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-21 18:38:41 +00:00
Niraj Tolia
da869aeba7
Fix CLI auto-generated docs headers (#1845)
## Description

Fixes warnings the H1 headers are missing

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix
- [x] 🗺️ Documentation
2022-12-16 21:10:05 +00:00
Niraj Tolia
692b8425b9
Website fixes (#1834)
## Description

Fix issues found by screaming frog

- Use https:// for links
- Add missing cookies page back
- Add missing alt-text
- Use dashes instead of underscores for better search engine indexing
- Fix internal redirects

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix
2022-12-16 04:10:56 +00:00
Keepers
8a29c52cdc
move graphService to graph as Service (#1790)
## Description

Relocates the graphService struct to graph as
the Service struct.  Replaces GC's embedded
graphService with a graph.Servicer reference.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-14 01:39:00 +00:00
Keepers
e2775aeb95
Refactor service failfast (#1789)
## Description

Configuration and attenion to the graphService
failFast is haphazard and has shared ownership.
This change removes that property from the
service, along with the ErrPolicy func, in favor of passing around a control.Options struct.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
* #302

## Test Plan

- [x]  Unit test
2022-12-13 23:06:27 +00:00