Adds a path.Build() func that can arbitrarily build a path according to expected standards without first initializing a path.Builder{}.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Github workflow - Release sanity test which can run pre and post
release.
Testing-
- version command
- repo init and connect
- backup and restore of exchange and onedrive
- verify correct no. of emails restored
- verify file sizes restored and roles on folder are correct.
- incremental backup and verification of the same
**Further enhancement-**
- check actual email data instead of just no. of emails
- verify contacts and calendar
- permission in onedrive
- slack notification on failure
- if we recieve an email in between a backup and restore, we will fail
the test as the counts won't match
**issues for handling enhancement-**
https://github.com/alcionai/corso/issues/2743https://github.com/alcionai/corso/issues/2742
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🤖 Test
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/2622/
## Test Plan
- [ ] 💪 Manual
## Description
Don't populate users when GraphConnector instance is created
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1994
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Renaming the funcs in the fault
package to be more clear about
their purpose and behavior. Largely
just find&replace changes, except
for fault.go and the fault examples.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Remove the display and interaction with .meta
files from details entries.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🐛 Bugfix
## Test Plan
- [x] ⚡ Unit test
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a new reference to the details ent: location-
ref. The location holds the human-readable
version of the item's location in whatever m365
service sourced the item. Hookup is incomplete,
following PRs will fill out functionality.
Also adds a LocationPather interface to data_
collections to pass this data back and forth
between producers and consumers.
Should be safe to merge into main.
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #2423
## Test Plan
- [x] ⚡ Unit test
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Add a function to fetch a file from the collection synchronously. This will help avoid data dependencies on the restore path created by splitting item information across multiple kopia files
Fetch function is currently unoptimized, though deeper analysis of memory footprint should be done before changing
Viewing by commit will help reduce chaff from updating tests to comply with the new interface
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1535
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Split the collection interface into stuff used during backup and stuff used during restore. Does not change other code beyond fixing types
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#1944
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Add backup format version information to the backups so that we can distinguish between backups which use a single file vs the ones that use both .data and .meta files.
Overrides https://github.com/alcionai/corso/pull/2297. I've also set it against `main` so that the diff shows up properly.
Ref: https://github.com/alcionai/corso/pull/2324#issuecomment-1409709118
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2230
* https://github.com/alcionai/corso/issues/2253
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual (Tested manually, will add an e2e test in a followup)
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
This PR adds option to backup and restore additional metadata(currently permissions) for OneDrive.
**This PR also adds the ability to restore empty folders for OneDrive along with their permissions.**
Breaking change: Any old backups will not work as we expect both `.data` and `.meta`/`.dirmeta` files to be available for OneDrive backups.
## Does this PR need a docs update or release note?
*Added changelog, docs pending.*
- [x] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1774
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
A couple minor refactors. First, combines the
graph/service_helper.go and graph/service.go
files into a single file (just lift'n'shift, no logic
changes). Second, renames CreateHTTPClient
to just HTTPClient. Third, replaces the new
LargeItemClient constructor with extensible
options for the http client, starting with a
NoTimeout option. This will provide longer
term maintenance over multiple constructors.
This change sets up QoL for building multiple services
and clients within api layer handlers, which will appear
in an immediately following PR.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2299
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client.
Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success
Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
onedrive currently constructs a new http client
for every file it downloads. This causes the OS
to generate extra sockets, and hang onto them
after the download is complete. Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* closes#2262
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Go does not allow importing code from a file marked as package main into another file.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2026
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces. This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.
No logic changes have occurred here. The only
modifications are what is required to utilize the
api client.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds restore commands to the cli for sharepoint.
The restore process is only partially functional at this time.
Library files that pass auth are able to be restored as expected.
However, auth issues (not directly related to these changes) prevent
restoration of all library items at this time.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1615
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Adds a lookup for all tenant site ids when a new
graph connector is created. Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💚 E2E
## Description
The secondary scripts in corso (factory, purge, etc) are not currently flushing their logs if an error occurrs since os.Exit(1) kills the current process without alloting time for deferred funcs to run.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Adds feature to be able to create mock events with attachments
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1005 <issue>
## Test Plan
Testing:
```
internal/connector/exchange$ go test -v . --run TestExchangeServiceSuite/TestRestoreExchangeObject/Test_Event_with_Attachment
```
- [x] ⚡ Unit test
## Description
Adds the capacity to programmatically generate events in the factory script. Status is currently functional in code completion, but experiencing unexpected results.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds scripted production of mock emails for building out
large/rolling datasets for load testing.
## Type of change
- [x] 🌻 Feature
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
## Description
Introduces a `factory` script to help devs generate
fake m365 data in bulk for testing purposes. Only
boilerplate at this time. Any TODO's or not-yet-
implemented statuses are present as placeholders
alone, without official deadlines.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
