## Description
Enables feature SharePoint to use "wildcard" to select all available sites with the ability to only backup selected categories.
Example
```bash
./corso backup create sharepoint --web-url "*" --data libraries
```
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2579<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
Create a `repo connect` event on CLI's repo connect command
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
https://github.com/alcionai/corso/issues/1762
## Test Plan
- [x] 💪 Manual
## Description
Due to recent changes with `Incrementals` restore of Library categories stopped working. This PR will allow for files that do not require special folder permissions to be restored. Changes involve the folder naming format that was coming from the CLI
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2489<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
Adds a new reference to the details ent: location-
ref. The location holds the human-readable
version of the item's location in whatever m365
service sourced the item. Hookup is incomplete,
following PRs will fill out functionality.
Also adds a LocationPather interface to data_
collections to pass this data back and forth
between producers and consumers.
Should be safe to merge into main.
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #2423
## Test Plan
- [x] ⚡ Unit test
## Description
Changes connect SharePoint Pages to restore CLI.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No Arguments are hidden
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2318
* closes#2070
* closes #2168<issue>
* closes#2218
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Create a wrapper struct that provides a `Fetch(ctx, name) (Stream, error)` function that always returns `ErrNotFound`. A future PR is going to expand the `RestoreCollection` interface to include that function and I wanted to reduce the amount of chaff that would come out of it
This PR just creates the wrapper and moves `ErrNotFound` from the kopia package to `data` package to avoid import cycles
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1944
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
adds fault.Errors to backupDetails() and backups().
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds clues and fault handling to selectors pkg.
Some bleed upward into the CLI occured from
where the cli directly calls selectors.Reduce.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Since the backup permissions increases the backup time by a lot(mostly from higher number of requests and increased throttling), it was decided to disable it by default. It is still enabled in tests so as to make sure the code does not have regressions.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
This PR adds option to backup and restore additional metadata(currently permissions) for OneDrive.
**This PR also adds the ability to restore empty folders for OneDrive along with their permissions.**
Breaking change: Any old backups will not work as we expect both `.data` and `.meta`/`.dirmeta` files to be available for OneDrive backups.
## Does this PR need a docs update or release note?
*Added changelog, docs pending.*
- [x] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1774
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Prevents showing the log file location when calling commands that defer to help output or env details.
Certain cases aren't caught, such as when calling a command with no flags (ex: `corso backup create exchange`).
In those cases we catch the lack of flags and manually display the usage within the command handler.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2329
## Test Plan
- [x] 💪 Manual
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2329
## Test Plan
- [x] 💪 Manual
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2329
## Test Plan
- [x] 💪 Manual
## Description
A basic mechanism to show a warning for the help output of pre-release commands
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Sharepoint support is still under active development and not ready for use. Hiding the command from
usage details.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
A couple minor refactors. First, combines the
graph/service_helper.go and graph/service.go
files into a single file (just lift'n'shift, no logic
changes). Second, renames CreateHTTPClient
to just HTTPClient. Third, replaces the new
LargeItemClient constructor with extensible
options for the http client, starting with a
NoTimeout option. This will provide longer
term maintenance over multiple constructors.
This change sets up QoL for building multiple services
and clients within api layer handlers, which will appear
in an immediately following PR.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2299
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client.
Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success
Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Don't split the Corso version information across lines. This is unexpected.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Description
onedrive currently constructs a new http client
for every file it downloads. This causes the OS
to generate extra sockets, and hang onto them
after the download is complete. Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* closes#2262
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds the functionality to select SharePoint Pages for backup.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
SharePoint commands remain hidden in terms of pages.
- [x] 🕐 Yes, but in a later PR
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2216 <issue>
* closes#2107
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
## Description
Removes references to M365 ID as instruction.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No, self-documenting
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#2190
## Test Plan
- [x] ⚡ Unit test
## Description
Expands SharePoint Selectors to include `SharePoint.Pages` within the Corso package.<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
Prerequisites
-----
- [ ] Merge #2114
- [x] Resolve#2112
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2110<issue>
* related #2107
## Test Plan
Testing will
- [x] ⚡ Unit test
## Description
This changes the behavior of logs and writes it to disk instead of `stdout` by default. It also adds a new flag `--log-file` to specify the filename for the log file.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2076
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Ensures that the details and restore commands
exit without attempting further processing if a
backupID doesn't point to either a valid backup
or backup details, and that the user is appropriately notified.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1877
## Test Plan
- [x] 💚 E2E
## Description
Previously the help output looked like:
```
--config-file string config file location (default is $HOME/.corso.toml) (default "$HOME/.corso.toml")
```
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
The selectors printable code wasn't being used in any valid way, so that's out. Backup List printing now refers to the DiscreteOwner in the embedded selector as the ResourceOwner reference. Renamed the "Selectors"
column in the List print table to "Resource Owner" to better match what should be contained in that field.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1617
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Sets the default run behavior for exchange to use
incremental backups. The cli feature flag for enabling exchange incrementals has been swapped for a toggle that disables incrementals, forcing a full backup
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
DataCollections validation step was still using the full resourceOwner list in the selector to validate every backup, rather than checking only the DiscreteOwner.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
## Description
Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Switches the CLI from calling `DiscreteScopes` to `SplitByResourceOwner`
on the selector itself. This func will take the original selector and produce
a slice of selectors, each one with a DiscreteOwner (the single user involved
in usage of that selector) and all include/filter scopes in that selector re-rooted
to that discrete owner.
Does not yet solve the per-category tuple, since we are still pivoting on the
scopes inside the selector. That comes as a later change.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
CLI commands were added to facilitate restore of `SharePoint.Lists` to M365
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #1963<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Make sure the options struct is passed through the entire stack so we actually have access to flag values.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1931
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
On backup create, sharepoint selectors also
need scopes, or else they do nothing.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1908
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Allows the usage of incrementals via feature-
flag controls. Feature flags can be enabled
on a per-flag basis in either the cli (the flag
is hidden from users, normally) or through the
sdk by directly toggling the flag property
in the control.Options.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] 💪 Manual
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Reverts alcionai/corso#1835 now that #1841 is in. That should help with the flakiness.
I will follow up with another PR after though that moves these tests to a post-CI job (we don't need to run these on every commit)
## Description
Initial changes to support SharePoint Lists being chosen for Backup Operations.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* close #1785<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
These tests have the highes rate of timeout
based failures, while providing the most loose
gurantees about system behavior. This will
temporarily disable them while we consider
separating our suite between integration
and e2e smoke sets.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Test Plan
- [x] 💚 E2E
## Description
Fixes two bugs: 1. building the correc tag key
when retrieving the prior backup id from a
manifest. 2. returning ErrNotFound when a
snapshot entry isn't found for restore. This
can occur for metadata if the service has
a prior backup that does not contain the
expected metadata files.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1823
## Test Plan
- [x] 💪 Manual
## Description
Extends the columns printed on sharePoint details
to include the expected data. Fixes some bugs with
flag processing for details and restore commands.
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #1616
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
In a backup operation, begins the operation by
retrieving all backup manifests and metadata
from prior operations.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds weburl identificatiotn and unioning in the
cli, for both backup and restore of sharepoint
data.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
