## Description
Make sure the options struct is passed through the entire stack so we actually have access to flag values.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1931
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Folder population now done when merging items in BackupOp. Also add more tests to make sure folders are actually populated properly in Details
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1800
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Factor out code to get a set of FolderEntries based on some path information. This code will be used in BackupOp when merging item details.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1800
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
- Adds accessor methods for size/modified
- Do not initialize folder in test with a modified time
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add helper functions that allows updating ItemInfo given a new path.Path. This is required because some items embed portions of the path in their details.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1800
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Later code in details package will need access to this function so it can update item details. However, importing the onedrive package in the details package leads to an import cycle. Moving it to the path package breaks the cycle while still allowing the code to be accessed.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1800
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Allows the usage of incrementals via feature-
flag controls. Feature flags can be enabled
on a per-flag basis in either the cli (the flag
is hidden from users, normally) or through the
sdk by directly toggling the flag property
in the control.Options.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] 💪 Manual
## Description
Caches folder info added during details construction in the details builder and keeps the size/modified time
updated as newer items are added.
As part of this, this PR refactors the details package to separate out building `details.Details` from the in-memory representation and model.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
replaces the common/maps.go code with the
golang experimental maps package. The
maps package provides standard api
such as Keys, Values, Copy, and Clone.
https://pkg.go.dev/golang.org/x/exp/maps
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
## Description
Adds logging for non-2xx http communication
within the graph API client. Logging req/resp
metadata is turned off by default, and requires
either a cli flag or os env var to enable.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1858
## Test Plan
- [x] 💚 E2E
## Description
Update the `FolderEntry` type with placeholder fields for folder size and modified time values.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Pick what data to source from a base snapshot by examining the reason the snapshot was selected as a base. This helps avoid two issues:
* pulling in unwanted data when the base has a superset of what is being backed up. Example: pulling in contacts from the base when only email is being backed up
* clobbering already selected data from a different base with data in the base currently being examined. Example: two snapshots, one with contacts and emails and the other with just emails. Second snapshot is newer than the first. The email items in the first snapshot should not clobber those in the first when building the hierarchy
This PR also has the effect of, under some conditions, reducing the amount of data that is pulled from the remote kopia repo when building the hierarchy. This occurs because only the subtrees that will be used in the new backup are traversed instead of traversing the entire snapshot
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Initial changes to support SharePoint Lists being chosen for Backup Operations.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* close #1785<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Adds a flag in backup details that indicates whether the item in that backup is new/updated.
Currently always set to `true` but once we implement #1800 - we will set this to false for existing items from
base snapshots.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Relocates the graphService struct to graph as
the Service struct. Replaces GC's embedded
graphService with a graph.Servicer reference.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Extends the columns printed on sharePoint details
to include the expected data. Fixes some bugs with
flag processing for details and restore commands.
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #1616
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
When backing up exchange data, parse the
metadata collection of delta urls from prior runs
(if any exist) and pass those tokens along to the
fetch functions for re-use.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
## Description
generates a unique repository ID on repo init, and retrieves it again on repo connect. This allows
the event bus (and any other tracker) to id a repo independent of other context.
I've been unable to find any good way to retrieve a unique repository ID from kopia. This seems like
the next quickest solution.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1602
## Test Plan
- [x] 💪 Manual
## Description
This PR uses a separate snapshot to store backup details instead of a Corso model (i.e. a kopia manifest).
Introduces a `StreamStore` that can be leveraged to store larger metadata objects. We can also leverage this
for incrementals or restartable backups going forward.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1735
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a PathCategories() func to selectors which returns
all the path categories exhibited by the scopes within the
selector.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
## Description
Adds a func in graphConnector that reduces
siteIDs and webURLs into a set of siteIDs. This
will be used by callers such as the CLI to
generate id-based selectors for sites even if
they handle webURLs as an alternative id.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Missed configuring working dir for staticcheck, plus fixes for errors reported.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Corso needs to store some transient information between backups so that it can perform more efficient incremental backups. This transient information will live at a path `tenant-id/service/user-id/categoryMetadata`. This patch adds an initializer, category enums, and tests for these new path locations
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds restore commands to the cli for sharepoint.
The restore process is only partially functional at this time.
Library files that pass auth are able to be restored as expected.
However, auth issues (not directly related to these changes) prevent
restoration of all library items at this time.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1615
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Move user discovery into a `discovery` package. Also use that to return user display name from `m365.Users()` call.
This will also allow us to remove `Users` from the GC struct going forward.
If this pattern works - will follow up with a similar change for `Sites()`.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
the new pathFilterFactory now properly cleans
the inputs and checks for all-pass and all-fail
conditions.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Adds webUrl scopes to the sharepoint selector.
Also introduces the idea of a scope categories
that can broadly match across all leaf types
within a service. This category union should be
reserved for root categories, and properties
that can be used interchangably with the root.
This is part 2 of exposing webURLs as an alternative
to siteIDs for sharepoint backup and restore. The
next change providing a webURL => siteID lookup
within the graph package.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Add size and stored size to cli output for `backup list`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1622
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Preparing for handling sharepoint webUrl
selectors, we will need suffix-matching filters.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Sketch of how to split backups by (resource owner, service) for exchange. Needs more support from selectors package to be complete.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1505
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use latest golangci-lint version. Also updates cache flags.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1629
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Helper function to fetch backup models by ID. Also renames `Backups` to `BackupsByTag`
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1505
separated from
* #1609
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the `backup details sharepoint` command
to the CLI. E2E support is already functional.
Additional filters will be added as needs are
identified.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1614
## Test Plan
- [x] ⚡ Unit test
## Description
Add a Set(ctx, logger) func to the logger package
to allow SDK users to provide their own zap
logger instance to corso.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1630
## Description
Adds a method to all scopes that produces a
discrete clone of the original scope. The result is a shallow clone of the scope, thus retaining all
original values, but is made discrete by replacing the resource owner target with a specific value.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1621
## Test Plan
- [x] ⚡ Unit test
## Description
Backup of SharePoint requires the extension of `path.CategoryType` to be able to support the creation of `data.Collections` for `SharePoint.Lists`.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Test Plan
- [x] ⚡ Unit test
## Description
Adds a LibraryCategory in paths, and changes
the current sharepoint code to use that cat
instead of the onedrive files cat. This is mostly to keep sharepoint library drives separate from
onedrive and sharepoint site list items.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] ⚡ Unit test
## Description
Adapts the graph onedrive library to handle
access to drive data across both onedrive and
sharepoint services.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This will now show elapsed time along with the spinner.
```
Connecting to repository: 4s done
Connecting to M365: 0s done
Discovering items to backup: 2s done
Discovered 1013 items to backup
Backing up data:4m37s done
Started At ID Status Selectors
2022-11-17T11:22:35Z cdbbf625-f925-4f59-9c42-12756108b752 Completed (0 errors) J...L@redacted.onmicrosoft.com
```
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a lookup for all tenant site ids when a new
graph connector is created. Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💚 E2E
## Description
Make the return type of looking up backup models more consistent by always returning pointers to items. This helps avoid some type casting that could occur in some situations and makes the function definition for print more consistent because they will always work with a pointer type
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1512
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Want to use the `split` function through the path package to split arbitrary escaped strings.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X] ⚡ Unit test
- [ ] 💚 E2E
Existing unit tests pass.
## Description
Removes the individual user load test set to
focus on backup-restore of the standard org.
Fixes the folder-creation time value provided to
the purge script from load tests.
## Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Add size to all objects. Not adding them to the output of details view.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
