## Description
Begin populating a global exclude set with files deleted from OneDrive. The set contains the IDs of files that have been deleted.
This is actually safe to return from GraphConnector now (though it's not returned) as it uses item IDs instead of item names.
Future PRs will need to address handling of (potentially) moved files
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2242
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Track folder moves and update the persisted metadata accordingly. Moves need to take into account subtree changes because OneDrive will not notify us of subfolders moving if nothing else was updated on the subfolder.
Handle updates by making a copy of the folder map for ease of updating and later on ease of finding out which collection paths have changed
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2120
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
SharePointInfo extended to include SitePageable objects.
**NOTE:** Merge includes the beta graph package and will affect the compile time.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2069<issue>
* related #2071
## Test Plan
- [x] ⚡ Unit test
## Description
Fix to allow selectors to include `SharePoint.Pages` during backup without causing backup workflow to fail.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #2113<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
Branch treats Drive `Site Pages` as a restricted directory. All `pages` within the directory, will be backed up via the Site Pages API. This adds an additional call to per collection to obtain the parent `driveName`.
<!-- Insert PR description-->
TODO: Create a Pipeline that distinguishes between SitePages, Libraries, and List for SharePoint
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
The documentation should state the assumption is that only `.aspx` files are to be found in the s
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2072<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Start persisting the folder path maps and delta URLs for backed up OneDrive/SharePoint drives. Delta URLs are saved in a map[drive ID]deltaURL while folder IDs are in a map[driveID]map[folder ID]folder path
Needs another patch to properly save the path for folders that match the selector, currently the selector comparison is only on the parent of an item
Later PRs can get the new folder map by taking the map from the previous backup and making changes to it when folder deletions/moves are encountered
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2120
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
add logging to the observe package, assume that
every instance where a message is observed, it
also gets logged.
Merger may want to wait until logging to a file is the standard behavior, else the terminal might get messy/buggy.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2061
## Test Plan
- [x] 💪 Manual
## Description
Refactoring and new code additions have led to some unit tests that fail unless testing is run with the proper env vars. Fixup code to either skip those tests or provide them with the proper information so they pass
* some tests moved from unit tests to integration tests
* function to get credentials with bogus info for unit tests
* skip some tests if env vars not passed
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2052
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Connects restore pipeline for List Category. CLI commands are not included, Documentation for List restore needs to be placed within the CLI PR
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* closes #1962<issue>
* closes#1978
* closes#1935
## Test Plan
- [x] ⚡ Unit test
## Description
Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds the ability to create non-empty lists within mocks
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes to #2018<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Switches the CLI from calling `DiscreteScopes` to `SplitByResourceOwner`
on the selector itself. This func will take the original selector and produce
a slice of selectors, each one with a DiscreteOwner (the single user involved
in usage of that selector) and all include/filter scopes in that selector re-rooted
to that discrete owner.
Does not yet solve the per-category tuple, since we are still pivoting on the
scopes inside the selector. That comes as a later change.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds the ability to delete a from a SharePoint Site for purge script
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*related to #2016 <issue>
## Description
Restore function for `SharePoint.List` logic within PR. Mock data changed as `SetName` is not allowed on item creation. Format for list restore is verified as below:
```java
TestSharePointCollectionSuite/TestRestoreList
collection_test.go:126: List created: Corso_Restore_26-Dec-2022_21-53-01.459183_MockListing
```
The expectation is that the correct folder name is passed to the function from the CLI. The restore pipeline is not connected for `SharePoint.List`.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Related to #1935 #<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Restore for `Libraries` to be housed within the main drive directory for the site drive.
- Change to `onedrive` package to allow for the export of `CreateRestoreFolders()` a utility function for creating a directory structure.
- Temporary test (`TestRestoreLocation()`) was added to ensure proper folder creation for SharePoint Libraries.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
- related to #1938
## Test Plan
- [x] ⚡ Unit test
## Description
Adds a new func to the data.Collection iface:
DoNotMergeItems() tells kopia to skip the
retention of items from prior snapshots when
generating the new snapshot for this collection.
A primary use case for this flag is when a delta
token expires, preventing an incremental lookup
and forcing gc to re-discover all items in the
container.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1914
## Test Plan
- [x] ⚡ Unit test
## Description
Caches folder info added during details construction in the details builder and keeps the size/modified time
updated as newer items are added.
As part of this, this PR refactors the details package to separate out building `details.Details` from the in-memory representation and model.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Framework for the SharePoint backup workflow.
### Special Instructions for Review
Ensure that the information used to build `path.Path` are in line with future PRs (@ashmrtn )
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* closes#1403
* closes #1474<issue>
* closes#1795
## Test Plan
- SharePoint Integration checked:
- `src/internal/connector/data_collections_test.go`
- `TestSharePointDataCollection()`
- SharePoint List basics checked:
- `src/internal/connector/sharepoint/collection_test.go`
- SharePoint Operational Backup:
- `src/internal/operations/backup_test.go`
- `TestBackup_Run_sharePoint() ` verified as operational during testing
- [x] ⚡ Unit test
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#1856 <issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Fulfills the data collection interfaces to provide item deleted
flags and collection state enums.
A note on the additional data consumption: I'm making an assumption
that the @removed property will appear in that map. This is awaiting
testing for verification, which I'll get into as a follow-up step.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1727
## Test Plan
- [x] ❓ In a follow-up PR
## Description
Configuration and attenion to the graphService
failFast is haphazard and has shared ownership.
This change removes that property from the
service, along with the ErrPolicy func, in favor of passing around a control.Options struct.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
* #302
## Test Plan
- [x] ⚡ Unit test
## Description
Addition of SharePoint-specific functions to test package. Verifies that support package functions are present and do not cause errors upon SharePoint Collection Creation
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
## Description
`graph.Service -> graph.Servicer`, no other changes.
More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
## Description
Move helper functions & test structs to `helper_test.go`
<!-- Insert PR description-->
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
## Description
Versioning upgraded for GC
Packages updated:
- [x] github.com/microsoftgraph/msgraph-sdk-go v0.41.0 => v0.48.0
- [x] github.com/microsoft/kiota-abstractions-go v0.14.0
- [x] github.com/microsoft/kiota-authentication-azure-go v0.5.0
- [x] github.com/microsoft/kiota-http-go v0.9.0
- [x] github.com/microsoft/kiota-serialization-json-go v0.7.2
- [x] github.com/microsoftgraph/msgraph-sdk-go-core v0.30.1
- [x] github.com/stretchr/testify v1.8.1
- [x] github.com/cjlapao/common-go v0.0.29 => v0.0.35
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
- Update library change. AdditionalData no longer contains `nextLink` for iteration. See commit [e7c75ca](e7c75ca1cc) for full details.
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1478<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Currently, all drive backup and restore actions
populatet a details.OneDriveInfo struct. This
change branches that struct between one-
drive and sharepoint info, depending on the
current source.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Add functions to Collection and Stream interfaces that allow for getting
more information about the difference between the previous backup and
the currently in-progress one. These will allow delta token-based
incremental backups to determine how the state has evolved.
Current code does not use these functions and return values for them
are "default" values that should result in full backups even if
KopiaWrapper is updated to start checking the values and GraphConnector
still pulls all items
These functions are not used during restore and can return "default"
values
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1700
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add ModTime to Exchange, OneDrive and SharePoint list stream items. This enables kopia-assisted incrementals for those items. Backup details still contains a complete set of information for all items in the backup regardless of if kopia uploaded data for the item or not.
Kopia-assisted incrementals does come with some caveats though. If changes are made to an item in M365 and that change does not cause the modified time reported by M365 to update, then the change will not be backed up. Currently, only marking an email as read/unread is known to hit this edge case.
This patch does not lazily fetch data from Graph API. This means that kopia may upload less data, but the same amount of data will still be pulled from Graph
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#622
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Logic for List retrieval from M365 back store. Keeping separate from Collection creation logic as there are a lot of calls to retrieve all supporting relationship data from the backstore.
Adds collection to
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1475<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Use latest golangci-lint version. Also updates cache flags.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1629
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a server-side filter that removes all
personal (aka: oneDrive) sites from the drive
retrieval.
The approach is not my preferred method. I'd rather
have used a filter in the query parameters. In fact, I
have a functioning query in postman:
`$filter=NOT contains(webUrl, 'sharepoint.com/personal/')`
But I can't seem to get the graph client query to build
when I add that to the query params.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💚 E2E
## Description
Adds a LibraryCategory in paths, and changes
the current sharepoint code to use that cat
instead of the onedrive files cat. This is mostly to keep sharepoint library drives separate from
onedrive and sharepoint site list items.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] ⚡ Unit test
## Description
Adapts the graph onedrive library to handle
access to drive data across both onedrive and
sharepoint services.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a lookup for all tenant site ids when a new
graph connector is created. Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💚 E2E
## Description
Since scopes are service specific, we cannot
easily house them within the graph QueryParam
struct, unless we bloat the struct with all types.
Alternatively, we could add a generic "scope"
with parsers, much like the Selector itself. But
really, the most simple solution is to only pass
the scope within the tree of service funcs that
use it.
## Type of change
- [x] 🐹 Refactor
## Issue(s)
* #1506
## Test Plan
- [x] ⚡ Unit test
## Description
boilerplate to graph connector. To minimize
changes, some dependencies or half-baked
solutions have been listed as TODOs for follow-
up.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Add size to all objects. Not adding them to the output of details view.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds the sharepoint service to /pkg/path. Currently uses the
"Files" category for its category type, which is just a placeholder
for kicking off development.
Additionally, uncomments selector tests that were dependent
upon the path service declaration.
## Description
Creates test package for SharePoint List Collection. Verifies Item and ListInfo for collection
<!-- Insert PR description-->
## Type of change
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #1474<issue>
## Test Plan
- [x] ⚡ Unit testt: E2E
## Description
Creates Sharepoint Info within the details package. SharepointInfo constructor and test package included
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Test Plan
- [x] ⚡ Unit test
