Make sure dependencies are properly installed before building
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds function call to the path package that returns the folder elements present in a path after the path prefix. This is safer than using `strings.Split()` because some paths may have `/` characters in elements.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#909
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Disable the kopia retention policy that it uses by default. All snapshot deletions in corso must be done manually.
There should really be a follow up change to this after making a method in kopia public. Right now this PR is going off [some code](7bda16ab33/snapshot/policy/retention_policy.go (L117)) that shows the effective number of "latest" snapshots will default to intMAX if all of the other fields are set. We really should have a regression test or something for this to ensure that's actually the case. I've manually tested this patch, but that's not a scalable solution
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#889
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Fix responsive ordering of key features
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Over-restrictive scope correlation caused the
reduce processor to only include filters which
matched the data type of the scope. Ironically,
this allowed a superset of information to match,
by evading the _all-match_ expectations of
filter scopes.
Also replaces the data category consts inside /details
with the path category types, since those are acting as
better canonical owners of data type identification
throughout the app.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #890
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Add messaging pillar information on the website
Bullet points will be added next.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Test Plan
- [x] 💪 Manual
## Description
Implements `data.Collection` restore for OneDrive and wires it up to the restore operation
Leverages previously added `onedrive` helpers that have integration tests.
An op/CLI level integration test will be added in a follow-up PR.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
```
$ ./corso restore onedrive --backup 7de4d68f-67c2-4e81-95a2-81cf0d1bd11d
Restored OneDrive in S3 for user
```
## Description
In order for purge to gracefully handle the variety of timestring values that may appear within a
folder name, the common/time.go file has added
a func to extract a time substring from a string.
Purge is switched to that extractor instead of
a len-based suffix check.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #805
## Test Plan
- [x] 💪 Manual
## Description
1/ introduce an ID() method int the account
package which provides the tenantID of the
configured account provider.
2/ introduce a event bus mock for testing.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
Allow users to provide bucket names prefixed
with the s3 scheme: s3://my-bucket
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #560
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
PR enables Filtering for `exchange.Contact` items wherein if an `exchange.ContactFolder` is requested specifically, only the requested folders are backed up.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #811<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Encode all paths given to kopia with base64URL to ensure no special characters that kopia can't handle end up in there. The encoded paths are not stored in backup details nor are they ever surfaced to the user. This also works around the previous limitation where Corso was unable to properly backup or restore exchange email layouts that had folders containing `/` characters
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#865
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Small refactor. Moves Path functions from iterator to `service_functions.go`
Introduces getCollectionPath which combines the two previous functions and reduces code complexity
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
The wrapper.go file was not properly grouping items during restore, instead returning a bunch of single item data.Collections. Update this to properly group items by folder. Also update the tests to be more strict about checking for this.
Remove dead code around restoring a single item as it was unused. Lose some visibility into what happens if the code is unable to open the reader for a given item.
Generally moves some logic around in wrapper.go
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#879
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
No change to logic in other code, just refactoring the tests
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Wires up the OneDrive Backup(create,list,details,delete) and Restore commands
Unit tests added but integration tests will be added after the underlying operation
PRs are merged.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#658#668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Allows kopia data collection consumers to query stream size.
This is required for OneDrive that needs to set the file size prior to uploading file data.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Other components may need to rebuild the directory hierarchy of items. As the paths Corso deals with can be hard to properly parse at times, store that information in the Corso backup details. The hierarchy can be rebuilt by following the `ParentRef` fields of items. The item at the root of the hierarchy has an empty `ParentRef` field.
Also hide these folders from end-users. They are not displayed during backup list nor are they eligible as a target for restore
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#862
* closes#861
* closes#818
merge after:
* #869
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This implements helper methods that support creating new items and uploading data from a `io.Reader` to OneDrive.
It includes an integration test that uses mock data.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Removed timestamp check which could fail if the test started right before the hour rolled over.
Couldn't think of a better check that wouldn't fail at *some point*
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Description
<!-- Insert PR description-->
Folder entries allow recording the directory hierarchy of a backup in backup details for other consumers.
Although folders are present in the backup details and they have the logic to be displayed, they are not expected to be shown to users (later PRs will ensure this).
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#860
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the ability to retrieve M365 IDs of contacts not in the default folder.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#804
* closes #767<issue>
* closes#669
## Test Plan
- [x] ⚡ Unit test
## Description
Fixup corner case on ShortRef so that empty paths generate the empty string. This makes some cases with generating short refs for each parent path to an item easier because the base case (empty path) is no different than other cases.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* part of #818
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use $HOME as the config default.
Allow CORSO_CONFIG_DIR env to
override the default. Allow the flag
--config-file to override either.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #718
## Test Plan
- [x] 💚 E2E
## Description
Graph clips time values ending in :00 from folder
names. In order for the purge handler to check
the time values correctly, we need to identify
those folders and reinstate the 00 seconds tail.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #805
## Test Plan
- [x] 💪 Manual
## Description
Adds events for repo init, backup start and end,
and restore start and end. Not all expected
values are available. TODOs have been marked
for values that need future implementation.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
The details entry shorthash can be treated as
equal to the leaf item ID. This adds support to
the selector reduce step to compare the leaf val
to either the path item ID or the shortHash.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* closes#572
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
An attempt to minimize and centralize how we're caching go builds.
Not necessarily going to solve all our CI issues, but should make
changes building on this state much easier.
## Type of change
- [x] 💻 CI/Deployment
## Issue(s)
* #790
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Updating logging to help display/control within testing envs.
Restore error tracking within gc iterators.
## Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Test
## Issue(s)
* #824
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Short refs allow the user to type less to select an item to restore. This PR
* Adds a short ref field to the BackupDetails entries
* Populates the short ref field during backup
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #572
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Places `exchange.Restore` logic into single file
<!-- Insert PR description-->
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Stops the generation of Inbox messages when events are created
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #855<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
Verify that Inbox doesn't grow with undeliverable messages.
## Description
Swaps the default corso config location from
$HOME to /app/corso in order to align with docs.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #718
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
The current action workflow skips cause requried
actions to be marked as passing until they fail,
not marked as incomplete until either passed,
failed, or skipped. This change should provide
the latter behavior.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #744
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Currently using 12 hex characters of a sha256 sum. This is expected to
start giving collisions after about 16.7M paths hashed.
Short refs are stable in the sense that calling ShortRef multiple times
on the same path will yield the same result. They can be used to
identify unique paths if desired, but extra lookup functionality will be
needed because short refs cannot be directly transformed back into a
path.
* Make data.Collection.FullPath return path.Path
* Fixup graph connector code for path struct
* Add Elements call to Path interface
Still should not be used in place of the named functions to get specific
path elements (e.x. ResourceOwner())
* Fixup kopia wrapper path handling
Mostly removing shim code and minor fixup for building the directory
tree.
* All the test fixes
* Helper function to append elements to a path
Kopia wrapper will need to create the complete path of an item by
joining the collection path and the item name. This allows it to do so
without having to drop to a path Builder (not service/category safe) and
go back to a resource path. Right now it does not handle escaping.
* Use path struct while streaming entries
Use new path struct while streaming entries to kopia. Preparation for
FullPath returning a path struct.
* Update tests to use valid path structures
## Description
Small refactor of the data collection restore helper so that we can plug-in the OneDrive helpers in a
follow-up PR
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
* Constants for OneDrive stuff
* Tests and constructor for OneDrive paths
* Populate onedrive path struct in data collection (#835)
* Helper function to make path structs for onedrive
* Use path struct in onedrive data collection
Does not change the external API at all, just the internals of how
FullPath functions and what is stored for the path.
* Wire up making data collections with path struct
Requires addition of tenant as input to Collections().
* Fixup onedrive Collections tests
* Wire up call to onedrive.NewCollections()
Just requires adding the tenant ID to the call.
* Use path struct in kopia DataCollection
Does not change the external API of the DataCollection any, just updates
internals in preparation for switching support of
data.Collection.FullPath.
* Expand Path interface slightly
kopia.Wrapper needs some extra functionality from paths, mostly along
the lines of directly manipulating the elements in the path. This gives
access to those functions.
* Use path struct in kopia.Wrapper for restore
Pass path structs to the newly created collections during restore.
* Add tests for new path functionality
