introduces the Chats service and Chats category
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5061
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config. This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
updaing the path package to the current naming convention. No logic changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Not sure if we wanna merge it as it might generate way too many conflicts, but this should help us add a linter in CI. If we are good, I'll add something that can do lints for this in a follow up PR.
Super hacky, but this fix was created using `while true ; do tree-grepper -q go '(argument_list "," @nope .)' | tail -n1| awk -F: "{print \$1,\"+\"\$2\" -c ':norm \$xJZZ'\"}" | xargs vim ; done`.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3654
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
populate a "test_name" clues field in the
tester context, so that logs can be associated
with the calling test.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
removes the microsoft item ID from the exchange pathValues set when using only-name selector configuration. This is the standard config for the cli, thus this removes the ability to filter exchange items from the cli by using their m365 id.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3313
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.
Only code movement, no logical changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
A few high-level things of note:
* things will no longer match on folder ID. Folder IDs weren't displayed
to the user via CLI and SDK consumers have no insight into folder IDs
so this shouldn't be an issue
* OneDrive and SharePoint match on ParentPath (derived from
LocationRef). ParentPath *does not* include root: in the path
Not matching on folder ID should be the only user-visible change in
this PR
First commit contains the required logic changes. All other changes
are test updates
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#3194
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds an new details entry field: itemRef.
This holds a stable, semi-unique identifier to the item represented by that entry.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3027
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Simple concept for a large footprint of change:
filters should always accept a slice of targets
to compare against, instead of a string. Currently,
we're faking a slice by way of using a const
separator anyway, so we might as well support
slices outright.
Lots of changes (simplification, mostly) in
selectors cascade from this shift. Any updates
you see here are purely to comply with the updated
filters interface.
This step will make concealing PII in selector and filter
logging significantly easier than before.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2024
#### Test Plan
- [x] ⚡ Unit test
Includes fixups for calls that need updated
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [x] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2373
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Renames a lot of the names in scopes that contain
the word "filter" with the word "info". The basis here
is to reduce confusion in the libraries. Filter as a term
is overloaded. It currently means: 1/
the set of scopes that comparisons _must_ pass
to succeed; 2/ the pkg and struct that handles
comparison logic; and 3/ the set of scopes that
compare values in details.ItemInfo instead of in
the item's storage or logical path. The third
category is the primary overload here, and is
better off with a different name.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
In order to retrieve all clues structured error data in tests, we need to extract it from the error using the clues library.
This change appends `clues.ToCore(err)` to all
variations of `assert.NoError(t, err)`. The only
other changes are those necessary to preserve
linting, or to produce an error variable for the
ToCore call.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Change to storing OneDrive files by ID instead of their OneDrive display name. This allows delta token-based incrementals to use an exclude list to remove previously backed up items from a backup during hierarchy merging
Also updates the following:
* selectors to match on file display name instead of kopia file name
* ShortRefs for OneDrive files to update when the file display name is updated
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#1535
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
With onedrive storage file names being changed from the file display name to the file id, we need a more granular form of indentification when using selectors to choose which values count as matchable fields.
This change modifies the selector PathValues to return slices of strings for each category instead of a single string, and the reducer matches on any. This will allow each service to decide what values are considered equivalent (id, shortRef, a value inside the info, etc) for each property.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2373
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Leave out account and storage subpackages as those are used by tester and create an import cycle.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2373
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a new reference to the details ent: location-
ref. The location holds the human-readable
version of the item's location in whatever m365
service sourced the item. Hookup is incomplete,
following PRs will fill out functionality.
Also adds a LocationPather interface to data_
collections to pass this data back and forth
between producers and consumers.
Should be safe to merge into main.
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #2423
## Test Plan
- [x] ⚡ Unit test
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds clues and fault handling to selectors pkg.
Some bleed upward into the CLI occured from
where the cli directly calls selectors.Reduce.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
DiscreteScopes is a vestigial func from when scopes contained the list of resource owners to track. That behavior is no longer in use.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds webUrl scopes to the sharepoint selector.
Also introduces the idea of a scope categories
that can broadly match across all leaf types
within a service. This category union should be
reserved for root categories, and properties
that can be used interchangably with the root.
This is part 2 of exposing webURLs as an alternative
to siteIDs for sharepoint backup and restore. The
next change providing a webURL => siteID lookup
within the graph package.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
For folder-level scopes (ie, scopes that compare
folder-hierarchy path segments), this change
replaces the standard "equals" and "prefix"
string comparators with the new PathContains
and PathPrefix comparators.
Next change is to interpret user inputs in the
cli to determine whether the comparator should
use contains or prefix behavior.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1224
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Builds a scope.Category() -> path.CategoryType trans- former func into scopes to standardize the relationship between the two properties.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
## Description
adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.
Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Fixes an issue with the path matching logic in selectors where if the path was a substring of what was
specified as the scope in the filter, the item would still be matched.
e.g.
Given 2 items - `/fold/contact1` and `/folderA/folderB/contact2` and a
selector `er.Include(er.ContactFolders("AnyUser", []string{"folderA/folderB"}))`,
the selector would match both items because `fold` is contained in the selector scope `folderA/folderB`
The fix is to invert the comparison - we check if the selector scope is contained in the item path instead.
In the example above, the selector string `folderA/folderB` will then only match the second items path.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Moves the `path` package to the `pkg` package so other code outside of Corso can use it if they need it
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#908
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Over-restrictive scope correlation caused the
reduce processor to only include filters which
matched the data type of the scope. Ironically,
this allowed a superset of information to match,
by evading the _all-match_ expectations of
filter scopes.
Also replaces the data category consts inside /details
with the path category types, since those are acting as
better canonical owners of data type identification
throughout the app.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #890
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
The details entry shorthash can be treated as
equal to the leaf item ID. This adds support to
the selector reduce step to compare the leaf val
to either the path item ID or the shortHash.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* closes#572
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
* Have exchange data collection store path.Path
Still complies with the old FullPath() string interface until we update
that.
* Pass path.Path to NewCollection for exchange
Basically fixes up errors introduced by previous commit.
* Fixup exchange recovery path indices
All exchange paths now use the path struct, meaning the service,
category, and user elements are in the standard positions.
* use path package in selector reduction (#822)
Currently, during a reduction process, scopes
compare their values to the raw split on repoRef.
This causes some brittle indexing to retrieve
values from the rr, carrying assumptions that are
difficult to track across changes. This PR trades
the string split for the paths package to better
integrate identification of the path values.
Adds some mocks and amends some error
behaviors in order to fit paths into the current
testing schema.
Co-authored-by: Keepers <ryanfkeepers@gmail.com>
* Remove pathType const
The path package const only provides information on the category, not
the service for the path.
* Fix var name/package name clashes
## Description
Adds foundational cli integration tests for backup/restore
of events and calendars
## Type of change
- [x] 🤖 Test
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
The path package changed the standard format
of fullPath and repoRef design. This should have
failed tests before being pushed to main, but was
able to slip in falsely while github actions were
configured to pass all tests until failed.
Will help make later PRs easier as the category will already be known.
## Description
<!-- Insert PR description-->
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
on path to:
* #456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the eventOrganizer exchange filter, wraps
up the addition of eventRecurs filtering, and fixes
a couple improperly designed filter constructors.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Exchange events were amended to use calendars
as a folder structure. This updates the selector
to treat events as having folders similar to mail and
contacts.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds filtering support for exchnage events in
the exchange selector.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
The filters package allows callers to specify both a target
to match on, and behavior of the comparison. While data-
type scopes always equate to "equals", the control over
different comparison behavior is useful for info-type
filters. This change integrates filters into scopes for
built-in control of those comparisons.
Though enumerated as ints, the selector service
categories are stored and managed as strings. The
only time that we use the int is when passing their
iota const to and from functions. Using a string
type instead of an int allows us to get rid of the
string builders and AtoI funcs (though not the
stringer requirement in the Iface, sadly).
