<!-- PR description-->
Add a killswitch for sliding window limiter. This is only relevant for exchange backups Setting this flag switches exchange backups to use the default token bucket rate limiter.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Concurrency limiter is stable and has been in use since several months. We had added a killswitch to disable it for safety reasons. It's not being used by SDK consumers, and hidden in corso cli.
This PR removes the `--disable-concurrency-limiter` flag & associated control option.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Follow up on #4557 [comment](https://github.com/alcionai/corso/pull/4557#pullrequestreview-1701971318) to consolidate test code for ease of use
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
centralizes details command processing in the cli
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2025
corrects the storage and provider flag positioning to attach to the child command instead of the parent. Also corrects unit tests for flags to ensure flags are preset as expected.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
allows cli users to limit the page size of delta queries by calling a new hidden flag: --delta-page-size.
This also adds the control.Options struct to the api client, so that configurations such as this can be easily handed into, and used by, the client.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
The goal of this PR is to normalize the cli packages in a way that 1/ showcases clear ownership of data, 2/ minimizes package bloat, and 3/ helps avoid circular import issues.
To achieve this, two primary changes were made.
First, the cli/options package was folded into cli/utils, so that all "shared functionality" is owned by a single package. Second, all flag values, globals, declarations, and mutator funcs (in the cli layer, logging package was not changed) were extracted from cli/utils and placed into cli/flags. This divides ownership between the declaration and population of the flags (cli/flags) from the utilization of values derived from flags in command processing (cli/utils).
This PR contains zero logical changes. Only code
movement and renaming.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3664
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
populate a "test_name" clues field in the
tester context, so that logs can be associated
with the calling test.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Add tests for older backup details versions
including things like:
* SharePoint using OneDriveItem type
* not having LocationRef
* not having ItemRef
* folder names in RepoRef
* file names in RepoRef
Recommend viewing with ignore whitespace changes
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3269
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
When the user's mailbox is full, we cannot make use of delta apis. This adds initial changes needed to create separate delta and non delta pagers for all of exchange.
*I would suggest looking commit wise when reviewing the PR.*
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
It seems to have been accidentally removed in e4be050.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
In order to retrieve all clues structured error data in tests, we need to extract it from the error using the clues library.
This change appends `clues.ToCore(err)` to all
variations of `assert.NoError(t, err)`. The only
other changes are those necessary to preserve
linting, or to produce an error variable for the
ToCore call.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
adds a hidden flag to backup details that
allows the caller to see all details retrieved
by the repository, without running the results
through the selector reduce step.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2675
#### Test Plan
- [x] 💪 Manual
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2373
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
adds fault.Errors to backupDetails() and backups().
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds clues and fault handling to selectors pkg.
Some bleed upward into the CLI occured from
where the cli directly calls selectors.Reduce.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
* add bool format checks for string flags
Adds a boolean-parseable validator for cli flags
that accept strings representing boolean values.
* fix ctx production in cli integration test
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Removes the -all flag from the exchange cli. The
same functionality can be gained with the flag
--user *.
Additionally does some minor tidying of other
comments and outputs throughout the cli.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1032
## Test Plan
- [x] ⚡ Unit test
## Description
* Clean up pass on all CLI output
* Added basic extended usage
* Mark required params in generated docs
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes#527
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add tests for the `backup list exchange` subcommand. Tests mostly center around indirectly testing how selectors are created and used in this subcommand by checking the output of running the `Reduce` call on a known set of `details.DetailsEntry`s. Also check various error cases
Tests for invalid formats of flag values are disabled as that code does not currently exist. These tests can be enabled when #943 is resolved
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
removes the requriement that higher order data
must be specified in order to specify granular
data ids. Ex: previously, cli users needed to add
--user and --folder flags in order to use a --mail
flag. Now the user and folder will default to Any()
if they're not provided.
## Description
Adds foundational cli integration tests for backup/restore
of events and calendars
## Type of change
- [x] 🤖 Test
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
## Description
Adds the new event filter selectors to the CLI for
backup detail and restore commands.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the event-calendar flag to the cli to prepare
for selectors including the calendar as the events'
"folder" structure.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
The filters package allows callers to specify both a target
to match on, and behavior of the comparison. While data-
type scopes always equate to "equals", the control over
different comparison behavior is useful for info-type
filters. This change integrates filters into scopes for
built-in control of those comparisons.
The alias of testing to ctesting prevented auto-importing
of the testing library. This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
* reduce filter selector flags to single strings
Since it doesn't make sense for many of the filter-type
selector flags to receive multiple input values (ex: what
would --received-after date1,date2 result in?), the filter
flags are getting reduced to single-string values.
* add must-pass filters to selectors
Extends the selectors scope set to include Filters. This
allows users to define all-pass matchers (filters),
separate from any-pass matchers (includes), and global
exclusions.
When a caller constructs an Users() exchange scope,
instead of building a single scope that contains all of the
provided users and all of the data beneath, follow the
pattern of the other scopes and generate one scope per
user-category pair.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* add per-data type flags to backup details
Adds the contacts, contact folders, emails, email folders, events,
and users flag support to `corso backup create exchange`
to enable selector support.
* add --all and --data to backup create
Adds flags for backing up all exchange data, and
for isolating the data in the backup by data type.
Introduces validation and selector creation in
backup create. Switches the --user flag variable
type from a string to a []string.
