527 Commits

Author SHA1 Message Date
Keepers
f574fc9729
remove model.StableID from repository iface (#2997)
model.StableID is an internal value.  SDK facing
interfaces should only need to provide strings.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-03-31 17:50:40 +00:00
Keepers
6190b1ba05
use equalfold in filter equals checks (#2995)
Because it's nifty.

-----

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-03-31 16:36:19 +00:00
Abin Simon
446ccfe491
Skip large OneNote items that cannot be downloaded (#2925)
OneNote files >2GB cannot be downloaded via the graph API. This change will gracefully skip them and prevent the backup from erroring out.

A OneNote file is represented by a Package(folder) which contains two file within it, `<file>.one` and `<file>.onetoc2`.  From what I can tell the `onetoc2` file is the metadata file and the `one` file is the one that contains data and can get big.

**The current implementation has a limitation(or feature depending on how you see it) in that we will end up doing a partial backup with just the `onetoc2` file but avoiding the big `one` file if that is big.**

I did not try to change the behaviour as even currently if for somehow we ended up with an error like 404 when trying to backup one of these files, we do skip it. I've added a comment in for now, let me know if we should go back and revisit the behaviour. *The main issue is the fact that we only hit this problem once it reaches kopia and removing it at that point is harder.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2910

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-31 04:43:34 +00:00
Keepers
4a395d44a8
move graph connector creation to new op (#2923)
Moves the generation of graphConnector to the
NewBackupOp and NewRestoreOp constructors
inside the repository.  This removes gc creation
from the backup and restore operations, and
requires a reference to gc to exist in operations.

This sets up two changes: 1/ mocking of GC
within operation (the next PR will replace it with
an interface).  2/ the ability to perform a gc
validation step within the operation construction,
and thus the capacity to look up a resource owner
display name or id, and vice versa.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-29 22:00:30 +00:00
Keepers
e09c120778
add linter for using clues to create and wrap errs (#2959)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-03-29 21:16:05 +00:00
Keepers
940892fc07
correct input aliasing (#2960)
Updates and corrects input aliasing according to
the following rules (in priority order):
1. if the library name is usable, use it
2. if not, alias to the package name
3. if the package name is weird, alias sensibly
4. in case of collision, alias more distant imports
5. aliases should be consistent throughout

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-03-28 20:48:22 +00:00
Keepers
9f0e259764
metrics PoC for github.com/armon/go-metrics (#2884)
Adds a logger-oriented metrics tracker to aggregate statistics like requests-per-minute and average
http request time.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2212

#### Test Plan

- [x] 💪 Manual
2023-03-28 15:57:55 +00:00
Keepers
9d73d0c8e2
replace errors.* with clues.* (#2924)
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.

Might follow up with a linter to enforce this change.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-28 04:01:54 +00:00
Keepers
8acff1ad93
fix sharepoint library folder selection (#2962)
Bugfix:
Sharepoint LibraryFolder scopes always return
zero matches in filtering with a prefix due to the
path not having the drive root prefix truncated, as is
expected of onedrive-based storage paths.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-03-28 01:21:20 +00:00
ashmrtn
dfceb79e0a
Update Exchange backup detail ParentPath when merging details (#2826)
Exchange now holds a reference to the parent folder so be sure to update
that if needed when merging backup details

Calendar is the only thing that populates location ref right now so this
alone doesn't completely solve the issue

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2819

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-27 20:44:41 +00:00
Keepers
ddeaaf2ff2
Sanity test cleanups and additional info (#2816)
updating sanity tests: fix the permissions check by not using reflect.DeepEquals, and various code cleanup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-03-27 16:25:08 +00:00
Abin Simon
c3610d1832
Revert "force omission of meta files (#2662)" (#2706)
This reverts commit 1945db6b65827e6dd6d57de8ac42fe4a119db685.

We do add the `isMeta` flags for missing items at f60207ee32/src/pkg/repository/repository.go (L363)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-24 07:48:28 +00:00
Keepers
f76222df73
introduce the resource owner name to op.backup (#2921)
adds a resourceOwnerName property to the op
backup struct.  Also adds a resource owner id
and owner name to the backup.backup.  These
values will be used for both identification and for
end user display.  Values are not currently populated
reliably, nor are they used.  This change only
allows them to exist.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-24 01:23:53 +00:00
Keepers
285f8f01ef
update sharepoint cli examples (#2822)
Updates the sharepoint cli examples to better
reflect the current support and utilities.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2786

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-03-21 17:33:24 +00:00
Keepers
bdc854c1da
scopes typing rename, filter -> info (#2842)
Renames a lot of the names in scopes that contain
the word "filter" with the word "info".  The basis here
is to reduce confusion in the libraries.  Filter as a term
is overloaded.  It currently means: 1/
the set of scopes that comparisons _must_ pass
to succeed; 2/ the pkg and  struct that handles
comparison logic; and 3/  the set of scopes that
compare values in details.ItemInfo instead of in
the item's storage or logical path.  The third
category is the primary overload here, and is
better off with a different name.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
2023-03-21 06:51:14 +00:00
Keepers
8b1c3a7073
add labels to logged errors (#2824)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-03-20 21:12:05 +00:00
ashmrtn
2176f4b831
Add option to control parallelism getting items (#2864)
This option is only consumed by Exchange right now and will control the number of items fetched in parallel per collection. Note that the total number of parallel item fetches may be higher is kopia is concurrently uploading multiple collections.

This PR does not expose this option at the CLI layer

Manually tested to ensure it
* uses the default if options value is 0
* uses the default if options value > default
* uses option value otherwise

There's another PR open for this but it's not expected to main. We still
want to make sure the change goes in though, hence this PR.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-20 20:04:49 +00:00
Keepers
fa2cf046bb
fixes, logging improvements for fault.Errors (#2848)
A last-second change in 2708 caused us to
pass along the wrong fault.Errors into backup
persistence, thus slicing the count of skipped
items.  That's been fixed, along with improved
end-of-operation logging of fault errors.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2708

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-03-20 06:01:04 +00:00
Keepers
8dfb00f308
add fault errors streamstore (#2731)
Adds a new streamstore controller for fault.Errors. This provides large scale, extensible file storage for fault errors to be persisted, much like we do for backup details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x] 💚 E2E
2023-03-17 01:07:07 +00:00
Keepers
26bf24c77e
add load tester suite (#2809)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💚 E2E
2023-03-16 23:04:35 +00:00
Keepers
4515d3a4b2
remove unused test scoping envs (#2648)
Doesn't seem like we're using these

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2373

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-16 08:00:49 +00:00
Keepers
1b6ad654e8
fix sharepoint filter consts (#2776)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2757

#### Test Plan

- [x]  Unit test
2023-03-15 20:58:43 +00:00
Keepers
76b3fe3b86
append ToCore to all errors tests (#2793)
In order to retrieve all clues structured error data in tests, we need to extract it from the error using the clues library.

This change appends `clues.ToCore(err)` to all
variations of `assert.NoError(t, err)`.  The only
other changes are those necessary to preserve
linting, or to produce an error variable for the
ToCore call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-15 19:02:47 +00:00
Abin Simon
0125876192
Skip items returning 404 (#2805)
Treat a 404 as an unavailable item and skip it similar to malware items.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
2023-03-15 17:52:22 +00:00
Vaibhav Kamra
3d96507719
Do not query repo ID if metrics are disabled (#2803)
This skips the call to `getRepoModel` if metrics are disabled and the repository ID is not required.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2802 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-15 13:12:28 +00:00
ashmrtn
8a81624b98
Change to storing files by ID instead of display name (#2496)
## Description

Change to storing OneDrive files by ID instead of their OneDrive display name. This allows delta token-based incrementals to use an exclude list to remove previously backed up items from a backup during hierarchy merging

Also updates the following:
* selectors to match on file display name instead of kopia file name
* ShortRefs for OneDrive files to update when the file display name is updated

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1535

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-03-14 22:32:06 +00:00
Keepers
635e4f6bc4
rename libraries -> libraryFolder (#2774)
The selector moniker "libraries" is incorrect.  It should refer to the library folder, since it only
matches on directory structures within a given
drive.  The 'library' is analogous to the drive
itself, and will need a separate selector of its
own.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2757

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-14 01:38:54 +00:00
Aarti Jivrajani
120eb0d256
Add a helper function that returns site ID and site URL in a Site struct (#2778)
This PR adds a helper function that returns a newly introduced `Site` struct that has a site ID and site webURL for a given m365 tenant.

---

#### Does this PR need a docs update or release note?
- [X]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [X] 🌻 Feature

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2777 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [X]  Unit test

```
--- PASS: TestM365IntegrationSuite (2.49s)
    --- PASS: TestM365IntegrationSuite/TestSites (1.82s)
        --- PASS: TestM365IntegrationSuite/TestSites/site (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#01 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#02 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#03 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#04 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#05 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#06 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#07 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#08 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#09 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#10 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#11 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#12 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#13 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#14 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#15 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#16 (0.00s)
        --- PASS: TestM365IntegrationSuite/TestSites/site#17 (0.00s)
```
2023-03-14 01:00:11 +00:00
Keepers
dc21f4ce07
add a QoL path.Build func (#2763)
Adds a path.Build() func that can arbitrarily build a path according to expected standards without first initializing a path.Builder{}.
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-13 16:56:31 +00:00
Keepers
a83f16c6e4
add logger CtxErr() qol func (#2599)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-11 01:41:13 +00:00
Keepers
871f0b406d
introduce fault items (#2710)
Adds the item struct to the fault package for tracking serializable and dedupliatable error sources.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
2023-03-10 17:23:19 +00:00
Abin Simon
71d2c3deb1
Augment paths for OneDrive restore (#2718)
This adds the missing .dirmeta files which are needed to make sure that permissions are being restored properly.

By adding a dirmeta entry for each folder that is part of the items, we make sure that all folders get their own collection which in turn will help us properly restore permissions for all folders.

**This also makes it so that meta items are not returned from older backups.**

*Although the newer version can fetch permissions file using the Fetch API, we still have to populate the items as it cannot fetch items from previous directories which is needed to make sure we can restore permissions correctly.*

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2697

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-10 05:44:23 +00:00
Danny
0766ee7afd
GC: SharePoint: ItemType Decouple (#2720)
<!-- Insert PR description-->
ItemTypes for SharePoint were `SharePointItem` only. This did not allow for data to be separated by category. 
Newly created categories:
- SharePointLibrary
- SharePointList
- SharePointPage

Breaks history for `SharePointItems`. All previous SharePoint items are available during `Corso List` command. However, when using the details command with the backup ID for `SharePointItems` restored prior to this PR , the return will be:
```
no items match the specified selectors
```
---

#### Does this PR need a docs update or release note?


- [x] 📝 : Will need documentation

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2709<issue>

#### Test Plan

- [x]  Unit test
2023-03-09 15:51:24 +00:00
ashmrtn
f28da24153
Use slice filter for In filter (#2735)
Have the `In` filter take a slice of values instead
of a joined string. Change negated slice filter behavior
to be the boolean negation instead of
(!filter-val-1 || !filter-val-2 || ...)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1535

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-09 01:02:32 +00:00
Keepers
23b90f9bec
selectors use mulit-value short-refs (#2736)
With onedrive storage file names being changed from the file display name to the file id, we need a more granular form of indentification when using selectors to choose which values count as matchable fields.

This change modifies the selector PathValues to return slices of strings for each category instead of a single string, and the reducer matches on any. This will allow each service to decide what values are considered equivalent (id, shortRef, a value inside the info, etc) for each property.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
2023-03-08 23:51:49 +00:00
Keepers
07b4900f58
remove stats.Errs (#2591)
## Description

Now that fault errors is in place, we don't need
the operation stats errs.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-08 20:19:19 +00:00
Keepers
0b2d0067ee
refactor streamstore for generic use (#2723)
Refactors the stream store so that it can comply with multiple different writers and readers beyond just the details.Details struct.  This will allow us to use the stream store to ingest large lists of errors fault items without burdening the backup manifest.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-08 17:16:09 +00:00
Keepers
1ca49c53a9
remove wrap and append support (#2589)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-08 01:24:48 +00:00
Keepers
bb61697a58
migrate streamstore details to own file (#2725)
Initial code movement before getting into larger refactor.  The goal is to refactor the stream store so that it can comply with multiple different writers and readers beyond just the details.Details struct. This will allow us to use the stream store to ingest large lists of errors fault items without burdening the backup manifest.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
2023-03-07 23:19:26 +00:00
Danny
030147ddcb
Corso: Exchange: Mail: Backup Details to include recipients --JSON ONLY (#2689)
<!-- Insert PR description-->
#### Adds  Email recipients to the backup details for Exchange Mail. If the recipients are of length greater than one, the list of individuals is included and separated by a `,`

---

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #2687<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-03-07 02:32:38 +00:00
Danny
9cfaf3c140
CLI: SharePoint.Libraries selector expansion (#2679)
<!-- Insert PR description-->
Selector expansion for filters includes the following:
- time flags for details filtering
- testing
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2602<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-03-06 22:34:50 +00:00
Danny
36d7d44bb4
Corso: Exchange: Details: add parentpath (#2690)
<!-- Insert PR description-->
Parent Path field added to  `Exchange` Backup Details 
---
- Only Exchange mail will display the path for  backup details. 
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2352<issue>

#### Test Plan


- [x]  Unit test
2023-03-06 18:01:26 +00:00
ashmrtn
83fe83a367
Exclude metadata files from the set that we merge during incrementals (#2650)
Metadata files don't need to appear in backup
details, which means they also shouldn't be
merged during incremental backups. If we
expect them to be merged then we'll fail the
item count check for the merge process

This excludes items with metadata extensions
from merging by not adding the to the list
of items to merge

Also make it so that metadata items don't
create backup details entries in the first
place

Manually tested to ensure
restore(backup(onedrive)) still works as
expected
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2498

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-02 13:49:19 +00:00
Keepers
e42e3f253b
add flag to output all details (#2676)
adds a hidden flag to backup details that
allows the caller to see all details retrieved
by the repository, without running the results
through the selector reduce step.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2675

#### Test Plan

- [x] 💪 Manual
2023-03-01 00:02:29 +00:00
ashmrtn
aef90f8b86
Remove GraphConnector user population (#2582)
## Description

Don't populate users when GraphConnector instance is created

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1994

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-28 21:26:12 +00:00
ashmrtn
ceb60a1c2b
Remove old test that's not needed (#2665)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2373

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-27 23:53:06 +00:00
Keepers
1945db6b65
force omission of meta files (#2662)
While the `isMeta` flag will solve this in the
future, the current state requires us to filter
out all meta and dirmeta files, even historically.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-27 22:46:55 +00:00
Danny
ad75540b03
GC: SharePoint: Library: Details: Change to Displayed data (#2632)
<!-- Insert PR description-->
Updates for `./corso backup details sharepoint --backup <bID>` 
- Ensures that Drive Name is part of the backup
- Adds field DisplayName to backup details

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #2601 <issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-02-27 15:07:42 +00:00
ashmrtn
40cb810861
Use test wrapper in pkg package (#2531)
## Description

Leave out account and storage subpackages as those are used by tester and create an import cycle.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-25 04:50:00 +00:00
Keepers
9e783efe3a
fault package funcs rename (#2583)
## Description

Renaming the funcs in the fault
package to be more clear about
their purpose and behavior.  Largely
just find&replace changes, except
for fault.go and the fault examples.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-25 03:29:02 +00:00