32 Commits

Author SHA1 Message Date
Danny
b3a1de89bb
GC: Item attachment contact support (#2465)
## Description
The logic for sanitizing Contactable data for restoration of `ItemAttachable.Contact` types. 

Contact `Item.Attachment`s required the removal of:
- `odata.Context` 
- `ETag` 
- `ParentFolder`

Otherwise, the following error occurs on POST. 
```bash
UnableToDeserializePostBody were unable to deserialize
```
<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2426<issue>

## Test Plan
- [x]  Unit test
2023-02-10 14:23:18 +00:00
Keepers
313f05fcb6
add clues, fault to small /internal pkgs (#2336) 2023-02-06 17:44:35 -07:00
Keepers
a13f1cc3a6
remove common maps, add golang maps (#1885)
## Description

replaces the common/maps.go code with the
golang experimental maps package. The
maps package provides standard api
such as Keys, Values, Copy, and Clone.

https://pkg.go.dev/golang.org/x/exp/maps

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Test Plan

- [x]  Unit test
2022-12-21 18:08:26 +00:00
Keepers
c8ad2e03ce
produce coll state and item deletion (#1815)
## Description

Fulfills the data collection interfaces to provide item deleted
flags and collection state enums.

A note on the additional data consumption: I'm making an assumption
that the @removed property will appear in that map.  This is awaiting
testing for verification, which I'll get into as a follow-up step.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1727

## Test Plan

- [x]  In a follow-up PR
2022-12-19 20:16:51 +00:00
Keepers
99b0f51980
consume prevPath on next backup (#1802)
## Description

Parses the previousPaths metadata collections
along with deltas, and hands paths down to
exchange backup collection producers.  Does
not yet scrutinize previous/current path diffs.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1726

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 19:59:34 +00:00
Vaibhav Kamra
c4333ad531
Use latest golangci-lint version (#1634)
## Description

Use latest golangci-lint version. Also updates cache flags.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1629 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:17:01 +00:00
Abin Simon
5089f9d949
Add EndTime to Exchange events (#1446)
## Description

Add EndTime to Exchange events. Also fixes a testcase around start time.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-07 09:28:27 +00:00
Vaibhav Kamra
09261d5474
Allow disabling TLS and TLS verification (#1417)
## Description

Introduces config options (`--disable-tls` and `--disable-tls-verification`) to turn off TLS and TLS verification

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1415 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-03 04:47:19 +00:00
Georgi Matev
0c921e8b40
Normalize repo suffix paths (#1153)
## Description

Repo `--prefix` values should be normalized with a trailing `/`

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* Fixes #1152  

## Test Plan

<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-13 02:11:13 +00:00
Keepers
0b65b20200
attach logger to test ctx, use flusher (#1083)
## Description

Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context.  The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton.  But in order for the logs to show up
within the test, they need to get flushed.

See my comments for focus on non-chaff changes.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-10 21:14:10 +00:00
Keepers
89668ed164
describe time value standards, validate time flags (#1066)
## Description

Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.

Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.

## Type of change

- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test

## Issue(s)

* #943
* #924

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-10-06 17:46:21 +00:00
ashmrtn
3f0951dea2
Higher resolution on test restore folder names (#1040)
## Description

\~Only affects folders created by tests\~
Create and use helper function that sets the restore destination folders to have microsecond granularity. This is to avoid naming collisions with second granularity (of which I've observed at least once)

Switches all tests for all services/categories to use OneDrive DateTime format+microseconds

Manually verified that a manually created folder with the given timestamp format was removed by the CI purge script

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1036

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-04 23:52:13 +00:00
Keepers
fe148af455
normalize bucket naming during config lookup (#1025)
## Description

Ensure s3 bucket name inputs are normalized
in all uses.  Currently, the normalized format is
applied when using storage, but not when passing
around --bucket flag values elsewhere.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #975

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-03 21:37:20 +00:00
ashmrtn
b612718723
Update time parser to handle OneDrive format (#1013)
## Description

OneDrive format uses '-' in place of ':'. Purge command needs to be able to clear folders with this format as well.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1012

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 22:38:43 +00:00
Vaibhav Kamra
921d77cc05
Implement OneDrive DataCollection restore (#877)
## Description

Implements `data.Collection` restore for OneDrive and wires it up to the restore operation

Leverages previously added `onedrive` helpers that have integration tests.

An op/CLI level integration test will be added in a follow-up PR.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #668 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

```
$ ./corso restore onedrive --backup 7de4d68f-67c2-4e81-95a2-81cf0d1bd11d
Restored OneDrive in S3 for user
```
2022-09-19 21:46:30 +00:00
Keepers
1203fd2b6a
extract timestamp from arbitrary string (#883)
## Description

In order for purge to gracefully handle the variety of timestring values that may appear within a
folder name, the common/time.go file has added
a func to extract a time substring from a string.
Purge is switched to that extractor instead of
a len-based suffix check.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #805

## Test Plan

- [x] 💪 Manual
2022-09-19 20:35:49 +00:00
Keepers
964a2fc39e
new format for tabular display of timestamps (#793) 2022-09-09 20:43:18 -06:00
Danny
2e3ee15fd4
Sent / Receive DateTime Stamp Incorrect (#792)
## Description
Single Legacy Policies added to allow the values for sent / receive times to correspond to the original content for `exchange.Mail` objects

## Type of change
- [x] 🐛 Bugfix


## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

*closes  #645

## Test Plan

<!-- How will this be tested prior to merging.-->

- [x] 💪 Manual
Values can be inspected using e2e restore pipelines.
2022-09-09 03:39:52 +00:00
Keepers
6cdc691c6f
correct the corso module (#749)
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.

All other changes in the repository update the
package imports to the new module path.
2022-09-07 15:50:54 -06:00
ashmrtn
09cc2769d9
Fix most remaining wsl lint errors (#656)
* Fix wsl lint errors in pkg package

* Fix wsl lint errors in most of internal package

Leave some sub-packages out that have higher churn at the moment.
2022-08-26 17:58:58 +00:00
Keepers
192c69c68f
add backup cli integration test (#517)
Adds the basic cli-based backup integration test.
Due to discovering some corner cases about panic
conditions and other error handling in bad runtime
state, updates many other packages for safety.
2022-08-12 20:04:44 +00:00
ashmrtn
da66cc4c2f
Enable more rigorous version of gofmt linting (#488)
* Enable a stricter linter

* Fix new lint errors
2022-08-05 13:33:20 -07:00
Keepers
20aa3d3a28
purge test user folders every half hour (#491) 2022-08-05 18:24:58 +00:00
Keepers
2d6c765ee8
add contains helpers to exchange scope (#487) 2022-08-04 23:17:52 +00:00
Keepers
a586512b42
add command to purge user's folders by prefix/date (#480) 2022-08-04 15:28:28 -06:00
Danny
da9a22271d
Mock Connector expansion (#437)
Mock Connector data collection is a Messageable item and has been changed to be utilized in other portions of the corso repo.
2022-07-29 15:08:36 -04:00
Keepers
7e60ec5073
move operation.Options into /pkg/control (#452)
In order for sdk users to build an operation using a repo,
without importing /inernal packages, the operation
options needs to be available via a /pkg package.
2022-07-29 12:52:19 -06:00
Keepers
0739ea7e09
add info-based selector flags to backup details (#380)
* filter backup details by flags

`backup details` should have its output filtered by the flags provided by
the user.  In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
2022-07-22 13:23:16 -06:00
Keepers
532922f662
introduce exchange info selector support (#379)
* introduce exchange info selector support

Adds support in selectors/exchange for queries based on
backup.ExchangeInfo entries.  This allows the declaration
of selectors based on non-identifier details such as sender,
subject, or receivedAt time.

Changes Exclude scope matching from being an Any-
match comparator to an All-match.  This keeps exclude
and include behavior identical, hopefully making less
confusion for users.
2022-07-21 15:08:03 -06:00
Danny
0422b9ecbe
Restore Policy options added (#350)
* Issue #349:
Added policy for restore to code base.  Policy Items are placed in the `internal/common` directory
2022-07-19 21:28:23 +00:00
Danny
5a9f2e4601
GC custom errors to inherit from internal/common (#337)
Custom error package added and extended to the internal/connector package
2022-07-15 21:21:11 +00:00
Keepers
0bd23ab2ab
move config unions to common code (#224)
* move config unions to common code

The configuration union handlers in Storage and Account
overlapped significantly in behavior.  Moving those helpers into
a common code folder was requested.  Although the behavior
was similar across the files, the types were not, requiring
the addition of generics to solve the need.
2022-06-23 12:16:20 -06:00