746 Commits

Author SHA1 Message Date
Keepers
cb467a3e46
correct the transformer used for contact folders (#814) 2022-09-09 15:34:28 -06:00
Danny
697701afeb
Reduce the amount of folders that are checked for serialization checks. (#807) 2022-09-09 14:23:01 -06:00
Keepers
6f7cf00188
add cli integration tests for event/contact (#760)
## Description

Adds foundational cli integration tests for backup/restore
of events and calendars

## Type of change

- [x] 🤖 Test

## Issue(s)

#501 

## Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-09-09 14:15:36 +00:00
Danny
2e3ee15fd4
Sent / Receive DateTime Stamp Incorrect (#792)
## Description
Single Legacy Policies added to allow the values for sent / receive times to correspond to the original content for `exchange.Mail` objects

## Type of change
- [x] 🐛 Bugfix


## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

*closes  #645

## Test Plan

<!-- How will this be tested prior to merging.-->

- [x] 💪 Manual
Values can be inspected using e2e restore pipelines.
2022-09-09 03:39:52 +00:00
Danny
ca6331b66e
[Fix] Issue #801: Error during Paging (#802)
## Description
Pagination fix for nill pointer

## Type of change

- [x] 🐛 Bugfix## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #801

## Test Plan

- [x]  Unit test
2022-09-09 02:27:47 +00:00
Danny
c02817ac86
CI: Bug Fix Iterator Tests (#800)
## Description

Fixes mapping alignment during iterator tests

## Type of change


- [x] 🐛 Bugfix
- [x] 💻 CI/Deployment


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #797
## Test Plan

- [x]  Unit test
2022-09-09 01:59:46 +00:00
Vaibhav Kamra
57bb229f35
Implement a logging middleware to log Graph requests (#791)
## Description

Enables a logging middleware that will log the Graph http calls if the `LOG_GRAPH_REQUESTS` environment variable
is set

## Type of change

Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E

```$ LOG_GRAPH_REQUESTS=true CORSO_CI_TESTS=true go test -v .
...
=== RUN   TestGraphConnectorIntegrationSuite
2022-09-08T13:39:25.556-0700	INFO	graph/service_helper.go:61	REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>


    integration_runners.go:58: TestGraphConnectorIntegrationSuite run at 2022-09-08T20:39:25.967824Z
=== RUN   TestGraphConnectorIntegrationSuite/TestAccessOfInboxAllUsers
2022-09-08T13:39:26.398-0700	INFO	graph/service_helper.go:61	REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>
```
2022-09-08 21:12:13 +00:00
Vaibhav Kamra
1246f51b22
OneDrive Backup Operation (#738)
## Description

Wires up the OneDrive collection logic to `operation.Backup`

Includes an integration test that runs against the test domain

Two bug fixes:
- Skip the "root" item that is returned by the delta query
- Fix incorrect usage of the `filepath.SplitList` function which does
  not split a path into components. Instead use `strings.Split`. This
  is ok because the paths returned here are not OS specific.
  Regardless - this logic will be refactored when we use the `path` pkg.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
#548 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-07 23:47:12 +00:00
Keepers
6cdc691c6f
correct the corso module (#749)
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.

All other changes in the repository update the
package imports to the new module path.
2022-09-07 15:50:54 -06:00
Danny
6640d42aac
GC Restore: Event pipeline not working [FIX] (#780)
Bug Fix to ensure E2E testing. Test suite expanded.
2022-09-07 17:14:40 -04:00
Keepers
c73ca6a4e5
correct paths in selectors (#789)
The path package changed the standard format
of fullPath and repoRef design.  This should have
failed tests before being pushed to main, but was
able to slip in falsely while github actions were
configured to pass all tests until failed.
2022-09-07 14:14:51 -06:00
ashmrtn
d707c22205
Resolve mail exchange folders to their real path in the hierarchy (#752)
## Description

Creates and uses a `ContainerResolver` interface to fetch container paths for items of different categories (when other resolvers are implemented). If the resolver is not available or fails to resolve a folder, defaults to the old implementation of using the folder's ID as its path.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

* #456 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-02 23:18:49 +00:00
ashmrtn
12f61dd71c
Switch to using path categories in graph connector (#734)
Will help make later PRs easier as the category will already be known.

## Description

<!-- Insert PR description-->

## Type of change

Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
on path to:
* #456 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-02 20:12:22 +00:00
Vaibhav Kamra
ed1a4bebce
Use a WaitGroup for AwaitStatus (#716)
## Description

This builds on the `MergeStatus` proposal and `WaitGroup` discussion proposed in #494 
Required for OneDrive where we are operating on multiple collections

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
#494 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-02 17:17:17 +00:00
ashmrtn
61ce920972
Cleanup parameters to functions to appease linters (#730)
## Description

Group like parameters to graph functions so SonarCloud is happier

## Type of change

Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
on the path to:
* #456 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-02 00:26:31 +00:00
ashmrtn
ae28971be3
Thread contexts through some GC functions (#728)
## Description

Will be used later and just generally useful if we ever add tracing or
such to these functions.

## Type of change

Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

#456 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-01 23:31:43 +00:00
ashmrtn
4a96f2571d
Second attempt at directory cache/resolver for a user (#724)
## Description

Directory cache capable of converting a folder ID to a complete path to the folder

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

#456 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-01 23:03:35 +00:00
Danny
4398a67a5a
Bug fix: Addresses (#717)
## Description
When a mock event is created the event is sent to personnel. Requires that personnel have valid email addresses.
Addresses created and updated in the mockconnector package. 
## Type of change

Please check the type of change your PR introduces:
- [x] 🐛 Bugfix


## Test Plan

- [x] 💪 Manual
No emails populate when `TestRestoreEvents` completes
2022-09-01 16:11:08 +00:00
Keepers
c29d93e655
adds calendars as a folder structure to events (#714)
## Description

Exchange events were amended to use calendars
as a folder structure.  This updates the selector
to treat events as having folders similar to mail and
contacts.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature

## Issue(s)

#501

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-01 15:39:14 +00:00
Danny
7a25a7b70e
GC: Restore Event to M365 user account feature (#711)
## Description
logic and orchestration to take byte representation of M365 event, create a Corso calendar, and place the event and calendar into the M365 user's account. 
## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
closes #599 
closes #710
Requires PR #708 to ensure test folder is where events are created. 
## Test Plan

<!-- How will this be tested prior to merging.-->

- [x]  Unit test
2022-09-01 00:02:36 +00:00
Danny
8ead744e9f
GC: Restore: Create / Delete Calendar Feature (#708)
Feature creation: Create/Delete Calendar for aM365 user added to main branch
2022-08-31 22:03:54 +00:00
Keepers
5c59522fc2
print separate tables per entry type (#709)
## Description

Since a backup can encapsulate multiple data
types (ex: mail, contacts, and events), it doesn't
make sense to print one table with all disjoint
values.  This change splits up the print output
so that each data type in the details gets its
own table.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature

## Issue(s)

#501 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-31 21:10:10 +00:00
Vaibhav Kamra
8634b1b1ac
Helpers to create OneDrive Collections (#692)
## Description

This adds the logic that materializes collections for a specified user

The collection paths are currently derived from OneDrive metadata but a future
PR will introduce `paths` pkg support to create these.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)

#388 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-31 18:39:18 +00:00
Danny
305de77188
Iterate-Test Suite Move (#701)
Test suite move of iterator functions from `exchange-service-test.go` to `iterators_test.go`
2022-08-31 17:55:36 +00:00
Danny
b5a5378113
GC: Restore: Retrieve CalendarID Feature (#687)
Feature added to be able to retrieve containerID for `exchange.Event` types.
2022-08-31 17:28:57 +00:00
Danny
068f0560e4
Issue #501: Event support (#704)
Added EventRecurs as option for backup details.
2022-08-31 10:04:13 -06:00
Danny
67bc038c55
GC: Add WSL Linting Formatting src/internal/connector (#694)
Lint remaining of `internal/connector` package for wsl
2022-08-30 22:12:38 +00:00
Keepers
6f04321a60
replace scope values with filters (#674)
The filters package allows callers to specify both a target
to match on, and behavior of the comparison.  While data-
type scopes always equate to "equals", the control over
different comparison behavior is useful for info-type
filters.  This change integrates filters into scopes for
built-in control of those comparisons.
2022-08-30 21:29:24 +00:00
ashmrtn
e3abc281d6
Enable wsl for most of internal/connector package (#680)
* Lint part of internal/connector package with wsl

* Cleanup some wsl lint errors in connector package
2022-08-30 13:13:00 -07:00
Danny
0b7fe0fdb9
GC: Small GC Refactor of Restore workflow (#682)
Removal of 'unused' if-clause and category variable being updated.
2022-08-30 18:50:31 +00:00
Danny
0b3f345727
GC: Restore Contact Interface (#642)
Functionality added to restore `exchange.Contact` to M365 back store. Unit tests added to support proper testing of new feature.
2022-08-30 17:46:47 +00:00
Danny
b2d3e62536
Contact Restore: Create / Delete Folders
Contact folder creation and deletion enabled by the PR. Test suites expanded
2022-08-29 21:23:34 +00:00
Danny
d55ee98d7d
GC: Iterate "Select All Messages/Contacts" Consolidation with new Interface (#663)
`Descendable` interface used to move "IterateSelectAll" functions into a singular function.
Test suites are modified to support changes.
2022-08-29 16:29:15 +00:00
Danny
d3bced28f9
Bug fix: (#662)
Fixes bug introduced by merge.
2022-08-26 16:28:13 -06:00
Danny
ab92358ea6
GC: Backup all users flag support for exchange (#631)
All-users flag support enabled for Graph-Connector
2022-08-26 15:05:14 -06:00
Danny
2ceafd8d62
Contact / Mail folder interfaces (2) (#659)
Added 2 interfaces for future abstractions.
2022-08-26 16:15:55 -04:00
Danny
02c2b0b4d6
GC: Iterate-Functions moved to separate file (#657)
* Issue #654: Prepwork: `service_iterate.go` contains iterate functions for the package.
2022-08-26 11:47:49 -06:00
Danny
a2a5260d3f
Gc option extract (#653)
Content Separation `query_options.go` created for `exchange package.
2022-08-26 12:47:45 -04:00
Keepers
15b12e634d
name cleanup in selectors and onedrive (#643) 2022-08-24 15:13:04 +00:00
Danny
3b9c8e284c
GC: Reduce the amount of folders upon restore (#637)
* Issue #595: All collections with the same `fullPath` name are to be stored within the same folder.
2022-08-23 20:46:59 +00:00
Danny
6e7335cc4d
Gc restore multi framework (#639)
* Abstraction extended to restore. Application logic moved to service_functions.go
2022-08-23 20:30:46 +00:00
Keepers
b524053e96
resolve lint errors (#638) 2022-08-23 20:22:25 +00:00
Danny
a1f79862f1
GC: Event collections to be referenced by calendarID instead of folder (#636)
`data.Collection` of type event  `fullPath()` contain the calendar id
2022-08-23 17:38:30 +00:00
ashmrtn
e8e4bf0914
WIP 316 kopia upload errors (#330)
* Update to kopia with required callback

* Support structs for materializing backup details

Kopia will not allow us to pass data to it that should be passed back to
us in the `FinishedFile` callback. To work around this, create a small
thread-safe support struct that handles information about files kopia is
currently processing. Entries are removed from the set when kopia is
done with them and if no error occurred, the item's info will be added
to the BackupDetails.

* Switch to best attempt for iterating through files

Defaulting to "best-attempt" error handling where all data that didn't
result in an error is handed to kopia and then all errors encountered
are returned at the end.

* Test for uploads that have an error

Simple error reading a file. BackupDetails should not contain
information about the file that had the error (needs update to kopia
code and this code to pass). All other files should be present in kopia
and in BackupDetails.

Co-authored-by: Danny <danny@alcion.ai>
2022-08-23 08:18:29 -07:00
Danny
d7abed1406
GC: setTenants not checking *users.GetMail() (#629)
User query updated from `*user.GetMail()` -> `*user.GetPrincipalName()`. Guard checks added and added GetUsers as one of the GraphQueryFunctions
2022-08-22 14:07:57 -07:00
Danny
b573882746
GC: Accept selectors.BackupExchange exchange.Event and exchange.Contacts (#605)
GraphConnector suite expanded to support backup of Mail, Contacts, and Events. e2e testing suite expanded for all 3 applications.
2022-08-22 20:52:01 +00:00
Danny
4f5bb2b995
Exchange event info (#569)
Exchange Info feature added and verified with `mockdata` to ensure correctness
2022-08-19 13:43:26 -06:00
Danny
74a8666956
Exchange Serialization test finalized (#611)
Finalize graph connector regression tests. Needed to change the use of GC to be singular due to parallel testing.
2022-08-19 15:39:16 +00:00
Danny
9d18d50cf8
Populate function Refactor to single function (#585)
Populate functions streamlined into one function with a few additional abstractions
2022-08-19 08:19:42 -07:00
Vaibhav Kamra
8fd867745f
Onedrive item reader (#582)
## Description

This contains the following changes:
- Support functions to enumerate a users drives and the files within it.
- `driveItemReader` method that that reads a drive item
- Integration tests for the above

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #388 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-18 23:38:23 +00:00