135 Commits

Author SHA1 Message Date
Keepers
07a8d13d1e
adds tenant site lookup on new graph conn (#1540)
## Description

Adds a lookup for all tenant site ids when a new
graph connector is created.  Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-18 00:31:41 +00:00
Keepers
93ad16dc30
extract scope from qp (#1510)
## Description

Since scopes are service specific, we cannot
easily house them within the graph QueryParam
struct, unless we bloat the struct with all types.
Alternatively, we could add a generic "scope"
with parsers, much like the Selector itself.  But
really, the most simple solution is to only pass
the scope within the tree of service funcs that
use it.

## Type of change

- [x] 🐹 Refactor

## Issue(s)

* #1506

## Test Plan

- [x]  Unit test
2022-11-17 22:08:37 +00:00
Keepers
67265a8c32
remove unused purge code (#1521)
## Description

Now that pwsh scripts do the purging, we can
get rid of the faultier graphapi exchange purge
code, along with the functions supporting it.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1520

## Test Plan

- [x] 💪 Manual
2022-11-16 22:37:25 +00:00
ashmrtn
147f96e69b
Limit container resolver recursion depth (#1524)
## Description

Now that the container resolver may not have all folders in-memory (we're relaxing the limits on when we return an error), add a recursion depth limit. Limit aligns with currently supported M365 service folder hierarchy depth limits

Exchange limitations for [mail folders](https://learn.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#mailbox-folder-limits) (search "Maximum folder hierarchy depth")
[OneDrive](https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa) limit on path length (search "file name and path lengths")

Minor refactoring of test code to make it easier to spin up tests for multiple cases

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1519

merge after
* #1523 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-16 20:41:39 +00:00
Danny
16e9f07d1e
GC. EventCalendarResolver Populate() refactor (#1518)
## Description
Iterator Removal from Event Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).  
- Stage 1: Reduce the number of error clauses in `Populate()` function
- Stage 2: Update the M365ID retrieval process per resolver

PR introduces Iterate approach used in `delta` functionality to populate the event calendar resolver.  Reduces the amount of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time `concurrent` Mailbox queries are being made (application throttling).
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐹 Trivial/Minor

## Issue(s)
Related to #1486 

## Test Plan

- [x]  Unit test
2022-11-16 16:17:59 +00:00
ashmrtn
caf8422558
Add container resolver tests back (#1523)
## Description

Mistakenly removed when renaming files.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1522 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 22:46:51 +00:00
Abin Simon
110684d5f7
Add size to all objects (#1491)
## Description
Add size to all objects. Not adding them to the output of details view.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 17:34:25 +05:30
Abin Simon
825cd6f21b
Add created and modified time to all (#1452)
## Description

Adds Created and Modified time to all objects. Need to go after
https://github.com/alcionai/corso/pull/1446 as it is built on top of
that. Keeping in draft until then.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2022-11-15 10:24:06 +05:30
Keepers
20795d3b56
applies folder matching behavior to cli input (#1443)
## Description

Allows users to specify whether they want to
select folders by prefix or search behavior.
Search/contains behavior is the default case,
with prefix being an optional deviation if the
folder input is prepended with a '/' character.

Also, propagates the PrefixMatch setting to
all integration tests that rely on selecting only
the default folder in exchange.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-08 23:10:40 +00:00
Keepers
b20e7af533
use path filters in folder selectors (#1429)
## Description

For folder-level scopes (ie, scopes that compare
folder-hierarchy path segments),  this change
replaces the standard "equals" and "prefix"
string comparators with the new PathContains
and PathPrefix comparators.

Next change is to interpret user inputs in the
cli to determine whether the comparator should
use contains or prefix behavior.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-11-08 22:27:09 +00:00
Abin Simon
5089f9d949
Add EndTime to Exchange events (#1446)
## Description

Add EndTime to Exchange events. Also fixes a testcase around start time.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-07 09:28:27 +00:00
ashmrtn
524c3ba853
Consolidate container resolver code (#1365)
## Description

As the exchange container resolvers are pretty close implementation-wise, consolidate the generic code. Also move some of the tests around to better fit what file has the code being tested

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1362 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-02 23:06:52 +00:00
Danny
0e29655645
GC: Container Resolver tests consolidation (#1363)
## Description
Grouping of event and contact folder caches to the same file. MailFolderCache left in a different folder until it can be decided where the tests before
<!-- Insert PR description-->

## Type of change

- [x] 🤖 Test


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
- [ ] 💚 E2E
2022-10-31 16:56:31 +00:00
Danny
6ddf02cf50
GC: Restore: MockEvent w/ Attachment (#1345)
## Description
Adds feature to be able to create mock events with attachments
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #1005 <issue>

## Test Plan
Testing:
```
internal/connector/exchange$ go test -v . --run TestExchangeServiceSuite/TestRestoreExchangeObject/Test_Event_with_Attachment
```
- [x]  Unit test
2022-10-27 18:54:29 +00:00
Keepers
953f038395
Add AWS X-Ray support (#1111)
## Description

Add aws xray observability tooling, plus ways to quickly and
easily run the tools/load tests.  Next steps are 1/ running the
daemon container alongside the load tests within github, and
2/ creating client credentials in aws's xray service so that
gathered trace info is propagated from the daemon proxy
to the xray service.

## Type of change

- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-27 17:55:50 +00:00
Keepers
77c703cee9
add scope category to path cat type func (#1344)
## Description

Builds a scope.Category() -> path.CategoryType trans- former func into scopes to standardize the relationship between the two properties.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1133

## Test Plan

- [x]  Unit test
2022-10-26 21:43:35 +00:00
Danny
39b47cc123
Gc mock package (#1313)
## Description
Command line utility that allows for the printing of mockData based on the m365ID. Supports:
- `exchange.Mail` 
- `exchange.Contacts`
- `exchange.Events`

<!-- Insert PR description-->

## Type of change

- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

Related to:
* #1255<issue>
* #999 
* #705
2022-10-25 18:50:39 +00:00
Danny
597e689417
GC: Update Error Messages (#1252)
## Description

<!-- Insert PR description-->

## Type of change

- [x] 🗺️ Documentation

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*Related  to #1225<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2022-10-21 22:11:29 +00:00
Vaibhav Kamra
6047805d6c
Switch test user accounts (#1291)
## Description

Switches the test accounts used in CI since the previously used account was hitting slowness due to
items in Deleted Folders.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1191 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-10-21 21:33:10 +00:00
Danny
63e1cec71c
GC: Remove DeadCode (#1253)
## Description
Code removal from GraphConnector
`AddItemsToCollection` removed from graph_connector.go
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-20 17:54:36 +00:00
Danny
bb7e48a82e
GC: Use of graph.ContainerResolver for directory structure and data retrieval (#1134)
## Description

`graph.ContainerResolver` has the capacity to keep the directory structure of the m365 objects that are helpful within a user's account. Leveraging this abstraction allows for a better flow of data from M365 into storage. 

## Type of change


- [x]  : Optimization

## Issue(s)

* closes #1125<issue>
* closes #1122
## Test Plan


- [x]  Unit test
2022-10-20 12:49:09 +00:00
Keepers
6055d8c16a
adds collection-level progress display (#1187)
## Description

Adds a progress spinner tracking the collection-level work on an incremental basis.  Useful for tracking collection progress when individual files may be too quickly resolved to visibly show progress bars.

In the future, we could combine the two so that the current item in progress always displays below the collection tracker.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1112

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-20 01:12:47 +00:00
Danny
5cd2583466
GC: Purge: Updates to use graph.ContainerResolver (#1190)
## Description
purge.go to use the ContainerResolvers to collect containers from the Graph Connector.

NOTE: CollectFolders in service_query.go unchanged. `CollectFolderrs()` is stubbed for another PR. This will reduce the amount of line changes to be used in the future. 
<!-- Insert PR description-->

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #1189<issue>

## Test Plan

- [x]  Unit test
2022-10-19 14:08:03 +00:00
Keepers
5d2bc1eb4e
add event production to the factory script (#1200)
## Description

Adds the capacity to programmatically generate events in the factory script.  Status is currently functional in code completion, but experiencing unexpected results.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-18 23:54:37 +00:00
ashmrtn
28c3bbcf7a
Remove (old) unused code for fetching messages (#1168)
## Description

Just code cleanup

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1156

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-18 23:09:58 +00:00
Danny
e79051318c
GC: Deprecated functions removed. (#1179)
Removal of deprecated functions from code base. 
-  GetRestoreContainer no longer used for restore workflow
- GetContainerID no longer used to find the M365ID of container objects. 
## Description

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->

- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-18 01:17:13 +00:00
Keepers
48cb751ee0
lookup contact folder as fallback (#1174)
## Description

In the event that a user only has a primary contact
folder, and no subfolders, the contact folder legacy
iter needs to fall back to checking for the contacts
default folder in an isolated query, because that
folder isn't provided as part of the contacts folders
get request.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1113

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-18 00:17:13 +00:00
Keepers
a32f86e40d
migrate naming of azure env vars (#1176)
## Description

migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.

⚠️  Your local environment will need to change in the following ways:  ⚠️ 
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #558

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-10-17 20:47:04 +00:00
Keepers
7d72cd12a4
unskip event restores (#1019)
## Type of change

- [x] 🤖 Test

## Issue(s)

* #1018

## Test Plan

- [x] 💚 E2E
2022-10-14 20:38:33 +00:00
Keepers
f737f58a0f
generate emails in factory script (#1162)
## Description

Adds scripted production of mock emails for building out
large/rolling datasets for load testing.

## Type of change

- [x] 🌻 Feature
- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
2022-10-14 18:48:00 +00:00
Danny
4a29d22216
GC: Restore: Directory Hierarchy Feature for Exchange (#1053)
## Description
Feature to add the folder hierarchy for folders when restored.
This required an overhaul of the `graph.ContainerResolver` interfaces:
- MailFolderCache
- ContactFolderCache
- ~EventFolderCache (placed in a separate PR)~ https://github.com/alcionai/corso/pull/1101

Restore Pipeline changed to separate the caching / container creation process from the rest of the restore pipeline.


## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

* closes #1046 
* #1004 
* closes #1091
* closes #1098 
* closes #1097 
* closes #1096
* closes #1095
* closes #991
* closes #895
* closes #798 

## Test Plan
- [x]  Unit test
2022-10-14 13:14:33 +00:00
Vaibhav Kamra
f1f6c06ba0
Use an upload session for large attachments (#1148)
## Description

Use an upload session and chunked writes for large attachments.

This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and 
leverages that for the upload.

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1115 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-13 03:35:40 +00:00
ashmrtn
41319c117f
Allow selective mail backup and change how mail is fetched from Graph (#1123)
* Move interfaces to common location

Upcoming PRs are using these interfaces across packages. Move them to a
common location so multiple packages can use them without import cycles
etc.

* Allow adding newly created folders to the cache (#1131)

* New function to add folders to cache

Allow adding new folders to the cache. Automatically cache the paths for
the new folders. Also add the new function to the interface.

* Reuse the AddToCache function during population

* Wire up ability to back up a single subfolder of mail (#1132)

* Expand cache to return items in it

Required to allow matching an item's path to a selector as the selector
will not provide which paths it matches on easily.

* Function to get collections from cached folders

Returned collections match any matchers given for the folders

* Thread resolver through iterator functions

Allow the folder resolver to be used in all iterator functions. The
resolver will be tied to the current category and user.

* Choose between using resolver and making queries

Allow either using the resolver to get folders with matching names or
using queries to get them.

* Wire up resolver at entry point

Create a resolver instance for each user/category of data being
backedup.

* Preparation for changing how mail enumeration is done (#1157)

* Step towards redoing mail fetching

Pull out old way to get data into a new function and setup some helper
functions etc.

* Switch to pulling mail items folder by folder (#1158)

* Function to pull mail items given collections

Given a set of collections and IDs for those collections pull the mail
items for each collection.

* Create helper function to fetch mail

New helper function to fetch mail items. This goes through each folder
and gets the items for them individually.

* Wire up new way to fetch mail

Leaves fetch logic for other data types undisturbed.

* Tests for new mail fetching logic

Remove tests that were previously in iterators_test.go and move them to
graph_connector_test.go. These tests only had to do with mail logic.
Tests that handled all data types in iterators_test.go have been updated
to skip mail now.
2022-10-12 16:44:24 -07:00
Keepers
0b65b20200
attach logger to test ctx, use flusher (#1083)
## Description

Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context.  The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton.  But in order for the logs to show up
within the test, they need to get flushed.

See my comments for focus on non-chaff changes.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-10 21:14:10 +00:00
Danny
211c248f35
GC: Iterative Function: Bug fix (#1110)
## Description
Contact Iterative sequence tests changed to reflect `exchange.Backup` functions used within `SetupExchangeCollectionVars()`. The previous bug was caused by misalignment from the old test configuration and the current implementation.
<!-- Insert PR description-->

## Type of change

- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #1107<issue>

## Test Plan

- [x]  Unit test
2022-10-10 18:22:31 +00:00
Danny
a5140d20b2
GC: Backup: CI: Stabilize Iterator test (#1079)
## Description
CI test timing during times of high activity. Reduced test suite to cover Inbox and adjusted outputs accordingly. 
<!-- Insert PR description-->

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #1078<issue>

## Test Plan

- [x]  Unit test
2022-10-07 18:55:17 +00:00
Danny
0724db938c
GC: CI: Streamline Tests and Reduction (#1082)
* Reduction of duplicated tests.  Removal of tests that are checked in other functions.
2022-10-07 13:01:37 -04:00
Keepers
89668ed164
describe time value standards, validate time flags (#1066)
## Description

Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.

Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.

## Type of change

- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test

## Issue(s)

* #943
* #924

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-10-06 17:46:21 +00:00
Danny
7843787cd3
GC: Restore: Mail: Create Subfolder Feature (#1055)
## Description
The feature allows for the subFolders to be created. Test Suite Expanded. 
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*Closes Issue #1054<issue>

## Test Plan

- [x]  Unit test
2022-10-05 22:33:41 +00:00
Keepers
f4556288a5
add iteminfo to restore details (#1041)
## Description

Extends the restore details with itemInfo from restored items.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #977

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-05 21:19:21 +00:00
Danny
5bcdaef769
GC: Interface: Cache Refactor (#1043)
## Description
Code changed to support caching on the default folder of `exchange.Mail` as well as an independent node within the Inbox.
<!-- Insert PR description-->

## Type of change

- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to Issue #1004<issue>

## Test Plan
- [x]  Unit test
2022-10-05 20:04:50 +00:00
ashmrtn
3f0951dea2
Higher resolution on test restore folder names (#1040)
## Description

\~Only affects folders created by tests\~
Create and use helper function that sets the restore destination folders to have microsecond granularity. This is to avoid naming collisions with second granularity (of which I've observed at least once)

Switches all tests for all services/categories to use OneDrive DateTime format+microseconds

Manually verified that a manually created folder with the given timestamp format was removed by the CI purge script

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1036

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-04 23:52:13 +00:00
Keepers
d0560500d2
add deets to gc restore ops (#1029)
## Description

In order for a restore operation to accurately print
out the list of items that were restored, it needs to
track item details within the GC restoration like is
done within kopia in backup.   These details will
not be stored in a modelStore.  But they will get
printed out on the CLI for end users.

Part 1 of multiple.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #977

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-04 17:56:34 +00:00
Danny
c6a9d2feb6
GC Status Progress Detail Extension (#1001)
## Description
Feature to show the number of bytes and additional details for backup / restore progress from GraphConncector
Full description [here](https://www.notion.so/alcion/Corso-Testing-Notes-b9867ac719d8459d8b46dbc7b07b33e0#da3869278b434b0398e7d245554b609b)
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #559<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual

Feature Output changed to:

```go
graph_connector_test.go:137: Action: Backup performed on 5 of 5 objects (227.63KB) within 1 directories. Downloaded from Inbox
        
graph_connector_test.go:137: Action: Backup performed on 8 of 8 objects (231.28KB) within 2 directories. Downloaded from Inbox, Contacts

graph_connector_test.go:137: Action: Backup performed on 23 of 23 objects (309.36KB) within 3 directories. Downloaded from Inbox, Contacts, Calendar
```
2022-10-03 23:25:26 +00:00
Danny
d60a771024
GC: TWO Message attachments (#1014)
## Description
Introduces mock testing to ensure that objects with two or more attachments are restored properly.

Minor change to restore pipeline for failure. attachments are restored separately in the event that one of the attachments is corrupted. 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1007<issue>

## Test Plan

- [x]  Unit test
2022-10-03 20:59:03 +00:00
Danny
a31a17ce1f
GC: Extend mockable to include attachments for mail (#990)
## Description
Mockable `exchange.Mail.Messages` extended to include direct attachments and oneDrive attachments. 

NOTE: There are changes to support as it was recognized that the lack of these variables during cloning caused Issue #978 for direct attachments. However, it does not solve it for OneDrive. 

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix


## Issue(s)


**closes** #988<issue>

## Test Plan

- [x]  Unit test
2022-09-30 14:08:17 +00:00
ashmrtn
4e86cd2f84
Fix bug fetching contact folders with filters (#986)
## Description

Stop fetching all folders for every new folder seen, erasing previous results. Regression test to come in a later patch as it requires some extra infrastructure

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #982 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 23:20:50 +00:00
Danny
114fec6059
GC: Restore: Simplified Event Restore (#981)
## Description
Implementation of simplified Restore based on the following [spec](https://www.notion.so/alcion/Event-restore-semantics-061aee5288244629b1c53337e4dea306#6e974540c8804c4fa832218675534e1c)

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #954<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2022-09-29 22:01:12 +00:00
Keepers
d4390ac5ea
add diagnostics to load_test (#983)
## Description

Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on.  Therefore,
just starting with something basic.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-09-29 16:47:09 +00:00
Danny
652bb9cd4a
GC: Restore: Contact: Restore to single Folder (#972)
## Description

The feature ensures that this Flat style directory only creates a single folder and restores all the contacts to the same folder during the restore process. 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #880<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-28 14:16:08 +00:00