746 Commits

Author SHA1 Message Date
ashmrtn
8fb55ec886
Implement ModTime interface in service streams (#1670)
## Description

Add ModTime to Exchange, OneDrive and SharePoint list stream items. This enables kopia-assisted incrementals for those items. Backup details still contains a complete set of information for all items in the backup regardless of if kopia uploaded data for the item or not.

Kopia-assisted incrementals does come with some caveats though. If changes are made to an item in M365 and that change does not cause the modified time reported by M365 to update, then the change will not be backed up. Currently, only marking an email as read/unread is known to hit this edge case.

This patch does not lazily fetch data from Graph API. This means that kopia may upload less data, but the same amount of data will still be pulled from Graph

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #622 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-05 18:01:38 +00:00
Danny
ef7d37e246
GC: Backup: Kiota Serialization Patch no longer required. (#1255)
## Description
Kiota serialization patch found located `exchange_data_collection.go`L 316 required a large collection of functions to make the message object serializable. Kiota upgrades since instantiation no longer require this collection of functions. 

Dead code removed from m365Transform.go

<!-- Insert PR description-->

## Type of change
- [x] 🗺️ Documentation


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-12-02 21:39:44 +00:00
ashmrtn
a8d855a9da
Manual iteration for calendar events (#1628)
## Description

Will fit better with how incremental backups are planned to work in the future as next links can be extracted.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* closes #1612 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 20:44:07 +00:00
Danny
e767bb0b77
Backup: SharePoint: List content retrieval (#1633)
## Description
Logic for List retrieval from M365 back store. Keeping separate from Collection creation logic as there are a lot of calls to retrieve all supporting relationship data from the backstore.  
Adds collection to 
## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #1475<issue>

## Test Plan

- [x]  Unit test
2022-12-02 19:31:20 +00:00
ashmrtn
6dcbc8f2a1
Set file mod time in KopiaWrapper (#1405)
## Description

Set the mod time of uploaded files to either the mod time of the item (if it has one) or the current time (if it does not have one). Also add tests to check caching in kopia works properly.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #621 

part of:
* #547 

merge after:
* #1427 
* #1430

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 16:49:27 +00:00
ashmrtn
9963e50b99
More consistent folder cleanup for OneDrive test (#1663)
## Description

Cleanup all folders instead of just some of them. Log errors instead of failing test

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1661

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 16:37:01 +00:00
ashmrtn
3eae0f969d
Contacts delta endpoint fetch (#1644)
## Description

Use delta endpoint to get contacts in each folder

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* #1612 

merge after:
* #1620 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 05:30:14 +00:00
ashmrtn
34832591ce
Use delta endpoint to fetch exchange mail items (#1620)
## Description

Use delta endpoint to fetch exchange mail items. Disables picking only some item attributes as query parameters are not implemented in this version of Graph SDK.

Switches to custom pagination iteration method so that next links and delta tokens can be extracted later on.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-02 03:57:15 +00:00
ashmrtn
af977f46ca
Fix OneDrive and Kopia Wrapper test flakes (#1662)
## Description

Minor fixes for flakey tests:

* Dedupe OneDrive folders by checking IDs. Should help remove some test flakes.
* Weaken checks on kopia snapshot manager tests as they use maps with indeterminate iteration orders

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1660 
* closes #1657 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 03:39:45 +00:00
Abin Simon
c58240ea3d
Parallelize OneDrive queries to get URL (#1607)
## Description

This improves the initial backup speed for OneDrive. OneDrive backup was mostly slow when we had a lot of tiny files. Case where we had mostly large files was pretty much the best case scenario and we were throttled by purely how fast we can get the files from MS and how fast kopia can process and upload it. But of small files, we were slowed by the loop which was taking quite a bit of time to fetch the download urls. We have now parallelized the query for getting the download URL. Under best case scenarios, I was able to speed it up to under 20s from ~4-5m starting point. That said, MS graph api still seems to throttle us and when that happen we still go back to around ~2m for worst case scenario. I've added 3 retries as some requests were failing when we continuously making many requests.

This should also take care of the issue of url expiring mentioned in https://github.com/alcionai/corso/issues/581 as we are only prefetching a few urls ahed of time.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1595
* fixes https://github.com/alcionai/corso/issues/581

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-01 06:11:11 +00:00
Abin Simon
784db9e12e
Speed up Exchange backup by parallelizing url fetch (#1608)
## Description

From rough numbers we can speedup an account with ~3000 emails, ~1000 contacts and ~1000 events from ~18m to <3m.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/362
* https://github.com/alcionai/corso/issues/1595
* https://github.com/alcionai/corso/pull/1607

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-01 04:46:24 +00:00
ashmrtn
27d0980526
Split backups by (resource owner, service) at CLI layer (#1609)
## Description

Sketch of how to split backups by (resource owner, service) for exchange. Needs more support from selectors package to be complete.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1505

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:50:49 +00:00
Vaibhav Kamra
c4333ad531
Use latest golangci-lint version (#1634)
## Description

Use latest golangci-lint version. Also updates cache flags.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1629 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:17:01 +00:00
ashmrtn
bd373bbfd4
Custom function to send get requests on delta queries (#1643)
## Description

The current version of graph SDK (v0.40.0) Corso uses does not support adding query parameters to delta requests on exchange mail folder messages or contact folder contacts. This reimplements the logic from a newer version of graph SDK so that we can add those parameters. Much of the code is lifted from the updated version of the SDK, with some combining of types that have the same fields.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 20:24:31 +00:00
ashmrtn
1d977105c0
Fix bad comparison due to timezone mismatch (#1627)
## Description

Exchange stores start/end times for events in UTC time. This fixes test failures due to comparing hours extracted from time structs directly. Without the normalization to UTC, the initial time may be different.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1625 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-29 17:51:34 +00:00
Danny
d766c30231
GC: Populate Refactor (Contacts) (#1570)
## Description

Iterator Removal from Cache Folder Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).

Stage 1: Reduce the number of error clauses in Populate() function
Stage 2: Update the M365ID retrieval process per resolver
PR introduces the `Iterate` approach used in delta functionality to populate the contact folder cache resolver. Reduces the number of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time concurrent Mailbox queries are being made (application throttling).

## Type of change
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-11-28 14:38:03 +00:00
Keepers
3f7860fd0f
retrieve all drives in each site (#1599)
## Description

Extends the site drive query to include a selector
for 'system' properties, which causes graph to
return an expanded set of drives including many
which are not normally visible to end users.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-23 23:42:30 +00:00
Keepers
d6522e01ae
filter out personal sites (#1554)
## Description

Adds a server-side filter that removes all
personal (aka: oneDrive) sites from the drive
retrieval.

The approach is not my preferred method.  I'd rather
have used a filter in the query parameters.  In fact, I
have a functioning query in postman:
`$filter=NOT contains(webUrl, 'sharepoint.com/personal/')`
But I can't seem to get the graph client query to build
when I add that to the query params.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-22 22:17:55 +00:00
Keepers
5b0549fb32
migrate sharepoint cat from file to library (#1555)
## Description

Adds a LibraryCategory in paths, and changes
the current sharepoint code to use that cat
instead of the onedrive files cat.  This is mostly to keep sharepoint library drives separate from
onedrive and sharepoint site list items.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x]  Unit test
2022-11-22 19:01:05 +00:00
Keepers
10acf0ccf6
generic drive retrieval for sharepoint (#1536)
## Description

Adapts the graph onedrive library to handle
access to drive data across both onedrive and
sharepoint services.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-22 18:05:47 +00:00
ashmrtn
86c2f2c284
OneDrive restore/backup test (#1173)
## Description

GraphConnector test for OneDrive backup and restore. Tests the following:
* restore properly builds the hierarchy for items
* items in folders that have the same name restore properly
* backup properly selects subfolders
* backup separates folders with same name in different parts of
  hierarchy
* input data == backup(restore(input data))

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-19 00:45:03 +00:00
Abin Simon
1f0338a80d
Use http client from msgraphcore (#1484)
## Description

Use http client from msgraphcore instead of the one from net/http as it has preconfigured middleware.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/581

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-11-18 12:01:12 +00:00
Keepers
07a8d13d1e
adds tenant site lookup on new graph conn (#1540)
## Description

Adds a lookup for all tenant site ids when a new
graph connector is created.  Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-18 00:31:41 +00:00
ashmrtn
9fe9cd1ce6
Use prefix matching in GraphConnector tests (#1548)
## Description

Switch to using prefix matchers for restore+backup tests as selectors now support them by default. Not using prefix matchers could eventually lead to failed tests as some tests may have subfolders that could lead to repeated items depending on how the selector is constructed. For example, a selector with scopes matching on `["inbox", "inbox/Work"]` could lead to matching `inbox/Work` twice if only prefix matching is done and GraphConnector iterates over all scopes

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-17 23:03:31 +00:00
Keepers
93ad16dc30
extract scope from qp (#1510)
## Description

Since scopes are service specific, we cannot
easily house them within the graph QueryParam
struct, unless we bloat the struct with all types.
Alternatively, we could add a generic "scope"
with parsers, much like the Selector itself.  But
really, the most simple solution is to only pass
the scope within the tree of service funcs that
use it.

## Type of change

- [x] 🐹 Refactor

## Issue(s)

* #1506

## Test Plan

- [x]  Unit test
2022-11-17 22:08:37 +00:00
Keepers
f1de0eb2b7
Adds the foundation of sharepoint data coll (#1508)
## Description

boilerplate to graph connector.  To minimize
changes, some dependencies or half-baked
solutions have been listed as TODOs for follow-
up.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-11-17 21:49:45 +00:00
Keepers
2c913f9ef7
connector data coll refactor (#1507)
A quick code movement before adding the
sharepoint datacollection production so that
we minimize graph_conn file bloat.
2022-11-17 14:04:16 -07:00
Keepers
67265a8c32
remove unused purge code (#1521)
## Description

Now that pwsh scripts do the purging, we can
get rid of the faultier graphapi exchange purge
code, along with the functions supporting it.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1520

## Test Plan

- [x] 💪 Manual
2022-11-16 22:37:25 +00:00
ashmrtn
147f96e69b
Limit container resolver recursion depth (#1524)
## Description

Now that the container resolver may not have all folders in-memory (we're relaxing the limits on when we return an error), add a recursion depth limit. Limit aligns with currently supported M365 service folder hierarchy depth limits

Exchange limitations for [mail folders](https://learn.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#mailbox-folder-limits) (search "Maximum folder hierarchy depth")
[OneDrive](https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa) limit on path length (search "file name and path lengths")

Minor refactoring of test code to make it easier to spin up tests for multiple cases

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1519

merge after
* #1523 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-16 20:41:39 +00:00
Danny
16e9f07d1e
GC. EventCalendarResolver Populate() refactor (#1518)
## Description
Iterator Removal from Event Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).  
- Stage 1: Reduce the number of error clauses in `Populate()` function
- Stage 2: Update the M365ID retrieval process per resolver

PR introduces Iterate approach used in `delta` functionality to populate the event calendar resolver.  Reduces the amount of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time `concurrent` Mailbox queries are being made (application throttling).
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐹 Trivial/Minor

## Issue(s)
Related to #1486 

## Test Plan

- [x]  Unit test
2022-11-16 16:17:59 +00:00
Vaibhav Kamra
b7330b2ba6
Reduce CI test times (#1532)
## Description

The bulk of the time being spent in the `Test-Suite` job was compiling the test binaries.
This is related to the size of the MS graph library.

The impact of this was even more obvious because the Go build cache didn't seem to be getting used
in our CI tests.

The issue was that the cache is being used (and primed) in multiple places and if it happens to
get populated during a build step that doesn't compile the test binaries - then the cache would
never get updated with those and we would keep building from scratch.

The fix is to add a cache suffix so that the testsuite cache and lint cache use different locations.

Going forward - another improvement would be to create a single cache and populate it up-front
in the workflow with all Go artifacts (i.e. compile src and test up front).

This change reduces `Test-Suite` time from ~32 minutes to ~14 minutes

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #790 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-16 07:29:21 +00:00
ashmrtn
caf8422558
Add container resolver tests back (#1523)
## Description

Mistakenly removed when renaming files.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1522 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 22:46:51 +00:00
Abin Simon
110684d5f7
Add size to all objects (#1491)
## Description
Add size to all objects. Not adding them to the output of details view.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 17:34:25 +05:30
Abin Simon
825cd6f21b
Add created and modified time to all (#1452)
## Description

Adds Created and Modified time to all objects. Need to go after
https://github.com/alcionai/corso/pull/1446 as it is built on top of
that. Keeping in draft until then.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2022-11-15 10:24:06 +05:30
Keepers
11371f6e94
add sharepoint to path (#1465)
Adds the sharepoint service to /pkg/path.  Currently uses the
"Files" category for its category type, which is just a placeholder
for kicking off development.

Additionally, uncomments selector tests that were dependent
upon the path service declaration.
2022-11-14 15:35:57 -07:00
Danny
765fd6222b
GC: Backup: Sharepoint: Collection Implementation (#1477)
## Description
Creates test package for SharePoint List Collection. Verifies Item and ListInfo for collection

<!-- Insert PR description-->

## Type of change

- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #1474<issue>

## Test Plan

- [x]  Unit testt: E2E
2022-11-14 17:51:44 +00:00
Keepers
20795d3b56
applies folder matching behavior to cli input (#1443)
## Description

Allows users to specify whether they want to
select folders by prefix or search behavior.
Search/contains behavior is the default case,
with prefix being an optional deviation if the
folder input is prepended with a '/' character.

Also, propagates the PrefixMatch setting to
all integration tests that rely on selecting only
the default folder in exchange.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-08 23:10:40 +00:00
Keepers
b20e7af533
use path filters in folder selectors (#1429)
## Description

For folder-level scopes (ie, scopes that compare
folder-hierarchy path segments),  this change
replaces the standard "equals" and "prefix"
string comparators with the new PathContains
and PathPrefix comparators.

Next change is to interpret user inputs in the
cli to determine whether the comparator should
use contains or prefix behavior.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-11-08 22:27:09 +00:00
Danny
bd285e7a62
GC: Restore: Regression Mocks (#1461)
Mock package data added for Regression testing.
2022-11-07 14:43:21 -05:00
Danny
70d9c2ed15
Sharepoint Feature: details.Sharepoint implementation (#1454)
## Description
Creates Sharepoint Info within the details package. SharepointInfo constructor and test package included
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Test Plan

- [x]  Unit test
2022-11-07 13:46:31 +00:00
Abin Simon
5089f9d949
Add EndTime to Exchange events (#1446)
## Description

Add EndTime to Exchange events. Also fixes a testcase around start time.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-07 09:28:27 +00:00
Danny
ecde9cc242
GC: Backup: Sharepoint: Mock Support (#1456)
Creates Sharepoint.List mock object. Tests support the creation and hydration of objects.
2022-11-04 16:57:28 -04:00
Keepers
2d66dcbd7a
replace golang purge script with powershell script (#1407)
## Description

Three primary changes:
1. move the list of load test users into a secret in the github test domain.
2. purge all data that was generated for users in the load test
3. fix all `m356` typos.

## Type of change

- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

* #1266

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-11-04 15:41:36 +00:00
Danny
c4632cb152
GC SharePoint Collection Framework (#1444)
## Description
Boilerplate for SharePoint collection 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
2022-11-04 14:11:10 +00:00
Abin Simon
935fe5cbec
Store/Retrieve owner info for OneDrive files (#1436)
## Description

Store and retrieve owner info for onedrive files.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-04 07:01:45 +00:00
ashmrtn
524c3ba853
Consolidate container resolver code (#1365)
## Description

As the exchange container resolvers are pretty close implementation-wise, consolidate the generic code. Also move some of the tests around to better fit what file has the code being tested

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1362 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-02 23:06:52 +00:00
Danny
0e29655645
GC: Container Resolver tests consolidation (#1363)
## Description
Grouping of event and contact folder caches to the same file. MailFolderCache left in a different folder until it can be decided where the tests before
<!-- Insert PR description-->

## Type of change

- [x] 🤖 Test


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
- [ ] 💚 E2E
2022-10-31 16:56:31 +00:00
Keepers
2a2af20f94
revert DPanic log to Error in support pkg (#1340)
## Description

The panic in development mode is interfering with test scrutiny and other general behavior.  Errors will be reported naturally without the need for this additional panic.

## Type of change

- [x] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-28 13:32:47 +00:00
Danny
6ddf02cf50
GC: Restore: MockEvent w/ Attachment (#1345)
## Description
Adds feature to be able to create mock events with attachments
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #1005 <issue>

## Test Plan
Testing:
```
internal/connector/exchange$ go test -v . --run TestExchangeServiceSuite/TestRestoreExchangeObject/Test_Event_with_Attachment
```
- [x]  Unit test
2022-10-27 18:54:29 +00:00
Keepers
953f038395
Add AWS X-Ray support (#1111)
## Description

Add aws xray observability tooling, plus ways to quickly and
easily run the tools/load tests.  Next steps are 1/ running the
daemon container alongside the load tests within github, and
2/ creating client credentials in aws's xray service so that
gathered trace info is propagated from the daemon proxy
to the xray service.

## Type of change

- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-27 17:55:50 +00:00