175 Commits

Author SHA1 Message Date
Keepers
402e6b5139
handle invalid or expired deltas (#1882)
## Description

If a delta token expires or is otherwise invalid,
the backup should fall back to the same
behavior as if the collection were new.  It must
collect the full delta of information, and create
a collection with New state.  If the previous
container moved between the two deltas,
it should be marked for deletion with a tomb-
stone collection.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1780

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-22 20:54:00 +00:00
Keepers
5b6b60de60
enable incrementals with feature flags (#1903)
## Description

Allows the usage of incrementals via feature-
flag controls.  Feature flags can be enabled
on a per-flag basis in either the cli (the flag
is hidden from users, normally) or through the
sdk by directly toggling the flag property
in the control.Options.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x] 💪 Manual
2022-12-21 21:31:48 +00:00
Vaibhav Kamra
a45aeda4a2
Update folder size and modified time in details (#1881)
## Description

Caches folder info added during details construction in the details builder and keeps the size/modified time
updated as newer items are added.

As part of this, this PR refactors the details package to separate out building `details.Details` from the in-memory representation and model.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-21 19:17:24 +00:00
Keepers
752ff20c6b
require resource owner set on selector create (#1887)
## Description

selector creation now includes a parameter for
a slice of resource owners (users or sites).  This
is step one in migrating resource owner lists out
of scopes and into the selector.  next step is to
have the selector utilize the primary list instead
of the per-scope list.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-21 18:38:41 +00:00
Keepers
a13f1cc3a6
remove common maps, add golang maps (#1885)
## Description

replaces the common/maps.go code with the
golang experimental maps package. The
maps package provides standard api
such as Keys, Values, Copy, and Clone.

https://pkg.go.dev/golang.org/x/exp/maps

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Test Plan

- [x]  Unit test
2022-12-21 18:08:26 +00:00
Keepers
ce7fb30a7f
move exchange data collections to exchange (#1818)
## Description

moves the DataCollections producer out of
collections and into exchange, along with the
integration tests. The only changes are the
code shuffles, passing down required values,
and the unexporting of funcs that were only
exported for the old design.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1727

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-20 22:46:06 +00:00
Keepers
2b0ccdc8de
require complete previous data for increment (#1869)
## Description

If an exchange collection only has previous
delta info, or only previous path info, and not
the other, remove that entry from the parsed
metadata map, causing a full backup of the
containere.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1804

## Test Plan

- [x]  Unit test
2022-12-20 01:31:30 +00:00
ashmrtn
fbbf6aa84c
Fetch isRead attribute in email delta queries (#1867)
## Description

Updating the isRead flag in M365 does not cause the mod time of the item to be updated. This means delta queries will not return items marked as [un]read when isRead is not one of the attributes requested in the $select clause.

Explicitly fetch isRead so that we get those items. Further changes will be required to ensure the mod time of the item is something that will not cause kopia-assisted dedupe to skip the item when only the isRead attribute has been changed.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1866 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-19 23:12:25 +00:00
Danny
20bfaee8db
GC: MSGRAPH Linked Library Upgrades (#1861)
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1856 <issue>

## Test Plan


- [x]  Unit test
2022-12-19 22:19:18 +00:00
Keepers
c8ad2e03ce
produce coll state and item deletion (#1815)
## Description

Fulfills the data collection interfaces to provide item deleted
flags and collection state enums.

A note on the additional data consumption: I'm making an assumption
that the @removed property will appear in that map.  This is awaiting
testing for verification, which I'll get into as a follow-up step.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1727

## Test Plan

- [x]  In a follow-up PR
2022-12-19 20:16:51 +00:00
ashmrtn
7a5a8c077e
Use leaf test instance for assertions (#1848)
## Description

For tests that run subtests, make sure to use the subtest instance of testing.T so that the error trace in the output is easier to read.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1846 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-17 04:27:30 +00:00
ashmrtn
715e436dd9
Do not return an error if folder was deleted (#1849)
## Description

When backing up Exchange, don't return an error if the folder/calendar we're trying to fetch item IDs for has been deleted.

Error codes pulled with graph explorer API

Manually tested with an Exchange mail backup

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1846 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-17 00:06:44 +00:00
Keepers
6c72eefdae
deduplicate cache folders (#1814)
## Description

These two structures are effectively identical.
This refactor deferrs to the graph version since
services may still wrap that for more specific
behavior as needed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1727

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 23:43:00 +00:00
Keepers
99b0f51980
consume prevPath on next backup (#1802)
## Description

Parses the previousPaths metadata collections
along with deltas, and hands paths down to
exchange backup collection producers.  Does
not yet scrutinize previous/current path diffs.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1726

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 19:59:34 +00:00
Keepers
703acbdcf7
produce previousPaths metadata (#1799)
## Description

Adds an additional metadata collection: a folder
id to path string mapping.  This collection is
created on backup, and retrieved along with
the delta metadata on the next backup, but is
not yet parsed or utilzed downstream.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1726

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 18:09:25 +00:00
Danny
0d5043aa1f
SharePoint List selector Expansion (#1786)
## Description
Initial changes to support SharePoint Lists being chosen for Backup Operations. 
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

* close #1785<issue>

## Test Plan

- [x]  Unit test
2022-12-16 13:24:15 +00:00
Vaibhav Kamra
debe74a113
Add an updated flag to backup details (#1813)
## Description

Adds a flag in backup details that indicates whether the item in that backup is new/updated.

Currently always set to `true` but once we implement #1800 - we will set this to false for existing items from
base snapshots.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-15 04:19:35 +00:00
Keepers
e2775aeb95
Refactor service failfast (#1789)
## Description

Configuration and attenion to the graphService
failFast is haphazard and has shared ownership.
This change removes that property from the
service, along with the ErrPolicy func, in favor of passing around a control.Options struct.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
* #302

## Test Plan

- [x]  Unit test
2022-12-13 23:06:27 +00:00
Keepers
414d2a490f
consume old deltas on exchange mail backup (#1768)
## Description

When backing up exchange data, parse the
metadata collection of delta urls from prior runs
(if any exist) and pass those tokens along to the
fetch functions for re-use.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-13 19:03:33 +00:00
Keepers
fc5f42545f
rename graph.Service to Servicer (#1787)
## Description

`graph.Service -> graph.Servicer`,  no other changes.

More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-13 17:37:03 +00:00
Keepers
edd0708285
aggregate categories from selector (#1742)
## Description

Adds a PathCategories() func to selectors which returns
all the path categories exhibited by the scopes within the
selector.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-09 18:57:06 +00:00
Keepers
ca5f197067
export ownerCats and fetchPrevSnap (#1741)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-09 01:34:20 +00:00
Danny
45e535c9b8
GC: Go.Mod update (#1479)
## Description
Versioning upgraded for GC
Packages updated:
- [x] github.com/microsoftgraph/msgraph-sdk-go v0.41.0 => v0.48.0
- [x] github.com/microsoft/kiota-abstractions-go v0.14.0
- [x] github.com/microsoft/kiota-authentication-azure-go v0.5.0
- [x] github.com/microsoft/kiota-http-go v0.9.0
- [x] github.com/microsoft/kiota-serialization-json-go v0.7.2
- [x] github.com/microsoftgraph/msgraph-sdk-go-core v0.30.1
- [x] github.com/stretchr/testify v1.8.1
- [x] github.com/cjlapao/common-go v0.0.29 => v0.0.35

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
- Update library change. AdditionalData no longer contains `nextLink` for iteration.  See commit [e7c75ca](e7c75ca1cc) for full details. 

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1478<issue>

## Test Plan

- [x]  Unit test
2022-12-08 21:54:22 +00:00
ashmrtn
e15d86e82f
Persist Exchange delta tokens (#1722)
## Description

Begin persisting Exchange delta tokens for data categories that support delta queries. Tokens are stored in a single file as a `map[M365 container ID]token` where both the container ID and token are of type `string`. The file is located in the kopia snapshot that has all the other backup data at the path `tenant-id/{service}Metadata/user/category/delta`. No information about the delta token file is stored in backup details.

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1685 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-08 18:01:00 +00:00
ashmrtn
09c48c1ec9
Expand Stream and Collection interfaces for delta token-based incrementals (#1710)
## Description

Add functions to Collection and Stream interfaces that allow for getting
more information about the difference between the previous backup and
the currently in-progress one. These will allow delta token-based
incremental backups to determine how the state has evolved.

Current code does not use these functions and return values for them
are "default" values that should result in full backups even if
KopiaWrapper is updated to start checking the values and GraphConnector
still pulls all items

These functions are not used during restore and can return "default"
values

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1700 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-07 22:20:59 +00:00
Vaibhav Kamra
cdc92b3929
Skip creating inline reference attachments (#1606)
## Description

This fixes #999 by skipping restore of reference attachments that are marked inline.

The attachment links (e.g. OneDrive links) are part of the message body and do not need to be
recreated.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #999 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-07 05:54:30 +00:00
ashmrtn
db020305f6
Return delta tokens from fetch functions (#1717)
## Description

When fetching Exchange items, return the resulting delta token from the query so it can be reused later.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1685 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-06 22:12:08 +00:00
ashmrtn
55a03af95f
Return interface values instead of concrete type (#1696)
## Description

Having a map of interface values instead of concrete structs allows adding collection implementations via different structs. This will be useful to inject a new collection that contains metadata information like delta links.

Also refactors the code for creating collections in Exchange so that it follows golang patterns of exiting early a bit better.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1685

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-06 18:10:36 +00:00
Vaibhav Kamra
5533ada9be
Refactor user query and return user display name (#1679)
## Description

Move user discovery into a `discovery` package. Also use that to return user display name from `m365.Users()` call.

This will also allow us to remove `Users` from the GC struct going forward.

If this pattern works - will follow up with a similar change for `Sites()`.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-06 01:45:51 +00:00
ashmrtn
858aa13f2a
Create helper function for getting data from map (#1694)
## Description

Helper function only deals with string values in the map. Returns empty string if the requested item was not present.

Will also be useful for retrieving delta tokens in future patches.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1685 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-05 22:09:57 +00:00
ashmrtn
8fb55ec886
Implement ModTime interface in service streams (#1670)
## Description

Add ModTime to Exchange, OneDrive and SharePoint list stream items. This enables kopia-assisted incrementals for those items. Backup details still contains a complete set of information for all items in the backup regardless of if kopia uploaded data for the item or not.

Kopia-assisted incrementals does come with some caveats though. If changes are made to an item in M365 and that change does not cause the modified time reported by M365 to update, then the change will not be backed up. Currently, only marking an email as read/unread is known to hit this edge case.

This patch does not lazily fetch data from Graph API. This means that kopia may upload less data, but the same amount of data will still be pulled from Graph

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #622 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-05 18:01:38 +00:00
Danny
ef7d37e246
GC: Backup: Kiota Serialization Patch no longer required. (#1255)
## Description
Kiota serialization patch found located `exchange_data_collection.go`L 316 required a large collection of functions to make the message object serializable. Kiota upgrades since instantiation no longer require this collection of functions. 

Dead code removed from m365Transform.go

<!-- Insert PR description-->

## Type of change
- [x] 🗺️ Documentation


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-12-02 21:39:44 +00:00
ashmrtn
a8d855a9da
Manual iteration for calendar events (#1628)
## Description

Will fit better with how incremental backups are planned to work in the future as next links can be extracted.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* closes #1612 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 20:44:07 +00:00
ashmrtn
3eae0f969d
Contacts delta endpoint fetch (#1644)
## Description

Use delta endpoint to get contacts in each folder

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* #1612 

merge after:
* #1620 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 05:30:14 +00:00
ashmrtn
34832591ce
Use delta endpoint to fetch exchange mail items (#1620)
## Description

Use delta endpoint to fetch exchange mail items. Disables picking only some item attributes as query parameters are not implemented in this version of Graph SDK.

Switches to custom pagination iteration method so that next links and delta tokens can be extracted later on.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-02 03:57:15 +00:00
Abin Simon
784db9e12e
Speed up Exchange backup by parallelizing url fetch (#1608)
## Description

From rough numbers we can speedup an account with ~3000 emails, ~1000 contacts and ~1000 events from ~18m to <3m.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/362
* https://github.com/alcionai/corso/issues/1595
* https://github.com/alcionai/corso/pull/1607

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-01 04:46:24 +00:00
Vaibhav Kamra
c4333ad531
Use latest golangci-lint version (#1634)
## Description

Use latest golangci-lint version. Also updates cache flags.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1629 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:17:01 +00:00
ashmrtn
bd373bbfd4
Custom function to send get requests on delta queries (#1643)
## Description

The current version of graph SDK (v0.40.0) Corso uses does not support adding query parameters to delta requests on exchange mail folder messages or contact folder contacts. This reimplements the logic from a newer version of graph SDK so that we can add those parameters. Much of the code is lifted from the updated version of the SDK, with some combining of types that have the same fields.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 20:24:31 +00:00
ashmrtn
1d977105c0
Fix bad comparison due to timezone mismatch (#1627)
## Description

Exchange stores start/end times for events in UTC time. This fixes test failures due to comparing hours extracted from time structs directly. Without the normalization to UTC, the initial time may be different.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1625 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-29 17:51:34 +00:00
Danny
d766c30231
GC: Populate Refactor (Contacts) (#1570)
## Description

Iterator Removal from Cache Folder Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).

Stage 1: Reduce the number of error clauses in Populate() function
Stage 2: Update the M365ID retrieval process per resolver
PR introduces the `Iterate` approach used in delta functionality to populate the contact folder cache resolver. Reduces the number of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time concurrent Mailbox queries are being made (application throttling).

## Type of change
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-11-28 14:38:03 +00:00
Keepers
07a8d13d1e
adds tenant site lookup on new graph conn (#1540)
## Description

Adds a lookup for all tenant site ids when a new
graph connector is created.  Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-18 00:31:41 +00:00
Keepers
93ad16dc30
extract scope from qp (#1510)
## Description

Since scopes are service specific, we cannot
easily house them within the graph QueryParam
struct, unless we bloat the struct with all types.
Alternatively, we could add a generic "scope"
with parsers, much like the Selector itself.  But
really, the most simple solution is to only pass
the scope within the tree of service funcs that
use it.

## Type of change

- [x] 🐹 Refactor

## Issue(s)

* #1506

## Test Plan

- [x]  Unit test
2022-11-17 22:08:37 +00:00
Keepers
67265a8c32
remove unused purge code (#1521)
## Description

Now that pwsh scripts do the purging, we can
get rid of the faultier graphapi exchange purge
code, along with the functions supporting it.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1520

## Test Plan

- [x] 💪 Manual
2022-11-16 22:37:25 +00:00
ashmrtn
147f96e69b
Limit container resolver recursion depth (#1524)
## Description

Now that the container resolver may not have all folders in-memory (we're relaxing the limits on when we return an error), add a recursion depth limit. Limit aligns with currently supported M365 service folder hierarchy depth limits

Exchange limitations for [mail folders](https://learn.microsoft.com/en-us/office365/servicedescriptions/exchange-online-service-description/exchange-online-limits#mailbox-folder-limits) (search "Maximum folder hierarchy depth")
[OneDrive](https://support.microsoft.com/en-us/office/restrictions-and-limitations-in-onedrive-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa) limit on path length (search "file name and path lengths")

Minor refactoring of test code to make it easier to spin up tests for multiple cases

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1519

merge after
* #1523 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-16 20:41:39 +00:00
Danny
16e9f07d1e
GC. EventCalendarResolver Populate() refactor (#1518)
## Description
Iterator Removal from Event Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).  
- Stage 1: Reduce the number of error clauses in `Populate()` function
- Stage 2: Update the M365ID retrieval process per resolver

PR introduces Iterate approach used in `delta` functionality to populate the event calendar resolver.  Reduces the amount of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time `concurrent` Mailbox queries are being made (application throttling).
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐹 Trivial/Minor

## Issue(s)
Related to #1486 

## Test Plan

- [x]  Unit test
2022-11-16 16:17:59 +00:00
ashmrtn
caf8422558
Add container resolver tests back (#1523)
## Description

Mistakenly removed when renaming files.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1522 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 22:46:51 +00:00
Abin Simon
110684d5f7
Add size to all objects (#1491)
## Description
Add size to all objects. Not adding them to the output of details view.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-15 17:34:25 +05:30
Abin Simon
825cd6f21b
Add created and modified time to all (#1452)
## Description

Adds Created and Modified time to all objects. Need to go after
https://github.com/alcionai/corso/pull/1446 as it is built on top of
that. Keeping in draft until then.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2022-11-15 10:24:06 +05:30
Keepers
20795d3b56
applies folder matching behavior to cli input (#1443)
## Description

Allows users to specify whether they want to
select folders by prefix or search behavior.
Search/contains behavior is the default case,
with prefix being an optional deviation if the
folder input is prepended with a '/' character.

Also, propagates the PrefixMatch setting to
all integration tests that rely on selecting only
the default folder in exchange.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-08 23:10:40 +00:00
Keepers
b20e7af533
use path filters in folder selectors (#1429)
## Description

For folder-level scopes (ie, scopes that compare
folder-hierarchy path segments),  this change
replaces the standard "equals" and "prefix"
string comparators with the new PathContains
and PathPrefix comparators.

Next change is to interpret user inputs in the
cli to determine whether the comparator should
use contains or prefix behavior.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1224

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-11-08 22:27:09 +00:00