314 Commits

Author SHA1 Message Date
Danny
d55ee98d7d
GC: Iterate "Select All Messages/Contacts" Consolidation with new Interface (#663)
`Descendable` interface used to move "IterateSelectAll" functions into a singular function.
Test suites are modified to support changes.
2022-08-29 16:29:15 +00:00
Danny
d3bced28f9
Bug fix: (#662)
Fixes bug introduced by merge.
2022-08-26 16:28:13 -06:00
Danny
ab92358ea6
GC: Backup all users flag support for exchange (#631)
All-users flag support enabled for Graph-Connector
2022-08-26 15:05:14 -06:00
Danny
2ceafd8d62
Contact / Mail folder interfaces (2) (#659)
Added 2 interfaces for future abstractions.
2022-08-26 16:15:55 -04:00
Danny
02c2b0b4d6
GC: Iterate-Functions moved to separate file (#657)
* Issue #654: Prepwork: `service_iterate.go` contains iterate functions for the package.
2022-08-26 11:47:49 -06:00
Danny
a2a5260d3f
Gc option extract (#653)
Content Separation `query_options.go` created for `exchange package.
2022-08-26 12:47:45 -04:00
Keepers
15b12e634d
name cleanup in selectors and onedrive (#643) 2022-08-24 15:13:04 +00:00
Danny
3b9c8e284c
GC: Reduce the amount of folders upon restore (#637)
* Issue #595: All collections with the same `fullPath` name are to be stored within the same folder.
2022-08-23 20:46:59 +00:00
Danny
6e7335cc4d
Gc restore multi framework (#639)
* Abstraction extended to restore. Application logic moved to service_functions.go
2022-08-23 20:30:46 +00:00
Keepers
b524053e96
resolve lint errors (#638) 2022-08-23 20:22:25 +00:00
Danny
a1f79862f1
GC: Event collections to be referenced by calendarID instead of folder (#636)
`data.Collection` of type event  `fullPath()` contain the calendar id
2022-08-23 17:38:30 +00:00
ashmrtn
e8e4bf0914
WIP 316 kopia upload errors (#330)
* Update to kopia with required callback

* Support structs for materializing backup details

Kopia will not allow us to pass data to it that should be passed back to
us in the `FinishedFile` callback. To work around this, create a small
thread-safe support struct that handles information about files kopia is
currently processing. Entries are removed from the set when kopia is
done with them and if no error occurred, the item's info will be added
to the BackupDetails.

* Switch to best attempt for iterating through files

Defaulting to "best-attempt" error handling where all data that didn't
result in an error is handed to kopia and then all errors encountered
are returned at the end.

* Test for uploads that have an error

Simple error reading a file. BackupDetails should not contain
information about the file that had the error (needs update to kopia
code and this code to pass). All other files should be present in kopia
and in BackupDetails.

Co-authored-by: Danny <danny@alcion.ai>
2022-08-23 08:18:29 -07:00
Danny
d7abed1406
GC: setTenants not checking *users.GetMail() (#629)
User query updated from `*user.GetMail()` -> `*user.GetPrincipalName()`. Guard checks added and added GetUsers as one of the GraphQueryFunctions
2022-08-22 14:07:57 -07:00
Danny
b573882746
GC: Accept selectors.BackupExchange exchange.Event and exchange.Contacts (#605)
GraphConnector suite expanded to support backup of Mail, Contacts, and Events. e2e testing suite expanded for all 3 applications.
2022-08-22 20:52:01 +00:00
Danny
4f5bb2b995
Exchange event info (#569)
Exchange Info feature added and verified with `mockdata` to ensure correctness
2022-08-19 13:43:26 -06:00
Danny
74a8666956
Exchange Serialization test finalized (#611)
Finalize graph connector regression tests. Needed to change the use of GC to be singular due to parallel testing.
2022-08-19 15:39:16 +00:00
Danny
9d18d50cf8
Populate function Refactor to single function (#585)
Populate functions streamlined into one function with a few additional abstractions
2022-08-19 08:19:42 -07:00
Vaibhav Kamra
8fd867745f
Onedrive item reader (#582)
## Description

This contains the following changes:
- Support functions to enumerate a users drives and the files within it.
- `driveItemReader` method that that reads a drive item
- Integration tests for the above

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #388 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-18 23:38:23 +00:00
Vaibhav Kamra
f8bfe93261
Scope restore CLI test (#592) 2022-08-18 14:19:08 +00:00
Vaibhav Kamra
c4e5915983
OneDrive Collection (#580)
## Description

Introduces a OneDrive data collection.

Follow up PRs will implement the `collection.driveItemReader()` method that uses the Graph API

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #387 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-17 19:38:05 +00:00
Danny
9049f3c2bf
552 gc mockdata events (#583)
Addition of mock event data structure and testing suite.
2022-08-17 11:22:39 -06:00
Danny
90be4200ca
GC: Events Support added. Not connected within ExchangeDataCollection (#516)
m365 events added for querying and iteration. Test suite added. Mock data is in a different PR.
2022-08-17 16:29:37 +00:00
Danny
ad2c17876f
ContactInfo added to exchange package (#574)
ContactInfo created for models.Contactable. ContactInfo test suite created.
2022-08-17 13:47:27 +00:00
Danny
41f22ad9a4
Added error trace reading for query calls. (#575)
Added error tracing that gives more details for failed GraphQuery reading for query calls.
2022-08-17 13:07:53 +00:00
Danny
2f7d8420ae
551 mockdata contacts (#553)
Creation of mock data for Contact M365 objects.
CreateContactFromBytes function added to support package
2022-08-15 16:00:58 -04:00
Danny
be7b778769
GC Contacts requires additional tests placed in Exchange Services (#475)
Test coverage extended to GraphQuery and GraphIterateFuncs for exchange.Mail and exchange.Contact use cases.
2022-08-15 17:50:06 +00:00
Keepers
6464a66b46
use standard user env var in graphconnector tests (#537) 2022-08-12 18:40:32 +00:00
ashmrtn
e76860fd80
Set line length to 120 characters (#506)
* Enable line width linter

Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
2022-08-12 16:05:46 +00:00
Vaibhav Kamra
88bcd053e5
Simplify test utilities (#532) 2022-08-12 02:49:22 +00:00
ashmrtn
da66cc4c2f
Enable more rigorous version of gofmt linting (#488)
* Enable a stricter linter

* Fix new lint errors
2022-08-05 13:33:20 -07:00
Danny
eff95b7702
GC: Contact Framework added (#473)
`exchange`.Contacts added to Collection retrieval option base through the use of a selector. 

The ExchangeDataCollection Function to be updated to make the function `Live` to be called externally
2022-08-05 16:19:48 -04:00
Keepers
20aa3d3a28
purge test user folders every half hour (#491) 2022-08-05 18:24:58 +00:00
Danny
97113aa80b
Folder Granularity scope Added to Collections Retrieval (#482)
Backup function `DataExchangeCollections` supports the ability to backup a single file using the `selectors` package. Support files and documentation updated accordingly
2022-08-05 11:20:36 -04:00
Keepers
73942fe952
fix up logger specs (#489) 2022-08-04 23:28:26 +00:00
ashmrtn
9e2e88f5f3
Enable some linters in revive (#486)
* Turn on revive linter, ignore only a few things

* Fix lint errors

Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
2022-08-04 14:42:51 -07:00
Keepers
a586512b42
add command to purge user's folders by prefix/date (#480) 2022-08-04 15:28:28 -06:00
ashmrtn
d920589507
Add linter for import groups and group order (#477)
* Add import order linter

Error messages aren't the most clear/helpful, but they will point you to
the lines that need modifying.

* Adjust max errors reported

* Fix lint errors
2022-08-03 16:21:38 -07:00
Danny
bb7531d95c
exchange.Collection to fill in key information from selectors (#457)
Selectors and scope information are used to create the correct populate methods. Populate methods changed to fit type abstractions definitions for iteration, querying, and collection population.
2022-08-03 19:07:26 -04:00
Danny
d656bc181c
Regression test added for serialization and (de)serialization process… (#461)
* Regression test added for serialization and (de)serialization process for graphConnector.
2022-08-03 17:53:19 -04:00
Danny
2b95553355
Gc serialize to collection (#467)
Abstractions: Exchange selector function introduced, GraphIterateFuncs and service functions extended
2022-08-03 14:26:06 -04:00
Danny
186c2d9dc4
Restore into Custom Folder (#391)
Custom folder added for restore process workflow
2022-08-03 12:14:45 -04:00
Keepers
640dfcf967
derive test userID from test config (#465)
In order to both maintain control of test env data, as
well as ensure open-source users are able to run the
integration tests on personal account data, the test
user must be derived from the test configuration, rather
than hardcoded within the tests.
2022-08-02 19:13:23 +00:00
Keepers
d7fe518007
rename /internal/testing to /tester (#463)
The alias of testing to ctesting prevented auto-importing
of the testing library.  This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
2022-08-02 12:21:14 -06:00
Danny
87427a52aa
Gc iterate abstraction (#459)
Abstraction functions and iterate functions created for the `serializeMessages` move to createCollections. 
Graph Connector refactor still underway.
2022-08-02 13:31:53 -04:00
Danny
a1b8868600
Removal of internal Contains methods (#462)
Removal of internal  `contains`methods
2022-08-02 12:49:21 -04:00
Danny
3eaebaed8f
Incremental change prior to selector (#460)
Use of abstract variable for creating page iterator.
2022-08-02 12:21:35 -04:00
Danny
bcbf0b1fb8
GetMailFolderID feature added (#455)
GetMaillFolderID add to `exchange` package. Test Suite Added.
2022-08-01 15:12:02 -04:00
Danny
890c8f2b7f
Exchange service: Abstractions (#450)
- Creation of exchange service-based query file. Test package expanded for support
2022-08-01 13:27:30 -04:00
Danny
bd559f690e
Exchange service functions (#454)
`exchange`.service_actions.go created. Create/Delete Mail folder functions inherited. Test suite updated accordingly.
2022-08-01 16:29:03 +00:00
Danny
2f56a849c9
Exchange.Collection.Items() to initiate populate method (#446)
`Items()` starts populate method for collection for the `exchange.Collection` package. Test packages expanded to verify the functionality of this feature.
2022-07-31 17:57:14 +00:00