23 Commits

Author SHA1 Message Date
Danny
9f05e83d43
CLI: Adds to the displayable headers for OneDrive and SharePoint. (#2090)
## Description
The update adds `DriveName` to SharePoint details. Values are currently the M365ID

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #1938<issue>
* related to #2064 


## Test Plan
- [x]  Unit test
2023-01-11 12:49:20 +00:00
Danny
85b5877987
Backup Details: Onedrive details expanded (#2085)
## Description
Updates the amount of metadata available for OneDrive backups.  This does not affect the way `OneDrive` information is displayed to the user. 
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #2074<issue>

## Test Plan

- [x]  Unit test
2023-01-10 18:56:38 +00:00
Abin Simon
a4147f5498
Speedup OneDrive backup by pulling urls in delta API (#1842)
## Description

This brings in another huge improvement in the backup speed dropping from ~15m to ~1.5m for an account with ~5000 files. This one also drastically reduces the number of requests we have to make for the same account from 5505 to just ~55. This also means that we don't get throttled anymore and we can easily run multiple backup jobs in parallel before we hit the 1024 limit.

~The code works as of now, but I have to double check the numbers as well as fix an issue with us opening too many files and causing program to crash with 'too many open files' when we bump up the numbers. With the current numbers, it works, but I wanna double check and optimize them. Plus some code cleanup.~

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-20 09:34:28 +00:00
Danny
20bfaee8db
GC: MSGRAPH Linked Library Upgrades (#1861)
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1856 <issue>

## Test Plan


- [x]  Unit test
2022-12-19 22:19:18 +00:00
Keepers
fc5f42545f
rename graph.Service to Servicer (#1787)
## Description

`graph.Service -> graph.Servicer`,  no other changes.

More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-13 17:37:03 +00:00
Abin Simon
4ee66c3d3d
Refactor http client creation for onedrive file download (#1736)
## Description

This way we are able to use any middlewares added in both places.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-12-09 04:59:17 +00:00
Keepers
38239e986c
add weburl identification in the CLI (#1712)
## Description

Adds weburl identificatiotn and unioning in the
cli, for both backup and restore of sharepoint
data.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-09 01:14:04 +00:00
Danny
45e535c9b8
GC: Go.Mod update (#1479)
## Description
Versioning upgraded for GC
Packages updated:
- [x] github.com/microsoftgraph/msgraph-sdk-go v0.41.0 => v0.48.0
- [x] github.com/microsoft/kiota-abstractions-go v0.14.0
- [x] github.com/microsoft/kiota-authentication-azure-go v0.5.0
- [x] github.com/microsoft/kiota-http-go v0.9.0
- [x] github.com/microsoft/kiota-serialization-json-go v0.7.2
- [x] github.com/microsoftgraph/msgraph-sdk-go-core v0.30.1
- [x] github.com/stretchr/testify v1.8.1
- [x] github.com/cjlapao/common-go v0.0.29 => v0.0.35

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
- Update library change. AdditionalData no longer contains `nextLink` for iteration.  See commit [e7c75ca](e7c75ca1cc) for full details. 

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1478<issue>

## Test Plan

- [x]  Unit test
2022-12-08 21:54:22 +00:00
Keepers
4c976298d4
populate sharepoint itemInfo (#1687)
## Description

Currently, all drive backup and restore actions
populatet a details.OneDriveInfo struct.  This
change branches that struct between one-
drive and sharepoint info, depending on the
current source.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-08 17:26:56 +00:00
Abin Simon
1f0338a80d
Use http client from msgraphcore (#1484)
## Description

Use http client from msgraphcore instead of the one from net/http as it has preconfigured middleware.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/581

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-11-18 12:01:12 +00:00
Abin Simon
935fe5cbec
Store/Retrieve owner info for OneDrive files (#1436)
## Description

Store and retrieve owner info for onedrive files.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-11-04 07:01:45 +00:00
Vaibhav Kamra
f1f6c06ba0
Use an upload session for large attachments (#1148)
## Description

Use an upload session and chunked writes for large attachments.

This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and 
leverages that for the upload.

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1115 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-13 03:35:40 +00:00
Keepers
64219c3ae7
track item size in onedrive restores (#1159)
## Description

Hacks in the bytes written as the iteminfo size during onedrive item restores.  Would use the drive itemable, but the size property isn't instantiated locally during creation, and cannot be mutated afterwards.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1113

## Test Plan

- [x] 💪 Manual
2022-10-13 01:13:35 +00:00
Keepers
2cc9b7bc1d
add progress display handler and api (#1116)
## Description

Adds a new package- Observe- for owning user-
oriented displays like progress bars.  This PR adds
an initial progress bar to onedrive backups as a
proof-of-concept.  The API is more important than
the specific progress bar package at this time.
Future changes may opt for a different pkg.

Display format currently looks like:
```
59% [=============>           ] (6.9/12 kB, 14 MB/s)  |  Item_Name.txt
```

Known Issues:
* the `progressbar` package does not support multiline output, and [the author is not planning to add support](https://github.com/schollz/progressbar/issues/6).  This causes concurrent items to overwrite each other.  We will either need to fork the library, or change to a different one.  

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1112

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-10 22:53:48 +00:00
Keepers
f4556288a5
add iteminfo to restore details (#1041)
## Description

Extends the restore details with itemInfo from restored items.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #977

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-05 21:19:21 +00:00
Vaibhav Kamra
3df3d68a3e
OneDrive details UX improvements (#1034)
## Description

- Rename `LastModified` to `Modified` in OneDrive backup details
- Humanize file size units

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #627 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-04 15:44:47 +00:00
Vaibhav Kamra
1df48997ae
Add additional OneDrive metadata to backup details (#952)
## Description

Adds file size and timestamps to enable selectors using this metadata

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #594 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 22:54:42 +00:00
Vaibhav Kamra
ea73873ffb
Fix OneDrive restore regressions (#953)
## Description

Fixes a couple of regressions that got introduced on the restore path via code review changes and
kopia wrapper refactor.

Found while writing the OneDrive restore test. Would have been prevented if we had said test.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-26 22:20:54 +00:00
Danny
3f3c8217a3
GC: Component: Version 40.0 (#917)
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.  
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-22 18:59:51 +00:00
ashmrtn
7850272a5e
Apply wsl linter to onedrive package (#911)
## Description

Finish enabling wsl linter on Corso repo by removing the exception for onedrive package

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #481

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-20 20:02:35 +00:00
Vaibhav Kamra
8587e7d1c2
Implement OneDrive item upload (#843)
## Description

This implements helper methods that support creating new items and uploading data from a `io.Reader` to OneDrive.

It includes an integration test that uses mock data.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #668 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-16 01:22:23 +00:00
Keepers
6cdc691c6f
correct the corso module (#749)
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.

All other changes in the repository update the
package imports to the new module path.
2022-09-07 15:50:54 -06:00
Vaibhav Kamra
8fd867745f
Onedrive item reader (#582)
## Description

This contains the following changes:
- Support functions to enumerate a users drives and the files within it.
- `driveItemReader` method that that reads a drive item
- Integration tests for the above

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #388 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-18 23:38:23 +00:00