187 Commits

Author SHA1 Message Date
Keepers
fc5f42545f
rename graph.Service to Servicer (#1787)
## Description

`graph.Service -> graph.Servicer`,  no other changes.

More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-13 17:37:03 +00:00
Abin Simon
99f35eb5a8
Fix staticcheck lint in CI (#1682)
## Description

Missed configuring working dir for staticcheck, plus fixes for errors reported.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-08 08:26:27 +00:00
Keepers
7e2fbbea4f
adds sharepoint restore cli commands (#1637)
## Description

Adds restore commands to the cli for sharepoint.
The restore process is only partially functional at this time.
Library files that pass auth are able to be restored as expected.
However, auth issues (not directly related to these changes) prevent
restoration of all library items at this time.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1615

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-12-06 17:00:37 +00:00
Keepers
07a8d13d1e
adds tenant site lookup on new graph conn (#1540)
## Description

Adds a lookup for all tenant site ids when a new
graph connector is created.  Also adds an enum
to flag which resource set (users, sites, or all)
that the connector should initialize.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-18 00:31:41 +00:00
Keepers
67265a8c32
remove unused purge code (#1521)
## Description

Now that pwsh scripts do the purging, we can
get rid of the faultier graphapi exchange purge
code, along with the functions supporting it.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1520

## Test Plan

- [x] 💪 Manual
2022-11-16 22:37:25 +00:00
Keepers
e6191f017b
replace golang purge script with powershell script (#1364)
## Description

Swap from GraphAPI-based data cleanup (which doesn't allow
us to access hidden folders like Audit retention or delete-restoration)
to powershell script-based cleanup (which allows us to make
SOAP requests against legacy exchange apis).

## Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Test

## Issue(s)

* #1266

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-31 22:40:42 +00:00
Keepers
a31b01458c
flush logs on cmd failure (#1338)
## Description

The secondary scripts in corso (factory, purge, etc) are not currently flushing their logs if an error occurrs since os.Exit(1) kills the current process without alloting time for deferred funcs to run.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-27 22:57:05 +00:00
Danny
6ddf02cf50
GC: Restore: MockEvent w/ Attachment (#1345)
## Description
Adds feature to be able to create mock events with attachments
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #1005 <issue>

## Test Plan
Testing:
```
internal/connector/exchange$ go test -v . --run TestExchangeServiceSuite/TestRestoreExchangeObject/Test_Event_with_Attachment
```
- [x]  Unit test
2022-10-27 18:54:29 +00:00
Danny
39b47cc123
Gc mock package (#1313)
## Description
Command line utility that allows for the printing of mockData based on the m365ID. Supports:
- `exchange.Mail` 
- `exchange.Contacts`
- `exchange.Events`

<!-- Insert PR description-->

## Type of change

- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

Related to:
* #1255<issue>
* #999 
* #705
2022-10-25 18:50:39 +00:00
Keepers
1aa2d987e1
log failure cause on purge (#1307)
## Description

Load tests fail on the post-test cleanup using the purge script,
but are not currently logging the cause.  Local repro has not
been successful.  This change adds additional error logging
output for review in case of failure.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
2022-10-24 21:54:48 +00:00
Keepers
f57b0f1f7e
add contact generation to the factory script (#1202)
## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-20 16:58:12 +00:00
Danny
5cd2583466
GC: Purge: Updates to use graph.ContainerResolver (#1190)
## Description
purge.go to use the ContainerResolvers to collect containers from the Graph Connector.

NOTE: CollectFolders in service_query.go unchanged. `CollectFolderrs()` is stubbed for another PR. This will reduce the amount of line changes to be used in the future. 
<!-- Insert PR description-->

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #1189<issue>

## Test Plan

- [x]  Unit test
2022-10-19 14:08:03 +00:00
Keepers
5d2bc1eb4e
add event production to the factory script (#1200)
## Description

Adds the capacity to programmatically generate events in the factory script.  Status is currently functional in code completion, but experiencing unexpected results.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-18 23:54:37 +00:00
Keepers
a32f86e40d
migrate naming of azure env vars (#1176)
## Description

migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.

⚠️  Your local environment will need to change in the following ways:  ⚠️ 
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #558

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-10-17 20:47:04 +00:00
Keepers
f737f58a0f
generate emails in factory script (#1162)
## Description

Adds scripted production of mock emails for building out
large/rolling datasets for load testing.

## Type of change

- [x] 🌻 Feature
- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
2022-10-14 18:48:00 +00:00
Keepers
39f9c7f50f
add script boilerplate for mock data generation (#1161)
## Description

Introduces a `factory` script to help devs generate
fake m365 data in bulk for testing purposes.  Only
boilerplate at this time.  Any TODO's or not-yet-
implemented statuses are present as placeholders
alone, without official deadlines.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
2022-10-14 17:19:40 +00:00
ashmrtn
f4befaf849
Allow purge command to delete OneDrive folders (#1094)
* Add functions for OneDrive purge

function is setup to return and take purgable instead of just userID,
itemID for deleter.

* Adjust other functions to take purgable

Adjust interface for deleter so that it allows more information to the
delete function. The called function can cast the purgable it was passed
back to the type that was returned by the getter function.

* Move struct definition

Per reviewer comment move struct definition to OneDrive package instead
of leaving in purge command.

* Fix lint error

Likely slipped through while GitHub actions was having issues.
2022-10-10 18:16:46 -07:00
Georgi Matev
c4eed04483
Fill in developer/build.md page placeholder (#1065)
## Description

* Fill in developer/build.md page placeholder
* Fix some liniting issues

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-10-06 17:57:32 +00:00
Keepers
30f35709be
purge restored load test folders (#1059)
## Description

purge restored folders for all users in the tenant following a load test to ensure we don't over-
produce data that causes future tests to time out.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1048

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-06 01:20:17 +00:00
Georgi Matev
3a1eb1efd2
Add Examples section to CLI usage and docs (#1050)
## Description

Adds Examples to CLI usage output and the auto-gen docs

Also cleans up:
* OneDrive flags and flags help
* Addition of OneDrive generated files to docs 


## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes #529  

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-10-05 07:06:32 +00:00
Georgi Matev
81a259cb32
Clean up CLI flags, add extended usage, enhance docs pages (#1022)
## Description

* Clean up pass on all CLI output
* Added basic extended usage
* Mark required params in generated docs

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes #527  

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-10-03 19:35:57 +00:00
Danny
3f3c8217a3
GC: Component: Version 40.0 (#917)
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.  
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-22 18:59:51 +00:00
Keepers
e8473e3c78
fix config home dir help text (#907)
## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #718

## Test Plan

- [x] 💪 Manual
2022-09-20 18:19:25 +00:00
Keepers
1203fd2b6a
extract timestamp from arbitrary string (#883)
## Description

In order for purge to gracefully handle the variety of timestring values that may appear within a
folder name, the common/time.go file has added
a func to extract a time substring from a string.
Purge is switched to that extractor instead of
a len-based suffix check.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #805

## Test Plan

- [x] 💪 Manual
2022-09-19 20:35:49 +00:00
Keepers
1d24d575ca
fix folder deletion failures (#867)
## Description

Graph clips time values ending in :00 from folder
names.  In order for the purge handler to check
the time values correctly, we need to identify
those folders and reinstate the 00 seconds tail.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #805

## Test Plan

- [x] 💪 Manual
2022-09-15 19:44:52 +00:00
Keepers
fa489782a3
purge both mail and calendars (#813)
expands the `purge` command to accept args for
purging mail folders, calendars, or both.  This
allows the test cleanup to ensure we aren't over-
populating either mail folders or calendars, thus
blocking CI actions
2022-09-12 17:35:16 -06:00
Georgi Matev
f509543856
Custom output template for auto-generated docs (#785)
## Description

Introduces a Corso specific (more desirable) formatting for the auto generated documentation. 

More or less an adaptation or https://github.com/spf13/cobra/blob/main/doc/md_docs.go. Unfortunately the original 
package does not have sufficient hooks to allow for plugging in a new renderer. 

Other approaches considered: 
* [Override the UsageTemplate ](https://github.com/spf13/cobra/blob/main/user_guide.md#defining-your-own-help) - unfortunately does not apply to Markdown generation
* Use a golang template for the docs output - unfortunately the `pflags` package does not have a suitable way of iterating over the flags that can be easily fed into a template. 

Follow on work:
* ~~Styling of flags tables as part of docs rendering - Done~~
* Opens up possibility for flag ordering - need to set `pflags.SortFlags = false` before parsing the flags - https://github.com/alcionai/corso/issues/783
* Can simplify sidebar command maintenance by naming the rendered docs files line similar to `<Num>_corso_command1_command2` where `<Num>` can be set as annotation on the commands. Order in the navbar can be set as part of command authoring - https://github.com/alcionai/corso/issues/784

## Type of change

Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

* https://github.com/alcionai/corso/issues/536
* https://github.com/alcionai/corso/issues/528

## Test Plan

<!-- How will this be tested prior to merging.-->

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-09 03:15:24 +00:00
Keepers
6cdc691c6f
correct the corso module (#749)
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.

All other changes in the repository update the
package imports to the new module path.
2022-09-07 15:50:54 -06:00
Georgi Matev
dcbfed15df
Prepend Docusorus front matter to auto-gen CLI commands (#713)
* Pre-pend dark to auto-gen CLI commands

* After `gofmt -s`

* Update src/cmd/mdgen/mdgen.go

Co-authored-by: Keepers <ryanfkeepers@gmail.com>

* More gofmt

Co-authored-by: Keepers <ryanfkeepers@gmail.com>
2022-08-31 22:56:05 +00:00
ashmrtn
88a318a7c0
Fix wsl errors in cmd and cli packages (#651) 2022-08-26 09:13:39 -07:00
Keepers
b524053e96
resolve lint errors (#638) 2022-08-23 20:22:25 +00:00
Keepers
a1b9f876ae
add final basic cli tests (#578)
Adds the base CLI integration tests for backup, list, details,
and restore.  Also refactors out the global root command
value in favor of a ctx-bound reference so that tests may
control safely overwriting stdout to scrutinize output.
2022-08-17 18:13:55 +00:00
Keepers
192c69c68f
add backup cli integration test (#517)
Adds the basic cli-based backup integration test.
Due to discovering some corner cases about panic
conditions and other error handling in bad runtime
state, updates many other packages for safety.
2022-08-12 20:04:44 +00:00
Keepers
9f1c8aa64c
do not auto-generate date updates in cli markdown (#526) 2022-08-12 16:43:19 +00:00
ashmrtn
da66cc4c2f
Enable more rigorous version of gofmt linting (#488)
* Enable a stricter linter

* Fix new lint errors
2022-08-05 13:33:20 -07:00
Keepers
9326690338
autopopulate cli markdown on ci build (#453) 2022-08-04 16:46:14 -06:00
Keepers
a586512b42
add command to purge user's folders by prefix/date (#480) 2022-08-04 15:28:28 -06:00