639 Commits

Author SHA1 Message Date
ashmrtn
09aebc4a31
Wire backup cleanup into maintenance operation (#4091)
Wire backup cleanup into maintenance operation.
Currently it runs under a different mode so it
won't affect existing users of the maintenance
command. Once we're satisfied with testing we
can remove the temporary const and run this
when complete maintenance is run

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3217

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-21 22:15:29 +00:00
Abin Simon
ccf8df3548
Bump msgraph sdk to 1.19.0 (#4314)
Needed some changes in the func names.

``` bash
$ sed -i "s/By\([a-zA-Z]*\)IdString/By\1Id/g" **/*.go
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-21 07:01:07 +00:00
Keepers
6159668b1d
utilize format validation check (#4288)
adds usage of the export format validation check
to export cli commands.  Also moves the restore
cfg validation check out of the service flags
validation checks and into the generic restore runner for better separation of concerns.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-21 00:24:12 +00:00
Keepers
c1ec3c6648
formalizes ErrApplicationThrottled (#4317)
adds graph error identification for application throttled and adds an ApplicationThrottled sentinel to pkg/errs for sdk consumer identification.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-20 21:27:01 +00:00
ashmrtn
83815eee0e
Expand readonly mode test slightly (#4304)
Must have forgotten to submit the suggestion
for this in #4302

Maintenance is a quick way to check if readonly
mode is working. It doesn't require pulling
any data from graph but will still allow us to
check for a sentinel error.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-20 16:07:34 +00:00
Vaibhav Kamra
4814154928
Support opening repository in read-only mode (#4302)
Useful for running read only operations against the repository

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-20 04:05:12 +00:00
Abhishek Pandey
d2c73827cb
Add CLI for local storage (#4243)
<!-- PR description-->

New commands to initialize & connect to a repo on local or network attached storage.
* `repo init filesystem --path /tmp/repo`
* `repo connect filesystem --path /tmp/repo`

Includes basic unit & e2e tests. More coverage to be added in a following PR to keep the size contained.

**Updates:**
* Added Repo path sanitization i.e. handle relative paths, make paths cross platform compatible, etc. 
* Removed retention artifacts, not supported for filesystem storage.
* cli docs - auto updated.
* Manually tested with all corso backup/restore/export commands.

**Doesn't include** 
1. Symlinks
2. User ids wiring into repo.
3. Repos documentation update - in an upcoming PR.
4. Prefix support -> kopia doesn't support prefixes for `filesystem` storage
5. More E2E tests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1416 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-19 09:46:54 +00:00
ashmrtn
8b9f63879b
Allow getting a path prefix from a string (#4132)
Will be required for selective subtree pruning since hierarchy merging starts at the category directory.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4117

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-18 23:05:10 +00:00
ashmrtn
4a9951b876
Fetch mod time when getting added and removed items (#4266)
Also return mod time when available if
getting the set of added and removed
items. This will be leveraged in later
PRs to implement kopia assisted
incrementals for exchange

Does not change any logic in
collections right now, just adds the
fields to be returned

Also adds an additional return value
denoting if the mod times are expected
to be valid. This is required because
events delta cannot return mod time

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2023

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-18 22:30:56 +00:00
Keepers
8f7070ffac
add humanstring to path category type (#4282)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-09-18 20:57:15 +00:00
Keepers
77c48ebf95
minor touchups from manual testing (#4265)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-18 16:55:15 +00:00
Abhishek Pandey
8590b24199
Introduce new interfaces for storage configuration (#4251)
<!-- PR description-->
Introducing a new `Configurer` interface to abstract out storage config information(for s3, filesystem etc) from caller code. I consider this as a short term solution. We need to consolidate overall config handling in a better way, but that's out of scope for this PR chain.

Testing
* Most of the changes here are code movement under the hood. So relying on existing tests.
* I'll address any test gaps in a later PR.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1416

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-18 14:32:54 +00:00
Abhishek Pandey
fb6398ea41
Remove tight CLI coupling with S3 storage provider (#4237)
<!-- PR description-->

One of the 2 remaining setup PRs before we can introduce local storage repos.

**Changes:**
1. Read storage provider(`provider`) from config file except for `repo init * ` or `repo connect *` commands.
2. Apply flag overrides based on provider type( e.g. `S3FlagOverrides` if provider is `S3`)
3. Propagate storage provider type to functions which read/write config. These functions arbitrate on config hierarchy - flags, env, config file, in that order. 

**Reasons**
* Reason 1 is needed is because config file is the source of truth for storage provider for all commands except `repo init` or `repo connect`. In the exception cases, we pick the provider in command (e.g. `s3`) as the source of truth. e.g. consider a `repo init s3`, followed by `repo init filesystem`.During `repo init filesystem`, config file would indicate `S3` provider, but the correct behavior here is to select `filesystem` provider.
* One alternative was to push provider from the init/connect cmds into an override flag, and let the config code decide on hierarchy. However, this felt hacky. provider here is not a flag to begin with. It's part of init/connect commands.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1416

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-18 13:01:00 +00:00
Abin Simon
4a7847936e
Add incremental backup test for groups (#4248)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/3990

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-09-16 05:04:28 +00:00
Keepers
f7042129f4
minimize channel messages exports (#4245)
reduces channel message export data to the minimal set of valuable info: message content, creator,
creation and modification time, and replies (each
reply has the same data, sans other replies).

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3991  

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-09-15 21:16:06 +00:00
Keepers
265a77f1cd
match item type in groups selectors info filter (#4255)
adds item type comparisons to the info filter during groups selector reduction.  This ensures cross-
contamination of item types on shared info properties does not occur.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-15 20:29:14 +00:00
Abin Simon
ce422f0b1d
Add linter for checking with empty string (#4246)
Finding failing cases:

``` bash
tree-grepper -q go '((binary_expression (identifier) ["==" "!="] (interpreted_string_literal) @ri) (#eq? @ri "\"\""))'
```

Fixing failing cases:

``` bash
comby 'if :[1~[^ ]*] == ""' 'if len(:[1]) == 0' -matcher .go -in-place
comby 'if :[1~[^ ]*] != ""' 'if len(:[1]) > 0' -matcher .go -in-place
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/3654

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-15 06:45:00 +00:00
Abin Simon
04628c7f91
SharePoint MatchesInfo tests (#4250)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #3990

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-15 03:52:05 +00:00
Keepers
14416094e6
get replies when getting message (#4232)
This was somehow sliced out of changes persisted
in prior branch merges.  It re-adds persisting replies as part of message content retrieval.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3989

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-09-14 21:42:05 +00:00
Keepers
cb319bb2ae
use display name to look up groups (#4242)
Allows lookup of groups using their display name in addition to their ID.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-14 17:03:14 +00:00
Abin Simon
edf753382e
CLI and selector changes for groups restore (#4218)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-14 09:29:01 +00:00
Abin Simon
80f11d9876
Restore for Groups SharePoint Sites (#4209)
Still a few pending items like CLI handling and restoring specific items. Will be handled in a follow up.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3992

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-14 08:49:34 +00:00
Abin Simon
6e1be1f08c
Incrementals for Groups SharePoint backup (#4205)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3990

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-14 06:54:39 +00:00
Abhishek Pandey
de062cd5de
Minor refactor of storage provider stringer (#4228)
<!-- PR description-->

One of the setup PRs before introducing local storage support for corso.

Changes
1. Export `storage.storageProvider` and rename it to `storage.Provider`. Without the rename, linter errors out with `exported: type name will be used as storage.StorageProvider by other packages, and that stutters; consider calling this Provider`
2. Rerun stringer - `stringer -type=Provider -linecomment`
3. Introduce a new Provider - `FS` for filesystem. This is in line with kopia naming for local/network attached storage.
4. Fix hardcoded string values for storage providers in test code.
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-13 11:44:40 +00:00
Keepers
e3bde06457
add non-delta message paging path (#4229)
similar to exchange, there are conditions under
which channel messages are unable to use the
delta api.  In particular, when the channel has no valid email property.  This change ensures that
we enumerate the entire channel under those
conditions, instead of failing out.

Technically, the only situation we know of where
the email property is missing, and thus the channel cannot make delta queries, is when there are
zero messages in the channel already.  However,
there may be unknown cases that we haven't caught, so instead of skipping the channel we're going to
handle non-delta enumeration for future safeguarding.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3989 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-13 01:48:44 +00:00
Keepers
9a8c413b52
some code cleanup before groups export (#4202)
various tidbits of data cleanup before moving forward with adding export behavior to groups.
* move duplicate collections mocks into data/mock
* move the export collection struct into pkg/export (to prevent future duplicates in the next PR)
* rename export.Collection to Collectioner, because it's an interface.
* some other non-logic rearrangement

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3991 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-12 22:35:03 +00:00
Keepers
a2e80a178a
path pkg rename resouceOwner to protectedResource (#4193)
updaing the path package to the current naming convention. No logic changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-12 22:05:23 +00:00
neha_gupta
75c5cb3582
fix sanity test for groups (#4226)
<!-- PR description-->

Handle-
- empty channel
-  do to add item if deleted

#### Does this PR need a docs update or release note?


- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
2023-09-12 18:30:55 +00:00
Keepers
a1a33a2c2d
fixing bugs found during extensive manual tests (#4173)
fix up the following bugs:
* reinstate previousPath in the metadata retrieval
* implement DoNotMergeItems for real
* nil pointer protection for the 'from' property
* some additional logging

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-12 16:54:06 +00:00
Keepers
d3eda28989
enable groups integration tests (#4161)
enables nightly cli e2e tests, operations layer integration tests, and sanity tests for groups and teams.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-08 20:18:59 +00:00
Abin Simon
be59928f98
Fix cases where we had a trailing comma (#4208)
Not sure if we wanna merge it as it might generate way too many conflicts, but this should help us add a linter in CI. If we are good, I'll add something that can do lints for this in a follow up PR.

Super hacky, but this fix was created using `while true ; do tree-grepper -q go '(argument_list "," @nope .)' | tail -n1| awk -F: "{print \$1,\"+\"\$2\" -c ':norm \$xJZZ'\"}" | xargs vim ; done`.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3654

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-08 17:10:29 +00:00
Keepers
ce082162b5
add site owner to site struct (#4206)
Adds the owner category to the minified site struct in the m365 service api.  Instead of generating a "SiteByID" func in that layer, the ParseSite func is exported, which allows the user to provide an existing siteable
interface for transformation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-08 16:29:36 +00:00
Abhishek Pandey
35ccb985df
Cleanup old user info code (#4189)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-08 08:41:58 +00:00
Abhishek Pandey
9bedee9c72
Wire service isolation changes into SDK code (#4186)
<!-- PR description-->

Wires `IsServiceEnabled` code into `UserHasMailbox`, `UserHasDrive`, and `UserGetMailboxInfo` SDK APIs.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*  https://github.com/alcionai/corso/issues/3844

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-08 07:58:38 +00:00
Keepers
f94f5bbc76
add drive info to site get by id (#4197)
extends the site get-by-id call to include drive info. Useful for detecting whether the site is owned by
a user, or a group or team.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-07 22:13:46 +00:00
Keepers
bf02f57527
add a BuildMetadata func to path package (#4192)
just a QoL shorthand to match the Build func.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [=x] 💚 E2E
2023-09-07 19:05:58 +00:00
Keepers
c74585eafc
remove redundant type check (#4199)
info.itemType already looks through each service entry to find the item type.  A service nil check is redundant. Adds a qol func for asking if an itemInfo is a driveish item.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3990 

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-07 18:30:46 +00:00
Abin Simon
eb8f3131f9
Misc fixes for handling groups (#4195)
A bunch of places where we missed handling groups
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-07 06:36:18 +00:00
Abhishek Pandey
59bf350603
Add service isolation (#4096)
<!-- PR description-->

1. Renamed `IsBackupRunnable` to `IsServiceEnabled` & extended to restore operation. Removed `checkServiceEnabled`. Reasons:
- The above 2 functions were doing the same thing. Removed `checkServiceEnabled` in favor of `IsBackupRunnable` since we want this check to be as soon as possible during backup/restore op initialization
- Renamed `IsBackupRunnable` to `IsServiceEnabled`, because a) we are only doing service enabled checks right now, b) common code that can be used for both restore & backup.

2. Wire corso code to use new helpers in internal/m365/common.go
3. Note: SDK wiring and related integ tests will be added in a follow up PR

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3844

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-06 17:15:40 +00:00
Keepers
86e1cef3a9
make teams and groups work in the cli (#4139)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3989

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-09-05 21:14:34 +00:00
Keepers
5aee3cc2ae
use channels api to run backups (#4136)
This is the first in a series of PRs to get v0 backups working for channels.  In this change, the current api enumerators get plugged into the collections handler to produce backup data. Follow-up PRs will:
* hook backup to the CLI
* swap full-item enumeration for id-first-get-later pattern
* populate each message with all its replies on the get-later
* turn on integration testing at the operations and ci layers

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-05 19:54:02 +00:00
dependabot[bot]
ccdb672026
⬆️ Bump github.com/microsoftgraph/msgraph-sdk-go from 1.16.0 to 1.17.0 in /src (#4167)
Bumps [github.com/microsoftgraph/msgraph-sdk-go](https://github.com/microsoftgraph/msgraph-sdk-go) from 1.16.0 to 1.17.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/microsoftgraph/msgraph-sdk-go/blob/main/CHANGELOG.md">github.com/microsoftgraph/msgraph-sdk-go's changelog</a>.</em></p>
<blockquote>
<h2>[1.17.0]- 2023-08-30</h2>
<h3>Changed</h3>
<ul>
<li>Weekly generation.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="bd9689c25d"><code>bd9689c</code></a> Generated  models and request builders (<a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/563">#563</a>)</li>
<li>See full diff in <a href="https://github.com/microsoftgraph/msgraph-sdk-go/compare/v1.16.0...v1.17.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/microsoftgraph/msgraph-sdk-go&package-manager=go_modules&previous-version=1.16.0&new-version=1.17.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-09-05 16:45:27 +00:00
Abhishek Pandey
c1c4218994
Add common code to check if services are enabled (#4127)
<!-- PR description-->

Currently corso CLI & SDK have different checks for services enabled. These checks have diverged which can lead to bugs. For e.g. SDK users use [code](3e43028a88/src/pkg/services/m365/users.go (L92)) to check if OD is enabled, while corso uses [this code](https://github.com/alcionai/corso/blob/main/src/pkg/services/m365/api/users.go#L174). These funcs have different checks. This PR introduces common helpers to consolidate these checks in one place.


- Note: I decided against absorbing these helpers into api/users.go, because separation of concerns -  getters shouldn't arbitrate on return vals. 
- Note that this code is not yet wired up to SDK/CLI. It'll be added in a following [PR](https://github.com/alcionai/corso/pull/4096). 


**Changes**
1. `Is*ServiceEnabled` helpers in internal/m365/common.go. 

2. Add `GetMailboxInfo` to common code. This is currently a duplicate of code in `GetInfo`. That function will be removed once we port SDK users to `GetMailboxInfo`. 

3. Unit tests for common code. Integration tests will be added in a later PR
- Note:`TestIsOneDriveServiceEnabled` is copied from `TestCheckUserHasDrives`. The older test will be removed in a later PR.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3844

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-04 18:59:03 +00:00
Keepers
8d2070409f
fix language issues with default contacts (#4176)
prepends the default contacts folder display name to the location instead of the well-known defaultContacts value.  This ensures the folder gets compared properly for non-english mailboxes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4175

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-01 20:38:34 +00:00
Abin Simon
91e4f455b3
Use path prefixes for backup paths (#4155)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4154

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-08-31 06:34:13 +00:00
Keepers
4c452bcd8e
add channel and messages enumerators (#4135)
adds enumerators for channels, and a delta
of channel messages.
Also updates the its struct to include group ids, and reformats some of the struct schema.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-30 17:21:53 +00:00
ashmrtn
cc5864f852
Fix data clobbering issue when getting Quota Exceeded status (#4144)
Don't clobber the Quota Exceeded status once we get it

Add unit test to make sure this doesn't happen

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-08-29 22:08:43 +00:00
Keepers
ee321863ed
unify api pager interfaces (#4088)
We have multitple different pager interfaces that all utilize the same (effective) set of functions.  This change reduces those to two different interfaces: a delta- and non-delta- pair of pagers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-28 17:24:46 +00:00
Keepers
e96c21d342
add channels info filters to selectors (#4093)
Adds groups channel message filters to group selectors. Includes scopes for message creation time, last response time, and message created by.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3989

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-08-25 18:09:20 +00:00
ashmrtn
bf29443ad4
Use interface for test helper functions (#4114)
Take an interface as the parameter for test helper functions. This allows them to be used with *testing.B (benchmark) as well as *testing.T (test).

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-08-25 17:28:09 +00:00