16 Commits

Author SHA1 Message Date
Keepers
62daf10213
migrate onedrive using prefix collection (#3122)
#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-24 17:36:10 +00:00
Keepers
0458d04b9e
use slices for all filter targets (#3028)
Simple concept for a large footprint of change:
filters should always accept a slice of targets
to compare against, instead of a string.  Currently,
we're faking a slice by way of using a const
separator anyway, so we might as well support
slices outright.

Lots of changes (simplification, mostly) in
selectors cascade from this shift.  Any updates
you see here are purely to comply with the updated
filters interface.

This step will make concealing PII in selector and filter
logging significantly easier than before.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-17 22:07:17 +00:00
ashmrtn
1aa315412a
Add more logging to middleware and incremental backup operation logic (#2899)
Ensures:
* context clues are added to logged messages
* kv-pairs added with `log.With` are added to logged messages
* add log messages for how many details items were sourced from a base snapshot during merging
* ensure log messages for base selection have all the info they should

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 22:33:19 +00:00
ashmrtn
ac4d7d047a
Add tester suite wrapper to most of internal pkg (#2614)
## Description

Skip connector subpackage for now as that is fairly large in itself.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-25 02:13:50 +00:00
Keepers
565c33af4e
generate a unique repository id on init (#1716)
## Description

generates a unique repository ID on repo init, and retrieves it again on repo connect.  This allows
the event bus (and any other tracker) to id a repo independent of other context.

I've been unable to find any good way to retrieve a unique repository ID from kopia.  This seems like
the next quickest solution.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1602

## Test Plan

- [x] 💪 Manual
2022-12-12 22:14:37 +00:00
Vaibhav Kamra
93a7ff27e3
Store backup details in a separate snapshot (#1755)
## Description

This PR uses a separate snapshot to store backup details instead of a Corso model (i.e. a kopia manifest).

Introduces a `StreamStore` that can be leveraged to store larger metadata objects. We can also leverage this
for incrementals or restartable backups going forward.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1735 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-10 00:47:49 +00:00
ashmrtn
2524523298
Add a comment for updating Schema constants (#1759)
## Description

Note what things need to stay the same and what things can change when updating schema constant values. This is required because some information about the constants are persisted in the model store.

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1757 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-09 18:00:45 +00:00
Keepers
8ddbc16077
Add GetID() to model.BaseModel (#1336)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1334

## Test Plan

- [x]  Unit test
2022-10-25 23:46:49 +00:00
Keepers
286a74e819
persist service tag in backup model, list by service (#1233)
## Description

Adds a tag describing the service within the backup model and the backup details model.  Adds a filter builder pattern to the store wrapper for filtering results according to certain tags.  Finally, adds a filter builder to specify the service type when listing backups.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1226

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-21 01:04:54 +00:00
ashmrtn
b9e33901f4
Add versions to models (#1064)
## Description

Currently only the ModelStore populates and checks versions.

Some high-level points:
* fail any sort of get (just metadata or full model) if there's a version mismatch
* update operations automatically set things to the current version
* versions are stored as tags so there's some int->string (and vice versa) munging
* versions stored as tags so they can be repopulated even if only the metadata (BaseModel) for a model is pulled. This is done mostly for consistency

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #284 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-06 00:46:49 +00:00
Keepers
6cdc691c6f
correct the corso module (#749)
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.

All other changes in the repository update the
package imports to the new module path.
2022-09-07 15:50:54 -06:00
ashmrtn
d920589507
Add linter for import groups and group order (#477)
* Add import order linter

Error messages aren't the most clear/helpful, but they will point you to
the lines that need modifying.

* Adjust max errors reported

* Fix lint errors
2022-08-03 16:21:38 -07:00
Keepers
e35acb79ca
rename stableID to ID (#425)
embedding a baseModel means that objects like backup
have a backup.StableID, but no backup.ID nor backup.
backupID.  This could be confusing for users.  This change
swaps the value and type names, so that baseModels
include an ID value of type StableID.
2022-07-27 20:54:58 +00:00
Keepers
6c22d5c0ce
adds store package for wrapping model_store (#346)
* adds store package for wrapping model_store

Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
2022-07-19 11:54:53 -06:00
ashmrtn
5ec6d4f286
Base model revision (#263)
* Update model interface and base struct

* Update code and tests for new Model interface
2022-07-01 14:17:15 -07:00
ashmrtn
99691f46d5
ModelStore Put/Get implmentations (#261)
* Simple Get and Put implementations

Get implementation is currently the one that uses the kopia ID of the
model/manifest.

* Basic tests for ModelStore Get/Put
2022-07-01 11:07:01 -07:00