## Description
Record information about the error Graph returned if something went wrong when fetching an item.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2194
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Item not found errors may appear because an item was moved or deleted between enumerating the items in a folder and fetching the data for the items in the folder. This keeps them from being reported as there is nothing we can do if the item data is no longer available.
This applies to all Exchange data categories
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2198
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Make error appending thread-safe since multiple goroutines may attempt to add an error at the same time.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2197
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Images copy/pasted into a message were not being saved previously. Verified with Corso binary
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ✅ Yes, in the ChangeLog.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2163<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
## Description
Adds a little extra process into the calendar
resolver so that it mimics the mail and contact
resolvers. This will allow us to collapse the
three resolvers into a more common handler
or interface.
We can take this change or drop it. I added
the code in exploration of the event test failures,
and figured I'd throw it out just in case.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2022
## Test Plan
- [x] 💚 E2E
## Description
Hacks the beta version call into the events api
when iterating through items, allowing us to run
delta-based queries for calendar events.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #2022
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds info logging on all throttling responses,
and some extra error handling in container
resolvers.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Test Plan
- [x] 💪 Manual
## Description
Adds the per item collection streaming calls to
the api interface. Primarily migrates a "getItem"
and a "serializeItem" acton into the api pkg, out
from exchange_data_collection. Building an
ExchangeInfo is now also housed in api.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1996
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
#2116 implemented a "last change wins" strategy for deduping IDs. However, it appears Graph API doesn't really enforce an ordering for returned items so that strategy may not work. Instead, implement a "delete wins" strategy.
Items that are added, deleted, and then restored in M365 will be restored with a different ID
This PR does the following:
* reverts changes to return a unified list of IDs and added/removed status
* switches to a "delete wins" merge strategy for duplicate IDs
* updates tests
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#1954
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
add logging to the observe package, assume that
every instance where a message is observed, it
also gets logged.
Merger may want to wait until logging to a file is the standard behavior, else the terminal might get messy/buggy.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2061
## Test Plan
- [x] 💪 Manual
## Description
Under some conditions, items can be returned multiple times when enumerating the items in a folder in Exchange. Begin deduping those items by doing the following:
* use a map in ExchangeDataCollection to dedupe on item IDs
* have enumeration function return a single slice of items containing both additions and deletion so a "last-change-wins" policy can be implemented
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1954
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
DiscreteScopes is a vestigial func from when scopes contained the list of resource owners to track. That behavior is no longer in use.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
## Description
Transitions the fetchIDs service iterators to a
set of interfaces to consolidate code across
multiple nearly identical variations of "fetch id
for directory".
This was originally written in another PR (1780), then
separated out to isolate changes.
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] 💚 E2E
## Description
Refactoring and new code additions have led to some unit tests that fail unless testing is run with the proper env vars. Fixup code to either skip those tests or provide them with the proper information so they pass
* some tests moved from unit tests to integration tests
* function to get credentials with bogus info for unit tests
* skip some tests if env vars not passed
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#2052
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
The selectors printable code wasn't being used in any valid way, so that's out. Backup List printing now refers to the DiscreteOwner in the embedded selector as the ResourceOwner reference. Renamed the "Selectors"
column in the List print table to "Resource Owner" to better match what should be contained in that field.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1617
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Sets the default run behavior for exchange to use
incremental backups. The cli feature flag for enabling exchange incrementals has been swapped for a toggle that disables incrementals, forcing a full backup
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Now that we have a mocked api layer in place, we can test filterContainersAndFillCollections without
integration dependencies.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
replaces the new api client methods with interfaces, to prepare for testing funcions with mocks instead of integration.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Previous changes held out on migrating the
resolver iterators into the api package because
they embedded function calls from the resolvers
themselves. This change migrates that code
into the api package, and accepts a callback
function to hook in the resolver updates. This
change sets up resolvers to better utilize an
interface in the next PR.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces. This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.
No logic changes have occurred here. The only
modifications are what is required to utilize the
api client.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
In order to establish a standard api around our
graph client usage, and thus be able to mock
for testing, we need to migrate graph client
usage into another pacakge. This is the first
step in that process of refactoring.
All changes are code relocation importing and
exporting may change as needed. No logic
was altered.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Incremental backups can display progress bars
with 0 items to add or remove. This can cause
end-user confusion when hte bar completes
with 0 items involved. We should instead keep
those bars hidden.
Also removes the "heuristics" progress step.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1984
## Test Plan
- [x] 💪 Manual
## Description
the exchangeService struct is a vestigial prop
from before we had centralized around the
graph.Service, and is no longer needed. These
changes will cascade into the migration of
graph client usage into an api package for mocking.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
the exchange optionIdentifier performs almost
all the same functions as the path category. In
many cases, it is actually a translation of the
other value. This cleanup will cascade into
the creation of an api layer for the graph client.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1967
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a delta test case for exchange incremental
backups by moving one folder into another one.
Also sets up retrieving container IDs in the test
in preparation for other test control.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1966
## Test Plan
- [x] 💚 E2E
## Description
In order to track deleted items, we need to build
separate job sets when aggregating item IDs:
one for added items, and one for removed.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1777
## Test Plan
- [x] 💪 Manual
## Description
Ensure that we do not generate both a regular
collection and a tombstone for the same id.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1923
## Test Plan
- [x] 💚 E2E
## Description
Adds a flag to the BackupCollections interface
that identifies whether the caller is running an
incremental backup or not. If they are, kopia
will utilize the previous base snapshots when
building the directory tree.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] 💚 E2E
## Description
Make sure the options struct is passed through the entire stack so we actually have access to flag values.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1931
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Fixes a bug where we could incorrectly mark a previousPath as both
deleted and moved/created at the same time.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1780
## Test Plan
- [x] 💚 E2E
## Description
Flag doNotMergeItems for exchange collections
where the delta token was marked as reset (ie,
expired or otherwise unusable). Do this instead
of making a (delete, new) collection tuple for
that path.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1914
## Test Plan
- [x] 💚 E2E
## Description
Adds a new func to the data.Collection iface:
DoNotMergeItems() tells kopia to skip the
retention of items from prior snapshots when
generating the new snapshot for this collection.
A primary use case for this flag is when a delta
token expires, preventing an incremental lookup
and forcing gc to re-discover all items in the
container.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1914
## Test Plan
- [x] ⚡ Unit test
## Description
If a delta token expires or is otherwise invalid,
the backup should fall back to the same
behavior as if the collection were new. It must
collect the full delta of information, and create
a collection with New state. If the previous
container moved between the two deltas,
it should be marked for deletion with a tomb-
stone collection.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1780
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Allows the usage of incrementals via feature-
flag controls. Feature flags can be enabled
on a per-flag basis in either the cli (the flag
is hidden from users, normally) or through the
sdk by directly toggling the flag property
in the control.Options.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1901
## Test Plan
- [x] 💪 Manual
## Description
Caches folder info added during details construction in the details builder and keeps the size/modified time
updated as newer items are added.
As part of this, this PR refactors the details package to separate out building `details.Details` from the in-memory representation and model.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
replaces the common/maps.go code with the
golang experimental maps package. The
maps package provides standard api
such as Keys, Values, Copy, and Clone.
https://pkg.go.dev/golang.org/x/exp/maps
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
## Description
moves the DataCollections producer out of
collections and into exchange, along with the
integration tests. The only changes are the
code shuffles, passing down required values,
and the unexporting of funcs that were only
exported for the old design.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1727
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
If an exchange collection only has previous
delta info, or only previous path info, and not
the other, remove that entry from the parsed
metadata map, causing a full backup of the
containere.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1804
## Test Plan
- [x] ⚡ Unit test
## Description
Updating the isRead flag in M365 does not cause the mod time of the item to be updated. This means delta queries will not return items marked as [un]read when isRead is not one of the attributes requested in the $select clause.
Explicitly fetch isRead so that we get those items. Further changes will be required to ensure the mod time of the item is something that will not cause kopia-assisted dedupe to skip the item when only the isRead attribute has been changed.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1866
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#1856 <issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Fulfills the data collection interfaces to provide item deleted
flags and collection state enums.
A note on the additional data consumption: I'm making an assumption
that the @removed property will appear in that map. This is awaiting
testing for verification, which I'll get into as a follow-up step.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1727
## Test Plan
- [x] ❓ In a follow-up PR
## Description
For tests that run subtests, make sure to use the subtest instance of testing.T so that the error trace in the output is easier to read.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1846
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
When backing up Exchange, don't return an error if the folder/calendar we're trying to fetch item IDs for has been deleted.
Error codes pulled with graph explorer API
Manually tested with an Exchange mail backup
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1846
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
These two structures are effectively identical.
This refactor deferrs to the graph version since
services may still wrap that for more specific
behavior as needed.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1727
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
