diff --git a/README.md b/README.md index 3e2905473..3356dbdfd 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,18 @@ # Corso -Corso is the first open-source tool that aims to assist admins with the -critical task of protecting SaaS service data. It provides a reliable, secure, -and efficient data protection engine. Admins decide where backup data is -stored and have the flexibility to execute backups of their desired service -through an intuitive interface. As Corso evolves, it can become a great -building block for more complex data protection workflows. +Corso is the first open-source tool that aims to assist IT admins with the critical task of protecting their +Microsoft 365 data. It provides a reliable, secure, and efficient data protection engine. Admins decide where backup +data is stored and have the flexibility to perform backups of their desired service through an intuitive interface. +As Corso evolves, it can become a great building block for more complex data protection workflows. -Corso is initially focused on protecting data in Microsoft 365 services such -as Exchange, OneDrive, SharePoint, and Teams. The goals is to expand coverage -for additional services, M365 or others, over time based on the interest and -needs of the community. +Corso supports supports M365 Exchange, OneDrive, SharePoint, and Teams. Coverage for additional services, possibly +even beyond M365, will expand based on the interest and needs of the community. -# Getting Started +## Getting Started TODO - Link to the appropriate page in the published docs. -# Building Corso +## Building Corso ```sh # Build a binary. Will be placed in bin/ @@ -26,17 +22,17 @@ TODO - Link to the appropriate page in the published docs. ./build/build-container.sh ``` -# Contribution Guidelines +## Contribution Guidelines TODO -# Code of Conduct +## Code of Conduct -It is important that our community is inclusive and respectful of everyone. +It's important that our community is inclusive and respectful of everyone. We ask that all Corso users and contributors take a few minutes to review our [Code of Conduct](CODE_OF_CONDUCT.md) -# License +## License TODO ` diff --git a/docs/.markdownlint.json b/docs/.markdownlint.json index 933f52b79..645ab8403 100644 --- a/docs/.markdownlint.json +++ b/docs/.markdownlint.json @@ -1,3 +1,6 @@ { - "MD013": { "line_length": 120 } + "MD013": { "line_length": 120 }, + "no-inline-html": { + "allowed_elements": [ "TOCInline", "br", "img" ] + }, } diff --git a/docs/README.md b/docs/README.md index b7be8fdaa..7c475d388 100644 --- a/docs/README.md +++ b/docs/README.md @@ -34,11 +34,20 @@ This command generates static content into the `build` directory for integration ## Style and linting +```bash +# Lint all docs +make dockercheck +# Lint specific files and/or folders +make dockercheck VALE_TARGET="README.md docs/concepts" +``` + +This command will lint all Markdown files and check them for style issues using the Docker container + ```bash make check ``` -This command will lint all Markdown files and check them for style issues. +Same as `make dockercheck` but runs locally. Requires `vale` to be installed. ## Documentation platform development diff --git a/docs/docs/concepts/_category_.json b/docs/docs/concepts/_category_.json deleted file mode 100644 index af6ca99a5..000000000 --- a/docs/docs/concepts/_category_.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "label": "Concepts", - "position": 2, - "link": { - "type": "generated-index", - "description": "Core Corso Concepts." - } -} diff --git a/docs/docs/concepts/corso-repository-and-configuration.md b/docs/docs/concepts/corso-repository-and-configuration.md deleted file mode 100644 index 8557e4c30..000000000 --- a/docs/docs/concepts/corso-repository-and-configuration.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Repositories and configuration - -## Object storage configuration - -Corso stores all backup data in an object storage system. Corso supports Amazon Web -Services (AWS) S3, Google Cloud Storage (GCS), Azure Blob, and S3-compatible systems. - -```mermaid -graph TD; - Corso-->ObjectStorage; -``` - -### Amazon S3 and S3-compatible systems - -Corso needs the following permissions for AWS S3 and S3-compatible object storage systems: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "s3:PutObject", - "s3:GetObject", - "s3:ListBucket", - "s3:DeleteObject", - "s3:GetBucketLocation", - ], - "Resource": [ - "arn:aws:s3:::${BUCKET_NAME}", - "arn:aws:s3:::${BUCKET_NAME}/*" - ] - } - ] -} -``` diff --git a/docs/docs/configuration/concepts.md b/docs/docs/configuration/concepts.md new file mode 100644 index 000000000..1a8fd6687 --- /dev/null +++ b/docs/docs/configuration/concepts.md @@ -0,0 +1,27 @@ +--- +description: "Core Corso concepts." +--- + +# Concepts + +Before using Corso, it's important to familiarize yourself with some key concepts. + +## Microsoft 365 concepts {#m365-concepts} + +* **M365 Tenant** is typically associated with a unique domain (for example, `contoso.com`) and represents a dedicated +and logically segregated instance of the Microsoft 365 services plus associated data available to your organization. + +* **M365 Service** refer to a cloud-applications available through the Microsoft 365 platform. Corso supports +backup and recovery for Exchange Online, OneDrive, SharePoint, and Teams. + +* **Azure AD Application** represents an Azure AD digital identity/service principal and associated configuration which +define the accessible resources and permitted actions on these resources through the application. Corso uses an Azure AD +application to connect to your *M365 tenant* and transfer data during backup and restore operations. + +## Corso concepts {#corso-concepts} + +* **Repository** refers to the storage location where Corso securely and efficiently stores encrypted *backups* of your +*M365 Services* data. See [Repositories](/configuration/repos) for more information. + +* **Backup** is a copy of your *M365 Services* data that can be restored if the original data is deleted, lost, or +corrupted. Corso performs backups incrementally; each backup only captures data that has changed between backup iterations. diff --git a/docs/docs/configuration/m365_access.md b/docs/docs/configuration/m365_access.md new file mode 100644 index 000000000..ab51b1bf4 --- /dev/null +++ b/docs/docs/configuration/m365_access.md @@ -0,0 +1,78 @@ +--- +description: "Connect to M365 Tenant" +--- + +# M365 access + +To perform backup and restore operations, Corso requires access to your [M365 tenant](/concepts#m365-concepts) +through a properly configured [Azure AD application](/concepts#m365-concepts). + +## Create Azure AD application + +For the official documentation for adding an Azure AD Application and Service Principal using the Azure Portal see +[here](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal). + +The following steps outline a simplified procedure for creating an Azure Ad application suitable for use with Corso. + +1. **Create a new application** + + Select **Azure Active Directory → App Registrations → New Registration** + + +2. **Configure basic settings** + + * Give the application a name + * Select **Accounts in this organizational directory only** + * Skip the **Redirect URI** option + +
+ +3. **Configure required permissions** + + Select **App Permission** from the app management panel. + + + + The required permissions are as follows: + + **TODO: Complete list of permissions** + + | API / Permissions Name | Type | + |--|--| + | Permission 1 | Application | + | Permission 2 | Delegated | + +4. **Grant admin consent** + + + +## Export application credentials + +Now that the Corso Azure AD application is configured, you need to capture as environment variables the information that +Corso will need to connect to the application. + +### Tenant ID and client ID + +To obtain these, select Overview from the app management panel and export the corresponding environment variables. + +```bash +export AZURE_TENANT_ID= +export AZURE_CLIENT_ID= +``` + + + +### Azure client secret + +Lastly, you need to configure a client secret associated with the app using **Certificates & Secrets** from the app +management panel. + +Click **New Client Secret** and follow the instructions to create a secret. After creating the secret, copy the secret +value right away because it's only available after creation. In the end, export the secret value as an environment +variable. + +```bash +export AZURE_CLIENT_SECRET= +``` + + diff --git a/docs/docs/configuration/repos.md b/docs/docs/configuration/repos.md new file mode 100644 index 000000000..af3023016 --- /dev/null +++ b/docs/docs/configuration/repos.md @@ -0,0 +1,73 @@ +--- +description: "Configure backup repository" +--- + +# Repositories + +A Corso [repository](/concepts#corso-concepts) stores encrypted copies of your backup data. Repositories are +supported on the following object storage systems: + +import TOCInline from '@theme/TOCInline'; + +
+ +:::note +Depending on community interest, Corso may support other object storage backends in the future. +::: + +## Amazon S3 + +### Prerequisites + +Before setting you your Corso S3 repository, the following prerequisites must be met: + +* S3 bucket for the repository already exists. Corso won't create it for you. +* You have access to credentials for a user or an IAM role that represent the following permissions + +**TODO: Verify if these permissions are correct? What about multi-part upload permissions?** + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:ListBucket", + "s3:DeleteObject", + "s3:GetBucketLocation", + "s3:AbortMultipartUpload", + "s3:ListMultipartUploadParts", + "s3:ListBucketMultipartUploads" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + } + ] +} +``` + +### Credential setup {#s3-creds-setup} + +Corso supports the credential options offered by the Go SDK. For Full details, see the *Specifying Credentials* +section of the [official documentation](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html). + +* **Environment variables** - set and export `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. If using temporary + credentials derived by assuming an IAM Role, you will also need `AWS_SESSION_TOKEN`. + +* **Credentials file** - ensure that the credentials file is available to Corso (for example, may need to map it if + using Corso as a container). You may also want to set and export `AWS_PROFILE`, if not using the default profile, and + `AWS_SHARED_CREDENTIALS_FILE`, if not using the default file location. You can learn more about the AWS CLI + environment variables [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html). + +### Initialize repository + +Before first use, you need to initialize a Corso repository with `corso repo init s3`. See command details +[here](/cli/corso_repo_init_s3). + +If a repository already exists, you can connect to it with `corso repo connect s3`. See command details +[here](/cli/corso_repo_connect_s3). diff --git a/docs/docs/developers/architecture.md b/docs/docs/developers/architecture.md new file mode 100644 index 000000000..7fb2ee7ae --- /dev/null +++ b/docs/docs/developers/architecture.md @@ -0,0 +1,3 @@ +# Architecture + +TODO diff --git a/docs/docs/developers/build.md b/docs/docs/developers/build.md new file mode 100644 index 000000000..c42f179fb --- /dev/null +++ b/docs/docs/developers/build.md @@ -0,0 +1,3 @@ +# Build from source + +TODO diff --git a/docs/docs/developers/linters.md b/docs/docs/developers/linters.md index 8f921bb83..9d20c17f2 100644 --- a/docs/docs/developers/linters.md +++ b/docs/docs/developers/linters.md @@ -1,7 +1,3 @@ ---- -sidebar_position: 1 ---- - # Corso linters Corso uses the golangci-lint GitHub action to run linters on every PR to `main`. @@ -79,7 +75,7 @@ This applies to golangci-lint v1.45.2 for the `gci` linter and is due to an impo ordering issue. It occurs because imports in the file aren't grouped according to the import rules for Corso. Corso code should have three distinct import groups, system imports, third party imports, and imports of other Corso code -like below. The most likely cause of a `gci` lint error is a Corso import in the +like below. Typically the cause of a `gci` lint error is a Corso import in the block for third party libraries. ```go diff --git a/docs/docs/developers/testing.md b/docs/docs/developers/testing.md index 3c87d4287..a9190f4b7 100644 --- a/docs/docs/developers/testing.md +++ b/docs/docs/developers/testing.md @@ -1,11 +1,15 @@ # Running tests locally ## Prerequisites + - Set `CORSO_PASSWORD` environment variable + ```bash export CORSO_PASSWORD= ``` + - Set AWS credential (needed for tests that use S3) environment variables + ```bash export AWS_ACCESS_KEY_ID="...." export AWS_SECRET_ACCESS_KEY="..." @@ -13,34 +17,39 @@ ``` - Create a config file with the S3 bucket used for testing + ```toml bucket = '' ``` + - Set `CORSO_TEST_CONFIG_FILE` to use the test config file + ```bash export CORSO_TEST_CONFIG_FILE=~/.corso_test.toml ``` + - Set M365 Credentials environment variables + ```bash export TENANT_ID= export CLIENT_ID= export CLIENT_SECRET= ``` -## Running Tests +## Running tests + Standard `go test ./...` will run unit tests -Integration style tests are configured to run only if enabled by setting the -appropriate ENV variable. +Integration style tests run when enabled by setting the appropriate environment variable. -e.g. `CORSO_CI_TESTS=true go test ./...` +For example, `CORSO_CI_TESTS=true go test ./...` The complete list of environment constants is available at `.../src/internal/tester/integration_runners.go`. -## Advanced Options +## Advanced options -- Use `CORSO_M356_TEST_USER_ID` to override the M365 user tests are run against +- To override the M365 user for tests, use `CORSO_M356_TEST_USER_ID` ```bash export CORSO_M356_TEST_USER_ID="..." diff --git a/docs/docs/install.md b/docs/docs/install.md new file mode 100644 index 000000000..fbc4278fe --- /dev/null +++ b/docs/docs/install.md @@ -0,0 +1,83 @@ +# Installation + +Corso releases are available using the following options: + +import TOCInline from '@theme/TOCInline'; + +
+ +:::note + +To maximize portability across platforms, Corso is available as a container image. In the future, +releases may also be available as operating system specific pre-built binaries. + +In the meantime, if you want to run Corso as a binary, refer to the +[instructions on how to build from source](developers/build). + +::: + +## Docker image + +To use Corso as a Docker image, you need to have [Docker installed](https://docs.docker.com/engine/install/) +on your machine. + +### Docker command + +To run the Corso container, it's recommended that you: + +* Export [Corso key configuration environment variables](cli/corso_env) and add their names to an +[environment variables file](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file) +* Map a local directory to `/app/corso`. Corso will look for or create the `corso.toml` config file there. This will preserve + configuration across container runs. Corso will use the directoy for logs, if enabled. + +To create the environment variables file, you can run the following. + +```bash +# create an env vars file +$ cat < +``` + +:::note +Your first backup may take some time if your mailbox has many items so please be patient. +::: + +**TODO:** Update ^^^ after Corso output from operations is finalized. + +## Restore an email + +Now lets explore how you can restore data from one of your backups. + +You can see all Exchange backups available with the following command: + +```bash +$ docker run -e CORSO_PASSPHRASE \ + --env-file ~/.corso/corso.env \ + -v ~/.corso/config:/app/config \ + -v ~/.corso/logs:/app/logs corso/corso:latest \ + backup list exchange --user +``` + +**TODO:** Update after Corso output from operations is finalized. + +Select one of the available backups and search through its contents. + +```bash +$ docker run -e CORSO_PASSPHRASE \ + --env-file ~/.corso/corso.env \ + -v ~/.corso/config:/app/config \ + -v ~/.corso/logs:/app/logs corso/corso:latest \ + backup details exchange \ + --backup \ + --user \ + --email-subject +``` + +The above should give you a list of any matching emails. Note the ID of the one you would like to +use for testing restore. + +When you are ready to restore, use the following command: + +```bash +$ docker run -e CORSO_PASSPHRASE \ + --env-file ~/.corso/corso.env \ + -v ~/.corso/config:/app/config \ + -v ~/.corso/logs:/app/logs corso/corso:latest \ + backup details exchange \ + --backup \ + --user \ + --email +``` + +You can now find the recovered email in a folder named "Corso_Restore_DD-MMM-YYYY_HH:MM:SS" in your mailbox. + +You are now ready to explore the [Command Line Reference](cli) and try everything that Corso can do for you. diff --git a/docs/docusaurus.config.js b/docs/docusaurus.config.js index 6e359cfc8..1cfc68f42 100644 --- a/docs/docusaurus.config.js +++ b/docs/docusaurus.config.js @@ -12,7 +12,7 @@ const config = { baseUrl: '/', onBrokenLinks: 'throw', onBrokenMarkdownLinks: 'throw', - favicon: 'img/corso.svg', + favicon: 'img/corso_logo.svg', // GitHub pages deployment config. // If you aren't using GitHub pages, you don't need these. @@ -26,6 +26,7 @@ const config = { defaultLocale: 'en', locales: ['en'], }, + staticDirectories: ['static', 'public'], presets: [ [ @@ -54,8 +55,8 @@ const config = { title: 'Corso', logo: { alt: 'Corso Logo', - src: 'img/corso.svg', - srcDark: 'img/corso_white.svg', + src: '/img/corso_logo.svg', + srcDark: 'img/corso_logo_white.svg', }, items: [ { diff --git a/docs/sidebars.js b/docs/sidebars.js index eeade5309..1c415299b 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -14,18 +14,32 @@ /** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */ const sidebars = { // By default, Docusaurus generates a sidebar from the docs folder structure - docsSidebar: [{type: 'autogenerated', dirName: '.'}], - - // But you can create a sidebar manually - /* - tutorialSidebar: [ + docsSidebar: [ + 'intro', + 'install', { type: 'category', - label: 'Tutorial', - items: ['hello'], + label: 'Initial Configuration', + items: ['configuration/concepts', 'configuration/m365_access', 'configuration/repos'], }, + { + type: 'category', + label: 'Command Line Reference', + items: [ + 'cli/corso_repo_init_s3', 'cli/corso_repo_connect_s3', + 'cli/corso_backup_create_exchange', 'cli/corso_backup_list_exchange', 'cli/corso_backup_details_exchange', + 'cli/corso_restore_exchange', + ] + }, + { + type: 'category', + label: 'Developer Guide', + items: [ + 'developers/architecture', 'developers/build', 'developers/testing', 'developers/linters' + ], + }, + ], - */ }; module.exports = sidebars; diff --git a/docs/src/css/custom.css b/docs/src/css/custom.css index 2bc6a4cfd..43c8ccfe8 100644 --- a/docs/src/css/custom.css +++ b/docs/src/css/custom.css @@ -28,3 +28,9 @@ --ifm-color-primary-lightest: #4fddbf; --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3); } + +.guideImages { + max-width: 650px; + width: 100%; + border: 1px solid #80808029 +} \ No newline at end of file diff --git a/docs/static/img/corso_logo.svg b/docs/static/img/corso_logo.svg new file mode 100755 index 000000000..161de0c00 --- /dev/null +++ b/docs/static/img/corso_logo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/static/img/corso_logo_white.svg b/docs/static/img/corso_logo_white.svg new file mode 100755 index 000000000..cad446b3b --- /dev/null +++ b/docs/static/img/corso_logo_white.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/static/img/logo.svg b/docs/static/img/logo.svg deleted file mode 100644 index 9db6d0d06..000000000 --- a/docs/static/img/logo.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/docs/static/img/m365app_configure.png b/docs/static/img/m365app_configure.png new file mode 100644 index 000000000..1e2998712 Binary files /dev/null and b/docs/static/img/m365app_configure.png differ diff --git a/docs/static/img/m365app_consent.png b/docs/static/img/m365app_consent.png new file mode 100644 index 000000000..ccf58aef0 Binary files /dev/null and b/docs/static/img/m365app_consent.png differ diff --git a/docs/static/img/m365app_create_new.png b/docs/static/img/m365app_create_new.png new file mode 100644 index 000000000..805725a53 Binary files /dev/null and b/docs/static/img/m365app_create_new.png differ diff --git a/docs/static/img/m365app_ids.png b/docs/static/img/m365app_ids.png new file mode 100644 index 000000000..f7e4421ac Binary files /dev/null and b/docs/static/img/m365app_ids.png differ diff --git a/docs/static/img/m365app_permissions.png b/docs/static/img/m365app_permissions.png new file mode 100644 index 000000000..d8135c427 Binary files /dev/null and b/docs/static/img/m365app_permissions.png differ diff --git a/docs/static/img/m365app_secret.png b/docs/static/img/m365app_secret.png new file mode 100644 index 000000000..997f5267c Binary files /dev/null and b/docs/static/img/m365app_secret.png differ