From a178cb528d96f536b410f5ebfd0c0b8a4c7aa392 Mon Sep 17 00:00:00 2001 From: Lunny Xiao Date: Thu, 30 May 2024 15:33:50 +0800 Subject: [PATCH] Azure blob storage support (#30995) This PR implemented object storages(LFS/Packages/Attachments and etc.) for Azure Blob Storage. It depends on azure official golang SDK and can support both the azure blob storage cloud service and azurite mock server. Replace #25458 Fix #22527 - [x] CI Tests - [x] integration test, MSSQL integration tests will now based on azureblob - [x] unit test - [x] CLI Migrate Storage - [x] Documentation for configuration added ------ TODO (other PRs): - [ ] Improve performance of `blob download`. --------- Co-authored-by: yp05327 <576951401@qq.com> --- administration/config-cheat-sheet.en-us.md | 8 +++++++- administration/config-cheat-sheet.zh-cn.md | 7 ++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/administration/config-cheat-sheet.en-us.md b/administration/config-cheat-sheet.en-us.md index 2c15d161..aabf1b20 100644 --- a/administration/config-cheat-sheet.en-us.md +++ b/administration/config-cheat-sheet.en-us.md @@ -1287,7 +1287,7 @@ is `data/lfs` and the default of `MINIO_BASE_PATH` is `lfs/`. Default storage configuration for attachments, lfs, avatars, repo-avatars, repo-archive, packages, actions_log, actions_artifact. -- `STORAGE_TYPE`: **local**: Storage type, `local` for local disk or `minio` for s3 compatible object storage service. +- `STORAGE_TYPE`: **local**: Storage type, `local` for local disk, `minio` for s3 compatible object storage service, `azureblob` for azure blob storage service. - `SERVE_DIRECT`: **false**: Allows the storage driver to redirect to authenticated URLs to serve files directly. Currently, only Minio/S3 is supported via signed URLs, local does nothing. - `MINIO_ENDPOINT`: **localhost:9000**: Minio endpoint to connect only available when `STORAGE_TYPE` is `minio` - `MINIO_ACCESS_KEY_ID`: Minio accessKeyID to connect only available when STORAGE_TYPE is `minio`. If not provided and STORAGE_TYPE is `minio`, will search for credentials in known environment variables (MINIO_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID), credentials files (~/.mc/config.json, ~/.aws/credentials), and EC2 instance metadata. @@ -1298,6 +1298,12 @@ Default storage configuration for attachments, lfs, avatars, repo-avatars, repo- - `MINIO_INSECURE_SKIP_VERIFY`: **false**: Minio skip SSL verification available when STORAGE_TYPE is `minio` - `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio bucket lookup method defaults to auto mode; set it to `dns` for virtual host style or `path` for path style, only available when STORAGE_TYPE is `minio` +- `AZURE_BLOB_ENDPOINT`: **_empty_**: Azure Blob endpoint to connect only available when STORAGE_TYPE is `azureblob`, + e.g. https://accountname.blob.core.windows.net or http://127.0.0.1:10000/devstoreaccount1 +- `AZURE_BLOB_ACCOUNT_NAME`: **_empty_**: Azure Blob account name to connect only available when STORAGE_TYPE is `azureblob` +- `AZURE_BLOB_ACCOUNT_KEY`: **_empty_**: Azure Blob account key to connect only available when STORAGE_TYPE is `azureblob` +- `AZURE_BLOB_CONTAINER`: **gitea**: Azure Blob container to store the data only available when STORAGE_TYPE is `azureblob` + The recommended storage configuration for minio like below: ```ini diff --git a/administration/config-cheat-sheet.zh-cn.md b/administration/config-cheat-sheet.zh-cn.md index 3c6ac8c0..7d51c758 100644 --- a/administration/config-cheat-sheet.zh-cn.md +++ b/administration/config-cheat-sheet.zh-cn.md @@ -1208,7 +1208,7 @@ ALLOW_DATA_URI_IMAGES = true 默认的附件、lfs、头像、仓库头像、仓库归档、软件包、操作日志、操作艺术品的存储配置。 -- `STORAGE_TYPE`：**local**：存储类型，`local` 表示本地磁盘，`minio` 表示 S3 兼容的对象存储服务。 +- `STORAGE_TYPE`：**local**：存储类型，`local` 表示本地磁盘，`minio` 表示 S3，`azureblob` 表示 azure 对象存储。 - `SERVE_DIRECT`：**false**：允许存储驱动程序重定向到经过身份验证的 URL 以直接提供文件。目前，仅支持通过签名的 URL 提供 Minio/S3，本地不执行任何操作。 - `MINIO_ENDPOINT`：**localhost:9000**：连接的 Minio 终端点，仅在 `STORAGE_TYPE` 为 `minio` 时可用。 - `MINIO_ACCESS_KEY_ID`：Minio 的 accessKeyID，仅在 `STORAGE_TYPE` 为 `minio` 时可用。 @@ -1219,6 +1219,11 @@ ALLOW_DATA_URI_IMAGES = true - `MINIO_INSECURE_SKIP_VERIFY`：**false**：Minio 跳过 SSL 验证，仅在 `STORAGE_TYPE` 为 `minio` 时可用。 - `MINIO_BUCKET_LOOKUP_TYPE`: **auto**: Minio的bucket查找方式默认为`auto`模式，可将其设置为`dns`（虚拟托管样式）或`path`（路径样式），仅当`STORAGE_TYPE`为`minio`时可用。 +- `AZURE_BLOB_ENDPOINT`: **_empty_**: Azure Blob 终端点，仅在 `STORAGE_TYPE` 为 `azureblob` 时可用。例如：https://accountname.blob.core.windows.net 或 http://127.0.0.1:10000/devstoreaccount1 +- `AZURE_BLOB_ACCOUNT_NAME`: **_empty_**: Azure Blob 账号名，仅在 `STORAGE_TYPE` 为 `azureblob` 时可用。 +- `AZURE_BLOB_ACCOUNT_KEY`: **_empty_**: Azure Blob 访问密钥，仅在 `STORAGE_TYPE` 为 `azureblob` 时可用。 +- `AZURE_BLOB_CONTAINER`: **gitea**: 用于存储数据的 Azure Blob 容器名，仅在 `STORAGE_TYPE` 为 `azureblob` 时可用。 + 建议的 minio 存储配置如下： ```ini