Сегодня будем учиться делать бэкапы с помощью Heptio ARK. У нас kubernetes находится на AWS
ARK требует наличие S3 бакета для складывания архива состояния пода из etcd
Создаем бакет:
aws s3api create-bucket \
--bucket <YOUR_BUCKET> \
--region <YOUR_REGION> \
--create-bucket-configuration LocationConstraint=<YOUR_REGION>
Создаем IAM юзера:
aws iam create-user --user-name heptio-ark
Применяем ему права:
BUCKET=<YOUR_BUCKET>
cat > heptio-ark-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::${BUCKET}/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::${BUCKET}"
]
}
]
}
EOF
aws iam put-user-policy \
--user-name heptio-ark \
--policy-name heptio-ark \
--policy-document file://heptio-ark-policy.json
Создаем access key и secret key:
aws iam create-access-key --user-name heptio-ark
Получаем:
{
"AccessKey": {
"UserName": "heptio-ark",
"Status": "Active",
"CreateDate": "2017-07-31T22:24:41.576Z",
"SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
"AccessKeyId": <AWS_ACCESS_KEY_ID>
}
}
Cоздаем файл credentials-ark:
[default] aws_access_key_id=<AWS_ACCESS_KEY_ID> aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
Запускаем сервер в Kubernetes
Создаем файл prereqs.yaml:
# Copyright 2017 the Heptio Ark contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: backups.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: backups
kind: Backup
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: schedules.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: schedules
kind: Schedule
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: restores.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: restores
kind: Restore
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: configs.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: configs
kind: Config
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: downloadrequests.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: downloadrequests
kind: DownloadRequest
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: deletebackuprequests.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: deletebackuprequests
kind: DeleteBackupRequest
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: podvolumebackups.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: podvolumebackups
kind: PodVolumeBackup
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: podvolumerestores.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: podvolumerestores
kind: PodVolumeRestore
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: resticrepositories.ark.heptio.com
labels:
component: ark
spec:
group: ark.heptio.com
version: v1
scope: Namespaced
names:
plural: resticrepositories
kind: ResticRepository
---
apiVersion: v1
kind: Namespace
metadata:
name: heptio-ark
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ark
namespace: heptio-ark
labels:
component: ark
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ark
labels:
component: ark
subjects:
- kind: ServiceAccount
namespace: heptio-ark
name: ark
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Деплоим в kubernetes:
kubectl apply -f prereqs.yaml
Создаем kubernetes secret с файла credentials-ark:
kubectl create secret generic cloud-credentials \
--namespace <ARK_NAMESPACE> \
--from-file cloud=credentials-ark
Далее создаем файлы config.yaml и deployment.yaml:
# Copyright 2017 the Heptio Ark contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: ark.heptio.com/v1
kind: Config
metadata:
namespace: heptio-ark
name: default
persistentVolumeProvider:
name: aws
config:
region: <YOUR_REGION>
backupStorageProvider:
name: aws
bucket: <YOUR_BUCKET>
# Uncomment the below line to enable restic integration.
# The format for resticLocation is <bucket>[/<prefix>],
# e.g. "my-restic-bucket" or "my-restic-bucket/repos".
# This MUST be a different bucket than the main Ark bucket
# specified just above.
# resticLocation: <YOUR_RESTIC_LOCATION>
config:
region: <YOUR_REGION>
backupSyncPeriod: 30m
gcSyncPeriod: 30m
scheduleSyncPeriod: 1m
restoreOnlyMode: false
# Copyright 2017 the Heptio Ark contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
namespace: heptio-ark
name: ark
spec:
replicas: 1
template:
metadata:
labels:
component: ark
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8085"
prometheus.io/path: "/metrics"
spec:
restartPolicy: Always
serviceAccountName: ark
containers:
- name: ark
image: gcr.io/heptio-images/ark:latest
command:
- /ark
args:
- server
volumeMounts:
- name: cloud-credentials
mountPath: /credentials
- name: plugins
mountPath: /plugins
- name: scratch
mountPath: /scratch
env:
- name: AWS_SHARED_CREDENTIALS_FILE
value: /credentials/cloud
- name: ARK_SCRATCH_DIR
value: /scratch
volumes:
- name: cloud-credentials
secret:
secretName: cloud-credentials
- name: plugins
emptyDir: {}
- name: scratch
emptyDir: {}
Деплоим в kubernetes:
kubectl apply -f config.yaml kubectl apply -f deployment.yaml
Настройка бэкапов
Создаем файл для ежедневного выполнения daily.yaml:
apiVersion: ark.heptio.com/v1
kind: Schedule
metadata:
name: daily
namespace: heptio-ark
spec:
schedule: 5 0 * * *
template:
excludedNamespaces: null
excludedResources: null
hooks:
resources: null
includeClusterResources: null
includedNamespaces:
- '*'
includedResources: null
labelSelector: null
snapshotVolumes: true
ttl: 168h0m0s
Деплоим:
kubectl apply -f daily.yaml
Проверяем:
kubectl -n heptio-ark get schedules.ark.heptio.com NAME AGE daily 13h
kubectl -n heptio-ark get backups.ark.heptio.com NAME AGE daily-20180814220346 13h daily-20180815000534 11h
В AWS проверяем s3 бакет:
aws s3 ls <BUCKET_NAME>
PRE daily-20180814220346/
PRE daily-20180815000534/
Проверяем snapshots:
aws ec2 describe-snapshots
{
"Description": "",
"Tags": [
{
"Value": "owned",
"Key": "kubernetes.io/cluster/aws.cluster.com"
},
{
"Value": "internal-services",
"Key": "kubernetes.io/created-for/pvc/namespace"
},
{
"Value": "aws.cluster.com",
"Key": "KubernetesCluster"
},
{
"Value": "daily-20180814220346",
"Key": "ark.heptio.com/backup"
},
{
"Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",
"Key": "ark.heptio.com/pv"
},
{
"Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",
"Key": "kubernetes.io/created-for/pv/name"
},
{
"Value": "prometheus-claim0",
"Key": "kubernetes.io/created-for/pvc/name"
},
{
"Value": "aws.cluster.com-dynamic-pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",
"Key": "Name"
}
],
"Encrypted": false,
"VolumeId": "vol-0733b80fe4057339a",
"State": "completed",
"VolumeSize": 489,
"StartTime": "2018-08-14T22:03:49.000Z",
"Progress": "100%",
"OwnerId": "412224592913",
"SnapshotId": "snap-04173b3945360a237"
},
Для удобства можно скачать ark клиент https://github.com/heptio/ark/releases