Automate running backups and restores w/ fusion #2157

bluzarraga · 2024-08-22T15:18:15Z

What this PR does / why we need it:
Creates an automation script to trigger and wait for Fusion backups and restores

Which issue(s) this PR fixes:
Work for https://github.ibm.com/IBMPrivateCloud/roadmap/issues/64245 & https://github.ibm.com/IBMPrivateCloud/roadmap/issues/64247

Special notes for your reviewer:

How the test is done?
Setup cluster using instructions from Automate setup of Hub and Spoke clusters for backup and restore using Spectrum Fusion #2142
Update variables in script to match expected (ie BACKUP_STORAGE_LOCATION_NAME)
run auto-br.sh with appropriate parameters
verify backup and restore complete

Outstanding items:

Usage function needs to be updated
Variables at the top of the script need to either be parameterized or updated via env.properties
prereq checks for proper variables set before running the script
update logging statements
clean up commented out code and remove defaults for values that shouldn't have one

How to backport this PR to other branch:

Add label to this PR with the target branch name backport <branch-name>
The PR will be automatically created in the target branch after merging this PR
If this PR is already merged, you can still add the label with the target branch name backport <branch-name> and leave a comment /backport to trigger the backport action

…ce install Signed-off-by: Ben Luzarraga <[email protected]>

Signed-off-by: Ben Luzarraga <[email protected]>

bluzarraga · 2024-08-22T15:21:09Z

This pr is a bit ugly because I started work on this script before 2142 was merged in. The changes specific to this pr start from f9ce24b. I also opened the same pr in my fork branch for clarity on the difference https://github.com/bluzarraga/ibm-common-service-operator/pull/4/files

YCShen1010 · 2024-08-27T19:16:04Z

I have passed the test for Backup automation part.

Result:

./auto-br.sh --backup --backup-name cs-application-backuo-test --cluster-type hub --target-cluster cutie1
All arguments passed into the auto-br.sh: --backup --backup-name cs-application-backuo-test --cluster-type hub --target-cluster cutie1


[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
# Creating Spectrum Fusion backup resource for hub cluster.
[INFO] Copying template files...
[INFO] Editing backup yaml...
backup.data-protection.isf.ibm.com/cs-application-backuo-test unchanged
[✔] Backup cs-application-backuo-test successfully applied on hub server https://api.cutie1.cp.fyre.ibm.com:6443 to backup target cluster cutie1
# Waiting for backup cs-application-backuo-test to complete...
Completed && Completed && oc get backup.data-protection.isf.ibm.com cs-application-backuo-test -n ibm-spectrum-fusion-ns -o jsonpath='{.status.phase}'
[INFO] backup cs-application-backuo-test can be further tracked in the UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backuo-test
[✔] backup cs-application-backuo-test completed successfully for cutie1.
[INFO] For more info, see job in the UI (https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backuo-test) or use "oc get backup cs-application-backuo-test -n ibm-spectrum-fusion-ns -o yaml | yq '.status'".
[✔] Backup cs-application-backuo-test of cluster cutie1 completed. See results in Fusion UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backuo-test

br-testing-automation/auto-br.sh

Signed-off-by: Allen Li <[email protected]>

qpdpQ · 2024-09-06T20:28:07Z

it should be good now

Signed-off-by: Ben Luzarraga <[email protected]>

bluzarraga · 2024-09-11T19:47:49Z

Updated some of the parameters and prereq checking after testing the script. I've got it running backups and restores consistently so I think this is ready to merge and ready to hand over to SERT team for further testing/use

YCShen1010 · 2024-09-12T18:46:06Z

Backup test passed

./auto-br.sh --backup --backup-name cs-application-backup-test
All arguments passed into the auto-br.sh: --backup --backup-name cs-application-backup-test

[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
# Creating Spectrum Fusion backup.data-protection.isf.ibm.com resource for  cluster.
[INFO] Copying template files...
[INFO] Editing backup yaml...
backup.data-protection.isf.ibm.com/cs-application-backup-test created
[✔] Backup cs-application-backup-test successfully applied on hub server https://api.cutie1.cp.fyre.ibm.com:6443 to backup target cluster 
# Waiting for backup.data-protection.isf.ibm.com cs-application-backup-test to complete...
[INFO] backup.data-protection.isf.ibm.com cs-application-backup-test can be further tracked in the UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backup-test
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: InventoryInProgress
[INFO] Current sequence status:
...
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: RecipeInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: SnapshotInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: SnapshotInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: DataTransferInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: DataTransferInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: DataTransferInProgress
[INFO] Waiting on backup.data-protection.isf.ibm.com cs-application-backup-test to complete. Current status: Completed
[✔] backup.data-protection.isf.ibm.com cs-application-backup-test completed successfully for .
[INFO] For more info, see job in the UI (https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backup-test) or use "oc get backup.data-protection.isf.ibm.com cs-application-backup-test -n ibm-spectrum-fusion-ns -o yaml | yq '.status'".
[✔] Backup cs-application-backup-test of cluster  completed. See results in Fusion UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/backups/cs-application-backup-test

YCShen1010 · 2024-09-12T18:50:07Z

I have another question when testing restore, from the template I see some operator and services namespace are hard code like cs-serv, cs-op, are we availabe to replace this namespaces with our own custom namespace?
I have tested restore process with my own namespace and got restore status failed:
Failed validation There was an error when processing the job in the Backup service

spec:
  backup: cs-application-backup-test
  objectsToRestore:
    RESOURCES:
      - ALL
    v1/persistentvolumeclaim:
      - cs-op/setup-tenant-job-pvc
      - cs-serv/cs-db-backup-pvc
      - cs-serv/zen5-backup-pvc
      - ibm-lsr/lsr-backup-pvc

Signed-off-by: Allen Li <[email protected]>

qpdpQ · 2024-09-12T19:11:45Z

update the script to modify the pvc namespaces

Signed-off-by: Allen Li <[email protected]>

YCShen1010 · 2024-09-12T21:19:21Z

Restore process also passed:

./auto-br.sh --restore --backup-name cs-application-backup-test --restore-name cs-application-backup-test-1 --cluster-type spoke --target-cluster apps.cutie1.cp.fyre.ibm.com
All arguments passed into the auto-br.sh: --restore --backup-name cs-application-backup-test --restore-name cs-application-backup-test-1 --cluster-type spoke --target-cluster apps.cutie1.cp.fyre.ibm.com

[✔] oc command available
[✔] yq command available
[✔] oc command logged in as kube:admin
# Creating Spectrum Fusion restore.data-protection.isf.ibm.com resource for spoke cluster.
[INFO] Editing restore yaml...
restore.data-protection.isf.ibm.com/cs-application-backup-test-1 created
[✔] Restore cs-application-backup-test-1 successfully applied on hub server https://api.cutie1.cp.fyre.ibm.com:6443 to restore target cluster apps.cutie1.cp.fyre.ibm.com
# Waiting for restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete...
[INFO] restore.data-protection.isf.ibm.com cs-application-backup-test-1 can be further tracked in the UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/restores/cs-application-backup-test-1
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: InventoryInProgress
[INFO] Current sequence status:
null
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Current sequence status:
...
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestorePvcsInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: RestoreEtcdInProgress
[INFO] Waiting on restore.data-protection.isf.ibm.com cs-application-backup-test-1 to complete. Current status: Completed
[✔] restore.data-protection.isf.ibm.com cs-application-backup-test-1 completed successfully for apps.cutie1.cp.fyre.ibm.com.
[INFO] For more info, see job in the UI (https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/restores/cs-application-backup-test-1) or use "oc get restore.data-protection.isf.ibm.com cs-application-backup-test-1 -n ibm-spectrum-fusion-ns -o yaml | yq '.status'".
[✔] Restore cs-application-backup-test-1 to cluster apps.cutie1.cp.fyre.ibm.com completed. See results in Fusion UI here: https://console-ibm-spectrum-fusion-ns.apps.cutie1.cp.fyre.ibm.com/backupAndRestore/jobs/restores/cs-application-backup-test-1

YCShen1010

/lgtm

ibm-ci-bot · 2024-09-12T21:20:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bluzarraga, YCShen1010

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [YCShen1010,bluzarraga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bluzarraga added 30 commits August 5, 2024 10:59

initialize backup env setup script. Add sc validation and sf br servi…

5bdfdc6

…ce install Signed-off-by: Ben Luzarraga <[email protected]>

make executable

ea2b3b0

Signed-off-by: Ben Luzarraga <[email protected]>

add ability to apply SF resources

471a48e

Signed-off-by: Ben Luzarraga <[email protected]>

change file copy

6897eed

Signed-off-by: Ben Luzarraga <[email protected]>

debug applying secret

a5b9454

Signed-off-by: Ben Luzarraga <[email protected]>

remove templates dir

7ab8688

Signed-off-by: Ben Luzarraga <[email protected]>

need encoded

b57d927

Signed-off-by: Ben Luzarraga <[email protected]>

add CPFS resource labeling and deployment

0ed5267

Signed-off-by: Ben Luzarraga <[email protected]>

test without changing directory

31fc458

Signed-off-by: Ben Luzarraga <[email protected]>

update deploy br resources script name

f2a8afa

Signed-off-by: Ben Luzarraga <[email protected]>

add base dir for use outside of velero dir

3005206

Signed-off-by: Ben Luzarraga <[email protected]>

include lsr

b6eebad

Signed-off-by: Ben Luzarraga <[email protected]>

change json path to yq due to inconsistent results

a87d31b

Signed-off-by: Ben Luzarraga <[email protected]>

update application and recipe yamls

f39c1c7

Signed-off-by: Ben Luzarraga <[email protected]>

update lsr recipe

09a6987

Signed-off-by: Ben Luzarraga <[email protected]>

update zen br, remove unused cm items

8e9f221

Signed-off-by: Ben Luzarraga <[email protected]>

update cert manager resource def

bd1c7b0

Signed-off-by: Ben Luzarraga <[email protected]>

install spoke cluster

5fd0318

Signed-off-by: Ben Luzarraga <[email protected]>

apply spoke yaml

6919e48

Signed-off-by: Ben Luzarraga <[email protected]>

remove newline character from secret encoding

0363683

Signed-off-by: Ben Luzarraga <[email protected]>

remove newline character

1a6a6b1

Signed-off-by: Ben Luzarraga <[email protected]>

add wait logic for spoek cluster install

abe8d1f

Signed-off-by: Ben Luzarraga <[email protected]>

move wait to after applying file

389844f

Signed-off-by: Ben Luzarraga <[email protected]>

update setup tenant job deployment to include parameters

3c41548

Signed-off-by: Ben Luzarraga <[email protected]>

update cpfs utils image version

bd18b25

Signed-off-by: Ben Luzarraga <[email protected]>

add lsr resources to backup

d1d7b6e

Signed-off-by: Ben Luzarraga <[email protected]>

log back into hub cluster on failed spoke install attempt

c34ea50

Signed-off-by: Ben Luzarraga <[email protected]>

typo

dfd9bdf

Signed-off-by: Ben Luzarraga <[email protected]>

typo

c2e5748

Signed-off-by: Ben Luzarraga <[email protected]>

change how tethered ns is checked

cff6f70

Signed-off-by: Ben Luzarraga <[email protected]>

ibm-ci-bot added approved size/XXL labels Aug 22, 2024

Merge branch 'scripts-dev' into scriptsfbr

bc52ec3

ibm-ci-bot added size/L and removed size/XXL labels Aug 22, 2024

YCShen1010 reviewed Aug 27, 2024

View reviewed changes

br-testing-automation/auto-br.sh Show resolved Hide resolved

qpdpQ added 4 commits September 4, 2024 13:04

update backup/restore type

1c5d99d

Signed-off-by: Allen Li <[email protected]>

update script usage

6b311c0

Signed-off-by: Allen Li <[email protected]>

add more pre-request checks

b81c002

Signed-off-by: Allen Li <[email protected]>

clean up commented out

a80fc97

Signed-off-by: Allen Li <[email protected]>

bluzarraga added 3 commits September 11, 2024 13:58

tweak variable checking and usage output

30fc612

Signed-off-by: Ben Luzarraga <[email protected]>

update parameters

85ef4f3

Signed-off-by: Ben Luzarraga <[email protected]>

update default CPFS version

c3f4056

Signed-off-by: Ben Luzarraga <[email protected]>

bluzarraga changed the title ~~[WIP] Automate running backups and restores w/ fusion~~ Automate running backups and restores w/ fusion Sep 11, 2024

ibm-ci-bot removed the do-not-merge/work-in-progress label Sep 11, 2024

update restore pvc

705de59

Signed-off-by: Allen Li <[email protected]>

typo in lsr ns

9b621d8

Signed-off-by: Allen Li <[email protected]>

YCShen1010 approved these changes Sep 12, 2024

View reviewed changes

ibm-ci-bot assigned YCShen1010 Sep 12, 2024

ibm-ci-bot added the lgtm label Sep 12, 2024

ibm-ci-bot merged commit a61e662 into IBM:scripts-dev Sep 12, 2024
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate running backups and restores w/ fusion #2157

Automate running backups and restores w/ fusion #2157

bluzarraga commented Aug 22, 2024 •

edited by qpdpQ

Loading

bluzarraga commented Aug 22, 2024

YCShen1010 commented Aug 27, 2024

qpdpQ commented Sep 6, 2024

bluzarraga commented Sep 11, 2024

YCShen1010 commented Sep 12, 2024 •

edited

Loading

YCShen1010 commented Sep 12, 2024 •

edited

Loading

qpdpQ commented Sep 12, 2024

YCShen1010 commented Sep 12, 2024

YCShen1010 left a comment

ibm-ci-bot commented Sep 12, 2024

Automate running backups and restores w/ fusion #2157

Automate running backups and restores w/ fusion #2157

Conversation

bluzarraga commented Aug 22, 2024 • edited by qpdpQ Loading

bluzarraga commented Aug 22, 2024

YCShen1010 commented Aug 27, 2024

qpdpQ commented Sep 6, 2024

bluzarraga commented Sep 11, 2024

YCShen1010 commented Sep 12, 2024 • edited Loading

YCShen1010 commented Sep 12, 2024 • edited Loading

qpdpQ commented Sep 12, 2024

YCShen1010 commented Sep 12, 2024

YCShen1010 left a comment

Choose a reason for hiding this comment

ibm-ci-bot commented Sep 12, 2024

bluzarraga commented Aug 22, 2024 •

edited by qpdpQ

Loading

YCShen1010 commented Sep 12, 2024 •

edited

Loading

YCShen1010 commented Sep 12, 2024 •

edited

Loading