Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make mirror-images-qe-test-images re-usable outside QE env #56937

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+43 −16
Open

Make mirror-images-qe-test-images re-usable outside QE env #56937

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 36 additions & 16 deletions ...rator/step-registry/mirror-images/qe-test-images/mirror-images-qe-test-images-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,22 @@ trap 'CHILDREN=$(jobs -p); if test -n "${CHILDREN}"; then kill ${CHILDREN} && wa

new_pull_secret="${SHARED_DIR}/new_pull_secret"

if [ ! -f "${SHARED_DIR}/mirror_registry_url" ]; then
echo "File ${SHARED_DIR}/mirror_registry_url does not exist, skipping..."
exit 0
fi
# private mirror registry host
# <public_dns>:<port>
MIRROR_REGISTRY_HOST=`head -n 1 "${SHARED_DIR}/mirror_registry_url"`
if [ ! -f "${SHARED_DIR}/mirror_registry_url" ]; then
echo "File ${SHARED_DIR}/mirror_registry_url does not exist."
exit 1
fi
echo "MIRROR_REGISTRY_HOST: $MIRROR_REGISTRY_HOST"

# since ci-operator gives steps KUBECONFIG pointing to cluster under test under some circumstances,
# unset KUBECONFIG to ensure this step always interact with the build farm.
unset KUBECONFIG

# Create qe test image list
cat <<EOF > "/tmp/mirror-images-list.yaml"
if [ "${QE_IMAGES}" == "true" ]; then
echo "Create qe test image list"
cat <<EOF > "/tmp/mirror-images-list.yaml"
quay.io/openshifttest/iperf3:latest=MIRROR_REGISTRY_PLACEHOLDER/openshifttest/iperf3:latest
quay.io/openshifttest/hello-sdn:1.2.0=MIRROR_REGISTRY_PLACEHOLDER/openshifttest/hello-sdn:1.2.0
quay.io/openshifttest/hello-openshift-fedora:latest=MIRROR_REGISTRY_PLACEHOLDER/openshifttest/hello-openshift-fedora:latest
Expand Down Expand Up @@ -98,6 +99,18 @@ quay.io/openshifttest/uiauto-operators-index:1.2.0=MIRROR_REGISTRY_PLACEHOLDER/o
quay.io/openshifttest/ui-auto-operators:latest=MIRROR_REGISTRY_PLACEHOLDER/openshifttest/ui-auto-operators:latest
quay.io/openshifttest/prometheus-example-app:multiarch=MIRROR_REGISTRY_PLACEHOLDER/openshifttest/prometheus-example-app:multiarch
EOF
# Creating ICSP for quay.io/openshifttest is in enable-qe-catalogsource-disconnected step
# Set Node CA for Mirror Registry is in enable-qe-catalogsource-disconnected step
sed -i "s/MIRROR_REGISTRY_PLACEHOLDER/${MIRROR_REGISTRY_HOST}/g" "/tmp/mirror-images-list.yaml"
elif [ -f "${SHARED_DIR}/mirror-images-list.yaml" ]; then
echo "${SHARED_DIR}/mirror-images-list.yaml was found and will be used"
cp "${SHARED_DIR}/mirror-images-list.yaml" "/tmp/mirror-images-list.yaml"
# Format must be SRC=DEST
sed -i 's/ /=/g' "/tmp/mirror-images-list.yaml"
else
echo "No images to mirror."
exit 0
fi

# combine custom registry credential and default pull secret
registry_cred=`head -n 1 "/var/run/vault/mirror-registry/registry_creds" | base64 -w 0`
Expand All @@ -107,16 +120,23 @@ openshifttest_auth_password=$(cat "/var/run/vault/mirror-registry/registry_quay_
openshifttest_registry_auth=`echo -n "${openshifttest_auth_user}:${openshifttest_auth_password}" | base64 -w 0`
jq --argjson a "{\"${MIRROR_REGISTRY_HOST}\": {\"auth\": \"$registry_cred\"},\"quay.io/openshifttest\": {\"auth\": \"${openshifttest_registry_auth}\"}}" '.auths |= . + $a' "${CLUSTER_PROFILE_DIR}/pull-secret" > "${new_pull_secret}"

# MIRROR IMAGES
# Creating ICSP for quay.io/openshifttest is in enable-qe-catalogsource-disconnected step
# Set Node CA for Mirror Registry is in enable-qe-catalogsource-disconnected step
sed -i "s/MIRROR_REGISTRY_PLACEHOLDER/${MIRROR_REGISTRY_HOST}/g" "/tmp/mirror-images-list.yaml"
# For disconnected or otherwise unreachable environments, we want to
# have steps use an HTTP(S) proxy to reach the API server. This proxy
# configuration file should export HTTP_PROXY, HTTPS_PROXY, and NO_PROXY
# environment variables, as well as their lowercase equivalents (note
# that libcurl doesn't recognize the uppercase variables).
if test -f "${SHARED_DIR}/proxy-conf.sh"
Copy link
Contributor

@jianlinliu jianlinliu Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once enabled this block, all the mirror traffic would go though proxy to mirror registry, even mirror registry can be directly connected (especially in some QE's scenario on public clouds). Per my original understanding, "${SHARED_DIR}/proxy-conf.sh" is mainly for reaching cluster api endpoints which is not exposed to public.

mirror-images-qe-test-images step is a bit different with mirror-images-by-oc-adm, mirror-images-qe-test-images step generally is executed in post-installation, while mirror-images-by-oc-adm is executed in pre-installation.

In pre-installation phase, when mirror-images-by-oc-adm is being executed, ${SHARED_DIR}/proxy-conf.sh is not created yet in most of cases, so it seldom brings side effect for QE's ci jobs, while in post-installation phase, ${SHARED_DIR}/proxy-conf.sh is created, so all the traffic will go through proxy to mirror registry.

To avoid side effect as much as possible, I am thinking the following solutions for your reference:

  1. Introduce a separate env to decide source "${SHARED_DIR}/proxy-conf.sh" or not, we can set it on job config level as our demand.
  2. Create a new proxy config file with a different name, e.g: "${SHARED_DIR}/mirror-proxy-conf.sh" to indicated the file is only used to reach mirror registry behind a proxy.
  3. Based on the naming of mirror-images-qe-test-images, it was designed to mirror images for QE's test suite, or maybe it would be low cost to develop a new step.

then
# shellcheck disable=SC1090
source "${SHARED_DIR}/proxy-conf.sh"
fi

# To avoid 409 too many request error, mirroring image one by one
for image in `cat /tmp/mirror-images-list.yaml`
do
oc image mirror $image --insecure=true -a "${new_pull_secret}" \
--skip-missing=true --skip-verification=true --keep-manifest-list=true --filter-by-os='.*'
done
# MIRROR IMAGES
# To avoid 409 too many request error, mirroring image one by one
for image in `cat /tmp/mirror-images-list.yaml`
do
oc image mirror $image --insecure=true -a "${new_pull_secret}" \
--skip-missing=true --skip-verification=true --keep-manifest-list=true --filter-by-os='.*' --insecure=true
done

rm -f "${new_pull_secret}"
7 changes: 7 additions & 0 deletions ci-operator/step-registry/mirror-images/qe-test-images/mirror-images-qe-test-images-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ ref:
requests:
cpu: 10m
memory: 100Mi
env:
- name: QE_IMAGES
default: 'true'
documentation: |-
Whether or not mirror QE images. This can be disabled if someone wants
to pre-create "${SHARED_DIR}/mirror-images-list.yaml" and mirror
these images instead.
credentials:
- namespace: test-credentials
name: openshift-custom-mirror-registry
Expand Down