[BUG] Cannot pull image from local registry while running on Macos

[AyliD]

What did you do

• How was the cluster created?
  • k3d registry create local-registry --port 5050
  • k3d cluster create mycluster --registry-use k3d-local-registry:5050 --config cluster_config.yaml

apiVersion: k3d.io/v1alpha3
kind: Simple
name: salut
servers: 1
agents: 1
ports:
  - port: 8000:32080
    nodeFilters:
      - server:0:direct
  - port: 8443:32443
    nodeFilters:
      - server:0:direct
  - port: 9000:32090
    nodeFilters:
      - server:0:direct
  - port: 20017:30017
    nodeFilters:
      - server:0:direct
  - port: 20018:30018
    nodeFilters:
      - server:0:direct
registries:
  config: |
    mirrors:
      "registry.localhost:5000":
        endpoint:
          - http://host.k3d.internal:5000
options:
  k3d:
    wait: true
    timeout: "60s"
    disableLoadbalancer: true
    disableImageVolume: false
    disableRollback: true
  k3s:
    extraArgs:
      - arg: '--disable=traefik,servicelb'
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

• What did you do afterwards?
  • updated docker engine insecure registries:

 $ cat ~/.docker/daemon.json  | jq -r .
{
 "experimental": false,
 "debug": true,
 "insecure-registries": [
   "k3d-local-registry:5050",
   "host.k3d.internal:5050",
   "registry.localhost:5050"
 ],
 "registry-mirrors": [
   "https://registry-1.docker.io"
 ]
}

• docker push localhost:5050/nginx
• Was able to curl from some pod the registry via:

$ kubectl run curl --image=radial/busyboxplus:curl -i --tty --rm
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$ curl host.k3d.internal:5050/v2/_catalog
{"repositories":["nginx"]}

Wasnt able to start new pod using image from local registry:

kubectl run mynginx --image=host.k3d.internal:5050/nginx -i --tty --rm

Error:

  Normal   Pulling    13s   kubelet            Pulling image "host.k3d.internal:5050/nginx"
  Warning  Failed     13s   kubelet            Failed to pull image "host.k3d.internal:5050/nginx": rpc error: code = Unknown desc = failed to pull and unpack image "host.k3d.internal:5050/nginx:latest": failed to resolve reference "host.k3d.internal:5050/nginx:latest": failed to do request: Head "https://host.k3d.internal:5050/v2/nginx/manifests/latest": http: server gave HTTP response to HTTPS client

What did you expect to happen

Was hoping image will be pulled properly and pod will be running.

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture

• MacOS

Which version of k3d

• k3d version v5.3.0 k3s version v1.22.6-k3s1 (default)

Which version of docker

$ docker version
Client:
 Cloud integration: 1.0.14
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:57 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:56 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

 docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 22
  Running: 3
  Paused: 0
  Stopped: 19
 Images: 17
 Server Version: 20.10.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 15.64GiB
 Name: docker-desktop
 ID: NIYI:V2ON:M2DQ:Q6QK:D6MM:4Z5A:V5NP:IXAY:JLZG:JEFI:KND7:47J3
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 67
  Goroutines: 66
  System Time: 2022-02-16T15:31:47.200312Z
  EventsListeners: 4
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  localhost:5000
  localhost:5001
  registry.localhost:5000
  registry.localhost:5001
  registry.localhost:5050
  host.k3d.internal:5050
  k3d-local-registry:5050
  127.0.0.0/8
 Registry Mirrors:
  https://registry-1.docker.io/
 Live Restore Enabled: false

[iwilltry42]

Hi @AyliD , thanks for opening this issue!
So you're trying to run the pod with --image=host.k3d.internal:5050/nginx, but you never registered the registry under the name host.k3d.internal... During cluster creation, you configured --registry-use k3d-local-registry:5050 via flag and registry.localhost:5000 via configfile.

[AyliD]

All is now working with these setup:

$ cat  ~/.docker/daemon.json
{
  "experimental": false,
  "debug": true,
  "insecure-registries": [
    "host.k3d.internal:5050",
    "k3d-host.k3d.internal:5050",
  ],
  "registry-mirrors": [
    "https://registry-1.docker.io"
  ]
}

$ cat ./k3d/cluster_config.yaml
apiVersion: k3d.io/v1alpha3
kind: Simple
name: mycluster
servers: 1
agents: 1
ports:
  - port: 8000:32080
    nodeFilters:
      - server:0:direct
  - port: 8443:32443
    nodeFilters:
      - server:0:direct
  - port: 9000:32090
    nodeFilters:
      - server:0:direct
  - port: 20017:30017
    nodeFilters:
      - server:0:direct
  - port: 20018:30018
    nodeFilters:
      - server:0:direct
registries:
  config: |
    mirrors:
      "k3d-host.k3d.internal:5050":
        endpoint:
          - http://host.k3d.internal:5050
options:
  k3d:
    wait: true
    timeout: "60s"
    disableLoadbalancer: true
    disableImageVolume: false
    disableRollback: true
  k3s:
    extraArgs:
      - arg: '--disable=traefik,servicelb'
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

$ k3d registry create host.k3d.internal --port 5050
$ k3d cluster create mycluster --config ./k3d/cluster_config.yaml
$ kubectl run mynginx --image=k3d-host.k3d.internal:5050/nginx -i --tty --rm