Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-77515

[4.21] metal3-baremetal-operator pod not honoring NO_PROXY and routing metal3-state service traffic via proxy (works in 4.19.z)

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • In Progress
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      Summary:
      In OpenShift Container Platform (OCP) 4.20.z versions, the metal3-baremetal-operator pod is not honoring the configured NO_PROXY environment variable and is attempting to connect to the internal metal3-state service via the configured HTTP/HTTPS proxy. This behavior is not observed in OCP 4.19.z, where the operator correctly bypasses the proxy for internal service communication.

      Environment:

      • OCP Version: 4.20.z (multiple z-streams observed)
      • Working Version: 4.19.z
      • Platform: Baremetal (IPI)
      • Proxy configuration enabled (cluster-wide proxy configured)
      • Affected Pod: metal3-baremetal-operator
      • Target Service: metal3-state (ClusterIP service)

       

      Please find the reproducer details below. All cluster information is from my test environment and does not contain any sensitive data. I will also share the cluster must-gather shortly.

       

       

      > oc get clusterversion                            
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.14   True        False         6h8m    Cluster version is 4.20.14

> oc get proxy -oyaml  
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Proxy
  metadata:
    creationTimestamp: "2026-02-27T07:44:48Z"
    generation: 2
    name: cluster
    resourceVersion: "154487"
    uid: dc9dbe2b-645a-4b2b-98d1-41ab0ef7eebb
  spec:
    httpProxy: http://10.74.234.38:3128
    httpsProxy: http://10.74.234.38:3128
    noProxy: example.com
    trustedCA:
      name: ""
  status:
    httpProxy: http://10.74.234.38:3128
    httpsProxy: http://10.74.234.38:3128
    noProxy: .cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost
kind: List
metadata:
  resourceVersion: ""

> oc get infrastructure/cluster -oyaml 
apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2026-02-27T07:44:46Z"
  generation: 1
  name: cluster
  resourceVersion: "558"
  uid: 446c09a0-2b33-4ef1-bdf9-683d2cb00203
spec:
  cloudConfig:
    name: ""
  platformSpec:
    baremetal:
      apiServerInternalIPs:
      - 10.74.236.126
      ingressIPs:
      - 10.74.236.127
      machineNetworks:
      - 10.74.232.0/21
    type: BareMetal
status:
  apiServerInternalURI: https://api-int.ayush.example.com:6443
  apiServerURL: https://api.ayush.example.com:6443
  controlPlaneTopology: HighlyAvailable
  cpuPartitioning: None
  etcdDiscoveryDomain: ""
  infrastructureName: ayush-hvsdf
  infrastructureTopology: HighlyAvailable
  platform: BareMetal
  platformStatus:
    baremetal:
      apiServerInternalIP: 10.74.236.126
      apiServerInternalIPs:
      - 10.74.236.126
      ingressIP: 10.74.236.127
      ingressIPs:
      - 10.74.236.127
      loadBalancer:
        type: OpenShiftManagedDefault
      machineNetworks:
      - 10.74.232.0/21
    type: BareMetal

> oc get pod                           
NAME                                                  READY   STATUS    RESTARTS   AGE
cluster-autoscaler-operator-794cbb6fb-qfqmm           2/2     Running   0          14m
cluster-baremetal-operator-5d44678794-bvdk6           2/2     Running   0          14m
control-plane-machine-set-operator-675b5f7b45-xzxt8   1/1     Running   0          14m
ironic-proxy-b8km9                                    1/1     Running   0          14m
ironic-proxy-flvtt                                    1/1     Running   1          14m
ironic-proxy-vmbjm                                    1/1     Running   1          14m
machine-api-controllers-6687f5fd9f-5mj5t              7/7     Running   0          10m
machine-api-operator-bc77b9d7b-trm8j                  2/2     Running   0          14m
metal3-6b6ddc8cf9-nl6hn                               3/3     Running   0          14m
metal3-baremetal-operator-84f9846555-cqpfx            1/1     Running   0          10m
metal3-image-customization-b4847b9d6-7hlfz            1/1     Running   0          14m

> oc logs metal3-baremetal-operator-84f9846555-cqpfx | tail -n5                                
{"level":"info","ts":1772211615.0165863,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"master2.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}
{"level":"info","ts":1772211615.0166247,"logger":"provisioner.ironic","msg":"error caught while checking endpoint, will retry","host":"openshift-machine-api~compute1.ayush.example.com","endpoint":"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/","error":"Get \"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/\": Internal Server Error"}
{"level":"info","ts":1772211615.0166388,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"compute1.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}
{"level":"info","ts":1772211615.0168898,"logger":"provisioner.ironic","msg":"error caught while checking endpoint, will retry","host":"openshift-machine-api~master0.ayush.example.com","endpoint":"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/","error":"Get \"https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/\": Internal Server Error"}
{"level":"info","ts":1772211615.0169315,"logger":"controllers.BareMetalHost","msg":"provisioner is not ready","baremetalhost":{"name":"master0.ayush.example.com","namespace":"openshift-machine-api"},"Error":"Not ready","RequeueAfter":30}

> oc rsh metal3-baremetal-operator-84f9846555-cqpfx            
sh-5.1$ 
sh-5.1$ curl -kv https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/
* Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost,'
* Uses proxy env variable HTTPS_PROXY == 'http://10.74.234.38:3128'
*   Trying 10.74.234.38:3128...
* Connected to 10.74.234.38 (10.74.234.38) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to metal3-state.openshift-machine-api.svc.cluster.local:6388
> CONNECT metal3-state.openshift-machine-api.svc.cluster.local:6388 HTTP/1.1
> Host: metal3-state.openshift-machine-api.svc.cluster.local:6388
> User-Agent: curl/7.76.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 500 Internal Server Error
< Server: squid/5.5
< Mime-Version: 1.0
< Date: Fri, 27 Feb 2026 17:00:39 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3450
< X-Squid-Error: ERR_CANNOT_FORWARD 0
< Vary: Accept-Language
< Content-Language: en
< 
* Received HTTP code 500 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 500 from proxy after CONNECT
sh-5.1$ 
sh-5.1$ 
sh-5.1$  
sh-5.1$ 
sh-5.1$ unset HTTPS_PROXY
sh-5.1$ 
sh-5.1$ curl -kv https://metal3-state.openshift-machine-api.svc.cluster.local.:6388/v1/
* Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.128.0.0/14,10.74.232.0/21,127.0.0.1,172.30.0.0/16,api-int.ayush.example.com,example.com,localhost,'
*   Trying 172.30.114.184:6388...
* Connected to metal3-state.openshift-machine-api.svc.cluster.local (172.30.114.184) port 6388 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=localhost
*  start date: Feb 27 07:51:49 2026 GMT
*  expire date: Feb 27 07:51:50 2028 GMT
*  issuer: CN=metal3-ironic
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /v1/ HTTP/1.1
> Host: metal3-state.openshift-machine-api.svc.cluster.local:6388
> User-Agent: curl/7.76.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 27 Feb 2026 17:00:51 GMT
< Server: ironic_api
< X-OpenStack-Ironic-API-Minimum-Version: 1.1
< X-OpenStack-Ironic-API-Maximum-Version: 1.99
< X-OpenStack-Ironic-API-Version: 1.1
< Content-Length: 1029
< Content-Type: application/json
< Openstack-Request-Id: req-1d437637-879b-4af3-a8dd-9d7f9d11b172
< 
{"id": "v1", "links": [{"href": "https://10.74.236.31:6385/v1/", "rel": "self"}, {"href": "https://docs.openstack.org//ironic/latest/contributor//webapi.html", "rel": "describedby", "type": "text/html"}], "media_types": {"base": "application/json", "type": "application/vnd.openstack.ironic.v1+json"}, "chassis": [{"href": "https://10.74.236.31:6385/v1/chassis/", "rel": "self"}, {"href": "https://10.74.236.31:6385/chassis/", "rel": "bookmark"}], "nodes": [{"href": "https://10.74.236.31:6385/v1/nodes/", "rel": "self"}, {"href": "https://10.74.236.31:6385/nodes/", "rel": "bookmark"}], "ports": [{"href": "https://10.74.236.31:6385/v1/ports/", "rel": "self"}, {"href": "https://10.74.236.31:6385/ports/", "rel": "bookmark"}], "drivers": [{"href": "https://10.74.236.31:6385/v1/drivers/", "rel": "self"}, {"href": "https://10.74.236.31:6385/drivers/", "rel": "bookmark"}], "version": {"id": "v1", "links": [{"href": "https://10.74.236.31:6385/v1/", "rel": "self"}], "status": "CURRENT", "min_version": "1.1", "version": "1.* Connection #0 to host metal3-state.openshift-machine-api.svc.cluster.local left intact

      We can ignore the curl behavior for now, as this is expected. Curl does not consistently honor the no_proxy variables in certain scenarios. The output is shared solely to illustrate that the same response is observed when accessing the service via the proxy as when the pod process connects to the service.

       

              eterrell@redhat.com Eduardo Otubo
              rhn-support-aygarg Ayush Garg
              Jad Haj Yahya Jad Haj Yahya
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: